2024-02-09T22:23:50.164 INFO:root:teuthology version: 0.0.1.dev256+gd9fdb22 2024-02-09T22:23:50.164 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417 --owner scheduled_yuriw@teuthology --verbose --name yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} -- /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/orig.config.yaml 2024-02-09T22:23:50.223 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T22:23:50.308 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417 branch: main description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} email: yweinste@redhat.com first_in_suite: false job_id: '7554417' kernel: client: sha1: distro kdb: true sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - slow metadata IO - SLOW_OPS - slow request sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount install: ceph: flavor: default sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - wsync syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3558 sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi003.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnknuxCHqeCcyv4s+OGlpe8Tl9GA6RYMCPgoFCu2QyippZdl+Pw2d3Z7h3vDVnCLx16H2LezgTBUCACXEaIHXM= smithi064.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP68kaqh9+DKptpT9uZPbsRZtpU1kknaEztXtPYckqc9jLQfCK/fjAsQNuqBAKrc+7xJ2N2S2q2W9w981cSuuDQ= smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPVzyAieWIz1DOzU0iw73w3FWDXUNw4nVIkzpIOvG9scdUtcs9DRg0ScH2fxg/OUY1rZsEKXYdtwNIyvi0lgIf4= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: d9fdb2209e15b39d9f061fd85399f352ce0f0894 timestamp: 2024-02-09_19:35:09 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-02-09T22:23:50.308 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa; will attempt to use it 2024-02-09T22:23:50.309 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks 2024-02-09T22:23:50.309 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-02-09T22:23:50.311 INFO:teuthology.task.internal:Checking packages... 2024-02-09T22:23:50.334 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90' 2024-02-09T22:23:50.334 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-02-09T22:23:50.334 INFO:teuthology.packaging:ref: None 2024-02-09T22:23:50.334 INFO:teuthology.packaging:tag: None 2024-02-09T22:23:50.334 INFO:teuthology.packaging:branch: main 2024-02-09T22:23:50.334 INFO:teuthology.packaging:sha1: 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:23:50.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-02-09T22:23:50.576 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1361.gf8e4e331 2024-02-09T22:23:50.577 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-02-09T22:23:50.584 INFO:teuthology.task.internal:no buildpackages task found 2024-02-09T22:23:50.584 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-02-09T22:23:50.600 INFO:teuthology.task.internal:Saving configuration 2024-02-09T22:23:50.620 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-02-09T22:23:50.627 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T22:23:50.652 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 22:18:55.584249', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnknuxCHqeCcyv4s+OGlpe8Tl9GA6RYMCPgoFCu2QyippZdl+Pw2d3Z7h3vDVnCLx16H2LezgTBUCACXEaIHXM='} 2024-02-09T22:23:50.675 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 22:18:55.583139', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP68kaqh9+DKptpT9uZPbsRZtpU1kknaEztXtPYckqc9jLQfCK/fjAsQNuqBAKrc+7xJ2N2S2q2W9w981cSuuDQ='} 2024-02-09T22:23:50.697 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 22:18:55.585176', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPVzyAieWIz1DOzU0iw73w3FWDXUNw4nVIkzpIOvG9scdUtcs9DRg0ScH2fxg/OUY1rZsEKXYdtwNIyvi0lgIf4='} 2024-02-09T22:23:50.697 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-02-09T22:23:50.704 INFO:teuthology.task.internal:roles: ubuntu@smithi003.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-02-09T22:23:50.704 INFO:teuthology.task.internal:roles: ubuntu@smithi064.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-02-09T22:23:50.704 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-02-09T22:23:50.704 INFO:teuthology.run_tasks:Running task console_log... 2024-02-09T22:23:50.811 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f78c25ec160>, signals=[15]) 2024-02-09T22:23:50.811 INFO:teuthology.run_tasks:Running task internal.connect... 2024-02-09T22:23:50.818 INFO:teuthology.task.internal:Opening connections... 2024-02-09T22:23:50.818 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T22:23:50.820 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:23:50.926 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi064.front.sepia.ceph.com 2024-02-09T22:23:50.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:23:51.030 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-02-09T22:23:51.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:23:51.121 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-02-09T22:23:51.127 DEBUG:teuthology.orchestra.run.smithi003:> uname -m 2024-02-09T22:23:51.145 INFO:teuthology.orchestra.run.smithi003.stdout:x86_64 2024-02-09T22:23:51.146 DEBUG:teuthology.orchestra.run.smithi003:> cat /etc/os-release 2024-02-09T22:23:51.203 INFO:teuthology.orchestra.run.smithi003.stdout:NAME="CentOS Stream" 2024-02-09T22:23:51.203 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION="9" 2024-02-09T22:23:51.203 INFO:teuthology.orchestra.run.smithi003.stdout:ID="centos" 2024-02-09T22:23:51.203 INFO:teuthology.orchestra.run.smithi003.stdout:ID_LIKE="rhel fedora" 2024-02-09T22:23:51.203 INFO:teuthology.orchestra.run.smithi003.stdout:VERSION_ID="9" 2024-02-09T22:23:51.203 INFO:teuthology.orchestra.run.smithi003.stdout:PLATFORM_ID="platform:el9" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:ANSI_COLOR="0;31" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:LOGO="fedora-logo-icon" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:HOME_URL="https://centos.org/" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-09T22:23:51.204 INFO:teuthology.orchestra.run.smithi003.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T22:23:51.205 INFO:teuthology.lock.ops:Updating smithi003.front.sepia.ceph.com on lock server 2024-02-09T22:23:51.227 DEBUG:teuthology.orchestra.run.smithi064:> uname -m 2024-02-09T22:23:51.245 INFO:teuthology.orchestra.run.smithi064.stdout:x86_64 2024-02-09T22:23:51.245 DEBUG:teuthology.orchestra.run.smithi064:> cat /etc/os-release 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:NAME="CentOS Stream" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION="9" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:ID="centos" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:ID_LIKE="rhel fedora" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:VERSION_ID="9" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:PLATFORM_ID="platform:el9" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:ANSI_COLOR="0;31" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:LOGO="fedora-logo-icon" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-09T22:23:51.303 INFO:teuthology.orchestra.run.smithi064.stdout:HOME_URL="https://centos.org/" 2024-02-09T22:23:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T22:23:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-09T22:23:51.304 INFO:teuthology.orchestra.run.smithi064.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T22:23:51.304 INFO:teuthology.lock.ops:Updating smithi064.front.sepia.ceph.com on lock server 2024-02-09T22:23:51.327 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-02-09T22:23:51.345 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-02-09T22:23:51.345 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-02-09T22:23:51.402 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-02-09T22:23:51.403 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-02-09T22:23:51.403 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-02-09T22:23:51.403 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-02-09T22:23:51.403 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-02-09T22:23:51.403 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-02-09T22:23:51.403 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-02-09T22:23:51.426 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-02-09T22:23:51.437 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-02-09T22:23:51.442 INFO:teuthology.task.internal:Checking for old test directory... 2024-02-09T22:23:51.442 DEBUG:teuthology.orchestra.run.smithi003:> test '!' -e /home/ubuntu/cephtest 2024-02-09T22:23:51.444 DEBUG:teuthology.orchestra.run.smithi064:> test '!' -e /home/ubuntu/cephtest 2024-02-09T22:23:51.446 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-02-09T22:23:51.461 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-02-09T22:23:51.467 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-02-09T22:23:51.467 DEBUG:teuthology.orchestra.run.smithi003:> test -z $(ls -A /var/lib/ceph) 2024-02-09T22:23:51.502 DEBUG:teuthology.orchestra.run.smithi064:> test -z $(ls -A /var/lib/ceph) 2024-02-09T22:23:51.504 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-02-09T22:23:51.609 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-02-09T22:23:51.685 INFO:teuthology.run_tasks:Running task kernel... 2024-02-09T22:23:51.700 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2024-02-09T22:23:51.700 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-02-09T22:23:51.700 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-02-09T22:23:51.700 DEBUG:teuthology.orchestra.run.smithi003:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T22:23:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:23:51.718 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-02-09T22:23:51.777 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-361.el9.x86_64 2024-02-09T22:23:51.777 INFO:teuthology.task.kernel:Running kernel on smithi003: 5.14.0-361.el9.x86_64 2024-02-09T22:23:51.777 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum install -y kernel 2024-02-09T22:23:54.301 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 7.7 MB/s | 6.7 MB 00:00 2024-02-09T22:23:55.876 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2024-02-09T22:23:57.666 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.0 MB 00:01 2024-02-09T22:24:12.749 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 1.3 MB/s | 19 MB 00:13 2024-02-09T22:24:17.117 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 5.9 MB 00:00 2024-02-09T22:24:18.592 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 53 kB/s | 15 kB 00:00 2024-02-09T22:24:21.879 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 20 MB 00:03 2024-02-09T22:24:26.960 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 49 kB/s | 1.7 kB 00:00 2024-02-09T22:24:29.376 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-02-09T22:24:29.376 INFO:teuthology.orchestra.run.smithi003.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-02-09T22:24:29.474 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-02-09T22:24:29.482 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: kernel x86_64 5.14.0-419.el9 CentOS-BaseOS 6.0 M 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core x86_64 5.14.0-419.el9 CentOS-BaseOS 21 M 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules x86_64 5.14.0-419.el9 CentOS-BaseOS 39 M 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core x86_64 5.14.0-419.el9 CentOS-BaseOS 33 M 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:================================================================================ 2024-02-09T22:24:29.483 INFO:teuthology.orchestra.run.smithi003.stdout:Install 4 Packages 2024-02-09T22:24:29.484 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:24:29.484 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 99 M 2024-02-09T22:24:29.484 INFO:teuthology.orchestra.run.smithi003.stdout:Installed size: 122 M 2024-02-09T22:24:29.484 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-02-09T22:24:30.045 INFO:teuthology.orchestra.run.smithi003.stdout:(1/4): kernel-5.14.0-419.el9.x86_64.rpm 11 MB/s | 6.0 MB 00:00 2024-02-09T22:24:30.479 INFO:teuthology.orchestra.run.smithi003.stdout:(2/4): kernel-core-5.14.0-419.el9.x86_64.rpm 21 MB/s | 21 MB 00:00 2024-02-09T22:24:31.112 INFO:teuthology.orchestra.run.smithi003.stdout:(3/4): kernel-modules-5.14.0-419.el9.x86_64.rpm 24 MB/s | 39 MB 00:01 2024-02-09T22:24:31.637 INFO:teuthology.orchestra.run.smithi003.stdout:(4/4): kernel-modules-core-5.14.0-419.el9.x86_6 21 MB/s | 33 MB 00:01 2024-02-09T22:24:31.640 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:24:31.641 INFO:teuthology.orchestra.run.smithi003.stdout:Total 46 MB/s | 99 MB 00:02 2024-02-09T22:24:31.641 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-02-09T22:24:32.096 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-02-09T22:24:32.097 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-02-09T22:24:33.147 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-02-09T22:24:33.147 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-02-09T22:24:35.672 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-02-09T22:24:36.341 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-core-5.14.0-419.el9.x86_64 1/4 2024-02-09T22:24:36.473 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T22:24:39.356 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T22:24:39.532 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T22:24:44.588 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T22:24:44.696 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:24:49.779 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:25:20.333 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:25:20.333 INFO:teuthology.orchestra.run.smithi003.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-02-09T22:25:20.333 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:25:20.403 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-modules-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:25:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: kernel-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:25:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-5.14.0-419.el9.x86_64 1/4 2024-02-09T22:25:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-core-5.14.0-419.el9.x86_64 2/4 2024-02-09T22:25:20.819 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-5.14.0-419.el9.x86_64 3/4 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : kernel-modules-core-5.14.0-419.el9.x86_64 4/4 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-5.14.0-419.el9.x86_64 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-core-5.14.0-419.el9.x86_64 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-5.14.0-419.el9.x86_64 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: kernel-modules-core-5.14.0-419.el9.x86_64 2024-02-09T22:25:21.270 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:25:21.271 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-02-09T22:25:21.809 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T22:25:21.939 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T22:25:21.940 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T22:25:21.940 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-419.el9.x86_64 2024-02-09T22:25:21.940 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-02-09T22:25:21.940 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-02-09T22:25:21.940 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-02-09T22:25:21.940 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi003.front.sepia.ceph.com, path=None, version=distro) 2024-02-09T22:25:21.940 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q kernel | sort -rV | head -n 1 2024-02-09T22:25:22.052 INFO:teuthology.orchestra.run.smithi003.stdout:kernel-5.14.0-419.el9.x86_64 2024-02-09T22:25:22.053 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-419.el9.x86_64 2024-02-09T22:25:22.053 DEBUG:teuthology.orchestra.run.smithi003:> sudo rpm -qi grub2-tools 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Name : grub2-tools 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Epoch : 1 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Version : 2.06 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Release : 68.el9 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Architecture: x86_64 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Group : Unspecified 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Size : 8264254 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:License : GPLv3+ 2024-02-09T22:25:22.097 INFO:teuthology.orchestra.run.smithi003.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Packager : builder@centos.org 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Vendor : CentOS 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:URL : http://www.gnu.org/software/grub/ 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Summary : Support tools for GRUB. 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:Description : 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:customizable bootloader with modular architecture. It supports a rich 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:variety of kernel formats, file systems, computer architectures and 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout:hardware devices. 2024-02-09T22:25:22.098 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:25:22.099 INFO:teuthology.orchestra.run.smithi003.stdout:This subpackage provides tools for support of all platforms. 2024-02-09T22:25:22.099 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-02-09T22:25:22.100 INFO:teuthology.task.kernel:Updating grub on smithi003 to boot 5.14.0-419.el9.x86_64 2024-02-09T22:25:22.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-02-09T22:25:22.753 INFO:teuthology.orchestra.run.smithi003.stderr:Generating grub configuration file ... 2024-02-09T22:25:24.135 INFO:teuthology.orchestra.run.smithi003.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-02-09T22:25:24.168 INFO:teuthology.orchestra.run.smithi003.stderr:done 2024-02-09T22:25:24.171 DEBUG:teuthology.orchestra.run.smithi003:> mktemp 2024-02-09T22:25:24.186 INFO:teuthology.orchestra.run.smithi003.stdout:/tmp/tmp.ldPsvSxRpi 2024-02-09T22:25:24.187 DEBUG:teuthology.orchestra.run.smithi003:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ldPsvSxRpi 2024-02-09T22:25:24.260 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0666 /tmp/tmp.ldPsvSxRpi 2024-02-09T22:25:24.406 DEBUG:teuthology.orchestra.remote:smithi003:/tmp/tmp.ldPsvSxRpi is 6KB 2024-02-09T22:25:24.455 DEBUG:teuthology.orchestra.run.smithi003:> rm -fr /tmp/tmp.ldPsvSxRpi 2024-02-09T22:25:24.469 DEBUG:teuthology.orchestra.run.smithi003:> sudo /bin/ls /boot/loader/entries || true 2024-02-09T22:25:24.547 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-02-09T22:25:24.547 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-02-09T22:25:24.547 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-02-09T22:25:24.547 INFO:teuthology.orchestra.run.smithi003.stdout:15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64.conf 2024-02-09T22:25:24.548 DEBUG:teuthology.orchestra.run.smithi003:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-419.el9.x86_64 2024-02-09T22:25:24.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo shutdown -r now 2024-02-09T22:25:54.708 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-02-09T22:25:54.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-02-09T22:25:54.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:26:13.180 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.3 2024-02-09T22:26:22.181 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-02-09T22:26:22.182 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:27:22.184 DEBUG:teuthology.orchestra.remote:timed out 2024-02-09T22:27:34.186 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-02-09T22:27:34.187 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:27:34.534 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-02-09T22:27:34.917 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-02-09T22:27:34.918 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-419.el9.x86_64"... 2024-02-09T22:27:34.918 DEBUG:teuthology.orchestra.run.smithi003:> uname -r 2024-02-09T22:27:34.933 INFO:teuthology.orchestra.run.smithi003.stdout:5.14.0-419.el9.x86_64 2024-02-09T22:27:34.933 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-419.el9.x86_64 vs 5.14.0-419.el9.x86_64 2024-02-09T22:27:34.933 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-02-09T22:27:34.933 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-02-09T22:27:35.934 DEBUG:teuthology.parallel:result is None 2024-02-09T22:27:35.935 INFO:teuthology.run_tasks:Running task internal.base... 2024-02-09T22:27:35.943 INFO:teuthology.task.internal:Creating test directory... 2024-02-09T22:27:35.943 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T22:27:35.946 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T22:27:35.948 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-02-09T22:27:35.966 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-02-09T22:27:35.972 INFO:teuthology.run_tasks:Running task internal.archive... 2024-02-09T22:27:35.981 INFO:teuthology.task.internal:Creating archive directory... 2024-02-09T22:27:35.981 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T22:27:36.003 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T22:27:36.007 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-02-09T22:27:36.043 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-02-09T22:27:36.055 INFO:teuthology.task.internal:Enabling coredump saving... 2024-02-09T22:27:36.055 DEBUG:teuthology.orchestra.run.smithi064:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T22:27:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:27:36.086 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-02-09T22:27:36.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:27:36.102 DEBUG:teuthology.orchestra.run.smithi003:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T22:27:36.105 DEBUG:teuthology.orchestra.run.smithi064:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T22:27:36.129 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-02-09T22:27:36.170 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:27:36.182 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:27:36.188 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:27:36.191 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:27:36.216 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:27:36.224 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-02-09T22:27:36.225 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-02-09T22:27:36.232 INFO:teuthology.task.internal:Configuring sudo... 2024-02-09T22:27:36.233 DEBUG:teuthology.orchestra.run.smithi003:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T22:27:36.235 DEBUG:teuthology.orchestra.run.smithi064:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T22:27:36.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-02-09T22:27:36.287 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-02-09T22:27:36.296 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-02-09T22:27:36.296 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T22:27:36.306 DEBUG:teuthology.orchestra.run.smithi064:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T22:27:36.310 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-02-09T22:27:36.343 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:27:36.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:27:36.455 DEBUG:teuthology.orchestra.run.smithi003:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:27:36.541 DEBUG:teuthology.orchestra.run.smithi003:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:27:36.581 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:27:36.581 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T22:27:36.648 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:27:36.695 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:27:36.745 DEBUG:teuthology.orchestra.run.smithi064:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:27:36.827 DEBUG:teuthology.orchestra.run.smithi064:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:27:36.874 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:27:36.875 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T22:27:36.946 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:27:36.992 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-02-09T22:27:37.046 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:27:37.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-02-09T22:27:37.176 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:27:37.176 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-02-09T22:27:37.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo service rsyslog restart 2024-02-09T22:27:37.247 DEBUG:teuthology.orchestra.run.smithi064:> sudo service rsyslog restart 2024-02-09T22:27:37.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-02-09T22:27:37.295 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T22:27:37.299 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T22:27:37.334 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T22:27:37.812 INFO:teuthology.run_tasks:Running task internal.timer... 2024-02-09T22:27:37.821 INFO:teuthology.task.internal:Starting timer... 2024-02-09T22:27:37.821 INFO:teuthology.run_tasks:Running task pcp... 2024-02-09T22:27:37.833 INFO:teuthology.run_tasks:Running task selinux... 2024-02-09T22:27:37.843 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-02-09T22:27:37.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo service auditd rotate 2024-02-09T22:27:37.904 INFO:teuthology.orchestra.run.smithi003.stdout:Rotating logs: 2024-02-09T22:27:37.906 DEBUG:teuthology.orchestra.run.smithi064:> sudo service auditd rotate 2024-02-09T22:27:37.977 INFO:teuthology.orchestra.run.smithi064.stdout:Rotating logs: 2024-02-09T22:27:37.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-02-09T22:27:38.029 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-02-09T22:27:38.031 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-02-09T22:27:38.031 DEBUG:teuthology.orchestra.run.smithi003:> /usr/sbin/getenforce 2024-02-09T22:27:38.053 INFO:teuthology.orchestra.run.smithi003.stdout:Permissive 2024-02-09T22:27:38.053 DEBUG:teuthology.orchestra.run.smithi064:> /usr/sbin/getenforce 2024-02-09T22:27:38.074 INFO:teuthology.orchestra.run.smithi064.stdout:Permissive 2024-02-09T22:27:38.074 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-02-09T22:27:38.093 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-02-09T22:27:38.094 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi003.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi064.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi089.front.sepia.ceph.com': 'permissive'} 2024-02-09T22:27:38.094 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T22:27:38.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:27:38.124 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T22:27:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:27:38.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T22:27:38.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:27:38.183 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-02-09T22:27:38.183 DEBUG:teuthology.orchestra.run.smithi003:> sudo /usr/sbin/setenforce permissive 2024-02-09T22:27:38.213 DEBUG:teuthology.orchestra.run.smithi064:> sudo /usr/sbin/setenforce permissive 2024-02-09T22:27:38.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-02-09T22:27:38.287 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-02-09T22:27:38.297 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-02-09T22:27:38.314 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-02-09T22:27:38.316 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi003.front.sepia.ceph.com,smithi064.front.sepia.ceph.com,smithi089.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-02-09T22:32:26.373 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi003.front.sepia.ceph.com'), Remote(name='ubuntu@smithi064.front.sepia.ceph.com'), Remote(name='ubuntu@smithi089.front.sepia.ceph.com')] 2024-02-09T22:32:26.374 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-02-09T22:32:26.375 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi003.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:32:26.464 DEBUG:teuthology.orchestra.run.smithi003:> true 2024-02-09T22:32:26.538 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi003.front.sepia.ceph.com' 2024-02-09T22:32:26.539 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-09T22:32:26.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi064.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:32:26.639 DEBUG:teuthology.orchestra.run.smithi064:> true 2024-02-09T22:32:26.708 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi064.front.sepia.ceph.com' 2024-02-09T22:32:26.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-09T22:32:26.708 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-02-09T22:32:26.799 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-02-09T22:32:26.867 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-02-09T22:32:26.867 INFO:teuthology.run_tasks:Running task clock... 2024-02-09T22:32:26.878 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-02-09T22:32:26.878 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T22:32:26.879 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T22:32:26.881 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T22:32:26.881 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T22:32:26.883 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-02-09T22:32:26.884 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T22:32:26.911 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T22:32:26.918 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T22:32:26.928 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T22:32:26.940 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T22:32:26.940 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-02-09T22:32:26.956 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-02-09T22:32:26.959 INFO:teuthology.orchestra.run.smithi003.stderr:sudo: ntpd: command not found 2024-02-09T22:32:26.973 INFO:teuthology.orchestra.run.smithi003.stdout:506 Cannot talk to daemon 2024-02-09T22:32:26.976 INFO:teuthology.orchestra.run.smithi064.stderr:sudo: ntpd: command not found 2024-02-09T22:32:26.986 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-02-09T22:32:26.989 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T22:32:26.994 INFO:teuthology.orchestra.run.smithi064.stdout:506 Cannot talk to daemon 2024-02-09T22:32:26.999 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-02-09T22:32:27.007 INFO:teuthology.orchestra.run.smithi003.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T22:32:27.014 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T22:32:27.016 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-02-09T22:32:27.032 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T22:32:27.034 INFO:teuthology.orchestra.run.smithi064.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-02-09T22:32:27.056 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-02-09T22:32:27.059 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T22:32:27.060 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-02-09T22:32:27.060 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.060 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.060 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.060 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.076 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-02-09T22:32:27.079 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T22:32:27.079 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-02-09T22:32:27.079 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.079 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.079 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.079 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.082 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-02-09T22:32:27.085 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T22:32:27.086 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-02-09T22:32:27.086 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.086 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.086 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.086 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T22:32:27.086 INFO:teuthology.run_tasks:Running task install... 2024-02-09T22:32:27.094 DEBUG:teuthology.task.install:project ceph 2024-02-09T22:32:27.095 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'}} 2024-02-09T22:32:27.095 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'} 2024-02-09T22:32:27.095 INFO:teuthology.task.install:Using flavor: default 2024-02-09T22:32:27.102 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-02-09T22:32:27.102 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-02-09T22:32:27.103 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-09T22:32:27.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:32:27.105 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-09T22:32:27.105 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:32:27.106 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'tag': None, 'wait_for_package': False} 2024-02-09T22:32:27.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:32:27.261 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-09T22:32:27.261 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-09T22:32:27.269 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-09T22:32:27.270 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-09T22:32:27.274 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/ 2024-02-09T22:32:27.275 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1361.g290212a1 2024-02-09T22:32:27.503 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T22:32:27.503 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:32:27.503 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T22:32:27.505 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T22:32:27.506 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:32:27.506 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T22:32:27.535 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-09T22:32:27.535 DEBUG:teuthology.orchestra.run.smithi003:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T22:32:27.537 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-02-09T22:32:27.538 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:32:27.538 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-02-09T22:32:27.539 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-09T22:32:27.539 DEBUG:teuthology.orchestra.run.smithi064:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T22:32:27.566 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-02-09T22:32:27.567 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-02-09T22:32:27.603 DEBUG:teuthology.orchestra.run.smithi003:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T22:32:27.610 DEBUG:teuthology.orchestra.run.smithi064:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T22:32:27.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-02-09T22:32:27.683 DEBUG:teuthology.orchestra.run.smithi003:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T22:32:27.698 DEBUG:teuthology.orchestra.run.smithi064:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T22:32:27.716 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-02-09T22:32:27.728 INFO:teuthology.orchestra.run.smithi064.stdout:check_obsoletes = 1 2024-02-09T22:32:27.729 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum clean all 2024-02-09T22:32:27.749 INFO:teuthology.orchestra.run.smithi003.stdout:check_obsoletes = 1 2024-02-09T22:32:27.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum clean all 2024-02-09T22:32:27.782 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-02-09T22:32:27.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-02-09T22:32:27.975 INFO:teuthology.orchestra.run.smithi064.stdout:74 files removed 2024-02-09T22:32:27.996 INFO:teuthology.orchestra.run.smithi003.stdout:74 files removed 2024-02-09T22:32:28.003 DEBUG:teuthology.orchestra.run.smithi064:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-09T22:32:28.017 INFO:teuthology.orchestra.run.smithi089.stdout:74 files removed 2024-02-09T22:32:28.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-09T22:32:28.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-02-09T22:32:28.673 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 240 kB/s | 82 kB 00:00 2024-02-09T22:32:28.692 INFO:teuthology.orchestra.run.smithi064.stdout:ceph packages for x86_64 216 kB/s | 82 kB 00:00 2024-02-09T22:32:28.715 INFO:teuthology.orchestra.run.smithi003.stdout:ceph packages for x86_64 223 kB/s | 82 kB 00:00 2024-02-09T22:32:29.090 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-02-09T22:32:29.109 INFO:teuthology.orchestra.run.smithi064.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-02-09T22:32:29.157 INFO:teuthology.orchestra.run.smithi003.stdout:ceph noarch packages 26 kB/s | 10 kB 00:00 2024-02-09T22:32:29.440 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 5.9 kB/s | 1.9 kB 00:00 2024-02-09T22:32:29.443 INFO:teuthology.orchestra.run.smithi064.stdout:ceph source packages 6.3 kB/s | 1.9 kB 00:00 2024-02-09T22:32:29.524 INFO:teuthology.orchestra.run.smithi003.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-02-09T22:32:29.870 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-02-09T22:32:29.935 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-02-09T22:32:30.201 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-02-09T22:32:31.315 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS-9 - BaseOS 7.0 MB/s | 2.0 MB 00:00 2024-02-09T22:32:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-02-09T22:32:31.677 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-02-09T22:32:32.678 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - BaseOS 7.8 MB/s | 8.0 MB 00:01 2024-02-09T22:32:33.901 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 3.6 MB/s | 8.0 MB 00:02 2024-02-09T22:32:33.934 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - BaseOS 4.2 MB/s | 8.0 MB 00:01 2024-02-09T22:32:34.571 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-02-09T22:32:40.647 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - CRB 2.8 MB/s | 5.9 MB 00:02 2024-02-09T22:32:42.397 INFO:teuthology.orchestra.run.smithi064.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 15 kB 00:00 2024-02-09T22:33:00.658 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - AppStream 750 kB/s | 19 MB 00:25 2024-02-09T22:33:05.058 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - CRB 7.7 MB/s | 5.9 MB 00:00 2024-02-09T22:33:06.409 INFO:teuthology.orchestra.run.smithi003.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 15 kB 00:00 2024-02-09T22:33:11.055 INFO:teuthology.orchestra.run.smithi003.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 20 MB 00:04 2024-02-09T22:33:16.152 INFO:teuthology.orchestra.run.smithi003.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-02-09T22:33:18.872 INFO:teuthology.orchestra.run.smithi003.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-09T22:33:18.873 INFO:teuthology.orchestra.run.smithi003.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-09T22:33:18.885 INFO:teuthology.orchestra.run.smithi003.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-09T22:33:18.886 INFO:teuthology.orchestra.run.smithi003.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-09T22:33:18.887 INFO:teuthology.orchestra.run.smithi003.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-09T22:33:18.888 INFO:teuthology.orchestra.run.smithi003.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-09T22:33:18.889 INFO:teuthology.orchestra.run.smithi003.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-09T22:33:18.892 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-09T22:33:18.893 INFO:teuthology.orchestra.run.smithi003.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-09T22:33:18.979 INFO:teuthology.orchestra.run.smithi003.stdout:Dependencies resolved. 2024-02-09T22:33:19.004 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-02-09T22:33:19.004 INFO:teuthology.orchestra.run.smithi003.stdout: Package Arch Version Repository Size 2024-02-09T22:33:19.004 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-02-09T22:33:19.004 INFO:teuthology.orchestra.run.smithi003.stdout:Installing: 2024-02-09T22:33:19.004 INFO:teuthology.orchestra.run.smithi003.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-02-09T22:33:19.004 INFO:teuthology.orchestra.run.smithi003.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-09T22:33:19.005 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 217 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-shell x86_64 2:19.0.0-1361.g290212a1.el9 ceph 27 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 18 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-09T22:33:19.006 INFO:teuthology.orchestra.run.smithi003.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-09T22:33:19.007 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrading: 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager x86_64 1:1.45.90-1.el9 CentOS-BaseOS 2.3 M 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-libnm x86_64 1:1.45.90-1.el9 CentOS-BaseOS 1.8 M 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-team x86_64 1:1.45.90-1.el9 CentOS-BaseOS 42 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-tui x86_64 1:1.45.90-1.el9 CentOS-BaseOS 247 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-02-09T22:33:19.008 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-02-09T22:33:19.009 INFO:teuthology.orchestra.run.smithi003.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout:Installing dependencies: 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-02-09T22:33:19.010 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-09T22:33:19.011 INFO:teuthology.orchestra.run.smithi003.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-09T22:33:19.012 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-09T22:33:19.013 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-02-09T22:33:19.014 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-09T22:33:19.015 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:33:19.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-09T22:33:19.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-09T22:33:19.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-09T22:33:19.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-09T22:33:19.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-09T22:33:19.016 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-09T22:33:19.017 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-09T22:33:19.018 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-09T22:33:19.019 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-09T22:33:19.020 INFO:teuthology.orchestra.run.smithi003.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout:Installing weak dependencies: 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction Summary 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout:============================================================================================ 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout:Install 150 Packages 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout:Upgrade 26 Packages 2024-02-09T22:33:19.021 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:33:19.022 INFO:teuthology.orchestra.run.smithi003.stdout:Total download size: 222 M 2024-02-09T22:33:19.022 INFO:teuthology.orchestra.run.smithi003.stdout:Downloading Packages: 2024-02-09T22:33:19.362 INFO:teuthology.orchestra.run.smithi003.stdout:(1/176): ceph-19.0.0-1361.g290212a1.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-02-09T22:33:19.596 INFO:teuthology.orchestra.run.smithi003.stdout:(2/176): ceph-fuse-19.0.0-1361.g290212a1.el9.x8 3.7 MB/s | 885 kB 00:00 2024-02-09T22:33:19.679 INFO:teuthology.orchestra.run.smithi003.stdout:(3/176): ceph-immutable-object-cache-19.0.0-136 1.7 MB/s | 145 kB 00:00 2024-02-09T22:33:19.788 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 428 kB/s | 19 MB 00:44 2024-02-09T22:33:19.863 INFO:teuthology.orchestra.run.smithi003.stdout:(4/176): ceph-base-19.0.0-1361.g290212a1.el9.x8 7.9 MB/s | 5.3 MB 00:00 2024-02-09T22:33:19.963 INFO:teuthology.orchestra.run.smithi003.stdout:(5/176): ceph-mds-19.0.0-1361.g290212a1.el9.x86 7.5 MB/s | 2.1 MB 00:00 2024-02-09T22:33:20.022 INFO:teuthology.orchestra.run.smithi003.stdout:(6/176): ceph-mgr-19.0.0-1361.g290212a1.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-02-09T22:33:20.622 INFO:teuthology.orchestra.run.smithi003.stdout:(7/176): ceph-mon-19.0.0-1361.g290212a1.el9.x86 7.1 MB/s | 4.7 MB 00:00 2024-02-09T22:33:21.280 INFO:teuthology.orchestra.run.smithi003.stdout:(8/176): ceph-common-19.0.0-1361.g290212a1.el9. 9.5 MB/s | 20 MB 00:02 2024-02-09T22:33:21.339 INFO:teuthology.orchestra.run.smithi003.stdout:(9/176): ceph-selinux-19.0.0-1361.g290212a1.el9 432 kB/s | 25 kB 00:00 2024-02-09T22:33:21.922 INFO:teuthology.orchestra.run.smithi003.stdout:(10/176): ceph-osd-19.0.0-1361.g290212a1.el9.x8 8.8 MB/s | 17 MB 00:01 2024-02-09T22:33:22.064 INFO:teuthology.orchestra.run.smithi003.stdout:(11/176): ceph-radosgw-19.0.0-1361.g290212a1.el 6.8 MB/s | 9.7 MB 00:01 2024-02-09T22:33:22.090 INFO:teuthology.orchestra.run.smithi003.stdout:(12/176): cephfs-mirror-19.0.0-1361.g290212a1.e 1.3 MB/s | 217 kB 00:00 2024-02-09T22:33:22.123 INFO:teuthology.orchestra.run.smithi003.stdout:(13/176): cephfs-shell-19.0.0-1361.g290212a1.el 463 kB/s | 27 kB 00:00 2024-02-09T22:33:22.157 INFO:teuthology.orchestra.run.smithi003.stdout:(14/176): libcephfs-devel-19.0.0-1361.g290212a1 467 kB/s | 31 kB 00:00 2024-02-09T22:33:22.240 INFO:teuthology.orchestra.run.smithi003.stdout:(15/176): libcephsqlite-19.0.0-1361.g290212a1.e 1.9 MB/s | 163 kB 00:00 2024-02-09T22:33:22.274 INFO:teuthology.orchestra.run.smithi003.stdout:(16/176): libcephfs2-19.0.0-1361.g290212a1.el9. 4.8 MB/s | 743 kB 00:00 2024-02-09T22:33:22.300 INFO:teuthology.orchestra.run.smithi003.stdout:(17/176): librados-devel-19.0.0-1361.g290212a1. 2.1 MB/s | 127 kB 00:00 2024-02-09T22:33:22.416 INFO:teuthology.orchestra.run.smithi003.stdout:(18/176): libradosstriper1-19.0.0-1361.g290212a 3.4 MB/s | 499 kB 00:00 2024-02-09T22:33:22.492 INFO:teuthology.orchestra.run.smithi003.stdout:(19/176): python3-ceph-argparse-19.0.0-1361.g29 601 kB/s | 45 kB 00:00 2024-02-09T22:33:22.575 INFO:teuthology.orchestra.run.smithi003.stdout:(20/176): python3-ceph-common-19.0.0-1361.g2902 1.5 MB/s | 131 kB 00:00 2024-02-09T22:33:22.668 INFO:teuthology.orchestra.run.smithi003.stdout:(21/176): python3-cephfs-19.0.0-1361.g290212a1. 1.7 MB/s | 161 kB 00:00 2024-02-09T22:33:22.768 INFO:teuthology.orchestra.run.smithi003.stdout:(22/176): python3-rados-19.0.0-1361.g290212a1.e 3.1 MB/s | 319 kB 00:00 2024-02-09T22:33:22.860 INFO:teuthology.orchestra.run.smithi003.stdout:(23/176): python3-rbd-19.0.0-1361.g290212a1.el9 3.2 MB/s | 298 kB 00:00 2024-02-09T22:33:22.927 INFO:teuthology.orchestra.run.smithi003.stdout:(24/176): librgw2-19.0.0-1361.g290212a1.el9.x86 8.1 MB/s | 5.1 MB 00:00 2024-02-09T22:33:22.952 INFO:teuthology.orchestra.run.smithi003.stdout:(25/176): python3-rgw-19.0.0-1361.g290212a1.el9 1.1 MB/s | 99 kB 00:00 2024-02-09T22:33:22.985 INFO:teuthology.orchestra.run.smithi003.stdout:(26/176): rbd-fuse-19.0.0-1361.g290212a1.el9.x8 1.4 MB/s | 85 kB 00:00 2024-02-09T22:33:23.069 INFO:teuthology.orchestra.run.smithi003.stdout:(27/176): rbd-nbd-19.0.0-1361.g290212a1.el9.x86 2.0 MB/s | 171 kB 00:00 2024-02-09T22:33:23.145 INFO:teuthology.orchestra.run.smithi003.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1361.g 378 kB/s | 28 kB 00:00 2024-02-09T22:33:23.228 INFO:teuthology.orchestra.run.smithi003.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1361.g290212a 1.6 MB/s | 138 kB 00:00 2024-02-09T22:33:23.404 INFO:teuthology.orchestra.run.smithi003.stdout:(30/176): rbd-mirror-19.0.0-1361.g290212a1.el9. 6.8 MB/s | 3.1 MB 00:00 2024-02-09T22:33:23.462 INFO:teuthology.orchestra.run.smithi003.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1361.g29021 7.5 MB/s | 1.8 MB 00:00 2024-02-09T22:33:23.554 INFO:teuthology.orchestra.run.smithi003.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1361.g29 2.6 MB/s | 247 kB 00:00 2024-02-09T22:33:23.629 INFO:teuthology.orchestra.run.smithi003.stdout:(33/176): ceph-mgr-rook-19.0.0-1361.g290212a1.e 697 kB/s | 52 kB 00:00 2024-02-09T22:33:23.705 INFO:teuthology.orchestra.run.smithi003.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1361.g2 204 kB/s | 15 kB 00:00 2024-02-09T22:33:23.797 INFO:teuthology.orchestra.run.smithi003.stdout:(35/176): ceph-volume-19.0.0-1361.g290212a1.el9 2.8 MB/s | 264 kB 00:00 2024-02-09T22:33:23.964 INFO:teuthology.orchestra.run.smithi003.stdout:(36/176): cephadm-19.0.0-1361.g290212a1.el9.noa 4.4 MB/s | 748 kB 00:00 2024-02-09T22:33:24.372 INFO:teuthology.orchestra.run.smithi003.stdout:(37/176): ceph-test-19.0.0-1361.g290212a1.el9.x 16 MB/s | 47 MB 00:03 2024-02-09T22:33:24.397 INFO:teuthology.orchestra.run.smithi003.stdout:(38/176): cephfs-top-19.0.0-1361.g290212a1.el9. 42 kB/s | 18 kB 00:00 2024-02-09T22:33:24.565 INFO:teuthology.orchestra.run.smithi003.stdout:(39/176): ceph-mgr-diskprediction-local-19.0.0- 6.4 MB/s | 7.4 MB 00:01 2024-02-09T22:33:24.648 INFO:teuthology.orchestra.run.smithi003.stdout:(40/176): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-02-09T22:33:24.673 INFO:teuthology.orchestra.run.smithi003.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 992 kB/s | 106 kB 00:00 2024-02-09T22:33:24.698 INFO:teuthology.orchestra.run.smithi003.stdout:(42/176): elfutils-libelf-devel-0.190-2.el9.x86 410 kB/s | 20 kB 00:00 2024-02-09T22:33:24.732 INFO:teuthology.orchestra.run.smithi003.stdout:(43/176): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-02-09T22:33:24.758 INFO:teuthology.orchestra.run.smithi003.stdout:(44/176): bison-3.7.4-5.el9.x86_64.rpm 2.5 MB/s | 920 kB 00:00 2024-02-09T22:33:24.775 INFO:teuthology.orchestra.run.smithi003.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 422 kB/s | 32 kB 00:00 2024-02-09T22:33:24.858 INFO:teuthology.orchestra.run.smithi003.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-02-09T22:33:24.875 INFO:teuthology.orchestra.run.smithi003.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 175 kB/s | 17 kB 00:00 2024-02-09T22:33:24.892 INFO:teuthology.orchestra.run.smithi003.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-02-09T22:33:24.917 INFO:teuthology.orchestra.run.smithi003.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-02-09T22:33:24.934 INFO:teuthology.orchestra.run.smithi003.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-09T22:33:24.959 INFO:teuthology.orchestra.run.smithi003.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-02-09T22:33:24.993 INFO:teuthology.orchestra.run.smithi003.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 7.6 MB/s | 578 kB 00:00 2024-02-09T22:33:25.010 INFO:teuthology.orchestra.run.smithi003.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 496 kB/s | 24 kB 00:00 2024-02-09T22:33:25.027 INFO:teuthology.orchestra.run.smithi003.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 704 kB/s | 46 kB 00:00 2024-02-09T22:33:25.052 INFO:teuthology.orchestra.run.smithi003.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-02-09T22:33:25.069 INFO:teuthology.orchestra.run.smithi003.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 833 kB/s | 34 kB 00:00 2024-02-09T22:33:25.094 INFO:teuthology.orchestra.run.smithi003.stdout:(57/176): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 187 kB 00:00 2024-02-09T22:33:25.178 INFO:teuthology.orchestra.run.smithi003.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 38 MB/s | 4.7 MB 00:00 2024-02-09T22:33:25.236 INFO:teuthology.orchestra.run.smithi003.stdout:(59/176): openssl-devel-3.0.7-25.el9.x86_64.rpm 18 MB/s | 3.0 MB 00:00 2024-02-09T22:33:25.278 INFO:teuthology.orchestra.run.smithi003.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 8.4 MB/s | 1.5 MB 00:00 2024-02-09T22:33:25.304 INFO:teuthology.orchestra.run.smithi003.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 6.5 MB/s | 811 kB 00:00 2024-02-09T22:33:25.329 INFO:teuthology.orchestra.run.smithi003.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.5 MB/s | 134 kB 00:00 2024-02-09T22:33:25.362 INFO:teuthology.orchestra.run.smithi003.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-02-09T22:33:25.455 INFO:teuthology.orchestra.run.smithi003.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 38 MB/s | 5.7 MB 00:00 2024-02-09T22:33:25.480 INFO:teuthology.orchestra.run.smithi003.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 518 kB/s | 78 kB 00:00 2024-02-09T22:33:25.572 INFO:teuthology.orchestra.run.smithi003.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-02-09T22:33:25.597 INFO:teuthology.orchestra.run.smithi003.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.6 MB/s | 227 kB 00:00 2024-02-09T22:33:25.622 INFO:teuthology.orchestra.run.smithi003.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 176 kB/s | 24 kB 00:00 2024-02-09T22:33:25.647 INFO:teuthology.orchestra.run.smithi003.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 242 kB/s | 18 kB 00:00 2024-02-09T22:33:25.681 INFO:teuthology.orchestra.run.smithi003.stdout:(70/176): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-02-09T22:33:25.815 INFO:teuthology.orchestra.run.smithi003.stdout:(71/176): python3-markupsafe-1.1.1-12.el9.x86_6 164 kB/s | 32 kB 00:00 2024-02-09T22:33:25.906 INFO:teuthology.orchestra.run.smithi003.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-02-09T22:33:25.932 INFO:teuthology.orchestra.run.smithi003.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2024-02-09T22:33:25.957 INFO:teuthology.orchestra.run.smithi003.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-02-09T22:33:25.982 INFO:teuthology.orchestra.run.smithi003.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 921 kB/s | 69 kB 00:00 2024-02-09T22:33:26.008 INFO:teuthology.orchestra.run.smithi003.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 821 kB/s | 41 kB 00:00 2024-02-09T22:33:26.033 INFO:teuthology.orchestra.run.smithi003.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-02-09T22:33:26.058 INFO:teuthology.orchestra.run.smithi003.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-09T22:33:26.083 INFO:teuthology.orchestra.run.smithi003.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 949 kB/s | 47 kB 00:00 2024-02-09T22:33:26.109 INFO:teuthology.orchestra.run.smithi003.stdout:(80/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-02-09T22:33:26.134 INFO:teuthology.orchestra.run.smithi003.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 570 kB/s | 43 kB 00:00 2024-02-09T22:33:26.276 INFO:teuthology.orchestra.run.smithi003.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 297 kB/s | 41 kB 00:00 2024-02-09T22:33:26.301 INFO:teuthology.orchestra.run.smithi003.stdout:(83/176): python3-toml-0.10.2-6.el9.noarch.rpm 236 kB/s | 39 kB 00:00 2024-02-09T22:33:26.485 INFO:teuthology.orchestra.run.smithi003.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 39 MB/s | 16 MB 00:00 2024-02-09T22:33:26.518 INFO:teuthology.orchestra.run.smithi003.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 890 kB/s | 215 kB 00:00 2024-02-09T22:33:26.543 INFO:teuthology.orchestra.run.smithi003.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 141 kB/s | 34 kB 00:00 2024-02-09T22:33:26.569 INFO:teuthology.orchestra.run.smithi003.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 522 kB/s | 43 kB 00:00 2024-02-09T22:33:26.594 INFO:teuthology.orchestra.run.smithi003.stdout:(88/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-02-09T22:33:26.619 INFO:teuthology.orchestra.run.smithi003.stdout:(89/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.9 MB/s | 300 kB 00:00 2024-02-09T22:33:26.654 INFO:teuthology.orchestra.run.smithi003.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.0 MB/s | 87 kB 00:00 2024-02-09T22:33:26.687 INFO:teuthology.orchestra.run.smithi003.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 684 kB/s | 63 kB 00:00 2024-02-09T22:33:26.712 INFO:teuthology.orchestra.run.smithi003.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 771 kB/s | 44 kB 00:00 2024-02-09T22:33:26.737 INFO:teuthology.orchestra.run.smithi003.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 955 kB/s | 79 kB 00:00 2024-02-09T22:33:26.787 INFO:teuthology.orchestra.run.smithi003.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 1.8 MB/s | 186 kB 00:00 2024-02-09T22:33:26.813 INFO:teuthology.orchestra.run.smithi003.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 194 kB 00:00 2024-02-09T22:33:26.838 INFO:teuthology.orchestra.run.smithi003.stdout:(96/176): libgfortran-11.4.1-3.el9.x86_64.rpm 7.8 MB/s | 802 kB 00:00 2024-02-09T22:33:26.863 INFO:teuthology.orchestra.run.smithi003.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-02-09T22:33:26.897 INFO:teuthology.orchestra.run.smithi003.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.8 MB/s | 241 kB 00:00 2024-02-09T22:33:26.922 INFO:teuthology.orchestra.run.smithi003.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 1.7 MB/s | 103 kB 00:00 2024-02-09T22:33:26.956 INFO:teuthology.orchestra.run.smithi003.stdout:(100/176): python3-cryptography-36.0.1-4.el9.x8 12 MB/s | 1.1 MB 00:00 2024-02-09T22:33:26.981 INFO:teuthology.orchestra.run.smithi003.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.4 MB/s | 124 kB 00:00 2024-02-09T22:33:27.014 INFO:teuthology.orchestra.run.smithi003.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.2 MB/s | 113 kB 00:00 2024-02-09T22:33:27.048 INFO:teuthology.orchestra.run.smithi003.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.7 MB/s | 187 kB 00:00 2024-02-09T22:33:27.157 INFO:teuthology.orchestra.run.smithi003.stdout:(104/176): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-02-09T22:33:27.207 INFO:teuthology.orchestra.run.smithi003.stdout:(105/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.1 MB/s | 245 kB 00:00 2024-02-09T22:33:27.332 INFO:teuthology.orchestra.run.smithi003.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 3.0 MB/s | 863 kB 00:00 2024-02-09T22:33:27.450 INFO:teuthology.orchestra.run.smithi003.stdout:(107/176): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-02-09T22:33:27.517 INFO:teuthology.orchestra.run.smithi003.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 999 kB/s | 309 kB 00:00 2024-02-09T22:33:27.550 INFO:teuthology.orchestra.run.smithi003.stdout:(109/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.4 MB/s | 548 kB 00:00 2024-02-09T22:33:27.575 INFO:teuthology.orchestra.run.smithi003.stdout:(110/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 443 kB/s | 26 kB 00:00 2024-02-09T22:33:27.609 INFO:teuthology.orchestra.run.smithi003.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 827 kB/s | 49 kB 00:00 2024-02-09T22:33:27.643 INFO:teuthology.orchestra.run.smithi003.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-02-09T22:33:27.685 INFO:teuthology.orchestra.run.smithi003.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-02-09T22:33:27.785 INFO:teuthology.orchestra.run.smithi003.stdout:(114/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.8 MB/s | 838 kB 00:00 2024-02-09T22:33:27.819 INFO:teuthology.orchestra.run.smithi003.stdout:(115/176): python3-asyncssh-2.13.2-3.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-02-09T22:33:27.886 INFO:teuthology.orchestra.run.smithi003.stdout:(116/176): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-02-09T22:33:27.911 INFO:teuthology.orchestra.run.smithi003.stdout:(117/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 346 kB/s | 43 kB 00:00 2024-02-09T22:33:27.936 INFO:teuthology.orchestra.run.smithi003.stdout:(118/176): python3-cachetools-4.2.4-1.el9.noarc 274 kB/s | 32 kB 00:00 2024-02-09T22:33:27.962 INFO:teuthology.orchestra.run.smithi003.stdout:(119/176): python3-certifi-2021.10.8-2.el9.noar 199 kB/s | 15 kB 00:00 2024-02-09T22:33:28.020 INFO:teuthology.orchestra.run.smithi003.stdout:(120/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-02-09T22:33:28.045 INFO:teuthology.orchestra.run.smithi003.stdout:(121/176): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-02-09T22:33:28.071 INFO:teuthology.orchestra.run.smithi003.stdout:(122/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-02-09T22:33:28.096 INFO:teuthology.orchestra.run.smithi003.stdout:(123/176): python3-colorama-0.4.4-12.el9.noarch 468 kB/s | 35 kB 00:00 2024-02-09T22:33:28.130 INFO:teuthology.orchestra.run.smithi003.stdout:(124/176): python3-google-auth-2.27.0-1.el9.noa 2.4 MB/s | 203 kB 00:00 2024-02-09T22:33:28.180 INFO:teuthology.orchestra.run.smithi003.stdout:(125/176): python3-grpcio-1.46.7-10.el9.x86_64. 19 MB/s | 2.0 MB 00:00 2024-02-09T22:33:28.205 INFO:teuthology.orchestra.run.smithi003.stdout:(126/176): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-02-09T22:33:28.230 INFO:teuthology.orchestra.run.smithi003.stdout:(127/176): python3-grpcio-tools-1.46.7-10.el9.x 1.1 MB/s | 144 kB 00:00 2024-02-09T22:33:28.256 INFO:teuthology.orchestra.run.smithi003.stdout:(128/176): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-02-09T22:33:28.281 INFO:teuthology.orchestra.run.smithi003.stdout:(129/176): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-02-09T22:33:28.306 INFO:teuthology.orchestra.run.smithi003.stdout:(130/176): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-02-09T22:33:28.331 INFO:teuthology.orchestra.run.smithi003.stdout:(131/176): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-02-09T22:33:28.373 INFO:teuthology.orchestra.run.smithi003.stdout:(132/176): python3-kubernetes-26.1.0-2.el9.noar 11 MB/s | 1.0 MB 00:00 2024-02-09T22:33:28.398 INFO:teuthology.orchestra.run.smithi003.stdout:(133/176): python3-logutils-0.3.5-21.el9.noarch 501 kB/s | 46 kB 00:00 2024-02-09T22:33:28.423 INFO:teuthology.orchestra.run.smithi003.stdout:(134/176): python3-more-itertools-8.12.0-2.el9. 856 kB/s | 79 kB 00:00 2024-02-09T22:33:28.449 INFO:teuthology.orchestra.run.smithi003.stdout:(135/176): python3-natsort-7.1.1-5.el9.noarch.r 766 kB/s | 58 kB 00:00 2024-02-09T22:33:28.483 INFO:teuthology.orchestra.run.smithi003.stdout:(136/176): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-02-09T22:33:28.500 INFO:teuthology.orchestra.run.smithi003.stdout:(137/176): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-02-09T22:33:28.525 INFO:teuthology.orchestra.run.smithi003.stdout:(138/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-02-09T22:33:28.550 INFO:teuthology.orchestra.run.smithi003.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 610 kB/s | 31 kB 00:00 2024-02-09T22:33:28.567 INFO:teuthology.orchestra.run.smithi003.stdout:(140/176): python3-pyperclip-1.8.2-2.el9.noarch 335 kB/s | 28 kB 00:00 2024-02-09T22:33:28.601 INFO:teuthology.orchestra.run.smithi003.stdout:(141/176): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-02-09T22:33:28.626 INFO:teuthology.orchestra.run.smithi003.stdout:(142/176): python3-rsa-4.9-2.el9.noarch.rpm 781 kB/s | 59 kB 00:00 2024-02-09T22:33:28.660 INFO:teuthology.orchestra.run.smithi003.stdout:(143/176): python3-tempora-5.0.0-2.el9.noarch.r 388 kB/s | 36 kB 00:00 2024-02-09T22:33:28.685 INFO:teuthology.orchestra.run.smithi003.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-02-09T22:33:28.727 INFO:teuthology.orchestra.run.smithi003.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-02-09T22:33:28.786 INFO:teuthology.orchestra.run.smithi003.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 714 kB/s | 90 kB 00:00 2024-02-09T22:33:28.811 INFO:teuthology.orchestra.run.smithi003.stdout:(147/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-02-09T22:33:28.836 INFO:teuthology.orchestra.run.smithi003.stdout:(148/176): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-02-09T22:33:28.887 INFO:teuthology.orchestra.run.smithi003.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-02-09T22:33:29.037 INFO:teuthology.orchestra.run.smithi003.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 7.0 MB/s | 1.6 MB 00:00 2024-02-09T22:33:29.521 INFO:teuthology.orchestra.run.smithi003.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 140 kB/s | 68 kB 00:00 2024-02-09T22:33:29.579 INFO:teuthology.orchestra.run.smithi003.stdout:(152/176): librados2-19.0.0-1361.g290212a1.el9. 4.5 MB/s | 3.4 MB 00:00 2024-02-09T22:33:30.854 INFO:teuthology.orchestra.run.smithi003.stdout:(153/176): librbd1-19.0.0-1361.g290212a1.el9.x8 1.6 MB/s | 3.1 MB 00:01 2024-02-09T22:33:32.538 INFO:teuthology.orchestra.run.smithi003.stdout:(154/176): NetworkManager-1.45.90-1.el9.x86_64. 764 kB/s | 2.3 MB 00:03 2024-02-09T22:33:32.654 INFO:teuthology.orchestra.run.smithi003.stdout:(155/176): NetworkManager-libnm-1.45.90-1.el9.x 600 kB/s | 1.8 MB 00:03 2024-02-09T22:33:32.680 INFO:teuthology.orchestra.run.smithi003.stdout:(156/176): NetworkManager-team-1.45.90-1.el9.x8 23 kB/s | 42 kB 00:01 2024-02-09T22:33:32.705 INFO:teuthology.orchestra.run.smithi003.stdout:(157/176): NetworkManager-tui-1.45.90-1.el9.x86 1.5 MB/s | 247 kB 00:00 2024-02-09T22:33:32.730 INFO:teuthology.orchestra.run.smithi003.stdout:(158/176): acl-2.3.1-4.el9.x86_64.rpm 920 kB/s | 69 kB 00:00 2024-02-09T22:33:32.756 INFO:teuthology.orchestra.run.smithi003.stdout:(159/176): elfutils-debuginfod-client-0.190-2.e 706 kB/s | 35 kB 00:00 2024-02-09T22:33:32.783 INFO:teuthology.orchestra.run.smithi003.stdout:(160/176): elfutils-libelf-0.190-2.el9.x86_64.r 2.5 MB/s | 191 kB 00:00 2024-02-09T22:33:32.815 INFO:teuthology.orchestra.run.smithi003.stdout:(161/176): elfutils-libs-0.190-2.el9.x86_64.rpm 3.0 MB/s | 254 kB 00:00 2024-02-09T22:33:32.864 INFO:teuthology.orchestra.run.smithi003.stdout:(162/176): iproute-6.7.0-2.el9.x86_64.rpm 9.4 MB/s | 802 kB 00:00 2024-02-09T22:33:32.898 INFO:teuthology.orchestra.run.smithi003.stdout:(163/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 3.6 MB/s | 433 kB 00:00 2024-02-09T22:33:32.923 INFO:teuthology.orchestra.run.smithi003.stdout:(164/176): libacl-2.3.1-4.el9.x86_64.rpm 204 kB/s | 22 kB 00:00 2024-02-09T22:33:32.949 INFO:teuthology.orchestra.run.smithi003.stdout:(165/176): libblkid-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 107 kB 00:00 2024-02-09T22:33:32.974 INFO:teuthology.orchestra.run.smithi003.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-02-09T22:33:33.000 INFO:teuthology.orchestra.run.smithi003.stdout:(167/176): libfdisk-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 154 kB 00:00 2024-02-09T22:33:33.025 INFO:teuthology.orchestra.run.smithi003.stdout:(168/176): libsmartcols-2.37.4-18.el9.x86_64.rp 833 kB/s | 63 kB 00:00 2024-02-09T22:33:33.052 INFO:teuthology.orchestra.run.smithi003.stdout:(169/176): libuuid-2.37.4-18.el9.x86_64.rpm 369 kB/s | 28 kB 00:00 2024-02-09T22:33:33.150 INFO:teuthology.orchestra.run.smithi003.stdout:(170/176): openssl-3.0.7-25.el9.x86_64.rpm 9.2 MB/s | 1.2 MB 00:00 2024-02-09T22:33:33.242 INFO:teuthology.orchestra.run.smithi003.stdout:(171/176): openssl-libs-3.0.7-25.el9.x86_64.rpm 9.9 MB/s | 2.1 MB 00:00 2024-02-09T22:33:33.334 INFO:teuthology.orchestra.run.smithi003.stdout:(172/176): util-linux-2.37.4-18.el9.x86_64.rpm 7.7 MB/s | 2.2 MB 00:00 2024-02-09T22:33:33.359 INFO:teuthology.orchestra.run.smithi003.stdout:(173/176): util-linux-core-2.37.4-18.el9.x86_64 2.1 MB/s | 438 kB 00:00 2024-02-09T22:33:33.376 INFO:teuthology.orchestra.run.smithi003.stdout:(174/176): util-linux-user-2.37.4-18.el9.x86_64 228 kB/s | 30 kB 00:00 2024-02-09T22:33:33.410 INFO:teuthology.orchestra.run.smithi003.stdout:(175/176): xfsdump-3.1.12-4.el9.x86_64.rpm 4.2 MB/s | 321 kB 00:00 2024-02-09T22:33:33.452 INFO:teuthology.orchestra.run.smithi003.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 14 MB/s | 1.1 MB 00:00 2024-02-09T22:33:33.457 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:33:33.457 INFO:teuthology.orchestra.run.smithi003.stdout:Total 15 MB/s | 222 MB 00:14 2024-02-09T22:33:33.509 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction check 2024-02-09T22:33:33.833 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction check succeeded. 2024-02-09T22:33:33.833 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction test 2024-02-09T22:33:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:Transaction test succeeded. 2024-02-09T22:33:35.900 INFO:teuthology.orchestra.run.smithi003.stdout:Running transaction 2024-02-09T22:33:43.146 INFO:teuthology.orchestra.run.smithi003.stdout: Preparing : 1/1 2024-02-09T22:33:43.268 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/202 2024-02-09T22:33:43.300 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-02-09T22:33:43.346 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-09T22:33:43.376 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-09T22:33:43.452 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-02-09T22:33:43.498 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-02-09T22:33:43.550 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-02-09T22:33:43.590 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-02-09T22:33:43.623 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-02-09T22:33:43.683 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-02-09T22:33:44.026 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-02-09T22:33:44.082 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:33:44.190 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:33:44.217 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:33:44.310 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:33:44.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 13/202 2024-02-09T22:33:44.514 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-02-09T22:33:44.569 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-02-09T22:33:44.596 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:33:44.702 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:33:44.731 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 17/202 2024-02-09T22:33:44.796 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 17/202 2024-02-09T22:33:44.840 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-02-09T22:33:44.866 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-09T22:33:44.881 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-09T22:33:45.039 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-09T22:33:45.072 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-09T22:33:45.104 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-02-09T22:33:45.307 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-02-09T22:33:45.434 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-libnm-1:1.45.90-1.el9.x86_64 23/202 2024-02-09T22:33:45.502 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.90-1.el9.x86_64 23/202 2024-02-09T22:33:45.598 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:33:45.662 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:33:45.876 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:33:45.988 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-02-09T22:33:46.083 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-02-09T22:33:46.117 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-02-09T22:33:46.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-02-09T22:33:46.249 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-02-09T22:33:46.287 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-02-09T22:33:46.323 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-02-09T22:33:46.363 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-02-09T22:33:46.438 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-02-09T22:33:46.479 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-02-09T22:33:46.522 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-02-09T22:33:46.555 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-02-09T22:33:46.582 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-02-09T22:33:46.654 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-02-09T22:33:46.717 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9. 39/202 2024-02-09T22:33:46.756 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el 40/202 2024-02-09T22:33:46.784 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_6 41/202 2024-02-09T22:33:46.858 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-02-09T22:33:46.899 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-02-09T22:33:46.977 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-02-09T22:33:47.113 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-02-09T22:33:47.170 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-02-09T22:33:47.733 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-02-09T22:33:47.816 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-02-09T22:33:47.898 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-02-09T22:33:47.950 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-09T22:33:47.959 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-09T22:33:47.997 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-02-09T22:33:48.034 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_6 52/202 2024-02-09T22:33:48.071 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-02-09T22:33:48.179 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-02-09T22:33:48.251 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-02-09T22:33:48.556 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-02-09T22:33:48.710 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-02-09T22:33:48.877 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-02-09T22:33:49.074 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-02-09T22:33:49.214 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-02-09T22:33:49.274 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 61/202 2024-02-09T22:33:49.331 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-02-09T22:33:49.405 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-02-09T22:33:49.441 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-02-09T22:33:49.478 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-02-09T22:33:49.533 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-02-09T22:33:49.569 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-02-09T22:33:49.612 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-02-09T22:33:49.655 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-02-09T22:33:49.793 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-02-09T22:33:49.843 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-02-09T22:33:49.873 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-02-09T22:33:49.915 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-02-09T22:33:50.032 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-02-09T22:33:50.064 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 75/202 2024-02-09T22:33:50.105 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-02-09T22:33:50.138 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-02-09T22:33:50.163 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-02-09T22:33:50.240 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-02-09T22:33:50.398 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-02-09T22:33:50.508 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-02-09T22:33:50.556 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-02-09T22:33:50.613 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-02-09T22:33:50.669 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-02-09T22:33:50.871 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-02-09T22:33:50.925 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-02-09T22:33:51.072 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-02-09T22:33:51.116 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 88/202 2024-02-09T22:33:51.148 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-02-09T22:33:51.180 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-02-09T22:33:51.224 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-02-09T22:33:51.262 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-02-09T22:33:51.321 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-02-09T22:33:51.466 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-02-09T22:33:51.656 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-02-09T22:33:51.788 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-02-09T22:33:51.817 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-02-09T22:33:51.897 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-02-09T22:33:51.936 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-02-09T22:33:51.977 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-02-09T22:33:52.019 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-02-09T22:33:52.053 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-02-09T22:33:52.145 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-02-09T22:33:52.235 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-02-09T22:33:52.285 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-02-09T22:33:52.315 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-02-09T22:33:52.372 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-02-09T22:33:52.472 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-02-09T22:33:52.533 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-02-09T22:33:53.298 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-02-09T22:33:53.346 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-02-09T22:33:53.379 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-02-09T22:33:53.429 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-02-09T22:33:53.520 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-02-09T22:33:53.555 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-02-09T22:33:53.740 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-02-09T22:33:53.778 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-02-09T22:33:53.858 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-02-09T22:33:54.533 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-02-09T22:33:54.653 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-02-09T22:33:55.957 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-02-09T22:33:56.040 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-02-09T22:33:56.066 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-02-09T22:33:56.390 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-02-09T22:33:56.429 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 125/202 2024-02-09T22:33:56.537 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 125/202 2024-02-09T22:33:56.576 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 126/202 2024-02-09T22:33:56.621 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-02-09T22:33:57.102 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-02-09T22:33:57.132 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 129/202 2024-02-09T22:33:57.204 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 129/202 2024-02-09T22:33:57.234 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 130/202 2024-02-09T22:33:59.549 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:33:59.590 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:34:00.160 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:34:00.194 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 132/202 2024-02-09T22:34:00.293 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 132/202 2024-02-09T22:34:00.293 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-09T22:34:00.293 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-09T22:34:00.293 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:00.328 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 133/202 2024-02-09T22:34:09.934 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 130 kB/s | 5.9 MB 00:46 2024-02-09T22:34:11.434 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 15 kB 00:00 2024-02-09T22:34:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 133/202 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /sys 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /proc 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /mnt 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /var/tmp 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /home 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /root 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout:skipping the directory /tmp 2024-02-09T22:34:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 134/202 2024-02-09T22:34:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 134/202 2024-02-09T22:34:12.420 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:12.420 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-09T22:34:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:13.948 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 135/202 2024-02-09T22:34:13.974 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 135/202 2024-02-09T22:34:13.975 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:13.975 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-09T22:34:13.975 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:34:13.975 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:34:13.976 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:14.193 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 136/202 2024-02-09T22:34:14.214 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 136/202 2024-02-09T22:34:14.214 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:14.214 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-09T22:34:14.214 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:34:14.214 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:34:14.215 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:14.677 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 137/202 2024-02-09T22:34:14.699 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 137/202 2024-02-09T22:34:14.700 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:14.700 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-09T22:34:14.700 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:34:14.700 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:34:14.700 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:14.804 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-02-09T22:34:14.997 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 20 MB 00:03 2024-02-09T22:34:15.262 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 139/202 2024-02-09T22:34:15.293 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 139/202 2024-02-09T22:34:15.323 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.e 140/202 2024-02-09T22:34:15.388 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1. 141/202 2024-02-09T22:34:15.425 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 142/202 2024-02-09T22:34:15.664 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 142/202 2024-02-09T22:34:15.700 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 143/202 2024-02-09T22:34:16.718 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 143/202 2024-02-09T22:34:16.742 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 144/202 2024-02-09T22:34:16.847 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 144/202 2024-02-09T22:34:17.021 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el 145/202 2024-02-09T22:34:17.052 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 146/202 2024-02-09T22:34:17.076 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 146/202 2024-02-09T22:34:17.076 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:17.076 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-09T22:34:17.076 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:34:17.076 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:34:17.076 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:17.128 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 147/202 2024-02-09T22:34:17.140 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 147/202 2024-02-09T22:34:17.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 148/202 2024-02-09T22:34:18.085 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-02-09T22:34:18.122 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 150/202 2024-02-09T22:34:18.145 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 150/202 2024-02-09T22:34:18.145 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:18.145 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-09T22:34:18.145 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:34:18.145 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:34:18.145 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:18.189 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g29021 151/202 2024-02-09T22:34:18.211 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g29021 151/202 2024-02-09T22:34:18.211 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:18.211 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-09T22:34:18.211 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:18.265 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 152/202 2024-02-09T22:34:18.288 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 152/202 2024-02-09T22:34:18.288 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:18.288 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-09T22:34:18.288 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-09T22:34:18.288 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-09T22:34:18.288 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:18.607 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:34:18.630 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:34:18.630 INFO:teuthology.orchestra.run.smithi003.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:34:18.630 INFO:teuthology.orchestra.run.smithi003.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-09T22:34:18.630 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:34:18.630 INFO:teuthology.orchestra.run.smithi003.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:34:18.630 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:19.927 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-02-09T22:34:22.504 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-09T22:34:22.505 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-09T22:34:22.517 INFO:teuthology.orchestra.run.smithi089.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-09T22:34:22.518 INFO:teuthology.orchestra.run.smithi089.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-09T22:34:22.519 INFO:teuthology.orchestra.run.smithi089.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-09T22:34:22.519 INFO:teuthology.orchestra.run.smithi089.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-09T22:34:22.521 INFO:teuthology.orchestra.run.smithi089.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-09T22:34:22.523 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-09T22:34:22.524 INFO:teuthology.orchestra.run.smithi089.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-09T22:34:22.604 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-02-09T22:34:22.626 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-02-09T22:34:22.626 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-02-09T22:34:22.626 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-02-09T22:34:22.626 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-02-09T22:34:22.626 INFO:teuthology.orchestra.run.smithi089.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-09T22:34:22.627 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 217 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-shell x86_64 2:19.0.0-1361.g290212a1.el9 ceph 27 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 18 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-02-09T22:34:22.628 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-02-09T22:34:22.629 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager x86_64 1:1.45.90-1.el9 CentOS-BaseOS 2.3 M 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm x86_64 1:1.45.90-1.el9 CentOS-BaseOS 1.8 M 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team x86_64 1:1.45.90-1.el9 CentOS-BaseOS 42 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui x86_64 1:1.45.90-1.el9 CentOS-BaseOS 247 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-02-09T22:34:22.630 INFO:teuthology.orchestra.run.smithi089.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:34:22.631 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-09T22:34:22.632 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-09T22:34:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-09T22:34:22.634 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-09T22:34:22.635 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-09T22:34:22.636 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-09T22:34:22.637 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-09T22:34:22.638 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-09T22:34:22.639 INFO:teuthology.orchestra.run.smithi089.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout:Installing weak dependencies: 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout:============================================================================================ 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout:Install 150 Packages 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 26 Packages 2024-02-09T22:34:22.640 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:34:22.641 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 222 M 2024-02-09T22:34:22.641 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-02-09T22:34:23.328 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 154/202 2024-02-09T22:34:23.369 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 155/202 2024-02-09T22:34:23.399 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 156/202 2024-02-09T22:34:23.426 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-02-09T22:34:23.428 INFO:teuthology.orchestra.run.smithi089.stdout:(1/176): ceph-19.0.0-1361.g290212a1.el9.x86_64. 37 kB/s | 6.6 kB 00:00 2024-02-09T22:34:23.629 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 158/202 2024-02-09T22:34:23.662 INFO:teuthology.orchestra.run.smithi089.stdout:(2/176): ceph-fuse-19.0.0-1361.g290212a1.el9.x8 3.7 MB/s | 885 kB 00:00 2024-02-09T22:34:23.737 INFO:teuthology.orchestra.run.smithi089.stdout:(3/176): ceph-immutable-object-cache-19.0.0-136 1.9 MB/s | 145 kB 00:00 2024-02-09T22:34:23.748 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-02-09T22:34:23.794 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 160/202 2024-02-09T22:34:23.851 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-02-09T22:34:23.918 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-02-09T22:34:23.995 INFO:teuthology.orchestra.run.smithi089.stdout:(4/176): ceph-base-19.0.0-1361.g290212a1.el9.x8 7.2 MB/s | 5.3 MB 00:00 2024-02-09T22:34:24.012 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-02-09T22:34:24.061 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-02-09T22:34:24.105 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-02-09T22:34:24.135 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_ 166/202 2024-02-09T22:34:24.162 INFO:teuthology.orchestra.run.smithi089.stdout:(5/176): ceph-mds-19.0.0-1361.g290212a1.el9.x86 5.0 MB/s | 2.1 MB 00:00 2024-02-09T22:34:24.167 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-team-1:1.45.90-1.el9.x86_64 167/202 2024-02-09T22:34:24.204 INFO:teuthology.orchestra.run.smithi089.stdout:(6/176): ceph-mgr-19.0.0-1361.g290212a1.el9.x86 7.2 MB/s | 1.5 MB 00:00 2024-02-09T22:34:24.239 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : NetworkManager-tui-1:1.45.90-1.el9.x86_64 168/202 2024-02-09T22:34:24.298 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-02-09T22:34:24.339 INFO:teuthology.orchestra.run.smithi003.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-02-09T22:34:24.973 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:34:25.019 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:34:25.040 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:34:25.046 INFO:teuthology.orchestra.run.smithi089.stdout:(7/176): ceph-mon-19.0.0-1361.g290212a1.el9.x86 5.3 MB/s | 4.7 MB 00:00 2024-02-09T22:34:25.296 INFO:teuthology.orchestra.run.smithi089.stdout:(8/176): ceph-common-19.0.0-1361.g290212a1.el9. 9.7 MB/s | 20 MB 00:02 2024-02-09T22:34:25.355 INFO:teuthology.orchestra.run.smithi089.stdout:(9/176): ceph-selinux-19.0.0-1361.g290212a1.el9 433 kB/s | 25 kB 00:00 2024-02-09T22:34:25.897 INFO:teuthology.orchestra.run.smithi089.stdout:(10/176): ceph-osd-19.0.0-1361.g290212a1.el9.x8 9.9 MB/s | 17 MB 00:01 2024-02-09T22:34:25.903 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 172/202 2024-02-09T22:34:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:(11/176): cephfs-mirror-19.0.0-1361.g290212a1.e 2.8 MB/s | 217 kB 00:00 2024-02-09T22:34:25.989 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 173/202 2024-02-09T22:34:26.031 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-02-09T22:34:26.047 INFO:teuthology.orchestra.run.smithi089.stdout:(12/176): cephfs-shell-19.0.0-1361.g290212a1.el 360 kB/s | 27 kB 00:00 2024-02-09T22:34:26.122 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-02-09T22:34:26.122 INFO:teuthology.orchestra.run.smithi089.stdout:(13/176): libcephfs-devel-19.0.0-1361.g290212a1 417 kB/s | 31 kB 00:00 2024-02-09T22:34:26.151 INFO:teuthology.orchestra.run.smithi003.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-02-09T22:34:26.176 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-02-09T22:34:26.210 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-02-09T22:34:26.245 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-02-09T22:34:26.245 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-09T22:34:26.281 INFO:teuthology.orchestra.run.smithi089.stdout:(14/176): libcephfs2-19.0.0-1361.g290212a1.el9. 4.6 MB/s | 743 kB 00:00 2024-02-09T22:34:26.329 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-09T22:34:26.358 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-02-09T22:34:26.364 INFO:teuthology.orchestra.run.smithi089.stdout:(15/176): libcephsqlite-19.0.0-1361.g290212a1.e 1.9 MB/s | 163 kB 00:00 2024-02-09T22:34:26.387 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-02-09T22:34:26.387 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-09T22:34:26.448 INFO:teuthology.orchestra.run.smithi089.stdout:(16/176): librados-devel-19.0.0-1361.g290212a1. 1.5 MB/s | 127 kB 00:00 2024-02-09T22:34:26.472 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-09T22:34:26.498 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-02-09T22:34:26.540 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-02-09T22:34:26.557 INFO:teuthology.orchestra.run.smithi089.stdout:(17/176): ceph-radosgw-19.0.0-1361.g290212a1.el 6.4 MB/s | 9.7 MB 00:01 2024-02-09T22:34:26.573 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-02-09T22:34:26.590 INFO:teuthology.orchestra.run.smithi089.stdout:(18/176): libradosstriper1-19.0.0-1361.g290212a 3.4 MB/s | 499 kB 00:00 2024-02-09T22:34:26.601 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-02-09T22:34:26.633 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-02-09T22:34:26.683 INFO:teuthology.orchestra.run.smithi089.stdout:(19/176): python3-ceph-argparse-19.0.0-1361.g29 490 kB/s | 45 kB 00:00 2024-02-09T22:34:26.684 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-02-09T22:34:26.731 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-02-09T22:34:26.766 INFO:teuthology.orchestra.run.smithi089.stdout:(20/176): python3-ceph-common-19.0.0-1361.g2902 1.5 MB/s | 131 kB 00:00 2024-02-09T22:34:26.771 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-02-09T22:34:26.796 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-02-09T22:34:26.824 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-02-09T22:34:26.847 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-02-09T22:34:26.850 INFO:teuthology.orchestra.run.smithi089.stdout:(21/176): python3-cephfs-19.0.0-1361.g290212a1. 1.9 MB/s | 161 kB 00:00 2024-02-09T22:34:26.872 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-02-09T22:34:26.879 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:34:26.879 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:34:26.942 INFO:teuthology.orchestra.run.smithi089.stdout:(22/176): python3-rados-19.0.0-1361.g290212a1.e 3.4 MB/s | 319 kB 00:00 2024-02-09T22:34:27.034 INFO:teuthology.orchestra.run.smithi089.stdout:(23/176): python3-rbd-19.0.0-1361.g290212a1.el9 3.2 MB/s | 298 kB 00:00 2024-02-09T22:34:27.067 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:34:27.067 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-09T22:34:27.118 INFO:teuthology.orchestra.run.smithi089.stdout:(24/176): python3-rgw-19.0.0-1361.g290212a1.el9 1.2 MB/s | 99 kB 00:00 2024-02-09T22:34:27.158 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-09T22:34:27.183 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-02-09T22:34:27.201 INFO:teuthology.orchestra.run.smithi089.stdout:(25/176): rbd-fuse-19.0.0-1361.g290212a1.el9.x8 1.0 MB/s | 85 kB 00:00 2024-02-09T22:34:27.208 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-02-09T22:34:27.233 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-02-09T22:34:27.266 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-02-09T22:34:27.301 INFO:teuthology.orchestra.run.smithi089.stdout:(26/176): librgw2-19.0.0-1361.g290212a1.el9.x86 6.9 MB/s | 5.1 MB 00:00 2024-02-09T22:34:27.344 INFO:teuthology.orchestra.run.smithi003.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-09T22:34:27.402 INFO:teuthology.orchestra.run.smithi089.stdout:(27/176): rbd-nbd-19.0.0-1361.g290212a1.el9.x86 1.7 MB/s | 171 kB 00:00 2024-02-09T22:34:27.493 INFO:teuthology.orchestra.run.smithi089.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1361.g 310 kB/s | 28 kB 00:00 2024-02-09T22:34:27.577 INFO:teuthology.orchestra.run.smithi089.stdout:(29/176): rbd-mirror-19.0.0-1361.g290212a1.el9. 8.2 MB/s | 3.1 MB 00:00 2024-02-09T22:34:27.594 INFO:teuthology.orchestra.run.smithi089.stdout:(30/176): ceph-mgr-cephadm-19.0.0-1361.g290212a 1.3 MB/s | 138 kB 00:00 2024-02-09T22:34:27.861 INFO:teuthology.orchestra.run.smithi089.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1361.g29021 6.2 MB/s | 1.8 MB 00:00 2024-02-09T22:34:27.969 INFO:teuthology.orchestra.run.smithi089.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1361.g29 2.2 MB/s | 247 kB 00:00 2024-02-09T22:34:28.062 INFO:teuthology.orchestra.run.smithi089.stdout:(33/176): ceph-mgr-rook-19.0.0-1361.g290212a1.e 569 kB/s | 52 kB 00:00 2024-02-09T22:34:28.162 INFO:teuthology.orchestra.run.smithi089.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1361.g2 153 kB/s | 15 kB 00:00 2024-02-09T22:34:28.812 INFO:teuthology.orchestra.run.smithi089.stdout:(35/176): ceph-test-19.0.0-1361.g290212a1.el9.x 14 MB/s | 47 MB 00:03 2024-02-09T22:34:28.837 INFO:teuthology.orchestra.run.smithi089.stdout:(36/176): ceph-volume-19.0.0-1361.g290212a1.el9 390 kB/s | 264 kB 00:00 2024-02-09T22:34:28.887 INFO:teuthology.orchestra.run.smithi089.stdout:(37/176): cephfs-top-19.0.0-1361.g290212a1.el9. 367 kB/s | 18 kB 00:00 2024-02-09T22:34:28.921 INFO:teuthology.orchestra.run.smithi089.stdout:(38/176): cephadm-19.0.0-1361.g290212a1.el9.noa 6.7 MB/s | 748 kB 00:00 2024-02-09T22:34:29.030 INFO:teuthology.orchestra.run.smithi089.stdout:(39/176): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-02-09T22:34:29.172 INFO:teuthology.orchestra.run.smithi089.stdout:(40/176): automake-1.16.2-8.el9.noarch.rpm 2.3 MB/s | 662 kB 00:00 2024-02-09T22:34:29.197 INFO:teuthology.orchestra.run.smithi089.stdout:(41/176): boost-program-options-1.75.0-8.el9.x8 759 kB/s | 106 kB 00:00 2024-02-09T22:34:29.222 INFO:teuthology.orchestra.run.smithi089.stdout:(42/176): bison-3.7.4-5.el9.x86_64.rpm 3.3 MB/s | 920 kB 00:00 2024-02-09T22:34:29.247 INFO:teuthology.orchestra.run.smithi089.stdout:(43/176): elfutils-libelf-devel-0.190-2.el9.x86 273 kB/s | 20 kB 00:00 2024-02-09T22:34:29.281 INFO:teuthology.orchestra.run.smithi089.stdout:(44/176): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-02-09T22:34:29.306 INFO:teuthology.orchestra.run.smithi089.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 547 kB/s | 32 kB 00:00 2024-02-09T22:34:29.373 INFO:teuthology.orchestra.run.smithi089.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 24 MB/s | 3.0 MB 00:00 2024-02-09T22:34:29.398 INFO:teuthology.orchestra.run.smithi089.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 146 kB/s | 17 kB 00:00 2024-02-09T22:34:29.424 INFO:teuthology.orchestra.run.smithi089.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 88 kB/s | 10 kB 00:00 2024-02-09T22:34:29.449 INFO:teuthology.orchestra.run.smithi089.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 159 kB 00:00 2024-02-09T22:34:29.483 INFO:teuthology.orchestra.run.smithi089.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 757 kB/s | 44 kB 00:00 2024-02-09T22:34:29.516 INFO:teuthology.orchestra.run.smithi089.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 7.0 MB/s | 661 kB 00:00 2024-02-09T22:34:29.542 INFO:teuthology.orchestra.run.smithi089.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-02-09T22:34:29.568 INFO:teuthology.orchestra.run.smithi089.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 497 kB/s | 24 kB 00:00 2024-02-09T22:34:29.592 INFO:teuthology.orchestra.run.smithi089.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 627 kB/s | 46 kB 00:00 2024-02-09T22:34:29.617 INFO:teuthology.orchestra.run.smithi089.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-02-09T22:34:29.642 INFO:teuthology.orchestra.run.smithi089.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 688 kB/s | 34 kB 00:00 2024-02-09T22:34:29.668 INFO:teuthology.orchestra.run.smithi089.stdout:(57/176): lua-5.4.4-4.el9.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2024-02-09T22:34:29.776 INFO:teuthology.orchestra.run.smithi089.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-02-09T22:34:29.835 INFO:teuthology.orchestra.run.smithi089.stdout:(59/176): openssl-devel-3.0.7-25.el9.x86_64.rpm 16 MB/s | 3.0 MB 00:00 2024-02-09T22:34:29.877 INFO:teuthology.orchestra.run.smithi089.stdout:(60/176): postgresql-13.11-1.el9.x86_64.rpm 7.4 MB/s | 1.5 MB 00:00 2024-02-09T22:34:29.910 INFO:teuthology.orchestra.run.smithi089.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 6.1 MB/s | 811 kB 00:00 2024-02-09T22:34:29.936 INFO:teuthology.orchestra.run.smithi089.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.3 MB/s | 134 kB 00:00 2024-02-09T22:34:29.977 INFO:teuthology.orchestra.run.smithi089.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-02-09T22:34:30.103 INFO:teuthology.orchestra.run.smithi089.stdout:(64/176): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-02-09T22:34:30.120 INFO:teuthology.orchestra.run.smithi089.stdout:(65/176): python3-attrs-20.3.0-7.el9.noarch.rpm 425 kB/s | 78 kB 00:00 2024-02-09T22:34:30.212 INFO:teuthology.orchestra.run.smithi089.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 25 MB/s | 5.8 MB 00:00 2024-02-09T22:34:30.237 INFO:teuthology.orchestra.run.smithi089.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.7 MB/s | 227 kB 00:00 2024-02-09T22:34:30.262 INFO:teuthology.orchestra.run.smithi089.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-02-09T22:34:30.287 INFO:teuthology.orchestra.run.smithi089.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 241 kB/s | 18 kB 00:00 2024-02-09T22:34:30.312 INFO:teuthology.orchestra.run.smithi089.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-02-09T22:34:30.338 INFO:teuthology.orchestra.run.smithi089.stdout:(71/176): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-02-09T22:34:30.438 INFO:teuthology.orchestra.run.smithi089.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-02-09T22:34:30.463 INFO:teuthology.orchestra.run.smithi089.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-02-09T22:34:30.489 INFO:teuthology.orchestra.run.smithi089.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-02-09T22:34:30.514 INFO:teuthology.orchestra.run.smithi089.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-02-09T22:34:30.531 INFO:teuthology.orchestra.run.smithi089.stdout:(76/176): python3-prettytable-0.7.2-27.el9.noar 986 kB/s | 41 kB 00:00 2024-02-09T22:34:30.556 INFO:teuthology.orchestra.run.smithi089.stdout:(77/176): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-02-09T22:34:30.581 INFO:teuthology.orchestra.run.smithi089.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-02-09T22:34:30.598 INFO:teuthology.orchestra.run.smithi089.stdout:(79/176): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-02-09T22:34:30.623 INFO:teuthology.orchestra.run.smithi089.stdout:(80/176): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-02-09T22:34:30.640 INFO:teuthology.orchestra.run.smithi089.stdout:(81/176): python3-requests-oauthlib-1.3.0-12.el 729 kB/s | 43 kB 00:00 2024-02-09T22:34:30.841 INFO:teuthology.orchestra.run.smithi089.stdout:(82/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 208 kB/s | 41 kB 00:00 2024-02-09T22:34:30.866 INFO:teuthology.orchestra.run.smithi089.stdout:(83/176): python3-toml-0.10.2-6.el9.noarch.rpm 173 kB/s | 39 kB 00:00 2024-02-09T22:34:31.025 INFO:teuthology.orchestra.run.smithi089.stdout:(84/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-02-09T22:34:31.050 INFO:teuthology.orchestra.run.smithi089.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 215 kB 00:00 2024-02-09T22:34:31.075 INFO:teuthology.orchestra.run.smithi089.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 163 kB/s | 34 kB 00:00 2024-02-09T22:34:31.106 INFO:teuthology.orchestra.run.smithi089.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 783 kB/s | 43 kB 00:00 2024-02-09T22:34:31.131 INFO:teuthology.orchestra.run.smithi089.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 3.6 MB/s | 300 kB 00:00 2024-02-09T22:34:31.156 INFO:teuthology.orchestra.run.smithi089.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 701 kB/s | 56 kB 00:00 2024-02-09T22:34:31.181 INFO:teuthology.orchestra.run.smithi089.stdout:(90/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-02-09T22:34:31.207 INFO:teuthology.orchestra.run.smithi089.stdout:(91/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-02-09T22:34:31.232 INFO:teuthology.orchestra.run.smithi089.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 581 kB/s | 44 kB 00:00 2024-02-09T22:34:31.257 INFO:teuthology.orchestra.run.smithi089.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-02-09T22:34:31.283 INFO:teuthology.orchestra.run.smithi089.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 3.6 MB/s | 186 kB 00:00 2024-02-09T22:34:31.308 INFO:teuthology.orchestra.run.smithi089.stdout:(95/176): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 802 kB 00:00 2024-02-09T22:34:31.333 INFO:teuthology.orchestra.run.smithi089.stdout:(96/176): libquadmath-11.4.1-3.el9.x86_64.rpm 2.5 MB/s | 194 kB 00:00 2024-02-09T22:34:31.359 INFO:teuthology.orchestra.run.smithi089.stdout:(97/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-02-09T22:34:31.384 INFO:teuthology.orchestra.run.smithi089.stdout:(98/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-02-09T22:34:31.417 INFO:teuthology.orchestra.run.smithi089.stdout:(99/176): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-02-09T22:34:31.443 INFO:teuthology.orchestra.run.smithi089.stdout:(100/176): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-02-09T22:34:31.468 INFO:teuthology.orchestra.run.smithi089.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 2.4 MB/s | 124 kB 00:00 2024-02-09T22:34:31.493 INFO:teuthology.orchestra.run.smithi089.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.5 MB/s | 113 kB 00:00 2024-02-09T22:34:31.518 INFO:teuthology.orchestra.run.smithi089.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 2.4 MB/s | 187 kB 00:00 2024-02-09T22:34:31.594 INFO:teuthology.orchestra.run.smithi089.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 2.4 MB/s | 245 kB 00:00 2024-02-09T22:34:31.619 INFO:teuthology.orchestra.run.smithi089.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-02-09T22:34:31.844 INFO:teuthology.orchestra.run.smithi089.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.6 MB/s | 863 kB 00:00 2024-02-09T22:34:31.869 INFO:teuthology.orchestra.run.smithi089.stdout:(107/176): gperftools-libs-2.9.1-2.el9.x86_64.r 1.2 MB/s | 309 kB 00:00 2024-02-09T22:34:31.920 INFO:teuthology.orchestra.run.smithi089.stdout:(108/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-02-09T22:34:31.962 INFO:teuthology.orchestra.run.smithi089.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 644 kB/s | 26 kB 00:00 2024-02-09T22:34:32.004 INFO:teuthology.orchestra.run.smithi089.stdout:(110/176): grpc-data-1.46.7-10.el9.noarch.rpm 122 kB/s | 19 kB 00:00 2024-02-09T22:34:32.029 INFO:teuthology.orchestra.run.smithi089.stdout:(111/176): liboath-2.6.7-2.el9.x86_64.rpm 726 kB/s | 49 kB 00:00 2024-02-09T22:34:32.071 INFO:teuthology.orchestra.run.smithi089.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-02-09T22:34:32.113 INFO:teuthology.orchestra.run.smithi089.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-02-09T22:34:32.253 INFO:teuthology.orchestra.run.smithi003.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 3/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 4/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g29021 5/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 6/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 7/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 8/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 9/202 2024-02-09T22:34:32.254 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 10/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 13/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 14/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_ 15/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 17/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_6 18/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 19/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 20/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el 21/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9. 22/202 2024-02-09T22:34:32.255 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_6 23/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 24/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 25/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 26/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 27/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 28/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 29/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1. 30/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 31/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 32/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 33/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el 34/202 2024-02-09T22:34:32.256 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 35/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.e 36/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 37/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 38/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 39/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-02-09T22:34:32.257 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 59/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-02-09T22:34:32.258 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-02-09T22:34:32.259 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-02-09T22:34:32.260 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-02-09T22:34:32.261 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-02-09T22:34:32.262 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 116/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-02-09T22:34:32.263 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-02-09T22:34:32.264 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-02-09T22:34:32.265 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 151/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-02-09T22:34:32.266 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-1:1.45.90-1.el9.x86_64 157/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-libnm-1:1.45.90-1.el9.x86_64 159/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-team-1:1.45.90-1.el9.x86_64 161/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-tui-1:1.45.90-1.el9.x86_64 163/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-02-09T22:34:32.267 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-02-09T22:34:32.268 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-02-09T22:34:32.269 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 189/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 191/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-02-09T22:34:32.270 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-02-09T22:34:32.271 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-02-09T22:34:32.271 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-02-09T22:34:32.271 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-02-09T22:34:32.271 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-02-09T22:34:32.313 INFO:teuthology.orchestra.run.smithi089.stdout:(114/176): python3-asyncssh-2.13.2-3.el9.noarch 2.7 MB/s | 548 kB 00:00 2024-02-09T22:34:32.338 INFO:teuthology.orchestra.run.smithi089.stdout:(115/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.1 MB/s | 838 kB 00:00 2024-02-09T22:34:32.355 INFO:teuthology.orchestra.run.smithi089.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2024-02-09T22:34:32.372 INFO:teuthology.orchestra.run.smithi089.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 959 kB/s | 32 kB 00:00 2024-02-09T22:34:32.389 INFO:teuthology.orchestra.run.smithi089.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 447 kB/s | 15 kB 00:00 2024-02-09T22:34:32.456 INFO:teuthology.orchestra.run.smithi089.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-02-09T22:34:32.532 INFO:teuthology.orchestra.run.smithi089.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 2.5 MB/s | 358 kB 00:00 2024-02-09T22:34:32.573 INFO:teuthology.orchestra.run.smithi089.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-02-09T22:34:32.599 INFO:teuthology.orchestra.run.smithi089.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 527 kB/s | 35 kB 00:00 2024-02-09T22:34:32.649 INFO:teuthology.orchestra.run.smithi089.stdout:(123/176): python3-google-auth-2.27.0-1.el9.noa 2.6 MB/s | 203 kB 00:00 2024-02-09T22:34:32.708 INFO:teuthology.orchestra.run.smithi089.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-02-09T22:34:32.758 INFO:teuthology.orchestra.run.smithi089.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 214 kB/s | 11 kB 00:00 2024-02-09T22:34:32.808 INFO:teuthology.orchestra.run.smithi089.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 354 kB/s | 18 kB 00:00 2024-02-09T22:34:32.858 INFO:teuthology.orchestra.run.smithi089.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 465 kB/s | 23 kB 00:00 2024-02-09T22:34:32.908 INFO:teuthology.orchestra.run.smithi089.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 390 kB/s | 19 kB 00:00 2024-02-09T22:34:32.951 INFO:teuthology.orchestra.run.smithi089.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 473 kB/s | 20 kB 00:00 2024-02-09T22:34:33.234 INFO:teuthology.orchestra.run.smithi089.stdout:(130/176): libarrow-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 4.4 MB 00:01 2024-02-09T22:34:33.276 INFO:teuthology.orchestra.run.smithi089.stdout:(131/176): python3-grpcio-1.46.7-10.el9.x86_64. 3.0 MB/s | 2.0 MB 00:00 2024-02-09T22:34:33.301 INFO:teuthology.orchestra.run.smithi089.stdout:(132/176): python3-logutils-0.3.5-21.el9.noarch 692 kB/s | 46 kB 00:00 2024-02-09T22:34:33.335 INFO:teuthology.orchestra.run.smithi089.stdout:(133/176): python3-kubernetes-26.1.0-2.el9.noar 2.7 MB/s | 1.0 MB 00:00 2024-02-09T22:34:33.360 INFO:teuthology.orchestra.run.smithi089.stdout:(134/176): python3-natsort-7.1.1-5.el9.noarch.r 986 kB/s | 58 kB 00:00 2024-02-09T22:34:33.386 INFO:teuthology.orchestra.run.smithi089.stdout:(135/176): python3-more-itertools-8.12.0-2.el9. 723 kB/s | 79 kB 00:00 2024-02-09T22:34:33.428 INFO:teuthology.orchestra.run.smithi089.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 244 kB/s | 16 kB 00:00 2024-02-09T22:34:33.469 INFO:teuthology.orchestra.run.smithi089.stdout:(137/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.0 MB/s | 90 kB 00:00 2024-02-09T22:34:33.503 INFO:teuthology.orchestra.run.smithi089.stdout:(138/176): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-02-09T22:34:33.529 INFO:teuthology.orchestra.run.smithi089.stdout:(139/176): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-02-09T22:34:33.554 INFO:teuthology.orchestra.run.smithi089.stdout:(140/176): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-02-09T22:34:33.579 INFO:teuthology.orchestra.run.smithi089.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-02-09T22:34:33.606 INFO:teuthology.orchestra.run.smithi089.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 691 kB/s | 36 kB 00:00 2024-02-09T22:34:33.648 INFO:teuthology.orchestra.run.smithi089.stdout:(143/176): python3-typing-extensions-4.4.0-2.el 745 kB/s | 51 kB 00:00 2024-02-09T22:34:33.690 INFO:teuthology.orchestra.run.smithi089.stdout:(144/176): python3-routes-2.5.1-5.el9.noarch.rp 1.0 MB/s | 188 kB 00:00 2024-02-09T22:34:33.715 INFO:teuthology.orchestra.run.smithi089.stdout:(145/176): python3-websocket-client-1.2.3-2.el9 1.3 MB/s | 90 kB 00:00 2024-02-09T22:34:33.757 INFO:teuthology.orchestra.run.smithi089.stdout:(146/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.5 MB/s | 230 kB 00:00 2024-02-09T22:34:33.782 INFO:teuthology.orchestra.run.smithi089.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-02-09T22:34:33.858 INFO:teuthology.orchestra.run.smithi089.stdout:(148/176): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-02-09T22:34:33.966 INFO:teuthology.orchestra.run.smithi089.stdout:(149/176): python3-werkzeug-2.0.3-3.el9.1.noarc 1.5 MB/s | 427 kB 00:00 2024-02-09T22:34:34.266 INFO:teuthology.orchestra.run.smithi089.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 3.3 MB/s | 1.6 MB 00:00 2024-02-09T22:34:34.292 INFO:teuthology.orchestra.run.smithi089.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.6 MB/s | 68 kB 00:00 2024-02-09T22:34:34.375 INFO:teuthology.orchestra.run.smithi089.stdout:(152/176): NetworkManager-1.45.90-1.el9.x86_64. 27 MB/s | 2.3 MB 00:00 2024-02-09T22:34:34.442 INFO:teuthology.orchestra.run.smithi089.stdout:(153/176): NetworkManager-libnm-1.45.90-1.el9.x 27 MB/s | 1.8 MB 00:00 2024-02-09T22:34:34.476 INFO:teuthology.orchestra.run.smithi089.stdout:(154/176): NetworkManager-team-1.45.90-1.el9.x8 1.2 MB/s | 42 kB 00:00 2024-02-09T22:34:34.517 INFO:teuthology.orchestra.run.smithi089.stdout:(155/176): NetworkManager-tui-1.45.90-1.el9.x86 5.8 MB/s | 247 kB 00:00 2024-02-09T22:34:34.576 INFO:teuthology.orchestra.run.smithi089.stdout:(156/176): librados2-19.0.0-1361.g290212a1.el9. 4.7 MB/s | 3.4 MB 00:00 2024-02-09T22:34:34.593 INFO:teuthology.orchestra.run.smithi089.stdout:(157/176): acl-2.3.1-4.el9.x86_64.rpm 914 kB/s | 69 kB 00:00 2024-02-09T22:34:34.618 INFO:teuthology.orchestra.run.smithi089.stdout:(158/176): elfutils-debuginfod-client-0.190-2.e 844 kB/s | 35 kB 00:00 2024-02-09T22:34:34.644 INFO:teuthology.orchestra.run.smithi089.stdout:(159/176): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 191 kB 00:00 2024-02-09T22:34:34.669 INFO:teuthology.orchestra.run.smithi089.stdout:(160/176): elfutils-libs-0.190-2.el9.x86_64.rpm 5.0 MB/s | 254 kB 00:00 2024-02-09T22:34:34.694 INFO:teuthology.orchestra.run.smithi089.stdout:(161/176): iproute-6.7.0-2.el9.x86_64.rpm 16 MB/s | 802 kB 00:00 2024-02-09T22:34:34.719 INFO:teuthology.orchestra.run.smithi089.stdout:(162/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.7 MB/s | 433 kB 00:00 2024-02-09T22:34:34.745 INFO:teuthology.orchestra.run.smithi089.stdout:(163/176): libacl-2.3.1-4.el9.x86_64.rpm 443 kB/s | 22 kB 00:00 2024-02-09T22:34:34.770 INFO:teuthology.orchestra.run.smithi089.stdout:(164/176): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 107 kB 00:00 2024-02-09T22:34:34.795 INFO:teuthology.orchestra.run.smithi089.stdout:(165/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-02-09T22:34:34.821 INFO:teuthology.orchestra.run.smithi089.stdout:(166/176): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 135 kB 00:00 2024-02-09T22:34:34.846 INFO:teuthology.orchestra.run.smithi089.stdout:(167/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 63 kB 00:00 2024-02-09T22:34:34.871 INFO:teuthology.orchestra.run.smithi089.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 551 kB/s | 28 kB 00:00 2024-02-09T22:34:34.913 INFO:teuthology.orchestra.run.smithi089.stdout:(169/176): openssl-3.0.7-25.el9.x86_64.rpm 17 MB/s | 1.2 MB 00:00 2024-02-09T22:34:34.955 INFO:teuthology.orchestra.run.smithi089.stdout:(170/176): openssl-libs-3.0.7-25.el9.x86_64.rpm 26 MB/s | 2.1 MB 00:00 2024-02-09T22:34:34.997 INFO:teuthology.orchestra.run.smithi089.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 27 MB/s | 2.2 MB 00:00 2024-02-09T22:34:35.023 INFO:teuthology.orchestra.run.smithi089.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 6.7 MB/s | 438 kB 00:00 2024-02-09T22:34:35.048 INFO:teuthology.orchestra.run.smithi089.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 609 kB/s | 30 kB 00:00 2024-02-09T22:34:35.081 INFO:teuthology.orchestra.run.smithi089.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2024-02-09T22:34:35.100 INFO:teuthology.orchestra.run.smithi003.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout:Upgraded: 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-1:1.45.90-1.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-libnm-1:1.45.90-1.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-team-1:1.45.90-1.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: NetworkManager-tui-1:1.45.90-1.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: acl-2.3.1-4.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: iproute-6.7.0-2.el9.x86_64 2024-02-09T22:34:35.101 INFO:teuthology.orchestra.run.smithi003.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libblkid-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libmount-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.102 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout:Installed: 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: automake-1.16.2-8.el9.noarch 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: bison-3.7.4-5.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.103 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.104 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: flex-2.6.4-9.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-09T22:34:35.105 INFO:teuthology.orchestra.run.smithi003.stdout: jq-1.6-16.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.106 INFO:teuthology.orchestra.run.smithi003.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: lua-5.4.4-4.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-09T22:34:35.107 INFO:teuthology.orchestra.run.smithi003.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-13.11-1.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.108 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-09T22:34:35.109 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-09T22:34:35.110 INFO:teuthology.orchestra.run.smithi003.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-09T22:34:35.111 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-09T22:34:35.111 INFO:teuthology.orchestra.run.smithi003.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-09T22:34:35.111 INFO:teuthology.orchestra.run.smithi003.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-09T22:34:35.111 INFO:teuthology.orchestra.run.smithi003.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-09T22:34:35.111 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-09T22:34:35.111 INFO:teuthology.orchestra.run.smithi003.stdout: python3-ply-3.11-14.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-09T22:34:35.112 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-09T22:34:35.113 INFO:teuthology.orchestra.run.smithi003.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: re2-1:20211101-3.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-09T22:34:35.114 INFO:teuthology.orchestra.run.smithi003.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-09T22:34:35.115 INFO:teuthology.orchestra.run.smithi003.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-09T22:34:35.115 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:34:35.115 INFO:teuthology.orchestra.run.smithi003.stdout:Complete! 2024-02-09T22:34:35.115 INFO:teuthology.orchestra.run.smithi089.stdout:(175/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 16 MB/s | 1.1 MB 00:00 2024-02-09T22:34:35.265 INFO:teuthology.orchestra.run.smithi089.stdout:(176/176): librbd1-19.0.0-1361.g290212a1.el9.x8 2.4 MB/s | 3.1 MB 00:01 2024-02-09T22:34:35.272 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:34:35.273 INFO:teuthology.orchestra.run.smithi089.stdout:Total 18 MB/s | 222 MB 00:12 2024-02-09T22:34:35.320 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-02-09T22:34:35.596 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-02-09T22:34:35.596 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-02-09T22:34:35.792 DEBUG:teuthology.parallel:result is None 2024-02-09T22:34:37.261 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-02-09T22:34:37.261 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-02-09T22:34:44.212 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-02-09T22:34:44.335 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/202 2024-02-09T22:34:44.369 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-02-09T22:34:44.400 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-09T22:34:44.426 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-09T22:34:44.479 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-02-09T22:34:44.517 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-02-09T22:34:44.570 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-02-09T22:34:44.604 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-02-09T22:34:44.631 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-02-09T22:34:44.672 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-02-09T22:34:44.991 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-02-09T22:34:45.040 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:34:45.143 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:34:45.174 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:34:45.249 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:34:45.278 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 13/202 2024-02-09T22:34:45.434 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-02-09T22:34:45.486 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-02-09T22:34:45.514 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:34:45.617 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:34:45.640 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 17/202 2024-02-09T22:34:45.686 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 17/202 2024-02-09T22:34:45.727 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-02-09T22:34:45.750 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-09T22:34:45.764 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-09T22:34:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-09T22:34:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-09T22:34:46.029 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-02-09T22:34:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-02-09T22:34:46.232 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-libnm-1:1.45.90-1.el9.x86_64 23/202 2024-02-09T22:34:46.276 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.90-1.el9.x86_64 23/202 2024-02-09T22:34:46.338 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:34:46.394 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:34:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:34:46.737 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-02-09T22:34:47.048 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-02-09T22:34:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-02-09T22:34:47.160 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-02-09T22:34:47.215 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-02-09T22:34:47.245 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-02-09T22:34:47.280 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-02-09T22:34:47.321 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-02-09T22:34:47.386 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-02-09T22:34:47.428 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-02-09T22:34:47.471 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-02-09T22:34:47.504 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-02-09T22:34:47.532 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-02-09T22:34:47.602 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-02-09T22:34:47.645 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9. 39/202 2024-02-09T22:34:47.689 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el 40/202 2024-02-09T22:34:47.718 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_6 41/202 2024-02-09T22:34:47.788 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-02-09T22:34:47.832 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-02-09T22:34:47.916 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-02-09T22:34:48.059 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-02-09T22:34:48.111 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-02-09T22:34:48.634 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-02-09T22:34:48.706 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-02-09T22:34:48.793 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-02-09T22:34:48.851 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-09T22:34:48.859 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-09T22:34:48.899 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-02-09T22:34:48.936 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_6 52/202 2024-02-09T22:34:48.967 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-02-09T22:34:49.058 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-02-09T22:34:49.110 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-02-09T22:34:49.353 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-02-09T22:34:49.514 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-02-09T22:34:49.572 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-02-09T22:34:49.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-02-09T22:34:50.009 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-02-09T22:34:50.079 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 61/202 2024-02-09T22:34:50.140 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-02-09T22:34:50.209 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-02-09T22:34:50.249 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-02-09T22:34:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-02-09T22:34:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-02-09T22:34:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-02-09T22:34:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-02-09T22:34:50.451 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-02-09T22:34:50.586 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-02-09T22:34:50.640 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-02-09T22:34:50.670 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-02-09T22:34:50.711 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-02-09T22:34:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-02-09T22:34:50.845 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 75/202 2024-02-09T22:34:50.886 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-02-09T22:34:50.912 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-02-09T22:34:50.936 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-02-09T22:34:51.008 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-02-09T22:34:51.138 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-02-09T22:34:51.238 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-02-09T22:34:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-02-09T22:34:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-02-09T22:34:51.407 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-02-09T22:34:51.592 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-02-09T22:34:51.641 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-02-09T22:34:51.770 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-02-09T22:34:51.815 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 88/202 2024-02-09T22:34:51.848 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-02-09T22:34:51.879 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-02-09T22:34:51.915 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-02-09T22:34:51.946 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-02-09T22:34:51.974 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-02-09T22:34:52.008 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-02-09T22:34:52.048 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-02-09T22:34:52.079 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-02-09T22:34:52.110 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-02-09T22:34:52.177 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-02-09T22:34:52.220 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-02-09T22:34:52.253 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-02-09T22:34:52.294 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-02-09T22:34:52.329 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-02-09T22:34:52.418 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-02-09T22:34:52.508 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-02-09T22:34:52.552 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-02-09T22:34:52.592 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-02-09T22:34:52.639 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-02-09T22:34:52.736 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-02-09T22:34:52.794 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-02-09T22:34:53.509 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-02-09T22:34:53.558 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-02-09T22:34:53.592 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-02-09T22:34:53.641 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-02-09T22:34:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-02-09T22:34:53.777 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-02-09T22:34:53.953 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-02-09T22:34:53.983 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-02-09T22:34:54.068 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-02-09T22:34:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-02-09T22:34:54.806 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-02-09T22:34:56.001 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-02-09T22:34:56.081 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-02-09T22:34:56.108 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-02-09T22:34:56.417 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-02-09T22:34:56.446 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 125/202 2024-02-09T22:34:56.527 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 125/202 2024-02-09T22:34:56.559 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 126/202 2024-02-09T22:34:56.595 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-02-09T22:34:57.056 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-02-09T22:34:57.083 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 129/202 2024-02-09T22:34:57.162 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 129/202 2024-02-09T22:34:57.184 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 130/202 2024-02-09T22:34:59.719 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:34:59.765 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:35:00.302 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:35:00.336 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 132/202 2024-02-09T22:35:00.429 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 132/202 2024-02-09T22:35:00.429 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-09T22:35:00.429 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-09T22:35:00.429 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:00.447 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 133/202 2024-02-09T22:35:12.040 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 133/202 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-02-09T22:35:12.041 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 134/202 2024-02-09T22:35:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 134/202 2024-02-09T22:35:12.222 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:12.222 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-09T22:35:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:13.718 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 135/202 2024-02-09T22:35:13.743 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 135/202 2024-02-09T22:35:13.743 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:13.743 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-09T22:35:13.743 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:35:13.743 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:35:13.743 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:13.952 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 136/202 2024-02-09T22:35:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 136/202 2024-02-09T22:35:13.974 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:13.974 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-09T22:35:13.974 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:35:13.974 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:35:13.974 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:14.411 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 137/202 2024-02-09T22:35:14.433 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 137/202 2024-02-09T22:35:14.434 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:14.434 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-09T22:35:14.434 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:35:14.434 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:35:14.434 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:14.529 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-02-09T22:35:14.947 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 139/202 2024-02-09T22:35:14.978 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 139/202 2024-02-09T22:35:15.008 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.e 140/202 2024-02-09T22:35:15.069 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1. 141/202 2024-02-09T22:35:15.094 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 142/202 2024-02-09T22:35:15.322 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 142/202 2024-02-09T22:35:15.361 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 143/202 2024-02-09T22:35:16.326 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 143/202 2024-02-09T22:35:16.353 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 144/202 2024-02-09T22:35:16.452 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 144/202 2024-02-09T22:35:16.623 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el 145/202 2024-02-09T22:35:16.655 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 146/202 2024-02-09T22:35:16.678 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 146/202 2024-02-09T22:35:16.678 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:16.678 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-09T22:35:16.678 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:35:16.678 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:35:16.678 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:16.723 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 147/202 2024-02-09T22:35:16.733 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 147/202 2024-02-09T22:35:16.779 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 148/202 2024-02-09T22:35:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-02-09T22:35:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 150/202 2024-02-09T22:35:17.681 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 150/202 2024-02-09T22:35:17.681 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:17.681 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-09T22:35:17.681 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:35:17.681 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:35:17.681 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:17.728 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g29021 151/202 2024-02-09T22:35:17.749 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g29021 151/202 2024-02-09T22:35:17.749 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:17.749 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-09T22:35:17.749 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:17.787 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 152/202 2024-02-09T22:35:17.809 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 152/202 2024-02-09T22:35:17.809 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:17.809 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-09T22:35:17.809 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-09T22:35:17.809 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-09T22:35:17.809 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:18.121 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:35:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:35:18.143 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:35:18.143 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-09T22:35:18.143 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:35:18.143 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:35:18.143 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 154/202 2024-02-09T22:35:22.567 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 155/202 2024-02-09T22:35:22.606 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 156/202 2024-02-09T22:35:22.633 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-02-09T22:35:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 158/202 2024-02-09T22:35:22.941 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-02-09T22:35:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 160/202 2024-02-09T22:35:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-02-09T22:35:23.074 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-02-09T22:35:23.163 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-02-09T22:35:23.210 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-02-09T22:35:23.253 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-02-09T22:35:23.283 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_ 166/202 2024-02-09T22:35:23.313 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-team-1:1.45.90-1.el9.x86_64 167/202 2024-02-09T22:35:23.369 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : NetworkManager-tui-1:1.45.90-1.el9.x86_64 168/202 2024-02-09T22:35:23.418 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-02-09T22:35:23.454 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-02-09T22:35:23.988 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:35:24.028 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:35:24.048 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:35:24.856 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 172/202 2024-02-09T22:35:24.933 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 173/202 2024-02-09T22:35:24.973 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-02-09T22:35:25.052 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-02-09T22:35:25.103 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-02-09T22:35:25.128 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-02-09T22:35:25.162 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-02-09T22:35:25.189 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-02-09T22:35:25.189 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-09T22:35:25.273 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-09T22:35:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-02-09T22:35:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-02-09T22:35:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-09T22:35:25.416 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-09T22:35:25.442 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-02-09T22:35:25.476 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-02-09T22:35:25.510 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-02-09T22:35:25.553 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-02-09T22:35:25.586 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-02-09T22:35:25.619 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-02-09T22:35:25.676 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-02-09T22:35:25.717 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-02-09T22:35:25.742 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-02-09T22:35:25.767 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-02-09T22:35:25.793 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-02-09T22:35:25.818 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-02-09T22:35:25.824 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:35:25.824 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:35:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:35:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-09T22:35:26.088 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-09T22:35:26.112 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-02-09T22:35:26.138 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-02-09T22:35:26.163 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-02-09T22:35:26.196 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-02-09T22:35:26.261 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-09T22:35:31.151 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-09T22:35:31.151 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/202 2024-02-09T22:35:31.151 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2/202 2024-02-09T22:35:31.151 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 3/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 4/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g29021 5/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 6/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 7/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 8/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 9/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 10/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 13/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 14/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_ 15/202 2024-02-09T22:35:31.152 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 17/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_6 18/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 19/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 20/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el 21/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9. 22/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_6 23/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 24/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 25/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 26/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 27/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 28/202 2024-02-09T22:35:31.153 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 29/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1. 30/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 31/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 32/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 33/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el 34/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 35/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.e 36/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 37/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 38/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 39/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-02-09T22:35:31.154 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-02-09T22:35:31.155 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 59/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-02-09T22:35:31.156 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-02-09T22:35:31.157 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-02-09T22:35:31.158 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-02-09T22:35:31.159 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 116/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-02-09T22:35:31.160 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-02-09T22:35:31.161 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-02-09T22:35:31.162 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 151/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.45.90-1.el9.x86_64 157/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.45.90-1.el9.x86_64 159/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.45.90-1.el9.x86_64 161/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.45.90-1.el9.x86_64 163/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-02-09T22:35:31.163 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-02-09T22:35:31.164 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 189/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 191/202 2024-02-09T22:35:31.165 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-02-09T22:35:31.166 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-02-09T22:35:33.978 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-02-09T22:35:33.978 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:33.978 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-02-09T22:35:33.978 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-1:1.45.90-1.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-libnm-1:1.45.90-1.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-team-1:1.45.90-1.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: NetworkManager-tui-1:1.45.90-1.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: acl-2.3.1-4.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: iproute-6.7.0-2.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: libblkid-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.979 INFO:teuthology.orchestra.run.smithi089.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: libmount-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-02-09T22:35:33.980 INFO:teuthology.orchestra.run.smithi089.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: automake-1.16.2-8.el9.noarch 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: bison-3.7.4-5.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.981 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.982 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: flex-2.6.4-9.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: jq-1.6-16.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-09T22:35:33.983 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-09T22:35:33.984 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: lua-5.4.4-4.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-13.11-1.el9.x86_64 2024-02-09T22:35:33.985 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-09T22:35:33.986 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-09T22:35:33.987 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-09T22:35:33.988 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-09T22:35:33.989 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-09T22:35:33.990 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-3.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-09T22:35:33.991 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-09T22:35:33.992 INFO:teuthology.orchestra.run.smithi089.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-09T22:35:33.992 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:35:33.992 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-02-09T22:35:34.580 DEBUG:teuthology.parallel:result is None 2024-02-09T22:38:43.276 INFO:teuthology.orchestra.run.smithi064.stdout:Extra Packages for Enterprise Linux 58 kB/s | 20 MB 06:00 2024-02-09T22:38:48.499 INFO:teuthology.orchestra.run.smithi064.stdout:lab-extras 38 kB/s | 1.7 kB 00:00 2024-02-09T22:38:51.206 INFO:teuthology.orchestra.run.smithi064.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-09T22:38:51.207 INFO:teuthology.orchestra.run.smithi064.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-02-09T22:38:51.222 INFO:teuthology.orchestra.run.smithi064.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-02-09T22:38:51.223 INFO:teuthology.orchestra.run.smithi064.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-02-09T22:38:51.225 INFO:teuthology.orchestra.run.smithi064.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-02-09T22:38:51.226 INFO:teuthology.orchestra.run.smithi064.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-02-09T22:38:51.228 INFO:teuthology.orchestra.run.smithi064.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-02-09T22:38:51.231 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-02-09T22:38:51.232 INFO:teuthology.orchestra.run.smithi064.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-02-09T22:38:51.321 INFO:teuthology.orchestra.run.smithi064.stdout:Dependencies resolved. 2024-02-09T22:38:51.352 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-02-09T22:38:51.352 INFO:teuthology.orchestra.run.smithi064.stdout: Package Arch Version Repository Size 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout:Installing: 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph x86_64 2:19.0.0-1361.g290212a1.el9 ceph 6.6 k 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.3 M 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 885 k 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1361.g290212a1.el9 ceph 145 k 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr x86_64 2:19.0.0-1361.g290212a1.el9 ceph 1.5 M 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 138 k 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 1.8 M 2024-02-09T22:38:51.353 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 7.4 M 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 52 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 9.7 M 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test x86_64 2:19.0.0-1361.g290212a1.el9 ceph 47 M 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 264 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 748 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 217 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell x86_64 2:19.0.0-1361.g290212a1.el9 ceph 27 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 18 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 CentOS-AppStream 20 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-02-09T22:38:51.354 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 31 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 743 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel x86_64 2:19.0.0-1361.g290212a1.el9 ceph 127 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel x86_64 2.37.4-18.el9 CentOS-AppStream 24 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel x86_64 1:3.0.7-25.el9 CentOS-AppStream 3.0 M 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs x86_64 2:19.0.0-1361.g290212a1.el9 ceph 161 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados x86_64 2:19.0.0-1361.g290212a1.el9 ceph 319 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 298 k 2024-02-09T22:38:51.355 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw x86_64 2:19.0.0-1361.g290212a1.el9 ceph 99 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 85 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 171 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 CentOS-AppStream 87 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrading: 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager x86_64 1:1.45.90-1.el9 CentOS-BaseOS 2.3 M 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-libnm x86_64 1:1.45.90-1.el9 CentOS-BaseOS 1.8 M 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-team x86_64 1:1.45.90-1.el9 CentOS-BaseOS 42 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-tui x86_64 1:1.45.90-1.el9 CentOS-BaseOS 247 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 CentOS-BaseOS 35 k 2024-02-09T22:38:51.356 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf x86_64 0.190-2.el9 CentOS-BaseOS 191 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libs x86_64 0.190-2.el9 CentOS-BaseOS 254 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: iproute x86_64 6.7.0-2.el9 CentOS-BaseOS 802 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: iproute-tc x86_64 6.7.0-2.el9 CentOS-BaseOS 433 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libblkid x86_64 2.37.4-18.el9 CentOS-BaseOS 107 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libfdisk x86_64 2.37.4-18.el9 CentOS-BaseOS 154 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libmount x86_64 2.37.4-18.el9 CentOS-BaseOS 135 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: librados2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.4 M 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 3.1 M 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libsmartcols x86_64 2.37.4-18.el9 CentOS-BaseOS 63 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid x86_64 2.37.4-18.el9 CentOS-BaseOS 28 k 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: openssl x86_64 1:3.0.7-25.el9 CentOS-BaseOS 1.2 M 2024-02-09T22:38:51.357 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-libs x86_64 1:3.0.7-25.el9 CentOS-BaseOS 2.1 M 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: util-linux x86_64 2.37.4-18.el9 CentOS-BaseOS 2.2 M 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: util-linux-core x86_64 2.37.4-18.el9 CentOS-BaseOS 438 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: util-linux-user x86_64 2.37.4-18.el9 CentOS-BaseOS 30 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs x86_64 6.3.0-1.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout:Installing dependencies: 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 20 M 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 28 k 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds x86_64 2:19.0.0-1361.g290212a1.el9 ceph 2.1 M 2024-02-09T22:38:51.358 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 247 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon x86_64 2:19.0.0-1361.g290212a1.el9 ceph 4.7 M 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd x86_64 2:19.0.0-1361.g290212a1.el9 ceph 17 M 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1361.g290212a1.el9 ceph-noarch 15 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux x86_64 2:19.0.0-1361.g290212a1.el9 ceph 25 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: jq x86_64 1.6-16.el9 CentOS-BaseOS 186 k 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-02-09T22:38:51.359 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite x86_64 2:19.0.0-1361.g290212a1.el9 ceph 163 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran x86_64 11.4.1-3.el9 CentOS-BaseOS 802 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath x86_64 11.4.1-3.el9 CentOS-BaseOS 194 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 499 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2 x86_64 2:19.0.0-1361.g290212a1.el9 ceph 5.1 M 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-02-09T22:38:51.360 INFO:teuthology.orchestra.run.smithi064.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-BaseOS 217 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-02-09T22:38:51.361 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse x86_64 2:19.0.0-1361.g290212a1.el9 ceph 45 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common x86_64 2:19.0.0-1361.g290212a1.el9 ceph 131 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth noarch 1:2.27.0-1.el9 epel 203 k 2024-02-09T22:38:51.362 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2 noarch 2.11.3-5.el9 CentOS-AppStream 227 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-02-09T22:38:51.363 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-02-09T22:38:51.364 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests noarch 2.25.1-8.el9 CentOS-BaseOS 113 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-02-09T22:38:51.365 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib x86_64 23.11.0-1.el9 CentOS-AppStream 215 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs x86_64 1.1.2-2.el9 CentOS-AppStream 43 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout:Installing weak dependencies: 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service x86_64 23.11.0-1.el9 CentOS-AppStream 34 k 2024-02-09T22:38:51.366 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction Summary 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout:============================================================================================ 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout:Install 150 Packages 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout:Upgrade 26 Packages 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout:Total download size: 222 M 2024-02-09T22:38:51.367 INFO:teuthology.orchestra.run.smithi064.stdout:Downloading Packages: 2024-02-09T22:38:51.776 INFO:teuthology.orchestra.run.smithi064.stdout:(1/176): ceph-19.0.0-1361.g290212a1.el9.x86_64. 36 kB/s | 6.6 kB 00:00 2024-02-09T22:38:51.959 INFO:teuthology.orchestra.run.smithi064.stdout:(2/176): ceph-fuse-19.0.0-1361.g290212a1.el9.x8 4.7 MB/s | 885 kB 00:00 2024-02-09T22:38:52.026 INFO:teuthology.orchestra.run.smithi064.stdout:(3/176): ceph-immutable-object-cache-19.0.0-136 2.1 MB/s | 145 kB 00:00 2024-02-09T22:38:52.310 INFO:teuthology.orchestra.run.smithi064.stdout:(4/176): ceph-base-19.0.0-1361.g290212a1.el9.x8 7.5 MB/s | 5.3 MB 00:00 2024-02-09T22:38:52.360 INFO:teuthology.orchestra.run.smithi064.stdout:(5/176): ceph-mds-19.0.0-1361.g290212a1.el9.x86 6.4 MB/s | 2.1 MB 00:00 2024-02-09T22:38:52.469 INFO:teuthology.orchestra.run.smithi064.stdout:(6/176): ceph-mgr-19.0.0-1361.g290212a1.el9.x86 9.4 MB/s | 1.5 MB 00:00 2024-02-09T22:38:53.119 INFO:teuthology.orchestra.run.smithi064.stdout:(7/176): ceph-mon-19.0.0-1361.g290212a1.el9.x86 6.2 MB/s | 4.7 MB 00:00 2024-02-09T22:38:53.753 INFO:teuthology.orchestra.run.smithi064.stdout:(8/176): ceph-osd-19.0.0-1361.g290212a1.el9.x86 13 MB/s | 17 MB 00:01 2024-02-09T22:38:53.811 INFO:teuthology.orchestra.run.smithi064.stdout:(9/176): ceph-selinux-19.0.0-1361.g290212a1.el9 432 kB/s | 25 kB 00:00 2024-02-09T22:38:54.287 INFO:teuthology.orchestra.run.smithi064.stdout:(10/176): ceph-common-19.0.0-1361.g290212a1.el9 7.4 MB/s | 20 MB 00:02 2024-02-09T22:38:54.354 INFO:teuthology.orchestra.run.smithi064.stdout:(11/176): cephfs-mirror-19.0.0-1361.g290212a1.e 3.2 MB/s | 217 kB 00:00 2024-02-09T22:38:54.429 INFO:teuthology.orchestra.run.smithi064.stdout:(12/176): cephfs-shell-19.0.0-1361.g290212a1.el 362 kB/s | 27 kB 00:00 2024-02-09T22:38:54.504 INFO:teuthology.orchestra.run.smithi064.stdout:(13/176): libcephfs-devel-19.0.0-1361.g290212a1 415 kB/s | 31 kB 00:00 2024-02-09T22:38:54.621 INFO:teuthology.orchestra.run.smithi064.stdout:(14/176): ceph-radosgw-19.0.0-1361.g290212a1.el 6.5 MB/s | 9.7 MB 00:01 2024-02-09T22:38:54.664 INFO:teuthology.orchestra.run.smithi064.stdout:(15/176): libcephfs2-19.0.0-1361.g290212a1.el9. 4.6 MB/s | 743 kB 00:00 2024-02-09T22:38:54.688 INFO:teuthology.orchestra.run.smithi064.stdout:(16/176): libcephsqlite-19.0.0-1361.g290212a1.e 2.4 MB/s | 163 kB 00:00 2024-02-09T22:38:54.738 INFO:teuthology.orchestra.run.smithi064.stdout:(17/176): librados-devel-19.0.0-1361.g290212a1. 1.6 MB/s | 127 kB 00:00 2024-02-09T22:38:54.805 INFO:teuthology.orchestra.run.smithi064.stdout:(18/176): libradosstriper1-19.0.0-1361.g290212a 4.2 MB/s | 499 kB 00:00 2024-02-09T22:38:54.888 INFO:teuthology.orchestra.run.smithi064.stdout:(19/176): python3-ceph-argparse-19.0.0-1361.g29 542 kB/s | 45 kB 00:00 2024-02-09T22:38:54.972 INFO:teuthology.orchestra.run.smithi064.stdout:(20/176): python3-ceph-common-19.0.0-1361.g2902 1.5 MB/s | 131 kB 00:00 2024-02-09T22:38:55.056 INFO:teuthology.orchestra.run.smithi064.stdout:(21/176): python3-cephfs-19.0.0-1361.g290212a1. 1.9 MB/s | 161 kB 00:00 2024-02-09T22:38:55.148 INFO:teuthology.orchestra.run.smithi064.stdout:(22/176): python3-rados-19.0.0-1361.g290212a1.e 3.4 MB/s | 319 kB 00:00 2024-02-09T22:38:55.256 INFO:teuthology.orchestra.run.smithi064.stdout:(23/176): python3-rbd-19.0.0-1361.g290212a1.el9 2.7 MB/s | 298 kB 00:00 2024-02-09T22:38:55.381 INFO:teuthology.orchestra.run.smithi064.stdout:(24/176): librgw2-19.0.0-1361.g290212a1.el9.x86 7.9 MB/s | 5.1 MB 00:00 2024-02-09T22:38:55.407 INFO:teuthology.orchestra.run.smithi064.stdout:(25/176): python3-rgw-19.0.0-1361.g290212a1.el9 660 kB/s | 99 kB 00:00 2024-02-09T22:38:55.440 INFO:teuthology.orchestra.run.smithi064.stdout:(26/176): rbd-fuse-19.0.0-1361.g290212a1.el9.x8 1.4 MB/s | 85 kB 00:00 2024-02-09T22:38:55.524 INFO:teuthology.orchestra.run.smithi064.stdout:(27/176): rbd-nbd-19.0.0-1361.g290212a1.el9.x86 2.0 MB/s | 171 kB 00:00 2024-02-09T22:38:55.608 INFO:teuthology.orchestra.run.smithi064.stdout:(28/176): ceph-grafana-dashboards-19.0.0-1361.g 340 kB/s | 28 kB 00:00 2024-02-09T22:38:55.700 INFO:teuthology.orchestra.run.smithi064.stdout:(29/176): ceph-mgr-cephadm-19.0.0-1361.g290212a 1.5 MB/s | 138 kB 00:00 2024-02-09T22:38:55.934 INFO:teuthology.orchestra.run.smithi064.stdout:(30/176): rbd-mirror-19.0.0-1361.g290212a1.el9. 5.9 MB/s | 3.1 MB 00:00 2024-02-09T22:38:55.975 INFO:teuthology.orchestra.run.smithi064.stdout:(31/176): ceph-mgr-dashboard-19.0.0-1361.g29021 6.4 MB/s | 1.8 MB 00:00 2024-02-09T22:38:56.076 INFO:teuthology.orchestra.run.smithi064.stdout:(32/176): ceph-mgr-modules-core-19.0.0-1361.g29 2.4 MB/s | 247 kB 00:00 2024-02-09T22:38:56.159 INFO:teuthology.orchestra.run.smithi064.stdout:(33/176): ceph-mgr-rook-19.0.0-1361.g290212a1.e 629 kB/s | 52 kB 00:00 2024-02-09T22:38:56.243 INFO:teuthology.orchestra.run.smithi064.stdout:(34/176): ceph-prometheus-alerts-19.0.0-1361.g2 183 kB/s | 15 kB 00:00 2024-02-09T22:38:56.718 INFO:teuthology.orchestra.run.smithi064.stdout:(35/176): ceph-test-19.0.0-1361.g290212a1.el9.x 16 MB/s | 47 MB 00:02 2024-02-09T22:38:56.743 INFO:teuthology.orchestra.run.smithi064.stdout:(36/176): ceph-volume-19.0.0-1361.g290212a1.el9 527 kB/s | 264 kB 00:00 2024-02-09T22:38:56.802 INFO:teuthology.orchestra.run.smithi064.stdout:(37/176): cephfs-top-19.0.0-1361.g290212a1.el9. 312 kB/s | 18 kB 00:00 2024-02-09T22:38:56.836 INFO:teuthology.orchestra.run.smithi064.stdout:(38/176): cephadm-19.0.0-1361.g290212a1.el9.noa 6.2 MB/s | 748 kB 00:00 2024-02-09T22:38:56.928 INFO:teuthology.orchestra.run.smithi064.stdout:(39/176): ceph-mgr-diskprediction-local-19.0.0- 7.4 MB/s | 7.4 MB 00:00 2024-02-09T22:38:57.078 INFO:teuthology.orchestra.run.smithi064.stdout:(40/176): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 106 kB 00:00 2024-02-09T22:38:57.112 INFO:teuthology.orchestra.run.smithi064.stdout:(41/176): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 662 kB 00:00 2024-02-09T22:38:57.145 INFO:teuthology.orchestra.run.smithi064.stdout:(42/176): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 920 kB 00:00 2024-02-09T22:38:57.170 INFO:teuthology.orchestra.run.smithi064.stdout:(43/176): elfutils-libelf-devel-0.190-2.el9.x86 225 kB/s | 20 kB 00:00 2024-02-09T22:38:57.195 INFO:teuthology.orchestra.run.smithi064.stdout:(44/176): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-02-09T22:38:57.221 INFO:teuthology.orchestra.run.smithi064.stdout:(45/176): flexiblas-3.0.4-8.el9.x86_64.rpm 637 kB/s | 32 kB 00:00 2024-02-09T22:38:57.298 INFO:teuthology.orchestra.run.smithi064.stdout:(46/176): flexiblas-netlib-3.0.4-8.el9.x86_64.r 23 MB/s | 3.0 MB 00:00 2024-02-09T22:38:57.315 INFO:teuthology.orchestra.run.smithi064.stdout:(47/176): flexiblas-openblas-openmp-3.0.4-8.el9 145 kB/s | 17 kB 00:00 2024-02-09T22:38:57.332 INFO:teuthology.orchestra.run.smithi064.stdout:(48/176): libaio-devel-0.3.111-13.el9.x86_64.rp 92 kB/s | 10 kB 00:00 2024-02-09T22:38:57.357 INFO:teuthology.orchestra.run.smithi064.stdout:(49/176): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-02-09T22:38:57.375 INFO:teuthology.orchestra.run.smithi064.stdout:(50/176): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-02-09T22:38:57.408 INFO:teuthology.orchestra.run.smithi064.stdout:(51/176): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-02-09T22:38:57.434 INFO:teuthology.orchestra.run.smithi064.stdout:(52/176): libtool-2.4.6-45.el9.x86_64.rpm 7.5 MB/s | 578 kB 00:00 2024-02-09T22:38:57.459 INFO:teuthology.orchestra.run.smithi064.stdout:(53/176): libuuid-devel-2.37.4-18.el9.x86_64.rp 493 kB/s | 24 kB 00:00 2024-02-09T22:38:57.484 INFO:teuthology.orchestra.run.smithi064.stdout:(54/176): libzstd-devel-1.5.1-2.el9.x86_64.rpm 624 kB/s | 46 kB 00:00 2024-02-09T22:38:57.518 INFO:teuthology.orchestra.run.smithi064.stdout:(55/176): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-02-09T22:38:57.543 INFO:teuthology.orchestra.run.smithi064.stdout:(56/176): openblas-0.3.21-2.el9.x86_64.rpm 590 kB/s | 34 kB 00:00 2024-02-09T22:38:57.568 INFO:teuthology.orchestra.run.smithi064.stdout:(57/176): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-02-09T22:38:57.677 INFO:teuthology.orchestra.run.smithi064.stdout:(58/176): openblas-openmp-0.3.21-2.el9.x86_64.r 30 MB/s | 4.7 MB 00:00 2024-02-09T22:38:57.727 INFO:teuthology.orchestra.run.smithi064.stdout:(59/176): postgresql-13.11-1.el9.x86_64.rpm 9.8 MB/s | 1.5 MB 00:00 2024-02-09T22:38:57.794 INFO:teuthology.orchestra.run.smithi064.stdout:(60/176): openssl-devel-3.0.7-25.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-02-09T22:38:57.827 INFO:teuthology.orchestra.run.smithi064.stdout:(61/176): postgresql-contrib-13.11-1.el9.x86_64 5.4 MB/s | 811 kB 00:00 2024-02-09T22:38:57.844 INFO:teuthology.orchestra.run.smithi064.stdout:(62/176): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2024-02-09T22:38:57.895 INFO:teuthology.orchestra.run.smithi064.stdout:(63/176): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-02-09T22:38:57.954 INFO:teuthology.orchestra.run.smithi064.stdout:(64/176): python3-attrs-20.3.0-7.el9.noarch.rpm 720 kB/s | 78 kB 00:00 2024-02-09T22:38:58.037 INFO:teuthology.orchestra.run.smithi064.stdout:(65/176): postgresql-server-13.11-1.el9.x86_64. 27 MB/s | 5.7 MB 00:00 2024-02-09T22:38:58.121 INFO:teuthology.orchestra.run.smithi064.stdout:(66/176): python3-babel-2.9.1-2.el9.noarch.rpm 26 MB/s | 5.8 MB 00:00 2024-02-09T22:38:58.146 INFO:teuthology.orchestra.run.smithi064.stdout:(67/176): python3-jinja2-2.11.3-5.el9.noarch.rp 1.2 MB/s | 227 kB 00:00 2024-02-09T22:38:58.171 INFO:teuthology.orchestra.run.smithi064.stdout:(68/176): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-02-09T22:38:58.196 INFO:teuthology.orchestra.run.smithi064.stdout:(69/176): python3-jsonpointer-2.0-4.el9.noarch. 241 kB/s | 18 kB 00:00 2024-02-09T22:38:58.222 INFO:teuthology.orchestra.run.smithi064.stdout:(70/176): python3-markupsafe-1.1.1-12.el9.x86_6 629 kB/s | 32 kB 00:00 2024-02-09T22:38:58.247 INFO:teuthology.orchestra.run.smithi064.stdout:(71/176): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-02-09T22:38:58.339 INFO:teuthology.orchestra.run.smithi064.stdout:(72/176): python3-numpy-1.20.1-5.el9.x86_64.rpm 34 MB/s | 4.8 MB 00:00 2024-02-09T22:38:58.364 INFO:teuthology.orchestra.run.smithi064.stdout:(73/176): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-02-09T22:38:58.389 INFO:teuthology.orchestra.run.smithi064.stdout:(74/176): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-02-09T22:38:58.414 INFO:teuthology.orchestra.run.smithi064.stdout:(75/176): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-02-09T22:38:58.439 INFO:teuthology.orchestra.run.smithi064.stdout:(76/176): python3-protobuf-3.14.0-13.el9.noarch 4.7 MB/s | 238 kB 00:00 2024-02-09T22:38:58.465 INFO:teuthology.orchestra.run.smithi064.stdout:(77/176): python3-prettytable-0.7.2-27.el9.noar 546 kB/s | 41 kB 00:00 2024-02-09T22:38:58.490 INFO:teuthology.orchestra.run.smithi064.stdout:(78/176): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-02-09T22:38:58.515 INFO:teuthology.orchestra.run.smithi064.stdout:(79/176): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-02-09T22:38:58.540 INFO:teuthology.orchestra.run.smithi064.stdout:(80/176): python3-requests-oauthlib-1.3.0-12.el 866 kB/s | 43 kB 00:00 2024-02-09T22:38:58.566 INFO:teuthology.orchestra.run.smithi064.stdout:(81/176): python3-pytz-2021.1-5.el9.noarch.rpm 633 kB/s | 47 kB 00:00 2024-02-09T22:38:58.741 INFO:teuthology.orchestra.run.smithi064.stdout:(82/176): python3-toml-0.10.2-6.el9.noarch.rpm 195 kB/s | 39 kB 00:00 2024-02-09T22:38:58.892 INFO:teuthology.orchestra.run.smithi064.stdout:(83/176): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-02-09T22:38:58.908 INFO:teuthology.orchestra.run.smithi064.stdout:(84/176): python3-wcwidth-0.2.5-8.el9.noarch.rp 121 kB/s | 41 kB 00:00 2024-02-09T22:38:58.934 INFO:teuthology.orchestra.run.smithi064.stdout:(85/176): qatlib-23.11.0-1.el9.x86_64.rpm 1.1 MB/s | 215 kB 00:00 2024-02-09T22:38:58.958 INFO:teuthology.orchestra.run.smithi064.stdout:(86/176): qatlib-service-23.11.0-1.el9.x86_64.r 681 kB/s | 34 kB 00:00 2024-02-09T22:38:58.976 INFO:teuthology.orchestra.run.smithi064.stdout:(87/176): qatzip-libs-1.1.2-2.el9.x86_64.rpm 653 kB/s | 43 kB 00:00 2024-02-09T22:38:59.001 INFO:teuthology.orchestra.run.smithi064.stdout:(88/176): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2024-02-09T22:38:59.026 INFO:teuthology.orchestra.run.smithi064.stdout:(89/176): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 56 kB 00:00 2024-02-09T22:38:59.051 INFO:teuthology.orchestra.run.smithi064.stdout:(90/176): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 87 kB 00:00 2024-02-09T22:38:59.076 INFO:teuthology.orchestra.run.smithi064.stdout:(91/176): xmlstarlet-1.6.1-20.el9.x86_64.rpm 833 kB/s | 63 kB 00:00 2024-02-09T22:38:59.102 INFO:teuthology.orchestra.run.smithi064.stdout:(92/176): zlib-devel-1.2.11-40.el9.x86_64.rpm 877 kB/s | 44 kB 00:00 2024-02-09T22:38:59.127 INFO:teuthology.orchestra.run.smithi064.stdout:(93/176): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-02-09T22:38:59.152 INFO:teuthology.orchestra.run.smithi064.stdout:(94/176): jq-1.6-16.el9.x86_64.rpm 2.4 MB/s | 186 kB 00:00 2024-02-09T22:38:59.178 INFO:teuthology.orchestra.run.smithi064.stdout:(95/176): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 194 kB 00:00 2024-02-09T22:38:59.203 INFO:teuthology.orchestra.run.smithi064.stdout:(96/176): libgfortran-11.4.1-3.el9.x86_64.rpm 10 MB/s | 802 kB 00:00 2024-02-09T22:38:59.228 INFO:teuthology.orchestra.run.smithi064.stdout:(97/176): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-02-09T22:38:59.253 INFO:teuthology.orchestra.run.smithi064.stdout:(98/176): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-02-09T22:38:59.279 INFO:teuthology.orchestra.run.smithi064.stdout:(99/176): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-02-09T22:38:59.320 INFO:teuthology.orchestra.run.smithi064.stdout:(100/176): python3-cryptography-36.0.1-4.el9.x8 12 MB/s | 1.1 MB 00:00 2024-02-09T22:38:59.346 INFO:teuthology.orchestra.run.smithi064.stdout:(101/176): python3-pycparser-2.20-6.el9.noarch. 1.3 MB/s | 124 kB 00:00 2024-02-09T22:38:59.371 INFO:teuthology.orchestra.run.smithi064.stdout:(102/176): python3-requests-2.25.1-8.el9.noarch 1.2 MB/s | 113 kB 00:00 2024-02-09T22:38:59.396 INFO:teuthology.orchestra.run.smithi064.stdout:(103/176): python3-urllib3-1.26.5-5.el9.noarch. 3.7 MB/s | 187 kB 00:00 2024-02-09T22:38:59.505 INFO:teuthology.orchestra.run.smithi064.stdout:(104/176): python3-devel-3.9.17-2.el9.x86_64.rp 1.5 MB/s | 245 kB 00:00 2024-02-09T22:38:59.539 INFO:teuthology.orchestra.run.smithi064.stdout:(105/176): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-02-09T22:38:59.789 INFO:teuthology.orchestra.run.smithi064.stdout:(106/176): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-02-09T22:38:59.831 INFO:teuthology.orchestra.run.smithi064.stdout:(107/176): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-02-09T22:38:59.856 INFO:teuthology.orchestra.run.smithi064.stdout:(108/176): gperftools-libs-2.9.1-2.el9.x86_64.r 975 kB/s | 309 kB 00:00 2024-02-09T22:38:59.923 INFO:teuthology.orchestra.run.smithi064.stdout:(109/176): libarrow-doc-9.0.0-10.el9.noarch.rpm 643 kB/s | 26 kB 00:00 2024-02-09T22:38:59.965 INFO:teuthology.orchestra.run.smithi064.stdout:(110/176): liboath-2.6.7-2.el9.x86_64.rpm 1.2 MB/s | 49 kB 00:00 2024-02-09T22:39:00.007 INFO:teuthology.orchestra.run.smithi064.stdout:(111/176): grpc-data-1.46.7-10.el9.noarch.rpm 89 kB/s | 19 kB 00:00 2024-02-09T22:39:00.032 INFO:teuthology.orchestra.run.smithi064.stdout:(112/176): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-02-09T22:39:00.091 INFO:teuthology.orchestra.run.smithi064.stdout:(113/176): luarocks-3.9.2-1.el9.noarch.rpm 1.8 MB/s | 151 kB 00:00 2024-02-09T22:39:00.224 INFO:teuthology.orchestra.run.smithi064.stdout:(114/176): python3-asyncssh-2.13.2-3.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-02-09T22:39:00.258 INFO:teuthology.orchestra.run.smithi064.stdout:(115/176): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.6 MB/s | 838 kB 00:00 2024-02-09T22:39:00.283 INFO:teuthology.orchestra.run.smithi064.stdout:(116/176): python3-bcrypt-3.2.2-1.el9.x86_64.rp 743 kB/s | 43 kB 00:00 2024-02-09T22:39:00.309 INFO:teuthology.orchestra.run.smithi064.stdout:(117/176): python3-cachetools-4.2.4-1.el9.noarc 642 kB/s | 32 kB 00:00 2024-02-09T22:39:00.334 INFO:teuthology.orchestra.run.smithi064.stdout:(118/176): python3-certifi-2021.10.8-2.el9.noar 298 kB/s | 15 kB 00:00 2024-02-09T22:39:00.376 INFO:teuthology.orchestra.run.smithi064.stdout:(119/176): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-02-09T22:39:00.459 INFO:teuthology.orchestra.run.smithi064.stdout:(120/176): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-02-09T22:39:00.485 INFO:teuthology.orchestra.run.smithi064.stdout:(121/176): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-02-09T22:39:00.510 INFO:teuthology.orchestra.run.smithi064.stdout:(122/176): python3-colorama-0.4.4-12.el9.noarch 703 kB/s | 35 kB 00:00 2024-02-09T22:39:00.552 INFO:teuthology.orchestra.run.smithi064.stdout:(123/176): python3-google-auth-2.27.0-1.el9.noa 3.0 MB/s | 203 kB 00:00 2024-02-09T22:39:00.611 INFO:teuthology.orchestra.run.smithi064.stdout:(124/176): python3-grpcio-tools-1.46.7-10.el9.x 2.4 MB/s | 144 kB 00:00 2024-02-09T22:39:00.652 INFO:teuthology.orchestra.run.smithi064.stdout:(125/176): python3-jaraco-8.2.1-3.el9.noarch.rp 258 kB/s | 11 kB 00:00 2024-02-09T22:39:00.694 INFO:teuthology.orchestra.run.smithi064.stdout:(126/176): python3-jaraco-classes-3.2.1-5.el9.n 426 kB/s | 18 kB 00:00 2024-02-09T22:39:00.736 INFO:teuthology.orchestra.run.smithi064.stdout:(127/176): python3-jaraco-collections-3.0.0-8.e 557 kB/s | 23 kB 00:00 2024-02-09T22:39:00.779 INFO:teuthology.orchestra.run.smithi064.stdout:(128/176): python3-jaraco-functools-3.5.0-2.el9 466 kB/s | 19 kB 00:00 2024-02-09T22:39:00.812 INFO:teuthology.orchestra.run.smithi064.stdout:(129/176): python3-jaraco-text-3.2.0-6.el9.noar 593 kB/s | 20 kB 00:00 2024-02-09T22:39:00.879 INFO:teuthology.orchestra.run.smithi064.stdout:(130/176): libarrow-9.0.0-10.el9.x86_64.rpm 4.4 MB/s | 4.4 MB 00:00 2024-02-09T22:39:00.921 INFO:teuthology.orchestra.run.smithi064.stdout:(131/176): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-02-09T22:39:00.963 INFO:teuthology.orchestra.run.smithi064.stdout:(132/176): python3-more-itertools-8.12.0-2.el9. 1.8 MB/s | 79 kB 00:00 2024-02-09T22:39:00.996 INFO:teuthology.orchestra.run.smithi064.stdout:(133/176): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-02-09T22:39:01.080 INFO:teuthology.orchestra.run.smithi064.stdout:(134/176): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-02-09T22:39:01.105 INFO:teuthology.orchestra.run.smithi064.stdout:(135/176): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-02-09T22:39:01.122 INFO:teuthology.orchestra.run.smithi064.stdout:(136/176): python3-portend-3.1.0-2.el9.noarch.r 391 kB/s | 16 kB 00:00 2024-02-09T22:39:01.164 INFO:teuthology.orchestra.run.smithi064.stdout:(137/176): python3-grpcio-1.46.7-10.el9.x86_64. 3.1 MB/s | 2.0 MB 00:00 2024-02-09T22:39:01.190 INFO:teuthology.orchestra.run.smithi064.stdout:(138/176): python3-pyperclip-1.8.2-2.el9.noarch 422 kB/s | 28 kB 00:00 2024-02-09T22:39:01.215 INFO:teuthology.orchestra.run.smithi064.stdout:(139/176): python3-repoze-lru-0.7-16.el9.noarch 614 kB/s | 31 kB 00:00 2024-02-09T22:39:01.240 INFO:teuthology.orchestra.run.smithi064.stdout:(140/176): python3-pyOpenSSL-21.0.0-1.el9.noarc 670 kB/s | 90 kB 00:00 2024-02-09T22:39:01.265 INFO:teuthology.orchestra.run.smithi064.stdout:(141/176): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-02-09T22:39:01.291 INFO:teuthology.orchestra.run.smithi064.stdout:(142/176): python3-tempora-5.0.0-2.el9.noarch.r 709 kB/s | 36 kB 00:00 2024-02-09T22:39:01.441 INFO:teuthology.orchestra.run.smithi064.stdout:(143/176): python3-routes-2.5.1-5.el9.noarch.rp 752 kB/s | 188 kB 00:00 2024-02-09T22:39:01.466 INFO:teuthology.orchestra.run.smithi064.stdout:(144/176): python3-typing-extensions-4.4.0-2.el 254 kB/s | 51 kB 00:00 2024-02-09T22:39:01.508 INFO:teuthology.orchestra.run.smithi064.stdout:(145/176): python3-webob-1.8.7-6.el9.noarch.rpm 1.0 MB/s | 230 kB 00:00 2024-02-09T22:39:01.533 INFO:teuthology.orchestra.run.smithi064.stdout:(146/176): python3-websocket-client-1.2.3-2.el9 974 kB/s | 90 kB 00:00 2024-02-09T22:39:01.559 INFO:teuthology.orchestra.run.smithi064.stdout:(147/176): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-02-09T22:39:01.600 INFO:teuthology.orchestra.run.smithi064.stdout:(148/176): python3-werkzeug-2.0.3-3.el9.1.noarc 3.1 MB/s | 427 kB 00:00 2024-02-09T22:39:01.676 INFO:teuthology.orchestra.run.smithi064.stdout:(149/176): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-02-09T22:39:02.126 INFO:teuthology.orchestra.run.smithi064.stdout:(150/176): thrift-0.15.0-2.el9.x86_64.rpm 2.8 MB/s | 1.6 MB 00:00 2024-02-09T22:39:02.160 INFO:teuthology.orchestra.run.smithi064.stdout:(151/176): libacl-devel-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 68 kB 00:00 2024-02-09T22:39:02.251 INFO:teuthology.orchestra.run.smithi064.stdout:(152/176): NetworkManager-1.45.90-1.el9.x86_64. 25 MB/s | 2.3 MB 00:00 2024-02-09T22:39:02.335 INFO:teuthology.orchestra.run.smithi064.stdout:(153/176): NetworkManager-libnm-1.45.90-1.el9.x 22 MB/s | 1.8 MB 00:00 2024-02-09T22:39:02.369 INFO:teuthology.orchestra.run.smithi064.stdout:(154/176): NetworkManager-team-1.45.90-1.el9.x8 1.2 MB/s | 42 kB 00:00 2024-02-09T22:39:02.411 INFO:teuthology.orchestra.run.smithi064.stdout:(155/176): NetworkManager-tui-1.45.90-1.el9.x86 5.8 MB/s | 247 kB 00:00 2024-02-09T22:39:02.444 INFO:teuthology.orchestra.run.smithi064.stdout:(156/176): acl-2.3.1-4.el9.x86_64.rpm 2.0 MB/s | 69 kB 00:00 2024-02-09T22:39:02.478 INFO:teuthology.orchestra.run.smithi064.stdout:(157/176): elfutils-debuginfod-client-0.190-2.e 1.0 MB/s | 35 kB 00:00 2024-02-09T22:39:02.520 INFO:teuthology.orchestra.run.smithi064.stdout:(158/176): elfutils-libelf-0.190-2.el9.x86_64.r 4.5 MB/s | 191 kB 00:00 2024-02-09T22:39:02.562 INFO:teuthology.orchestra.run.smithi064.stdout:(159/176): elfutils-libs-0.190-2.el9.x86_64.rpm 6.0 MB/s | 254 kB 00:00 2024-02-09T22:39:02.620 INFO:teuthology.orchestra.run.smithi064.stdout:(160/176): iproute-6.7.0-2.el9.x86_64.rpm 13 MB/s | 802 kB 00:00 2024-02-09T22:39:02.671 INFO:teuthology.orchestra.run.smithi064.stdout:(161/176): iproute-tc-6.7.0-2.el9.x86_64.rpm 8.5 MB/s | 433 kB 00:00 2024-02-09T22:39:02.704 INFO:teuthology.orchestra.run.smithi064.stdout:(162/176): libacl-2.3.1-4.el9.x86_64.rpm 661 kB/s | 22 kB 00:00 2024-02-09T22:39:02.729 INFO:teuthology.orchestra.run.smithi064.stdout:(163/176): libblkid-2.37.4-18.el9.x86_64.rpm 4.2 MB/s | 107 kB 00:00 2024-02-09T22:39:02.780 INFO:teuthology.orchestra.run.smithi064.stdout:(164/176): libfdisk-2.37.4-18.el9.x86_64.rpm 3.0 MB/s | 154 kB 00:00 2024-02-09T22:39:02.824 INFO:teuthology.orchestra.run.smithi064.stdout:(165/176): libmount-2.37.4-18.el9.x86_64.rpm 3.2 MB/s | 135 kB 00:00 2024-02-09T22:39:02.856 INFO:teuthology.orchestra.run.smithi064.stdout:(166/176): libsmartcols-2.37.4-18.el9.x86_64.rp 1.8 MB/s | 63 kB 00:00 2024-02-09T22:39:02.906 INFO:teuthology.orchestra.run.smithi064.stdout:(167/176): librbd1-19.0.0-1361.g290212a1.el9.x8 2.5 MB/s | 3.1 MB 00:01 2024-02-09T22:39:02.931 INFO:teuthology.orchestra.run.smithi064.stdout:(168/176): libuuid-2.37.4-18.el9.x86_64.rpm 368 kB/s | 28 kB 00:00 2024-02-09T22:39:02.965 INFO:teuthology.orchestra.run.smithi064.stdout:(169/176): openssl-3.0.7-25.el9.x86_64.rpm 20 MB/s | 1.2 MB 00:00 2024-02-09T22:39:03.023 INFO:teuthology.orchestra.run.smithi064.stdout:(170/176): openssl-libs-3.0.7-25.el9.x86_64.rpm 24 MB/s | 2.1 MB 00:00 2024-02-09T22:39:03.073 INFO:teuthology.orchestra.run.smithi064.stdout:(171/176): util-linux-2.37.4-18.el9.x86_64.rpm 21 MB/s | 2.2 MB 00:00 2024-02-09T22:39:03.107 INFO:teuthology.orchestra.run.smithi064.stdout:(172/176): util-linux-core-2.37.4-18.el9.x86_64 5.3 MB/s | 438 kB 00:00 2024-02-09T22:39:03.133 INFO:teuthology.orchestra.run.smithi064.stdout:(173/176): util-linux-user-2.37.4-18.el9.x86_64 520 kB/s | 30 kB 00:00 2024-02-09T22:39:03.157 INFO:teuthology.orchestra.run.smithi064.stdout:(174/176): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-02-09T22:39:03.225 INFO:teuthology.orchestra.run.smithi064.stdout:(175/176): librados2-19.0.0-1361.g290212a1.el9. 2.1 MB/s | 3.4 MB 00:01 2024-02-09T22:39:03.258 INFO:teuthology.orchestra.run.smithi064.stdout:(176/176): xfsprogs-6.3.0-1.el9.x86_64.rpm 8.5 MB/s | 1.1 MB 00:00 2024-02-09T22:39:03.267 INFO:teuthology.orchestra.run.smithi064.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:39:03.267 INFO:teuthology.orchestra.run.smithi064.stdout:Total 19 MB/s | 222 MB 00:11 2024-02-09T22:39:03.315 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction check 2024-02-09T22:39:03.621 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction check succeeded. 2024-02-09T22:39:03.621 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction test 2024-02-09T22:39:05.397 INFO:teuthology.orchestra.run.smithi064.stdout:Transaction test succeeded. 2024-02-09T22:39:05.397 INFO:teuthology.orchestra.run.smithi064.stdout:Running transaction 2024-02-09T22:39:12.520 INFO:teuthology.orchestra.run.smithi064.stdout: Preparing : 1/1 2024-02-09T22:39:12.620 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : openssl-libs-1:3.0.7-25.el9.x86_64 1/202 2024-02-09T22:39:12.646 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/202 2024-02-09T22:39:12.677 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-09T22:39:12.703 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/202 2024-02-09T22:39:12.763 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 4/202 2024-02-09T22:39:12.803 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 5/202 2024-02-09T22:39:12.857 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/202 2024-02-09T22:39:12.899 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : thrift-0.15.0-2.el9.x86_64 7/202 2024-02-09T22:39:12.923 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 8/202 2024-02-09T22:39:12.966 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 9/202 2024-02-09T22:39:13.356 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 10/202 2024-02-09T22:39:13.399 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:39:13.523 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:39:13.550 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:39:13.642 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:39:13.670 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 13/202 2024-02-09T22:39:13.829 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 14/202 2024-02-09T22:39:13.876 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/202 2024-02-09T22:39:13.913 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:39:14.017 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:39:14.047 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 17/202 2024-02-09T22:39:14.102 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 17/202 2024-02-09T22:39:14.142 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/202 2024-02-09T22:39:14.165 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-09T22:39:14.180 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/202 2024-02-09T22:39:14.373 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-09T22:39:14.403 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/202 2024-02-09T22:39:14.444 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/202 2024-02-09T22:39:14.552 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/202 2024-02-09T22:39:14.665 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : NetworkManager-libnm-1:1.45.90-1.el9.x86_64 23/202 2024-02-09T22:39:14.725 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.90-1.el9.x86_64 23/202 2024-02-09T22:39:14.803 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:39:14.843 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:39:15.078 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-1:1.45.90-1.el9.x86_64 24/202 2024-02-09T22:39:15.270 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/202 2024-02-09T22:39:15.418 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/202 2024-02-09T22:39:15.466 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/202 2024-02-09T22:39:15.580 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/202 2024-02-09T22:39:15.649 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/202 2024-02-09T22:39:15.679 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/202 2024-02-09T22:39:15.731 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/202 2024-02-09T22:39:15.778 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/202 2024-02-09T22:39:15.885 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/202 2024-02-09T22:39:15.924 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/202 2024-02-09T22:39:15.984 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/202 2024-02-09T22:39:16.025 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/202 2024-02-09T22:39:16.056 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/202 2024-02-09T22:39:16.135 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/202 2024-02-09T22:39:16.168 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-common-2:19.0.0-1361.g290212a1.el9. 39/202 2024-02-09T22:39:16.211 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el 40/202 2024-02-09T22:39:16.240 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_6 41/202 2024-02-09T22:39:16.313 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/202 2024-02-09T22:39:16.354 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/202 2024-02-09T22:39:16.432 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/202 2024-02-09T22:39:16.587 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/202 2024-02-09T22:39:16.642 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/202 2024-02-09T22:39:17.204 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/202 2024-02-09T22:39:17.280 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/202 2024-02-09T22:39:17.360 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/202 2024-02-09T22:39:17.405 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-09T22:39:17.413 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/202 2024-02-09T22:39:17.452 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/202 2024-02-09T22:39:17.490 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_6 52/202 2024-02-09T22:39:17.521 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 53/202 2024-02-09T22:39:17.616 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 54/202 2024-02-09T22:39:17.655 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : iproute-6.7.0-2.el9.x86_64 55/202 2024-02-09T22:39:18.024 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 56/202 2024-02-09T22:39:18.159 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 57/202 2024-02-09T22:39:18.225 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 58/202 2024-02-09T22:39:18.270 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 59/202 2024-02-09T22:39:18.408 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 60/202 2024-02-09T22:39:18.480 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : openssl-1:3.0.7-25.el9.x86_64 61/202 2024-02-09T22:39:18.536 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 62/202 2024-02-09T22:39:18.611 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 63/202 2024-02-09T22:39:18.657 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 64/202 2024-02-09T22:39:18.719 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 65/202 2024-02-09T22:39:18.767 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 66/202 2024-02-09T22:39:18.801 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 67/202 2024-02-09T22:39:18.845 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 68/202 2024-02-09T22:39:18.888 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 69/202 2024-02-09T22:39:19.030 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 70/202 2024-02-09T22:39:19.083 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 71/202 2024-02-09T22:39:19.114 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 72/202 2024-02-09T22:39:19.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 73/202 2024-02-09T22:39:19.254 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 74/202 2024-02-09T22:39:19.288 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-google-auth-1:2.27.0-1.el9.noarch 75/202 2024-02-09T22:39:19.328 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 76/202 2024-02-09T22:39:19.354 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 77/202 2024-02-09T22:39:19.379 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 78/202 2024-02-09T22:39:19.455 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 79/202 2024-02-09T22:39:19.594 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 80/202 2024-02-09T22:39:19.709 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 81/202 2024-02-09T22:39:19.754 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 82/202 2024-02-09T22:39:19.811 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-ply-3.11-14.el9.noarch 83/202 2024-02-09T22:39:19.867 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 84/202 2024-02-09T22:39:20.060 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 85/202 2024-02-09T22:39:20.116 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 86/202 2024-02-09T22:39:20.261 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 87/202 2024-02-09T22:39:20.298 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 88/202 2024-02-09T22:39:20.331 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 89/202 2024-02-09T22:39:20.362 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 90/202 2024-02-09T22:39:20.397 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : jq-1.6-16.el9.x86_64 91/202 2024-02-09T22:39:20.429 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : fuse-2.9.9-15.el9.x86_64 92/202 2024-02-09T22:39:20.474 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 93/202 2024-02-09T22:39:20.508 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 94/202 2024-02-09T22:39:20.547 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : uuid-1.6.2-55.el9.x86_64 95/202 2024-02-09T22:39:20.579 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 96/202 2024-02-09T22:39:20.608 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 97/202 2024-02-09T22:39:20.688 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 98/202 2024-02-09T22:39:20.719 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 99/202 2024-02-09T22:39:20.752 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 100/202 2024-02-09T22:39:20.794 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 101/202 2024-02-09T22:39:20.828 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 102/202 2024-02-09T22:39:20.921 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 103/202 2024-02-09T22:39:21.011 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 104/202 2024-02-09T22:39:21.068 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 105/202 2024-02-09T22:39:21.099 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 106/202 2024-02-09T22:39:21.147 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 107/202 2024-02-09T22:39:21.256 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-packaging-20.9-5.el9.noarch 108/202 2024-02-09T22:39:21.310 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 109/202 2024-02-09T22:39:22.078 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 110/202 2024-02-09T22:39:22.122 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 111/202 2024-02-09T22:39:22.156 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 112/202 2024-02-09T22:39:22.205 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 113/202 2024-02-09T22:39:22.305 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 114/202 2024-02-09T22:39:22.341 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 115/202 2024-02-09T22:39:22.525 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-0.3.21-2.el9.x86_64 116/202 2024-02-09T22:39:22.563 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 117/202 2024-02-09T22:39:22.643 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 118/202 2024-02-09T22:39:23.330 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 119/202 2024-02-09T22:39:23.439 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 120/202 2024-02-09T22:39:24.740 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 121/202 2024-02-09T22:39:24.809 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 122/202 2024-02-09T22:39:24.836 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 123/202 2024-02-09T22:39:25.159 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 124/202 2024-02-09T22:39:25.182 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 125/202 2024-02-09T22:39:25.265 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 125/202 2024-02-09T22:39:25.295 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 126/202 2024-02-09T22:39:25.354 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 127/202 2024-02-09T22:39:25.846 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 128/202 2024-02-09T22:39:25.877 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 129/202 2024-02-09T22:39:25.948 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 129/202 2024-02-09T22:39:25.979 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 130/202 2024-02-09T22:39:28.272 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:39:28.308 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:39:28.874 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 131/202 2024-02-09T22:39:28.912 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 132/202 2024-02-09T22:39:28.977 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 132/202 2024-02-09T22:39:28.978 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-02-09T22:39:28.978 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-02-09T22:39:28.978 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:29.005 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 133/202 2024-02-09T22:39:41.283 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 133/202 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /sys 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /proc 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /mnt 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /var/tmp 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /home 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /root 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout:skipping the directory /tmp 2024-02-09T22:39:41.284 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:41.443 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 134/202 2024-02-09T22:39:41.459 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 134/202 2024-02-09T22:39:41.460 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:41.460 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-02-09T22:39:41.460 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:42.980 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 135/202 2024-02-09T22:39:43.007 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 135/202 2024-02-09T22:39:43.007 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:43.007 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-02-09T22:39:43.007 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:39:43.007 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-02-09T22:39:43.007 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:43.222 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 136/202 2024-02-09T22:39:43.246 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 136/202 2024-02-09T22:39:43.246 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:43.246 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-02-09T22:39:43.246 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:39:43.246 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-02-09T22:39:43.246 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:43.706 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 137/202 2024-02-09T22:39:43.730 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 137/202 2024-02-09T22:39:43.730 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:43.730 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-02-09T22:39:43.730 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:39:43.730 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-02-09T22:39:43.731 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:43.833 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : automake-1.16.2-8.el9.noarch 138/202 2024-02-09T22:39:44.260 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 139/202 2024-02-09T22:39:44.289 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 139/202 2024-02-09T22:39:44.320 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.e 140/202 2024-02-09T22:39:44.384 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1. 141/202 2024-02-09T22:39:44.413 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 142/202 2024-02-09T22:39:44.655 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 142/202 2024-02-09T22:39:44.688 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 143/202 2024-02-09T22:39:45.704 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 143/202 2024-02-09T22:39:45.730 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 144/202 2024-02-09T22:39:45.839 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 144/202 2024-02-09T22:39:46.010 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el 145/202 2024-02-09T22:39:46.041 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 146/202 2024-02-09T22:39:46.067 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 146/202 2024-02-09T22:39:46.068 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:46.068 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-02-09T22:39:46.068 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:39:46.068 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-02-09T22:39:46.068 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:46.116 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 147/202 2024-02-09T22:39:46.127 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 147/202 2024-02-09T22:39:46.174 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 148/202 2024-02-09T22:39:47.074 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libtool-2.4.6-45.el9.x86_64 149/202 2024-02-09T22:39:47.102 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 150/202 2024-02-09T22:39:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 150/202 2024-02-09T22:39:47.126 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:47.126 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-02-09T22:39:47.126 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:39:47.126 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-02-09T22:39:47.126 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:47.169 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1361.g29021 151/202 2024-02-09T22:39:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1361.g29021 151/202 2024-02-09T22:39:47.193 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:47.193 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-02-09T22:39:47.193 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:47.236 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 152/202 2024-02-09T22:39:47.260 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 152/202 2024-02-09T22:39:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-02-09T22:39:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-09T22:39:47.261 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-02-09T22:39:47.261 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:47.579 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:39:47.603 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:39:47.603 INFO:teuthology.orchestra.run.smithi064.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-02-09T22:39:47.603 INFO:teuthology.orchestra.run.smithi064.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-02-09T22:39:47.603 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:39:47.603 INFO:teuthology.orchestra.run.smithi064.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-02-09T22:39:47.603 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:39:52.259 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 154/202 2024-02-09T22:39:52.300 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 155/202 2024-02-09T22:39:52.330 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 156/202 2024-02-09T22:39:52.357 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 157/202 2024-02-09T22:39:52.550 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 158/202 2024-02-09T22:39:52.670 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 159/202 2024-02-09T22:39:52.714 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 160/202 2024-02-09T22:39:52.753 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 161/202 2024-02-09T22:39:52.805 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 162/202 2024-02-09T22:39:52.875 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 163/202 2024-02-09T22:39:52.907 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : iproute-tc-6.7.0-2.el9.x86_64 164/202 2024-02-09T22:39:52.942 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 165/202 2024-02-09T22:39:52.973 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_ 166/202 2024-02-09T22:39:53.002 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : NetworkManager-team-1:1.45.90-1.el9.x86_64 167/202 2024-02-09T22:39:53.059 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : NetworkManager-tui-1:1.45.90-1.el9.x86_64 168/202 2024-02-09T22:39:53.108 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 169/202 2024-02-09T22:39:53.142 INFO:teuthology.orchestra.run.smithi064.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 170/202 2024-02-09T22:39:53.735 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:39:53.774 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:39:53.794 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 171/202 2024-02-09T22:39:54.694 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 172/202 2024-02-09T22:39:54.779 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : openssl-devel-1:3.0.7-25.el9.x86_64 173/202 2024-02-09T22:39:54.821 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 174/202 2024-02-09T22:39:54.904 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : flex-2.6.4-9.el9.x86_64 175/202 2024-02-09T22:39:54.941 INFO:teuthology.orchestra.run.smithi064.stdout: Installing : bison-3.7.4-5.el9.x86_64 176/202 2024-02-09T22:39:54.966 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 177/202 2024-02-09T22:39:54.991 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 178/202 2024-02-09T22:39:55.018 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 179/202 2024-02-09T22:39:55.018 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-09T22:39:55.086 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 180/202 2024-02-09T22:39:55.114 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 181/202 2024-02-09T22:39:55.144 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 182/202 2024-02-09T22:39:55.144 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-09T22:39:55.228 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 183/202 2024-02-09T22:39:55.254 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 184/202 2024-02-09T22:39:55.305 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 185/202 2024-02-09T22:39:55.330 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : iproute-tc-6.2.0-5.el9.x86_64 186/202 2024-02-09T22:39:55.357 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 187/202 2024-02-09T22:39:55.398 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : iproute-6.2.0-5.el9.x86_64 188/202 2024-02-09T22:39:55.431 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 189/202 2024-02-09T22:39:55.470 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 190/202 2024-02-09T22:39:55.503 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 191/202 2024-02-09T22:39:55.528 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 192/202 2024-02-09T22:39:55.553 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 193/202 2024-02-09T22:39:55.578 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 194/202 2024-02-09T22:39:55.604 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 195/202 2024-02-09T22:39:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:39:55.611 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:39:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 196/202 2024-02-09T22:39:55.807 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-09T22:39:55.890 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 197/202 2024-02-09T22:39:55.915 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 198/202 2024-02-09T22:39:55.940 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 199/202 2024-02-09T22:39:55.965 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 200/202 2024-02-09T22:39:55.998 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 201/202 2024-02-09T22:39:56.068 INFO:teuthology.orchestra.run.smithi064.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-09T22:40:00.677 INFO:teuthology.orchestra.run.smithi064.stdout: Running scriptlet: openssl-libs-1:3.0.7-24.el9.x86_64 202/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-2:19.0.0-1361.g290212a1.el9.x86_64 1/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 3/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 4/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1361.g29021 5/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 6/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 7/202 2024-02-09T22:40:00.678 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 8/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 9/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 10/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 11/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 12/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 13/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 14/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_ 15/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 16/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 17/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados-devel-2:19.0.0-1361.g290212a1.el9.x86_6 18/202 2024-02-09T22:40:00.679 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86 19/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 20/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1361.g290212a1.el 21/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ceph-common-2:19.0.0-1361.g290212a1.el9. 22/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_6 23/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 24/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 25/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 26/202 2024-02-09T22:40:00.680 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 27/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 28/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 29/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1361.g290212a1. 30/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noa 31/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.n 32/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1361.g290 33/202 2024-02-09T22:40:00.697 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el 34/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 35/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.e 36/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 37/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephadm-2:19.0.0-1361.g290212a1.el9.noarch 38/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 39/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 43/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flex-2.6.4-9.el9.x86_64 44/202 2024-02-09T22:40:00.698 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 45/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 46/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 47/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 48/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 49/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 50/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 51/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 52/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 53/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 54/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 55/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-5.4.4-4.el9.x86_64 56/202 2024-02-09T22:40:00.699 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 57/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 58/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-devel-1:3.0.7-25.el9.x86_64 59/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-13.11-1.el9.x86_64 60/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 61/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 62/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 63/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 64/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 65/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 66/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 67/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 68/202 2024-02-09T22:40:00.700 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 69/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 70/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 71/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 72/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 73/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 74/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 75/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 76/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 77/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 78/202 2024-02-09T22:40:00.701 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 79/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 80/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 82/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 83/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 84/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 85/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 86/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 87/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 88/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 89/202 2024-02-09T22:40:00.702 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 90/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 91/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 92/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 93/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : jq-1.6-16.el9.x86_64 94/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 95/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 96/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 97/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 98/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 99/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-ply-3.11-14.el9.noarch 100/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 101/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 102/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 103/202 2024-02-09T22:40:00.703 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 104/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/202 2024-02-09T22:40:00.704 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 116/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-google-auth-1:2.27.0-1.el9.noarch 124/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/202 2024-02-09T22:40:00.705 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/202 2024-02-09T22:40:00.706 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:19.0.0-1361.g290212a1.el9.x86_64 151/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 153/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 155/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 156/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-1:1.45.90-1.el9.x86_64 157/202 2024-02-09T22:40:00.707 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-libnm-1:1.45.90-1.el9.x86_64 159/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-team-1:1.45.90-1.el9.x86_64 161/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-tui-1:1.45.90-1.el9.x86_64 163/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/202 2024-02-09T22:40:00.708 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : iproute-6.7.0-2.el9.x86_64 173/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : iproute-6.2.0-5.el9.x86_64 174/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : iproute-tc-6.7.0-2.el9.x86_64 175/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : iproute-tc-6.2.0-5.el9.x86_64 176/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 177/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 178/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 179/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 180/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 181/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 182/202 2024-02-09T22:40:00.709 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 183/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 184/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 185/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 186/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 187/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 188/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-1:3.0.7-25.el9.x86_64 189/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 190/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-libs-1:3.0.7-25.el9.x86_64 191/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 192/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 193/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 194/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 195/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 196/202 2024-02-09T22:40:00.710 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 197/202 2024-02-09T22:40:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 198/202 2024-02-09T22:40:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 199/202 2024-02-09T22:40:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 200/202 2024-02-09T22:40:00.711 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 201/202 2024-02-09T22:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 202/202 2024-02-09T22:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout:Upgraded: 2024-02-09T22:40:03.658 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-1:1.45.90-1.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-libnm-1:1.45.90-1.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-team-1:1.45.90-1.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: NetworkManager-tui-1:1.45.90-1.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: acl-2.3.1-4.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: iproute-6.7.0-2.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: iproute-tc-6.7.0-2.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: libacl-2.3.1-4.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-02-09T22:40:03.659 INFO:teuthology.orchestra.run.smithi064.stdout: libblkid-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: libmount-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: librados2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: librbd1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-1:3.0.7-25.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-libs-1:3.0.7-25.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: util-linux-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.660 INFO:teuthology.orchestra.run.smithi064.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout:Installed: 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: automake-1.16.2-8.el9.noarch 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: bison-3.7.4-5.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-base-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-grafana-dashboards-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.661 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-immutable-object-cache-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mds-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-dashboard-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-modules-core-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mgr-rook-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-mon-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-osd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-prometheus-alerts-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-radosgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-selinux-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.662 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-test-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: ceph-volume-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephadm-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-shell-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: cephfs-top-2:19.0.0-1361.g290212a1.el9.noarch 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: flex-2.6.4-9.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: fuse-2.9.9-15.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-02-09T22:40:03.663 INFO:teuthology.orchestra.run.smithi064.stdout: grpc-data-1.46.7-10.el9.noarch 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: jq-1.6-16.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-9.0.0-10.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libcephfs2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libcephsqlite-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: liboath-2.6.7-2.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-02-09T22:40:03.664 INFO:teuthology.orchestra.run.smithi064.stdout: librados-devel-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: libradosstriper1-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: librgw2-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: libtool-2.4.6-45.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: libunwind-1.6.2-1.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: lua-5.4.4-4.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: luarocks-3.9.2-1.el9.noarch 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-02-09T22:40:03.665 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-0.3.21-2.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: openssl-devel-1:3.0.7-25.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-13.11-1.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: postgresql-server-13.11-1.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-3.14.0-13.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: python3-babel-2.9.1-2.el9.noarch 2024-02-09T22:40:03.666 INFO:teuthology.orchestra.run.smithi064.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-argparse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ceph-common-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cephfs-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-google-auth-1:2.27.0-1.el9.noarch 2024-02-09T22:40:03.667 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-02-09T22:40:03.668 INFO:teuthology.orchestra.run.smithi064.stdout: python3-mako-1.1.4-6.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-packaging-20.9-5.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-ply-3.11-14.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-portend-3.1.0-2.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-02-09T22:40:03.669 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pycparser-2.20-6.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-pytz-2021.1-5.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rados-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-2.25.1-8.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rgw-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-routes-2.5.1-5.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-rsa-4.9-2.el9.noarch 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-02-09T22:40:03.670 INFO:teuthology.orchestra.run.smithi064.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-toml-0.10.2-6.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-webob-1.8.7-6.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-23.11.0-1.el9.x86_64 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-fuse-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-mirror-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: rbd-nbd-2:19.0.0-1361.g290212a1.el9.x86_64 2024-02-09T22:40:03.671 INFO:teuthology.orchestra.run.smithi064.stdout: re2-1:20211101-3.el9.x86_64 2024-02-09T22:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: socat-1.7.4.1-5.el9.x86_64 2024-02-09T22:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: thrift-0.15.0-2.el9.x86_64 2024-02-09T22:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: uuid-1.6.2-55.el9.x86_64 2024-02-09T22:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-02-09T22:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-02-09T22:40:03.672 INFO:teuthology.orchestra.run.smithi064.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-02-09T22:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout: 2024-02-09T22:40:03.673 INFO:teuthology.orchestra.run.smithi064.stdout:Complete! 2024-02-09T22:40:04.311 DEBUG:teuthology.parallel:result is None 2024-02-09T22:40:04.312 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:40:04.469 DEBUG:teuthology.orchestra.run.smithi003:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T22:40:04.496 INFO:teuthology.orchestra.run.smithi003.stdout:19.0.0-1361.g290212a1.el9 2024-02-09T22:40:04.500 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-09T22:40:04.500 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-09T22:40:04.503 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:40:04.679 DEBUG:teuthology.orchestra.run.smithi064:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T22:40:04.702 INFO:teuthology.orchestra.run.smithi064.stdout:19.0.0-1361.g290212a1.el9 2024-02-09T22:40:04.702 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-09T22:40:04.702 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-09T22:40:04.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:40:04.987 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-02-09T22:40:05.013 INFO:teuthology.orchestra.run.smithi089.stdout:19.0.0-1361.g290212a1.el9 2024-02-09T22:40:05.014 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1361.g290212a1.el9 2024-02-09T22:40:05.014 INFO:teuthology.task.install:The correct ceph version 19.0.0-1361.g290212a1 is installed. 2024-02-09T22:40:05.016 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-02-09T22:40:05.016 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:40:05.016 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T22:40:05.060 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:40:05.060 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T22:40:05.097 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:40:05.097 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-02-09T22:40:05.131 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-02-09T22:40:05.132 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:40:05.132 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T22:40:05.163 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T22:40:05.234 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:40:05.234 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T22:40:05.268 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T22:40:05.341 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:40:05.341 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-02-09T22:40:05.373 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-02-09T22:40:05.443 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-02-09T22:40:05.443 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:40:05.443 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T22:40:05.474 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T22:40:05.545 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:40:05.545 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T22:40:05.577 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T22:40:05.649 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:40:05.649 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-02-09T22:40:05.679 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-02-09T22:40:05.747 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-02-09T22:40:05.748 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:40:05.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T22:40:05.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T22:40:05.850 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:40:05.851 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T22:40:05.882 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T22:40:05.954 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:40:05.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-02-09T22:40:05.982 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-02-09T22:40:06.051 INFO:teuthology.run_tasks:Running task cephadm... 2024-02-09T22:40:06.159 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-02-09T22:40:06.160 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:40:06.160 INFO:tasks.cephadm:Cluster fsid is 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:40:06.160 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-02-09T22:40:06.160 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.3', 'mon.b': '172.21.15.64', 'mon.c': '172.21.15.89'} 2024-02-09T22:40:06.161 INFO:tasks.cephadm:First mon is mon.a on smithi003 2024-02-09T22:40:06.161 INFO:tasks.cephadm:First mgr is x 2024-02-09T22:40:06.161 INFO:tasks.cephadm:Normalizing hostnames... 2024-02-09T22:40:06.161 DEBUG:teuthology.orchestra.run.smithi003:> sudo hostname $(hostname -s) 2024-02-09T22:40:06.193 DEBUG:teuthology.orchestra.run.smithi064:> sudo hostname $(hostname -s) 2024-02-09T22:40:06.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo hostname $(hostname -s) 2024-02-09T22:40:06.254 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-02-09T22:40:06.254 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:40:06.420 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/76638/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1361-g290212a1', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1361.g290212a1'}, 'url': 'https://2.chacra.ceph.com/r/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-02-09 18:34:21.832169', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-02-09T22:40:06.588 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref main, sha1 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:40:06.589 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm 2024-02-09T22:40:06.590 DEBUG:teuthology.orchestra.run.smithi003:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:07.528 INFO:teuthology.orchestra.run.smithi003.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 9 22:40 /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:07.529 DEBUG:teuthology.orchestra.run.smithi064:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.077 INFO:teuthology.orchestra.run.smithi064.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 9 22:40 /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.077 DEBUG:teuthology.orchestra.run.smithi089:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/main/290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.625 INFO:teuthology.orchestra.run.smithi089.stdout:-rw-r--r--. 1 ubuntu ubuntu 766809 Feb 9 22:40 /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.625 DEBUG:teuthology.orchestra.run.smithi003:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.644 DEBUG:teuthology.orchestra.run.smithi064:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.663 DEBUG:teuthology.orchestra.run.smithi089:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-02-09T22:40:08.698 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 on all hosts... 2024-02-09T22:40:08.698 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-09T22:40:08.700 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-09T22:40:08.705 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 pull 2024-02-09T22:40:09.527 INFO:teuthology.orchestra.run.smithi089.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-09T22:40:09.538 INFO:teuthology.orchestra.run.smithi064.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-09T22:40:09.598 INFO:teuthology.orchestra.run.smithi003.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout: "repo_digests": [ 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout: ] 2024-02-09T22:40:37.699 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "repo_digests": [ 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-02-09T22:40:55.199 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-09T22:40:59.638 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:40:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_version": "ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev)", 2024-02-09T22:40:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: "image_id": "c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6", 2024-02-09T22:40:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: "repo_digests": [ 2024-02-09T22:40:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58" 2024-02-09T22:40:59.639 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T22:40:59.639 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:40:59.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo mkdir -p /etc/ceph 2024-02-09T22:40:59.711 DEBUG:teuthology.orchestra.run.smithi064:> sudo mkdir -p /etc/ceph 2024-02-09T22:40:59.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph 2024-02-09T22:40:59.789 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 777 /etc/ceph 2024-02-09T22:40:59.816 DEBUG:teuthology.orchestra.run.smithi064:> sudo chmod 777 /etc/ceph 2024-02-09T22:40:59.851 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /etc/ceph 2024-02-09T22:40:59.883 INFO:tasks.cephadm:Writing seed config... 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [client] debug client = 20 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-02-09T22:40:59.884 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-02-09T22:40:59.885 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-02-09T22:40:59.886 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-02-09T22:40:59.887 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-02-09T22:40:59.887 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-02-09T22:40:59.887 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-02-09T22:40:59.887 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-02-09T22:40:59.887 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-02-09T22:40:59.887 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-02-09T22:40:59.888 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:40:59.888 DEBUG:teuthology.orchestra.run.smithi003:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-02-09T22:40:59.902 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2bf8a73a-c79c-11ee-95b8-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-02-09T22:40:59.903 DEBUG:teuthology.orchestra.run.smithi003:mon.a> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a.service 2024-02-09T22:40:59.945 DEBUG:teuthology.orchestra.run.smithi003:mgr.x> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x.service 2024-02-09T22:40:59.988 INFO:tasks.cephadm:Bootstrapping... 2024-02-09T22:40:59.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 -v bootstrap --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.3 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-02-09T22:41:00.185 INFO:teuthology.orchestra.run.smithi003.stdout:-------------------------------------------------------------------------------- 2024-02-09T22:41:00.185 INFO:teuthology.orchestra.run.smithi003.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90', '-v', 'bootstrap', '--fsid', '2bf8a73a-c79c-11ee-95b8-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.3', '--skip-admin-label'] 2024-02-09T22:41:00.210 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-09T22:41:00.210 INFO:teuthology.orchestra.run.smithi003.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-02-09T22:41:00.210 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying podman|docker is present... 2024-02-09T22:41:00.236 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-09T22:41:00.236 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying lvm2 is present... 2024-02-09T22:41:00.236 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying time synchronization is in place... 2024-02-09T22:41:00.241 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-09T22:41:00.241 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-09T22:41:00.246 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-09T22:41:00.247 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-02-09T22:41:00.253 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-02-09T22:41:00.259 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-02-09T22:41:00.259 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-02-09T22:41:00.259 INFO:teuthology.orchestra.run.smithi003.stdout:Repeating the final host check... 2024-02-09T22:41:00.282 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout 4.9.3-dev 2024-02-09T22:41:00.283 INFO:teuthology.orchestra.run.smithi003.stdout:podman (/bin/podman) version 4.9.3 is present 2024-02-09T22:41:00.283 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl is present 2024-02-09T22:41:00.283 INFO:teuthology.orchestra.run.smithi003.stdout:lvcreate is present 2024-02-09T22:41:00.288 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-02-09T22:41:00.288 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-02-09T22:41:00.294 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-02-09T22:41:00.294 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-02-09T22:41:00.300 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout enabled 2024-02-09T22:41:00.307 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout active 2024-02-09T22:41:00.307 INFO:teuthology.orchestra.run.smithi003.stdout:Unit chronyd.service is enabled and running 2024-02-09T22:41:00.307 INFO:teuthology.orchestra.run.smithi003.stdout:Host looks OK 2024-02-09T22:41:00.308 INFO:teuthology.orchestra.run.smithi003.stdout:Cluster fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:00.308 INFO:teuthology.orchestra.run.smithi003.stdout:Acquiring lock 140707952866928 on /run/cephadm/2bf8a73a-c79c-11ee-95b8-87774f69a715.lock 2024-02-09T22:41:00.308 INFO:teuthology.orchestra.run.smithi003.stdout:Lock 140707952866928 acquired on /run/cephadm/2bf8a73a-c79c-11ee-95b8-87774f69a715.lock 2024-02-09T22:41:00.308 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 3300 ... 2024-02-09T22:41:00.308 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying IP 172.21.15.3 port 6789 ... 2024-02-09T22:41:00.309 INFO:teuthology.orchestra.run.smithi003.stdout:Base mon IP(s) is [172.21.15.3:3300, 172.21.15.3:6789], mon addrv is [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-02-09T22:41:00.311 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.3 metric 100 2024-02-09T22:41:00.311 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.3 metric 100 2024-02-09T22:41:00.313 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-02-09T22:41:00.313 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-02-09T22:41:00.314 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-02-09T22:41:00.316 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-02-09T22:41:00.316 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-02-09T22:41:00.316 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-09T22:41:00.316 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-02-09T22:41:00.316 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-02-09T22:41:00.316 INFO:teuthology.orchestra.run.smithi003.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-02-09T22:41:00.317 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-02-09T22:41:00.317 INFO:teuthology.orchestra.run.smithi003.stdout:Mon IP `172.21.15.3` is in CIDR network `172.21.0.0/20` 2024-02-09T22:41:00.317 INFO:teuthology.orchestra.run.smithi003.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-02-09T22:41:00.317 INFO:teuthology.orchestra.run.smithi003.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-02-09T22:41:00.318 INFO:teuthology.orchestra.run.smithi003.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stdout c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90... 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Getting image source signatures 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:5fe0e97a267976e17c7dcb240cf9455472de6876101ae78af4bdf182158b6d8d 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying blob sha256:58df4e03ba066accf3a7e07426bab5f1c867e27a139801c642bdafb42b9a2d9a 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Copying config sha256:c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 2024-02-09T22:41:00.766 INFO:teuthology.orchestra.run.smithi003.stdout:/bin/podman: stderr Writing manifest to image destination 2024-02-09T22:41:01.687 INFO:teuthology.orchestra.run.smithi003.stdout:ceph: stdout ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-09T22:41:01.687 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph version: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev) 2024-02-09T22:41:01.687 INFO:teuthology.orchestra.run.smithi003.stdout:Extracting ceph user uid/gid from container image... 2024-02-09T22:41:02.702 INFO:teuthology.orchestra.run.smithi003.stdout:stat: stdout 167 167 2024-02-09T22:41:02.702 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial keys... 2024-02-09T22:41:03.674 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQD/qcZlvggfAhAAr/uwHHjHIcK+K7bHbsuADQ== 2024-02-09T22:41:04.597 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQAAqsZlnFBWARAAHuUw8Nn3dzPGynFK39SzaQ== 2024-02-09T22:41:05.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph-authtool: stdout AQAAqsZlXkjsNxAAB7alBy3tcdaL73lh/1wstA== 2024-02-09T22:41:05.544 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial monmap... 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:monmaptool for a [v2:172.21.15.3:3300,v1:172.21.15.3:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:setting min_mon_release = quincy 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: set fsid to 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:41:06.417 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mon... 2024-02-09T22:41:07.405 INFO:teuthology.orchestra.run.smithi003.stdout:create mon.a on 2024-02-09T22:41:07.624 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-02-09T22:41:07.778 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-02-09T22:41:07.954 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715.target. 2024-02-09T22:41:07.954 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715.target → /etc/systemd/system/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715.target. 2024-02-09T22:41:08.202 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a 2024-02-09T22:41:08.203 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a.service: Unit ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a.service not loaded. 2024-02-09T22:41:08.375 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715.target.wants/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a.service → /etc/systemd/system/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@.service. 2024-02-09T22:41:08.945 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T22:41:08.946 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-02-09T22:41:08.952 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T22:41:08.952 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-02-09T22:41:08.953 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-02-09T22:41:08.953 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-02-09T22:41:08.953 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon to start... 2024-02-09T22:41:08.953 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mon... 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout cluster: 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout id: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout services: 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.554762s) 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout data: 2024-02-09T22:41:10.031 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-02-09T22:41:10.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-02-09T22:41:10.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-02-09T22:41:10.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout pgs: 2024-02-09T22:41:10.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:10.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.390+0000 7fc15e93c700 1 Processor -- start 2024-02-09T22:41:10.032 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.390+0000 7fc15e93c700 1 -- start start 2024-02-09T22:41:10.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc15e93c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc158106ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:10.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc15e93c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1581070a0 con 0x7fc1581066b0 2024-02-09T22:41:10.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc157fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc158106ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:10.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc157fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc158106ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50632/0 (socket says 172.21.15.3:50632) 2024-02-09T22:41:10.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc157fff700 1 -- 172.21.15.3:0/886761402 learned_addr learned my addr 172.21.15.3:0/886761402 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:10.033 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc157fff700 1 -- 172.21.15.3:0/886761402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1581078b0 con 0x7fc1581066b0 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc157fff700 1 --2- 172.21.15.3:0/886761402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc158106ad0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc1480098a0 tx=0x7fc148009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a40fe3d58fb2c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.391+0000 7fc156ffd700 1 -- 172.21.15.3:0/886761402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148004160 con 0x7fc1581066b0 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc156ffd700 1 -- 172.21.15.3:0/886761402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1480042c0 con 0x7fc1581066b0 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc156ffd700 1 -- 172.21.15.3:0/886761402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1480045a0 con 0x7fc1581066b0 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 -- 172.21.15.3:0/886761402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 msgr2=0x7fc158106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 --2- 172.21.15.3:0/886761402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc158106ad0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fc1480098a0 tx=0x7fc148009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 -- 172.21.15.3:0/886761402 shutdown_connections 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 --2- 172.21.15.3:0/886761402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc158106ad0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 -- 172.21.15.3:0/886761402 >> 172.21.15.3:0/886761402 conn(0x7fc158101cd0 msgr2=0x7fc1581040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 -- 172.21.15.3:0/886761402 shutdown_connections 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.392+0000 7fc15e93c700 1 -- 172.21.15.3:0/886761402 wait complete. 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc15e93c700 1 Processor -- start 2024-02-09T22:41:10.034 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc15e93c700 1 -- start start 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc15e93c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc15819ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc15e93c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15819b250 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc157fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc15819ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc157fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc15819ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50644/0 (socket says 172.21.15.3:50644) 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.393+0000 7fc157fff700 1 -- 172.21.15.3:0/1553588127 learned_addr learned my addr 172.21.15.3:0/1553588127 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc157fff700 1 -- 172.21.15.3:0/1553588127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc148009550 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc157fff700 1 --2- 172.21.15.3:0/1553588127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc15819ad10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc148004000 tx=0x7fc148004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148011630 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc148011790 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc15819b450 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc148011a30 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15819b8f0 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fc148021070 con 0x7fc1581066b0 2024-02-09T22:41:10.035 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.394+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc14801b820 con 0x7fc1581066b0 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.395+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc158194930 con 0x7fc1581066b0 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.397+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7fc14801ba80 con 0x7fc1581066b0 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.470+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc158107df0 con 0x7fc1581066b0 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.471+0000 7fc1557fa700 1 -- 172.21.15.3:0/1553588127 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7fc14801bc20 con 0x7fc1581066b0 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.471+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 msgr2=0x7fc15819ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.471+0000 7fc15e93c700 1 --2- 172.21.15.3:0/1553588127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc15819ad10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc148004000 tx=0x7fc148004750 comp rx=0 tx=0).stop 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.472+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 shutdown_connections 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.472+0000 7fc15e93c700 1 --2- 172.21.15.3:0/1553588127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1581066b0 0x7fc15819ad10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.472+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 >> 172.21.15.3:0/1553588127 conn(0x7fc158101cd0 msgr2=0x7fc158191a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.472+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 shutdown_connections 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:09.472+0000 7fc15e93c700 1 -- 172.21.15.3:0/1553588127 wait complete. 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:mon is available 2024-02-09T22:41:10.036 INFO:teuthology.orchestra.run.smithi003.stdout:Assimilating anything we can from ceph.conf... 2024-02-09T22:41:11.163 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:10 smithi003 ceph-mon[32306]: from='client.? 172.21.15.3:0/2129154698' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.3:3300,v1:172.21.15.3:6789] 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-09T22:41:11.171 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.491+0000 7f5ac7f4e700 1 Processor -- start 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.491+0000 7f5ac7f4e700 1 -- start start 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.491+0000 7f5ac7f4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac010a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.491+0000 7f5ac7f4e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac00757d0 con 0x7f5ac0108280 2024-02-09T22:41:11.172 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac5cea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac010a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac5cea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac010a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50652/0 (socket says 172.21.15.3:50652) 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac5cea700 1 -- 172.21.15.3:0/2598105303 learned_addr learned my addr 172.21.15.3:0/2598105303 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac5cea700 1 -- 172.21.15.3:0/2598105303 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac010abb0 con 0x7f5ac0108280 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac5cea700 1 --2- 172.21.15.3:0/2598105303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac010a670 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80098a0 tx=0x7f5ab8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724d761e2dd73c29 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac4ce8700 1 -- 172.21.15.3:0/2598105303 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ab8004160 con 0x7f5ac0108280 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac4ce8700 1 -- 172.21.15.3:0/2598105303 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab80042c0 con 0x7f5ac0108280 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.492+0000 7f5ac4ce8700 1 -- 172.21.15.3:0/2598105303 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ab80045a0 con 0x7f5ac0108280 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2598105303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 msgr2=0x7f5ac010a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 --2- 172.21.15.3:0/2598105303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac010a670 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80098a0 tx=0x7f5ab8009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2598105303 shutdown_connections 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 --2- 172.21.15.3:0/2598105303 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac010a670 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2598105303 >> 172.21.15.3:0/2598105303 conn(0x7f5ac0101cd0 msgr2=0x7f5ac01040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2598105303 shutdown_connections 2024-02-09T22:41:11.173 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2598105303 wait complete. 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.493+0000 7f5ac7f4e700 1 Processor -- start 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac7f4e700 1 -- start start 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac7f4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac019acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac7f4e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac019b200 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac5cea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac019acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac5cea700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac019acc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50666/0 (socket says 172.21.15.3:50666) 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac5cea700 1 -- 172.21.15.3:0/2129154698 learned_addr learned my addr 172.21.15.3:0/2129154698 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac5cea700 1 -- 172.21.15.3:0/2129154698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab8009550 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac5cea700 1 --2- 172.21.15.3:0/2129154698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac019acc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0074c10 tx=0x7f5ab8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ab8011630 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab8011790 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5ab8011a30 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac019b400 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.494+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac019b8a0 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.495+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5ab8021070 con 0x7f5ac0108280 2024-02-09T22:41:11.174 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.495+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5ab801b7d0 con 0x7f5ac0108280 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.495+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac0194ab0 con 0x7f5ac0108280 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.498+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f5ab801ba30 con 0x7f5ac0108280 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.568+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5ac010ae00 con 0x7f5ac0108280 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.571+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+427 (secure 0 0 0) 0x7f5ab801bc80 con 0x7f5ac0108280 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.572+0000 7f5abeffd700 1 -- 172.21.15.3:0/2129154698 <== mon.0 v2:172.21.15.3:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5ab801be20 con 0x7f5ac0108280 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.572+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 msgr2=0x7f5ac019acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.573+0000 7f5ac7f4e700 1 --2- 172.21.15.3:0/2129154698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac019acc0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f5ac0074c10 tx=0x7f5ab8004750 comp rx=0 tx=0).stop 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.573+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 shutdown_connections 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.573+0000 7f5ac7f4e700 1 --2- 172.21.15.3:0/2129154698 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac0108280 0x7f5ac019acc0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.573+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 >> 172.21.15.3:0/2129154698 conn(0x7f5ac0101cd0 msgr2=0x7f5ac0103f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.573+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 shutdown_connections 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:10.573+0000 7f5ac7f4e700 1 -- 172.21.15.3:0/2129154698 wait complete. 2024-02-09T22:41:11.175 INFO:teuthology.orchestra.run.smithi003.stdout:Generating new minimal ceph.conf... 2024-02-09T22:41:12.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:11 smithi003 ceph-mon[32306]: from='client.? 172.21.15.3:0/3033817962' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:41:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.610+0000 7f3791a51700 1 Processor -- start 2024-02-09T22:41:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.610+0000 7f3791a51700 1 -- start start 2024-02-09T22:41:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f3791a51700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f3791a51700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c108170 con 0x7f378c107780 2024-02-09T22:41:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f378affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f378affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50678/0 (socket says 172.21.15.3:50678) 2024-02-09T22:41:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f378affd700 1 -- 172.21.15.3:0/2986579472 learned_addr learned my addr 172.21.15.3:0/2986579472 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f378affd700 1 -- 172.21.15.3:0/2986579472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378c108980 con 0x7f378c107780 2024-02-09T22:41:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.611+0000 7f378affd700 1 --2- 172.21.15.3:0/2986579472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c107ba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098a0 tx=0x7f377c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca0d74a14a957948 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3789ffb700 1 -- 172.21.15.3:0/2986579472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c004160 con 0x7f378c107780 2024-02-09T22:41:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3789ffb700 1 -- 172.21.15.3:0/2986579472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f377c0042c0 con 0x7f378c107780 2024-02-09T22:41:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3789ffb700 1 -- 172.21.15.3:0/2986579472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c011ac0 con 0x7f378c107780 2024-02-09T22:41:12.262 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 -- 172.21.15.3:0/2986579472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 msgr2=0x7f378c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 --2- 172.21.15.3:0/2986579472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c107ba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f377c0098a0 tx=0x7f377c009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 -- 172.21.15.3:0/2986579472 shutdown_connections 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 --2- 172.21.15.3:0/2986579472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c107ba0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 -- 172.21.15.3:0/2986579472 >> 172.21.15.3:0/2986579472 conn(0x7f378c102dc0 msgr2=0x7f378c1051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 -- 172.21.15.3:0/2986579472 shutdown_connections 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.612+0000 7f3791a51700 1 -- 172.21.15.3:0/2986579472 wait complete. 2024-02-09T22:41:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f3791a51700 1 Processor -- start 2024-02-09T22:41:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f3791a51700 1 -- start start 2024-02-09T22:41:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f3791a51700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c19b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f3791a51700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f378c19bac0 con 0x7f378c107780 2024-02-09T22:41:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f378affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c19b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f378affd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c19b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50688/0 (socket says 172.21.15.3:50688) 2024-02-09T22:41:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.613+0000 7f378affd700 1 -- 172.21.15.3:0/3033817962 learned_addr learned my addr 172.21.15.3:0/3033817962 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f378affd700 1 -- 172.21.15.3:0/3033817962 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c009550 con 0x7f378c107780 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f378affd700 1 --2- 172.21.15.3:0/3033817962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c19b580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f377c009fb0 tx=0x7f377c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c0038b0 con 0x7f378c107780 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f377c003a10 con 0x7f378c107780 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f378c19bcc0 con 0x7f378c107780 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f377c0195e0 con 0x7f378c107780 2024-02-09T22:41:12.265 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c19c160 con 0x7f378c107780 2024-02-09T22:41:12.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f377c019c00 con 0x7f378c107780 2024-02-09T22:41:12.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.614+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f377c024990 con 0x7f378c107780 2024-02-09T22:41:12.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.615+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f378c10bbb0 con 0x7f378c107780 2024-02-09T22:41:12.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.617+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f377c003b80 con 0x7f378c107780 2024-02-09T22:41:12.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.688+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f378c19ecc0 con 0x7f378c107780 2024-02-09T22:41:12.266 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.688+0000 7f3790a4f700 1 -- 172.21.15.3:0/3033817962 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+173 (secure 0 0 0) 0x7f377c01aa10 con 0x7f378c107780 2024-02-09T22:41:12.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 msgr2=0x7f378c19b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:12.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 --2- 172.21.15.3:0/3033817962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c19b580 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f377c009fb0 tx=0x7f377c011b20 comp rx=0 tx=0).stop 2024-02-09T22:41:12.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 shutdown_connections 2024-02-09T22:41:12.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 --2- 172.21.15.3:0/3033817962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f378c107780 0x7f378c19b580 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:12.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 >> 172.21.15.3:0/3033817962 conn(0x7f378c102dc0 msgr2=0x7f378c1038b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:12.267 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 shutdown_connections 2024-02-09T22:41:12.268 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:11.689+0000 7f3791a51700 1 -- 172.21.15.3:0/3033817962 wait complete. 2024-02-09T22:41:12.268 INFO:teuthology.orchestra.run.smithi003.stdout:Restarting the monitor... 2024-02-09T22:41:12.515 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:12 smithi003 systemd[1]: Stopping Ceph mon.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:41:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a[32302]: 2024-02-09T22:41:12.510+0000 7f5e37df0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T22:41:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a[32302]: 2024-02-09T22:41:12.510+0000 7f5e37df0700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-02-09T22:41:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:12 smithi003 podman[32551]: 2024-02-09 22:41:12.736567497 +0000 UTC m=+0.321267183 container died 60a38d6f4ddd24eb84726b17b8e1d316949d2f1ef88076ddbd9e5634119b0e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:41:13.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 podman[32551]: 2024-02-09 22:41:13.085808009 +0000 UTC m=+0.670507694 container cleanup 60a38d6f4ddd24eb84726b17b8e1d316949d2f1ef88076ddbd9e5634119b0e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-02-09T22:41:13.268 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 bash[32551]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a 2024-02-09T22:41:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 podman[32565]: 2024-02-09 22:41:13.378963793 +0000 UTC m=+0.641186016 container remove 60a38d6f4ddd24eb84726b17b8e1d316949d2f1ef88076ddbd9e5634119b0e5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-09T22:41:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a.service: Deactivated successfully. 2024-02-09T22:41:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 systemd[1]: Stopped Ceph mon.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:41:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 systemd[1]: Starting Ceph mon.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:41:13.968 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 podman[32659]: 2024-02-09 22:41:13.610466275 +0000 UTC m=+0.018063691 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:41:13.968 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 podman[32659]: 2024-02-09 22:41:13.721485163 +0000 UTC m=+0.129082604 container create 1d1b447cf16c1daa0cdca5ec400e57a5384f9272d0625114939f13e26b5ef894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS) 2024-02-09T22:41:13.969 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 podman[32659]: 2024-02-09 22:41:13.940554507 +0000 UTC m=+0.348151921 container init 1d1b447cf16c1daa0cdca5ec400e57a5384f9272d0625114939f13e26b5ef894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-09T22:41:13.969 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 podman[32659]: 2024-02-09 22:41:13.943409412 +0000 UTC m=+0.351006811 container start 1d1b447cf16c1daa0cdca5ec400e57a5384f9272d0625114939f13e26b5ef894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-09T22:41:14.047 INFO:teuthology.orchestra.run.smithi003.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: set uid:gid to 167:167 (ceph:ceph) 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: pidfile_write: ignore empty --pid-file 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: load: jerasure load: lrc 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: RocksDB version: 7.9.2 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Git sha 0 2024-02-09T22:41:14.228 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: DB SUMMARY 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: DB Session ID: MKS23ZT5B8474R9GXGEG 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: CURRENT file: CURRENT 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: IDENTITY file: IDENTITY 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100514 ; 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.error_if_exists: 0 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.create_if_missing: 0 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.paranoid_checks: 1 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-09T22:41:14.229 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.env: 0x5569f4777ae0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.fs: PosixFileSystem 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.info_log: 0x5569f52218c0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_file_opening_threads: 16 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.statistics: (nil) 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.use_fsync: 0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_log_file_size: 0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.keep_log_file_num: 1000 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.recycle_log_file_num: 0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.allow_fallocate: 1 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.allow_mmap_reads: 0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.allow_mmap_writes: 0 2024-02-09T22:41:14.230 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.use_direct_reads: 0 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.create_missing_column_families: 0 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.db_log_dir: 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.wal_dir: 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-09T22:41:14.231 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.advise_random_on_open: 1 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.db_write_buffer_size: 0 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.write_buffer_manager: 0x5569f51f5400 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.rate_limiter: (nil) 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-09T22:41:14.232 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.wal_recovery_mode: 2 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.enable_thread_tracking: 0 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.enable_pipelined_write: 0 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.unordered_write: 0 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.row_cache: None 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.wal_filter: None 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.allow_ingest_behind: 0 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.two_write_queues: 0 2024-02-09T22:41:14.233 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.manual_wal_flush: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.wal_compression: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.atomic_flush: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.log_readahead_size: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.best_efforts_recovery: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.allow_data_in_errors: 0 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.db_host_id: __hostname__ 2024-02-09T22:41:14.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-09T22:41:14.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_background_jobs: 2 2024-02-09T22:41:14.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_background_compactions: -1 2024-02-09T22:41:14.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_subcompactions: 1 2024-02-09T22:41:14.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-09T22:41:14.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-09T22:41:14.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-09T22:41:14.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_total_wal_size: 0 2024-02-09T22:41:14.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-09T22:41:14.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-09T22:41:14.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-09T22:41:14.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_open_files: -1 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bytes_per_sync: 0 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_readahead_size: 0 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_background_flushes: -1 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Compression algorithms supported: 2024-02-09T22:41:14.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kZSTD supported: 0 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kXpressCompression supported: 0 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kLZ4HCCompression supported: 1 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kZlibCompression supported: 1 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kSnappyCompression supported: 1 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kLZ4Compression supported: 1 2024-02-09T22:41:14.240 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: kBZip2Compression supported: 0 2024-02-09T22:41:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-09T22:41:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-09T22:41:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-02-09T22:41:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-09T22:41:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-09T22:41:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.merge_operator: 2024-02-09T22:41:14.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_filter: None 2024-02-09T22:41:14.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_filter_factory: None 2024-02-09T22:41:14.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.sst_partitioner_factory: None 2024-02-09T22:41:14.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-09T22:41:14.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-09T22:41:14.242 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5569f52219c0) 2024-02-09T22:41:14.243 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks: 1 2024-02-09T22:41:14.243 INFO:journalctl@ceph.mon.a.smithi003.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-09T22:41:14.243 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-09T22:41:14.243 INFO:journalctl@ceph.mon.a.smithi003.stdout: pin_top_level_index_and_filter: 1 2024-02-09T22:41:14.243 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_type: 0 2024-02-09T22:41:14.243 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_index_type: 0 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_shortening: 1 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: checksum: 4 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: no_block_cache: 0 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache: 0x5569f5225090 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_name: BinnedLRUCache 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_options: 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: capacity : 536870912 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_shard_bits : 4 2024-02-09T22:41:14.244 INFO:journalctl@ceph.mon.a.smithi003.stdout: strict_capacity_limit : 0 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: high_pri_pool_ratio: 0.000 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_cache_compressed: (nil) 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: persistent_cache: (nil) 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size: 4096 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_size_deviation: 10 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_restart_interval: 16 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: index_block_restart_interval: 1 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: metadata_block_size: 4096 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: partition_filters: 0 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: use_delta_encoding: 1 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: filter_policy: bloomfilter 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: whole_key_filtering: 1 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: verify_compression: 0 2024-02-09T22:41:14.245 INFO:journalctl@ceph.mon.a.smithi003.stdout: read_amp_bytes_per_bit: 0 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: format_version: 5 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_index_compression: 1 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: block_align: 0 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: max_auto_readahead_size: 262144 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: prepopulate_block_cache: 0 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: initial_auto_readahead_size: 8192 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout: num_file_reads_for_auto_readahead: 2 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.write_buffer_size: 33554432 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_write_buffer_number: 2 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression: NoCompression 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression: Disabled 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.prefix_extractor: nullptr 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-09T22:41:14.246 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.num_levels: 7 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.level: 32767 2024-02-09T22:41:14.247 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.strategy: 0 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.enabled: false 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.target_file_size_base: 67108864 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-09T22:41:14.248 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.arena_block_size: 1048576 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-09T22:41:14.249 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.disable_auto_compactions: 0 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.table_properties_collectors: 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.inplace_update_support: 0 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-09T22:41:14.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.bloom_locality: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.max_successive_merges: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.paranoid_file_checks: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.force_consistency_checks: 1 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.report_bg_io_stats: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.ttl: 2592000 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.enable_blob_files: false 2024-02-09T22:41:14.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.min_blob_size: 0 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.blob_file_size: 268435456 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.blob_file_starting_level: 0 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c1bc7042-a8c9-48e8-a5b5-d0fddcde7026 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707518473972246, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-02-09T22:41:14.252 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707518473973179, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95575, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 320, "table_properties": {"data_size": 93565, "index_size": 292, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13989, "raw_average_key_size": 50, "raw_value_size": 86006, "raw_average_value_size": 308, "num_data_blocks": 11, "num_entries": 279, "num_filter_entries": 279, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707518473, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c1bc7042-a8c9-48e8-a5b5-d0fddcde7026", "db_session_id": "MKS23ZT5B8474R9GXGEG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707518473973262, "job": 1, "event": "recovery_finished"} 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5569f5316000 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: DB pointer 0x5569f5300000 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** DB Stats ** 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T22:41:14.253 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: L0 2/0 95.17 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: Sum 2/0 95.17 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** Compaction Stats [default] ** 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T22:41:14.254 INFO:journalctl@ceph.mon.a.smithi003.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 158.8 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Cumulative compaction: 0.00 GB write, 31.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Interval compaction: 0.00 GB write, 31.63 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache BinnedLRUCache@0x5569f5225090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-02-09T22:41:14.255 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: starting mon.a rank 0 at public addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] at bind addrs [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???) e1 preinit fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).mds e1 new map 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).mds e1 print_map 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: e1 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: legacy client fscid: -1 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout: No filesystems configured 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T22:41:14.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:13 smithi003 ceph-mon[32678]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 bash[32659]: 1d1b447cf16c1daa0cdca5ec400e57a5384f9272d0625114939f13e26b5ef894 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 ceph-mon[32678]: mon.a is new leader, mons a in quorum (ranks 0) 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 ceph-mon[32678]: monmap e1: 1 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 ceph-mon[32678]: fsmap 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 ceph-mon[32678]: osdmap e1: 0 total, 0 up, 0 in 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 ceph-mon[32678]: mgrmap e1: no daemons active 2024-02-09T22:41:14.257 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:14 smithi003 systemd[1]: Started Ceph mon.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:41:15.195 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.573+0000 7f682c274700 1 Processor -- start 2024-02-09T22:41:15.195 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.573+0000 7f682c274700 1 -- start start 2024-02-09T22:41:15.195 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.573+0000 7f682c274700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f6824107bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:15.195 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.573+0000 7f682c274700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6824108190 con 0x7f68241077a0 2024-02-09T22:41:15.195 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682a010700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f6824107bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682a010700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f6824107bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50696/0 (socket says 172.21.15.3:50696) 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682a010700 1 -- 172.21.15.3:0/3321058473 learned_addr learned my addr 172.21.15.3:0/3321058473 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682a010700 1 -- 172.21.15.3:0/3321058473 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6824108950 con 0x7f68241077a0 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682a010700 1 --2- 172.21.15.3:0/3321058473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f6824107bc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f681c0098a0 tx=0x7f681c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1220f1381bea77 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682900e700 1 -- 172.21.15.3:0/3321058473 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f681c004160 con 0x7f68241077a0 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682900e700 1 -- 172.21.15.3:0/3321058473 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f681c0042c0 con 0x7f68241077a0 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.574+0000 7f682900e700 1 -- 172.21.15.3:0/3321058473 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f681c011ac0 con 0x7f68241077a0 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 -- 172.21.15.3:0/3321058473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 msgr2=0x7f6824107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 --2- 172.21.15.3:0/3321058473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f6824107bc0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f681c0098a0 tx=0x7f681c009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 -- 172.21.15.3:0/3321058473 shutdown_connections 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 --2- 172.21.15.3:0/3321058473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f6824107bc0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 -- 172.21.15.3:0/3321058473 >> 172.21.15.3:0/3321058473 conn(0x7f6824102dc0 msgr2=0x7f68241051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:15.196 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 -- 172.21.15.3:0/3321058473 shutdown_connections 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.575+0000 7f682c274700 1 -- 172.21.15.3:0/3321058473 wait complete. 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682c274700 1 Processor -- start 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682c274700 1 -- start start 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682c274700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f682419b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682c274700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f682419bb00 con 0x7f68241077a0 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682a010700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f682419b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682a010700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f682419b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50712/0 (socket says 172.21.15.3:50712) 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682a010700 1 -- 172.21.15.3:0/2165736013 learned_addr learned my addr 172.21.15.3:0/2165736013 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682a010700 1 -- 172.21.15.3:0/2165736013 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c009550 con 0x7f68241077a0 2024-02-09T22:41:15.197 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f682a010700 1 --2- 172.21.15.3:0/2165736013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f682419b5c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f681c011780 tx=0x7f681c011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:15.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.576+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f681c003af0 con 0x7f68241077a0 2024-02-09T22:41:15.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f681c003c50 con 0x7f68241077a0 2024-02-09T22:41:15.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f682419bd00 con 0x7f68241077a0 2024-02-09T22:41:15.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f681c0197e0 con 0x7f68241077a0 2024-02-09T22:41:15.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f682419c1a0 con 0x7f68241077a0 2024-02-09T22:41:15.199 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f681c025460 con 0x7f68241077a0 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f681c00bb70 con 0x7f68241077a0 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.577+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f682410bb80 con 0x7f68241077a0 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.580+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f681c02a060 con 0x7f68241077a0 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.649+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f682410bd90 con 0x7f68241077a0 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.651+0000 7f68237fe700 1 -- 172.21.15.3:0/2165736013 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f681c019db0 con 0x7f68241077a0 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 msgr2=0x7f682419b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 --2- 172.21.15.3:0/2165736013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f682419b5c0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f681c011780 tx=0x7f681c011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 shutdown_connections 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 --2- 172.21.15.3:0/2165736013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f68241077a0 0x7f682419b5c0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 >> 172.21.15.3:0/2165736013 conn(0x7f6824102dc0 msgr2=0x7f6824103950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:15.200 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 shutdown_connections 2024-02-09T22:41:15.201 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:14.652+0000 7f682c274700 1 -- 172.21.15.3:0/2165736013 wait complete. 2024-02-09T22:41:15.201 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote config to /etc/ceph/ceph.conf 2024-02-09T22:41:15.216 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-02-09T22:41:15.217 INFO:teuthology.orchestra.run.smithi003.stdout:Creating mgr... 2024-02-09T22:41:15.217 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:9283 ... 2024-02-09T22:41:15.217 INFO:teuthology.orchestra.run.smithi003.stdout:Verifying port 0.0.0.0:8765 ... 2024-02-09T22:41:15.475 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x 2024-02-09T22:41:15.475 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Failed to reset failed state of unit ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x.service: Unit ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x.service not loaded. 2024-02-09T22:41:15.622 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715.target.wants/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@.service. 2024-02-09T22:41:16.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:15 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2165736013' entity='client.admin' 2024-02-09T22:41:16.133 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:15 smithi003 podman[32895]: 2024-02-09 22:41:15.756084664 +0000 UTC m=+0.017139769 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:41:17.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:16 smithi003 podman[32895]: 2024-02-09 22:41:16.96697456 +0000 UTC m=+1.228029658 container create 0ae12abd5b181113b24a7287707a0239b73ba19a6b6bbc8ae26b568588cee8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-09T22:41:19.082 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:18 smithi003 podman[32895]: 2024-02-09 22:41:18.759873826 +0000 UTC m=+3.020928925 container init 0ae12abd5b181113b24a7287707a0239b73ba19a6b6bbc8ae26b568588cee8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a) 2024-02-09T22:41:19.082 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:18 smithi003 podman[32895]: 2024-02-09 22:41:18.763380056 +0000 UTC m=+3.024435154 container start 0ae12abd5b181113b24a7287707a0239b73ba19a6b6bbc8ae26b568588cee8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-09T22:41:19.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:19 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:19.342+0000 7f524056b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:41:19.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:19 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:19.438+0000 7f524056b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:41:19.651 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T22:41:19.651 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-02-09T22:41:19.658 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T22:41:19.658 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-02-09T22:41:19.658 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-02-09T22:41:19.658 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to enable service . firewalld.service is not available 2024-02-09T22:41:19.666 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T22:41:19.666 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-02-09T22:41:19.673 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T22:41:19.673 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-02-09T22:41:19.674 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-02-09T22:41:19.674 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-02-09T22:41:19.674 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr to start... 2024-02-09T22:41:19.674 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr... 2024-02-09T22:41:20.113 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:19 smithi003 bash[32895]: 0ae12abd5b181113b24a7287707a0239b73ba19a6b6bbc8ae26b568588cee8e5 2024-02-09T22:41:20.113 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:19 smithi003 systemd[1]: Started Ceph mgr.x for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:41:20.113 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.108+0000 7f524056b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:41:20.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.203+0000 7f524056b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:41:20.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.347+0000 7f524056b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:41:20.802 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.445+0000 7f524056b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:41:20.803 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.530+0000 7f524056b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:41:20.803 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.798+0000 7f524056b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:41:21.100 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:20 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.911+0000 7f524056b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:41:21.100 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:21 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:20.999+0000 7f524056b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:41:21.100 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:21 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:21.096+0000 7f524056b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:41:21.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:21 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:21.274+0000 7f524056b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:41:22.585 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:22 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:22.287+0000 7f524056b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:41:22.585 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:22 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:22.424+0000 7f524056b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:41:22.868 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:22 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/900009382' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T22:41:22.868 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:22 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:22.771+0000 7f524056b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:41:22.868 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:22 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:22.864+0000 7f524056b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:41:23.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:23 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:23.064+0000 7f524056b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:41:23.550 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:23 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:23.253+0000 7f524056b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:41:23.814 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:23 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:23.545+0000 7f524056b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:41:23.814 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:23 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:23.809+0000 7f524056b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:41:24.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:24 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:24.052+0000 7f524056b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:41:25.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:24 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:24.884+0000 7f524056b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-02-09T22:41:25.149 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T22:41:25.150 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.151 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": false, 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T22:41:25.152 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T22:41:08.918280+0000", 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.499+0000 7f57d9cf8700 1 Processor -- start 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.499+0000 7f57d9cf8700 1 -- start start 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.499+0000 7f57d9cf8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d4106ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.499+0000 7f57d9cf8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d41070a0 con 0x7f57d41066b0 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.499+0000 7f57d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d4106ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d4106ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51714/0 (socket says 172.21.15.3:51714) 2024-02-09T22:41:25.153 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d37fe700 1 -- 172.21.15.3:0/2529829421 learned_addr learned my addr 172.21.15.3:0/2529829421 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d37fe700 1 -- 172.21.15.3:0/2529829421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d41078b0 con 0x7f57d41066b0 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d37fe700 1 --2- 172.21.15.3:0/2529829421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d4106ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57c40098a0 tx=0x7f57c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e5582bfe6bbcc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d2ffd700 1 -- 172.21.15.3:0/2529829421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57c4004160 con 0x7f57d41066b0 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d2ffd700 1 -- 172.21.15.3:0/2529829421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57c40042c0 con 0x7f57d41066b0 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.500+0000 7f57d2ffd700 1 -- 172.21.15.3:0/2529829421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57c4011ac0 con 0x7f57d41066b0 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 -- 172.21.15.3:0/2529829421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 msgr2=0x7f57d4106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 --2- 172.21.15.3:0/2529829421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d4106ad0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f57c40098a0 tx=0x7f57c4009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 -- 172.21.15.3:0/2529829421 shutdown_connections 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 --2- 172.21.15.3:0/2529829421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d4106ad0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 -- 172.21.15.3:0/2529829421 >> 172.21.15.3:0/2529829421 conn(0x7f57d4101cd0 msgr2=0x7f57d41040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 -- 172.21.15.3:0/2529829421 shutdown_connections 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 -- 172.21.15.3:0/2529829421 wait complete. 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.501+0000 7f57d9cf8700 1 Processor -- start 2024-02-09T22:41:25.154 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d9cf8700 1 -- start start 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d9cf8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d40753f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d9cf8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d419e350 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d40753f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d37fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d40753f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51730/0 (socket says 172.21.15.3:51730) 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d37fe700 1 -- 172.21.15.3:0/900009382 learned_addr learned my addr 172.21.15.3:0/900009382 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d37fe700 1 -- 172.21.15.3:0/900009382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c4009550 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.502+0000 7f57d37fe700 1 --2- 172.21.15.3:0/900009382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d40753f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57c4011780 tx=0x7f57c4011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57c4003af0 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d419e490 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f57c4003c50 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57c40197e0 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d419e870 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f57c4025460 con 0x7f57d41066b0 2024-02-09T22:41:25.155 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.503+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f57c400bb70 con 0x7f57d41066b0 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.504+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d410aae0 con 0x7f57d41066b0 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.506+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f57c402a060 con 0x7f57d41066b0 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.580+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f57d410acf0 con 0x7f57d41066b0 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.581+0000 7f57d17fa700 1 -- 172.21.15.3:0/900009382 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f57c4019db0 con 0x7f57d41066b0 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 msgr2=0x7f57d40753f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 --2- 172.21.15.3:0/900009382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d40753f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f57c4011780 tx=0x7f57c4011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 shutdown_connections 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 --2- 172.21.15.3:0/900009382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f57d41066b0 0x7f57d40753f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 >> 172.21.15.3:0/900009382 conn(0x7f57d4101cd0 msgr2=0x7f57d4102880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 shutdown_connections 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:22.582+0000 7f57d9cf8700 1 -- 172.21.15.3:0/900009382 wait complete. 2024-02-09T22:41:25.156 INFO:teuthology.orchestra.run.smithi003.stdout:mgr not available, waiting (1/15)... 2024-02-09T22:41:25.793 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:25.468+0000 7f524056b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:41:25.793 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:25.569+0000 7f524056b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:41:25.793 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:25.789+0000 7f524056b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: Activating manager daemon x 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: mgrmap e2: x(active, starting, since 0.00284853s) 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:41:26.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T22:41:26.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: Manager daemon x is now available 2024-02-09T22:41:26.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-09T22:41:26.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-09T22:41:26.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' 2024-02-09T22:41:26.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' 2024-02-09T22:41:26.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:25 smithi003 ceph-mon[32678]: from='mgr.14100 172.21.15.3:0/3791916571' entity='mgr.x' 2024-02-09T22:41:28.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:27 smithi003 ceph-mon[32678]: mgrmap e3: x(active, since 1.00709s) 2024-02-09T22:41:28.534 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:28.534 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-02-09T22:41:28.534 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "health": { 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 0 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-02-09T22:41:28.535 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "a" 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "monmap": { 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-02-09T22:41:28.536 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-02-09T22:41:28.537 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-02-09T22:41:28.539 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modules": [ 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "iostat", 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "nfs", 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "restful" 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ], 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "modified": "2024-02-09T22:41:08.918280+0000", 2024-02-09T22:41:28.540 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "services": {} 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout }, 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.624+0000 7f6860049700 1 Processor -- start 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.624+0000 7f6860049700 1 -- start start 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f6860049700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685810a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f6860049700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68580757d0 con 0x7f6858108280 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685dde5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685810a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685dde5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685810a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40614/0 (socket says 172.21.15.3:40614) 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685dde5700 1 -- 172.21.15.3:0/3469358385 learned_addr learned my addr 172.21.15.3:0/3469358385 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685dde5700 1 -- 172.21.15.3:0/3469358385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685810abb0 con 0x7f6858108280 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685dde5700 1 --2- 172.21.15.3:0/3469358385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685810a670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f684c0098a0 tx=0x7f684c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=699e1ec4f1756887 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:28.541 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685cde3700 1 -- 172.21.15.3:0/3469358385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f684c004160 con 0x7f6858108280 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685cde3700 1 -- 172.21.15.3:0/3469358385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f684c0042c0 con 0x7f6858108280 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.625+0000 7f685cde3700 1 -- 172.21.15.3:0/3469358385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f684c011ac0 con 0x7f6858108280 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 -- 172.21.15.3:0/3469358385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 msgr2=0x7f685810a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 --2- 172.21.15.3:0/3469358385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685810a670 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f684c0098a0 tx=0x7f684c009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 -- 172.21.15.3:0/3469358385 shutdown_connections 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 --2- 172.21.15.3:0/3469358385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685810a670 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 -- 172.21.15.3:0/3469358385 >> 172.21.15.3:0/3469358385 conn(0x7f6858101cd0 msgr2=0x7f68581040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 -- 172.21.15.3:0/3469358385 shutdown_connections 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.626+0000 7f6860049700 1 -- 172.21.15.3:0/3469358385 wait complete. 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f6860049700 1 Processor -- start 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f6860049700 1 -- start start 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f6860049700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685819b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f6860049700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685819bad0 con 0x7f6858108280 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f685dde5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685819b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:28.542 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f685dde5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685819b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40624/0 (socket says 172.21.15.3:40624) 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f685dde5700 1 -- 172.21.15.3:0/88367173 learned_addr learned my addr 172.21.15.3:0/88367173 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.627+0000 7f685dde5700 1 -- 172.21.15.3:0/88367173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c009550 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.628+0000 7f685dde5700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685819b590 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f684c009fb0 tx=0x7f684c011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.628+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f684c0038b0 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.628+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685819bcd0 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.628+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f684c003a10 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.628+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f684c0195e0 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.628+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685819c170 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.629+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 3) v1 ==== 45815+0+0 (secure 0 0 0) 0x7f684c0198a0 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.629+0000 7f684affd700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f6844038d30 0x7f684403b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.629+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f684c04d080 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.629+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685810b9b0 con 0x7f6858108280 2024-02-09T22:41:28.543 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.629+0000 7f685d5e4700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f6844038d30 0x7f684403b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.629+0000 7f685d5e4700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f6844038d30 0x7f684403b1f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6854006fd0 tx=0x7f6854006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.634+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f684c019b50 con 0x7f6858108280 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.920+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f685810bb70 con 0x7f6858108280 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.921+0000 7f684affd700 1 -- 172.21.15.3:0/88367173 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f684c018be0 con 0x7f6858108280 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f6844038d30 msgr2=0x7f684403b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f6844038d30 0x7f684403b1f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6854006fd0 tx=0x7f6854006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 msgr2=0x7f685819b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685819b590 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f684c009fb0 tx=0x7f684c011b20 comp rx=0 tx=0).stop 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 shutdown_connections 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f6844038d30 0x7f684403b1f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 --2- 172.21.15.3:0/88367173 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6858108280 0x7f685819b590 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 >> 172.21.15.3:0/88367173 conn(0x7f6858101cd0 msgr2=0x7f6858102810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:28.544 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 shutdown_connections 2024-02-09T22:41:28.545 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:27.923+0000 7f6860049700 1 -- 172.21.15.3:0/88367173 wait complete. 2024-02-09T22:41:28.545 INFO:teuthology.orchestra.run.smithi003.stdout:mgr is available 2024-02-09T22:41:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:28 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/88367173' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-02-09T22:41:30.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:29 smithi003 ceph-mon[32678]: mgrmap e4: x(active, since 3s) 2024-02-09T22:41:30.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:29 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3214759716' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-02-09T22:41:31.369 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:31.369 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [global] 2024-02-09T22:41:31.369 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout fsid = 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:31.369 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [mgr] 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-02-09T22:41:31.370 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 2024-02-09T22:41:31.371 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout [osd] 2024-02-09T22:41:31.371 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-02-09T22:41:31.371 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-02-09T22:41:31.371 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-02-09T22:41:31.371 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-02-09T22:41:31.371 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-02-09T22:41:31.372 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb815737700 1 Processor -- start 2024-02-09T22:41:31.372 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb815737700 1 -- start start 2024-02-09T22:41:31.373 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb815737700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81010a610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:31.373 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb815737700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81007aec0 con 0x7fb810108220 2024-02-09T22:41:31.373 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb80effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81010a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:31.373 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb80effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81010a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40634/0 (socket says 172.21.15.3:40634) 2024-02-09T22:41:31.373 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb80effd700 1 -- 172.21.15.3:0/698640571 learned_addr learned my addr 172.21.15.3:0/698640571 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:31.374 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.919+0000 7fb80effd700 1 -- 172.21.15.3:0/698640571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81010ab50 con 0x7fb810108220 2024-02-09T22:41:31.374 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb80effd700 1 --2- 172.21.15.3:0/698640571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81010a610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb8000098a0 tx=0x7fb800009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d6fda2c1efaeabf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:31.374 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb80dffb700 1 -- 172.21.15.3:0/698640571 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb800004160 con 0x7fb810108220 2024-02-09T22:41:31.374 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb80dffb700 1 -- 172.21.15.3:0/698640571 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb8000042c0 con 0x7fb810108220 2024-02-09T22:41:31.374 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb80dffb700 1 -- 172.21.15.3:0/698640571 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb800011ac0 con 0x7fb810108220 2024-02-09T22:41:31.374 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 -- 172.21.15.3:0/698640571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 msgr2=0x7fb81010a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:31.375 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 --2- 172.21.15.3:0/698640571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81010a610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb8000098a0 tx=0x7fb800009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:31.375 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 -- 172.21.15.3:0/698640571 shutdown_connections 2024-02-09T22:41:31.375 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 --2- 172.21.15.3:0/698640571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81010a610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:31.375 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 -- 172.21.15.3:0/698640571 >> 172.21.15.3:0/698640571 conn(0x7fb810101cf0 msgr2=0x7fb810104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:31.375 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 -- 172.21.15.3:0/698640571 shutdown_connections 2024-02-09T22:41:31.375 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.920+0000 7fb815737700 1 -- 172.21.15.3:0/698640571 wait complete. 2024-02-09T22:41:31.376 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb815737700 1 Processor -- start 2024-02-09T22:41:31.376 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb815737700 1 -- start start 2024-02-09T22:41:31.376 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb815737700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81019b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:31.377 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb815737700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb81019bb00 con 0x7fb810108220 2024-02-09T22:41:31.377 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb80effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81019b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:31.377 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb80effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81019b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40640/0 (socket says 172.21.15.3:40640) 2024-02-09T22:41:31.377 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.921+0000 7fb80effd700 1 -- 172.21.15.3:0/3214759716 learned_addr learned my addr 172.21.15.3:0/3214759716 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:31.377 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb80effd700 1 -- 172.21.15.3:0/3214759716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb800009550 con 0x7fb810108220 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb80effd700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81019b5c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb800009fb0 tx=0x7fb800011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8000038b0 con 0x7fb810108220 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb800003a10 con 0x7fb810108220 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81019bd00 con 0x7fb810108220 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8000195e0 con 0x7fb810108220 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.922+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81019c1a0 con 0x7fb810108220 2024-02-09T22:41:31.378 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.923+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 45921+0+0 (secure 0 0 0) 0x7fb8000198a0 con 0x7fb810108220 2024-02-09T22:41:31.379 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.923+0000 7fb7f7fff700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7fb7f8038e40 0x7fb7f803b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:31.379 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.923+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb80004d040 con 0x7fb810108220 2024-02-09T22:41:31.379 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.923+0000 7fb80e7fc700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7fb7f8038e40 0x7fb7f803b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:31.379 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.923+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81010b9b0 con 0x7fb810108220 2024-02-09T22:41:31.379 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.923+0000 7fb80e7fc700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7fb7f8038e40 0x7fb7f803b300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb804006fd0 tx=0x7fb804006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:31.379 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:28.928+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb80002b340 con 0x7fb810108220 2024-02-09T22:41:31.380 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.132+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fb81010bb90 con 0x7fb810108220 2024-02-09T22:41:31.380 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.133+0000 7fb7f7fff700 1 -- 172.21.15.3:0/3214759716 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fb800018be0 con 0x7fb810108220 2024-02-09T22:41:31.380 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7fb7f8038e40 msgr2=0x7fb7f803b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:31.380 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7fb7f8038e40 0x7fb7f803b300 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb804006fd0 tx=0x7fb804006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:31.380 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 msgr2=0x7fb81019b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:31.380 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81019b5c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb800009fb0 tx=0x7fb800011b20 comp rx=0 tx=0).stop 2024-02-09T22:41:31.381 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 shutdown_connections 2024-02-09T22:41:31.381 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7fb7f8038e40 0x7fb7f803b300 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:31.381 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 --2- 172.21.15.3:0/3214759716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb810108220 0x7fb81019b5c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:31.381 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 >> 172.21.15.3:0/3214759716 conn(0x7fb810101cf0 msgr2=0x7fb810102830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:31.381 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 shutdown_connections 2024-02-09T22:41:31.381 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:29.135+0000 7fb815737700 1 -- 172.21.15.3:0/3214759716 wait complete. 2024-02-09T22:41:31.382 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling cephadm module... 2024-02-09T22:41:32.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:32 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1534089028' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-02-09T22:41:32.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:32 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: ignoring --setuser ceph since I am not root 2024-02-09T22:41:32.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:32 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: ignoring --setgroup ceph since I am not root 2024-02-09T22:41:32.680 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.762+0000 7f209661f700 1 Processor -- start 2024-02-09T22:41:32.680 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.762+0000 7f209661f700 1 -- start start 2024-02-09T22:41:32.680 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.762+0000 7f209661f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f2090106aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.762+0000 7f209661f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2090107070 con 0x7f2090106680 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f2090106aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f2090106aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40648/0 (socket says 172.21.15.3:40648) 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208ffff700 1 -- 172.21.15.3:0/3156242923 learned_addr learned my addr 172.21.15.3:0/3156242923 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208ffff700 1 -- 172.21.15.3:0/3156242923 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2090107880 con 0x7f2090106680 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208ffff700 1 --2- 172.21.15.3:0/3156242923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f2090106aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f20780098a0 tx=0x7f2078009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d6b605df35a5500 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208effd700 1 -- 172.21.15.3:0/3156242923 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2078004160 con 0x7f2090106680 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208effd700 1 -- 172.21.15.3:0/3156242923 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f20780042c0 con 0x7f2090106680 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.763+0000 7f208effd700 1 -- 172.21.15.3:0/3156242923 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2078011ac0 con 0x7f2090106680 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 -- 172.21.15.3:0/3156242923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 msgr2=0x7f2090106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 --2- 172.21.15.3:0/3156242923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f2090106aa0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f20780098a0 tx=0x7f2078009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 -- 172.21.15.3:0/3156242923 shutdown_connections 2024-02-09T22:41:32.681 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 --2- 172.21.15.3:0/3156242923 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f2090106aa0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 -- 172.21.15.3:0/3156242923 >> 172.21.15.3:0/3156242923 conn(0x7f2090101cc0 msgr2=0x7f20901040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 -- 172.21.15.3:0/3156242923 shutdown_connections 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 -- 172.21.15.3:0/3156242923 wait complete. 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 Processor -- start 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.764+0000 7f209661f700 1 -- start start 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f209661f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f20901917f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f209661f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2090191d30 con 0x7f2090106680 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f20901917f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f20901917f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40654/0 (socket says 172.21.15.3:40654) 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208ffff700 1 -- 172.21.15.3:0/1534089028 learned_addr learned my addr 172.21.15.3:0/1534089028 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208ffff700 1 -- 172.21.15.3:0/1534089028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078009550 con 0x7f2090106680 2024-02-09T22:41:32.682 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208ffff700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f20901917f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2078000c00 tx=0x7f2078011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2078003aa0 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2078003c00 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.765+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209018ff20 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.766+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f20780248a0 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.766+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20901903c0 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.766+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 4) v1 ==== 45921+0+0 (secure 0 0 0) 0x7f207802d3f0 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.766+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209010aab0 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.767+0000 7f208d7fa700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f207c038ee0 0x7f207c03b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.767+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f207804cc50 con 0x7f2090106680 2024-02-09T22:41:32.683 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.767+0000 7f208f7fe700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f207c038ee0 0x7f207c03b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.767+0000 7f208f7fe700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f207c038ee0 0x7f207c03b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2080006fd0 tx=0x7f2080006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:31.772+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2078010740 con 0x7f2090106680 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.030+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f2090195370 con 0x7f2090106680 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.082+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f2078010380 con 0x7f2090106680 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.083+0000 7f208d7fa700 1 -- 172.21.15.3:0/1534089028 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 5) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f2078019410 con 0x7f2090106680 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f207c038ee0 msgr2=0x7f207c03b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f207c038ee0 0x7f207c03b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2080006fd0 tx=0x7f2080006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 msgr2=0x7f20901917f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f20901917f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2078000c00 tx=0x7f2078011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 shutdown_connections 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f207c038ee0 0x7f207c03b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 --2- 172.21.15.3:0/1534089028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2090106680 0x7f20901917f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:32.684 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 >> 172.21.15.3:0/1534089028 conn(0x7f2090101cc0 msgr2=0x7f2090102b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:32.685 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 shutdown_connections 2024-02-09T22:41:32.685 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:32.084+0000 7f209661f700 1 -- 172.21.15.3:0/1534089028 wait complete. 2024-02-09T22:41:32.756 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:32 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:32.653+0000 7fcf4b78e200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:41:32.756 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:32 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:32.753+0000 7fcf4b78e200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:41:33.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1534089028' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-02-09T22:41:33.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-mon[32678]: mgrmap e5: x(active, since 6s) 2024-02-09T22:41:33.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:33.430+0000 7fcf4b78e200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:41:33.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:33.525+0000 7fcf4b78e200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:41:33.978 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:33.671+0000 7fcf4b78e200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:41:33.978 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:33.770+0000 7fcf4b78e200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:41:33.978 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:33.856+0000 7fcf4b78e200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.096+0000 7f07cf96e700 1 Processor -- start 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.096+0000 7f07cf96e700 1 -- start start 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.096+0000 7f07cf96e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c8107bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.096+0000 7f07cf96e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c8108190 con 0x7f07c81077a0 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07cd70a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c8107bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:33.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07cd70a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c8107bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40682/0 (socket says 172.21.15.3:40682) 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07cd70a700 1 -- 172.21.15.3:0/1191503736 learned_addr learned my addr 172.21.15.3:0/1191503736 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07cd70a700 1 -- 172.21.15.3:0/1191503736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c81089a0 con 0x7f07c81077a0 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07cd70a700 1 --2- 172.21.15.3:0/1191503736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c8107bc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f07b80098a0 tx=0x7f07b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=209f789608ce421f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07bffff700 1 -- 172.21.15.3:0/1191503736 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07b8004160 con 0x7f07c81077a0 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07bffff700 1 -- 172.21.15.3:0/1191503736 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07b80042c0 con 0x7f07c81077a0 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.097+0000 7f07bffff700 1 -- 172.21.15.3:0/1191503736 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07b8011ac0 con 0x7f07c81077a0 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 -- 172.21.15.3:0/1191503736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 msgr2=0x7f07c8107bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 --2- 172.21.15.3:0/1191503736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c8107bc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f07b80098a0 tx=0x7f07b8009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 -- 172.21.15.3:0/1191503736 shutdown_connections 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 --2- 172.21.15.3:0/1191503736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c8107bc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 -- 172.21.15.3:0/1191503736 >> 172.21.15.3:0/1191503736 conn(0x7f07c8102da0 msgr2=0x7f07c81051c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 -- 172.21.15.3:0/1191503736 shutdown_connections 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.098+0000 7f07cf96e700 1 -- 172.21.15.3:0/1191503736 wait complete. 2024-02-09T22:41:33.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cf96e700 1 Processor -- start 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cf96e700 1 -- start start 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cf96e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c819b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cf96e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c819bb20 con 0x7f07c81077a0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cd70a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c819b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cd70a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c819b5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40698/0 (socket says 172.21.15.3:40698) 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cd70a700 1 -- 172.21.15.3:0/3251282998 learned_addr learned my addr 172.21.15.3:0/3251282998 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cd70a700 1 -- 172.21.15.3:0/3251282998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b8009550 con 0x7f07c81077a0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07cd70a700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c819b5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009fb0 tx=0x7f07b8011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.099+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07b80038b0 con 0x7f07c81077a0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.100+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07b8003a10 con 0x7f07c81077a0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.100+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c819bd20 con 0x7f07c81077a0 2024-02-09T22:41:33.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.100+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07b80195e0 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.100+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c819c1c0 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.101+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c810bbd0 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.101+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f07b801a430 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.101+0000 7f07be7fc700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 0x7f07b403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.101+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f07b804cfd0 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.101+0000 7f07ccf09700 1 -- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 msgr2=0x7f07b403b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.101+0000 7f07ccf09700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 0x7f07b403b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.110+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f07b8003b80 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.302+0000 7f07ccf09700 1 -- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 msgr2=0x7f07b403b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.302+0000 7f07ccf09700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 0x7f07b403b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.369+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f07c819ef90 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.369+0000 7f07be7fc700 1 -- 172.21.15.3:0/3251282998 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f07b802ce00 con 0x7f07c81077a0 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 msgr2=0x7f07b403b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:41:33.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 0x7f07b403b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 msgr2=0x7f07c819b5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c819b5e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f07b8009fb0 tx=0x7f07b8011b20 comp rx=0 tx=0).stop 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 shutdown_connections 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7f07b4038ee0 0x7f07b403b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 --2- 172.21.15.3:0/3251282998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07c81077a0 0x7f07c819b5e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 >> 172.21.15.3:0/3251282998 conn(0x7f07c8102da0 msgr2=0x7f07c81038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 shutdown_connections 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:33.371+0000 7f07cf96e700 1 -- 172.21.15.3:0/3251282998 wait complete. 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-02-09T22:41:33.990 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 5... 2024-02-09T22:41:34.235 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:34.131+0000 7fcf4b78e200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:41:34.235 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:34 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3251282998' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-09T22:41:34.617 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:34.246+0000 7fcf4b78e200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:41:34.617 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:34.335+0000 7fcf4b78e200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:41:34.617 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:34.434+0000 7fcf4b78e200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:41:34.617 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:34.613+0000 7fcf4b78e200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:41:35.881 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:35.599+0000 7fcf4b78e200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:41:35.881 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:35.733+0000 7fcf4b78e200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:41:36.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:36.070+0000 7fcf4b78e200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:41:36.553 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:36.162+0000 7fcf4b78e200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:41:36.553 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:36.362+0000 7fcf4b78e200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:41:36.553 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:36.550+0000 7fcf4b78e200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:41:37.104 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:36.840+0000 7fcf4b78e200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:41:37.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:37 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:37.101+0000 7fcf4b78e200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:41:37.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:37 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:37.343+0000 7fcf4b78e200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:41:38.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:38.167+0000 7fcf4b78e200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:41:39.074 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:38.750+0000 7fcf4b78e200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:41:39.074 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:38.851+0000 7fcf4b78e200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:41:39.074 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:41:39.071+0000 7fcf4b78e200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:41:40.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: Active manager daemon x restarted 2024-02-09T22:41:40.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: Activating manager daemon x 2024-02-09T22:41:40.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: osdmap e2: 0 total, 0 up, 0 in 2024-02-09T22:41:40.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: mgrmap e6: x(active, starting, since 0.003843s) 2024-02-09T22:41:40.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:41:40.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: Manager daemon x is now available 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: Found migration_current of "None". Setting to last migration. 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:41:40.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-09T22:41:40.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-09T22:41:40.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:40.134 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:39 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-02-09T22:41:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-02-09T22:41:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-09T22:41:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-02-09T22:41:40.761 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff463bc0700 1 Processor -- start 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff463bc0700 1 -- start start 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff463bc0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff463bc0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c107050 con 0x7ff45c106660 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff46195c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff46195c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40710/0 (socket says 172.21.15.3:40710) 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.470+0000 7ff46195c700 1 -- 172.21.15.3:0/3388390281 learned_addr learned my addr 172.21.15.3:0/3388390281 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.471+0000 7ff46195c700 1 -- 172.21.15.3:0/3388390281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45c107860 con 0x7ff45c106660 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.471+0000 7ff46195c700 1 --2- 172.21.15.3:0/3388390281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c106a80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff4580098a0 tx=0x7ff458009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa8c2d8b27d75ff6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.471+0000 7ff46095a700 1 -- 172.21.15.3:0/3388390281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff458004160 con 0x7ff45c106660 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.471+0000 7ff46095a700 1 -- 172.21.15.3:0/3388390281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff4580042c0 con 0x7ff45c106660 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.471+0000 7ff46095a700 1 -- 172.21.15.3:0/3388390281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff458011ac0 con 0x7ff45c106660 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 -- 172.21.15.3:0/3388390281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 msgr2=0x7ff45c106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3388390281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c106a80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff4580098a0 tx=0x7ff458009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:40.762 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 -- 172.21.15.3:0/3388390281 shutdown_connections 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3388390281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c106a80 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 -- 172.21.15.3:0/3388390281 >> 172.21.15.3:0/3388390281 conn(0x7ff45c101c00 msgr2=0x7ff45c104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 -- 172.21.15.3:0/3388390281 shutdown_connections 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.472+0000 7ff463bc0700 1 -- 172.21.15.3:0/3388390281 wait complete. 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff463bc0700 1 Processor -- start 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff463bc0700 1 -- start start 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff463bc0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c1993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff463bc0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45c199910 con 0x7ff45c106660 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff46195c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c1993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff46195c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c1993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40726/0 (socket says 172.21.15.3:40726) 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff46195c700 1 -- 172.21.15.3:0/3668518233 learned_addr learned my addr 172.21.15.3:0/3668518233 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff46195c700 1 -- 172.21.15.3:0/3668518233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff458009550 con 0x7ff45c106660 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff46195c700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c1993d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff458011780 tx=0x7ff458011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:40.763 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.473+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff458003af0 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.474+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff45c199b10 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.474+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff458003c50 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.474+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff4580197e0 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.474+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45c199fb0 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.475+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 5) v1 ==== 45932+0+0 (secure 0 0 0) 0x7ff45801a430 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.475+0000 7ff452ffd700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.475+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 --> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff44803dd10 con 0x7ff44803b140 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.475+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff45804d190 con 0x7ff45c106660 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.475+0000 7ff46115b700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 msgr2=0x7ff44803d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:40.764 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.475+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.676+0000 7ff46115b700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 msgr2=0x7ff44803d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:34.676+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:35.077+0000 7ff46115b700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 msgr2=0x7ff44803d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:35.077+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:35.878+0000 7ff46115b700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 msgr2=0x7ff44803d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:35.878+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:37.479+0000 7ff46115b700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 msgr2=0x7ff44803d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2830848370 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:37.479+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:39.086+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 6) v1 ==== 45699+0+0 (secure 0 0 0) 0x7ff458019a30 con 0x7ff45c106660 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:39.087+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 msgr2=0x7ff44803d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:39.087+0000 7ff452ffd700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.087+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 7) v1 ==== 45826+0+0 (secure 0 0 0) 0x7ff45804ea70 con 0x7ff45c106660 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.088+0000 7ff452ffd700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7ff44803ea80 0x7ff448040e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:40.765 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.088+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff44803dd10 con 0x7ff44803ea80 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.088+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7ff44803ea80 0x7ff448040e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.088+0000 7ff46115b700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7ff44803ea80 0x7ff448040e70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff44c003a10 tx=0x7ff44c009290 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.089+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7ff44803dd10 con 0x7ff44803ea80 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.096+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7ff45c068e00 con 0x7ff44803ea80 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.096+0000 7ff452ffd700 1 -- 172.21.15.3:0/3668518233 <== mgr.14116 v2:172.21.15.3:6800/2045974484 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff45c068e00 con 0x7ff44803ea80 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.096+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7ff44803ea80 msgr2=0x7ff448040e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.096+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7ff44803ea80 0x7ff448040e70 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff44c003a10 tx=0x7ff44c009290 comp rx=0 tx=0).stop 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.096+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 msgr2=0x7ff45c1993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.096+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c1993d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7ff458011780 tx=0x7ff458011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 shutdown_connections 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7ff44803ea80 0x7ff448040e70 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff45c106660 0x7ff45c1993d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 --2- 172.21.15.3:0/3668518233 >> [v2:172.21.15.3:6800/2830848370,v1:172.21.15.3:6801/2830848370] conn(0x7ff44803b140 0x7ff44803d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:40.766 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 >> 172.21.15.3:0/3668518233 conn(0x7ff45c101c00 msgr2=0x7ff45c18fb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:40.767 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 shutdown_connections 2024-02-09T22:41:40.767 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:40.097+0000 7ff463bc0700 1 -- 172.21.15.3:0/3668518233 wait complete. 2024-02-09T22:41:40.767 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 5 is available 2024-02-09T22:41:40.767 INFO:teuthology.orchestra.run.smithi003.stdout:Setting orchestrator backend to cephadm... 2024-02-09T22:41:41.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:41 smithi003 ceph-mon[32678]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-09T22:41:41.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:41 smithi003 ceph-mon[32678]: mgrmap e7: x(active, since 1.00555s) 2024-02-09T22:41:41.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:41 smithi003 ceph-mon[32678]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-09T22:41:42.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.234+0000 7f6ddce38700 1 Processor -- start 2024-02-09T22:41:42.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.234+0000 7f6ddce38700 1 -- start start 2024-02-09T22:41:42.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6ddce38700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd8106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:42.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6ddce38700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd8107090 con 0x7f6dd81066a0 2024-02-09T22:41:42.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd8106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:42.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd8106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41492/0 (socket says 172.21.15.3:41492) 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd659c700 1 -- 172.21.15.3:0/2245112582 learned_addr learned my addr 172.21.15.3:0/2245112582 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd659c700 1 -- 172.21.15.3:0/2245112582 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd81078a0 con 0x7f6dd81066a0 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd659c700 1 --2- 172.21.15.3:0/2245112582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd8106ac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80098a0 tx=0x7f6dc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3319148d1ac4cf78 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd559a700 1 -- 172.21.15.3:0/2245112582 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc8004160 con 0x7f6dd81066a0 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd559a700 1 -- 172.21.15.3:0/2245112582 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc80042c0 con 0x7f6dd81066a0 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.235+0000 7f6dd559a700 1 -- 172.21.15.3:0/2245112582 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc8011ac0 con 0x7f6dd81066a0 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 -- 172.21.15.3:0/2245112582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 msgr2=0x7f6dd8106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 --2- 172.21.15.3:0/2245112582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd8106ac0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6dc80098a0 tx=0x7f6dc8009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 -- 172.21.15.3:0/2245112582 shutdown_connections 2024-02-09T22:41:42.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 --2- 172.21.15.3:0/2245112582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd8106ac0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 -- 172.21.15.3:0/2245112582 >> 172.21.15.3:0/2245112582 conn(0x7f6dd8101ca0 msgr2=0x7f6dd81040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 -- 172.21.15.3:0/2245112582 shutdown_connections 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.236+0000 7f6ddce38700 1 -- 172.21.15.3:0/2245112582 wait complete. 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6ddce38700 1 Processor -- start 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6ddce38700 1 -- start start 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6ddce38700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd819b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6ddce38700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd819bb80 con 0x7f6dd81066a0 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6dd659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd819b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6dd659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd819b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41504/0 (socket says 172.21.15.3:41504) 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6dd659c700 1 -- 172.21.15.3:0/1706903359 learned_addr learned my addr 172.21.15.3:0/1706903359 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6dd659c700 1 -- 172.21.15.3:0/1706903359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc8009550 con 0x7f6dd81066a0 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6dd659c700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd819b640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009fb0 tx=0x7f6dc8011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.237+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc80038b0 con 0x7f6dd81066a0 2024-02-09T22:41:42.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd819bd80 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6dc8003a10 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6dc80195e0 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd819c220 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 7) v1 ==== 45826+0+0 (secure 0 0 0) 0x7f6dc80198a0 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd8195740 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.238+0000 7f6dc77fe700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6dc0038d80 0x7f6dc003b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.239+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6dc804d190 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.239+0000 7f6dd5d9b700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6dc0038d80 0x7f6dc003b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.239+0000 7f6dd5d9b700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6dc0038d80 0x7f6dc003b240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc006fd0 tx=0x7f6dcc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.244+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6dc802b340 con 0x7f6dd81066a0 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.462+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f6dd802da40 con 0x7f6dc0038d80 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.470+0000 7f6dc77fe700 1 -- 172.21.15.3:0/1706903359 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6dd802da40 con 0x7f6dc0038d80 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6dc0038d80 msgr2=0x7f6dc003b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:42.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6dc0038d80 0x7f6dc003b240 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc006fd0 tx=0x7f6dcc006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 msgr2=0x7f6dd819b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd819b640 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8009fb0 tx=0x7f6dc8011b20 comp rx=0 tx=0).stop 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 shutdown_connections 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6dc0038d80 0x7f6dc003b240 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 --2- 172.21.15.3:0/1706903359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6dd81066a0 0x7f6dd819b640 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 >> 172.21.15.3:0/1706903359 conn(0x7f6dd8101ca0 msgr2=0x7f6dd8102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 shutdown_connections 2024-02-09T22:41:42.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:41.472+0000 7f6ddce38700 1 -- 172.21.15.3:0/1706903359 wait complete. 2024-02-09T22:41:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:42 smithi003 ceph-mon[32678]: [09/Feb/2024:22:41:40] ENGINE Bus STARTING 2024-02-09T22:41:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:42 smithi003 ceph-mon[32678]: [09/Feb/2024:22:41:41] ENGINE Serving on https://172.21.15.3:7150 2024-02-09T22:41:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:42 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:41:42.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:42 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:42.312 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:42 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:41:43.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:43 smithi003 ceph-mon[32678]: [09/Feb/2024:22:41:41] ENGINE Serving on http://172.21.15.3:8765 2024-02-09T22:41:43.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:43 smithi003 ceph-mon[32678]: [09/Feb/2024:22:41:41] ENGINE Bus STARTED 2024-02-09T22:41:43.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:43 smithi003 ceph-mon[32678]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:43.315 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:43 smithi003 ceph-mon[32678]: mgrmap e8: x(active, since 3s) 2024-02-09T22:41:43.322 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout value unchanged 2024-02-09T22:41:43.322 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.505+0000 7f0d98e41700 1 Processor -- start 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.505+0000 7f0d98e41700 1 -- start start 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d98e41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d94107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d98e41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d94108170 con 0x7f0d94107780 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d94107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d94107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41516/0 (socket says 172.21.15.3:41516) 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9259c700 1 -- 172.21.15.3:0/1384917776 learned_addr learned my addr 172.21.15.3:0/1384917776 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9259c700 1 -- 172.21.15.3:0/1384917776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d94108980 con 0x7f0d94107780 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9259c700 1 --2- 172.21.15.3:0/1384917776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d94107ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d840098a0 tx=0x7f0d84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e62694804539dd0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9159a700 1 -- 172.21.15.3:0/1384917776 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d84004160 con 0x7f0d94107780 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9159a700 1 -- 172.21.15.3:0/1384917776 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d840042c0 con 0x7f0d94107780 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.506+0000 7f0d9159a700 1 -- 172.21.15.3:0/1384917776 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d84011ac0 con 0x7f0d94107780 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 -- 172.21.15.3:0/1384917776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 msgr2=0x7f0d94107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:43.323 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 --2- 172.21.15.3:0/1384917776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d94107ba0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0d840098a0 tx=0x7f0d84009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 -- 172.21.15.3:0/1384917776 shutdown_connections 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 --2- 172.21.15.3:0/1384917776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d94107ba0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 -- 172.21.15.3:0/1384917776 >> 172.21.15.3:0/1384917776 conn(0x7f0d94102dc0 msgr2=0x7f0d941051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 -- 172.21.15.3:0/1384917776 shutdown_connections 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.507+0000 7f0d98e41700 1 -- 172.21.15.3:0/1384917776 wait complete. 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d98e41700 1 Processor -- start 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d98e41700 1 -- start start 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d98e41700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d9419b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d98e41700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9419bae0 con 0x7f0d94107780 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d9259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d9419b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d9259c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d9419b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41528/0 (socket says 172.21.15.3:41528) 2024-02-09T22:41:43.324 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.508+0000 7f0d9259c700 1 -- 172.21.15.3:0/126425799 learned_addr learned my addr 172.21.15.3:0/126425799 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d9259c700 1 -- 172.21.15.3:0/126425799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d84009550 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d9259c700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d9419b5a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009fb0 tx=0x7f0d84011b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d840038b0 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9419bce0 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9419c180 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0d84003a10 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.509+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0d840195e0 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.510+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f0d840198d0 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.510+0000 7f0d8b7fe700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f0d80038e90 0x7f0d8003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.510+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d74005360 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.510+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0d8404d170 con 0x7f0d94107780 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.510+0000 7f0d91d9b700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f0d80038e90 0x7f0d8003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.510+0000 7f0d91d9b700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f0d80038e90 0x7f0d8003b350 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c006fd0 tx=0x7f0d7c006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:43.325 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.515+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0d84019e00 con 0x7f0d94107780 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.718+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f0d74000bf0 con 0x7f0d80038e90 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.718+0000 7f0d8b7fe700 1 -- 172.21.15.3:0/126425799 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f0d74000bf0 con 0x7f0d80038e90 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f0d80038e90 msgr2=0x7f0d8003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f0d80038e90 0x7f0d8003b350 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c006fd0 tx=0x7f0d7c006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 msgr2=0x7f0d9419b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d9419b5a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0d84009fb0 tx=0x7f0d84011b20 comp rx=0 tx=0).stop 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 shutdown_connections 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f0d80038e90 0x7f0d8003b350 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 --2- 172.21.15.3:0/126425799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d94107780 0x7f0d9419b5a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 >> 172.21.15.3:0/126425799 conn(0x7f0d94102dc0 msgr2=0x7f0d941038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 shutdown_connections 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:42.720+0000 7f0d98e41700 1 -- 172.21.15.3:0/126425799 wait complete. 2024-02-09T22:41:43.326 INFO:teuthology.orchestra.run.smithi003.stdout:Generating ssh key... 2024-02-09T22:41:44.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-mon[32678]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:44.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: Generating public/private rsa key pair. 2024-02-09T22:41:44.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: Your identification has been saved in /tmp/tmppittms13/key. 2024-02-09T22:41:44.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: Your public key has been saved in /tmp/tmppittms13/key.pub. 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: The key fingerprint is: 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: SHA256:hz5e/ovP59qBbRr47tTeIP6EmhzDGukSUk9RJf0TNdA ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: The key's randomart image is: 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: +---[RSA 3072]----+ 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | .oo..oo.| 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | . .. .E.| 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | . . . | 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | . .. o | 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | . oS . . | 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | . ...+ . = | 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | . .= * * B | 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | .o B @ O.+ | 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: | .+ *=%*= .| 2024-02-09T22:41:44.383 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: +----[SHA256]-----+ 2024-02-09T22:41:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e14db4700 1 Processor -- start 2024-02-09T22:41:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e14db4700 1 -- start start 2024-02-09T22:41:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e14db4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e10107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e14db4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e10108170 con 0x7f6e10107780 2024-02-09T22:41:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e0e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e10107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:44.729 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e0e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e10107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41540/0 (socket says 172.21.15.3:41540) 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.735+0000 7f6e0e59c700 1 -- 172.21.15.3:0/530107541 learned_addr learned my addr 172.21.15.3:0/530107541 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e0e59c700 1 -- 172.21.15.3:0/530107541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e10108980 con 0x7f6e10107780 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e0e59c700 1 --2- 172.21.15.3:0/530107541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e10107ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e000098a0 tx=0x7f6e00009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80d272ef6da7d5c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e0d59a700 1 -- 172.21.15.3:0/530107541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e00004160 con 0x7f6e10107780 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e0d59a700 1 -- 172.21.15.3:0/530107541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e000042c0 con 0x7f6e10107780 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e0d59a700 1 -- 172.21.15.3:0/530107541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e00011ac0 con 0x7f6e10107780 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e14db4700 1 -- 172.21.15.3:0/530107541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 msgr2=0x7f6e10107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.736+0000 7f6e14db4700 1 --2- 172.21.15.3:0/530107541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e10107ba0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6e000098a0 tx=0x7f6e00009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 -- 172.21.15.3:0/530107541 shutdown_connections 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 --2- 172.21.15.3:0/530107541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e10107ba0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 -- 172.21.15.3:0/530107541 >> 172.21.15.3:0/530107541 conn(0x7f6e10102dc0 msgr2=0x7f6e101051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 -- 172.21.15.3:0/530107541 shutdown_connections 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 -- 172.21.15.3:0/530107541 wait complete. 2024-02-09T22:41:44.730 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 Processor -- start 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.737+0000 7f6e14db4700 1 -- start start 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e14db4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e1019b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e14db4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1019bae0 con 0x7f6e10107780 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e0e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e1019b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e0e59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e1019b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41556/0 (socket says 172.21.15.3:41556) 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e0e59c700 1 -- 172.21.15.3:0/2047249439 learned_addr learned my addr 172.21.15.3:0/2047249439 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e0e59c700 1 -- 172.21.15.3:0/2047249439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e00009550 con 0x7f6e10107780 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e0e59c700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e1019b5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e00011780 tx=0x7f6e00011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e00003af0 con 0x7f6e10107780 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6e00003c50 con 0x7f6e10107780 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1019bce0 con 0x7f6e10107780 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6e000197e0 con 0x7f6e10107780 2024-02-09T22:41:44.731 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.738+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1019c180 con 0x7f6e10107780 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.739+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f6e000199d0 con 0x7f6e10107780 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.739+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1010bbb0 con 0x7f6e10107780 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.739+0000 7f6dff7fe700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6df8038f30 0x7f6df803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.739+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f6e0004d2a0 con 0x7f6e10107780 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.739+0000 7f6e0dd9b700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6df8038f30 0x7f6df803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.740+0000 7f6e0dd9b700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6df8038f30 0x7f6df803b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006fd0 tx=0x7f6e04006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.744+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6e00018c20 con 0x7f6e10107780 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:43.948+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e10103fe0 con 0x7f6df8038f30 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.129+0000 7f6dff7fe700 1 -- 172.21.15.3:0/2047249439 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f6e10103fe0 con 0x7f6df8038f30 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6df8038f30 msgr2=0x7f6df803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6df8038f30 0x7f6df803b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6e04006fd0 tx=0x7f6e04006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:44.732 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 msgr2=0x7f6e1019b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e1019b5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6e00011780 tx=0x7f6e00011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 shutdown_connections 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f6df8038f30 0x7f6df803b3f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 --2- 172.21.15.3:0/2047249439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6e10107780 0x7f6e1019b5a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 >> 172.21.15.3:0/2047249439 conn(0x7f6e10102dc0 msgr2=0x7f6e101038d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 shutdown_connections 2024-02-09T22:41:44.733 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:44.131+0000 7f6e14db4700 1 -- 172.21.15.3:0/2047249439 wait complete. 2024-02-09T22:41:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:45 smithi003 ceph-mon[32678]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:45 smithi003 ceph-mon[32678]: Generating ssh key... 2024-02-09T22:41:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:45 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:45 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:45.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:41:45.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.185+0000 7fe49fccd700 1 Processor -- start 2024-02-09T22:41:45.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.185+0000 7fe49fccd700 1 -- start start 2024-02-09T22:41:45.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49fccd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe498106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:45.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49fccd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe498107050 con 0x7fe498106660 2024-02-09T22:41:45.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49da69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe498106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:45.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49da69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe498106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41570/0 (socket says 172.21.15.3:41570) 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49da69700 1 -- 172.21.15.3:0/378151505 learned_addr learned my addr 172.21.15.3:0/378151505 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49da69700 1 -- 172.21.15.3:0/378151505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe498107860 con 0x7fe498106660 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.186+0000 7fe49da69700 1 --2- 172.21.15.3:0/378151505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe498106a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4880098a0 tx=0x7fe488009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97287be4e91c65ca server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49ca67700 1 -- 172.21.15.3:0/378151505 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe488004160 con 0x7fe498106660 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49ca67700 1 -- 172.21.15.3:0/378151505 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe4880042c0 con 0x7fe498106660 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49ca67700 1 -- 172.21.15.3:0/378151505 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe488011ac0 con 0x7fe498106660 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 -- 172.21.15.3:0/378151505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 msgr2=0x7fe498106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 --2- 172.21.15.3:0/378151505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe498106a80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4880098a0 tx=0x7fe488009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 -- 172.21.15.3:0/378151505 shutdown_connections 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 --2- 172.21.15.3:0/378151505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe498106a80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 -- 172.21.15.3:0/378151505 >> 172.21.15.3:0/378151505 conn(0x7fe498101c00 msgr2=0x7fe498104040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:45.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 -- 172.21.15.3:0/378151505 shutdown_connections 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.187+0000 7fe49fccd700 1 -- 172.21.15.3:0/378151505 wait complete. 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49fccd700 1 Processor -- start 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49fccd700 1 -- start start 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49fccd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe49819a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49fccd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe49819aa10 con 0x7fe498106660 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49da69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe49819a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49da69700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe49819a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41586/0 (socket says 172.21.15.3:41586) 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49da69700 1 -- 172.21.15.3:0/3689347034 learned_addr learned my addr 172.21.15.3:0/3689347034 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.188+0000 7fe49da69700 1 -- 172.21.15.3:0/3689347034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe488009550 con 0x7fe498106660 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.189+0000 7fe49da69700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe49819a4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe488011780 tx=0x7fe488011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.189+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe488003af0 con 0x7fe498106660 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.189+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe488003c50 con 0x7fe498106660 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.189+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe49819ac10 con 0x7fe498106660 2024-02-09T22:41:45.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.189+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe4880197e0 con 0x7fe498106660 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.189+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe49819b0b0 con 0x7fe498106660 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.190+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7fe48801a430 con 0x7fe498106660 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.190+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe49810aa90 con 0x7fe498106660 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.190+0000 7fe48effd700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fe484038f30 0x7fe48403b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.190+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe48804d240 con 0x7fe498106660 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.190+0000 7fe49d268700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fe484038f30 0x7fe48403b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.191+0000 7fe49d268700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fe484038f30 0x7fe48403b3f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe494006fd0 tx=0x7fe494006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.195+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe48802da40 con 0x7fe498106660 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.396+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe498102ee0 con 0x7fe484038f30 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.396+0000 7fe48effd700 1 -- 172.21.15.3:0/3689347034 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fe498102ee0 con 0x7fe484038f30 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fe484038f30 msgr2=0x7fe48403b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fe484038f30 0x7fe48403b3f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe494006fd0 tx=0x7fe494006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:45.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 msgr2=0x7fe49819a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe49819a4d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe488011780 tx=0x7fe488011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 shutdown_connections 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fe484038f30 0x7fe48403b3f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 --2- 172.21.15.3:0/3689347034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe498106660 0x7fe49819a4d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 >> 172.21.15.3:0/3689347034 conn(0x7fe498101c00 msgr2=0x7fe4981027d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 shutdown_connections 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:45.398+0000 7fe49fccd700 1 -- 172.21.15.3:0/3689347034 wait complete. 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:Adding key to root@localhost authorized_keys... 2024-02-09T22:41:45.991 INFO:teuthology.orchestra.run.smithi003.stdout:Adding host smithi003... 2024-02-09T22:41:46.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:46 smithi003 ceph-mon[32678]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:47.251 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:47 smithi003 ceph-mon[32678]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:48 smithi003 ceph-mon[32678]: Deploying cephadm binary to smithi003 2024-02-09T22:41:51.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Added host 'smithi003' with addr '172.21.15.3' 2024-02-09T22:41:51.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f91f1700 1 Processor -- start 2024-02-09T22:41:51.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f91f1700 1 -- start start 2024-02-09T22:41:51.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f91f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:51.662 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f91f1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f4107050 con 0x7fc1f4106660 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41600/0 (socket says 172.21.15.3:41600) 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.400+0000 7fc1f2d9d700 1 -- 172.21.15.3:0/656585869 learned_addr learned my addr 172.21.15.3:0/656585869 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.401+0000 7fc1f2d9d700 1 -- 172.21.15.3:0/656585869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1f4107860 con 0x7fc1f4106660 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.401+0000 7fc1f2d9d700 1 --2- 172.21.15.3:0/656585869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4106a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00bb40 tx=0x7fc1ec00be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade750d1233b0efc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.401+0000 7fc1f1d9b700 1 -- 172.21.15.3:0/656585869 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec00dbe0 con 0x7fc1f4106660 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.401+0000 7fc1f1d9b700 1 -- 172.21.15.3:0/656585869 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1ec0044f0 con 0x7fc1f4106660 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.401+0000 7fc1f1d9b700 1 -- 172.21.15.3:0/656585869 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec010470 con 0x7fc1f4106660 2024-02-09T22:41:51.663 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 -- 172.21.15.3:0/656585869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 msgr2=0x7fc1f4106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 --2- 172.21.15.3:0/656585869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4106a80 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00bb40 tx=0x7fc1ec00be50 comp rx=0 tx=0).stop 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 -- 172.21.15.3:0/656585869 shutdown_connections 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 --2- 172.21.15.3:0/656585869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4106a80 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 -- 172.21.15.3:0/656585869 >> 172.21.15.3:0/656585869 conn(0x7fc1f4101c60 msgr2=0x7fc1f4104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 -- 172.21.15.3:0/656585869 shutdown_connections 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 -- 172.21.15.3:0/656585869 wait complete. 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.402+0000 7fc1f91f1700 1 Processor -- start 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f91f1700 1 -- start start 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f91f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4199370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:51.664 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f91f1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1f41998b0 con 0x7fc1f4106660 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4199370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f2d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4199370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41610/0 (socket says 172.21.15.3:41610) 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f2d9d700 1 -- 172.21.15.3:0/3255534472 learned_addr learned my addr 172.21.15.3:0/3255534472 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f2d9d700 1 -- 172.21.15.3:0/3255534472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1ec00b7f0 con 0x7fc1f4106660 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.403+0000 7fc1f2d9d700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4199370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00b340 tx=0x7fc1ec010520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.404+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec01e070 con 0x7fc1f4106660 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.404+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc1ec004250 con 0x7fc1f4106660 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.404+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1f4199ab0 con 0x7fc1f4106660 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.404+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc1ec022a80 con 0x7fc1f4106660 2024-02-09T22:41:51.665 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.404+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1f4199f50 con 0x7fc1f4106660 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.404+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7fc1ec02c440 con 0x7fc1f4106660 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.405+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1f4079fe0 con 0x7fc1f4106660 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.405+0000 7fc1ebfff700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fc1e0038ee0 0x7fc1e003b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.405+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc1ec04f300 con 0x7fc1f4106660 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.405+0000 7fc1f259c700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fc1e0038ee0 0x7fc1e003b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.405+0000 7fc1f259c700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fc1e0038ee0 0x7fc1e003b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc006fd0 tx=0x7fc1dc006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.410+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc1ec02a930 con 0x7fc1f4106660 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:46.617+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi003", "addr": "172.21.15.3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1f4062160 con 0x7fc1e0038ee0 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.049+0000 7fc1ebfff700 1 -- 172.21.15.3:0/3255534472 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fc1f4062160 con 0x7fc1e0038ee0 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fc1e0038ee0 msgr2=0x7fc1e003b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:51.666 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fc1e0038ee0 0x7fc1e003b3a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc1dc006fd0 tx=0x7fc1dc006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 msgr2=0x7fc1f4199370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4199370 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc1ec00b340 tx=0x7fc1ec010520 comp rx=0 tx=0).stop 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 shutdown_connections 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fc1e0038ee0 0x7fc1e003b3a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 --2- 172.21.15.3:0/3255534472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1f4106660 0x7fc1f4199370 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 >> 172.21.15.3:0/3255534472 conn(0x7fc1f4101c60 msgr2=0x7fc1f418fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 shutdown_connections 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:51.052+0000 7fc1f91f1700 1 -- 172.21.15.3:0/3255534472 wait complete. 2024-02-09T22:41:51.667 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mon service... 2024-02-09T22:41:52.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:52 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:52.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:52 smithi003 ceph-mon[32678]: Added host smithi003 2024-02-09T22:41:52.292 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:52 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:41:52.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-02-09T22:41:52.986 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effd65b9700 1 Processor -- start 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effd65b9700 1 -- start start 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effd65b9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd0107bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effd65b9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd0108180 con 0x7effd0107790 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd0107bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd0107bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52448/0 (socket says 172.21.15.3:52448) 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effcffff700 1 -- 172.21.15.3:0/2166431423 learned_addr learned my addr 172.21.15.3:0/2166431423 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effcffff700 1 -- 172.21.15.3:0/2166431423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd0108940 con 0x7effd0107790 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.132+0000 7effcffff700 1 --2- 172.21.15.3:0/2166431423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd0107bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7effc00098a0 tx=0x7effc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2182ca479d51b4d7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effceffd700 1 -- 172.21.15.3:0/2166431423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc0004160 con 0x7effd0107790 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effceffd700 1 -- 172.21.15.3:0/2166431423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc00042c0 con 0x7effd0107790 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effceffd700 1 -- 172.21.15.3:0/2166431423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc0011ac0 con 0x7effd0107790 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 -- 172.21.15.3:0/2166431423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 msgr2=0x7effd0107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 --2- 172.21.15.3:0/2166431423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd0107bb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7effc00098a0 tx=0x7effc0009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:52.987 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 -- 172.21.15.3:0/2166431423 shutdown_connections 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 --2- 172.21.15.3:0/2166431423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd0107bb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 -- 172.21.15.3:0/2166431423 >> 172.21.15.3:0/2166431423 conn(0x7effd0102e50 msgr2=0x7effd0105270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 -- 172.21.15.3:0/2166431423 shutdown_connections 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.133+0000 7effd65b9700 1 -- 172.21.15.3:0/2166431423 wait complete. 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effd65b9700 1 Processor -- start 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effd65b9700 1 -- start start 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effd65b9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd019b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effd65b9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effd019ba60 con 0x7effd0107790 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd019b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effcffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd019b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52454/0 (socket says 172.21.15.3:52454) 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effcffff700 1 -- 172.21.15.3:0/3888845524 learned_addr learned my addr 172.21.15.3:0/3888845524 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.134+0000 7effcffff700 1 -- 172.21.15.3:0/3888845524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc0009550 con 0x7effd0107790 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.135+0000 7effcffff700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd019b520 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7effc0003f70 tx=0x7effc0011b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:52.988 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.135+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc00039f0 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.135+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7effc0003b50 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.135+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effd019bc60 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.135+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7effc001a690 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.135+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd019c100 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.136+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7effc001a950 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.136+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effd010bb70 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.136+0000 7effcd7fa700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7effb8038e90 0x7effb803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.136+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7effc004d3f0 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.136+0000 7effcf7fe700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7effb8038e90 0x7effb803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.136+0000 7effcf7fe700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7effb8038e90 0x7effb803b350 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7effc4006fd0 tx=0x7effc4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.141+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7effc0018be0 con 0x7effd0107790 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.361+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7effd0104040 con 0x7effb8038e90 2024-02-09T22:41:52.989 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.366+0000 7effcd7fa700 1 -- 172.21.15.3:0/3888845524 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7effd0104040 con 0x7effb8038e90 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7effb8038e90 msgr2=0x7effb803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7effb8038e90 0x7effb803b350 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7effc4006fd0 tx=0x7effc4006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 msgr2=0x7effd019b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd019b520 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7effc0003f70 tx=0x7effc0011b40 comp rx=0 tx=0).stop 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 shutdown_connections 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7effb8038e90 0x7effb803b350 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 --2- 172.21.15.3:0/3888845524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effd0107790 0x7effd019b520 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 >> 172.21.15.3:0/3888845524 conn(0x7effd0102e50 msgr2=0x7effd0103930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 shutdown_connections 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:52.368+0000 7effd65b9700 1 -- 172.21.15.3:0/3888845524 wait complete. 2024-02-09T22:41:52.990 INFO:teuthology.orchestra.run.smithi003.stdout:Deploying unmanaged mgr service... 2024-02-09T22:41:53.583 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:53 smithi003 ceph-mon[32678]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:53.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:53 smithi003 ceph-mon[32678]: Saving service mon spec with placement count:5 2024-02-09T22:41:53.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:53 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:54.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:54 smithi003 ceph-mon[32678]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:41:54.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:54 smithi003 ceph-mon[32678]: Saving service mgr spec with placement count:2 2024-02-09T22:41:54.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:54 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:54.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:54 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:54.890 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-02-09T22:41:54.890 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc3d929700 1 Processor -- start 2024-02-09T22:41:54.890 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc3d929700 1 -- start start 2024-02-09T22:41:54.890 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc3d929700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38106a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:54.890 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc3d929700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc38107050 con 0x7efc38106660 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc36ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38106a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc36ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38106a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52464/0 (socket says 172.21.15.3:52464) 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.495+0000 7efc36ffd700 1 -- 172.21.15.3:0/2971279335 learned_addr learned my addr 172.21.15.3:0/2971279335 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.496+0000 7efc36ffd700 1 -- 172.21.15.3:0/2971279335 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc38107860 con 0x7efc38106660 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.496+0000 7efc36ffd700 1 --2- 172.21.15.3:0/2971279335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38106a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc30009e80 tx=0x7efc3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea137c6ba250389 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.496+0000 7efc35ffb700 1 -- 172.21.15.3:0/2971279335 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc3000e070 con 0x7efc38106660 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.496+0000 7efc35ffb700 1 -- 172.21.15.3:0/2971279335 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc3000b8c0 con 0x7efc38106660 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.496+0000 7efc35ffb700 1 -- 172.21.15.3:0/2971279335 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc30004550 con 0x7efc38106660 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 -- 172.21.15.3:0/2971279335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 msgr2=0x7efc38106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 --2- 172.21.15.3:0/2971279335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38106a80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7efc30009e80 tx=0x7efc3000b330 comp rx=0 tx=0).stop 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 -- 172.21.15.3:0/2971279335 shutdown_connections 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 --2- 172.21.15.3:0/2971279335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38106a80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 -- 172.21.15.3:0/2971279335 >> 172.21.15.3:0/2971279335 conn(0x7efc38101c60 msgr2=0x7efc38104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:54.891 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 -- 172.21.15.3:0/2971279335 shutdown_connections 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 -- 172.21.15.3:0/2971279335 wait complete. 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.497+0000 7efc3d929700 1 Processor -- start 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3d929700 1 -- start start 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3d929700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38104010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3d929700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc38104550 con 0x7efc38106660 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc36ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38104010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc36ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38104010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52474/0 (socket says 172.21.15.3:52474) 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc36ffd700 1 -- 172.21.15.3:0/1765838892 learned_addr learned my addr 172.21.15.3:0/1765838892 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc36ffd700 1 -- 172.21.15.3:0/1765838892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009b30 con 0x7efc38106660 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc36ffd700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38104010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc3000b7b0 tx=0x7efc30003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc3000e040 con 0x7efc38106660 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efc3000bd70 con 0x7efc38106660 2024-02-09T22:41:54.892 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc38102740 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efc3001c770 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.498+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc38102bc0 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.499+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7efc3001c920 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.500+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc38063350 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.500+0000 7efc3c927700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7efc24038ee0 0x7efc2403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.500+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efc3004d000 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.500+0000 7efc367fc700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7efc24038ee0 0x7efc2403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.500+0000 7efc367fc700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7efc24038ee0 0x7efc2403b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efc20006fd0 tx=0x7efc20006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.505+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc30080d20 con 0x7efc38106660 2024-02-09T22:41:54.893 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.723+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7efc3810aa60 con 0x7efc24038ee0 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.728+0000 7efc3c927700 1 -- 172.21.15.3:0/1765838892 <== mgr.14116 v2:172.21.15.3:6800/2045974484 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7efc3810aa60 con 0x7efc24038ee0 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7efc24038ee0 msgr2=0x7efc2403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7efc24038ee0 0x7efc2403b3a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7efc20006fd0 tx=0x7efc20006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 msgr2=0x7efc38104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38104010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7efc3000b7b0 tx=0x7efc30003d00 comp rx=0 tx=0).stop 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 shutdown_connections 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7efc24038ee0 0x7efc2403b3a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 --2- 172.21.15.3:0/1765838892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc38106660 0x7efc38104010 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 >> 172.21.15.3:0/1765838892 conn(0x7efc38101c60 msgr2=0x7efc38192260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 shutdown_connections 2024-02-09T22:41:54.894 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:53.730+0000 7efc3d929700 1 -- 172.21.15.3:0/1765838892 wait complete. 2024-02-09T22:41:56.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:55 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:56.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:55 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1317685060' entity='client.admin' 2024-02-09T22:41:56.767 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.337+0000 7fd053cc8700 1 Processor -- start 2024-02-09T22:41:56.767 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd053cc8700 1 -- start start 2024-02-09T22:41:56.767 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd053cc8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c106ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:56.767 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd053cc8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c1070a0 con 0x7fd04c1066b0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd051a64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c106ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd051a64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c106ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52482/0 (socket says 172.21.15.3:52482) 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd051a64700 1 -- 172.21.15.3:0/88944731 learned_addr learned my addr 172.21.15.3:0/88944731 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd051a64700 1 -- 172.21.15.3:0/88944731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04c1078b0 con 0x7fd04c1066b0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.338+0000 7fd051a64700 1 --2- 172.21.15.3:0/88944731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c106ad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd0400098a0 tx=0x7fd040009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=407d47fe400f122b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd050a62700 1 -- 172.21.15.3:0/88944731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd040004160 con 0x7fd04c1066b0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd050a62700 1 -- 172.21.15.3:0/88944731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0400042c0 con 0x7fd04c1066b0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd050a62700 1 -- 172.21.15.3:0/88944731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd040011a80 con 0x7fd04c1066b0 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 -- 172.21.15.3:0/88944731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 msgr2=0x7fd04c106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 --2- 172.21.15.3:0/88944731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c106ad0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd0400098a0 tx=0x7fd040009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 -- 172.21.15.3:0/88944731 shutdown_connections 2024-02-09T22:41:56.768 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 --2- 172.21.15.3:0/88944731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c106ad0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 -- 172.21.15.3:0/88944731 >> 172.21.15.3:0/88944731 conn(0x7fd04c101cf0 msgr2=0x7fd04c104110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 -- 172.21.15.3:0/88944731 shutdown_connections 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.339+0000 7fd053cc8700 1 -- 172.21.15.3:0/88944731 wait complete. 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd053cc8700 1 Processor -- start 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd053cc8700 1 -- start start 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd053cc8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c19b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd053cc8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd04c19bb10 con 0x7fd04c1066b0 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd051a64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c19b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd051a64700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c19b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52484/0 (socket says 172.21.15.3:52484) 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd051a64700 1 -- 172.21.15.3:0/1317685060 learned_addr learned my addr 172.21.15.3:0/1317685060 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:56.769 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd051a64700 1 -- 172.21.15.3:0/1317685060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd040009550 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.340+0000 7fd051a64700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c19b5d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd040003f20 tx=0x7fd040011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.341+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0400038b0 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.341+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd040003a10 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.341+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd04c19bd10 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.341+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd04001a5c0 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.341+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd04c19c1b0 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.341+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7fd04001a830 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.342+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd04c10aae0 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.342+0000 7fd03effd700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd038038e90 0x7fd03803b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.342+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd04004d280 con 0x7fd04c1066b0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.342+0000 7fd051263700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd038038e90 0x7fd03803b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.342+0000 7fd051263700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd038038e90 0x7fd03803b350 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd048006fd0 tx=0x7fd048006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:56.770 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.347+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd040018350 con 0x7fd04c1066b0 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.562+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd04c10acf0 con 0x7fd04c1066b0 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.565+0000 7fd03effd700 1 -- 172.21.15.3:0/1317685060 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd04004cbf0 con 0x7fd04c1066b0 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd038038e90 msgr2=0x7fd03803b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd038038e90 0x7fd03803b350 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fd048006fd0 tx=0x7fd048006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 msgr2=0x7fd04c19b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c19b5d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd040003f20 tx=0x7fd040011b00 comp rx=0 tx=0).stop 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 shutdown_connections 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd038038e90 0x7fd03803b350 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 --2- 172.21.15.3:0/1317685060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd04c1066b0 0x7fd04c19b5d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 >> 172.21.15.3:0/1317685060 conn(0x7fd04c101cf0 msgr2=0x7fd04c102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:56.771 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 shutdown_connections 2024-02-09T22:41:56.772 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:55.567+0000 7fd053cc8700 1 -- 172.21.15.3:0/1317685060 wait complete. 2024-02-09T22:41:57.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:57 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd87850b700 1 Processor -- start 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd87850b700 1 -- start start 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd87850b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd87010a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd87850b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8700757f0 con 0x7fd870108250 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd8762a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd87010a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd8762a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd87010a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41740/0 (socket says 172.21.15.3:41740) 2024-02-09T22:41:58.073 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.215+0000 7fd8762a7700 1 -- 172.21.15.3:0/1740073466 learned_addr learned my addr 172.21.15.3:0/1740073466 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.216+0000 7fd8762a7700 1 -- 172.21.15.3:0/1740073466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd87010ab80 con 0x7fd870108250 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.216+0000 7fd8762a7700 1 --2- 172.21.15.3:0/1740073466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd87010a640 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098a0 tx=0x7fd86c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96644a804b1fb074 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.216+0000 7fd8752a5700 1 -- 172.21.15.3:0/1740073466 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd86c004160 con 0x7fd870108250 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.216+0000 7fd8752a5700 1 -- 172.21.15.3:0/1740073466 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd86c0042c0 con 0x7fd870108250 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.216+0000 7fd8752a5700 1 -- 172.21.15.3:0/1740073466 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd86c011a80 con 0x7fd870108250 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 -- 172.21.15.3:0/1740073466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 msgr2=0x7fd87010a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 --2- 172.21.15.3:0/1740073466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd87010a640 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd86c0098a0 tx=0x7fd86c009bb0 comp rx=0 tx=0).stop 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 -- 172.21.15.3:0/1740073466 shutdown_connections 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 --2- 172.21.15.3:0/1740073466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd87010a640 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 -- 172.21.15.3:0/1740073466 >> 172.21.15.3:0/1740073466 conn(0x7fd870101ca0 msgr2=0x7fd8701040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 -- 172.21.15.3:0/1740073466 shutdown_connections 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 -- 172.21.15.3:0/1740073466 wait complete. 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 Processor -- start 2024-02-09T22:41:58.074 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.217+0000 7fd87850b700 1 -- start start 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd87850b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd870107ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd87850b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870108010 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8762a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd870107ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8762a7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd870107ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41752/0 (socket says 172.21.15.3:41752) 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8762a7700 1 -- 172.21.15.3:0/829629340 learned_addr learned my addr 172.21.15.3:0/829629340 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8762a7700 1 -- 172.21.15.3:0/829629340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c009550 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8762a7700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd870107ad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd86c003f20 tx=0x7fd86c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd86c0039f0 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd86c003b50 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd870106200 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd86c01a6d0 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.218+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8701066a0 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.219+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7fd86c01a990 con 0x7fd870108250 2024-02-09T22:41:58.075 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.220+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87010b9c0 con 0x7fd870108250 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.220+0000 7fd8637fe700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd85c038e90 0x7fd85c03b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.220+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd86c04d260 con 0x7fd870108250 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.220+0000 7fd875aa6700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd85c038e90 0x7fd85c03b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.220+0000 7fd875aa6700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd85c038e90 0x7fd85c03b350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd864006fd0 tx=0x7fd864006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.225+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd86c018dc0 con 0x7fd870108250 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.428+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd87010bbd0 con 0x7fd870108250 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.431+0000 7fd8637fe700 1 -- 172.21.15.3:0/829629340 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd86c018be0 con 0x7fd870108250 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd85c038e90 msgr2=0x7fd85c03b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd85c038e90 0x7fd85c03b350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd864006fd0 tx=0x7fd864006e40 comp rx=0 tx=0).stop 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 msgr2=0x7fd870107ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd870107ad0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd86c003f20 tx=0x7fd86c011b00 comp rx=0 tx=0).stop 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 shutdown_connections 2024-02-09T22:41:58.076 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7fd85c038e90 0x7fd85c03b350 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:58.077 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 --2- 172.21.15.3:0/829629340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd870108250 0x7fd870107ad0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:41:58.077 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 >> 172.21.15.3:0/829629340 conn(0x7fd870101ca0 msgr2=0x7fd870102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:41:58.077 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 shutdown_connections 2024-02-09T22:41:58.077 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:57.433+0000 7fd87850b700 1 -- 172.21.15.3:0/829629340 wait complete. 2024-02-09T22:41:58.077 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling the dashboard module... 2024-02-09T22:41:58.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:58 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/829629340' entity='client.admin' 2024-02-09T22:41:58.661 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:58 smithi003 ceph-mon[32678]: from='mgr.14116 172.21.15.3:0/660591805' entity='mgr.x' 2024-02-09T22:41:59.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:59 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3571867374' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-02-09T22:41:59.751 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:41:59 smithi003 ceph-mon[32678]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:41:59.751 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: ignoring --setuser ceph since I am not root 2024-02-09T22:41:59.751 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:41:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: ignoring --setgroup ceph since I am not root 2024-02-09T22:42:00.346 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:00.035+0000 7f8a07684200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:42:00.346 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:00.132+0000 7f8a07684200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:42:00.353 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39ebaf0700 1 Processor -- start 2024-02-09T22:42:00.353 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39ebaf0700 1 -- start start 2024-02-09T22:42:00.353 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39ebaf0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e4107bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:00.353 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39ebaf0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e4108180 con 0x7f39e4107790 2024-02-09T22:42:00.353 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39e988c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e4107bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39e988c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e4107bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41768/0 (socket says 172.21.15.3:41768) 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39e988c700 1 -- 172.21.15.3:0/3175392035 learned_addr learned my addr 172.21.15.3:0/3175392035 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39e988c700 1 -- 172.21.15.3:0/3175392035 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e4108940 con 0x7f39e4107790 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.484+0000 7f39e988c700 1 --2- 172.21.15.3:0/3175392035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e4107bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39e00098a0 tx=0x7f39e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2474ceebcfa6fb44 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39e888a700 1 -- 172.21.15.3:0/3175392035 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e0004160 con 0x7f39e4107790 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39e888a700 1 -- 172.21.15.3:0/3175392035 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39e00042c0 con 0x7f39e4107790 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39e888a700 1 -- 172.21.15.3:0/3175392035 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e0011a80 con 0x7f39e4107790 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3175392035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 msgr2=0x7f39e4107bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 --2- 172.21.15.3:0/3175392035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e4107bb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f39e00098a0 tx=0x7f39e0009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3175392035 shutdown_connections 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 --2- 172.21.15.3:0/3175392035 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e4107bb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3175392035 >> 172.21.15.3:0/3175392035 conn(0x7f39e4102df0 msgr2=0x7f39e4105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:00.354 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3175392035 shutdown_connections 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.485+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3175392035 wait complete. 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39ebaf0700 1 Processor -- start 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39ebaf0700 1 -- start start 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39ebaf0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e419b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39ebaf0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39e419ba40 con 0x7f39e4107790 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39e988c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e419b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39e988c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e419b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41778/0 (socket says 172.21.15.3:41778) 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39e988c700 1 -- 172.21.15.3:0/3571867374 learned_addr learned my addr 172.21.15.3:0/3571867374 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.486+0000 7f39e988c700 1 -- 172.21.15.3:0/3571867374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39e0009550 con 0x7f39e4107790 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.487+0000 7f39e988c700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e419b500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39e0011740 tx=0x7f39e0011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.487+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e00039e0 con 0x7f39e4107790 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.487+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39e0003b40 con 0x7f39e4107790 2024-02-09T22:42:00.355 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.487+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f39e0024710 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.487+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39e419bc40 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.487+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39e419c0e0 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.488+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 8) v1 ==== 45932+0+0 (secure 0 0 0) 0x7f39e001a430 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.488+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39e410bb70 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.488+0000 7f39daffd700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f39d0038e90 0x7f39d003b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.488+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f39e004d200 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.488+0000 7f39e908b700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f39d0038e90 0x7f39d003b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.488+0000 7f39e908b700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f39d0038e90 0x7f39d003b350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f39d4006fd0 tx=0x7f39d4006e40 comp rx=0 tx=0).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.493+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f39e0018530 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:58.757+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f39e410bd80 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.434+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f39e004cb70 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.436+0000 7f39daffd700 1 -- 172.21.15.3:0/3571867374 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 9) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f39e001abe0 con 0x7f39e4107790 2024-02-09T22:42:00.356 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f39d0038e90 msgr2=0x7f39d003b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f39d0038e90 0x7f39d003b350 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f39d4006fd0 tx=0x7f39d4006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 msgr2=0x7f39e419b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e419b500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39e0011740 tx=0x7f39e0011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 shutdown_connections 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f39d0038e90 0x7f39d003b350 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 --2- 172.21.15.3:0/3571867374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f39e4107790 0x7f39e419b500 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 >> 172.21.15.3:0/3571867374 conn(0x7f39e4102df0 msgr2=0x7f39e4103890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 shutdown_connections 2024-02-09T22:42:00.357 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:41:59.438+0000 7f39ebaf0700 1 -- 172.21.15.3:0/3571867374 wait complete. 2024-02-09T22:42:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:00 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3571867374' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-02-09T22:42:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:00 smithi003 ceph-mon[32678]: mgrmap e9: x(active, since 20s) 2024-02-09T22:42:00.983 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:00.814+0000 7f8a07684200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:42:00.983 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:00.909+0000 7f8a07684200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:42:01.243 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.053+0000 7f8a07684200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:42:01.243 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.155+0000 7f8a07684200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:42:01.243 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.240+0000 7f8a07684200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:42:01.511 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1415941158' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-02-09T22:42:01.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-02-09T22:42:01.736 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "available": true, 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.774+0000 7f2702ea4700 1 Processor -- start 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.774+0000 7f2702ea4700 1 -- start start 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.774+0000 7f2702ea4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.774+0000 7f2702ea4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc108170 con 0x7f26fc107780 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f2700c40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f2700c40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41800/0 (socket says 172.21.15.3:41800) 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f2700c40700 1 -- 172.21.15.3:0/4066874594 learned_addr learned my addr 172.21.15.3:0/4066874594 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f2700c40700 1 -- 172.21.15.3:0/4066874594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26fc108930 con 0x7f26fc107780 2024-02-09T22:42:01.737 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f2700c40700 1 --2- 172.21.15.3:0/4066874594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc107ba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009e80 tx=0x7f26f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b84309e391731075 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f26fb7fe700 1 -- 172.21.15.3:0/4066874594 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26f400e070 con 0x7f26fc107780 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f26fb7fe700 1 -- 172.21.15.3:0/4066874594 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26f400b8c0 con 0x7f26fc107780 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.775+0000 7f26fb7fe700 1 -- 172.21.15.3:0/4066874594 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26f40044a0 con 0x7f26fc107780 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 -- 172.21.15.3:0/4066874594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 msgr2=0x7f26fc107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 --2- 172.21.15.3:0/4066874594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc107ba0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f26f4009e80 tx=0x7f26f400b330 comp rx=0 tx=0).stop 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 -- 172.21.15.3:0/4066874594 shutdown_connections 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 --2- 172.21.15.3:0/4066874594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc107ba0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 -- 172.21.15.3:0/4066874594 >> 172.21.15.3:0/4066874594 conn(0x7f26fc102d60 msgr2=0x7f26fc105180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 -- 172.21.15.3:0/4066874594 shutdown_connections 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.776+0000 7f2702ea4700 1 -- 172.21.15.3:0/4066874594 wait complete. 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2702ea4700 1 Processor -- start 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2702ea4700 1 -- start start 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2702ea4700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc07cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:01.738 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2702ea4700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26fc07d0c0 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2700c40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc07cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2700c40700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc07cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41812/0 (socket says 172.21.15.3:41812) 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2700c40700 1 -- 172.21.15.3:0/1415941158 learned_addr learned my addr 172.21.15.3:0/1415941158 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2700c40700 1 -- 172.21.15.3:0/1415941158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26f4009b30 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.777+0000 7f2700c40700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc07cb80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26f4004110 tx=0x7f26f40046a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.778+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26f400e040 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.778+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26fc07b2b0 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.778+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26f4003d60 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.778+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26f401c700 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.778+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26fc07b7b0 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.779+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 9) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f26f401c940 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.779+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26fc068e00 con 0x7f26fc107780 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.779+0000 7f26f9ffb700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 0x7f26e803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:01.739 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.779+0000 7f26fbfff700 1 -- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 msgr2=0x7f26e803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.779+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f26f404d010 con 0x7f26fc107780 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.779+0000 7f26fbfff700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 0x7f26e803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.784+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f26f4011980 con 0x7f26fc107780 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.980+0000 7f26fbfff700 1 -- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 msgr2=0x7f26e803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:00.980+0000 7f26fbfff700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 0x7f26e803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.044+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f26fc07bcf0 con 0x7f26fc107780 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.045+0000 7f26f9ffb700 1 -- 172.21.15.3:0/1415941158 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f26f4017070 con 0x7f26fc107780 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 msgr2=0x7f26e803b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 0x7f26e803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 msgr2=0x7f26fc07cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc07cb80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f26f4004110 tx=0x7f26f40046a0 comp rx=0 tx=0).stop 2024-02-09T22:42:01.740 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 shutdown_connections 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f26e8038ee0 0x7f26e803b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 --2- 172.21.15.3:0/1415941158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26fc107780 0x7f26fc07cb80 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 >> 172.21.15.3:0/1415941158 conn(0x7f26fc102d60 msgr2=0x7f26fc103850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 shutdown_connections 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:01.047+0000 7f2702ea4700 1 -- 172.21.15.3:0/1415941158 wait complete. 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for the mgr to restart... 2024-02-09T22:42:01.741 INFO:teuthology.orchestra.run.smithi003.stdout:Waiting for mgr epoch 9... 2024-02-09T22:42:01.812 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.508+0000 7f8a07684200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:42:01.812 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.620+0000 7f8a07684200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:42:01.812 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.708+0000 7f8a07684200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:42:01.812 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.808+0000 7f8a07684200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:42:02.115 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:01.989+0000 7f8a07684200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:42:03.114 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:02 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:02.977+0000 7f8a07684200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:42:03.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:03.111+0000 7f8a07684200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:42:03.741 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:03.447+0000 7f8a07684200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:42:03.742 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:03.539+0000 7f8a07684200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:42:03.742 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:03.738+0000 7f8a07684200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:42:04.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:03.926+0000 7f8a07684200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:42:04.481 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:04.216+0000 7f8a07684200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:42:04.481 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:04.478+0000 7f8a07684200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:42:04.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:04.718+0000 7f8a07684200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:42:05.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:05 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:05.537+0000 7f8a07684200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:42:06.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:06.119+0000 7f8a07684200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:42:06.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:06.219+0000 7f8a07684200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:42:06.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:42:06.439+0000 7f8a07684200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:42:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: Active manager daemon x restarted 2024-02-09T22:42:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: Activating manager daemon x 2024-02-09T22:42:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: osdmap e3: 0 total, 0 up, 0 in 2024-02-09T22:42:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: mgrmap e10: x(active, starting, since 0.00322341s) 2024-02-09T22:42:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: Manager daemon x is now available 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:06.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:07.820 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-09T22:42:07.820 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-09T22:42:07.820 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: [09/Feb/2024:22:42:07] ENGINE Bus STARTING 2024-02-09T22:42:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: [09/Feb/2024:22:42:07] ENGINE Serving on http://172.21.15.3:8765 2024-02-09T22:42:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-02-09T22:42:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: mgrmap e11: x(active, since 1.00785s) 2024-02-09T22:42:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-02-09T22:42:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: [09/Feb/2024:22:42:07] ENGINE Serving on https://172.21.15.3:7150 2024-02-09T22:42:07.821 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:07 smithi003 ceph-mon[32678]: [09/Feb/2024:22:42:07] ENGINE Bus STARTED 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout { 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout "initialized": true 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout } 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.131+0000 7f07d4c76700 1 Processor -- start 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.131+0000 7f07d4c76700 1 -- start start 2024-02-09T22:42:08.064 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.131+0000 7f07d4c76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d010a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.131+0000 7f07d4c76700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d007ade0 con 0x7f07d01082d0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d010a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d010a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41814/0 (socket says 172.21.15.3:41814) 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07ce59c700 1 -- 172.21.15.3:0/3523348385 learned_addr learned my addr 172.21.15.3:0/3523348385 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07ce59c700 1 -- 172.21.15.3:0/3523348385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d007af20 con 0x7f07d01082d0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07ce59c700 1 --2- 172.21.15.3:0/3523348385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d010a6c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07c40098a0 tx=0x7f07c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80b5f394d03fb5fe server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07cd59a700 1 -- 172.21.15.3:0/3523348385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c4004160 con 0x7f07d01082d0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07cd59a700 1 -- 172.21.15.3:0/3523348385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f07c40042c0 con 0x7f07d01082d0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.132+0000 7f07cd59a700 1 -- 172.21.15.3:0/3523348385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c4011a80 con 0x7f07d01082d0 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 -- 172.21.15.3:0/3523348385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 msgr2=0x7f07d010a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:08.065 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 --2- 172.21.15.3:0/3523348385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d010a6c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f07c40098a0 tx=0x7f07c4009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 -- 172.21.15.3:0/3523348385 shutdown_connections 2024-02-09T22:42:08.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 --2- 172.21.15.3:0/3523348385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d010a6c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 -- 172.21.15.3:0/3523348385 >> 172.21.15.3:0/3523348385 conn(0x7f07d0101c60 msgr2=0x7f07d0104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:08.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 -- 172.21.15.3:0/3523348385 shutdown_connections 2024-02-09T22:42:08.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.133+0000 7f07d4c76700 1 -- 172.21.15.3:0/3523348385 wait complete. 2024-02-09T22:42:08.066 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07d4c76700 1 Processor -- start 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07d4c76700 1 -- start start 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07d4c76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d019b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07d4c76700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d019ba40 con 0x7f07d01082d0 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d019b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07ce59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d019b500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41830/0 (socket says 172.21.15.3:41830) 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07ce59c700 1 -- 172.21.15.3:0/2977398060 learned_addr learned my addr 172.21.15.3:0/2977398060 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.134+0000 7f07ce59c700 1 -- 172.21.15.3:0/2977398060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4009550 con 0x7f07d01082d0 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.135+0000 7f07ce59c700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d019b500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f07c4003f20 tx=0x7f07c4011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:08.067 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.135+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c40039f0 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.135+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d019bc40 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.135+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f07c4003b50 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.135+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f07c401a6d0 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.135+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d019c0e0 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.136+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 9) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f07c401a990 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.136+0000 7f07bf7fe700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.136+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 --> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07d010b9d0 con 0x7f07b8038ee0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.136+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f07c404e1d0 con 0x7f07d01082d0 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.136+0000 7f07cdd9b700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 msgr2=0x7f07b803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:08.068 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.136+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.337+0000 7f07cdd9b700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 msgr2=0x7f07b803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.337+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.738+0000 7f07cdd9b700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 msgr2=0x7f07b803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:02.738+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:03.539+0000 7f07cdd9b700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 msgr2=0x7f07b803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:03.539+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:05.140+0000 7f07cdd9b700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 msgr2=0x7f07b803b3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/2045974484 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:05.140+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:06.454+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mgrmap(e 10) v1 ==== 45712+0+0 (secure 0 0 0) 0x7f07c402ad60 con 0x7f07d01082d0 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:06.454+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 msgr2=0x7f07b803b3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:06.454+0000 7f07bf7fe700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.457+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 11) v1 ==== 45839+0+0 (secure 0 0 0) 0x7f07c4021d40 con 0x7f07d01082d0 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.457+0000 7f07bf7fe700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f07b803c7b0 0x7f07b803eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:08.069 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.457+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f07d010b9d0 con 0x7f07b803c7b0 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.458+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f07b803c7b0 0x7f07b803eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.459+0000 7f07cdd9b700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f07b803c7b0 0x7f07b803eba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f07c0003a10 tx=0x7f07c00092b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.459+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f07d010b9d0 con 0x7f07b803c7b0 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.466+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f07d010bb40 con 0x7f07b803c7b0 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07bf7fe700 1 -- 172.21.15.3:0/2977398060 <== mgr.14148 v2:172.21.15.3:6800/284621866 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f07d010bb40 con 0x7f07b803c7b0 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f07b803c7b0 msgr2=0x7f07b803eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f07b803c7b0 0x7f07b803eba0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f07c0003a10 tx=0x7f07c00092b0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 msgr2=0x7f07d019b500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d019b500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f07c4003f20 tx=0x7f07c4011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 shutdown_connections 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f07b803c7b0 0x7f07b803eba0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f07d01082d0 0x7f07d019b500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.070 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 --2- 172.21.15.3:0/2977398060 >> [v2:172.21.15.3:6800/2045974484,v1:172.21.15.3:6801/2045974484] conn(0x7f07b8038ee0 0x7f07b803b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:08.071 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 >> 172.21.15.3:0/2977398060 conn(0x7f07d0101c60 msgr2=0x7f07d0102750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:08.071 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 shutdown_connections 2024-02-09T22:42:08.071 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:07.467+0000 7f07d4c76700 1 -- 172.21.15.3:0/2977398060 wait complete. 2024-02-09T22:42:08.071 INFO:teuthology.orchestra.run.smithi003.stdout:mgr epoch 9 is available 2024-02-09T22:42:08.071 INFO:teuthology.orchestra.run.smithi003.stdout:Generating a dashboard self-signed certificate... 2024-02-09T22:42:09.395 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-02-09T22:42:09.395 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.473+0000 7f5facf25700 1 Processor -- start 2024-02-09T22:42:09.395 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.473+0000 7f5facf25700 1 -- start start 2024-02-09T22:42:09.395 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.473+0000 7f5facf25700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa8105e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:09.395 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.473+0000 7f5facf25700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa8106450 con 0x7f5fa8105a60 2024-02-09T22:42:09.395 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa8105e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa8105e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46882/0 (socket says 172.21.15.3:46882) 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa659c700 1 -- 172.21.15.3:0/2229045358 learned_addr learned my addr 172.21.15.3:0/2229045358 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa659c700 1 -- 172.21.15.3:0/2229045358 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8106590 con 0x7f5fa8105a60 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa659c700 1 --2- 172.21.15.3:0/2229045358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa8105e80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f980098a0 tx=0x7f5f98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0ac5ca5e548cec server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa559a700 1 -- 172.21.15.3:0/2229045358 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f98004160 con 0x7f5fa8105a60 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa559a700 1 -- 172.21.15.3:0/2229045358 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f980042c0 con 0x7f5fa8105a60 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.474+0000 7f5fa559a700 1 -- 172.21.15.3:0/2229045358 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f98011a80 con 0x7f5fa8105a60 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 -- 172.21.15.3:0/2229045358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 msgr2=0x7f5fa8105e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 --2- 172.21.15.3:0/2229045358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa8105e80 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f980098a0 tx=0x7f5f98009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 -- 172.21.15.3:0/2229045358 shutdown_connections 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 --2- 172.21.15.3:0/2229045358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa8105e80 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 -- 172.21.15.3:0/2229045358 >> 172.21.15.3:0/2229045358 conn(0x7f5fa8101420 msgr2=0x7f5fa8103840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:09.396 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 -- 172.21.15.3:0/2229045358 shutdown_connections 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 -- 172.21.15.3:0/2229045358 wait complete. 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.475+0000 7f5facf25700 1 Processor -- start 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5facf25700 1 -- start start 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5facf25700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa819b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5facf25700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa819ba60 con 0x7f5fa8105a60 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5fa659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa819b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5fa659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa819b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46896/0 (socket says 172.21.15.3:46896) 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5fa659c700 1 -- 172.21.15.3:0/3023061139 learned_addr learned my addr 172.21.15.3:0/3023061139 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5fa659c700 1 -- 172.21.15.3:0/3023061139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f98009550 con 0x7f5fa8105a60 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5fa659c700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa819b520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5f98000c00 tx=0x7f5f98011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f98003aa0 con 0x7f5fa8105a60 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f98003c00 con 0x7f5fa8105a60 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa819bc60 con 0x7f5fa8105a60 2024-02-09T22:42:09.397 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5f98024930 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.476+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa819c100 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.477+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 11) v1 ==== 45839+0+0 (secure 0 0 0) 0x7f5f9802d3f0 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.477+0000 7f5f9f7fe700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5f94038e80 0x7f5f9403b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.477+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5f9804d030 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.478+0000 7f5fa5d9b700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5f94038e80 0x7f5f9403b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.478+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fa8068e00 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.478+0000 7f5fa5d9b700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5f94038e80 0x7f5f9403b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5f90006fd0 tx=0x7f5f90006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.483+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5f98010740 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.508+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f5f9801aca0 con 0x7f5fa8105a60 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.694+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fa810bce0 con 0x7f5f94038e80 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.748+0000 7f5f9f7fe700 1 -- 172.21.15.3:0/3023061139 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f5fa810bce0 con 0x7f5f94038e80 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5f94038e80 msgr2=0x7f5f9403b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5f94038e80 0x7f5f9403b340 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5f90006fd0 tx=0x7f5f90006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:09.398 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 msgr2=0x7f5fa819b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa819b520 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5f98000c00 tx=0x7f5f98011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 shutdown_connections 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5f94038e80 0x7f5f9403b340 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 --2- 172.21.15.3:0/3023061139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa8105a60 0x7f5fa819b520 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 >> 172.21.15.3:0/3023061139 conn(0x7f5fa8101420 msgr2=0x7f5fa8101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 shutdown_connections 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:08.750+0000 7f5facf25700 1 -- 172.21.15.3:0/3023061139 wait complete. 2024-02-09T22:42:09.399 INFO:teuthology.orchestra.run.smithi003.stdout:Creating initial admin user... 2024-02-09T22:42:09.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:09 smithi003 ceph-mon[32678]: mgrmap e12: x(active, since 2s) 2024-02-09T22:42:09.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:09 smithi003 ceph-mon[32678]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:42:09.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:09.706 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:10.884 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$d/EqdBBpyf4ggbNzJngO/Olm1ZONFmChtummmvHnAj8b715Yiyqcu", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1707518530, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e90bc9700 1 Processor -- start 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e90bc9700 1 -- start start 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e90bc9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c107ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e90bc9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c108170 con 0x7f7e8c107780 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c107ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c107ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46904/0 (socket says 172.21.15.3:46904) 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.815+0000 7f7e8a59c700 1 -- 172.21.15.3:0/513853072 learned_addr learned my addr 172.21.15.3:0/513853072 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.816+0000 7f7e8a59c700 1 -- 172.21.15.3:0/513853072 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e8c108980 con 0x7f7e8c107780 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.816+0000 7f7e8a59c700 1 --2- 172.21.15.3:0/513853072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c107ba0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0098a0 tx=0x7f7e7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb733b9a46d269b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.816+0000 7f7e8959a700 1 -- 172.21.15.3:0/513853072 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e7c004160 con 0x7f7e8c107780 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.816+0000 7f7e8959a700 1 -- 172.21.15.3:0/513853072 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e7c0042c0 con 0x7f7e8c107780 2024-02-09T22:42:10.885 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.816+0000 7f7e8959a700 1 -- 172.21.15.3:0/513853072 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e7c011a80 con 0x7f7e8c107780 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 -- 172.21.15.3:0/513853072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 msgr2=0x7f7e8c107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 --2- 172.21.15.3:0/513853072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c107ba0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0098a0 tx=0x7f7e7c009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 -- 172.21.15.3:0/513853072 shutdown_connections 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 --2- 172.21.15.3:0/513853072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c107ba0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 -- 172.21.15.3:0/513853072 >> 172.21.15.3:0/513853072 conn(0x7f7e8c102dc0 msgr2=0x7f7e8c1051e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 -- 172.21.15.3:0/513853072 shutdown_connections 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 -- 172.21.15.3:0/513853072 wait complete. 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.817+0000 7f7e90bc9700 1 Processor -- start 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e90bc9700 1 -- start start 2024-02-09T22:42:10.886 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e90bc9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c19b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e90bc9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e8c19ba80 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c19b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e8a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c19b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46918/0 (socket says 172.21.15.3:46918) 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e8a59c700 1 -- 172.21.15.3:0/3688966886 learned_addr learned my addr 172.21.15.3:0/3688966886 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e8a59c700 1 -- 172.21.15.3:0/3688966886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7c009550 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e8a59c700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c19b540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c003f20 tx=0x7f7e7c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e7c0038b0 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e7c003a10 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7e7c01a5c0 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e8c19bc80 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.818+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e8c19c120 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.819+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f7e7c01a830 con 0x7f7e8c107780 2024-02-09T22:42:10.887 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.819+0000 7f7e7b7fe700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7e74038ee0 0x7f7e7403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.819+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e8c10bbb0 con 0x7f7e8c107780 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.819+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7e7c04d440 con 0x7f7e8c107780 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.820+0000 7f7e89d9b700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7e74038ee0 0x7f7e7403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.820+0000 7f7e89d9b700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7e74038ee0 0x7f7e7403b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7e80006fd0 tx=0x7f7e80006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:09.825+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7e7c01ab30 con 0x7f7e8c107780 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.046+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7e8c063170 con 0x7f7e74038ee0 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.293+0000 7f7e7b7fe700 1 -- 172.21.15.3:0/3688966886 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7e8c063170 con 0x7f7e74038ee0 2024-02-09T22:42:10.888 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7e74038ee0 msgr2=0x7f7e7403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7e74038ee0 0x7f7e7403b3a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7e80006fd0 tx=0x7f7e80006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 msgr2=0x7f7e8c19b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c19b540 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c003f20 tx=0x7f7e7c011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 shutdown_connections 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7e74038ee0 0x7f7e7403b3a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 --2- 172.21.15.3:0/3688966886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7e8c107780 0x7f7e8c19b540 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 >> 172.21.15.3:0/3688966886 conn(0x7f7e8c102dc0 msgr2=0x7f7e8c103880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 shutdown_connections 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:10.295+0000 7f7e90bc9700 1 -- 172.21.15.3:0/3688966886 wait complete. 2024-02-09T22:42:10.889 INFO:teuthology.orchestra.run.smithi003.stdout:Fetching dashboard port number... 2024-02-09T22:42:11.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:11.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:10 smithi003 ceph-mon[32678]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:42:11.130 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:12.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:11 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1178059369' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-02-09T22:42:12.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:11 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:12.257 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stdout 8443 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.353+0000 7f3b65c4a700 1 Processor -- start 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.353+0000 7f3b65c4a700 1 -- start start 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b65c4a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b60106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b65c4a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b60107090 con 0x7f3b601066a0 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b5f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b60106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b5f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b60106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46926/0 (socket says 172.21.15.3:46926) 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b5f7fe700 1 -- 172.21.15.3:0/893096129 learned_addr learned my addr 172.21.15.3:0/893096129 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b5f7fe700 1 -- 172.21.15.3:0/893096129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b601078a0 con 0x7f3b601066a0 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.354+0000 7f3b5f7fe700 1 --2- 172.21.15.3:0/893096129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b60106ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b500098a0 tx=0x7f3b50009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fbef8b126ab8b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b5e7fc700 1 -- 172.21.15.3:0/893096129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b50004160 con 0x7f3b601066a0 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b5e7fc700 1 -- 172.21.15.3:0/893096129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b500042c0 con 0x7f3b601066a0 2024-02-09T22:42:12.258 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b5e7fc700 1 -- 172.21.15.3:0/893096129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b50011a80 con 0x7f3b601066a0 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 -- 172.21.15.3:0/893096129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 msgr2=0x7f3b60106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 --2- 172.21.15.3:0/893096129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b60106ac0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3b500098a0 tx=0x7f3b50009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 -- 172.21.15.3:0/893096129 shutdown_connections 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 --2- 172.21.15.3:0/893096129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b60106ac0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 -- 172.21.15.3:0/893096129 >> 172.21.15.3:0/893096129 conn(0x7f3b60101ca0 msgr2=0x7f3b601040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 -- 172.21.15.3:0/893096129 shutdown_connections 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.355+0000 7f3b65c4a700 1 -- 172.21.15.3:0/893096129 wait complete. 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.356+0000 7f3b65c4a700 1 Processor -- start 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.356+0000 7f3b65c4a700 1 -- start start 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.356+0000 7f3b65c4a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b6019b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.356+0000 7f3b65c4a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6019bb30 con 0x7f3b601066a0 2024-02-09T22:42:12.259 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.356+0000 7f3b5f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b6019b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b6019b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46942/0 (socket says 172.21.15.3:46942) 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5f7fe700 1 -- 172.21.15.3:0/1178059369 learned_addr learned my addr 172.21.15.3:0/1178059369 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5f7fe700 1 -- 172.21.15.3:0/1178059369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b50009550 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5f7fe700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b6019b5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3b50000c00 tx=0x7f3b50011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b500039e0 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b6019bd30 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b50003b40 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3b50024820 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.357+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b6019c1d0 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.358+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f3b50024980 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.358+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b60195740 con 0x7f3b601066a0 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.358+0000 7f3b5cff9700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3b48038f30 0x7f3b4803b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:12.260 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.358+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3b5004d140 con 0x7f3b601066a0 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.359+0000 7f3b5effd700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3b48038f30 0x7f3b4803b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.359+0000 7f3b5effd700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3b48038f30 0x7f3b4803b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b54006fd0 tx=0x7f3b54006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.368+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3b5002da50 con 0x7f3b601066a0 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.577+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f3b6010aad0 con 0x7f3b601066a0 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.578+0000 7f3b5cff9700 1 -- 172.21.15.3:0/1178059369 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f3b5004c9d0 con 0x7f3b601066a0 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3b48038f30 msgr2=0x7f3b4803b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3b48038f30 0x7f3b4803b3f0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b54006fd0 tx=0x7f3b54006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 msgr2=0x7f3b6019b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b6019b5f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3b50000c00 tx=0x7f3b50011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 shutdown_connections 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3b48038f30 0x7f3b4803b3f0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 --2- 172.21.15.3:0/1178059369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b601066a0 0x7f3b6019b5f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 >> 172.21.15.3:0/1178059369 conn(0x7f3b60101ca0 msgr2=0x7f3b60102b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 shutdown_connections 2024-02-09T22:42:12.261 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:11.580+0000 7f3b65c4a700 1 -- 172.21.15.3:0/1178059369 wait complete. 2024-02-09T22:42:12.263 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-02-09T22:42:12.264 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout disabled 2024-02-09T22:42:12.272 INFO:teuthology.orchestra.run.smithi003.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-02-09T22:42:12.272 INFO:teuthology.orchestra.run.smithi003.stdout:systemctl: stdout inactive 2024-02-09T22:42:12.272 INFO:teuthology.orchestra.run.smithi003.stdout:firewalld.service is not enabled 2024-02-09T22:42:12.272 INFO:teuthology.orchestra.run.smithi003.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-02-09T22:42:12.273 INFO:teuthology.orchestra.run.smithi003.stdout:Ceph Dashboard is now available at: 2024-02-09T22:42:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: URL: https://smithi003.front.sepia.ceph.com:8443/ 2024-02-09T22:42:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: User: admin 2024-02-09T22:42:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: Password: ocuf7x1xe6 2024-02-09T22:42:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:12.274 INFO:teuthology.orchestra.run.smithi003.stdout:Saving cluster configuration to /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config directory 2024-02-09T22:42:12.274 INFO:teuthology.orchestra.run.smithi003.stdout:Enabling autotune for osd_memory_target 2024-02-09T22:42:14.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:13 smithi003 ceph-mon[32678]: mgrmap e13: x(active, since 6s) 2024-02-09T22:42:14.481 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c684b6700 1 Processor -- start 2024-02-09T22:42:14.481 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c684b6700 1 -- start start 2024-02-09T22:42:14.481 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c684b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c60106ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:14.481 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c684b6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c60107090 con 0x7f4c601066a0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c66252700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c60106ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c66252700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c60106ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46950/0 (socket says 172.21.15.3:46950) 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.738+0000 7f4c66252700 1 -- 172.21.15.3:0/1486425182 learned_addr learned my addr 172.21.15.3:0/1486425182 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.739+0000 7f4c66252700 1 -- 172.21.15.3:0/1486425182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c601078a0 con 0x7f4c601066a0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.739+0000 7f4c66252700 1 --2- 172.21.15.3:0/1486425182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c60106ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098a0 tx=0x7f4c5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d9710bf3b3ef0fd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.739+0000 7f4c65250700 1 -- 172.21.15.3:0/1486425182 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c5c004160 con 0x7f4c601066a0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.739+0000 7f4c65250700 1 -- 172.21.15.3:0/1486425182 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c0042c0 con 0x7f4c601066a0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.739+0000 7f4c65250700 1 -- 172.21.15.3:0/1486425182 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c5c011a80 con 0x7f4c601066a0 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 -- 172.21.15.3:0/1486425182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 msgr2=0x7f4c60106ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 --2- 172.21.15.3:0/1486425182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c60106ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098a0 tx=0x7f4c5c009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 -- 172.21.15.3:0/1486425182 shutdown_connections 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 --2- 172.21.15.3:0/1486425182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c60106ac0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 -- 172.21.15.3:0/1486425182 >> 172.21.15.3:0/1486425182 conn(0x7f4c60101cc0 msgr2=0x7f4c601040e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:14.482 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 -- 172.21.15.3:0/1486425182 shutdown_connections 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.740+0000 7f4c684b6700 1 -- 172.21.15.3:0/1486425182 wait complete. 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c684b6700 1 Processor -- start 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c684b6700 1 -- start start 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c684b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c6019a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c684b6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c6019a9e0 con 0x7f4c601066a0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c66252700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c6019a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c66252700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c6019a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46966/0 (socket says 172.21.15.3:46966) 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c66252700 1 -- 172.21.15.3:0/2116230022 learned_addr learned my addr 172.21.15.3:0/2116230022 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.741+0000 7f4c66252700 1 -- 172.21.15.3:0/2116230022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c009550 con 0x7f4c601066a0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.742+0000 7f4c66252700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c6019a4a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c000c00 tx=0x7f4c5c011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.742+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c5c0039e0 con 0x7f4c601066a0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.742+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4c5c003b40 con 0x7f4c601066a0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.742+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c6019abe0 con 0x7f4c601066a0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.742+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4c5c024820 con 0x7f4c601066a0 2024-02-09T22:42:14.483 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.742+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6019b080 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.743+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 12) v1 ==== 45945+0+0 (secure 0 0 0) 0x7f4c5c0193c0 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.743+0000 7f4c577fe700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4c4c038fd0 0x7f4c4c03b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.743+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f4c5c04e3e0 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.744+0000 7f4c65a51700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4c4c038fd0 0x7f4c4c03b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.744+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c6010aad0 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.744+0000 7f4c65a51700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4c4c038fd0 0x7f4c4c03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c50006fd0 tx=0x7f4c50006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.749+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4c5c010970 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.932+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f4c5c02c4e0 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.967+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f4c6019dbe0 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.967+0000 7f4c577fe700 1 -- 172.21.15.3:0/2116230022 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f4c5c02b350 con 0x7f4c601066a0 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4c4c038fd0 msgr2=0x7f4c4c03b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4c4c038fd0 0x7f4c4c03b490 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4c50006fd0 tx=0x7f4c50006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 msgr2=0x7f4c6019a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:14.484 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c6019a4a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c000c00 tx=0x7f4c5c011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:14.485 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 shutdown_connections 2024-02-09T22:42:14.485 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4c4c038fd0 0x7f4c4c03b490 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:14.485 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 --2- 172.21.15.3:0/2116230022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4c601066a0 0x7f4c6019a4a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:14.485 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 >> 172.21.15.3:0/2116230022 conn(0x7f4c60101cc0 msgr2=0x7f4c60102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:14.485 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 shutdown_connections 2024-02-09T22:42:14.485 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:12.969+0000 7f4c684b6700 1 -- 172.21.15.3:0/2116230022 wait complete. 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f11bd6700 1 Processor -- start 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f11bd6700 1 -- start start 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f11bd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c10a600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f11bd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c07ae80 con 0x7f5f0c108210 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c10a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c10a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46980/0 (socket says 172.21.15.3:46980) 2024-02-09T22:42:15.877 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f0b7fe700 1 -- 172.21.15.3:0/3818009491 learned_addr learned my addr 172.21.15.3:0/3818009491 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f0b7fe700 1 -- 172.21.15.3:0/3818009491 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c10ab40 con 0x7f5f0c108210 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.933+0000 7f5f0b7fe700 1 --2- 172.21.15.3:0/3818009491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c10a600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0098a0 tx=0x7f5efc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51a14d5f236f2f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f0a7fc700 1 -- 172.21.15.3:0/3818009491 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5efc004160 con 0x7f5f0c108210 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f0a7fc700 1 -- 172.21.15.3:0/3818009491 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5efc0042c0 con 0x7f5f0c108210 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f0a7fc700 1 -- 172.21.15.3:0/3818009491 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5efc011a80 con 0x7f5f0c108210 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 -- 172.21.15.3:0/3818009491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 msgr2=0x7f5f0c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 --2- 172.21.15.3:0/3818009491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c10a600 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5efc0098a0 tx=0x7f5efc009bb0 comp rx=0 tx=0).stop 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 -- 172.21.15.3:0/3818009491 shutdown_connections 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 --2- 172.21.15.3:0/3818009491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c10a600 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 -- 172.21.15.3:0/3818009491 >> 172.21.15.3:0/3818009491 conn(0x7f5f0c101c60 msgr2=0x7f5f0c104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 -- 172.21.15.3:0/3818009491 shutdown_connections 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.934+0000 7f5f11bd6700 1 -- 172.21.15.3:0/3818009491 wait complete. 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.935+0000 7f5f11bd6700 1 Processor -- start 2024-02-09T22:42:15.878 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.935+0000 7f5f11bd6700 1 -- start start 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.935+0000 7f5f11bd6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c107a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.935+0000 7f5f11bd6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0c107fd0 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c107a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f0b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c107a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46988/0 (socket says 172.21.15.3:46988) 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f0b7fe700 1 -- 172.21.15.3:0/1124177514 learned_addr learned my addr 172.21.15.3:0/1124177514 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f0b7fe700 1 -- 172.21.15.3:0/1124177514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc009550 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f0b7fe700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c107a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5efc003f20 tx=0x7f5efc011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5efc0039f0 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5efc003b50 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0c1061c0 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5efc01a6d0 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.936+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f0c106660 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.937+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f5efc01a940 con 0x7f5f0c108210 2024-02-09T22:42:15.879 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.937+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f0c10b980 con 0x7f5f0c108210 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.937+0000 7f5f08ff9700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ef4038fe0 0x7f5ef403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.937+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f5efc04d540 con 0x7f5f0c108210 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.937+0000 7f5f0affd700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ef4038fe0 0x7f5ef403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.938+0000 7f5f0affd700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ef4038fe0 0x7f5ef403b4a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f00006fd0 tx=0x7f5f00006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:14.942+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5efc018710 con 0x7f5f0c108210 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.256+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f5f0c10bb90 con 0x7f5f0c108210 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.259+0000 7f5f08ff9700 1 -- 172.21.15.3:0/1124177514 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7f5efc018350 con 0x7f5f0c108210 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ef4038fe0 msgr2=0x7f5ef403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ef4038fe0 0x7f5ef403b4a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5f00006fd0 tx=0x7f5f00006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 msgr2=0x7f5f0c107a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c107a90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5efc003f20 tx=0x7f5efc011b00 comp rx=0 tx=0).stop 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 shutdown_connections 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ef4038fe0 0x7f5ef403b4a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:15.880 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 --2- 172.21.15.3:0/1124177514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5f0c108210 0x7f5f0c107a90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 >> 172.21.15.3:0/1124177514 conn(0x7f5f0c101c60 msgr2=0x7f5f0c1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 shutdown_connections 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr 2024-02-09T22:42:15.261+0000 7f5f11bd6700 1 -- 172.21.15.3:0/1124177514 wait complete. 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:Or, if you are only running a single cluster on this host: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout:Please consider enabling telemetry to help improve Ceph: 2024-02-09T22:42:15.881 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout: ceph telemetry on 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout:For more information see: 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:15.882 INFO:teuthology.orchestra.run.smithi003.stdout:Bootstrap complete. 2024-02-09T22:42:15.917 INFO:tasks.cephadm:Fetching config... 2024-02-09T22:42:15.917 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:42:15.917 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-02-09T22:42:15.933 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-02-09T22:42:15.934 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:42:15.934 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-02-09T22:42:15.989 INFO:tasks.cephadm:Fetching mon keyring... 2024-02-09T22:42:15.989 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:42:15.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/keyring of=/dev/stdout 2024-02-09T22:42:16.055 INFO:tasks.cephadm:Fetching pub ssh key... 2024-02-09T22:42:16.055 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:42:16.055 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-02-09T22:42:16.110 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-02-09T22:42:16.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T22:42:16.189 INFO:teuthology.orchestra.run.smithi003.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:42:16.202 DEBUG:teuthology.orchestra.run.smithi064:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T22:42:16.257 INFO:teuthology.orchestra.run.smithi064.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:42:16.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-02-09T22:42:16.324 INFO:teuthology.orchestra.run.smithi089.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZg3vJ4pHwl/8Q/knyyPzwr/VE1bP8aRZGrCceO8MnjVC+Y3SS3LTyokZ471E/V54YMN1jtPGfxSjnGPJxfIPMG6/99agXShykMLPM/0QPFhjUm0rfom0k6udr+T9T7THKc5Sx2O0wkKzgQj2KY4LNxgHTKv1geNFvEKMmkXvWwYU0U3VUvM/dE+nKGtsjNk9fiWS3o4q1yLtxm9cMYKobdFVzml4Rq0I+QsjKFOEcl4X23fWKzgG3JK0NB9YRAzBnX0iPX5lFoFvMdOkA6lpEypr/I2H5m0TaMjXnOkXQguKDAE30CjsuOpNiNcDW0Vxc2vGzcgFDw+q84KzOvdbNUcR7BgeZInRzBCmABTQrPa+28l+D+DL+hP2vLrn30NFffaJE1PVErTuCwEXOLl18b8/DquznMsl3ieceKwVbJbhwtbqbqQX/LT7/PBQKfaNM2+b4CsuCnBWqiq0lrI+uMt6qNqEuFWYsq4t5h/hTuL4TL8hQ79qphFcSx85iF/0= ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:42:16.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-02-09T22:42:16.430 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:16 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1124177514' entity='client.admin' 2024-02-09T22:42:16.632 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 -- 172.21.15.3:0/3052143330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 msgr2=0x7f02401045e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 --2- 172.21.15.3:0/3052143330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f02401045e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0230009910 tx=0x7f0230009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 -- 172.21.15.3:0/3052143330 shutdown_connections 2024-02-09T22:42:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 --2- 172.21.15.3:0/3052143330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f02401045e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:18.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 -- 172.21.15.3:0/3052143330 >> 172.21.15.3:0/3052143330 conn(0x7f02400fbc20 msgr2=0x7f02400fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 -- 172.21.15.3:0/3052143330 shutdown_connections 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 -- 172.21.15.3:0/3052143330 wait complete. 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 Processor -- start 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.569+0000 7f0246052700 1 -- start start 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f0246052700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f0240101a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f0246052700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0240101f60 con 0x7f02401021f0 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f0240101a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:18.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f0240101a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33646/0 (socket says 172.21.15.3:33646) 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023f7fe700 1 -- 172.21.15.3:0/2063376107 learned_addr learned my addr 172.21.15.3:0/2063376107 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023f7fe700 1 -- 172.21.15.3:0/2063376107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02300095f0 con 0x7f02401021f0 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023f7fe700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f0240101a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0230004f40 tx=0x7f0230005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f023001d070 con 0x7f02401021f0 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02401001a0 con 0x7f02401021f0 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0240100640 con 0x7f02401021f0 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f023000be40 con 0x7f02401021f0 2024-02-09T22:42:18.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.570+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f023000f460 con 0x7f02401021f0 2024-02-09T22:42:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.571+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f023000f670 con 0x7f02401021f0 2024-02-09T22:42:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.572+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0240107ef0 con 0x7f02401021f0 2024-02-09T22:42:18.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.572+0000 7f023cff9700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0228038fe0 0x7f022803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.572+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f023004d300 con 0x7f02401021f0 2024-02-09T22:42:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.572+0000 7f023effd700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0228038fe0 0x7f022803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:18.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.572+0000 7f023effd700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0228038fe0 0x7f022803b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0234006fd0 tx=0x7f0234006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:18.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.577+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f023002a360 con 0x7f02401021f0 2024-02-09T22:42:18.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.779+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0240063350 con 0x7f02401021f0 2024-02-09T22:42:18.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.781+0000 7f023cff9700 1 -- 172.21.15.3:0/2063376107 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0230027030 con 0x7f02401021f0 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0228038fe0 msgr2=0x7f022803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0228038fe0 0x7f022803b4a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f0234006fd0 tx=0x7f0234006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 msgr2=0x7f0240101a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f0240101a20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0230004f40 tx=0x7f0230005020 comp rx=0 tx=0).stop 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 shutdown_connections 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0228038fe0 0x7f022803b4a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 --2- 172.21.15.3:0/2063376107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02401021f0 0x7f0240101a20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 >> 172.21.15.3:0/2063376107 conn(0x7f02400fbc20 msgr2=0x7f02400fc600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.783+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 shutdown_connections 2024-02-09T22:42:18.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:18.784+0000 7f0246052700 1 -- 172.21.15.3:0/2063376107 wait complete. 2024-02-09T22:42:19.508 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-02-09T22:42:19.509 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-02-09T22:42:20.014 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:20.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:19 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2063376107' entity='client.admin' 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 -- 172.21.15.3:0/3215395426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 msgr2=0x7ff01c100a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 --2- 172.21.15.3:0/3215395426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c100a70 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff018009910 tx=0x7ff018009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 -- 172.21.15.3:0/3215395426 shutdown_connections 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 --2- 172.21.15.3:0/3215395426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c100a70 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 -- 172.21.15.3:0/3215395426 >> 172.21.15.3:0/3215395426 conn(0x7ff01c0fbbf0 msgr2=0x7ff01c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 -- 172.21.15.3:0/3215395426 shutdown_connections 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.226+0000 7ff0232bb700 1 -- 172.21.15.3:0/3215395426 wait complete. 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff0232bb700 1 Processor -- start 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff0232bb700 1 -- start start 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff0232bb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c196ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:22.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff0232bb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c1971e0 con 0x7ff01c100650 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff021057700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c196ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff021057700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c196ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33676/0 (socket says 172.21.15.3:33676) 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff021057700 1 -- 172.21.15.3:0/577010836 learned_addr learned my addr 172.21.15.3:0/577010836 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.227+0000 7ff021057700 1 -- 172.21.15.3:0/577010836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180095f0 con 0x7ff01c100650 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.228+0000 7ff021057700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c196ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff018005850 tx=0x7ff018004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.228+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff01801d070 con 0x7ff01c100650 2024-02-09T22:42:22.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.228+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff01800be40 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.228+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01c1973e0 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.228+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff018022ea0 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.228+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c197880 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.229+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7ff01800b960 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.229+0000 7ff0127fc700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff008038fe0 0x7ff00803b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.229+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c190770 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.229+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ff01804d0a0 con 0x7ff01c100650 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.229+0000 7ff020856700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff008038fe0 0x7ff00803b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:22.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.229+0000 7ff020856700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff008038fe0 0x7ff00803b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff00c006fd0 tx=0x7ff00c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:22.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.234+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff018027020 con 0x7ff01c100650 2024-02-09T22:42:22.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.444+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7ff01c062160 con 0x7ff008038fe0 2024-02-09T22:42:22.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.447+0000 7ff0127fc700 1 -- 172.21.15.3:0/577010836 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff01c062160 con 0x7ff008038fe0 2024-02-09T22:42:22.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff008038fe0 msgr2=0x7ff00803b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:22.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff008038fe0 0x7ff00803b4a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff00c006fd0 tx=0x7ff00c006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:22.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 msgr2=0x7ff01c196ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:22.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c196ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff018005850 tx=0x7ff018004ff0 comp rx=0 tx=0).stop 2024-02-09T22:42:22.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 shutdown_connections 2024-02-09T22:42:22.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff008038fe0 0x7ff00803b4a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:22.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 --2- 172.21.15.3:0/577010836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff01c100650 0x7ff01c196ca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:22.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 >> 172.21.15.3:0/577010836 conn(0x7ff01c0fbbf0 msgr2=0x7ff01c069e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:22.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 shutdown_connections 2024-02-09T22:42:22.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:22.449+0000 7ff0232bb700 1 -- 172.21.15.3:0/577010836 wait complete. 2024-02-09T22:42:22.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:22.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:22.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi003", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:42:22.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:22.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:22.463 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:23.232 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi064 2024-02-09T22:42:23.233 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:42:23.233 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/etc/ceph/ceph.conf 2024-02-09T22:42:23.251 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:42:23.251 DEBUG:teuthology.orchestra.run.smithi064:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:42:23.309 INFO:tasks.cephadm:Adding host smithi064 to orchestrator... 2024-02-09T22:42:23.309 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch host add smithi064 2024-02-09T22:42:23.550 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:24.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:24 smithi003 ceph-mon[32678]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:42:24.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:24 smithi003 ceph-mon[32678]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:42:24.718 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:24 smithi003 ceph-mon[32678]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:42:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 -- 172.21.15.3:0/1542001914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 msgr2=0x7f5a04102d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 --2- 172.21.15.3:0/1542001914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04102d10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f59fc00b0a0 tx=0x7f59fc00b3b0 comp rx=0 tx=0).stop 2024-02-09T22:42:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 -- 172.21.15.3:0/1542001914 shutdown_connections 2024-02-09T22:42:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 --2- 172.21.15.3:0/1542001914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04102d10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 -- 172.21.15.3:0/1542001914 >> 172.21.15.3:0/1542001914 conn(0x7f5a04077260 msgr2=0x7f5a040fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:25.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 -- 172.21.15.3:0/1542001914 shutdown_connections 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.213+0000 7f5a091fb700 1 -- 172.21.15.3:0/1542001914 wait complete. 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a091fb700 1 Processor -- start 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a091fb700 1 -- start start 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a091fb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a091fb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a041998d0 con 0x7f5a04102930 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a02d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a02d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33700/0 (socket says 172.21.15.3:33700) 2024-02-09T22:42:25.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a02d9d700 1 -- 172.21.15.3:0/2452901272 learned_addr learned my addr 172.21.15.3:0/2452901272 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.214+0000 7f5a02d9d700 1 -- 172.21.15.3:0/2452901272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59fc009d00 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.215+0000 7f5a02d9d700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04199390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f59fc000f80 tx=0x7f59fc0075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.215+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59fc00e040 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.215+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f59fc007bc0 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.215+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a04199ad0 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.215+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f59fc01d5d0 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.215+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a04199ef0 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.216+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f59fc01d890 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.216+0000 7f59fbfff700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f59f0038fd0 0x7f59f003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.216+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f59fc04d1e0 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.216+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a04063350 con 0x7f5a04102930 2024-02-09T22:42:25.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.216+0000 7f5a0259c700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f59f0038fd0 0x7f59f003b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:25.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.216+0000 7f5a0259c700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f59f0038fd0 0x7f59f003b490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006fd0 tx=0x7f59ec006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:25.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.221+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f59fc017070 con 0x7f5a04102930 2024-02-09T22:42:25.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:25.429+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi064", "target": ["mon-mgr", ""]}) v1 -- 0x7f5a0410bb90 con 0x7f59f0038fd0 2024-02-09T22:42:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:25 smithi003 ceph-mon[32678]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi064", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:42:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:26 smithi003 ceph-mon[32678]: Deploying cephadm binary to smithi064 2024-02-09T22:42:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:26 smithi003 ceph-mon[32678]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:28 smithi003 ceph-mon[32678]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:29.798 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi064' with addr '172.21.15.64' 2024-02-09T22:42:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.794+0000 7f59fbfff700 1 -- 172.21.15.3:0/2452901272 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f5a0410bb90 con 0x7f59f0038fd0 2024-02-09T22:42:29.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f59f0038fd0 msgr2=0x7f59f003b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f59f0038fd0 0x7f59f003b490 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f59ec006fd0 tx=0x7f59ec006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 msgr2=0x7f5a04199390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04199390 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f59fc000f80 tx=0x7f59fc0075c0 comp rx=0 tx=0).stop 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 shutdown_connections 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f59f0038fd0 0x7f59f003b490 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 --2- 172.21.15.3:0/2452901272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a04102930 0x7f5a04199390 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 >> 172.21.15.3:0/2452901272 conn(0x7f5a04077260 msgr2=0x7f5a04107af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 shutdown_connections 2024-02-09T22:42:29.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:29.796+0000 7f5a091fb700 1 -- 172.21.15.3:0/2452901272 wait complete. 2024-02-09T22:42:30.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-09T22:42:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:31.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:31.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:30 smithi003 ceph-mon[32678]: Added host smithi064 2024-02-09T22:42:31.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:31.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:31.061 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:30 smithi003 ceph-mon[32678]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:31.940 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:31 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:32.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/3716409927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 msgr2=0x7fb8dc106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:32.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 --2- 172.21.15.3:0/3716409927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc106a70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc009910 tx=0x7fb8cc009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:32.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/3716409927 shutdown_connections 2024-02-09T22:42:32.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 --2- 172.21.15.3:0/3716409927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc106a70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:32.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/3716409927 >> 172.21.15.3:0/3716409927 conn(0x7fb8dc0fc0f0 msgr2=0x7fb8dc0fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/3716409927 shutdown_connections 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.133+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/3716409927 wait complete. 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e2bcd700 1 Processor -- start 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e2bcd700 1 -- start start 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e2bcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e2bcd700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8dc06a980 con 0x7fb8dc106690 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e0969700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e0969700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc195c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32802/0 (socket says 172.21.15.3:32802) 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e0969700 1 -- 172.21.15.3:0/2073556706 learned_addr learned my addr 172.21.15.3:0/2073556706 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:32.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.134+0000 7fb8e0969700 1 -- 172.21.15.3:0/2073556706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8cc0095f0 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.135+0000 7fb8e0969700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc195c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc004f40 tx=0x7fb8cc005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.135+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8cc01d070 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.135+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb8cc00b6f0 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.135+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8dc196180 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.135+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb8cc003810 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.135+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8dc196680 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.136+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7fb8cc00b860 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.136+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8dc06aac0 con 0x7fb8dc106690 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.136+0000 7fb8d9ffb700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb8c4038fe0 0x7fb8c403b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:32.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.136+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb8cc04ce20 con 0x7fb8dc106690 2024-02-09T22:42:32.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.136+0000 7fb8dbfff700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb8c4038fe0 0x7fb8c403b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:32.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.137+0000 7fb8dbfff700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb8c4038fe0 0x7fb8c403b4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0006fd0 tx=0x7fb8d0006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:32.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.141+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb8cc029360 con 0x7fb8dc106690 2024-02-09T22:42:32.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.347+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb8dc06a270 con 0x7fb8c4038fe0 2024-02-09T22:42:32.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.348+0000 7fb8d9ffb700 1 -- 172.21.15.3:0/2073556706 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+158 (secure 0 0 0) 0x7fb8dc06a270 con 0x7fb8c4038fe0 2024-02-09T22:42:32.351 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:32.351 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}] 2024-02-09T22:42:32.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb8c4038fe0 msgr2=0x7fb8c403b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:32.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb8c4038fe0 0x7fb8c403b4a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb8d0006fd0 tx=0x7fb8d0006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:32.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 msgr2=0x7fb8dc195c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc195c40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8cc004f40 tx=0x7fb8cc005020 comp rx=0 tx=0).stop 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 shutdown_connections 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb8c4038fe0 0x7fb8c403b4a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 --2- 172.21.15.3:0/2073556706 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb8dc106690 0x7fb8dc195c40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 >> 172.21.15.3:0/2073556706 conn(0x7fb8dc0fc0f0 msgr2=0x7fb8dc0fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.350+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 shutdown_connections 2024-02-09T22:42:32.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:32.351+0000 7fb8e2bcd700 1 -- 172.21.15.3:0/2073556706 wait complete. 2024-02-09T22:42:32.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:32 smithi003 ceph-mon[32678]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:42:32.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:32 smithi003 ceph-mon[32678]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:32.977 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:33.005 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi089 2024-02-09T22:42:33.005 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:42:33.005 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/etc/ceph/ceph.conf 2024-02-09T22:42:33.024 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:42:33.024 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:42:33.081 INFO:tasks.cephadm:Adding host smithi089 to orchestrator... 2024-02-09T22:42:33.081 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch host add smithi089 2024-02-09T22:42:33.320 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:34.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.884+0000 7fb4960b6700 1 -- 172.21.15.3:0/3388487977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 msgr2=0x7fb490106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.884+0000 7fb4960b6700 1 --2- 172.21.15.3:0/3388487977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490106a60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb480009910 tx=0x7fb480009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 -- 172.21.15.3:0/3388487977 shutdown_connections 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 --2- 172.21.15.3:0/3388487977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490106a60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 -- 172.21.15.3:0/3388487977 >> 172.21.15.3:0/3388487977 conn(0x7fb490077260 msgr2=0x7fb4900fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 -- 172.21.15.3:0/3388487977 shutdown_connections 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 -- 172.21.15.3:0/3388487977 wait complete. 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 Processor -- start 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.885+0000 7fb4960b6700 1 -- start start 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb4960b6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:34.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb4960b6700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49006a0c0 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb48f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb48f7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32820/0 (socket says 172.21.15.3:32820) 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb48f7fe700 1 -- 172.21.15.3:0/2799079140 learned_addr learned my addr 172.21.15.3:0/2799079140 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb48f7fe700 1 -- 172.21.15.3:0/2799079140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4800095f0 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb48f7fe700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490197e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb4800092e0 tx=0x7fb480004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.886+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb48001d070 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.887+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb48000b750 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.887+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb490198400 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.887+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4800038a0 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.887+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb490198880 con 0x7fb490106680 2024-02-09T22:42:34.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.887+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7fb480003b60 con 0x7fb490106680 2024-02-09T22:42:34.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.887+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb490063350 con 0x7fb490106680 2024-02-09T22:42:34.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.888+0000 7fb48cff9700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb478038f80 0x7fb47803b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:34.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.888+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb48004ce40 con 0x7fb490106680 2024-02-09T22:42:34.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.888+0000 7fb48effd700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb478038f80 0x7fb47803b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:34.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.888+0000 7fb48effd700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb478038f80 0x7fb47803b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb484006fd0 tx=0x7fb484006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:34.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:34.893+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb480029900 con 0x7fb490106680 2024-02-09T22:42:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:35.098+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}) v1 -- 0x7fb490198be0 con 0x7fb478038f80 2024-02-09T22:42:35.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:34 smithi003 ceph-mon[32678]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:36.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:42:37.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:37 smithi003 ceph-mon[32678]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:42:37.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:37 smithi003 ceph-mon[32678]: Deploying cephadm binary to smithi089 2024-02-09T22:42:37.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:37 smithi003 ceph-mon[32678]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:42:37.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:37 smithi003 ceph-mon[32678]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:42:37.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:37 smithi003 ceph-mon[32678]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:38 smithi003 ceph-mon[32678]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:39.371 INFO:teuthology.orchestra.run.smithi003.stdout:Added host 'smithi089' with addr '172.21.15.89' 2024-02-09T22:42:39.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.367+0000 7fb48cff9700 1 -- 172.21.15.3:0/2799079140 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fb490198be0 con 0x7fb478038f80 2024-02-09T22:42:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb478038f80 msgr2=0x7fb47803b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb478038f80 0x7fb47803b440 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb484006fd0 tx=0x7fb484006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 msgr2=0x7fb490197e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490197e00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb4800092e0 tx=0x7fb480004ff0 comp rx=0 tx=0).stop 2024-02-09T22:42:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 shutdown_connections 2024-02-09T22:42:39.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb478038f80 0x7fb47803b440 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 --2- 172.21.15.3:0/2799079140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490106680 0x7fb490197e00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 >> 172.21.15.3:0/2799079140 conn(0x7fb490077260 msgr2=0x7fb4900fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 shutdown_connections 2024-02-09T22:42:39.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:39.369+0000 7fb4960b6700 1 -- 172.21.15.3:0/2799079140 wait complete. 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: Added host smithi089 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:39.973 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:40.000 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch host ls --format=json 2024-02-09T22:42:40.237 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:41.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:41.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:41 smithi003 ceph-mon[32678]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:41.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 -- 172.21.15.3:0/4134222606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 msgr2=0x7fc0fc107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:41.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 --2- 172.21.15.3:0/4134222606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc107ae0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009910 tx=0x7fc0f0009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:41.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 -- 172.21.15.3:0/4134222606 shutdown_connections 2024-02-09T22:42:41.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 --2- 172.21.15.3:0/4134222606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc107ae0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:41.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 -- 172.21.15.3:0/4134222606 >> 172.21.15.3:0/4134222606 conn(0x7fc0fc0fd230 msgr2=0x7fc0fc0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:41.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 -- 172.21.15.3:0/4134222606 shutdown_connections 2024-02-09T22:42:41.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.900+0000 7fc102e31700 1 -- 172.21.15.3:0/4134222606 wait complete. 2024-02-09T22:42:41.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc102e31700 1 Processor -- start 2024-02-09T22:42:41.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc102e31700 1 -- start start 2024-02-09T22:42:41.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc102e31700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc199220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:41.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc102e31700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0fc199760 con 0x7fc0fc107700 2024-02-09T22:42:41.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc100bcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc199220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc100bcd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc199220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38404/0 (socket says 172.21.15.3:38404) 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.901+0000 7fc100bcd700 1 -- 172.21.15.3:0/823390405 learned_addr learned my addr 172.21.15.3:0/823390405 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc100bcd700 1 -- 172.21.15.3:0/823390405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f00095f0 con 0x7fc0fc107700 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc100bcd700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc199220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0004f40 tx=0x7fc0f0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0f001d070 con 0x7fc0fc107700 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0fc1999c0 con 0x7fc0fc107700 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0f000bbd0 con 0x7fc0fc107700 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc0f0003900 con 0x7fc0fc107700 2024-02-09T22:42:41.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.902+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0fc199e40 con 0x7fc0fc107700 2024-02-09T22:42:41.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.903+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7fc0f0003bc0 con 0x7fc0fc107700 2024-02-09T22:42:41.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.903+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0fc063350 con 0x7fc0fc107700 2024-02-09T22:42:41.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.903+0000 7fc0f9ffb700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0ec038f80 0x7fc0ec03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:41.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.903+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc0f004cdc0 con 0x7fc0fc107700 2024-02-09T22:42:41.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.903+0000 7fc0fbfff700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0ec038f80 0x7fc0ec03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:41.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.904+0000 7fc0fbfff700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0ec038f80 0x7fc0ec03b440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006fd0 tx=0x7fc0e8006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:41.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:41.908+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0f0029910 con 0x7fc0fc107700 2024-02-09T22:42:42.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.117+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc0fc10bcf0 con 0x7fc0ec038f80 2024-02-09T22:42:42.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.118+0000 7fc0f9ffb700 1 -- 172.21.15.3:0/823390405 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+237 (secure 0 0 0) 0x7fc0fc10bcf0 con 0x7fc0ec038f80 2024-02-09T22:42:42.122 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:42:42.122 INFO:teuthology.orchestra.run.smithi003.stdout:[{"addr": "172.21.15.3", "hostname": "smithi003", "labels": [], "status": ""}, {"addr": "172.21.15.64", "hostname": "smithi064", "labels": [], "status": ""}, {"addr": "172.21.15.89", "hostname": "smithi089", "labels": [], "status": ""}] 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0ec038f80 msgr2=0x7fc0ec03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0ec038f80 0x7fc0ec03b440 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8006fd0 tx=0x7fc0e8006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 msgr2=0x7fc0fc199220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc199220 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0004f40 tx=0x7fc0f0005020 comp rx=0 tx=0).stop 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 shutdown_connections 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0ec038f80 0x7fc0ec03b440 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 --2- 172.21.15.3:0/823390405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0fc107700 0x7fc0fc199220 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 >> 172.21.15.3:0/823390405 conn(0x7fc0fc0fd230 msgr2=0x7fc0fc0759f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 shutdown_connections 2024-02-09T22:42:42.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:42.120+0000 7fc102e31700 1 -- 172.21.15.3:0/823390405 wait complete. 2024-02-09T22:42:42.724 INFO:tasks.cephadm:Setting crush tunables to default 2024-02-09T22:42:42.724 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd crush tunables default 2024-02-09T22:42:42.962 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:42:43.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:43.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:42 smithi003 ceph-mon[32678]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:42:43.313 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:42 smithi003 ceph-mon[32678]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 -- 172.21.15.3:0/4236958682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 msgr2=0x7fcf78106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 --2- 172.21.15.3:0/4236958682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78106a70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcf74009910 tx=0x7fcf74009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 -- 172.21.15.3:0/4236958682 shutdown_connections 2024-02-09T22:42:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 --2- 172.21.15.3:0/4236958682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78106a70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 -- 172.21.15.3:0/4236958682 >> 172.21.15.3:0/4236958682 conn(0x7fcf780fc0f0 msgr2=0x7fcf780fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 -- 172.21.15.3:0/4236958682 shutdown_connections 2024-02-09T22:42:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.495+0000 7fcf80724700 1 -- 172.21.15.3:0/4236958682 wait complete. 2024-02-09T22:42:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf80724700 1 Processor -- start 2024-02-09T22:42:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf80724700 1 -- start start 2024-02-09T22:42:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf80724700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78195ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf80724700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf78196420 con 0x7fcf78106690 2024-02-09T22:42:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf7e4c0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78195ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf7e4c0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78195ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38436/0 (socket says 172.21.15.3:38436) 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.496+0000 7fcf7e4c0700 1 -- 172.21.15.3:0/4283823472 learned_addr learned my addr 172.21.15.3:0/4283823472 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf7e4c0700 1 -- 172.21.15.3:0/4283823472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf740095f0 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf7e4c0700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78195ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf74004f40 tx=0x7fcf74005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf7401d070 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf7400b6f0 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf78196680 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcf74003810 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.497+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf78196b80 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.498+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf78063350 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.498+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7fcf7400b860 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.498+0000 7fcf6b7fe700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcf64038f80 0x7fcf6403b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.498+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcf7404ccf0 con 0x7fcf78106690 2024-02-09T22:42:44.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.498+0000 7fcf7dcbf700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcf64038f80 0x7fcf6403b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:44.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.499+0000 7fcf7dcbf700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcf64038f80 0x7fcf6403b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c006fd0 tx=0x7fcf6c006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:44.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.503+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcf74051070 con 0x7fcf78106690 2024-02-09T22:42:44.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.711+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fcf78197240 con 0x7fcf78106690 2024-02-09T22:42:44.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.994+0000 7fcf6b7fe700 1 -- 172.21.15.3:0/4283823472 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fcf74026020 con 0x7fcf78106690 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcf64038f80 msgr2=0x7fcf6403b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcf64038f80 0x7fcf6403b440 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c006fd0 tx=0x7fcf6c006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 msgr2=0x7fcf78195ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78195ee0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf74004f40 tx=0x7fcf74005020 comp rx=0 tx=0).stop 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 shutdown_connections 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcf64038f80 0x7fcf6403b440 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 --2- 172.21.15.3:0/4283823472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf78106690 0x7fcf78195ee0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 >> 172.21.15.3:0/4283823472 conn(0x7fcf780fc0f0 msgr2=0x7fcf780fd910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 shutdown_connections 2024-02-09T22:42:45.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:42:44.998+0000 7fcf80724700 1 -- 172.21.15.3:0/4283823472 wait complete. 2024-02-09T22:42:45.003 INFO:teuthology.orchestra.run.smithi003.stderr:adjusted tunables profile to default 2024-02-09T22:42:45.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:44 smithi003 ceph-mon[32678]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:45.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:44 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4283823472' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-09T22:42:45.710 INFO:tasks.cephadm:Adding mon.a on smithi003 2024-02-09T22:42:45.710 INFO:tasks.cephadm:Adding mon.b on smithi064 2024-02-09T22:42:45.710 INFO:tasks.cephadm:Adding mon.c on smithi089 2024-02-09T22:42:45.710 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch apply mon '3;smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c' 2024-02-09T22:42:45.946 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4283823472' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:42:46.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:42:46.385 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:45 smithi003 ceph-mon[32678]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:42:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:47 smithi003 ceph-mon[32678]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:42:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:47 smithi003 ceph-mon[32678]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:42:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:47 smithi003 ceph-mon[32678]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:47.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 -- 172.21.15.89:0/4166416944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 msgr2=0x7f0edc107ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:47.497 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 --2- 172.21.15.89:0/4166416944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc107ae0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0009910 tx=0x7f0ed0009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 -- 172.21.15.89:0/4166416944 shutdown_connections 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 --2- 172.21.15.89:0/4166416944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc107ae0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 -- 172.21.15.89:0/4166416944 >> 172.21.15.89:0/4166416944 conn(0x7f0edc0fd230 msgr2=0x7f0edc0ff650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 -- 172.21.15.89:0/4166416944 shutdown_connections 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.493+0000 7f0ee3240700 1 -- 172.21.15.89:0/4166416944 wait complete. 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee3240700 1 Processor -- start 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee3240700 1 -- start start 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee3240700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc197050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee3240700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0edc197590 con 0x7f0edc107700 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee0fdc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc197050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee0fdc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc197050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.89:39694/0 (socket says 172.21.15.89:39694) 2024-02-09T22:42:47.498 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.494+0000 7f0ee0fdc700 1 -- 172.21.15.89:0/2272697343 learned_addr learned my addr 172.21.15.89:0/2272697343 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ee0fdc700 1 -- 172.21.15.89:0/2272697343 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ed00095f0 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ee0fdc700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc197050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0004f40 tx=0x7f0ed0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ed001d070 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0edc1977f0 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ed000bbd0 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0ed0003900 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.495+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0edc197c70 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.496+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0edc063350 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.496+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f0ed000bd40 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.496+0000 7f0ed9ffb700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0ecc038f80 0x7f0ecc03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.496+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f0ed004ce80 con 0x7f0edc107700 2024-02-09T22:42:47.500 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.497+0000 7f0edbfff700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0ecc038f80 0x7f0ecc03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:47.501 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.497+0000 7f0edbfff700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0ecc038f80 0x7f0ecc03b440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:47.504 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.502+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0ed002ade0 con 0x7f0edc107700 2024-02-09T22:42:47.707 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.703+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f0edc10bd00 con 0x7f0ecc038f80 2024-02-09T22:42:47.713 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.710+0000 7f0ed9ffb700 1 -- 172.21.15.89:0/2272697343 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0edc10bd00 con 0x7f0ecc038f80 2024-02-09T22:42:47.713 INFO:teuthology.orchestra.run.smithi089.stdout:Scheduled mon update... 2024-02-09T22:42:47.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0ecc038f80 msgr2=0x7f0ecc03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:47.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0ecc038f80 0x7f0ecc03b440 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f0ec8006fd0 tx=0x7f0ec8006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:47.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 msgr2=0x7f0edc197050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:47.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc197050 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f0ed0004f40 tx=0x7f0ed0005020 comp rx=0 tx=0).stop 2024-02-09T22:42:47.715 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 shutdown_connections 2024-02-09T22:42:47.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0ecc038f80 0x7f0ecc03b440 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:47.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 --2- 172.21.15.89:0/2272697343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0edc107700 0x7f0edc197050 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:47.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 >> 172.21.15.89:0/2272697343 conn(0x7f0edc0fd230 msgr2=0x7f0edc109220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:47.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 shutdown_connections 2024-02-09T22:42:47.716 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:47.712+0000 7f0ee3240700 1 -- 172.21.15.89:0/2272697343 wait complete. 2024-02-09T22:42:48.257 DEBUG:teuthology.orchestra.run.smithi064:mon.b> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.b.service 2024-02-09T22:42:48.260 DEBUG:teuthology.orchestra.run.smithi089:mon.c> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.c.service 2024-02-09T22:42:48.263 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-09T22:42:48.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T22:42:48.954 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c;count:3 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:42:49.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:49.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: Deploying daemon mon.c on smithi089 2024-02-09T22:42:49.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:48 smithi003 ceph-mon[32678]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:50 smithi003 ceph-mon[32678]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:51.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 -- 172.21.15.89:0/1543619152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 msgr2=0x7f085c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:51.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 --2- 172.21.15.89:0/1543619152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c106a70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f084c009910 tx=0x7f084c009c20 comp rx=0 tx=0).stop 2024-02-09T22:42:51.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 -- 172.21.15.89:0/1543619152 shutdown_connections 2024-02-09T22:42:51.390 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 --2- 172.21.15.89:0/1543619152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c106a70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 -- 172.21.15.89:0/1543619152 >> 172.21.15.89:0/1543619152 conn(0x7f085c0fc110 msgr2=0x7f085c0fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 -- 172.21.15.89:0/1543619152 shutdown_connections 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 -- 172.21.15.89:0/1543619152 wait complete. 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 Processor -- start 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.386+0000 7f0862263700 1 -- start start 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f0862263700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c194e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f0862263700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f085c195340 con 0x7f085c106690 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f085bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c194e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f085bfff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c194e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.89:39704/0 (socket says 172.21.15.89:39704) 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f085bfff700 1 -- 172.21.15.89:0/771599762 learned_addr learned my addr 172.21.15.89:0/771599762 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f085bfff700 1 -- 172.21.15.89:0/771599762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084c0095f0 con 0x7f085c106690 2024-02-09T22:42:51.391 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.387+0000 7f085bfff700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c194e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f084c004f40 tx=0x7f084c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:51.392 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.388+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f084c01d070 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.388+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f085c1955a0 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.388+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f084c00b6f0 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.388+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f084c003810 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.388+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f085c195aa0 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.389+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f084c004260 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.389+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085c063350 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.389+0000 7f08597fa700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f084803b1e0 0x7f084803d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.389+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f084c04cc30 con 0x7f085c106690 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.389+0000 7f085b7fe700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f084803b1e0 0x7f084803d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:51.393 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.390+0000 7f085b7fe700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f084803b1e0 0x7f084803d6a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0844006fd0 tx=0x7f0844006e40 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:51.397 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.394+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f084c029360 con 0x7f085c106690 2024-02-09T22:42:51.673 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.670+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f085c196110 con 0x7f085c106690 2024-02-09T22:42:51.674 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.671+0000 7f08597fa700 1 -- 172.21.15.89:0/771599762 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+741 (secure 0 0 0) 0x7f084c026070 con 0x7f085c106690 2024-02-09T22:42:51.674 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:42:51.674 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":1,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","modified":"2024-02-09T22:41:05.848623Z","created":"2024-02-09T22:41:05.848623Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f084803b1e0 msgr2=0x7f084803d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f084803b1e0 0x7f084803d6a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0844006fd0 tx=0x7f0844006e40 comp rx=0 tx=0).stop 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 msgr2=0x7f085c194e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c194e00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f084c004f40 tx=0x7f084c005020 comp rx=0 tx=0).stop 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 shutdown_connections 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f084803b1e0 0x7f084803d6a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 --2- 172.21.15.89:0/771599762 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f085c106690 0x7f085c194e00 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 >> 172.21.15.89:0/771599762 conn(0x7f085c0fc110 msgr2=0x7f085c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:51.676 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 shutdown_connections 2024-02-09T22:42:51.677 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:51.673+0000 7f0862263700 1 -- 172.21.15.89:0/771599762 wait complete. 2024-02-09T22:42:51.677 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 1 2024-02-09T22:42:52.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/771599762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:42:53.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:52 smithi003 ceph-mon[32678]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:53.542 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-09T22:42:53.542 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T22:42:53.773 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:42:58.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:42:58 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:58.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.766+0000 7f4699d0d700 1 -- 172.21.15.89:0/1982013406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f467c0035c0 msgr2=0x7f467c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:58.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.766+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1982013406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f467c0035c0 0x7f467c005a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f4694078f50 tx=0x7f4684009e80 comp rx=0 tx=0).stop 2024-02-09T22:42:58.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.766+0000 7f4699d0d700 1 -- 172.21.15.89:0/1982013406 shutdown_connections 2024-02-09T22:42:58.772 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.766+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1982013406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f467c0035c0 0x7f467c005a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.766+0000 7f4699d0d700 1 -- 172.21.15.89:0/1982013406 >> 172.21.15.89:0/1982013406 conn(0x7f4694076e10 msgr2=0x7f4694077220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.766+0000 7f4699d0d700 1 -- 172.21.15.89:0/1982013406 shutdown_connections 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.767+0000 7f4699d0d700 1 -- 172.21.15.89:0/1982013406 wait complete. 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.767+0000 7f4699d0d700 1 Processor -- start 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4699d0d700 1 -- start start 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4699d0d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4699d0d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46941998d0 0x7f469419dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4699d0d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4694199e60 con 0x7f46941998d0 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4699d0d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4694199fd0 con 0x7f467c0035c0 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36206/0 (socket says 172.21.15.89:36206) 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 -- 172.21.15.89:0/1459112020 learned_addr learned my addr 172.21.15.89:0/1459112020 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:42:58.773 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4692ffd700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46941998d0 0x7f469419dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 -- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 msgr2=0x7f4694199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 -- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 msgr2=0x7f4694199390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f46937fe700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4692ffd700 1 -- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 msgr2=0x7f4694199390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4692ffd700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.768+0000 7f4692ffd700 1 -- 172.21.15.89:0/1459112020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46840098d0 con 0x7f46941998d0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.769+0000 7f4692ffd700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46941998d0 0x7f469419dc60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f468800b4d0 tx=0x7f468800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.769+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4688011630 con 0x7f46941998d0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.769+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4688011c70 con 0x7f46941998d0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.769+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f4688010de0 con 0x7f46941998d0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.769+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469419e200 con 0x7f46941998d0 2024-02-09T22:42:58.774 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.769+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469419e750 con 0x7f46941998d0 2024-02-09T22:42:58.775 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.770+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f468801a400 con 0x7f46941998d0 2024-02-09T22:42:58.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.770+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4694106b90 con 0x7f46941998d0 2024-02-09T22:42:58.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.770+0000 7f4690ff9700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4680038d90 0x7f468003b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:42:58.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.770+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f468804b780 con 0x7f46941998d0 2024-02-09T22:42:58.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.770+0000 7f46937fe700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4680038d90 0x7f468003b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:42:58.776 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.770+0000 7f46937fe700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4680038d90 0x7f468003b250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f468401b040 tx=0x7f468401ae50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:42:58.778 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:58.775+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4688017890 con 0x7f46941998d0 2024-02-09T22:42:58.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:42:58.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: mon.a calling monitor election 2024-02-09T22:42:58.815 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: mon.c calling monitor election 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:58.816 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: fsmap 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: mgrmap e13: x(active, since 51s) 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: overall HEALTH_OK 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:42:58.817 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:42:59.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.048+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f469419a6f0 con 0x7f46941998d0 2024-02-09T22:42:59.051 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.048+0000 7f4690ff9700 1 -- 172.21.15.89:0/1459112020 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1007 (secure 0 0 0) 0x7f46880172c0 con 0x7f46941998d0 2024-02-09T22:42:59.052 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:42:59.052 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":2,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","modified":"2024-02-09T22:42:53.371464Z","created":"2024-02-09T22:41:05.848623Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-02-09T22:42:59.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4680038d90 msgr2=0x7f468003b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:59.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4680038d90 0x7f468003b250 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f468401b040 tx=0x7f468401ae50 comp rx=0 tx=0).stop 2024-02-09T22:42:59.054 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46941998d0 msgr2=0x7f469419dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46941998d0 0x7f469419dc60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f468800b4d0 tx=0x7f468800b890 comp rx=0 tx=0).stop 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 shutdown_connections 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4680038d90 0x7f468003b250 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f467c0035c0 0x7f4694199390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 --2- 172.21.15.89:0/1459112020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46941998d0 0x7f469419dc60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 >> 172.21.15.89:0/1459112020 conn(0x7f4694076e10 msgr2=0x7f46941019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 shutdown_connections 2024-02-09T22:42:59.055 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:42:59.051+0000 7f4699d0d700 1 -- 172.21.15.89:0/1459112020 wait complete. 2024-02-09T22:42:59.056 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 2 2024-02-09T22:42:59.646 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:42:59 smithi089 ceph-mon[36665]: Deploying daemon mon.b on smithi064 2024-02-09T22:42:59.646 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:42:59 smithi089 ceph-mon[36665]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:59.646 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:42:59 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/1459112020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:42:59.647 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:42:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:42:59.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:59 smithi003 ceph-mon[32678]: Deploying daemon mon.b on smithi064 2024-02-09T22:42:59.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:59 smithi003 ceph-mon[32678]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:42:59.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:59 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/1459112020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:42:59.779 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:42:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:00.676 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-02-09T22:43:00.677 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph mon dump -f json 2024-02-09T22:43:00.919 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:43:01.701 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:01 smithi089 ceph-mon[36665]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:01.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:01 smithi003 ceph-mon[32678]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 podman[36619]: 2024-02-09 22:43:02.153565162 +0000 UTC m=+0.122644256 container create aea8794133553767c48cb5a79324f9325b077a82821c997c5b82c73e1d66f940 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 -- 172.21.15.89:0/1114036836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba941066f0 msgr2=0x7fba94106ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 --2- 172.21.15.89:0/1114036836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba941066f0 0x7fba94106ad0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fba8c009910 tx=0x7fba8c009c20 comp rx=0 tx=0).stop 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 -- 172.21.15.89:0/1114036836 shutdown_connections 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 --2- 172.21.15.89:0/1114036836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba941066f0 0x7fba94106ad0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 -- 172.21.15.89:0/1114036836 >> 172.21.15.89:0/1114036836 conn(0x7fba940fc1c0 msgr2=0x7fba940fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 -- 172.21.15.89:0/1114036836 shutdown_connections 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 -- 172.21.15.89:0/1114036836 wait complete. 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.317+0000 7fba9967b700 1 Processor -- start 2024-02-09T22:43:02.322 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba9967b700 1 -- start start 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba9967b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba9967b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba94196710 0x7fba9419aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba9967b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba94196ca0 con 0x7fba94196710 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba9967b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba94196e10 con 0x7fba941066f0 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36218/0 (socket says 172.21.15.89:36218) 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 -- 172.21.15.89:0/2217003074 learned_addr learned my addr 172.21.15.89:0/2217003074 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba927fc700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba94196710 0x7fba9419aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 -- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 msgr2=0x7fba941961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 -- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 msgr2=0x7fba941961d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba92ffd700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba927fc700 1 -- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 msgr2=0x7fba941961d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:43:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba927fc700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.318+0000 7fba927fc700 1 -- 172.21.15.89:0/2217003074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba8c0095f0 con 0x7fba94196710 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.319+0000 7fba927fc700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba94196710 0x7fba9419aaa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00b4d0 tx=0x7fba7c00b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.319+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fba7c011630 con 0x7fba94196710 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.319+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7c011c70 con 0x7fba94196710 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.319+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba9419b040 con 0x7fba94196710 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.319+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fba7c010de0 con 0x7fba94196710 2024-02-09T22:43:02.324 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.319+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba9419b590 con 0x7fba94196710 2024-02-09T22:43:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.320+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7fba7c01a400 con 0x7fba94196710 2024-02-09T22:43:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.320+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba94101e30 con 0x7fba94196710 2024-02-09T22:43:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.320+0000 7fba8bfff700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fba80038d20 0x7fba8003b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.320+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fba7c04b8c0 con 0x7fba94196710 2024-02-09T22:43:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.320+0000 7fba92ffd700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fba80038d20 0x7fba8003b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:02.325 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.321+0000 7fba92ffd700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fba80038d20 0x7fba8003b1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00b5c0 tx=0x7fba8c009ec0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:02.328 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.325+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba7c00fd50 con 0x7fba94196710 2024-02-09T22:43:02.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.421+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7c050020 con 0x7fba94196710 2024-02-09T22:43:02.601 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:02.598+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fba94063350 con 0x7fba94196710 2024-02-09T22:43:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 podman[36619]: 2024-02-09 22:43:02.364167101 +0000 UTC m=+0.333246159 container init aea8794133553767c48cb5a79324f9325b077a82821c997c5b82c73e1d66f940 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-09T22:43:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 podman[36619]: 2024-02-09 22:43:02.367217182 +0000 UTC m=+0.336296229 container start aea8794133553767c48cb5a79324f9325b077a82821c997c5b82c73e1d66f940 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2) 2024-02-09T22:43:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: set uid:gid to 167:167 (ceph:ceph) 2024-02-09T22:43:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: ceph version 19.0.0-1361-g290212a1 (290212a1bfd33563ba76ee5b7f0b944fdb6c5d90) squid (dev), process ceph-mon, pid 2 2024-02-09T22:43:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pidfile_write: ignore empty --pid-file 2024-02-09T22:43:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: load: jerasure load: lrc 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: RocksDB version: 7.9.2 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Git sha 0 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Compile date 2024-02-09 17:35:33 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: DB SUMMARY 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: DB Session ID: QR6L0CONB47VZQLX4MKQ 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: CURRENT file: CURRENT 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: IDENTITY file: IDENTITY 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.error_if_exists: 0 2024-02-09T22:43:02.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.create_if_missing: 0 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.paranoid_checks: 1 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.flush_verify_memtable_count: 1 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.env: 0x564b41359ae0 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.fs: PosixFileSystem 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.info_log: 0x564b439518c0 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_file_opening_threads: 16 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.statistics: (nil) 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.use_fsync: 0 2024-02-09T22:43:02.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_log_file_size: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.log_file_time_to_roll: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.keep_log_file_num: 1000 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.recycle_log_file_num: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.allow_fallocate: 1 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.allow_mmap_reads: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.allow_mmap_writes: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.use_direct_reads: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.create_missing_column_families: 0 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.db_log_dir: 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.wal_dir: 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.table_cache_numshardbits: 6 2024-02-09T22:43:02.689 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.WAL_ttl_seconds: 0 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.WAL_size_limit_MB: 0 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.is_fd_close_on_exec: 1 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.advise_random_on_open: 1 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.db_write_buffer_size: 0 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.write_buffer_manager: 0x564b43925400 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.use_adaptive_mutex: 0 2024-02-09T22:43:02.690 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.rate_limiter: (nil) 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.wal_recovery_mode: 2 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.enable_thread_tracking: 0 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.enable_pipelined_write: 0 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.unordered_write: 0 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.row_cache: None 2024-02-09T22:43:02.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.wal_filter: None 2024-02-09T22:43:02.693 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-02-09T22:43:02.693 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.allow_ingest_behind: 0 2024-02-09T22:43:02.693 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.two_write_queues: 0 2024-02-09T22:43:02.693 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.manual_wal_flush: 0 2024-02-09T22:43:02.693 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.wal_compression: 0 2024-02-09T22:43:02.693 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.atomic_flush: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.persist_stats_to_disk: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.write_dbid_to_manifest: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.log_readahead_size: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.best_efforts_recovery: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.allow_data_in_errors: 0 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.db_host_id: __hostname__ 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.enforce_single_del_contracts: true 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_background_jobs: 2 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_background_compactions: -1 2024-02-09T22:43:02.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_subcompactions: 1 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.delayed_write_rate : 16777216 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_total_wal_size: 0 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.stats_dump_period_sec: 600 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.stats_persist_period_sec: 600 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_open_files: -1 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bytes_per_sync: 0 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.wal_bytes_per_sync: 0 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.strict_bytes_per_sync: 0 2024-02-09T22:43:02.695 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_readahead_size: 0 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_background_flushes: -1 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Compression algorithms supported: 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kZSTD supported: 0 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kXpressCompression supported: 0 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kLZ4HCCompression supported: 1 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kZlibCompression supported: 1 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kSnappyCompression supported: 1 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kLZ4Compression supported: 1 2024-02-09T22:43:02.696 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: kBZip2Compression supported: 0 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Fast CRC32 supported: Supported on x86 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: DMutex implementation: pthread_mutex_t 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.merge_operator: 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_filter: None 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_filter_factory: None 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.sst_partitioner_factory: None 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.memtable_factory: SkipListFactory 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.table_factory: BlockBasedTable 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564b439519c0) 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout: cache_index_and_filter_blocks: 1 2024-02-09T22:43:02.697 INFO:journalctl@ceph.mon.b.smithi064.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: pin_top_level_index_and_filter: 1 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: index_type: 0 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: data_block_index_type: 0 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: index_shortening: 1 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: data_block_hash_table_util_ratio: 0.750000 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: checksum: 4 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: no_block_cache: 0 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache: 0x564b43955090 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache_name: BinnedLRUCache 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache_options: 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: capacity : 536870912 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: num_shard_bits : 4 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: strict_capacity_limit : 0 2024-02-09T22:43:02.698 INFO:journalctl@ceph.mon.b.smithi064.stdout: high_pri_pool_ratio: 0.000 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_cache_compressed: (nil) 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: persistent_cache: (nil) 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_size: 4096 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_size_deviation: 10 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_restart_interval: 16 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: index_block_restart_interval: 1 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: metadata_block_size: 4096 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: partition_filters: 0 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: use_delta_encoding: 1 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: filter_policy: bloomfilter 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: whole_key_filtering: 1 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: verify_compression: 0 2024-02-09T22:43:02.699 INFO:journalctl@ceph.mon.b.smithi064.stdout: read_amp_bytes_per_bit: 0 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: format_version: 5 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: enable_index_compression: 1 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: block_align: 0 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: max_auto_readahead_size: 262144 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: prepopulate_block_cache: 0 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: initial_auto_readahead_size: 8192 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout: num_file_reads_for_auto_readahead: 2 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.write_buffer_size: 33554432 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_write_buffer_number: 2 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression: NoCompression 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression: Disabled 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.prefix_extractor: nullptr 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-02-09T22:43:02.700 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.num_levels: 7 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.window_bits: -14 2024-02-09T22:43:02.701 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.level: 32767 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.strategy: 0 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.enabled: false 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.target_file_size_base: 67108864 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.target_file_size_multiplier: 1 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-02-09T22:43:02.702 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.arena_block_size: 1048576 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.disable_auto_compactions: 0 2024-02-09T22:43:02.703 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.table_properties_collectors: 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.inplace_update_support: 0 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.inplace_update_num_locks: 10000 2024-02-09T22:43:02.704 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.memtable_huge_page_size: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.bloom_locality: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.max_successive_merges: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.optimize_filters_for_hits: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.paranoid_file_checks: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.force_consistency_checks: 1 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.report_bg_io_stats: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.ttl: 2592000 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.periodic_compaction_seconds: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.enable_blob_files: false 2024-02-09T22:43:02.705 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.min_blob_size: 0 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.blob_file_size: 268435456 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.blob_compression_type: NoCompression 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.enable_blob_garbage_collection: false 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.blob_file_starting_level: 0 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 1db93cba-0634-42eb-ace9-8661a4e3b5d9 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707518582396924, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-02-09T22:43:02.706 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707518582397348, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1707518582, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "1db93cba-0634-42eb-ace9-8661a4e3b5d9", "db_session_id": "QR6L0CONB47VZQLX4MKQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: EVENT_LOG_v1 {"time_micros": 1707518582397431, "job": 1, "event": "recovery_finished"} 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564b43a46000 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: DB pointer 0x564b43a30000 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** DB Stats ** 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T22:43:02.707 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T22:43:02.708 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** Compaction Stats [default] ** 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-02-09T22:43:02.709 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: AddFile(Keys): cumulative 0, interval 0 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: Cumulative compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: Interval compaction: 0.00 GB write, 0.68 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: Block cache BinnedLRUCache@0x564b43955090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout: ** File Read Latency Histogram By Level [default] ** 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: using public_addr v2:172.21.15.64:0/0 -> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: starting mon.b rank -1 at public addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] at bind addrs [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(???) e0 preinit fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:43:02.710 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).mds e1 new map 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).mds e1 print_map 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout: e1 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout: legacy client fscid: -1 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout: 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout: No filesystems configured 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-02-09T22:43:02.711 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi089", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi064", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Deploying cephadm binary to smithi089 2024-02-09T22:43:02.712 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Added host smithi089 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.713 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4283823472' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4283823472' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi089", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:02.714 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Saving service mon spec with placement smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c;count:3 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:02.715 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Deploying daemon mon.c on smithi089 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/771599762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.a calling monitor election 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.716 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.c calling monitor election 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: monmap e2: 2 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],c=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: fsmap 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mgrmap e13: x(active, since 51s) 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: overall HEALTH_OK 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:02.717 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: Deploying daemon mon.b on smithi064 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/1459112020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 ceph-mon[36638]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 bash[36619]: aea8794133553767c48cb5a79324f9325b077a82821c997c5b82c73e1d66f940 2024-02-09T22:43:02.718 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:02 smithi064 systemd[1]: Started Ceph mon.b for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:43:07.460 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.457+0000 7fba8bfff700 1 -- 172.21.15.89:0/2217003074 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1273 (secure 0 0 0) 0x7fba7c00fb70 con 0x7fba94196710 2024-02-09T22:43:07.461 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-02-09T22:43:07.461 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":3,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","modified":"2024-02-09T22:43:02.420887Z","created":"2024-02-09T22:41:05.848623Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:3300","nonce":0},{"type":"v1","addr":"172.21.15.3:6789","nonce":0}]},"addr":"172.21.15.3:6789/0","public_addr":"172.21.15.3:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:3300","nonce":0},{"type":"v1","addr":"172.21.15.89:6789","nonce":0}]},"addr":"172.21.15.89:6789/0","public_addr":"172.21.15.89:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:3300","nonce":0},{"type":"v1","addr":"172.21.15.64:6789","nonce":0}]},"addr":"172.21.15.64:6789/0","public_addr":"172.21.15.64:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-02-09T22:43:07.463 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fba80038d20 msgr2=0x7fba8003b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fba80038d20 0x7fba8003b1e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fba8c00b5c0 tx=0x7fba8c009ec0 comp rx=0 tx=0).stop 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba94196710 msgr2=0x7fba9419aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba94196710 0x7fba9419aaa0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fba7c00b4d0 tx=0x7fba7c00b890 comp rx=0 tx=0).stop 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 shutdown_connections 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fba80038d20 0x7fba8003b1e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba941066f0 0x7fba941961d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 --2- 172.21.15.89:0/2217003074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba94196710 0x7fba9419aaa0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.460+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 >> 172.21.15.89:0/2217003074 conn(0x7fba940fc1c0 msgr2=0x7fba941021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:07.464 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.461+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 shutdown_connections 2024-02-09T22:43:07.465 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:07.461+0000 7fba9967b700 1 -- 172.21.15.89:0/2217003074 wait complete. 2024-02-09T22:43:07.465 INFO:teuthology.orchestra.run.smithi089.stderr:dumped monmap epoch 3 2024-02-09T22:43:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:43:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: mon.a calling monitor election 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: mon.c calling monitor election 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/2217003074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: mon.b calling monitor election 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-09T22:43:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: fsmap 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: mgrmap e13: x(active, since 61s) 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: overall HEALTH_OK 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:43:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: mon.a calling monitor election 2024-02-09T22:43:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: mon.c calling monitor election 2024-02-09T22:43:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/2217003074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: mon.b calling monitor election 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-09T22:43:07.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: fsmap 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: mgrmap e13: x(active, since 61s) 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: overall HEALTH_OK 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: mon.a calling monitor election 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: mon.c calling monitor election 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/2217003074' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: mon.b calling monitor election 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: monmap e3: 3 mons at {a=[v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0],b=[v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0],c=[v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: fsmap 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: osdmap e4: 0 total, 0 up, 0 in 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: mgrmap e13: x(active, since 61s) 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: overall HEALTH_OK 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:07.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:08.102 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-02-09T22:43:08.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph config generate-minimal-conf 2024-02-09T22:43:08.354 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:43:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:08 smithi089 ceph-mon[36665]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:08.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:08.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:08.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:08 smithi003 ceph-mon[32678]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:43:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:08 smithi064 ceph-mon[36638]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:10.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 -- 172.21.15.3:0/114384718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d901028c0 msgr2=0x7f3d90102ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/114384718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d901028c0 0x7f3d90102ca0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3d80009910 tx=0x7f3d80009c20 comp rx=0 tx=0).stop 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 -- 172.21.15.3:0/114384718 shutdown_connections 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/114384718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d901028c0 0x7f3d90102ca0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 -- 172.21.15.3:0/114384718 >> 172.21.15.3:0/114384718 conn(0x7f3d900fc110 msgr2=0x7f3d900fe530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 -- 172.21.15.3:0/114384718 shutdown_connections 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 -- 172.21.15.3:0/114384718 wait complete. 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.140+0000 7f3d95fbc700 1 Processor -- start 2024-02-09T22:43:10.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 -- start start 2024-02-09T22:43:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d901028c0 0x7f3d90199510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:10.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 0x7f3d9019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9019e320 0x7f3d9019e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d9019a140 con 0x7f3d90199a50 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901a6090 con 0x7f3d901028c0 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d95fbc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d901a61d0 con 0x7f3d9019e320 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 0x7f3d9019dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 0x7f3d9019dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39500/0 (socket says 172.21.15.3:39500) 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 -- 172.21.15.3:0/257920316 learned_addr learned my addr 172.21.15.3:0/257920316 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8ffff700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9019e320 0x7f3d9019e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8f7fe700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d901028c0 0x7f3d90199510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 -- 172.21.15.3:0/257920316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9019e320 msgr2=0x7f3d9019e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9019e320 0x7f3d9019e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 -- 172.21.15.3:0/257920316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d901028c0 msgr2=0x7f3d90199510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d901028c0 0x7f3d90199510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8effd700 1 -- 172.21.15.3:0/257920316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d800095f0 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.141+0000 7f3d8ffff700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9019e320 0x7f3d9019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.142+0000 7f3d8effd700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 0x7f3d9019dde0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400b780 tx=0x7f3d8400bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.142+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84004230 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.142+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d84004390 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.142+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d901a64b0 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.142+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d84011bf0 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.142+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d901a6a00 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.143+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f3d8401a400 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.143+0000 7f3d8cff9700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3d78039410 0x7f3d7803b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.143+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f3d8404ba10 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.143+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d90063350 con 0x7f3d90199a50 2024-02-09T22:43:10.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.143+0000 7f3d8f7fe700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3d78039410 0x7f3d7803b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:10.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.143+0000 7f3d8f7fe700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3d78039410 0x7f3d7803b8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3d80006010 tx=0x7f3d8000b540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:10.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.148+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3d84011d90 con 0x7f3d90199a50 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:10.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:10.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:10.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.298 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:10.299 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:10.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.354+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f3d9010bb50 con 0x7f3d90199a50 2024-02-09T22:43:10.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.355+0000 7f3d8cff9700 1 -- 172.21.15.3:0/257920316 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+269 (secure 0 0 0) 0x7f3d840151c0 con 0x7f3d90199a50 2024-02-09T22:43:10.358 INFO:teuthology.orchestra.run.smithi003.stdout:# minimal ceph.conf for 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:43:10.358 INFO:teuthology.orchestra.run.smithi003.stdout:[global] 2024-02-09T22:43:10.358 INFO:teuthology.orchestra.run.smithi003.stdout: fsid = 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:43:10.359 INFO:teuthology.orchestra.run.smithi003.stdout: mon_host = [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] 2024-02-09T22:43:10.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3d78039410 msgr2=0x7f3d7803b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:10.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3d78039410 0x7f3d7803b8d0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f3d80006010 tx=0x7f3d8000b540 comp rx=0 tx=0).stop 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 msgr2=0x7f3d9019dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 0x7f3d9019dde0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3d8400b780 tx=0x7f3d8400bb40 comp rx=0 tx=0).stop 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 shutdown_connections 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3d78039410 0x7f3d7803b8d0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3d9019e320 0x7f3d9019e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3d901028c0 0x7f3d90199510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 --2- 172.21.15.3:0/257920316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3d90199a50 0x7f3d9019dde0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.357+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 >> 172.21.15.3:0/257920316 conn(0x7f3d900fc110 msgr2=0x7f3d90107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.358+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 shutdown_connections 2024-02-09T22:43:10.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:10.358+0000 7f3d95fbc700 1 -- 172.21.15.3:0/257920316 wait complete. 2024-02-09T22:43:11.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:10 smithi064 ceph-mon[36638]: Reconfiguring mon.a (unknown last config time)... 2024-02-09T22:43:11.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:10 smithi064 ceph-mon[36638]: Reconfiguring daemon mon.a on smithi003 2024-02-09T22:43:11.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/257920316' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:11.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:10 smithi064 ceph-mon[36638]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:11.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:10 smithi089 ceph-mon[36665]: Reconfiguring mon.a (unknown last config time)... 2024-02-09T22:43:11.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:10 smithi089 ceph-mon[36665]: Reconfiguring daemon mon.a on smithi003 2024-02-09T22:43:11.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/257920316' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:11.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:10 smithi089 ceph-mon[36665]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:11.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:10 smithi003 ceph-mon[32678]: Reconfiguring mon.a (unknown last config time)... 2024-02-09T22:43:11.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:10 smithi003 ceph-mon[32678]: Reconfiguring daemon mon.a on smithi003 2024-02-09T22:43:11.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/257920316' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:11.348 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:10 smithi003 ceph-mon[32678]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:11.373 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-02-09T22:43:11.373 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:43:11.374 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T22:43:11.405 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:43:11.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:43:11.472 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:43:11.473 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T22:43:11.512 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:43:11.512 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:43:11.586 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:43:11.586 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.conf 2024-02-09T22:43:11.621 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:43:11.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:43:11.693 INFO:tasks.cephadm:Adding mgr.x on smithi003 2024-02-09T22:43:11.694 INFO:tasks.cephadm:Adding mgr.y on smithi064 2024-02-09T22:43:11.694 INFO:tasks.cephadm:Adding mgr.z on smithi089 2024-02-09T22:43:11.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch apply mgr '3;smithi003=x;smithi064=y;smithi089=z' 2024-02-09T22:43:11.967 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: Reconfiguring mon.b (monmap changed)... 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: Reconfiguring daemon mon.b on smithi064 2024-02-09T22:43:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:12 smithi089 ceph-mon[36665]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:12.781 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.781 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: Reconfiguring mon.b (monmap changed)... 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: Reconfiguring daemon mon.b on smithi064 2024-02-09T22:43:12.782 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:12 smithi064 ceph-mon[36638]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: Reconfiguring mon.b (monmap changed)... 2024-02-09T22:43:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:12.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:12.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: Reconfiguring daemon mon.b on smithi064 2024-02-09T22:43:12.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:12 smithi003 ceph-mon[32678]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.419+0000 7f4068cbb700 1 -- 172.21.15.89:0/656885353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 msgr2=0x7f40641022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:13.423 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.419+0000 7f4068cbb700 1 --2- 172.21.15.89:0/656885353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641022b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4054009910 tx=0x7f4054009c20 comp rx=0 tx=0).stop 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 -- 172.21.15.89:0/656885353 shutdown_connections 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 --2- 172.21.15.89:0/656885353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641022b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 -- 172.21.15.89:0/656885353 >> 172.21.15.89:0/656885353 conn(0x7f40640fb530 msgr2=0x7f40640fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 -- 172.21.15.89:0/656885353 shutdown_connections 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 -- 172.21.15.89:0/656885353 wait complete. 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 Processor -- start 2024-02-09T22:43:13.424 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.420+0000 7f4068cbb700 1 -- start start 2024-02-09T22:43:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641004b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f406419bfe0 0x7f406419e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 0x7f40641a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:13.426 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40641a13b0 con 0x7f4064101e90 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40641a1520 con 0x7f406419bfe0 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40641a16e0 con 0x7f406419e910 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 0x7f40641a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 0x7f40641a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:38316/0 (socket says 172.21.15.89:38316) 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 -- 172.21.15.89:0/473066234 learned_addr learned my addr 172.21.15.89:0/473066234 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4061d9b700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f406419bfe0 0x7f406419e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f406259c700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641004b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 -- 172.21.15.89:0/473066234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f406419bfe0 msgr2=0x7f406419e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f406419bfe0 0x7f406419e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 -- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 msgr2=0x7f40641004b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641004b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 -- 172.21.15.89:0/473066234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40540095f0 con 0x7f406419e910 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f406259c700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:43:13.427 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4062d9d700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 0x7f40641a0d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ba50 tx=0x7f405c00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c00c9b0 con 0x7f406419e910 2024-02-09T22:43:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f405c014440 con 0x7f406419e910 2024-02-09T22:43:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.421+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40641a19c0 con 0x7f406419e910 2024-02-09T22:43:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.422+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405c0075b0 con 0x7f406419e910 2024-02-09T22:43:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.422+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40641a1f90 con 0x7f406419e910 2024-02-09T22:43:13.428 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.423+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7f405c007710 con 0x7f406419e910 2024-02-09T22:43:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.423+0000 7f40537fe700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f404c039410 0x7f404c03b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.423+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406418f6c0 con 0x7f406419e910 2024-02-09T22:43:13.431 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.423+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f405c04b3a0 con 0x7f406419e910 2024-02-09T22:43:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.423+0000 7f406259c700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f404c039410 0x7f404c03b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.423+0000 7f406259c700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f404c039410 0x7f404c03b8d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4054004690 tx=0x7f40540058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:13.432 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.428+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f405c01b8b0 con 0x7f406419e910 2024-02-09T22:43:13.649 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.646+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi064=y;smithi089=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f406402d8e0 con 0x7f404c039410 2024-02-09T22:43:13.657 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.654+0000 7f40537fe700 1 -- 172.21.15.89:0/473066234 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f406402d8e0 con 0x7f404c039410 2024-02-09T22:43:13.657 INFO:teuthology.orchestra.run.smithi089.stdout:Scheduled mgr update... 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f404c039410 msgr2=0x7f404c03b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f404c039410 0x7f404c03b8d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4054004690 tx=0x7f40540058e0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 msgr2=0x7f40641a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 0x7f40641a0d00 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f405c00ba50 tx=0x7f405c00bd60 comp rx=0 tx=0).stop 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 shutdown_connections 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f404c039410 0x7f404c03b8d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f406419e910 0x7f40641a0d00 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4064101e90 0x7f40641004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 --2- 172.21.15.89:0/473066234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f406419bfe0 0x7f406419e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.656+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 >> 172.21.15.89:0/473066234 conn(0x7f40640fb530 msgr2=0x7f40640fc100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.657+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 shutdown_connections 2024-02-09T22:43:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:43:13.657+0000 7f4068cbb700 1 -- 172.21.15.89:0/473066234 wait complete. 2024-02-09T22:43:14.308 DEBUG:teuthology.orchestra.run.smithi064:mgr.y> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.y.service 2024-02-09T22:43:14.311 DEBUG:teuthology.orchestra.run.smithi089:mgr.z> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.z.service 2024-02-09T22:43:14.314 INFO:tasks.cephadm:Deploying OSDs... 2024-02-09T22:43:14.314 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:43:14.314 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T22:43:14.332 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T22:43:14.333 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_1 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-02-09 22:42:20.350151210 +0000 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-02-09 22:31:33.369731586 +0000 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-02-09 22:31:33.369731586 +0000 2024-02-09T22:43:14.391 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-02-09 22:31:33.369731586 +0000 2024-02-09T22:43:14.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T22:43:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-02-09T22:43:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-02-09T22:43:14.462 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000230578 s, 2.2 MB/s 2024-02-09T22:43:14.463 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T22:43:14.521 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_2 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-02-09 22:42:20.379150854 +0000 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-02-09 22:31:33.722727454 +0000 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-02-09 22:31:33.722727454 +0000 2024-02-09T22:43:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-02-09 22:31:33.722727454 +0000 2024-02-09T22:43:14.578 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T22:43:14.644 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-02-09T22:43:14.644 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-02-09T22:43:14.644 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000208093 s, 2.5 MB/s 2024-02-09T22:43:14.645 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T22:43:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: Reconfiguring mon.c (monmap changed)... 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: Reconfiguring daemon mon.c on smithi089 2024-02-09T22:43:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.702 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_3 2024-02-09T22:43:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: Reconfiguring mon.c (monmap changed)... 2024-02-09T22:43:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: Reconfiguring daemon mon.c on smithi089 2024-02-09T22:43:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-02-09 22:42:20.405150534 +0000 2024-02-09T22:43:14.757 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-02-09 22:31:34.062723475 +0000 2024-02-09T22:43:14.758 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-02-09 22:31:34.062723475 +0000 2024-02-09T22:43:14.758 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-02-09 22:31:34.062723475 +0000 2024-02-09T22:43:14.758 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: Reconfiguring mon.c (monmap changed)... 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: Reconfiguring daemon mon.c on smithi089 2024-02-09T22:43:14.822 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-02-09T22:43:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-02-09T22:43:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000203128 s, 2.5 MB/s 2024-02-09T22:43:14.826 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T22:43:14.882 DEBUG:teuthology.orchestra.run.smithi003:> stat /dev/vg_nvme/lv_4 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-02-09 22:42:20.431150214 +0000 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-02-09 22:31:34.425719227 +0000 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-02-09 22:31:34.425719227 +0000 2024-02-09T22:43:14.938 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-02-09 22:31:34.425719227 +0000 2024-02-09T22:43:14.939 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T22:43:15.003 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records in 2024-02-09T22:43:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:1+0 records out 2024-02-09T22:43:15.004 INFO:teuthology.orchestra.run.smithi003.stderr:512 bytes copied, 0.000197155 s, 2.6 MB/s 2024-02-09T22:43:15.005 DEBUG:teuthology.orchestra.run.smithi003:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T22:43:15.061 DEBUG:teuthology.orchestra.run.smithi064:> set -ex 2024-02-09T22:43:15.061 DEBUG:teuthology.orchestra.run.smithi064:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T22:43:15.080 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T22:43:15.080 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_1 2024-02-09T22:43:15.138 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T22:43:15.138 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:15.138 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-02-09T22:43:15.138 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:15.138 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:15.138 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-09 22:43:01.850145036 +0000 2024-02-09T22:43:15.139 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-09 22:32:02.937771140 +0000 2024-02-09T22:43:15.139 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-09 22:32:02.937771140 +0000 2024-02-09T22:43:15.139 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-09 22:32:02.937771140 +0000 2024-02-09T22:43:15.139 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T22:43:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-09T22:43:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-09T22:43:15.212 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000273338 s, 1.9 MB/s 2024-02-09T22:43:15.213 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T22:43:15.272 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_2 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-09 22:43:01.850145036 +0000 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-09 22:32:03.296763520 +0000 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-09 22:32:03.296763520 +0000 2024-02-09T22:43:15.329 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-09 22:32:03.296763520 +0000 2024-02-09T22:43:15.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T22:43:15.400 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-09T22:43:15.400 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-09T22:43:15.400 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000229667 s, 2.2 MB/s 2024-02-09T22:43:15.401 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T22:43:15.459 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_3 2024-02-09T22:43:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T22:43:15.515 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-09 22:43:01.851145015 +0000 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-09 22:32:03.644756133 +0000 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-09 22:32:03.644756133 +0000 2024-02-09T22:43:15.516 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-09 22:32:03.644756133 +0000 2024-02-09T22:43:15.516 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T22:43:15.582 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:15 smithi064 ceph-mon[36638]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi064=y;smithi089=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:15.583 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:15 smithi064 ceph-mon[36638]: Saving service mgr spec with placement smithi003=x;smithi064=y;smithi089=z;count:3 2024-02-09T22:43:15.583 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:15 smithi064 ceph-mon[36638]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:15.583 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:15 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:15.583 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:15 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:15.583 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:15 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:15.585 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-09T22:43:15.585 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-09T22:43:15.585 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000171594 s, 3.0 MB/s 2024-02-09T22:43:15.586 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T22:43:15.644 DEBUG:teuthology.orchestra.run.smithi064:> stat /dev/vg_nvme/lv_4 2024-02-09T22:43:15.701 INFO:teuthology.orchestra.run.smithi064.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Access: 2024-02-09 22:43:01.851145015 +0000 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Modify: 2024-02-09 22:32:03.983748938 +0000 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout:Change: 2024-02-09 22:32:03.983748938 +0000 2024-02-09T22:43:15.702 INFO:teuthology.orchestra.run.smithi064.stdout: Birth: 2024-02-09 22:32:03.983748938 +0000 2024-02-09T22:43:15.702 DEBUG:teuthology.orchestra.run.smithi064:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T22:43:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:15 smithi089 ceph-mon[36665]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi064=y;smithi089=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:15 smithi089 ceph-mon[36665]: Saving service mgr spec with placement smithi003=x;smithi064=y;smithi089=z;count:3 2024-02-09T22:43:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:15 smithi089 ceph-mon[36665]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:15 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:15 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:15 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records in 2024-02-09T22:43:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:1+0 records out 2024-02-09T22:43:15.772 INFO:teuthology.orchestra.run.smithi064.stderr:512 bytes copied, 0.000234088 s, 2.2 MB/s 2024-02-09T22:43:15.773 DEBUG:teuthology.orchestra.run.smithi064:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T22:43:15.829 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-02-09T22:43:15.829 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-02-09T22:43:15.845 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-02-09T22:43:15.846 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-02-09T22:43:15.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:15 smithi003 ceph-mon[32678]: from='client.24100 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi003=x;smithi064=y;smithi089=z", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:15.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:15 smithi003 ceph-mon[32678]: Saving service mgr spec with placement smithi003=x;smithi064=y;smithi089=z;count:3 2024-02-09T22:43:15.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:15 smithi003 ceph-mon[32678]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:15.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:15 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:15.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:15 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:15.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:15 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-09 22:42:52.845047654 +0000 2024-02-09T22:43:15.901 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-09 22:32:02.018613270 +0000 2024-02-09T22:43:15.902 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-09 22:32:02.018613270 +0000 2024-02-09T22:43:15.902 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-09 22:32:02.018613270 +0000 2024-02-09T22:43:15.902 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-02-09T22:43:15.967 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-09T22:43:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-09T22:43:15.968 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000203289 s, 2.5 MB/s 2024-02-09T22:43:15.969 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-02-09T22:43:16.025 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-02-09T22:43:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-09 22:42:52.845047654 +0000 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-09 22:32:02.350605591 +0000 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-09 22:32:02.350605591 +0000 2024-02-09T22:43:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-09 22:32:02.350605591 +0000 2024-02-09T22:43:16.081 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-02-09T22:43:16.146 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-09T22:43:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-09T22:43:16.147 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000209459 s, 2.4 MB/s 2024-02-09T22:43:16.148 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-02-09T22:43:16.203 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-02-09T22:43:16.257 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-09 22:42:52.845047654 +0000 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-09 22:32:02.694597635 +0000 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-09 22:32:02.694597635 +0000 2024-02-09T22:43:16.258 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-09 22:32:02.694597635 +0000 2024-02-09T22:43:16.258 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-02-09T22:43:16.324 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-09T22:43:16.325 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-09T22:43:16.325 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00017663 s, 2.9 MB/s 2024-02-09T22:43:16.326 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-02-09T22:43:16.381 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-02-09T22:43:16.436 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-02-09T22:43:16.436 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-02-09 22:42:52.846047632 +0000 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-02-09 22:32:03.046589493 +0000 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-02-09 22:32:03.046589493 +0000 2024-02-09T22:43:16.437 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-02-09 22:32:03.046589493 +0000 2024-02-09T22:43:16.437 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-02-09T22:43:16.503 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:16.504 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:16.504 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:16.504 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:16.504 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T22:43:16.504 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:16.504 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:16.505 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-02-09T22:43:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-02-09T22:43:16.506 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000177021 s, 2.9 MB/s 2024-02-09T22:43:16.507 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-02-09T22:43:16.564 INFO:tasks.cephadm:Deploying osd.0 on smithi003 with /dev/vg_nvme/lv_4... 2024-02-09T22:43:16.565 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T22:43:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T22:43:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:16.807 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:17 smithi003 ceph-mon[32678]: Deploying daemon mgr.z on smithi089 2024-02-09T22:43:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:17 smithi003 ceph-mon[32678]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:17.717 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:17 smithi089 ceph-mon[36665]: Deploying daemon mgr.z on smithi089 2024-02-09T22:43:17.717 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:17 smithi089 ceph-mon[36665]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:17 smithi064 ceph-mon[36638]: Deploying daemon mgr.z on smithi089 2024-02-09T22:43:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:17 smithi064 ceph-mon[36638]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:17.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:17 smithi089 systemd[1]: Starting Ceph mgr.z for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:43:18.343 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 podman[38509]: 2024-02-09 22:43:17.992336494 +0000 UTC m=+0.017507350 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:43:18.344 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 podman[38509]: 2024-02-09 22:43:18.094798883 +0000 UTC m=+0.119969732 container create bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-09T22:43:18.344 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 podman[38509]: 2024-02-09 22:43:18.338887048 +0000 UTC m=+0.364057914 container init bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-02-09T22:43:18.344 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 podman[38509]: 2024-02-09 22:43:18.3414748 +0000 UTC m=+0.366645649 container start bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:43:18.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 ceph-mgr[38529]: -- 172.21.15.89:0/3353727569 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ec2b86c1a0 con 0x55ec2b842800 2024-02-09T22:43:18.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 bash[38509]: bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f 2024-02-09T22:43:18.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 systemd[1]: Started Ceph mgr.z for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:43:18.740 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T22:43:18.740 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T22:43:18.741 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-02-09T22:43:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-02-09T22:43:18.741 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0133768 s, 784 MB/s 2024-02-09T22:43:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-02-09T22:43:19.227 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:18.896+0000 7fa884aac200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:43:19.228 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:18.989+0000 7fa884aac200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:43:19.392 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_4 2024-02-09T22:43:19.637 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:43:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T22:43:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:19.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:19.856 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:19.623+0000 7fa884aac200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:43:19.856 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:19.714+0000 7fa884aac200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:19.853+0000 7fa884aac200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:19.857 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:20.227 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:19.947+0000 7fa884aac200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:43:20.227 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:20.029+0000 7fa884aac200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:43:20.576 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-mon[36665]: Deploying daemon mgr.y on smithi064 2024-02-09T22:43:20.576 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:20.287+0000 7fa884aac200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:43:20.576 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:20.394+0000 7fa884aac200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:43:20.577 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:20.478+0000 7fa884aac200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:43:20.577 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:20.573+0000 7fa884aac200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:43:20.626 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:20 smithi064 ceph-mon[36638]: Deploying daemon mgr.y on smithi064 2024-02-09T22:43:20.761 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:20 smithi003 ceph-mon[32678]: Deploying daemon mgr.y on smithi064 2024-02-09T22:43:20.885 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:20 smithi064 ceph-mgr[38020]: -- 172.21.15.64:0/3365523580 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5630763d41a0 con 0x5630763ab000 2024-02-09T22:43:20.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:20.743+0000 7fa884aac200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:43:21.185 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:20 smithi064 bash[38001]: ea86b2fc3cf50a3df681a27ee124d57c6aa3d85a53849eae84fb3c157b6b6af0 2024-02-09T22:43:21.185 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:20 smithi064 systemd[1]: Started Ceph mgr.y for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:43:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 -- 172.21.15.3:0/3358626577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab8100f20 msgr2=0x7ffab8101380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 --2- 172.21.15.3:0/3358626577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab8100f20 0x7ffab8101380 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009990 tx=0x7ffaac009ca0 comp rx=0 tx=0).stop 2024-02-09T22:43:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 -- 172.21.15.3:0/3358626577 shutdown_connections 2024-02-09T22:43:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 --2- 172.21.15.3:0/3358626577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab8102270 0x7ffab8102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 --2- 172.21.15.3:0/3358626577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab81018c0 0x7ffab8109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:21.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 --2- 172.21.15.3:0/3358626577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab8100f20 0x7ffab8101380 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 -- 172.21.15.3:0/3358626577 >> 172.21.15.3:0/3358626577 conn(0x7ffab80fb9b0 msgr2=0x7ffab80fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 -- 172.21.15.3:0/3358626577 shutdown_connections 2024-02-09T22:43:21.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 -- 172.21.15.3:0/3358626577 wait complete. 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.257+0000 7ffabc92f700 1 Processor -- start 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 -- start start 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab8100f20 0x7ffab8197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 0x7ffab8198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab8102270 0x7ffab819f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab819f860 con 0x7ffab81018c0 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab819f9a0 con 0x7ffab8100f20 2024-02-09T22:43:21.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffabc92f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab819fb60 con 0x7ffab8102270 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 0x7ffab8198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 0x7ffab8198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47944/0 (socket says 172.21.15.3:47944) 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 -- 172.21.15.3:0/2868342219 learned_addr learned my addr 172.21.15.3:0/2868342219 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab659c700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab8100f20 0x7ffab8197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab6d9d700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab8102270 0x7ffab819f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 -- 172.21.15.3:0/2868342219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab8102270 msgr2=0x7ffab819f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab8102270 0x7ffab819f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 -- 172.21.15.3:0/2868342219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab8100f20 msgr2=0x7ffab8197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab8100f20 0x7ffab8197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 -- 172.21.15.3:0/2868342219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac0095f0 con 0x7ffab81018c0 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab659c700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab8100f20 0x7ffab8197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffab5d9b700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 0x7ffab8198250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00b820 tx=0x7ffaac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:21.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.258+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac01d070 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.259+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaac005030 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.259+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab819fde0 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.259+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac004cd0 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.259+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab81a0190 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.259+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 13) v1 ==== 45987+0+0 (secure 0 0 0) 0x7ffaac02a450 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.260+0000 7ffaa77fe700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ffaa00391c0 0x7ffaa003b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.260+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7ffaac0051a0 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.260+0000 7ffab659c700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ffaa00391c0 0x7ffaa003b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.260+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab80689e0 con 0x7ffab81018c0 2024-02-09T22:43:21.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.260+0000 7ffab659c700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ffaa00391c0 0x7ffaa003b680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffab81991f0 tx=0x7ffaa8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:21.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.265+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ffaac004e30 con 0x7ffab81018c0 2024-02-09T22:43:21.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:21.480+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7ffab80fdb30 con 0x7ffaa00391c0 2024-02-09T22:43:21.685 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:21.386+0000 7f05a3dec200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:21.493+0000 7f05a3dec200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:43:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:43:21.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:43:21.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:21.889 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:21.750+0000 7fa884aac200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:43:21.889 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:21.889 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.889 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.890 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.890 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:21.890 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:21.890 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:43:21.890 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:43:21.890 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:22.226 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:21.886+0000 7fa884aac200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:43:22.226 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:22.222+0000 7fa884aac200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:43:22.337 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:22.236+0000 7f05a3dec200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:43:22.337 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:22.333+0000 7f05a3dec200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:43:22.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:22.310+0000 7fa884aac200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:43:22.593 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-mon[36638]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:22.593 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.593 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.593 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.593 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:22.489+0000 7f05a3dec200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:43:22.593 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:22.589+0000 7f05a3dec200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:43:22.769 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:22 smithi003 ceph-mon[32678]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:22.770 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.770 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.770 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.935 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:22.675+0000 7f05a3dec200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:43:22.976 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:22.512+0000 7fa884aac200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:43:22.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:22.695+0000 7fa884aac200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:43:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-mon[36665]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.227 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:22.973+0000 7fa884aac200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:43:23.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:22.945+0000 7f05a3dec200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:43:23.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:23.058+0000 7f05a3dec200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:43:23.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:23.146+0000 7f05a3dec200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:43:23.247 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:23.243+0000 7f05a3dec200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:43:23.510 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:23.424+0000 7f05a3dec200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:43:23.510 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:23.229+0000 7fa884aac200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:43:23.510 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:23.462+0000 7fa884aac200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:23.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:23.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-02-09T22:43:23.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:24.511 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:24.250+0000 7fa884aac200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:43:24.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-mon[36665]: Reconfiguring mgr.x (unknown last config time)... 2024-02-09T22:43:24.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-mon[36665]: Reconfiguring daemon mgr.x on smithi003 2024-02-09T22:43:24.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3281976204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1313805-be20-4bd1-a375-4d8c740e1ff2"}]: dispatch 2024-02-09T22:43:24.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3281976204' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1313805-be20-4bd1-a375-4d8c740e1ff2"}]': finished 2024-02-09T22:43:24.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-mon[36665]: osdmap e5: 1 total, 0 up, 1 in 2024-02-09T22:43:24.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:24.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:24 smithi003 ceph-mon[32678]: Reconfiguring mgr.x (unknown last config time)... 2024-02-09T22:43:24.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:24 smithi003 ceph-mon[32678]: Reconfiguring daemon mgr.x on smithi003 2024-02-09T22:43:24.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:24 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3281976204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1313805-be20-4bd1-a375-4d8c740e1ff2"}]: dispatch 2024-02-09T22:43:24.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:24 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3281976204' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1313805-be20-4bd1-a375-4d8c740e1ff2"}]': finished 2024-02-09T22:43:24.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:24 smithi003 ceph-mon[32678]: osdmap e5: 1 total, 0 up, 1 in 2024-02-09T22:43:24.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:24 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:24.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-mon[36638]: Reconfiguring mgr.x (unknown last config time)... 2024-02-09T22:43:24.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-mon[36638]: Reconfiguring daemon mgr.x on smithi003 2024-02-09T22:43:24.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3281976204' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c1313805-be20-4bd1-a375-4d8c740e1ff2"}]: dispatch 2024-02-09T22:43:24.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3281976204' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c1313805-be20-4bd1-a375-4d8c740e1ff2"}]': finished 2024-02-09T22:43:24.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-mon[36638]: osdmap e5: 1 total, 0 up, 1 in 2024-02-09T22:43:24.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:24.931 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:24.447+0000 7f05a3dec200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:43:24.931 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:24.584+0000 7f05a3dec200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:43:24.931 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:24.926+0000 7f05a3dec200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:43:25.121 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:24.810+0000 7fa884aac200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:43:25.121 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:24.908+0000 7fa884aac200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:43:25.121 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:43:25.118+0000 7fa884aac200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:43:25.185 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:25.018+0000 7f05a3dec200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:43:25.515 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:25.218+0000 7f05a3dec200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:43:25.515 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:25.406+0000 7f05a3dec200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1538570995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: Standby manager daemon z started 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-09T22:43:25.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:25 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:43:25.935 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:25.698+0000 7f05a3dec200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1538570995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: Standby manager daemon z started 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-09T22:43:25.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:43:25.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:25.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1538570995' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:43:25.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: Standby manager daemon z started 2024-02-09T22:43:25.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-09T22:43:25.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:43:25.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-09T22:43:25.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:25 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/4283222057' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:43:26.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:26.127+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 14) v1 ==== 91349+0+0 (secure 0 0 0) 0x7ffaac00ed00 con 0x7ffab81018c0 2024-02-09T22:43:26.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:25 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:25.966+0000 7f05a3dec200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:43:26.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:26 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:26.209+0000 7f05a3dec200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:43:27.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:27.039+0000 7f05a3dec200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:43:27.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-mon[36638]: mgrmap e14: x(active, since 79s), standbys: z 2024-02-09T22:43:27.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-09T22:43:27.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-mon[36638]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:27 smithi089 ceph-mon[36665]: mgrmap e14: x(active, since 79s), standbys: z 2024-02-09T22:43:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:27 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-09T22:43:27.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:27 smithi089 ceph-mon[36665]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:27 smithi003 ceph-mon[32678]: mgrmap e14: x(active, since 79s), standbys: z 2024-02-09T22:43:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:27 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-09T22:43:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:27 smithi003 ceph-mon[32678]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:27.935 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:27.629+0000 7f05a3dec200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:43:27.935 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:27.729+0000 7f05a3dec200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:43:28.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:28.141+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 15) v1 ==== 136711+0+0 (secure 0 0 0) 0x7ffaac005550 con 0x7ffab81018c0 2024-02-09T22:43:28.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:28 smithi003 ceph-mon[32678]: Standby manager daemon y started 2024-02-09T22:43:28.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:28 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-09T22:43:28.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:28 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:43:28.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:28 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-09T22:43:28.363 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:28 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:43:28.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:43:27 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:43:27.949+0000 7f05a3dec200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:43:28.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:28 smithi064 ceph-mon[36638]: Standby manager daemon y started 2024-02-09T22:43:28.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:28 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-09T22:43:28.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:28 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:43:28.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:28 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-09T22:43:28.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:28 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:43:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:28 smithi089 ceph-mon[36665]: Standby manager daemon y started 2024-02-09T22:43:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:28 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-09T22:43:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:28 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:43:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:28 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-09T22:43:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:28 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/3193844535' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:43:29.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:29 smithi003 ceph-mon[32678]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-02-09T22:43:29.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T22:43:29.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:29 smithi003 ceph-mon[32678]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:29.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:29 smithi064 ceph-mon[36638]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-02-09T22:43:29.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T22:43:29.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:29 smithi064 ceph-mon[36638]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:29.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:29 smithi089 ceph-mon[36665]: mgrmap e15: x(active, since 81s), standbys: z, y 2024-02-09T22:43:29.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T22:43:29.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:29 smithi089 ceph-mon[36665]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:31.047 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:31.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:31.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:31.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.048 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:30 smithi003 ceph-mon[32678]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:31.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:31.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:31.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:31.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:30 smithi064 ceph-mon[36638]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:31.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:31.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:31.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:30 smithi089 ceph-mon[36665]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T22:43:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T22:43:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-02-09T22:43:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:33 smithi003 ceph-mon[32678]: Deploying daemon osd.0 on smithi003 2024-02-09T22:43:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:33 smithi003 ceph-mon[32678]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:33 smithi089 ceph-mon[36665]: Deploying daemon osd.0 on smithi003 2024-02-09T22:43:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:33 smithi089 ceph-mon[36665]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:33.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:33 smithi064 ceph-mon[36638]: Deploying daemon osd.0 on smithi003 2024-02-09T22:43:33.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:33 smithi064 ceph-mon[36638]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:35.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:35 smithi003 ceph-mon[32678]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:35 smithi064 ceph-mon[36638]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:35 smithi089 ceph-mon[36665]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:36.848 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:36 smithi003 ceph-mon[32678]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:36.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:36 smithi064 ceph-mon[36638]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:36.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:36 smithi089 ceph-mon[36665]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:39.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:39.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:39.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:39.116 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:38 smithi003 ceph-mon[32678]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:39.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:39.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:39.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:39.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:38 smithi064 ceph-mon[36638]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:39.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:39.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:39.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:39.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:38 smithi089 ceph-mon[36665]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:40.834 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:40 smithi003 ceph-mon[32678]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:40.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:40 smithi064 ceph-mon[36638]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:40 smithi089 ceph-mon[36665]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:41.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.435+0000 7ffaa77fe700 1 -- 172.21.15.3:0/2868342219 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ffab80fdb30 con 0x7ffaa00391c0 2024-02-09T22:43:41.438 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 0 on host 'smithi003' 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ffaa00391c0 msgr2=0x7ffaa003b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ffaa00391c0 0x7ffaa003b680 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffab81991f0 tx=0x7ffaa8009260 comp rx=0 tx=0).stop 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 msgr2=0x7ffab8198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 0x7ffab8198250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ffaac00b820 tx=0x7ffaac004730 comp rx=0 tx=0).stop 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 shutdown_connections 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ffaa00391c0 0x7ffaa003b680 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffab8102270 0x7ffab819f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffab8100f20 0x7ffab8197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 --2- 172.21.15.3:0/2868342219 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffab81018c0 0x7ffab8198250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 >> 172.21.15.3:0/2868342219 conn(0x7ffab80fb9b0 msgr2=0x7ffab80fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:41.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 shutdown_connections 2024-02-09T22:43:41.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:43:41.437+0000 7ffabc92f700 1 -- 172.21.15.3:0/2868342219 wait complete. 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.781 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.782 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:41.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:41.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:41.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:43:42.046 DEBUG:teuthology.orchestra.run.smithi003:osd.0> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.0.service 2024-02-09T22:43:42.048 INFO:tasks.cephadm:Deploying osd.1 on smithi064 with /dev/vg_nvme/lv_4... 2024-02-09T22:43:42.048 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T22:43:42.304 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:43:42.682 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:42 smithi064 ceph-mon[36638]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T22:43:42.683 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:42 smithi064 ceph-mon[36638]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:42 smithi003 ceph-mon[32678]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T22:43:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:42 smithi003 ceph-mon[32678]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:42 smithi089 ceph-mon[36665]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-02-09T22:43:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:42 smithi089 ceph-mon[36665]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:43 smithi003 ceph-mon[32678]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T22:43:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:43 smithi003 ceph-mon[32678]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T22:43:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:43 smithi003 ceph-mon[32678]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:43:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:43.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:43:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0[46960]: 2024-02-09T22:43:43.582+0000 7f2d0c13b700 -1 osd.0 0 waiting for initial osdmap 2024-02-09T22:43:43.633 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:43:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0[46960]: 2024-02-09T22:43:43.587+0000 7f2d0872f700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:43:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:43 smithi064 ceph-mon[36638]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T22:43:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:43 smithi064 ceph-mon[36638]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T22:43:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:43 smithi064 ceph-mon[36638]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:43:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:43.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:43 smithi089 ceph-mon[36665]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-02-09T22:43:43.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:43 smithi089 ceph-mon[36665]: osdmap e6: 1 total, 0 up, 1 in 2024-02-09T22:43:43.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:43 smithi089 ceph-mon[36665]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:43:43.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.158 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T22:43:44.159 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T22:43:44.159 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-09T22:43:44.159 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-09T22:43:44.159 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0111478 s, 941 MB/s 2024-02-09T22:43:44.159 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-09T22:43:44.844 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:44 smithi003 ceph-mon[32678]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:43:44.844 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:44 smithi003 ceph-mon[32678]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T22:43:44.844 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:44 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.844 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:44 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.844 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:44 smithi003 ceph-mon[32678]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:44.933 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:44 smithi064 ceph-mon[36638]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:43:44.934 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:44 smithi064 ceph-mon[36638]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T22:43:44.934 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:44 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.934 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:44 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.934 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:44 smithi064 ceph-mon[36638]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:44.964 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_4 2024-02-09T22:43:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:44 smithi089 ceph-mon[36665]: from='osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:43:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:44 smithi089 ceph-mon[36665]: osdmap e7: 1 total, 0 up, 1 in 2024-02-09T22:43:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:44 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:44 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:44 smithi089 ceph-mon[36665]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-02-09T22:43:45.235 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:43:45.653 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] boot 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: osdmap e8: 1 total, 1 up, 1 in 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi003 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi003 to 17093M 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.654 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:45.655 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:45.655 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] boot 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: osdmap e8: 1 total, 1 up, 1 in 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi003 2024-02-09T22:43:45.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi003 to 17093M 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:45.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:43:45.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:43:45.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: osd.0 [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] boot 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: osdmap e8: 1 total, 1 up, 1 in 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi003 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi003 to 17093M 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:43:45.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:43:46.663 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:46 smithi064 ceph-mon[36638]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T22:43:46.663 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:46 smithi064 ceph-mon[36638]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:46.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.864+0000 7f72fd42b700 1 -- 172.21.15.64:0/3577702669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f8102f50 msgr2=0x7f72f81033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:46.868 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.864+0000 7f72fd42b700 1 --2- 172.21.15.64:0/3577702669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f8102f50 0x7f72f81033b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f72ec0099c0 tx=0x7f72ec009cd0 comp rx=0 tx=0).stop 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.864+0000 7f72fd42b700 1 -- 172.21.15.64:0/3577702669 shutdown_connections 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.864+0000 7f72fd42b700 1 --2- 172.21.15.64:0/3577702669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f81038f0 0x7f72f810a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.864+0000 7f72fd42b700 1 --2- 172.21.15.64:0/3577702669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72f8101d50 0x7f72f8102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.864+0000 7f72fd42b700 1 --2- 172.21.15.64:0/3577702669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f8102f50 0x7f72f81033b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 -- 172.21.15.64:0/3577702669 >> 172.21.15.64:0/3577702669 conn(0x7f72f80fd330 msgr2=0x7f72f80ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 -- 172.21.15.64:0/3577702669 shutdown_connections 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 -- 172.21.15.64:0/3577702669 wait complete. 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 Processor -- start 2024-02-09T22:43:46.869 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 -- start start 2024-02-09T22:43:46.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f8101d50 0x7f72f8075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:46.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.865+0000 7f72fd42b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72f8102f50 0x7f72f8074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:46.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72fd42b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 0x7f72f8074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:46.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72fd42b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8074dd0 con 0x7f72f8102f50 2024-02-09T22:43:46.871 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72fd42b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8074f40 con 0x7f72f81038f0 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72fd42b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f81a1d40 con 0x7f72f8101d50 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 0x7f72f8074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 0x7f72f8074650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:56440/0 (socket says 172.21.15.64:56440) 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 -- 172.21.15.64:0/2945750770 learned_addr learned my addr 172.21.15.64:0/2945750770 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f6ffd700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f8101d50 0x7f72f8075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f67fc700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72f8102f50 0x7f72f8074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 -- 172.21.15.64:0/2945750770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f8101d50 msgr2=0x7f72f8075ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f8101d50 0x7f72f8075ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 -- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72f8102f50 msgr2=0x7f72f8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72f8102f50 0x7f72f8074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 -- 172.21.15.64:0/2945750770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ec0095f0 con 0x7f72f81038f0 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72f77fe700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 0x7f72f8074650 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f72f000ef30 tx=0x7f72f000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:46.872 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0018070 con 0x7f72f81038f0 2024-02-09T22:43:46.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72f000f040 con 0x7f72f81038f0 2024-02-09T22:43:46.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f81a1fc0 con 0x7f72f81038f0 2024-02-09T22:43:46.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f000cd50 con 0x7f72f81038f0 2024-02-09T22:43:46.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.866+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f81a23e0 con 0x7f72f81038f0 2024-02-09T22:43:46.873 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.868+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8109190 con 0x7f72f81038f0 2024-02-09T22:43:46.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.869+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 15) v1 ==== 136711+0+0 (secure 0 0 0) 0x7f72f0007500 con 0x7f72f81038f0 2024-02-09T22:43:46.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.869+0000 7f72dffff700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f72e00a20c0 0x7f72e00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:43:46.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.869+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f72f00cbd60 con 0x7f72f81038f0 2024-02-09T22:43:46.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.869+0000 7f72f6ffd700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f72e00a20c0 0x7f72e00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:43:46.874 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.870+0000 7f72f6ffd700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f72e00a20c0 0x7f72e00a4580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72f8102db0 tx=0x7f72e8009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:43:46.877 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:46.874+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f72f0096830 con 0x7f72f81038f0 2024-02-09T22:43:46.878 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:46 smithi089 ceph-mon[36665]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T22:43:46.878 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:46 smithi089 ceph-mon[36665]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:46.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:46 smithi003 ceph-mon[32678]: osdmap e9: 1 total, 1 up, 1 in 2024-02-09T22:43:46.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:46 smithi003 ceph-mon[32678]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:47.089 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:43:47.086+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f72f802d880 con 0x7f72e00a20c0 2024-02-09T22:43:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:47 smithi003 ceph-mon[32678]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:43:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:43:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:47.920 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:47 smithi064 ceph-mon[36638]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:47.920 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:43:47.921 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:43:47.921 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:47 smithi089 ceph-mon[36665]: from='client.24140 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:43:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:43:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:43:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:48.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:48 smithi003 ceph-mon[32678]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:48.930 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:48 smithi064 ceph-mon[36638]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:48.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:48 smithi089 ceph-mon[36665]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:50.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/559676380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]: dispatch 2024-02-09T22:43:50.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:50 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]: dispatch 2024-02-09T22:43:50.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:50 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]': finished 2024-02-09T22:43:50.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:50 smithi003 ceph-mon[32678]: osdmap e10: 2 total, 1 up, 2 in 2024-02-09T22:43:50.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:50 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:43:50.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/559676380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]: dispatch 2024-02-09T22:43:50.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:50 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]: dispatch 2024-02-09T22:43:50.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:50 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]': finished 2024-02-09T22:43:50.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:50 smithi064 ceph-mon[36638]: osdmap e10: 2 total, 1 up, 2 in 2024-02-09T22:43:50.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:50 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:43:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/559676380' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]: dispatch 2024-02-09T22:43:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:50 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]: dispatch 2024-02-09T22:43:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:50 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b4780ed6-656c-4d81-9520-d56cc32e39f9"}]': finished 2024-02-09T22:43:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:50 smithi089 ceph-mon[36665]: osdmap e10: 2 total, 1 up, 2 in 2024-02-09T22:43:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:50 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:43:51.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/3546021909' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:43:51.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:51 smithi003 ceph-mon[32678]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:51.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/3546021909' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:43:51.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:51 smithi064 ceph-mon[36638]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/3546021909' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:43:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:51 smithi089 ceph-mon[36665]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:52 smithi003 ceph-mon[32678]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:52.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:52 smithi064 ceph-mon[36638]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:52 smithi089 ceph-mon[36665]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:55.234 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:55 smithi064 ceph-mon[36638]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:55.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:55 smithi003 ceph-mon[32678]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:55.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:55 smithi089 ceph-mon[36665]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:56.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:56 smithi003 ceph-mon[32678]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:56.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:56 smithi064 ceph-mon[36638]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:56.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:56 smithi089 ceph-mon[36665]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:57.862 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T22:43:57.862 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:57.863 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:57 smithi064 ceph-mon[36638]: Deploying daemon osd.1 on smithi064 2024-02-09T22:43:57.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T22:43:57.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:57.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:57 smithi003 ceph-mon[32678]: Deploying daemon osd.1 on smithi064 2024-02-09T22:43:57.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-02-09T22:43:57.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:43:57.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:57 smithi089 ceph-mon[36665]: Deploying daemon osd.1 on smithi064 2024-02-09T22:43:58.813 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:43:58 smithi064 ceph-mon[36638]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:58.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:43:58 smithi003 ceph-mon[32678]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:43:58.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:43:58 smithi089 ceph-mon[36665]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:00.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:00 smithi003 ceph-mon[32678]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:00.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:00 smithi064 ceph-mon[36638]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:00 smithi089 ceph-mon[36665]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:03 smithi064 ceph-mon[36638]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:03 smithi089 ceph-mon[36665]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:03 smithi003 ceph-mon[32678]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:05.281 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:05 smithi064 ceph-mon[36638]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:05.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:05 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:05 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:05 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:05.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:05 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:05.282 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:05 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:05 smithi003 ceph-mon[32678]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:05 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:05 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:05 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:05 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:05 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:05 smithi089 ceph-mon[36665]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:05 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:05 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:05 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:05 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:05 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:05.606 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.602+0000 7f72dffff700 1 -- 172.21.15.64:0/2945750770 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f72f802d880 con 0x7f72e00a20c0 2024-02-09T22:44:05.606 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 1 on host 'smithi064' 2024-02-09T22:44:05.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f72e00a20c0 msgr2=0x7f72e00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:05.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f72e00a20c0 0x7f72e00a4580 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f72f8102db0 tx=0x7f72e8009260 comp rx=0 tx=0).stop 2024-02-09T22:44:05.608 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 msgr2=0x7f72f8074650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 0x7f72f8074650 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f72f000ef30 tx=0x7f72f000c3a0 comp rx=0 tx=0).stop 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 shutdown_connections 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f72e00a20c0 0x7f72e00a4580 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f72f8101d50 0x7f72f8075ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f72f81038f0 0x7f72f8074650 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 --2- 172.21.15.64:0/2945750770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f72f8102f50 0x7f72f8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 >> 172.21.15.64:0/2945750770 conn(0x7f72f80fd330 msgr2=0x7f72f81062d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 shutdown_connections 2024-02-09T22:44:05.609 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:05.605+0000 7f72fd42b700 1 -- 172.21.15.64:0/2945750770 wait complete. 2024-02-09T22:44:06.214 DEBUG:teuthology.orchestra.run.smithi064:osd.1> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.1.service 2024-02-09T22:44:06.216 INFO:tasks.cephadm:Deploying osd.2 on smithi089 with /dev/vg_nvme/lv_4... 2024-02-09T22:44:06.216 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-02-09T22:44:06.457 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:44:06.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:06.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:06.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:06.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:06 smithi089 ceph-mon[36665]: from='osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T22:44:06.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:06 smithi089 ceph-mon[36665]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T22:44:06.860 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:06 smithi089 ceph-mon[36665]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:06 smithi003 ceph-mon[32678]: from='osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T22:44:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:06 smithi003 ceph-mon[32678]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T22:44:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:06 smithi003 ceph-mon[32678]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:06.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:06.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:06.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:06.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:06 smithi064 ceph-mon[36638]: from='osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T22:44:06.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:06 smithi064 ceph-mon[36638]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-02-09T22:44:06.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:06 smithi064 ceph-mon[36638]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:07.656 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:07 smithi089 ceph-mon[36665]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T22:44:07.657 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:07 smithi089 ceph-mon[36665]: from='osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:44:07.657 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:07 smithi089 ceph-mon[36665]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T22:44:07.657 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:07.657 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:07 smithi089 ceph-mon[36665]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:44:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:07 smithi003 ceph-mon[32678]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T22:44:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:07 smithi003 ceph-mon[32678]: from='osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:44:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:07 smithi003 ceph-mon[32678]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T22:44:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:07 smithi003 ceph-mon[32678]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:44:07.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-mon[36638]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-02-09T22:44:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-mon[36638]: from='osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:44:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-mon[36638]: osdmap e11: 2 total, 1 up, 2 in 2024-02-09T22:44:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:07.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-mon[36638]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:44:07.936 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1[41263]: 2024-02-09T22:44:07.608+0000 7f0730513700 -1 osd.1 0 waiting for initial osdmap 2024-02-09T22:44:07.936 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1[41263]: 2024-02-09T22:44:07.613+0000 7f072cb07700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:44:08.192 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-02-09T22:44:08.192 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-02-09T22:44:08.192 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-09T22:44:08.192 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-09T22:44:08.192 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114992 s, 912 MB/s 2024-02-09T22:44:08.192 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-09T22:44:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:08 smithi064 ceph-mon[36638]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:44:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:08 smithi064 ceph-mon[36638]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T22:44:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:08 smithi064 ceph-mon[36638]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:08.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_4 2024-02-09T22:44:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:08 smithi003 ceph-mon[32678]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:44:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:08 smithi003 ceph-mon[32678]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T22:44:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:08 smithi003 ceph-mon[32678]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:08 smithi089 ceph-mon[36665]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:44:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:08 smithi089 ceph-mon[36665]: osdmap e12: 2 total, 1 up, 2 in 2024-02-09T22:44:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:08 smithi089 ceph-mon[36665]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-02-09T22:44:08.980 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:44:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] boot 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: osdmap e13: 2 total, 2 up, 2 in 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi064 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi064 to 17080M 2024-02-09T22:44:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:09.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:09.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] boot 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: osdmap e13: 2 total, 2 up, 2 in 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi064 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:09.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi064 to 17080M 2024-02-09T22:44:09.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:09.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:09.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: osd.1 [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] boot 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: osdmap e13: 2 total, 2 up, 2 in 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi064 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi064 to 17080M 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:10.340 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 -- 172.21.15.89:0/1087520802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 msgr2=0x7f6a2c104bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 --2- 172.21.15.89:0/1087520802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c104bd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6a20009990 tx=0x7f6a20009ca0 comp rx=0 tx=0).stop 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 -- 172.21.15.89:0/1087520802 shutdown_connections 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 --2- 172.21.15.89:0/1087520802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c104bd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 --2- 172.21.15.89:0/1087520802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a2c107a40 0x7f6a2c109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 --2- 172.21.15.89:0/1087520802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 0x7f6a2c107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.334+0000 7f6a32419700 1 -- 172.21.15.89:0/1087520802 >> 172.21.15.89:0/1087520802 conn(0x7f6a2c0fc210 msgr2=0x7f6a2c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.335+0000 7f6a32419700 1 -- 172.21.15.89:0/1087520802 shutdown_connections 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.335+0000 7f6a32419700 1 -- 172.21.15.89:0/1087520802 wait complete. 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.335+0000 7f6a32419700 1 Processor -- start 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.335+0000 7f6a32419700 1 -- start start 2024-02-09T22:44:10.341 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 0x7f6a2c198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a2c107a40 0x7f6a2c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c19f770 con 0x7f6a2c105110 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c19f8b0 con 0x7f6a2c107a40 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2c19fa70 con 0x7f6a2c1027e0 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c197c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:36592/0 (socket says 172.21.15.89:36592) 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 -- 172.21.15.89:0/453085401 learned_addr learned my addr 172.21.15.89:0/453085401 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a309b6700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a2c107a40 0x7f6a2c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2b7fe700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 0x7f6a2c198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 -- 172.21.15.89:0/453085401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a2c107a40 msgr2=0x7f6a2c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a2c107a40 0x7f6a2c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 -- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 msgr2=0x7f6a2c198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:10.342 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 0x7f6a2c198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 -- 172.21.15.89:0/453085401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200095f0 con 0x7f6a2c1027e0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2b7fe700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 0x7f6a2c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a2bfff700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c197c20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000c00 tx=0x7f6a2000f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a2001d070 con 0x7f6a2c1027e0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6a2000fb90 con 0x7f6a2c1027e0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2c19fcf0 con 0x7f6a2c1027e0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a20017650 con 0x7f6a2c1027e0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.336+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2c1a0160 con 0x7f6a2c1027e0 2024-02-09T22:44:10.343 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.338+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c0fdd90 con 0x7f6a2c1027e0 2024-02-09T22:44:10.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.338+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 15) v1 ==== 136711+0+0 (secure 0 0 0) 0x7f6a2000fd00 con 0x7f6a2c1027e0 2024-02-09T22:44:10.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.339+0000 7f6a297fa700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6a140a2140 0x7f6a140a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:10.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.339+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f6a200cd8b0 con 0x7f6a2c1027e0 2024-02-09T22:44:10.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.339+0000 7f6a2b7fe700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6a140a2140 0x7f6a140a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:10.344 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.340+0000 7f6a2b7fe700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6a140a2140 0x7f6a140a4600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c005fd0 tx=0x7f6a1c005e30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:44:10.346 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.344+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6a20098160 con 0x7f6a2c1027e0 2024-02-09T22:44:10.548 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:10.545+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6a2c02d7e0 con 0x7f6a140a2140 2024-02-09T22:44:11.315 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:11 smithi089 ceph-mon[36665]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T22:44:11.315 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:11 smithi089 ceph-mon[36665]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:11.315 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:11 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:44:11.315 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:11 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:44:11.315 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:11 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:11 smithi003 ceph-mon[32678]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T22:44:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:11 smithi003 ceph-mon[32678]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:11 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:44:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:11 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:44:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:11 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:11.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:11 smithi064 ceph-mon[36638]: osdmap e14: 2 total, 2 up, 2 in 2024-02-09T22:44:11.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:11 smithi064 ceph-mon[36638]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:11.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:11 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:44:11.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:11 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:44:11.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:11 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:12.195 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:12 smithi089 ceph-mon[36665]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:44:12.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:12 smithi003 ceph-mon[32678]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:44:12.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:12 smithi064 ceph-mon[36638]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:44:13.341 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:13 smithi089 ceph-mon[36665]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:13.342 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:13 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/822236633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]: dispatch 2024-02-09T22:44:13.342 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:13 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]: dispatch 2024-02-09T22:44:13.342 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:13 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]': finished 2024-02-09T22:44:13.342 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:13 smithi089 ceph-mon[36665]: osdmap e15: 3 total, 2 up, 3 in 2024-02-09T22:44:13.342 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:13.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:13 smithi003 ceph-mon[32678]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:13.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:13 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/822236633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]: dispatch 2024-02-09T22:44:13.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:13 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]: dispatch 2024-02-09T22:44:13.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:13 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]': finished 2024-02-09T22:44:13.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:13 smithi003 ceph-mon[32678]: osdmap e15: 3 total, 2 up, 3 in 2024-02-09T22:44:13.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:13.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:13 smithi064 ceph-mon[36638]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:13.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:13 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/822236633' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]: dispatch 2024-02-09T22:44:13.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:13 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]: dispatch 2024-02-09T22:44:13.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:13 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d1c14346-6703-4597-8214-860019453569"}]': finished 2024-02-09T22:44:13.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:13 smithi064 ceph-mon[36638]: osdmap e15: 3 total, 2 up, 3 in 2024-02-09T22:44:13.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:14.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:14 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/2695045025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:44:14.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:14 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/2695045025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:44:14.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:14 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/2695045025' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:44:15.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:15 smithi003 ceph-mon[32678]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:15.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:15 smithi064 ceph-mon[36638]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:15.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:15 smithi089 ceph-mon[36665]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:16.874 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:16 smithi089 ceph-mon[36665]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:16.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:16 smithi003 ceph-mon[32678]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:16.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:16 smithi064 ceph-mon[36638]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:18 smithi089 ceph-mon[36665]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:18.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:18 smithi003 ceph-mon[32678]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:18.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:18 smithi064 ceph-mon[36638]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:20 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T22:44:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:20 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:20.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:20 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T22:44:20.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:20 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:20 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-02-09T22:44:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:20 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:21.552 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:21 smithi089 ceph-mon[36665]: Deploying daemon osd.2 on smithi089 2024-02-09T22:44:21.552 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:21 smithi089 ceph-mon[36665]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:21 smithi003 ceph-mon[32678]: Deploying daemon osd.2 on smithi089 2024-02-09T22:44:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:21 smithi003 ceph-mon[32678]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:21.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:21 smithi064 ceph-mon[36638]: Deploying daemon osd.2 on smithi089 2024-02-09T22:44:21.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:21 smithi064 ceph-mon[36638]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:22.851 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:22 smithi089 ceph-mon[36665]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:22.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:22 smithi003 ceph-mon[32678]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:22.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:22 smithi064 ceph-mon[36638]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:25.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:25 smithi089 ceph-mon[36665]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:25 smithi003 ceph-mon[32678]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:25 smithi064 ceph-mon[36638]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:26.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:26.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:26.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:27.036 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:27 smithi089 ceph-mon[36665]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:27 smithi003 ceph-mon[32678]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:27.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:27 smithi064 ceph-mon[36638]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:28.720 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.714+0000 7f6a297fa700 1 -- 172.21.15.89:0/453085401 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6a2c02d7e0 con 0x7f6a140a2140 2024-02-09T22:44:28.720 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 2 on host 'smithi089' 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6a140a2140 msgr2=0x7f6a140a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6a140a2140 0x7f6a140a4600 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6a1c005fd0 tx=0x7f6a1c005e30 comp rx=0 tx=0).stop 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 msgr2=0x7f6a2c197c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c197c20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6a20000c00 tx=0x7f6a2000f690 comp rx=0 tx=0).stop 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 shutdown_connections 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6a140a2140 0x7f6a140a4600 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6a2c1027e0 0x7f6a2c197c20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6a2c107a40 0x7f6a2c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:28.721 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 --2- 172.21.15.89:0/453085401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6a2c105110 0x7f6a2c198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:28.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 >> 172.21.15.89:0/453085401 conn(0x7f6a2c0fc210 msgr2=0x7f6a2c1030e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:44:28.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 shutdown_connections 2024-02-09T22:44:28.722 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:44:28.716+0000 7f6a32419700 1 -- 172.21.15.89:0/453085401 wait complete. 2024-02-09T22:44:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:28.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:28.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:28.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:29.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:29.366 DEBUG:teuthology.orchestra.run.smithi089:osd.2> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.2.service 2024-02-09T22:44:29.368 INFO:tasks.cephadm:Deploying osd.3 on smithi003 with /dev/vg_nvme/lv_3... 2024-02-09T22:44:29.368 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T22:44:29.633 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:44:30.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:30 smithi089 ceph-mon[36665]: from='osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T22:44:30.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:30 smithi089 ceph-mon[36665]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T22:44:30.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:30 smithi003 ceph-mon[32678]: from='osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T22:44:30.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:30 smithi003 ceph-mon[32678]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T22:44:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:30 smithi064 ceph-mon[36638]: from='osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T22:44:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:30 smithi064 ceph-mon[36638]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-02-09T22:44:31.225 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:31 smithi003 ceph-mon[32678]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T22:44:31.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:31 smithi003 ceph-mon[32678]: from='osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:44:31.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:31 smithi003 ceph-mon[32678]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T22:44:31.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:31 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:31.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:31 smithi003 ceph-mon[32678]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:44:31.226 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:31 smithi003 ceph-mon[32678]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-mon[36665]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T22:44:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-mon[36665]: from='osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:44:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-mon[36665]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T22:44:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-mon[36665]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:44:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-mon[36665]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:31.228 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2[41795]: 2024-02-09T22:44:31.036+0000 7fb22f7a4700 -1 osd.2 0 waiting for initial osdmap 2024-02-09T22:44:31.228 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2[41795]: 2024-02-09T22:44:31.040+0000 7fb22bd98700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:44:31.427 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:31 smithi064 ceph-mon[36638]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-02-09T22:44:31.427 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:31 smithi064 ceph-mon[36638]: from='osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:44:31.427 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:31 smithi064 ceph-mon[36638]: osdmap e16: 3 total, 2 up, 3 in 2024-02-09T22:44:31.427 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:31 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:31.427 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:31 smithi064 ceph-mon[36638]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:44:31.427 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:31 smithi064 ceph-mon[36638]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:31.592 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T22:44:31.592 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T22:44:31.592 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-02-09T22:44:31.592 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-02-09T22:44:31.592 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122002 s, 859 MB/s 2024-02-09T22:44:31.592 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-02-09T22:44:32.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:32 smithi089 ceph-mon[36665]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:44:32.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:32 smithi089 ceph-mon[36665]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T22:44:32.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:32.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:32 smithi003 ceph-mon[32678]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:44:32.237 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:32 smithi003 ceph-mon[32678]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T22:44:32.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:32.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:32.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_3 2024-02-09T22:44:32.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:32 smithi064 ceph-mon[36638]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:44:32.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:32 smithi064 ceph-mon[36638]: osdmap e17: 3 total, 2 up, 3 in 2024-02-09T22:44:32.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:32.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:32.514 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:44:33.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] boot 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: osdmap e18: 3 total, 3 up, 3 in 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi089 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi089 to 17080M 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:33.302 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:33.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] boot 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: osdmap e18: 3 total, 3 up, 3 in 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:33.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi089 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi089 to 17080M 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:33.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-09T22:44:33.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: osd.2 [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] boot 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: osdmap e18: 3 total, 3 up, 3 in 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi089 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi089 to 17080M 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-02-09T22:44:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-02-09T22:44:34.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 -- 172.21.15.3:0/1484240738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c101e40 msgr2=0x7f579c1022a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:34.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 --2- 172.21.15.3:0/1484240738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c101e40 0x7f579c1022a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f57900099c0 tx=0x7f5790009cd0 comp rx=0 tx=0).stop 2024-02-09T22:44:34.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 -- 172.21.15.3:0/1484240738 shutdown_connections 2024-02-09T22:44:34.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 --2- 172.21.15.3:0/1484240738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 0x7f579c101060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:34.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 --2- 172.21.15.3:0/1484240738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c1027e0 0x7f579c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 --2- 172.21.15.3:0/1484240738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c101e40 0x7f579c1022a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 -- 172.21.15.3:0/1484240738 >> 172.21.15.3:0/1484240738 conn(0x7f579c0fc1e0 msgr2=0x7f579c0fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 -- 172.21.15.3:0/1484240738 shutdown_connections 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.071+0000 7f57a2745700 1 -- 172.21.15.3:0/1484240738 wait complete. 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 Processor -- start 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 -- start start 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 0x7f579c06ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 0x7f579c19c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 0x7f579c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c19f7d0 con 0x7f579c1027e0 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c19f940 con 0x7f579c101e40 2024-02-09T22:44:34.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a2745700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579c19fb00 con 0x7f579c100c40 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f579b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 0x7f579c19c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a0ce2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 0x7f579c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f579bfff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 0x7f579c06ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a0ce2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 0x7f579c19f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54388/0 (socket says 172.21.15.3:54388) 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f579b7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 0x7f579c19c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:38652/0 (socket says 172.21.15.3:38652) 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.072+0000 7f57a0ce2700 1 -- 172.21.15.3:0/2782878062 learned_addr learned my addr 172.21.15.3:0/2782878062 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a0ce2700 1 -- 172.21.15.3:0/2782878062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 msgr2=0x7f579c06ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a0ce2700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 0x7f579c06ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a0ce2700 1 -- 172.21.15.3:0/2782878062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 msgr2=0x7f579c19c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a0ce2700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 0x7f579c19c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:34.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a0ce2700 1 -- 172.21.15.3:0/2782878062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57900095f0 con 0x7f579c1027e0 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f579b7fe700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 0x7f579c19c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f579bfff700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 0x7f579c06ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a0ce2700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 0x7f579c19f030 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f579400ef30 tx=0x7f579400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5794018070 con 0x7f579c1027e0 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f579400f040 con 0x7f579c1027e0 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579c19fd80 con 0x7f579c1027e0 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f579400cd50 con 0x7f579c1027e0 2024-02-09T22:44:34.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.073+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f579c1a01f0 con 0x7f579c1027e0 2024-02-09T22:44:34.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.074+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579c108000 con 0x7f579c1027e0 2024-02-09T22:44:34.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.075+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 15) v1 ==== 136711+0+0 (secure 0 0 0) 0x7f5794007500 con 0x7f579c1027e0 2024-02-09T22:44:34.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.076+0000 7f57997fa700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f57840a2140 0x7f57840a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:34.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.076+0000 7f579bfff700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f57840a2140 0x7f57840a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:34.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.076+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f57940cc2a0 con 0x7f579c1027e0 2024-02-09T22:44:34.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.077+0000 7f579bfff700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f57840a2140 0x7f57840a4600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f579c101ca0 tx=0x7f578c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:44:34.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.081+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f57940967b0 con 0x7f579c1027e0 2024-02-09T22:44:34.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:34 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-09T22:44:34.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:34 smithi003 ceph-mon[32678]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T22:44:34.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:34 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T22:44:34.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:34.304+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f579c02da30 con 0x7f57840a2140 2024-02-09T22:44:34.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:34 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-09T22:44:34.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:34 smithi064 ceph-mon[36638]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T22:44:34.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:34 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T22:44:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:34 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-02-09T22:44:34.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:34 smithi089 ceph-mon[36665]: osdmap e19: 3 total, 3 up, 3 in 2024-02-09T22:44:34.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:34 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T22:44:34.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51740]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T22:44:34.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51740]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:34.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51740]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:34.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51740]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:34.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51743]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T22:44:34.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51743]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:34.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51743]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:34.940 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 22:44:34 smithi003 sudo[51743]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:35.258 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:35.259 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:35.435 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44988]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T22:44:35.435 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44988]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:35.436 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44988]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:35.436 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44988]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-02-09T22:44:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: osdmap e20: 3 total, 3 up, 3 in 2024-02-09T22:44:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:44:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:44:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:44:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:35.935 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44991]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T22:44:35.935 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44991]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:35.935 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44991]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:35.936 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 22:44:35 smithi064 sudo[44991]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:36.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:36.083+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f5794096b80 con 0x7f579c1027e0 2024-02-09T22:44:36.143 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:36 smithi089 ceph-mon[36665]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T22:44:36.143 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:36 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:36.144 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:35 smithi089 sudo[45889]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T22:44:36.144 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:35 smithi089 sudo[45889]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:36.144 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:35 smithi089 sudo[45889]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:36.144 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:35 smithi089 sudo[45889]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:36.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 ceph-mon[32678]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T22:44:36.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:36.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:36 smithi064 ceph-mon[36638]: osdmap e21: 3 total, 3 up, 3 in 2024-02-09T22:44:36.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:36 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:36.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45892]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T22:44:36.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45892]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:36.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45892]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:36.477 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45892]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51847]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T22:44:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51847]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51847]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51847]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:37.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51860]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T22:44:37.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51860]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:37.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51860]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:37.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:36 smithi003 sudo[51860]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45895]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45895]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45895]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:36 smithi089 sudo[45895]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 sudo[45898]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 sudo[45898]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 sudo[45898]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:37.187 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 sudo[45898]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:37.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-02-09T22:44:37.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:37.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:37.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:37.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-02-09T22:44:37.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:37.549 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:37.550 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:37.635 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-02-09T22:44:37.635 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:37.635 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:37.635 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:37.635 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:37.636 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:37.636 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:37.636 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:37.636 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:37.636 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:37.636 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:37.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 sudo[44994]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-02-09T22:44:37.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 sudo[44994]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:37.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 sudo[44994]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:37 smithi064 sudo[44994]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 sudo[44997]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 sudo[44997]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 sudo[44997]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 sudo[44997]: pam_unix(sudo:session): session closed for user root 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3315368215' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7129351-e0c4-4563-8e98-d818d30d891a"}]: dispatch 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3315368215' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7129351-e0c4-4563-8e98-d818d30d891a"}]': finished 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: osdmap e22: 4 total, 3 up, 4 in 2024-02-09T22:44:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2724891916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:44:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:38 smithi064 ceph-mon[36638]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3315368215' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7129351-e0c4-4563-8e98-d818d30d891a"}]: dispatch 2024-02-09T22:44:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3315368215' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7129351-e0c4-4563-8e98-d818d30d891a"}]': finished 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: osdmap e22: 4 total, 3 up, 4 in 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2724891916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:44:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:38 smithi089 ceph-mon[36665]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3315368215' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c7129351-e0c4-4563-8e98-d818d30d891a"}]: dispatch 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3315368215' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c7129351-e0c4-4563-8e98-d818d30d891a"}]': finished 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: osdmap e22: 4 total, 3 up, 4 in 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:44:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:44:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:44:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2724891916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:44:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:38 smithi003 ceph-mon[32678]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-02-09T22:44:39.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:39 smithi089 ceph-mon[36665]: pgmap v88: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:39.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:39 smithi003 ceph-mon[32678]: pgmap v88: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:39.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:39 smithi064 ceph-mon[36638]: pgmap v88: 1 pgs: 1 active+clean; 449 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:40.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:40 smithi003 ceph-mon[32678]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.4 KiB/s rd, 337 KiB/s wr, 23 op/s 2024-02-09T22:44:40.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:40 smithi064 ceph-mon[36638]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.4 KiB/s rd, 337 KiB/s wr, 23 op/s 2024-02-09T22:44:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:40 smithi089 ceph-mon[36665]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 7.4 KiB/s rd, 337 KiB/s wr, 23 op/s 2024-02-09T22:44:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:42 smithi003 ceph-mon[32678]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 246 KiB/s wr, 16 op/s 2024-02-09T22:44:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:42 smithi064 ceph-mon[36638]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 246 KiB/s wr, 16 op/s 2024-02-09T22:44:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:42 smithi089 ceph-mon[36665]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 5.4 KiB/s rd, 246 KiB/s wr, 16 op/s 2024-02-09T22:44:45.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:45 smithi003 ceph-mon[32678]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 194 KiB/s wr, 13 op/s 2024-02-09T22:44:45.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T22:44:45.316 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:45.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:45 smithi064 ceph-mon[36638]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 194 KiB/s wr, 13 op/s 2024-02-09T22:44:45.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T22:44:45.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:45 smithi089 ceph-mon[36665]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.2 KiB/s rd, 194 KiB/s wr, 13 op/s 2024-02-09T22:44:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-02-09T22:44:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:46.256 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:46 smithi003 ceph-mon[32678]: Deploying daemon osd.3 on smithi003 2024-02-09T22:44:46.379 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:46 smithi089 ceph-mon[36665]: Deploying daemon osd.3 on smithi003 2024-02-09T22:44:46.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:46 smithi064 ceph-mon[36638]: Deploying daemon osd.3 on smithi003 2024-02-09T22:44:47.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:47 smithi003 ceph-mon[32678]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-02-09T22:44:47.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:47 smithi064 ceph-mon[36638]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-02-09T22:44:47.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:47 smithi089 ceph-mon[36665]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 4.0 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-02-09T22:44:48.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:48 smithi003 ceph-mon[32678]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 162 KiB/s wr, 11 op/s 2024-02-09T22:44:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:48 smithi064 ceph-mon[36638]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 162 KiB/s wr, 11 op/s 2024-02-09T22:44:48.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:48 smithi089 ceph-mon[36665]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.5 KiB/s rd, 162 KiB/s wr, 11 op/s 2024-02-09T22:44:51.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:51.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:51.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:51.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:51 smithi003 ceph-mon[32678]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-02-09T22:44:51.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:51.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:51.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:51.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:51 smithi064 ceph-mon[36638]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-02-09T22:44:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:51 smithi089 ceph-mon[36665]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-02-09T22:44:52.722 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:52 smithi064 ceph-mon[36638]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:52.750 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:52 smithi003 ceph-mon[32678]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:52 smithi089 ceph-mon[36665]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:53.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.960+0000 7f57997fa700 1 -- 172.21.15.3:0/2782878062 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f579c02da30 con 0x7f57840a2140 2024-02-09T22:44:53.964 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 3 on host 'smithi003' 2024-02-09T22:44:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f57840a2140 msgr2=0x7f57840a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f57840a2140 0x7f57840a4600 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f579c101ca0 tx=0x7f578c009260 comp rx=0 tx=0).stop 2024-02-09T22:44:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 msgr2=0x7f579c19f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 0x7f579c19f030 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f579400ef30 tx=0x7f579400c3a0 comp rx=0 tx=0).stop 2024-02-09T22:44:53.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 shutdown_connections 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f57840a2140 0x7f57840a4600 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f579c100c40 0x7f579c06ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f579c1027e0 0x7f579c19f030 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 --2- 172.21.15.3:0/2782878062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f579c101e40 0x7f579c19c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 >> 172.21.15.3:0/2782878062 conn(0x7f579c0fc1e0 msgr2=0x7f579c105140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 shutdown_connections 2024-02-09T22:44:53.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:44:53.962+0000 7f57a2745700 1 -- 172.21.15.3:0/2782878062 wait complete. 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.276 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.277 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:54 smithi003 ceph-mon[32678]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T22:44:54.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:54 smithi064 ceph-mon[36638]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T22:44:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:54 smithi089 ceph-mon[36665]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-02-09T22:44:54.627 DEBUG:teuthology.orchestra.run.smithi003:osd.3> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.3.service 2024-02-09T22:44:54.630 INFO:tasks.cephadm:Deploying osd.4 on smithi064 with /dev/vg_nvme/lv_3... 2024-02-09T22:44:54.630 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T22:44:54.888 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:44:55.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:55 smithi003 ceph-mon[32678]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:55.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:55 smithi003 ceph-mon[32678]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T22:44:55.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:55 smithi003 ceph-mon[32678]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T22:44:55.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:55 smithi003 ceph-mon[32678]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:44:55.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:55 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:55.398 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:55 smithi064 ceph-mon[36638]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:55.398 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:55 smithi064 ceph-mon[36638]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T22:44:55.398 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:55 smithi064 ceph-mon[36638]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T22:44:55.398 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:55 smithi064 ceph-mon[36638]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:44:55.398 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:55 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:55.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:55 smithi089 ceph-mon[36665]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:55.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:55 smithi089 ceph-mon[36665]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-02-09T22:44:55.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:55 smithi089 ceph-mon[36665]: osdmap e23: 4 total, 3 up, 4 in 2024-02-09T22:44:55.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:55 smithi089 ceph-mon[36665]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:44:55.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:55 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.021 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T22:44:57.021 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T22:44:57.021 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-09T22:44:57.021 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-09T22:44:57.021 INFO:teuthology.orchestra.run.smithi064.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110993 s, 945 MB/s 2024-02-09T22:44:57.021 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-09T22:44:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-mon[32678]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:44:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-mon[32678]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T22:44:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-mon[32678]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-mon[32678]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' 2024-02-09T22:44:57.132 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T22:44:56.787+0000 7fce360e7700 -1 osd.3 0 waiting for initial osdmap 2024-02-09T22:44:57.132 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 22:44:56 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T22:44:56.798+0000 7fce2e6d3700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:44:57.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:56 smithi064 ceph-mon[36638]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:44:57.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:56 smithi064 ceph-mon[36638]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T22:44:57.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:56 smithi064 ceph-mon[36638]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:57.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:56 smithi064 ceph-mon[36638]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' 2024-02-09T22:44:57.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:56 smithi089 ceph-mon[36665]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:44:57.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:56 smithi089 ceph-mon[36665]: osdmap e24: 4 total, 3 up, 4 in 2024-02-09T22:44:57.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:57.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:56 smithi089 ceph-mon[36665]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-02-09T22:44:57.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:56 smithi089 ceph-mon[36665]: from='osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347]' entity='osd.3' 2024-02-09T22:44:57.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_3 2024-02-09T22:44:57.849 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:44:58.183 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:44:58.183 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:44:58.183 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] boot 2024-02-09T22:44:58.183 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: osdmap e25: 4 total, 4 up, 4 in 2024-02-09T22:44:58.183 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:58.183 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:58.184 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:44:58.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:44:58.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] boot 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: osdmap e25: 4 total, 4 up, 4 in 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:58.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:44:58.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:44:58.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: osd.3 [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] boot 2024-02-09T22:44:58.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: osdmap e25: 4 total, 4 up, 4 in 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:44:58.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:44:59.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:58 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi003 2024-02-09T22:44:59.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:58 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi003 to 8546M 2024-02-09T22:44:59.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:58 smithi064 ceph-mon[36638]: osdmap e26: 4 total, 4 up, 4 in 2024-02-09T22:44:59.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:58 smithi064 ceph-mon[36638]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:44:59.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:58 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi003 2024-02-09T22:44:59.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:58 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi003 to 8546M 2024-02-09T22:44:59.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:58 smithi089 ceph-mon[36665]: osdmap e26: 4 total, 4 up, 4 in 2024-02-09T22:44:59.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:58 smithi089 ceph-mon[36665]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:44:59.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:58 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi003 2024-02-09T22:44:59.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:58 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi003 to 8546M 2024-02-09T22:44:59.284 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:58 smithi003 ceph-mon[32678]: osdmap e26: 4 total, 4 up, 4 in 2024-02-09T22:44:59.285 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:58 smithi003 ceph-mon[32678]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 280 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 -- 172.21.15.64:0/2033433172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 msgr2=0x7fb5d010a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 --2- 172.21.15.64:0/2033433172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d010a240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0009990 tx=0x7fb5c0009ca0 comp rx=0 tx=0).stop 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 -- 172.21.15.64:0/2033433172 shutdown_connections 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 --2- 172.21.15.64:0/2033433172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0100010 0x7fb5d0100430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 --2- 172.21.15.64:0/2033433172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d010a240 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 --2- 172.21.15.64:0/2033433172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5d0102ff0 0x7fb5d0103450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 -- 172.21.15.64:0/2033433172 >> 172.21.15.64:0/2033433172 conn(0x7fb5d00fb930 msgr2=0x7fb5d00fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 -- 172.21.15.64:0/2033433172 shutdown_connections 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 -- 172.21.15.64:0/2033433172 wait complete. 2024-02-09T22:44:59.408 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.404+0000 7fb5d595b700 1 Processor -- start 2024-02-09T22:44:59.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 -- start start 2024-02-09T22:44:59.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5d0100010 0x7fb5d0102950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:59.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0102ff0 0x7fb5d0100fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:59.413 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d01014e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d0101c60 con 0x7fb5d0100010 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d0101dd0 con 0x7fb5d0103990 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5d595b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d01a1ca0 con 0x7fb5d0102ff0 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d01014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d01014e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:49418/0 (socket says 172.21.15.64:49418) 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 -- 172.21.15.64:0/3869531817 learned_addr learned my addr 172.21.15.64:0/3869531817 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5ceffd700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5d0100010 0x7fb5d0102950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5ce7fc700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0102ff0 0x7fb5d0100fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 -- 172.21.15.64:0/3869531817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0102ff0 msgr2=0x7fb5d0100fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0102ff0 0x7fb5d0100fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 -- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5d0100010 msgr2=0x7fb5d0102950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5d0100010 0x7fb5d0102950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:44:59.414 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5cf7fe700 1 -- 172.21.15.64:0/3869531817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c00095f0 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.405+0000 7fb5ce7fc700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0102ff0 0x7fb5d0100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.406+0000 7fb5cf7fe700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d01014e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0006010 tx=0x7fb5c000f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.406+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c001d070 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.406+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5c000fe60 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.406+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d01a1f20 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.406+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c0017650 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.406+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d01a22d0 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.407+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d0109150 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.408+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb5c0004830 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.409+0000 7fb5d4959700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5b80a2150 0x7fb5b80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.409+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fb5c00ce2e0 con 0x7fb5d0103990 2024-02-09T22:44:59.415 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.409+0000 7fb5ceffd700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5b80a2150 0x7fb5b80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:44:59.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.409+0000 7fb5ceffd700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5b80a2150 0x7fb5b80a4610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009550 tx=0x7fb5c80091b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:44:59.416 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.412+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb5c0098540 con 0x7fb5d0103990 2024-02-09T22:44:59.626 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:44:59.623+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5d002d960 con 0x7fb5b80a2150 2024-02-09T22:45:00.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:59 smithi089 ceph-mon[36665]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T22:45:00.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:00.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:00.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:44:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:00.250 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:59 smithi064 ceph-mon[36638]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T22:45:00.250 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:00.250 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:00.250 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:44:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:00.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:59 smithi003 ceph-mon[32678]: osdmap e27: 4 total, 4 up, 4 in 2024-02-09T22:45:00.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:00.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:00.300 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:44:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:01.351 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:01 smithi064 ceph-mon[36638]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:01.351 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:01 smithi064 ceph-mon[36638]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T22:45:01.351 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:01 smithi064 ceph-mon[36638]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 280 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:01.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:01 smithi003 ceph-mon[32678]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:01.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:01 smithi003 ceph-mon[32678]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T22:45:01.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:01 smithi003 ceph-mon[32678]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 280 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:01 smithi089 ceph-mon[36665]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:01 smithi089 ceph-mon[36665]: osdmap e28: 4 total, 4 up, 4 in 2024-02-09T22:45:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:01 smithi089 ceph-mon[36665]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 280 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:02.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:02 smithi003 ceph-mon[32678]: osdmap e29: 4 total, 4 up, 4 in 2024-02-09T22:45:02.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:02 smithi064 ceph-mon[36638]: osdmap e29: 4 total, 4 up, 4 in 2024-02-09T22:45:02.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:02 smithi089 ceph-mon[36665]: osdmap e29: 4 total, 4 up, 4 in 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/749698464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]: dispatch 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]: dispatch 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]': finished 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: osdmap e30: 5 total, 4 up, 5 in 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/3536062160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/749698464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]: dispatch 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]: dispatch 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]': finished 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: osdmap e30: 5 total, 4 up, 5 in 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/3536062160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/749698464' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]: dispatch 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]: dispatch 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bef8649b-bcc5-40c9-8df2-b59957282a8d"}]': finished 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: osdmap e30: 5 total, 4 up, 5 in 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: pgmap v108: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/3536062160' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:05 smithi003 ceph-mon[32678]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-02-09T22:45:05.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:05 smithi064 ceph-mon[36638]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-02-09T22:45:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:05 smithi089 ceph-mon[36665]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 104 KiB/s, 0 objects/s recovering 2024-02-09T22:45:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:06 smithi003 ceph-mon[32678]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:06.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:45:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:45:06.480+0000 7f899702c700 -1 calc_pg_upmaps abort due to max <= 0 2024-02-09T22:45:06.897 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:06 smithi064 ceph-mon[36638]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:06.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:06 smithi089 ceph-mon[36665]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:08.764 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:08 smithi064 ceph-mon[36638]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:08 smithi003 ceph-mon[32678]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:08 smithi089 ceph-mon[36665]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T22:45:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:10.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T22:45:10.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-02-09T22:45:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:11 smithi003 ceph-mon[32678]: Deploying daemon osd.4 on smithi064 2024-02-09T22:45:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:11 smithi003 ceph-mon[32678]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:11.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:11 smithi064 ceph-mon[36638]: Deploying daemon osd.4 on smithi064 2024-02-09T22:45:11.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:11 smithi064 ceph-mon[36638]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:11.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:11 smithi089 ceph-mon[36665]: Deploying daemon osd.4 on smithi064 2024-02-09T22:45:11.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:11 smithi089 ceph-mon[36665]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:12 smithi064 ceph-mon[36638]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:12 smithi003 ceph-mon[32678]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:12.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:12 smithi089 ceph-mon[36665]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:15.136 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:15 smithi064 ceph-mon[36638]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:15.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:15 smithi003 ceph-mon[32678]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:15.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:15 smithi089 ceph-mon[36665]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:17.457 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:17 smithi064 ceph-mon[36638]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:17 smithi089 ceph-mon[36665]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:17 smithi003 ceph-mon[32678]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:18.891 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 4 on host 'smithi064' 2024-02-09T22:45:18.892 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.887+0000 7fb5d4959700 1 -- 172.21.15.64:0/3869531817 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb5d002d960 con 0x7fb5b80a2150 2024-02-09T22:45:18.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5b80a2150 msgr2=0x7fb5b80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:18.893 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5b80a2150 0x7fb5b80a4610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009550 tx=0x7fb5c80091b0 comp rx=0 tx=0).stop 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 msgr2=0x7fb5d01014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d01014e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0006010 tx=0x7fb5c000f690 comp rx=0 tx=0).stop 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 shutdown_connections 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5b80a2150 0x7fb5b80a4610 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb5d0103990 0x7fb5d01014e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb5d0100010 0x7fb5d0102950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 --2- 172.21.15.64:0/3869531817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb5d0102ff0 0x7fb5d0100fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 >> 172.21.15.64:0/3869531817 conn(0x7fb5d00fb930 msgr2=0x7fb5d0106290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 shutdown_connections 2024-02-09T22:45:18.894 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:45:18.890+0000 7fb5d595b700 1 -- 172.21.15.64:0/3869531817 wait complete. 2024-02-09T22:45:19.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:19.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:19.508 DEBUG:teuthology.orchestra.run.smithi064:osd.4> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.4.service 2024-02-09T22:45:19.510 INFO:tasks.cephadm:Deploying osd.5 on smithi089 with /dev/vg_nvme/lv_3... 2024-02-09T22:45:19.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-02-09T22:45:19.755 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:45:20.155 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:20 smithi089 ceph-mon[36665]: from='osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T22:45:20.155 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:20 smithi089 ceph-mon[36665]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T22:45:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:20 smithi003 ceph-mon[32678]: from='osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T22:45:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:20 smithi003 ceph-mon[32678]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T22:45:20.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:20 smithi064 ceph-mon[36638]: from='osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T22:45:20.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:20 smithi064 ceph-mon[36638]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-02-09T22:45:21.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:21 smithi089 ceph-mon[36665]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T22:45:21.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:21 smithi089 ceph-mon[36665]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T22:45:21.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:21.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:21 smithi089 ceph-mon[36665]: from='osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:45:21.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:21 smithi089 ceph-mon[36665]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:45:21.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:21 smithi089 ceph-mon[36665]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:21 smithi003 ceph-mon[32678]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T22:45:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:21 smithi003 ceph-mon[32678]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T22:45:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:21 smithi003 ceph-mon[32678]: from='osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:45:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:21 smithi003 ceph-mon[32678]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:45:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:21 smithi003 ceph-mon[32678]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:21.423 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-02-09T22:45:21.423 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-02-09T22:45:21.424 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-09T22:45:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-09T22:45:21.424 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114114 s, 919 MB/s 2024-02-09T22:45:21.424 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-09T22:45:21.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:21 smithi064 ceph-mon[36638]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-02-09T22:45:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:21 smithi064 ceph-mon[36638]: osdmap e31: 5 total, 4 up, 5 in 2024-02-09T22:45:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:21 smithi064 ceph-mon[36638]: from='osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:45:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:21 smithi064 ceph-mon[36638]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:45:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:21 smithi064 ceph-mon[36638]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:21.990 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_3 2024-02-09T22:45:22.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-mon[36638]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:45:22.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-mon[36638]: osdmap e32: 5 total, 4 up, 5 in 2024-02-09T22:45:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-mon[36638]: from='osd.4 ' entity='osd.4' 2024-02-09T22:45:22.186 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T22:45:22.027+0000 7f2fb564b700 -1 osd.4 0 waiting for initial osdmap 2024-02-09T22:45:22.186 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 22:45:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T22:45:22.039+0000 7f2fadc37700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:45:22.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:22 smithi089 ceph-mon[36665]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:45:22.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:22 smithi089 ceph-mon[36665]: osdmap e32: 5 total, 4 up, 5 in 2024-02-09T22:45:22.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:22.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:22.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:22 smithi089 ceph-mon[36665]: from='osd.4 ' entity='osd.4' 2024-02-09T22:45:22.250 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:45:22.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:22 smithi003 ceph-mon[32678]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:45:22.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:22 smithi003 ceph-mon[32678]: osdmap e32: 5 total, 4 up, 5 in 2024-02-09T22:45:22.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:22.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:22.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:22 smithi003 ceph-mon[32678]: from='osd.4 ' entity='osd.4' 2024-02-09T22:45:23.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:45:23.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:45:23.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:23.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] boot 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: osdmap e33: 5 total, 5 up, 5 in 2024-02-09T22:45:23.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:23.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:23.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:23.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] boot 2024-02-09T22:45:23.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: osdmap e33: 5 total, 5 up, 5 in 2024-02-09T22:45:23.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:23.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:23.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: osd.4 [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] boot 2024-02-09T22:45:23.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: osdmap e33: 5 total, 5 up, 5 in 2024-02-09T22:45:23.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:45:23.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 -- 172.21.15.89:0/3310103424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf8105100 msgr2=0x7f7bf81074f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:23.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 --2- 172.21.15.89:0/3310103424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf8105100 0x7f7bf81074f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009990 tx=0x7f7bec009ca0 comp rx=0 tx=0).stop 2024-02-09T22:45:23.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 -- 172.21.15.89:0/3310103424 shutdown_connections 2024-02-09T22:45:23.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 --2- 172.21.15.89:0/3310103424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7bf8107a30 0x7f7bf8109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:23.686 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 --2- 172.21.15.89:0/3310103424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bf81027d0 0x7f7bf8104bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 --2- 172.21.15.89:0/3310103424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf8105100 0x7f7bf81074f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 -- 172.21.15.89:0/3310103424 >> 172.21.15.89:0/3310103424 conn(0x7f7bf80fc1e0 msgr2=0x7f7bf80fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 -- 172.21.15.89:0/3310103424 shutdown_connections 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.681+0000 7f7bfe684700 1 -- 172.21.15.89:0/3310103424 wait complete. 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 Processor -- start 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 -- start start 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 0x7f7bf8102130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bf8105100 0x7f7bf8100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7bf8107a30 0x7f7bf8100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:23.687 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf8101440 con 0x7f7bf8105100 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf81015b0 con 0x7f7bf8107a30 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfe684700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf81a1d50 con 0x7f7bf81027d0 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 0x7f7bf8102130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 0x7f7bf8102130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34108/0 (socket says 172.21.15.89:34108) 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 -- 172.21.15.89:0/2508220942 learned_addr learned my addr 172.21.15.89:0/2508220942 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf77fe700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bf8105100 0x7f7bf8100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bfcc21700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7bf8107a30 0x7f7bf8100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:23.688 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 -- 172.21.15.89:0/2508220942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7bf8107a30 msgr2=0x7f7bf8100cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7bf8107a30 0x7f7bf8100cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 -- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bf8105100 msgr2=0x7f7bf8100780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bf8105100 0x7f7bf8100780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 -- 172.21.15.89:0/2508220942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bec0095f0 con 0x7f7bf81027d0 2024-02-09T22:45:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.682+0000 7f7bf7fff700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 0x7f7bf8102130 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e9a0 tx=0x7f7be800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:45:23.689 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.683+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8009790 con 0x7f7bf81027d0 2024-02-09T22:45:23.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.683+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf81a1fd0 con 0x7f7bf81027d0 2024-02-09T22:45:23.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.683+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7be800cbd0 con 0x7f7bf81027d0 2024-02-09T22:45:23.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.683+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be8018680 con 0x7f7bf81027d0 2024-02-09T22:45:23.690 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.683+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf81a2490 con 0x7f7bf81027d0 2024-02-09T22:45:23.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.684+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf80fdd60 con 0x7f7bf81027d0 2024-02-09T22:45:23.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.685+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f7be800cd40 con 0x7f7bf81027d0 2024-02-09T22:45:23.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.685+0000 7f7bf57fa700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7be00a2150 0x7f7be00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:23.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.685+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f7be8014070 con 0x7f7bf81027d0 2024-02-09T22:45:23.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.686+0000 7f7bf77fe700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7be00a2150 0x7f7be00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:23.691 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.686+0000 7f7bf77fe700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7be00a2150 0x7f7be00a4610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00b650 tx=0x7f7bec005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:45:23.693 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.690+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7be8052d20 con 0x7f7bf81027d0 2024-02-09T22:45:23.891 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:23.888+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f7bf802d880 con 0x7f7be00a2150 2024-02-09T22:45:24.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:24 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi064 2024-02-09T22:45:24.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:24 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi064 to 8540M 2024-02-09T22:45:24.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:24 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:24.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:24 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:24.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:24 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:24.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:24 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi064 2024-02-09T22:45:24.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:24 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi064 to 8540M 2024-02-09T22:45:24.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:24 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:24.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:24 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:24.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:24 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:24.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:24 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi064 2024-02-09T22:45:24.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:24 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi064 to 8540M 2024-02-09T22:45:24.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:24 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:24.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:24 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:24.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:24 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:25 smithi003 ceph-mon[32678]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:25 smithi003 ceph-mon[32678]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T22:45:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:25 smithi003 ceph-mon[32678]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:25 smithi064 ceph-mon[36638]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:25 smithi064 ceph-mon[36638]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T22:45:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:25 smithi064 ceph-mon[36638]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:25.462 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:25 smithi089 ceph-mon[36665]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:25.462 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:25 smithi089 ceph-mon[36665]: osdmap e34: 5 total, 5 up, 5 in 2024-02-09T22:45:25.462 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:25 smithi089 ceph-mon[36665]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:26 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/1222637979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]: dispatch 2024-02-09T22:45:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:26 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]: dispatch 2024-02-09T22:45:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:26 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]': finished 2024-02-09T22:45:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:26 smithi064 ceph-mon[36638]: osdmap e35: 6 total, 5 up, 6 in 2024-02-09T22:45:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:26.700 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:26 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/1222637979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]: dispatch 2024-02-09T22:45:26.701 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:26 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]: dispatch 2024-02-09T22:45:26.701 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:26 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]': finished 2024-02-09T22:45:26.701 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:26 smithi089 ceph-mon[36665]: osdmap e35: 6 total, 5 up, 6 in 2024-02-09T22:45:26.701 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:26 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/1222637979' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]: dispatch 2024-02-09T22:45:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:26 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]: dispatch 2024-02-09T22:45:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:26 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "245a8e5f-2535-4346-8457-dd94e211574d"}]': finished 2024-02-09T22:45:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:26 smithi003 ceph-mon[32678]: osdmap e35: 6 total, 5 up, 6 in 2024-02-09T22:45:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:27 smithi064 ceph-mon[36638]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:27 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/1146305494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:27 smithi089 ceph-mon[36665]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:27 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/1146305494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:27.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:27 smithi003 ceph-mon[32678]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:27.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:27 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/1146305494' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:28.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:28 smithi003 ceph-mon[32678]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:28.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:28 smithi064 ceph-mon[36638]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:28 smithi089 ceph-mon[36665]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:30.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:30 smithi003 ceph-mon[32678]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:30.888 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:30 smithi089 ceph-mon[36665]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:30.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:30 smithi064 ceph-mon[36638]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:32 smithi003 ceph-mon[32678]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:32.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:32 smithi064 ceph-mon[36638]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:32 smithi089 ceph-mon[36665]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T22:45:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:34.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T22:45:34.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:34.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-02-09T22:45:34.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:35.364 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:35 smithi089 ceph-mon[36665]: Deploying daemon osd.5 on smithi089 2024-02-09T22:45:35.364 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:35 smithi089 ceph-mon[36665]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:35.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:35 smithi003 ceph-mon[32678]: Deploying daemon osd.5 on smithi089 2024-02-09T22:45:35.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:35 smithi003 ceph-mon[32678]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:35 smithi064 ceph-mon[36638]: Deploying daemon osd.5 on smithi089 2024-02-09T22:45:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:35 smithi064 ceph-mon[36638]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:36.877 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:36 smithi089 ceph-mon[36665]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:36.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:36 smithi003 ceph-mon[32678]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:36.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:36 smithi064 ceph-mon[36638]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:38.792 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:38 smithi089 ceph-mon[36665]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:38 smithi003 ceph-mon[32678]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:38.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:38 smithi064 ceph-mon[36638]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:40.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:39 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:40.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:39 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:40.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:39 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:40.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:39 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:40.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:39 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:40.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:39 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:39 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:41.136 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:40 smithi089 ceph-mon[36665]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:41.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:40 smithi003 ceph-mon[32678]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:41.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:40 smithi064 ceph-mon[36638]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:42.411 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.408+0000 7f7bf57fa700 1 -- 172.21.15.89:0/2508220942 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f7bf802d880 con 0x7f7be00a2150 2024-02-09T22:45:42.412 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 5 on host 'smithi089' 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.410+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7be00a2150 msgr2=0x7f7be00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.410+0000 7f7bfe684700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7be00a2150 0x7f7be00a4610 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7bec00b650 tx=0x7f7bec005fb0 comp rx=0 tx=0).stop 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.410+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 msgr2=0x7f7bf8102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.410+0000 7f7bfe684700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 0x7f7bf8102130 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7be800e9a0 tx=0x7f7be800ecb0 comp rx=0 tx=0).stop 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 shutdown_connections 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7be00a2150 0x7f7be00a4610 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7bf81027d0 0x7f7bf8102130 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7bf8107a30 0x7f7bf8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 --2- 172.21.15.89:0/2508220942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7bf8105100 0x7f7bf8100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 >> 172.21.15.89:0/2508220942 conn(0x7f7bf80fc1e0 msgr2=0x7f7bf8103070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 shutdown_connections 2024-02-09T22:45:42.414 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:45:42.411+0000 7f7bfe684700 1 -- 172.21.15.89:0/2508220942 wait complete. 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.711 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.712 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:42 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:42 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:42 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:45:42.986 DEBUG:teuthology.orchestra.run.smithi089:osd.5> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.5.service 2024-02-09T22:45:42.988 INFO:tasks.cephadm:Deploying osd.6 on smithi003 with /dev/vg_nvme/lv_2... 2024-02-09T22:45:42.988 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T22:45:43.241 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:45:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:43 smithi003 ceph-mon[32678]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:43 smithi003 ceph-mon[32678]: from='osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T22:45:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:43 smithi003 ceph-mon[32678]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T22:45:43.801 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:43 smithi089 ceph-mon[36665]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:43.801 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:43 smithi089 ceph-mon[36665]: from='osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T22:45:43.802 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:43 smithi089 ceph-mon[36665]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T22:45:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:43 smithi064 ceph-mon[36638]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:43 smithi064 ceph-mon[36638]: from='osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T22:45:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:43 smithi064 ceph-mon[36638]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-02-09T22:45:44.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:44 smithi003 ceph-mon[32678]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T22:45:44.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:44 smithi003 ceph-mon[32678]: osdmap e36: 6 total, 5 up, 6 in 2024-02-09T22:45:44.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:44 smithi003 ceph-mon[32678]: from='osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:45:44.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:44 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:44.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:44 smithi003 ceph-mon[32678]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:45:44.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:44 smithi064 ceph-mon[36638]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T22:45:44.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:44 smithi064 ceph-mon[36638]: osdmap e36: 6 total, 5 up, 6 in 2024-02-09T22:45:44.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:44 smithi064 ceph-mon[36638]: from='osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:45:44.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:44 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:44.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:44 smithi064 ceph-mon[36638]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:45:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:44 smithi089 ceph-mon[36665]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-02-09T22:45:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:44 smithi089 ceph-mon[36665]: osdmap e36: 6 total, 5 up, 6 in 2024-02-09T22:45:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:44 smithi089 ceph-mon[36665]: from='osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:45:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:44 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:44.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:44 smithi089 ceph-mon[36665]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:45:45.095 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T22:45:45.095 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T22:45:45.095 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-02-09T22:45:45.095 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-02-09T22:45:45.096 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132388 s, 792 MB/s 2024-02-09T22:45:45.096 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-02-09T22:45:45.625 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-mon[36665]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:45.625 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-mon[36665]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:45:45.625 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-mon[36665]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T22:45:45.625 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:45.625 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:45.626 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-mon[36665]: from='osd.5 ' entity='osd.5' 2024-02-09T22:45:45.626 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T22:45:45.368+0000 7f34a5f4e700 -1 osd.5 0 waiting for initial osdmap 2024-02-09T22:45:45.626 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 22:45:45 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T22:45:45.376+0000 7f349e53a700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:45:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:45 smithi003 ceph-mon[32678]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:45 smithi003 ceph-mon[32678]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:45:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:45 smithi003 ceph-mon[32678]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T22:45:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:45 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:45.700 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:45 smithi003 ceph-mon[32678]: from='osd.5 ' entity='osd.5' 2024-02-09T22:45:45.728 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_2 2024-02-09T22:45:45.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:45 smithi064 ceph-mon[36638]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:45.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:45 smithi064 ceph-mon[36638]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:45:45.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:45 smithi064 ceph-mon[36638]: osdmap e37: 6 total, 5 up, 6 in 2024-02-09T22:45:45.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:45.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:45 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:45.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:45 smithi064 ceph-mon[36638]: from='osd.5 ' entity='osd.5' 2024-02-09T22:45:45.984 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] boot 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: osdmap e38: 6 total, 6 up, 6 in 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.856 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:46.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:46.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:46.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:46.857 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] boot 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: osdmap e38: 6 total, 6 up, 6 in 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:46.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:46.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:45:46.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: osd.5 [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] boot 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: osdmap e38: 6 total, 6 up, 6 in 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:46.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:46.979 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:45:46.979 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 -- 172.21.15.3:0/2058249083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4105190 msgr2=0x7fc4b4107580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 --2- 172.21.15.3:0/2058249083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4105190 0x7fc4b4107580 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc4a8009990 tx=0x7fc4a8009ca0 comp rx=0 tx=0).stop 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 -- 172.21.15.3:0/2058249083 shutdown_connections 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 --2- 172.21.15.3:0/2058249083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4b4102860 0x7fc4b4104c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 --2- 172.21.15.3:0/2058249083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b4107ac0 0x7fc4b4109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 --2- 172.21.15.3:0/2058249083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4105190 0x7fc4b4107580 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 -- 172.21.15.3:0/2058249083 >> 172.21.15.3:0/2058249083 conn(0x7fc4b40fc1f0 msgr2=0x7fc4b40fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:45:47.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 -- 172.21.15.3:0/2058249083 shutdown_connections 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 -- 172.21.15.3:0/2058249083 wait complete. 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.589+0000 7fc4bc009700 1 Processor -- start 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 -- start start 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 0x7fc4b4199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4b4105190 0x7fc4b419a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b4107ac0 0x7fc4b41a1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41a1ae0 con 0x7fc4b4102860 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41a1c20 con 0x7fc4b4107ac0 2024-02-09T22:45:47.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4bc009700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4b41a1db0 con 0x7fc4b4105190 2024-02-09T22:45:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 0x7fc4b4199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 0x7fc4b4199f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36904/0 (socket says 172.21.15.3:36904) 2024-02-09T22:45:47.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b95a4700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4b4105190 0x7fc4b419a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 -- 172.21.15.3:0/442785674 learned_addr learned my addr 172.21.15.3:0/442785674 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4ba5a6700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b4107ac0 0x7fc4b41a1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 -- 172.21.15.3:0/442785674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4b4105190 msgr2=0x7fc4b419a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4b4105190 0x7fc4b419a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 -- 172.21.15.3:0/442785674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b4107ac0 msgr2=0x7fc4b41a1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b4107ac0 0x7fc4b41a1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:45:47.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 -- 172.21.15.3:0/442785674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a80095f0 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.590+0000 7fc4b9da5700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 0x7fc4b4199f50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4007ee0 tx=0x7fc4a400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.591+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a400dd20 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.591+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4a400f040 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.591+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4b41a2090 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.591+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4014600 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.591+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4b41a25e0 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.591+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4b4109640 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.593+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fc4a4015400 con 0x7fc4b4102860 2024-02-09T22:45:47.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.593+0000 7fc4b2ffd700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc49c0a21d0 0x7fc49c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:45:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.593+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7fc4a40ccb00 con 0x7fc4b4102860 2024-02-09T22:45:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.593+0000 7fc4b95a4700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc49c0a21d0 0x7fc49c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:45:47.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.594+0000 7fc4b95a4700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc49c0a21d0 0x7fc49c0a4690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800b5c0 tx=0x7fc4a8003860 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:45:47.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.597+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc4a4096970 con 0x7fc4b4102860 2024-02-09T22:45:47.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:47 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi089 2024-02-09T22:45:47.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:47 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi089 to 8540M 2024-02-09T22:45:47.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:47 smithi003 ceph-mon[32678]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:47.812 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:47 smithi003 ceph-mon[32678]: osdmap e39: 6 total, 6 up, 6 in 2024-02-09T22:45:47.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:45:47.809+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fc4b402d8b0 con 0x7fc49c0a21d0 2024-02-09T22:45:47.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:47 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi089 2024-02-09T22:45:47.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:47 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi089 to 8540M 2024-02-09T22:45:47.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:47 smithi064 ceph-mon[36638]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:47.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:47 smithi064 ceph-mon[36638]: osdmap e39: 6 total, 6 up, 6 in 2024-02-09T22:45:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:47 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi089 2024-02-09T22:45:47.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:47 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi089 to 8540M 2024-02-09T22:45:47.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:47 smithi089 ceph-mon[36665]: pgmap v138: 1 pgs: 1 peering; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-02-09T22:45:47.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:47 smithi089 ceph-mon[36665]: osdmap e39: 6 total, 6 up, 6 in 2024-02-09T22:45:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:48 smithi089 ceph-mon[36665]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T22:45:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:48 smithi089 ceph-mon[36665]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:48.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:48 smithi089 ceph-mon[36665]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:45:48.770 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:48 smithi003 ceph-mon[32678]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T22:45:48.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:48 smithi003 ceph-mon[32678]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:48.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:48.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:48.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:48.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:48 smithi003 ceph-mon[32678]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:45:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:48 smithi064 ceph-mon[36638]: osdmap e40: 6 total, 6 up, 6 in 2024-02-09T22:45:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:48 smithi064 ceph-mon[36638]: from='client.14367 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:45:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:45:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:45:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:48 smithi064 ceph-mon[36638]: pgmap v141: 1 pgs: 1 peering; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:45:49.704 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:49 smithi003 ceph-mon[32678]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T22:45:49.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:49 smithi064 ceph-mon[36638]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T22:45:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:49 smithi089 ceph-mon[36665]: osdmap e41: 6 total, 6 up, 6 in 2024-02-09T22:45:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:50 smithi003 ceph-mon[32678]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T22:45:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:50 smithi003 ceph-mon[32678]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3444853103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392d6f64-4fbc-472e-96ef-dfb2b5291327"}]: dispatch 2024-02-09T22:45:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3444853103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "392d6f64-4fbc-472e-96ef-dfb2b5291327"}]': finished 2024-02-09T22:45:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:50 smithi003 ceph-mon[32678]: osdmap e43: 7 total, 6 up, 7 in 2024-02-09T22:45:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:50 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:45:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:50 smithi064 ceph-mon[36638]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T22:45:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:50 smithi064 ceph-mon[36638]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3444853103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392d6f64-4fbc-472e-96ef-dfb2b5291327"}]: dispatch 2024-02-09T22:45:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3444853103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "392d6f64-4fbc-472e-96ef-dfb2b5291327"}]': finished 2024-02-09T22:45:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:50 smithi064 ceph-mon[36638]: osdmap e43: 7 total, 6 up, 7 in 2024-02-09T22:45:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:50 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:45:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:50 smithi089 ceph-mon[36665]: osdmap e42: 6 total, 6 up, 6 in 2024-02-09T22:45:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:50 smithi089 ceph-mon[36665]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-02-09T22:45:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3444853103' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "392d6f64-4fbc-472e-96ef-dfb2b5291327"}]: dispatch 2024-02-09T22:45:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3444853103' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "392d6f64-4fbc-472e-96ef-dfb2b5291327"}]': finished 2024-02-09T22:45:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:50 smithi089 ceph-mon[36665]: osdmap e43: 7 total, 6 up, 7 in 2024-02-09T22:45:50.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:50 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:45:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3142279938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:51.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3142279938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:51.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3142279938' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:45:52.852 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:52 smithi064 ceph-mon[36638]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-02-09T22:45:52.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:52 smithi003 ceph-mon[32678]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-02-09T22:45:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:52 smithi089 ceph-mon[36665]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 116 KiB/s, 0 objects/s recovering 2024-02-09T22:45:55.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:55 smithi003 ceph-mon[32678]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-09T22:45:55.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:55 smithi064 ceph-mon[36638]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-09T22:45:55.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:55 smithi089 ceph-mon[36665]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-09T22:45:56.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:56 smithi003 ceph-mon[32678]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-09T22:45:56.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:56 smithi064 ceph-mon[36638]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-09T22:45:56.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:56 smithi089 ceph-mon[36665]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 73 KiB/s, 0 objects/s recovering 2024-02-09T22:45:58.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:58 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-09T22:45:58.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:58 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:58.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-09T22:45:58.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:58.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:58 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-02-09T22:45:58.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:58 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:45:59.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:59 smithi003 ceph-mon[32678]: Deploying daemon osd.6 on smithi003 2024-02-09T22:45:59.250 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:45:59 smithi003 ceph-mon[32678]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-09T22:45:59.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:59 smithi064 ceph-mon[36638]: Deploying daemon osd.6 on smithi003 2024-02-09T22:45:59.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:45:59 smithi064 ceph-mon[36638]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-09T22:45:59.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:59 smithi089 ceph-mon[36665]: Deploying daemon osd.6 on smithi003 2024-02-09T22:45:59.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:45:59 smithi089 ceph-mon[36665]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-02-09T22:46:00.665 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:00 smithi003 ceph-mon[32678]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:00.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:00 smithi064 ceph-mon[36638]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:00 smithi089 ceph-mon[36665]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:02.825 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:02 smithi003 ceph-mon[32678]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:02.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:02 smithi064 ceph-mon[36638]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:02.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:02 smithi089 ceph-mon[36665]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:05.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:05.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:05.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:05.084 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:04 smithi003 ceph-mon[32678]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:05.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:05.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:05.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:05.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:04 smithi064 ceph-mon[36638]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:05.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:05.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:05.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:05.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:04 smithi089 ceph-mon[36665]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:06 smithi003 ceph-mon[32678]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:06.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:06 smithi064 ceph-mon[36638]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:06.979 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:06 smithi089 ceph-mon[36665]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.214+0000 7fc4b2ffd700 1 -- 172.21.15.3:0/442785674 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc4b402d8b0 con 0x7fc49c0a21d0 2024-02-09T22:46:07.217 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 6 on host 'smithi003' 2024-02-09T22:46:07.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc49c0a21d0 msgr2=0x7fc49c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:07.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc49c0a21d0 0x7fc49c0a4690 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800b5c0 tx=0x7fc4a8003860 comp rx=0 tx=0).stop 2024-02-09T22:46:07.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 msgr2=0x7fc4b4199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 0x7fc4b4199f50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4007ee0 tx=0x7fc4a400d3b0 comp rx=0 tx=0).stop 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 shutdown_connections 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc49c0a21d0 0x7fc49c0a4690 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc4b4107ac0 0x7fc4b41a1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc4b4102860 0x7fc4b4199f50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 --2- 172.21.15.3:0/442785674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc4b4105190 0x7fc4b419a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 >> 172.21.15.3:0/442785674 conn(0x7fc4b40fc1f0 msgr2=0x7fc4b4101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 shutdown_connections 2024-02-09T22:46:07.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:46:07.216+0000 7fc4bc009700 1 -- 172.21.15.3:0/442785674 wait complete. 2024-02-09T22:46:07.842 DEBUG:teuthology.orchestra.run.smithi003:osd.6> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.6.service 2024-02-09T22:46:07.845 INFO:tasks.cephadm:Deploying osd.7 on smithi064 with /dev/vg_nvme/lv_2... 2024-02-09T22:46:07.845 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T22:46:08.104 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:46:08.474 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.474 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.474 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:08 smithi064 ceph-mon[36638]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T22:46:08.475 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.475 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:08 smithi089 ceph-mon[36665]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T22:46:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:08 smithi003 ceph-mon[32678]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-02-09T22:46:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:09.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:09 smithi064 ceph-mon[36638]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-09T22:46:09.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:09 smithi064 ceph-mon[36638]: osdmap e44: 7 total, 6 up, 7 in 2024-02-09T22:46:09.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:09 smithi064 ceph-mon[36638]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:46:09.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:09 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:09.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:09 smithi064 ceph-mon[36638]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:09.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:09 smithi089 ceph-mon[36665]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-09T22:46:09.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:09 smithi089 ceph-mon[36665]: osdmap e44: 7 total, 6 up, 7 in 2024-02-09T22:46:09.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:09 smithi089 ceph-mon[36665]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:46:09.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:09 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:09.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:09 smithi089 ceph-mon[36665]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:09.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-mon[32678]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-02-09T22:46:09.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-mon[32678]: osdmap e44: 7 total, 6 up, 7 in 2024-02-09T22:46:09.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-mon[32678]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:46:09.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:09.624 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-mon[32678]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:09.624 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T22:46:09.202+0000 7f2ca0d06700 -1 osd.6 0 waiting for initial osdmap 2024-02-09T22:46:09.625 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 22:46:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T22:46:09.215+0000 7f2c9d2fa700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:46:10.313 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T22:46:10.313 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T22:46:10.313 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-09T22:46:10.313 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-09T22:46:10.313 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112857 s, 929 MB/s 2024-02-09T22:46:10.313 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-09T22:46:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:10 smithi089 ceph-mon[36665]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:46:10.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:10 smithi089 ceph-mon[36665]: osdmap e45: 7 total, 6 up, 7 in 2024-02-09T22:46:10.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:10.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:10.627 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:10 smithi064 ceph-mon[36638]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:46:10.627 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:10 smithi064 ceph-mon[36638]: osdmap e45: 7 total, 6 up, 7 in 2024-02-09T22:46:10.628 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:10.628 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:10 smithi003 ceph-mon[32678]: from='osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:46:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:10 smithi003 ceph-mon[32678]: osdmap e45: 7 total, 6 up, 7 in 2024-02-09T22:46:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.004 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_2 2024-02-09T22:46:11.255 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:46:11.377 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:46:11.378 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:46:11.378 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.378 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] boot 2024-02-09T22:46:11.378 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: osdmap e46: 7 total, 7 up, 7 in 2024-02-09T22:46:11.378 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.378 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:11 smithi064 ceph-mon[36638]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:11.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:46:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:46:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] boot 2024-02-09T22:46:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: osdmap e46: 7 total, 7 up, 7 in 2024-02-09T22:46:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:11 smithi003 ceph-mon[32678]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:11.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:46:11.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:46:11.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: osd.6 [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] boot 2024-02-09T22:46:11.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: osdmap e46: 7 total, 7 up, 7 in 2024-02-09T22:46:11.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:46:11.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:11 smithi089 ceph-mon[36665]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 175 MiB used, 536 GiB / 536 GiB avail 2024-02-09T22:46:12.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: osdmap e47: 7 total, 7 up, 7 in 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi003 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi003 to 5697M 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:12.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.499 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: osdmap e47: 7 total, 7 up, 7 in 2024-02-09T22:46:12.499 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi003 2024-02-09T22:46:12.499 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.499 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.499 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.499 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi003 to 5697M 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:12.500 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: osdmap e47: 7 total, 7 up, 7 in 2024-02-09T22:46:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi003 2024-02-09T22:46:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi003 to 5697M 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:12.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:13.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4144120882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941003d0 msgr2=0x7f2e941007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:13.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4144120882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941003d0 0x7f2e941007f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2e900099c0 tx=0x7f2e90009cd0 comp rx=0 tx=0).stop 2024-02-09T22:46:13.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4144120882 shutdown_connections 2024-02-09T22:46:13.043 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4144120882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e94101f70 0x7f2e94108820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4144120882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941003d0 0x7f2e941007f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4144120882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e941015d0 0x7f2e94101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4144120882 >> 172.21.15.64:0/4144120882 conn(0x7f2e940fb990 msgr2=0x7f2e940fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4144120882 shutdown_connections 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.031+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4144120882 wait complete. 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 Processor -- start 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 -- start start 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e941003d0 0x7f2e94197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 0x7f2e94198340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e94101f70 0x7f2e9419f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9419f900 con 0x7f2e94101f70 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9419fa40 con 0x7f2e941015d0 2024-02-09T22:46:13.044 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9b2b2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e9419fc00 con 0x7f2e941003d0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 0x7f2e94198340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 0x7f2e94198340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:34912/0 (socket says 172.21.15.64:34912) 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 -- 172.21.15.64:0/4181401350 learned_addr learned my addr 172.21.15.64:0/4181401350 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9984f700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e94101f70 0x7f2e9419f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9904e700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e941003d0 0x7f2e94197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 -- 172.21.15.64:0/4181401350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e941003d0 msgr2=0x7f2e94197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e941003d0 0x7f2e94197e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 -- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e94101f70 msgr2=0x7f2e9419f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e94101f70 0x7f2e9419f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.032+0000 7f2e9884d700 1 -- 172.21.15.64:0/4181401350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e900095f0 con 0x7f2e941015d0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.033+0000 7f2e9884d700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 0x7f2e94198340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400b530 tx=0x7f2e8400b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.033+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e84011830 con 0x7f2e941015d0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.033+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e84011e70 con 0x7f2e941015d0 2024-02-09T22:46:13.045 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.033+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8400f340 con 0x7f2e941015d0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.033+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9419fee0 con 0x7f2e941015d0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.033+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e941a0350 con 0x7f2e941015d0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.034+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e94068a10 con 0x7f2e941015d0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.035+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f2e840106d0 con 0x7f2e941015d0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.035+0000 7f2e8a7fc700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e800a21d0 0x7f2e800a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.036+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f2e840ccf70 con 0x7f2e941015d0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.036+0000 7f2e9904e700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e800a21d0 0x7f2e800a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.036+0000 7f2e9904e700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e800a21d0 0x7f2e800a4690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e90000c00 tx=0x7f2e90009f10 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:46:13.046 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.039+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2e84096c50 con 0x7f2e941015d0 2024-02-09T22:46:13.256 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:13.254+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e94107a20 con 0x7f2e800a21d0 2024-02-09T22:46:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:13 smithi003 ceph-mon[32678]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:13 smithi064 ceph-mon[36638]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:13 smithi089 ceph-mon[36665]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 201 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:14 smithi003 ceph-mon[32678]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:46:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:46:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:46:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:14 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:14 smithi064 ceph-mon[36638]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:46:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:46:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:46:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:14 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:14 smithi089 ceph-mon[36665]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:46:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:46:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:46:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:14 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:15.311 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:15 smithi064 ceph-mon[36638]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:15.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:15 smithi003 ceph-mon[32678]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:15 smithi089 ceph-mon[36665]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:16.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:16 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/1951327797' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]: dispatch 2024-02-09T22:46:16.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:16 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]: dispatch 2024-02-09T22:46:16.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:16 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]': finished 2024-02-09T22:46:16.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:16 smithi089 ceph-mon[36665]: osdmap e48: 8 total, 7 up, 8 in 2024-02-09T22:46:16.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:16 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/1951327797' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]: dispatch 2024-02-09T22:46:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:16 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]: dispatch 2024-02-09T22:46:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:16 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]': finished 2024-02-09T22:46:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:16 smithi003 ceph-mon[32678]: osdmap e48: 8 total, 7 up, 8 in 2024-02-09T22:46:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:16 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/1951327797' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]: dispatch 2024-02-09T22:46:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:16 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]: dispatch 2024-02-09T22:46:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:16 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c96f9dfa-d559-4d7f-9fde-00d3fbbe856a"}]': finished 2024-02-09T22:46:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:16 smithi064 ceph-mon[36638]: osdmap e48: 8 total, 7 up, 8 in 2024-02-09T22:46:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:17 smithi003 ceph-mon[32678]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:17 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/1363126214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:46:17.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:17 smithi064 ceph-mon[36638]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:17.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:17 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/1363126214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:46:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:17 smithi089 ceph-mon[36665]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:17 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/1363126214' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:46:18.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:18 smithi003 ceph-mon[32678]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:18.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:18 smithi064 ceph-mon[36638]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:18.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:18 smithi089 ceph-mon[36665]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:20.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:20 smithi003 ceph-mon[32678]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:20 smithi064 ceph-mon[36638]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:20 smithi089 ceph-mon[36665]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:22.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:22 smithi064 ceph-mon[36638]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:22.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:22 smithi003 ceph-mon[32678]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:22 smithi089 ceph-mon[36665]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-09T22:46:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:23 smithi064 ceph-mon[36638]: Deploying daemon osd.7 on smithi064 2024-02-09T22:46:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-09T22:46:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:23 smithi003 ceph-mon[32678]: Deploying daemon osd.7 on smithi064 2024-02-09T22:46:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-02-09T22:46:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:23 smithi089 ceph-mon[36665]: Deploying daemon osd.7 on smithi064 2024-02-09T22:46:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:25 smithi003 ceph-mon[32678]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:25.424 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:25 smithi064 ceph-mon[36638]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:25 smithi089 ceph-mon[36665]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:26.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:26 smithi003 ceph-mon[32678]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:26.891 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:26 smithi064 ceph-mon[36638]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:26.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:26 smithi089 ceph-mon[36665]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:28.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:28 smithi003 ceph-mon[32678]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:28.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:28 smithi064 ceph-mon[36638]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:28.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:28 smithi089 ceph-mon[36665]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:30.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:30.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:30.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:30.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:30.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:30.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:30.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:30.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:30.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:31.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:30 smithi003 ceph-mon[32678]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:31.159 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:30 smithi064 ceph-mon[36638]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:31.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:30 smithi089 ceph-mon[36665]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:32.271 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.267+0000 7f2e8a7fc700 1 -- 172.21.15.64:0/4181401350 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2e94107a20 con 0x7f2e800a21d0 2024-02-09T22:46:32.271 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 7 on host 'smithi064' 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e800a21d0 msgr2=0x7f2e800a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e800a21d0 0x7f2e800a4690 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2e90000c00 tx=0x7f2e90009f10 comp rx=0 tx=0).stop 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 msgr2=0x7f2e94198340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 0x7f2e94198340 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2e8400b530 tx=0x7f2e8400b840 comp rx=0 tx=0).stop 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 shutdown_connections 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e800a21d0 0x7f2e800a4690 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2e941003d0 0x7f2e94197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2e94101f70 0x7f2e9419f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:32.273 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 --2- 172.21.15.64:0/4181401350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2e941015d0 0x7f2e94198340 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:32.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 >> 172.21.15.64:0/4181401350 conn(0x7f2e940fb990 msgr2=0x7f2e940fd4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:46:32.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 shutdown_connections 2024-02-09T22:46:32.274 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:46:32.269+0000 7f2e9b2b2700 1 -- 172.21.15.64:0/4181401350 wait complete. 2024-02-09T22:46:32.918 DEBUG:teuthology.orchestra.run.smithi064:osd.7> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.7.service 2024-02-09T22:46:32.921 INFO:tasks.cephadm:Deploying osd.8 on smithi089 with /dev/vg_nvme/lv_2... 2024-02-09T22:46:32.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-02-09T22:46:33.165 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: from='osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.518 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:33 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: from='osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:33 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: from='osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:33.640 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:33 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:34 smithi089 ceph-mon[36665]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-09T22:46:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:34 smithi089 ceph-mon[36665]: osdmap e49: 8 total, 7 up, 8 in 2024-02-09T22:46:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:34 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:34 smithi089 ceph-mon[36665]: from='osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:46:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:34 smithi089 ceph-mon[36665]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:46:34.562 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-mon[36638]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-09T22:46:34.562 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-mon[36638]: osdmap e49: 8 total, 7 up, 8 in 2024-02-09T22:46:34.562 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:34.562 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-mon[36638]: from='osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:46:34.562 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-mon[36638]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:46:34.562 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T22:46:34.259+0000 7f9c4a620700 -1 osd.7 0 waiting for initial osdmap 2024-02-09T22:46:34.562 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 22:46:34 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T22:46:34.270+0000 7f9c46c14700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:46:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:34 smithi003 ceph-mon[32678]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-02-09T22:46:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:34 smithi003 ceph-mon[32678]: osdmap e49: 8 total, 7 up, 8 in 2024-02-09T22:46:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:34 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:34 smithi003 ceph-mon[32678]: from='osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:46:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:34 smithi003 ceph-mon[32678]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:46:35.047 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-02-09T22:46:35.048 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-02-09T22:46:35.048 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-09T22:46:35.048 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-09T22:46:35.048 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0151866 s, 690 MB/s 2024-02-09T22:46:35.048 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-09T22:46:35.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:35 smithi089 ceph-mon[36665]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:46:35.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:35 smithi089 ceph-mon[36665]: osdmap e50: 8 total, 7 up, 8 in 2024-02-09T22:46:35.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:35.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:35.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:35 smithi089 ceph-mon[36665]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:35.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:35 smithi089 ceph-mon[36665]: osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] boot 2024-02-09T22:46:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:35 smithi003 ceph-mon[32678]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:46:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:35 smithi003 ceph-mon[32678]: osdmap e50: 8 total, 7 up, 8 in 2024-02-09T22:46:35.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:35.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:35.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:35 smithi003 ceph-mon[32678]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:35.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:35 smithi003 ceph-mon[32678]: osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] boot 2024-02-09T22:46:35.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_2 2024-02-09T22:46:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:35 smithi064 ceph-mon[36638]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:46:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:35 smithi064 ceph-mon[36638]: osdmap e50: 8 total, 7 up, 8 in 2024-02-09T22:46:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:35.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:35 smithi064 ceph-mon[36638]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 202 MiB used, 626 GiB / 626 GiB avail 2024-02-09T22:46:35.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:35 smithi064 ceph-mon[36638]: osd.7 [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] boot 2024-02-09T22:46:35.901 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:46:36.360 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:36 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:46:36.360 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:36 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:46:36.360 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:36 smithi064 ceph-mon[36638]: osdmap e51: 8 total, 8 up, 8 in 2024-02-09T22:46:36.361 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:36 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:36.361 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:36 smithi064 ceph-mon[36638]: osdmap e52: 8 total, 8 up, 8 in 2024-02-09T22:46:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:36 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:46:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:36 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:46:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:36 smithi003 ceph-mon[32678]: osdmap e51: 8 total, 8 up, 8 in 2024-02-09T22:46:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:36 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:36 smithi003 ceph-mon[32678]: osdmap e52: 8 total, 8 up, 8 in 2024-02-09T22:46:36.633 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:36 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:46:36.633 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:36 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:46:36.634 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:36 smithi089 ceph-mon[36665]: osdmap e51: 8 total, 8 up, 8 in 2024-02-09T22:46:36.634 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:36 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:46:36.634 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:36 smithi089 ceph-mon[36665]: osdmap e52: 8 total, 8 up, 8 in 2024-02-09T22:46:37.369 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 -- 172.21.15.89:0/2467040927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c101ec0 msgr2=0x7f821c102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 --2- 172.21.15.89:0/2467040927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c101ec0 0x7f821c102320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8208009990 tx=0x7f8208009ca0 comp rx=0 tx=0).stop 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 -- 172.21.15.89:0/2467040927 shutdown_connections 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 --2- 172.21.15.89:0/2467040927 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f821c102860 0x7f821c109110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 --2- 172.21.15.89:0/2467040927 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c100cc0 0x7f821c1010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 --2- 172.21.15.89:0/2467040927 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c101ec0 0x7f821c102320 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 -- 172.21.15.89:0/2467040927 >> 172.21.15.89:0/2467040927 conn(0x7f821c0fc280 msgr2=0x7f821c0fe6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 -- 172.21.15.89:0/2467040927 shutdown_connections 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 -- 172.21.15.89:0/2467040927 wait complete. 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 Processor -- start 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 -- start start 2024-02-09T22:46:37.370 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.363+0000 7f822324f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f821c100cc0 0x7f821c197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f822324f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c101ec0 0x7f821c198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f822324f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 0x7f821c19f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f822324f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c19f840 con 0x7f821c101ec0 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f822324f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c19f980 con 0x7f821c100cc0 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f822324f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f821c19fb40 con 0x7f821c102860 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 0x7f821c19f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 0x7f821c19f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:34120/0 (socket says 172.21.15.89:34120) 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 -- 172.21.15.89:0/4205020433 learned_addr learned my addr 172.21.15.89:0/4205020433 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f8220feb700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f821c100cc0 0x7f821c197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f821bfff700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c101ec0 0x7f821c198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 -- 172.21.15.89:0/4205020433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f821c100cc0 msgr2=0x7f821c197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f821c100cc0 0x7f821c197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 -- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c101ec0 msgr2=0x7f821c198230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c101ec0 0x7f821c198230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:37.371 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 -- 172.21.15.89:0/4205020433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82080095f0 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f821bfff700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c101ec0 0x7f821c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f82217ec700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 0x7f821c19f270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8214007ee0 tx=0x7f821400d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214017070 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f821c19fe20 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f821400f040 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.364+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8214005760 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.365+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f821c1a01e0 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.366+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f821c108150 con 0x7f821c102860 2024-02-09T22:46:37.372 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.367+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f82140058c0 con 0x7f821c102860 2024-02-09T22:46:37.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.367+0000 7f8219ffb700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f820c0a2150 0x7f820c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:46:37.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.367+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f82140cc9c0 con 0x7f821c102860 2024-02-09T22:46:37.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.367+0000 7f8220feb700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f820c0a2150 0x7f820c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:46:37.373 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.368+0000 7f8220feb700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f820c0a2150 0x7f820c0a4610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f821c1991d0 tx=0x7f8210009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:46:37.374 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.372+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82140964d0 con 0x7f821c102860 2024-02-09T22:46:37.580 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:37.577+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f821c02d9a0 con 0x7f820c0a2150 2024-02-09T22:46:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi064 2024-02-09T22:46:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi064 to 5693M 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:37 smithi089 ceph-mon[36665]: osdmap e53: 8 total, 8 up, 8 in 2024-02-09T22:46:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi064 2024-02-09T22:46:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi064 to 5693M 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:37 smithi003 ceph-mon[32678]: osdmap e53: 8 total, 8 up, 8 in 2024-02-09T22:46:37.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi064 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi064 to 5693M 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:37.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:37.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:37.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:37.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:37.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:37 smithi064 ceph-mon[36638]: osdmap e53: 8 total, 8 up, 8 in 2024-02-09T22:46:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:46:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:46:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:38 smithi089 ceph-mon[36665]: osdmap e54: 8 total, 8 up, 8 in 2024-02-09T22:46:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:46:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:46:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:38 smithi003 ceph-mon[32678]: osdmap e54: 8 total, 8 up, 8 in 2024-02-09T22:46:38.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:46:38.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:46:38.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:38.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:38 smithi064 ceph-mon[36638]: osdmap e54: 8 total, 8 up, 8 in 2024-02-09T22:46:39.563 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:39 smithi089 ceph-mon[36665]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:46:39.563 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:39 smithi089 ceph-mon[36665]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:39.563 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:39 smithi089 ceph-mon[36665]: osdmap e55: 8 total, 8 up, 8 in 2024-02-09T22:46:39.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:39 smithi003 ceph-mon[32678]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:46:39.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:39 smithi003 ceph-mon[32678]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:39.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:39 smithi003 ceph-mon[32678]: osdmap e55: 8 total, 8 up, 8 in 2024-02-09T22:46:39.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:39 smithi064 ceph-mon[36638]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:46:39.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:39 smithi064 ceph-mon[36638]: pgmap v180: 1 pgs: 1 remapped+peering; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:39.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:39 smithi064 ceph-mon[36638]: osdmap e55: 8 total, 8 up, 8 in 2024-02-09T22:46:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:40 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/1263681030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]: dispatch 2024-02-09T22:46:40.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:40 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]: dispatch 2024-02-09T22:46:40.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:40 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]': finished 2024-02-09T22:46:40.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:40 smithi089 ceph-mon[36665]: osdmap e56: 9 total, 8 up, 9 in 2024-02-09T22:46:40.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:40.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:40 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/1263681030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]: dispatch 2024-02-09T22:46:40.869 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:40 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]: dispatch 2024-02-09T22:46:40.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:40 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]': finished 2024-02-09T22:46:40.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:40 smithi003 ceph-mon[32678]: osdmap e56: 9 total, 8 up, 9 in 2024-02-09T22:46:40.870 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:40.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:40 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/1263681030' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]: dispatch 2024-02-09T22:46:40.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:40 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]: dispatch 2024-02-09T22:46:40.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:40 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "49491de9-7b30-46a5-9869-b3ff6d2285f7"}]': finished 2024-02-09T22:46:40.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:40 smithi064 ceph-mon[36638]: osdmap e56: 9 total, 8 up, 9 in 2024-02-09T22:46:40.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:41.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:41 smithi003 ceph-mon[32678]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:41.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:41 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/3371007992' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:46:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:41 smithi064 ceph-mon[36638]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:41 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/3371007992' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:46:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:41 smithi089 ceph-mon[36665]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:41 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/3371007992' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:46:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:42 smithi003 ceph-mon[32678]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:42 smithi064 ceph-mon[36638]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:42 smithi089 ceph-mon[36665]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:45 smithi003 ceph-mon[32678]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:45.398 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:45 smithi089 ceph-mon[36665]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:45.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:45 smithi064 ceph-mon[36638]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:46 smithi089 ceph-mon[36665]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:46.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:46 smithi003 ceph-mon[32678]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:46.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:46 smithi064 ceph-mon[36638]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:47.668 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-09T22:46:47.668 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:47.668 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:47 smithi089 ceph-mon[36665]: Deploying daemon osd.8 on smithi089 2024-02-09T22:46:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-09T22:46:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:47.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:47 smithi003 ceph-mon[32678]: Deploying daemon osd.8 on smithi089 2024-02-09T22:46:47.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-02-09T22:46:47.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:47.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:47 smithi064 ceph-mon[36638]: Deploying daemon osd.8 on smithi089 2024-02-09T22:46:48.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:48 smithi003 ceph-mon[32678]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:48 smithi064 ceph-mon[36638]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:48.946 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:48 smithi089 ceph-mon[36665]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:50.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:50 smithi089 ceph-mon[36665]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:50 smithi003 ceph-mon[32678]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:50 smithi064 ceph-mon[36638]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:52 smithi003 ceph-mon[32678]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:52.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:52 smithi064 ceph-mon[36638]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:52 smithi089 ceph-mon[36665]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:53.777 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:53 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:53.777 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:53 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:53.777 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:53 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:54.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:53 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:54.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:53 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:54.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:53 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:54.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:53 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:54.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:53 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:54.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:53 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:55.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:55 smithi003 ceph-mon[32678]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:55.419 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:55 smithi089 ceph-mon[36665]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:55.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:55 smithi064 ceph-mon[36638]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:56.546 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.543+0000 7f8219ffb700 1 -- 172.21.15.89:0/4205020433 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f821c02d9a0 con 0x7f820c0a2150 2024-02-09T22:46:56.547 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 8 on host 'smithi089' 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.545+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f820c0a2150 msgr2=0x7f820c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.545+0000 7f822324f700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f820c0a2150 0x7f820c0a4610 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f821c1991d0 tx=0x7f8210009260 comp rx=0 tx=0).stop 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.545+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 msgr2=0x7f821c19f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.545+0000 7f822324f700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 0x7f821c19f270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8214007ee0 tx=0x7f821400d3b0 comp rx=0 tx=0).stop 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 shutdown_connections 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f820c0a2150 0x7f820c0a4610 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f821c102860 0x7f821c19f270 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f821c100cc0 0x7f821c197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 --2- 172.21.15.89:0/4205020433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f821c101ec0 0x7f821c198230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 >> 172.21.15.89:0/4205020433 conn(0x7f821c0fc280 msgr2=0x7f821c105290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 shutdown_connections 2024-02-09T22:46:56.549 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:46:56.546+0000 7f822324f700 1 -- 172.21.15.89:0/4205020433 wait complete. 2024-02-09T22:46:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:56.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:56.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:57.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:46:57.189 DEBUG:teuthology.orchestra.run.smithi089:osd.8> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.8.service 2024-02-09T22:46:57.191 INFO:tasks.cephadm:Deploying osd.9 on smithi003 with /dev/vg_nvme/lv_1... 2024-02-09T22:46:57.191 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-09T22:46:57.445 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:46:57.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:57 smithi003 ceph-mon[32678]: from='osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T22:46:57.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:57 smithi003 ceph-mon[32678]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T22:46:57.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:57 smithi064 ceph-mon[36638]: from='osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T22:46:57.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:57 smithi064 ceph-mon[36638]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T22:46:57.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:57 smithi089 ceph-mon[36665]: from='osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T22:46:57.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:57 smithi089 ceph-mon[36665]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-02-09T22:46:58.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:58 smithi003 ceph-mon[32678]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T22:46:58.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:58 smithi003 ceph-mon[32678]: osdmap e57: 9 total, 8 up, 9 in 2024-02-09T22:46:58.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:58 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:58.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:58 smithi003 ceph-mon[32678]: from='osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:46:58.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:58 smithi003 ceph-mon[32678]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:46:58.745 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:58 smithi003 ceph-mon[32678]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:58.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:58 smithi064 ceph-mon[36638]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T22:46:58.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:58 smithi064 ceph-mon[36638]: osdmap e57: 9 total, 8 up, 9 in 2024-02-09T22:46:58.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:58 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:58.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:58 smithi064 ceph-mon[36638]: from='osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:46:58.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:58 smithi064 ceph-mon[36638]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:46:58.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:58 smithi064 ceph-mon[36638]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:58.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-mon[36665]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-02-09T22:46:58.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-mon[36665]: osdmap e57: 9 total, 8 up, 9 in 2024-02-09T22:46:58.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:58.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-mon[36665]: from='osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:46:58.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-mon[36665]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:46:58.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-mon[36665]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 229 MiB used, 715 GiB / 715 GiB avail 2024-02-09T22:46:58.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T22:46:58.653+0000 7f9334d15700 -1 osd.8 0 waiting for initial osdmap 2024-02-09T22:46:58.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 22:46:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T22:46:58.664+0000 7f9331309700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi003.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10+0 records in 2024-02-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi003.stderr:10+0 records out 2024-02-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi003.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111085 s, 944 MB/s 2024-02-09T22:46:59.326 INFO:teuthology.orchestra.run.smithi003.stderr:--> Zapping successful for: 2024-02-09T22:46:59.675 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:46:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: osdmap e58: 9 total, 8 up, 9 in 2024-02-09T22:46:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] boot 2024-02-09T22:46:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: osdmap e59: 9 total, 9 up, 9 in 2024-02-09T22:46:59.676 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:46:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:46:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: osdmap e58: 9 total, 8 up, 9 in 2024-02-09T22:46:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] boot 2024-02-09T22:46:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: osdmap e59: 9 total, 9 up, 9 in 2024-02-09T22:46:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:46:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:46:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: osdmap e58: 9 total, 8 up, 9 in 2024-02-09T22:46:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:46:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: osd.8 [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] boot 2024-02-09T22:46:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: osdmap e59: 9 total, 9 up, 9 in 2024-02-09T22:46:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:46:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:47:00.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi003:vg_nvme/lv_1 2024-02-09T22:47:00.302 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: osdmap e60: 9 total, 9 up, 9 in 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi089 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:00.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi089 to 5693M 2024-02-09T22:47:00.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:00.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:00.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:00 smithi064 ceph-mon[36638]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: osdmap e60: 9 total, 9 up, 9 in 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi089 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi089 to 5693M 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:00.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:00.979 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:00 smithi089 ceph-mon[36665]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: osdmap e60: 9 total, 9 up, 9 in 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi089 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:01.091 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:01.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi089 to 5693M 2024-02-09T22:47:01.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:01.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:01.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:01.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:01.092 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:00 smithi003 ceph-mon[32678]: pgmap v197: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:01.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 -- 172.21.15.3:0/875107812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c107a40 msgr2=0x7f022c109eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:01.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 --2- 172.21.15.3:0/875107812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c107a40 0x7f022c109eb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f021c009990 tx=0x7f021c009ca0 comp rx=0 tx=0).stop 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 -- 172.21.15.3:0/875107812 shutdown_connections 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 --2- 172.21.15.3:0/875107812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022c1027e0 0x7f022c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 --2- 172.21.15.3:0/875107812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c107a40 0x7f022c109eb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 --2- 172.21.15.3:0/875107812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f022c105110 0x7f022c107500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 -- 172.21.15.3:0/875107812 >> 172.21.15.3:0/875107812 conn(0x7f022c0fc1f0 msgr2=0x7f022c0fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 -- 172.21.15.3:0/875107812 shutdown_connections 2024-02-09T22:47:01.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.906+0000 7f0230b0e700 1 -- 172.21.15.3:0/875107812 wait complete. 2024-02-09T22:47:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 Processor -- start 2024-02-09T22:47:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 -- start start 2024-02-09T22:47:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 0x7f022c102140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f022c105110 0x7f022c100790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022c107a40 0x7f022c100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:01.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c101450 con 0x7f022c1027e0 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c1015c0 con 0x7f022c105110 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0230b0e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c1a1c90 con 0x7f022c107a40 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f022a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 0x7f022c102140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f022a59c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 0x7f022c102140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51446/0 (socket says 172.21.15.3:51446) 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f022a59c700 1 -- 172.21.15.3:0/246518373 learned_addr learned my addr 172.21.15.3:0/246518373 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.907+0000 7f0229d9b700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f022c105110 0x7f022c100790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022ad9d700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022c107a40 0x7f022c100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022a59c700 1 -- 172.21.15.3:0/246518373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022c107a40 msgr2=0x7f022c100cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022a59c700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022c107a40 0x7f022c100cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022a59c700 1 -- 172.21.15.3:0/246518373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f022c105110 msgr2=0x7f022c100790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022a59c700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f022c105110 0x7f022c100790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022a59c700 1 -- 172.21.15.3:0/246518373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c0095f0 con 0x7f022c1027e0 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f022a59c700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 0x7f022c102140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0224007e20 tx=0x7f022400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:47:01.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022400ad40 con 0x7f022c1027e0 2024-02-09T22:47:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0224012900 con 0x7f022c1027e0 2024-02-09T22:47:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c1a1f70 con 0x7f022c1027e0 2024-02-09T22:47:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0224011840 con 0x7f022c1027e0 2024-02-09T22:47:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.908+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c1a2380 con 0x7f022c1027e0 2024-02-09T22:47:01.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.909+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c0fdd70 con 0x7f022c1027e0 2024-02-09T22:47:01.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.910+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f0224012a70 con 0x7f022c1027e0 2024-02-09T22:47:01.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.911+0000 7f02237fe700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f02180a2150 0x7f02180a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:01.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.911+0000 7f0229d9b700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f02180a2150 0x7f02180a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:01.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.911+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f02240cd680 con 0x7f022c1027e0 2024-02-09T22:47:01.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.911+0000 7f0229d9b700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f02180a2150 0x7f02180a4610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0214005950 tx=0x7f021400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:47:01.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:01.915+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0224096fe0 con 0x7f022c1027e0 2024-02-09T22:47:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:02.128+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f022c02da10 con 0x7f02180a2150 2024-02-09T22:47:02.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:02 smithi003 ceph-mon[32678]: osdmap e61: 9 total, 9 up, 9 in 2024-02-09T22:47:02.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:02 smithi064 ceph-mon[36638]: osdmap e61: 9 total, 9 up, 9 in 2024-02-09T22:47:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:02 smithi089 ceph-mon[36665]: osdmap e61: 9 total, 9 up, 9 in 2024-02-09T22:47:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:03 smithi003 ceph-mon[32678]: osdmap e62: 9 total, 9 up, 9 in 2024-02-09T22:47:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:03 smithi003 ceph-mon[32678]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:03.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:03.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:03 smithi003 ceph-mon[32678]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:03.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:03 smithi064 ceph-mon[36638]: osdmap e62: 9 total, 9 up, 9 in 2024-02-09T22:47:03.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:03 smithi064 ceph-mon[36638]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:03.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:03.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:03 smithi064 ceph-mon[36638]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:03 smithi089 ceph-mon[36665]: osdmap e62: 9 total, 9 up, 9 in 2024-02-09T22:47:03.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:03 smithi089 ceph-mon[36665]: from='client.14430 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi003:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:03.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:03.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:03.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:03.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:03 smithi089 ceph-mon[36665]: pgmap v200: 1 pgs: 1 remapped+peering; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:04.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:04 smithi003 ceph-mon[32678]: osdmap e63: 9 total, 9 up, 9 in 2024-02-09T22:47:04.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:04 smithi064 ceph-mon[36638]: osdmap e63: 9 total, 9 up, 9 in 2024-02-09T22:47:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:04 smithi089 ceph-mon[36665]: osdmap e63: 9 total, 9 up, 9 in 2024-02-09T22:47:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:05 smithi003 ceph-mon[32678]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:05 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/686931298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46d134d1-6b63-4909-8a0a-c5c431c9dfdb"}]: dispatch 2024-02-09T22:47:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:05 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/686931298' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46d134d1-6b63-4909-8a0a-c5c431c9dfdb"}]': finished 2024-02-09T22:47:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:05 smithi003 ceph-mon[32678]: osdmap e64: 10 total, 9 up, 10 in 2024-02-09T22:47:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:05 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:05.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:05 smithi064 ceph-mon[36638]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:05 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/686931298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46d134d1-6b63-4909-8a0a-c5c431c9dfdb"}]: dispatch 2024-02-09T22:47:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:05 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/686931298' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46d134d1-6b63-4909-8a0a-c5c431c9dfdb"}]': finished 2024-02-09T22:47:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:05 smithi064 ceph-mon[36638]: osdmap e64: 10 total, 9 up, 10 in 2024-02-09T22:47:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:05 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:05 smithi089 ceph-mon[36665]: pgmap v202: 1 pgs: 1 remapped+peering; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:05 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/686931298' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "46d134d1-6b63-4909-8a0a-c5c431c9dfdb"}]: dispatch 2024-02-09T22:47:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:05 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/686931298' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "46d134d1-6b63-4909-8a0a-c5c431c9dfdb"}]': finished 2024-02-09T22:47:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:05 smithi089 ceph-mon[36665]: osdmap e64: 10 total, 9 up, 10 in 2024-02-09T22:47:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:05 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:06.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:06 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4159120051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:06.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:06 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4159120051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:06 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4159120051' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:07.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:07 smithi003 ceph-mon[32678]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-02-09T22:47:07.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:07 smithi064 ceph-mon[36638]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-02-09T22:47:07.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:07 smithi089 ceph-mon[36665]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 106 KiB/s, 0 objects/s recovering 2024-02-09T22:47:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:08 smithi003 ceph-mon[32678]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-09T22:47:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:08 smithi064 ceph-mon[36638]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-09T22:47:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:08 smithi089 ceph-mon[36665]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-02-09T22:47:10.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:10 smithi003 ceph-mon[32678]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-09T22:47:10.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:10 smithi064 ceph-mon[36638]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-09T22:47:10.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:10 smithi089 ceph-mon[36665]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-02-09T22:47:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-09T22:47:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:12 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-09T22:47:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:12 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:12.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-02-09T22:47:12.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:12 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:13 smithi003 ceph-mon[32678]: Deploying daemon osd.9 on smithi003 2024-02-09T22:47:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:13 smithi003 ceph-mon[32678]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-02-09T22:47:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:13 smithi089 ceph-mon[36665]: Deploying daemon osd.9 on smithi003 2024-02-09T22:47:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:13 smithi089 ceph-mon[36665]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-02-09T22:47:13.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:13 smithi064 ceph-mon[36638]: Deploying daemon osd.9 on smithi003 2024-02-09T22:47:13.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:13 smithi064 ceph-mon[36638]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-02-09T22:47:15.322 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:15 smithi003 ceph-mon[32678]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-09T22:47:15.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:15 smithi064 ceph-mon[36638]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-09T22:47:15.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:15 smithi089 ceph-mon[36665]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-09T22:47:16.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:16 smithi003 ceph-mon[32678]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-02-09T22:47:16.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:16 smithi064 ceph-mon[36638]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-02-09T22:47:16.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:16 smithi089 ceph-mon[36665]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail; 49 KiB/s, 0 objects/s recovering 2024-02-09T22:47:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:19 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:19 smithi003 ceph-mon[32678]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:19.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:19.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:19 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:19 smithi064 ceph-mon[36638]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:19.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:19.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:19.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:19 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:19.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:19 smithi089 ceph-mon[36665]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:20.771 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:20 smithi003 ceph-mon[32678]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:20 smithi064 ceph-mon[36638]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:20 smithi089 ceph-mon[36665]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:21.936 INFO:teuthology.orchestra.run.smithi003.stdout:Created osd(s) 9 on host 'smithi003' 2024-02-09T22:47:21.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.932+0000 7f02237fe700 1 -- 172.21.15.3:0/246518373 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f022c02da10 con 0x7f02180a2150 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f02180a2150 msgr2=0x7f02180a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f02180a2150 0x7f02180a4610 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0214005950 tx=0x7f021400b410 comp rx=0 tx=0).stop 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 msgr2=0x7f022c102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 0x7f022c102140 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0224007e20 tx=0x7f022400a340 comp rx=0 tx=0).stop 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 shutdown_connections 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f02180a2150 0x7f02180a4610 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f022c107a40 0x7f022c100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f022c1027e0 0x7f022c102140 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 --2- 172.21.15.3:0/246518373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f022c105110 0x7f022c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 >> 172.21.15.3:0/246518373 conn(0x7f022c0fc1f0 msgr2=0x7f022c103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 shutdown_connections 2024-02-09T22:47:21.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:47:21.935+0000 7f0230b0e700 1 -- 172.21.15.3:0/246518373 wait complete. 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:22.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:22 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:22 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:22.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:22 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:22.649 DEBUG:teuthology.orchestra.run.smithi003:osd.9> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.9.service 2024-02-09T22:47:22.652 INFO:tasks.cephadm:Deploying osd.10 on smithi064 with /dev/vg_nvme/lv_1... 2024-02-09T22:47:22.652 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-09T22:47:22.913 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:47:23.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-mon[32678]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-09T22:47:23.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-mon[32678]: osdmap e65: 10 total, 9 up, 10 in 2024-02-09T22:47:23.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-mon[32678]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:47:23.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:23.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-mon[32678]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:23.133 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T22:47:23.067+0000 7f3c7a6ab700 -1 osd.9 0 waiting for initial osdmap 2024-02-09T22:47:23.133 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 22:47:23 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T22:47:23.083+0000 7f3c76c9f700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:47:23.319 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:23 smithi064 ceph-mon[36638]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-09T22:47:23.320 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:23 smithi064 ceph-mon[36638]: osdmap e65: 10 total, 9 up, 10 in 2024-02-09T22:47:23.320 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:23 smithi064 ceph-mon[36638]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:47:23.320 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:23 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:23.320 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:23 smithi064 ceph-mon[36638]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:23 smithi089 ceph-mon[36665]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-02-09T22:47:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:23 smithi089 ceph-mon[36665]: osdmap e65: 10 total, 9 up, 10 in 2024-02-09T22:47:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:23 smithi089 ceph-mon[36665]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]: dispatch 2024-02-09T22:47:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:23 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:23 smithi089 ceph-mon[36665]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:24.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:24 smithi003 ceph-mon[32678]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:47:24.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:24 smithi003 ceph-mon[32678]: osdmap e66: 10 total, 9 up, 10 in 2024-02-09T22:47:24.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:24 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:24.234 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:24 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:24.376 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:24 smithi064 ceph-mon[36638]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:47:24.376 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:24 smithi064 ceph-mon[36638]: osdmap e66: 10 total, 9 up, 10 in 2024-02-09T22:47:24.376 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:24 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:24.376 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:24 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:24 smithi089 ceph-mon[36665]: from='osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi003", "root=default"]}]': finished 2024-02-09T22:47:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:24 smithi089 ceph-mon[36665]: osdmap e66: 10 total, 9 up, 10 in 2024-02-09T22:47:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:24 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:24 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:24.832 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-09T22:47:24.833 INFO:teuthology.orchestra.run.smithi064.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-09T22:47:24.833 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10+0 records in 2024-02-09T22:47:24.833 INFO:teuthology.orchestra.run.smithi064.stderr:10+0 records out 2024-02-09T22:47:24.833 INFO:teuthology.orchestra.run.smithi064.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111462 s, 941 MB/s 2024-02-09T22:47:24.833 INFO:teuthology.orchestra.run.smithi064.stderr:--> Zapping successful for: 2024-02-09T22:47:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:47:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:47:25.310 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] boot 2024-02-09T22:47:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: osdmap e67: 10 total, 10 up, 10 in 2024-02-09T22:47:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:25.311 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:25 smithi003 ceph-mon[32678]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:47:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:47:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] boot 2024-02-09T22:47:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: osdmap e67: 10 total, 10 up, 10 in 2024-02-09T22:47:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:25.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:25 smithi064 ceph-mon[36638]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:47:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:47:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:25.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: osd.9 [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] boot 2024-02-09T22:47:25.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: osdmap e67: 10 total, 10 up, 10 in 2024-02-09T22:47:25.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:47:25.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:25 smithi089 ceph-mon[36665]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 257 MiB used, 804 GiB / 805 GiB avail 2024-02-09T22:47:25.510 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi064:vg_nvme/lv_1 2024-02-09T22:47:25.770 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T22:47:26.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: osdmap e68: 10 total, 10 up, 10 in 2024-02-09T22:47:26.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.089 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:26 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: osdmap e68: 10 total, 10 up, 10 in 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:26.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:26 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: osdmap e68: 10 total, 10 up, 10 in 2024-02-09T22:47:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:26.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:26.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:26.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:26.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:26 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:27.125 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:27 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi003 2024-02-09T22:47:27.125 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:27 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi003 to 4273M 2024-02-09T22:47:27.125 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:27 smithi064 ceph-mon[36638]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:27.330 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 -- 172.21.15.64:0/352711900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 msgr2=0x7f96d8101060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 --2- 172.21.15.64:0/352711900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8101060 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f96d000b0d0 tx=0x7f96d000b3e0 comp rx=0 tx=0).stop 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 -- 172.21.15.64:0/352711900 shutdown_connections 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 --2- 172.21.15.64:0/352711900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d81027e0 0x7f96d8109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 --2- 172.21.15.64:0/352711900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8101060 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 --2- 172.21.15.64:0/352711900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 0x7f96d81022a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 -- 172.21.15.64:0/352711900 >> 172.21.15.64:0/352711900 conn(0x7f96d80fc1e0 msgr2=0x7f96d80fe620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 -- 172.21.15.64:0/352711900 shutdown_connections 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.324+0000 7f96e0243700 1 -- 172.21.15.64:0/352711900 wait complete. 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 Processor -- start 2024-02-09T22:47:27.331 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 -- start start 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 0x7f96d8198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d81027e0 0x7f96d819f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d819f9c0 con 0x7f96d8101e40 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d819fb00 con 0x7f96d8100c40 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96e0243700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d819fc90 con 0x7f96d81027e0 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96ddfdf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96ddfdf700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.64:38880/0 (socket says 172.21.15.64:38880) 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.325+0000 7f96ddfdf700 1 -- 172.21.15.64:0/2762750955 learned_addr learned my addr 172.21.15.64:0/2762750955 (peer_addr_for_me v2:172.21.15.64:0/0) 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96de7e0700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d81027e0 0x7f96d819f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96dd7de700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 0x7f96d8198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:27.332 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96ddfdf700 1 -- 172.21.15.64:0/2762750955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d81027e0 msgr2=0x7f96d819f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96ddfdf700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d81027e0 0x7f96d819f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96ddfdf700 1 -- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 msgr2=0x7f96d8198370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96ddfdf700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 0x7f96d8198370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96ddfdf700 1 -- 172.21.15.64:0/2762750955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d0009d00 con 0x7f96d8100c40 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96dd7de700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 0x7f96d8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96ddfdf700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8197e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000f80 tx=0x7f96d0009380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d000e040 con 0x7f96d8100c40 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96d001dd70 con 0x7f96d8100c40 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d819ff10 con 0x7f96d8100c40 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96d001be80 con 0x7f96d8100c40 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.326+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d81a02c0 con 0x7f96d8100c40 2024-02-09T22:47:27.333 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.327+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d80689e0 con 0x7f96d8100c40 2024-02-09T22:47:27.334 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.328+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f96d001b5e0 con 0x7f96d8100c40 2024-02-09T22:47:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.329+0000 7f96d6ffd700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96c00a2150 0x7f96c00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.329+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f96d00cf4d0 con 0x7f96d8100c40 2024-02-09T22:47:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.329+0000 7f96dd7de700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96c00a2150 0x7f96c00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:27.335 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.330+0000 7f96dd7de700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96c00a2150 0x7f96c00a4610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f96cc005950 tx=0x7f96cc0058e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:47:27.336 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.333+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96d0098c90 con 0x7f96d8100c40 2024-02-09T22:47:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:27 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi003 2024-02-09T22:47:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:27 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi003 to 4273M 2024-02-09T22:47:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:27 smithi003 ceph-mon[32678]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:27 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi003 2024-02-09T22:47:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:27 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi003 to 4273M 2024-02-09T22:47:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:27 smithi089 ceph-mon[36665]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:27.549 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:27.546+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f96d806a970 con 0x7f96c00a2150 2024-02-09T22:47:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:28 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:28.400 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:28.400 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:28.400 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:28 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:28 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:29.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:29 smithi003 ceph-mon[32678]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:29.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:29 smithi003 ceph-mon[32678]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:29.403 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:29 smithi064 ceph-mon[36638]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:29.403 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:29 smithi064 ceph-mon[36638]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:29.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:29 smithi089 ceph-mon[36665]: from='client.24344 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi064:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:29.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:29 smithi089 ceph-mon[36665]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:30.596 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/3258038918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]: dispatch 2024-02-09T22:47:30.597 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:30 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]: dispatch 2024-02-09T22:47:30.597 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:30 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]': finished 2024-02-09T22:47:30.597 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:30 smithi064 ceph-mon[36638]: osdmap e69: 11 total, 10 up, 11 in 2024-02-09T22:47:30.597 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:30 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/3258038918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]: dispatch 2024-02-09T22:47:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:30 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]: dispatch 2024-02-09T22:47:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:30 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]': finished 2024-02-09T22:47:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:30 smithi003 ceph-mon[32678]: osdmap e69: 11 total, 10 up, 11 in 2024-02-09T22:47:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:30 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/3258038918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]: dispatch 2024-02-09T22:47:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:30 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]: dispatch 2024-02-09T22:47:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:30 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c065623e-ebf8-4100-8d19-26bcc2a837aa"}]': finished 2024-02-09T22:47:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:30 smithi089 ceph-mon[36665]: osdmap e69: 11 total, 10 up, 11 in 2024-02-09T22:47:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:30 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:31 smithi003 ceph-mon[32678]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:31 smithi003 ceph-mon[32678]: from='client.? 172.21.15.64:0/2635816424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:31 smithi064 ceph-mon[36638]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:31 smithi064 ceph-mon[36638]: from='client.? 172.21.15.64:0/2635816424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:31 smithi089 ceph-mon[36665]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 284 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:31 smithi089 ceph-mon[36665]: from='client.? 172.21.15.64:0/2635816424' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:32 smithi003 ceph-mon[32678]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:32.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:32 smithi064 ceph-mon[36638]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:32 smithi089 ceph-mon[36665]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:35.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:35 smithi003 ceph-mon[32678]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:35 smithi064 ceph-mon[36638]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:35 smithi089 ceph-mon[36665]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:36.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:36 smithi003 ceph-mon[32678]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:36.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:36 smithi064 ceph-mon[36638]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:36.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:36 smithi089 ceph-mon[36665]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-09T22:47:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-09T22:47:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-02-09T22:47:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:39.432 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:39 smithi064 ceph-mon[36638]: Deploying daemon osd.10 on smithi064 2024-02-09T22:47:39.432 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:39 smithi064 ceph-mon[36638]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:39.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:39 smithi089 ceph-mon[36665]: Deploying daemon osd.10 on smithi064 2024-02-09T22:47:39.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:39 smithi089 ceph-mon[36665]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:39.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:39 smithi003 ceph-mon[32678]: Deploying daemon osd.10 on smithi064 2024-02-09T22:47:39.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:39 smithi003 ceph-mon[32678]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:40 smithi064 ceph-mon[36638]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:40.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:40 smithi003 ceph-mon[32678]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:40 smithi089 ceph-mon[36665]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:42.837 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:42 smithi064 ceph-mon[36638]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:42 smithi003 ceph-mon[32678]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:42 smithi089 ceph-mon[36665]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:44 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:44 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:44.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:44 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:44.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:44 smithi089 ceph-mon[36665]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:45.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:44 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:45.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:44 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:45.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:44 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:45.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:44 smithi003 ceph-mon[32678]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:45.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:44 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:45.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:44 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:45.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:44 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:45.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:44 smithi064 ceph-mon[36638]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:46.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:46 smithi003 ceph-mon[32678]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:46.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:46 smithi064 ceph-mon[36638]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:46.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:46 smithi089 ceph-mon[36665]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:47.716 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.713+0000 7f96d6ffd700 1 -- 172.21.15.64:0/2762750955 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f96d806a970 con 0x7f96c00a2150 2024-02-09T22:47:47.717 INFO:teuthology.orchestra.run.smithi064.stdout:Created osd(s) 10 on host 'smithi064' 2024-02-09T22:47:47.718 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96c00a2150 msgr2=0x7f96c00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96c00a2150 0x7f96c00a4610 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f96cc005950 tx=0x7f96cc0058e0 comp rx=0 tx=0).stop 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 msgr2=0x7f96d8197e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8197e30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f96d0000f80 tx=0x7f96d0009380 comp rx=0 tx=0).stop 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 shutdown_connections 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96c00a2150 0x7f96c00a4610 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96d81027e0 0x7f96d819f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96d8100c40 0x7f96d8197e30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 --2- 172.21.15.64:0/2762750955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96d8101e40 0x7f96d8198370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 >> 172.21.15.64:0/2762750955 conn(0x7f96d80fc1e0 msgr2=0x7f96d806a260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 shutdown_connections 2024-02-09T22:47:47.719 INFO:teuthology.orchestra.run.smithi064.stderr:2024-02-09T22:47:47.715+0000 7f96e0243700 1 -- 172.21.15.64:0/2762750955 wait complete. 2024-02-09T22:47:48.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:48.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:48.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T22:47:48.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T22:47:48.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:47 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:48.184 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.184 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.184 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:48.184 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:48.184 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:47 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:48.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:47 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:47:48.353 DEBUG:teuthology.orchestra.run.smithi064:osd.10> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.10.service 2024-02-09T22:47:48.355 INFO:tasks.cephadm:Deploying osd.11 on smithi089 with /dev/vg_nvme/lv_1... 2024-02-09T22:47:48.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-02-09T22:47:48.590 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:47:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:48 smithi064 ceph-mon[36638]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-09T22:47:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:48 smithi064 ceph-mon[36638]: osdmap e70: 11 total, 10 up, 11 in 2024-02-09T22:47:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:48 smithi064 ceph-mon[36638]: from='osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:47:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:48 smithi064 ceph-mon[36638]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:47:48.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:48 smithi064 ceph-mon[36638]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:48.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:48 smithi089 ceph-mon[36665]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-09T22:47:48.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:48 smithi089 ceph-mon[36665]: osdmap e70: 11 total, 10 up, 11 in 2024-02-09T22:47:48.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:48.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:48 smithi089 ceph-mon[36665]: from='osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:47:48.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:48 smithi089 ceph-mon[36665]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:47:48.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:48 smithi089 ceph-mon[36665]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:48 smithi003 ceph-mon[32678]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-02-09T22:47:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:48 smithi003 ceph-mon[32678]: osdmap e70: 11 total, 10 up, 11 in 2024-02-09T22:47:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:48 smithi003 ceph-mon[32678]: from='osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:47:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:48 smithi003 ceph-mon[32678]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]: dispatch 2024-02-09T22:47:49.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:48 smithi003 ceph-mon[32678]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:47:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: osdmap e71: 11 total, 10 up, 11 in 2024-02-09T22:47:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: osdmap e72: 11 total, 10 up, 11 in 2024-02-09T22:47:49.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:49.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:49 smithi089 ceph-mon[36665]: from='osd.10 ' entity='osd.10' 2024-02-09T22:47:50.028 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T22:47:49.764+0000 7fb156a86700 -1 osd.10 0 waiting for initial osdmap 2024-02-09T22:47:50.028 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T22:47:49.782+0000 7fb14f072700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: osdmap e71: 11 total, 10 up, 11 in 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: osdmap e72: 11 total, 10 up, 11 in 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.029 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:49 smithi064 ceph-mon[36638]: from='osd.10 ' entity='osd.10' 2024-02-09T22:47:50.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi064", "root=default"]}]': finished 2024-02-09T22:47:50.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: osdmap e71: 11 total, 10 up, 11 in 2024-02-09T22:47:50.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: osdmap e72: 11 total, 10 up, 11 in 2024-02-09T22:47:50.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.133 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:49 smithi003 ceph-mon[32678]: from='osd.10 ' entity='osd.10' 2024-02-09T22:47:50.596 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-02-09T22:47:50.596 INFO:teuthology.orchestra.run.smithi089.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-02-09T22:47:50.596 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10+0 records in 2024-02-09T22:47:50.596 INFO:teuthology.orchestra.run.smithi089.stderr:10+0 records out 2024-02-09T22:47:50.596 INFO:teuthology.orchestra.run.smithi089.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111778 s, 938 MB/s 2024-02-09T22:47:50.597 INFO:teuthology.orchestra.run.smithi089.stderr:--> Zapping successful for: 2024-02-09T22:47:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:47:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:47:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] boot 2024-02-09T22:47:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: osdmap e73: 11 total, 11 up, 11 in 2024-02-09T22:47:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:50 smithi064 ceph-mon[36638]: pgmap v235: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] boot 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: osdmap e73: 11 total, 11 up, 11 in 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:51.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:50 smithi003 ceph-mon[32678]: pgmap v235: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:51.159 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:47:51.160 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:47:51.160 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:51.160 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: osd.10 [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] boot 2024-02-09T22:47:51.160 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: osdmap e73: 11 total, 11 up, 11 in 2024-02-09T22:47:51.160 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:47:51.160 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:50 smithi089 ceph-mon[36665]: pgmap v235: 1 pgs: 1 remapped+peering; 577 KiB data, 285 MiB used, 894 GiB / 894 GiB avail 2024-02-09T22:47:51.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch daemon add osd smithi089:vg_nvme/lv_1 2024-02-09T22:47:51.426 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:47:52.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: osdmap e74: 11 total, 11 up, 11 in 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi064 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi064 to 4270M 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:52.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:52 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: osdmap e74: 11 total, 11 up, 11 in 2024-02-09T22:47:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi064 2024-02-09T22:47:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi064 to 4270M 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:52 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: osdmap e74: 11 total, 11 up, 11 in 2024-02-09T22:47:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi064 2024-02-09T22:47:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi064 to 4270M 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:47:52.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:52 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:47:53.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 -- 172.21.15.89:0/3991678389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c1039b0 msgr2=0x7f111c10a260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:53.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 --2- 172.21.15.89:0/3991678389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c1039b0 0x7f111c10a260 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f111400b0a0 tx=0x7f111400b3b0 comp rx=0 tx=0).stop 2024-02-09T22:47:53.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 -- 172.21.15.89:0/3991678389 shutdown_connections 2024-02-09T22:47:53.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 --2- 172.21.15.89:0/3991678389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c1039b0 0x7f111c10a260 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:53.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 --2- 172.21.15.89:0/3991678389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c100030 0x7f111c100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:53.153 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 --2- 172.21.15.89:0/3991678389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f111c103010 0x7f111c103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 -- 172.21.15.89:0/3991678389 >> 172.21.15.89:0/3991678389 conn(0x7f111c0fb970 msgr2=0x7f111c0fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 -- 172.21.15.89:0/3991678389 shutdown_connections 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.147+0000 7f1123a97700 1 -- 172.21.15.89:0/3991678389 wait complete. 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 Processor -- start 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 -- start start 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 0x7f111c102970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f111c103010 0x7f111c100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c1039b0 0x7f111c101500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c101c80 con 0x7f111c103010 2024-02-09T22:47:53.154 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c101df0 con 0x7f111c1039b0 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1123a97700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f111c1a1ee0 con 0x7f111c100030 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 0x7f111c102970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 0x7f111c102970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:35622/0 (socket says 172.21.15.89:35622) 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 -- 172.21.15.89:0/1997286333 learned_addr learned my addr 172.21.15.89:0/1997286333 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1122034700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c1039b0 0x7f111c101500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 -- 172.21.15.89:0/1997286333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c1039b0 msgr2=0x7f111c101500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c1039b0 0x7f111c101500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 -- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f111c103010 msgr2=0x7f111c100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f111c103010 0x7f111c100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1121833700 1 -- 172.21.15.89:0/1997286333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1114009d00 con 0x7f111c100030 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.148+0000 7f1122034700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c1039b0 0x7f111c101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:47:53.155 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.149+0000 7f1121833700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 0x7f111c102970 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b500 tx=0x7f110c00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:47:53.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.149+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c011830 con 0x7f111c100030 2024-02-09T22:47:53.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.149+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f111c1a21c0 con 0x7f111c100030 2024-02-09T22:47:53.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.149+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f110c011e70 con 0x7f111c100030 2024-02-09T22:47:53.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.149+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f110c00f2e0 con 0x7f111c100030 2024-02-09T22:47:53.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.149+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f111c1a2740 con 0x7f111c100030 2024-02-09T22:47:53.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.150+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f111c1091d0 con 0x7f111c100030 2024-02-09T22:47:53.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.151+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f110c0105f0 con 0x7f111c100030 2024-02-09T22:47:53.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.151+0000 7f1112ffd700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f11080a2150 0x7f11080a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:47:53.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.152+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6723+0+0 (secure 0 0 0) 0x7f110c0cd1a0 con 0x7f111c100030 2024-02-09T22:47:53.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.152+0000 7f1121032700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f11080a2150 0x7f11080a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:47:53.157 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.152+0000 7f1121032700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f11080a2150 0x7f11080a4610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f111800a990 tx=0x7f1118005c80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:47:53.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.156+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f110c0967a0 con 0x7f111c100030 2024-02-09T22:47:53.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:53 smithi089 ceph-mon[36665]: osdmap e75: 11 total, 11 up, 11 in 2024-02-09T22:47:53.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:53 smithi089 ceph-mon[36665]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:47:53.364 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:47:53.361+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f111c02d810 con 0x7f11080a2150 2024-02-09T22:47:53.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:53 smithi064 ceph-mon[36638]: osdmap e75: 11 total, 11 up, 11 in 2024-02-09T22:47:53.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:53 smithi064 ceph-mon[36638]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:47:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:53 smithi003 ceph-mon[32678]: osdmap e75: 11 total, 11 up, 11 in 2024-02-09T22:47:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:53 smithi003 ceph-mon[32678]: pgmap v238: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:47:54.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:54 smithi089 ceph-mon[36665]: osdmap e76: 11 total, 11 up, 11 in 2024-02-09T22:47:54.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:54 smithi089 ceph-mon[36665]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:54.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:54.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:54.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:54.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:54 smithi064 ceph-mon[36638]: osdmap e76: 11 total, 11 up, 11 in 2024-02-09T22:47:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:54 smithi064 ceph-mon[36638]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:54 smithi003 ceph-mon[32678]: osdmap e76: 11 total, 11 up, 11 in 2024-02-09T22:47:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:54 smithi003 ceph-mon[32678]: from='client.24361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi089:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:47:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-02-09T22:47:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-02-09T22:47:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:47:55.244 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:55 smithi089 ceph-mon[36665]: osdmap e77: 11 total, 11 up, 11 in 2024-02-09T22:47:55.244 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:55 smithi089 ceph-mon[36665]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:47:55.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:55 smithi064 ceph-mon[36638]: osdmap e77: 11 total, 11 up, 11 in 2024-02-09T22:47:55.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:55 smithi064 ceph-mon[36638]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:47:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:55 smithi003 ceph-mon[32678]: osdmap e77: 11 total, 11 up, 11 in 2024-02-09T22:47:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:55 smithi003 ceph-mon[32678]: pgmap v241: 1 pgs: 1 remapped+peering; 577 KiB data, 316 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:47:56.595 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:56 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/1132219514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]: dispatch 2024-02-09T22:47:56.595 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:56 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]: dispatch 2024-02-09T22:47:56.595 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:56 smithi089 ceph-mon[36665]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]': finished 2024-02-09T22:47:56.595 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:56 smithi089 ceph-mon[36665]: osdmap e78: 12 total, 11 up, 12 in 2024-02-09T22:47:56.596 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:47:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:56 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/1132219514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]: dispatch 2024-02-09T22:47:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:56 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]: dispatch 2024-02-09T22:47:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:56 smithi003 ceph-mon[32678]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]': finished 2024-02-09T22:47:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:56 smithi003 ceph-mon[32678]: osdmap e78: 12 total, 11 up, 12 in 2024-02-09T22:47:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:47:56.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:56 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/1132219514' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]: dispatch 2024-02-09T22:47:56.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:56 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]: dispatch 2024-02-09T22:47:56.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:56 smithi064 ceph-mon[36638]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "33affb4f-8f05-46cd-b347-fe905769c48c"}]': finished 2024-02-09T22:47:56.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:56 smithi064 ceph-mon[36638]: osdmap e78: 12 total, 11 up, 12 in 2024-02-09T22:47:56.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:47:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:57 smithi003 ceph-mon[32678]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-02-09T22:47:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:57 smithi003 ceph-mon[32678]: from='client.? 172.21.15.89:0/752230181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:57 smithi064 ceph-mon[36638]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-02-09T22:47:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:57 smithi064 ceph-mon[36638]: from='client.? 172.21.15.89:0/752230181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:57 smithi089 ceph-mon[36665]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 132 KiB/s, 0 objects/s recovering 2024-02-09T22:47:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:57 smithi089 ceph-mon[36665]: from='client.? 172.21.15.89:0/752230181' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-02-09T22:47:58.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:47:58 smithi003 ceph-mon[32678]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-09T22:47:58.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:47:58 smithi064 ceph-mon[36638]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-09T22:47:58.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:47:58 smithi089 ceph-mon[36665]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-02-09T22:48:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:00 smithi089 ceph-mon[36665]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-09T22:48:00.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:00 smithi003 ceph-mon[32678]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-09T22:48:00.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:00 smithi064 ceph-mon[36638]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-02-09T22:48:02.801 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:02 smithi089 ceph-mon[36665]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-09T22:48:02.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:02 smithi003 ceph-mon[32678]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-09T22:48:02.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:02 smithi064 ceph-mon[36638]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-02-09T22:48:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-09T22:48:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:04.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-09T22:48:04.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:04.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-02-09T22:48:04.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:05.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:05 smithi089 ceph-mon[36665]: Deploying daemon osd.11 on smithi089 2024-02-09T22:48:05.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:05 smithi089 ceph-mon[36665]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-09T22:48:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:05 smithi003 ceph-mon[32678]: Deploying daemon osd.11 on smithi089 2024-02-09T22:48:05.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:05 smithi003 ceph-mon[32678]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-09T22:48:05.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:05 smithi064 ceph-mon[36638]: Deploying daemon osd.11 on smithi089 2024-02-09T22:48:05.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:05 smithi064 ceph-mon[36638]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-02-09T22:48:06.823 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:06 smithi089 ceph-mon[36665]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:06.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:06 smithi003 ceph-mon[32678]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:06.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:06 smithi064 ceph-mon[36638]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:08.689 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:08 smithi089 ceph-mon[36665]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:08 smithi003 ceph-mon[32678]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:08 smithi064 ceph-mon[36638]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:10.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:10.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:10.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:10 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:10.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:10 smithi089 ceph-mon[36665]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:11.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:11.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:11.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:10 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:11.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:10 smithi003 ceph-mon[32678]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:11.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:11.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:11.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:10 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:11.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:10 smithi064 ceph-mon[36638]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:12.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:12 smithi003 ceph-mon[32678]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:12.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:12 smithi064 ceph-mon[36638]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:12.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:12 smithi089 ceph-mon[36665]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:13.484 INFO:teuthology.orchestra.run.smithi089.stdout:Created osd(s) 11 on host 'smithi089' 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.479+0000 7f1112ffd700 1 -- 172.21.15.89:0/1997286333 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f111c02d810 con 0x7f11080a2150 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f11080a2150 msgr2=0x7f11080a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f11080a2150 0x7f11080a4610 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f111800a990 tx=0x7f1118005c80 comp rx=0 tx=0).stop 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 msgr2=0x7f111c102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 0x7f111c102970 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f110c00b500 tx=0x7f110c00b8c0 comp rx=0 tx=0).stop 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 shutdown_connections 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f11080a2150 0x7f11080a4610 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f111c100030 0x7f111c102970 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f111c1039b0 0x7f111c101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:13.485 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 --2- 172.21.15.89:0/1997286333 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f111c103010 0x7f111c100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:13.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 >> 172.21.15.89:0/1997286333 conn(0x7f111c0fb970 msgr2=0x7f111c106310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:13.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 shutdown_connections 2024-02-09T22:48:13.486 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:13.481+0000 7f1123a97700 1 -- 172.21.15.89:0/1997286333 wait complete. 2024-02-09T22:48:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T22:48:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:13 smithi089 ceph-mon[36665]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T22:48:13.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:13 smithi003 ceph-mon[32678]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T22:48:13.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T22:48:13.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:13 smithi064 ceph-mon[36638]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-02-09T22:48:14.073 DEBUG:teuthology.orchestra.run.smithi089:osd.11> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.11.service 2024-02-09T22:48:14.075 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-02-09T22:48:14.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-09T22:48:14.329 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:48:15.694 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-mon[36665]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-09T22:48:15.694 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-mon[36665]: from='osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:48:15.694 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-mon[36665]: osdmap e79: 12 total, 11 up, 12 in 2024-02-09T22:48:15.695 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:15.695 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-mon[36665]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:48:15.695 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-mon[36665]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:15.695 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T22:48:15.458+0000 7fe6a3a00700 -1 osd.11 0 waiting for initial osdmap 2024-02-09T22:48:15.695 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 22:48:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T22:48:15.477+0000 7fe69fff4700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-02-09T22:48:15.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:15 smithi003 ceph-mon[32678]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-09T22:48:15.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:15 smithi003 ceph-mon[32678]: from='osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:48:15.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:15 smithi003 ceph-mon[32678]: osdmap e79: 12 total, 11 up, 12 in 2024-02-09T22:48:15.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:15 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:15.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:15 smithi003 ceph-mon[32678]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:48:15.741 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:15 smithi003 ceph-mon[32678]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:15.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:15 smithi064 ceph-mon[36638]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-02-09T22:48:15.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:15 smithi064 ceph-mon[36638]: from='osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:48:15.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:15 smithi064 ceph-mon[36638]: osdmap e79: 12 total, 11 up, 12 in 2024-02-09T22:48:15.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:15 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:15.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:15 smithi064 ceph-mon[36638]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]: dispatch 2024-02-09T22:48:15.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:15 smithi064 ceph-mon[36638]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 -- 172.21.15.3:0/33223188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c04100c50 msgr2=0x7f7c04101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/33223188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c04100c50 0x7f7c04101070 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009920 tx=0x7f7c00009c30 comp rx=0 tx=0).stop 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 -- 172.21.15.3:0/33223188 shutdown_connections 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/33223188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c041027f0 0x7f7c041090a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/33223188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c04100c50 0x7f7c04101070 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/33223188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c04101e50 0x7f7c041022b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 -- 172.21.15.3:0/33223188 >> 172.21.15.3:0/33223188 conn(0x7f7c040fc1f0 msgr2=0x7f7c040fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 -- 172.21.15.3:0/33223188 shutdown_connections 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 -- 172.21.15.3:0/33223188 wait complete. 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.071+0000 7f7c0c53e700 1 Processor -- start 2024-02-09T22:48:16.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 -- start start 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c04100c50 0x7f7c04197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c04101e50 0x7f7c04198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 0x7f7c0419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419fa00 con 0x7f7c041027f0 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419fb40 con 0x7f7c04101e50 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0c53e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0419fcd0 con 0x7f7c04100c50 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0aadb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 0x7f7c0419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c04100c50 0x7f7c04197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0aadb700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 0x7f7c0419f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59118/0 (socket says 172.21.15.3:59118) 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0aadb700 1 -- 172.21.15.3:0/203176978 learned_addr learned my addr 172.21.15.3:0/203176978 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c09ad9700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c04101e50 0x7f7c04198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 -- 172.21.15.3:0/203176978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c04101e50 msgr2=0x7f7c04198400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:16.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c04101e50 0x7f7c04198400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 -- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 msgr2=0x7f7c0419f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 0x7f7c0419f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 -- 172.21.15.3:0/203176978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00009520 con 0x7f7c04100c50 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0aadb700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 0x7f7c0419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.072+0000 7f7c0a2da700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c04100c50 0x7f7c04197ec0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009230 tx=0x7f7c00005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.073+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0001c070 con 0x7f7c04100c50 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.073+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c00005700 con 0x7f7c04100c50 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.073+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0419ff50 con 0x7f7c04100c50 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.073+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c00003bb0 con 0x7f7c04100c50 2024-02-09T22:48:16.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.073+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c041a02b0 con 0x7f7c04100c50 2024-02-09T22:48:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.074+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c0410c590 con 0x7f7c04100c50 2024-02-09T22:48:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.075+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f7c00003d10 con 0x7f7c04100c50 2024-02-09T22:48:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.075+0000 7f7bfb7fe700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7bf00a2150 0x7f7bf00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.075+0000 7f7c09ad9700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7bf00a2150 0x7f7bf00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.075+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6847+0+0 (secure 0 0 0) 0x7f7c000ce670 con 0x7f7c04100c50 2024-02-09T22:48:16.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.076+0000 7f7c09ad9700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7bf00a2150 0x7f7bf00a4610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4005fd0 tx=0x7f7bf4009310 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:16.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.079+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7c00097bf0 con 0x7f7c04100c50 2024-02-09T22:48:16.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.281+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f7c0410c720 con 0x7f7c04100c50 2024-02-09T22:48:16.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.282+0000 7f7bfb7fe700 1 -- 172.21.15.3:0/203176978 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) v1 ==== 74+0+133 (secure 0 0 0) 0x7f7c0009b2b0 con 0x7f7c04100c50 2024-02-09T22:48:16.286 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.283+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7bf00a2150 msgr2=0x7f7bf00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.283+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7bf00a2150 0x7f7bf00a4610 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4005fd0 tx=0x7f7bf4009310 comp rx=0 tx=0).stop 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c04100c50 msgr2=0x7f7c04197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c04100c50 0x7f7c04197ec0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7c00009230 tx=0x7f7c00005250 comp rx=0 tx=0).stop 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 shutdown_connections 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7bf00a2150 0x7f7bf00a4610 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c04100c50 0x7f7c04197ec0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c041027f0 0x7f7c0419f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 --2- 172.21.15.3:0/203176978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c04101e50 0x7f7c04198400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 >> 172.21.15.3:0/203176978 conn(0x7f7c040fc1f0 msgr2=0x7f7c04069a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 shutdown_connections 2024-02-09T22:48:16.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:16.284+0000 7f7c0c53e700 1 -- 172.21.15.3:0/203176978 wait complete. 2024-02-09T22:48:16.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:16 smithi003 ceph-mon[32678]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:48:16.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:16 smithi003 ceph-mon[32678]: osdmap e80: 12 total, 11 up, 12 in 2024-02-09T22:48:16.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:16.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:16 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:16.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:16 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/203176978' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T22:48:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:16 smithi089 ceph-mon[36665]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:48:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:16 smithi089 ceph-mon[36665]: osdmap e80: 12 total, 11 up, 12 in 2024-02-09T22:48:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:16 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:16 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/203176978' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T22:48:16.924 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":80,"num_osds":12,"num_up_osds":11,"osd_up_since":1707518870,"num_in_osds":12,"osd_in_since":1707518876,"num_remapped_pgs":0} 2024-02-09T22:48:16.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:16 smithi064 ceph-mon[36638]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi089", "root=default"]}]': finished 2024-02-09T22:48:16.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:16 smithi064 ceph-mon[36638]: osdmap e80: 12 total, 11 up, 12 in 2024-02-09T22:48:16.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:16.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:16 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:16.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:16 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/203176978' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T22:48:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: purged_snaps scrub starts 2024-02-09T22:48:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: purged_snaps scrub ok 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] boot 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: osdmap e81: 12 total, 12 up, 12 in 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:17.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:17.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:17 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: purged_snaps scrub starts 2024-02-09T22:48:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: purged_snaps scrub ok 2024-02-09T22:48:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] boot 2024-02-09T22:48:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: osdmap e81: 12 total, 12 up, 12 in 2024-02-09T22:48:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:17.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:17 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.926 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd stat -f json 2024-02-09T22:48:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: purged_snaps scrub starts 2024-02-09T22:48:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: purged_snaps scrub ok 2024-02-09T22:48:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: osd.11 [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] boot 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: osdmap e81: 12 total, 12 up, 12 in 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:48:17.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:17.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:17.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:17.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:17 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:18.169 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:48:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:18 smithi003 ceph-mon[32678]: Detected new or changed devices on smithi089 2024-02-09T22:48:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:18 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi089 to 4270M 2024-02-09T22:48:18.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:18 smithi003 ceph-mon[32678]: osdmap e82: 12 total, 12 up, 12 in 2024-02-09T22:48:18.706 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:18 smithi089 ceph-mon[36665]: Detected new or changed devices on smithi089 2024-02-09T22:48:18.707 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:18 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi089 to 4270M 2024-02-09T22:48:18.707 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:18 smithi089 ceph-mon[36665]: osdmap e82: 12 total, 12 up, 12 in 2024-02-09T22:48:18.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:18 smithi064 ceph-mon[36638]: Detected new or changed devices on smithi089 2024-02-09T22:48:18.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:18 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi089 to 4270M 2024-02-09T22:48:18.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:18 smithi064 ceph-mon[36638]: osdmap e82: 12 total, 12 up, 12 in 2024-02-09T22:48:19.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:19 smithi003 ceph-mon[32678]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:19.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 -- 172.21.15.3:0/3688043057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd88101f90 msgr2=0x7efd88108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 --2- 172.21.15.3:0/3688043057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd88101f90 0x7efd88108840 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7efd8000c240 tx=0x7efd8000c550 comp rx=0 tx=0).stop 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 -- 172.21.15.3:0/3688043057 shutdown_connections 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 --2- 172.21.15.3:0/3688043057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd88101f90 0x7efd88108840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 --2- 172.21.15.3:0/3688043057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd881003f0 0x7efd88100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 --2- 172.21.15.3:0/3688043057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd881015f0 0x7efd88101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 -- 172.21.15.3:0/3688043057 >> 172.21.15.3:0/3688043057 conn(0x7efd880fb970 msgr2=0x7efd880fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 -- 172.21.15.3:0/3688043057 shutdown_connections 2024-02-09T22:48:19.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.809+0000 7efd8c88f700 1 -- 172.21.15.3:0/3688043057 wait complete. 2024-02-09T22:48:19.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 Processor -- start 2024-02-09T22:48:19.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 -- start start 2024-02-09T22:48:19.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 0x7efd88197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:19.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd881015f0 0x7efd88198360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:19.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd88101f90 0x7efd8819f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:19.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8819f9b0 con 0x7efd881003f0 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8819faf0 con 0x7efd881015f0 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8c88f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8819fc80 con 0x7efd88101f90 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 0x7efd88197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 0x7efd88197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60726/0 (socket says 172.21.15.3:60726) 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.810+0000 7efd8659c700 1 -- 172.21.15.3:0/711287407 learned_addr learned my addr 172.21.15.3:0/711287407 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd86d9d700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd88101f90 0x7efd8819f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd85d9b700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd881015f0 0x7efd88198360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:19.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8659c700 1 -- 172.21.15.3:0/711287407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd88101f90 msgr2=0x7efd8819f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8659c700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd88101f90 0x7efd8819f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8659c700 1 -- 172.21.15.3:0/711287407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd881015f0 msgr2=0x7efd88198360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8659c700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd881015f0 0x7efd88198360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8659c700 1 -- 172.21.15.3:0/711287407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8000ae10 con 0x7efd881003f0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd86d9d700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd88101f90 0x7efd8819f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8659c700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 0x7efd88197e20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efd7800e9a0 tx=0x7efd7800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78009790 con 0x7efd881003f0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8819ff60 con 0x7efd881003f0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd7800ccf0 con 0x7efd881003f0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd78003d20 con 0x7efd881003f0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.811+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd881a0380 con 0x7efd881003f0 2024-02-09T22:48:19.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.812+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd88068a10 con 0x7efd881003f0 2024-02-09T22:48:19.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.813+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7efd78004390 con 0x7efd881003f0 2024-02-09T22:48:19.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.814+0000 7efd7f7fe700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd740a21d0 0x7efd740a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:19.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.814+0000 7efd85d9b700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd740a21d0 0x7efd740a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:19.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.814+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efd78014070 con 0x7efd881003f0 2024-02-09T22:48:19.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.814+0000 7efd85d9b700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd740a21d0 0x7efd740a4690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd70009c60 tx=0x7efd70009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:19.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:19.818+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd78096f10 con 0x7efd881003f0 2024-02-09T22:48:19.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:19 smithi064 ceph-mon[36638]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:19.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:19 smithi089 ceph-mon[36665]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.022+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7efd8810b7d0 con 0x7efd881003f0 2024-02-09T22:48:20.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.022+0000 7efd7f7fe700 1 -- 172.21.15.3:0/711287407 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v82) v1 ==== 74+0+133 (secure 0 0 0) 0x7efd78003e80 con 0x7efd881003f0 2024-02-09T22:48:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd740a21d0 msgr2=0x7efd740a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd740a21d0 0x7efd740a4690 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7efd70009c60 tx=0x7efd70009260 comp rx=0 tx=0).stop 2024-02-09T22:48:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 msgr2=0x7efd88197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 0x7efd88197e20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efd7800e9a0 tx=0x7efd7800ecb0 comp rx=0 tx=0).stop 2024-02-09T22:48:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 shutdown_connections 2024-02-09T22:48:20.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd740a21d0 0x7efd740a4690 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd88101f90 0x7efd8819f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd881003f0 0x7efd88197e20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 --2- 172.21.15.3:0/711287407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd881015f0 0x7efd88198360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 >> 172.21.15.3:0/711287407 conn(0x7efd880fb970 msgr2=0x7efd88106aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 shutdown_connections 2024-02-09T22:48:20.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:20.024+0000 7efd8c88f700 1 -- 172.21.15.3:0/711287407 wait complete. 2024-02-09T22:48:20.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:20 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/711287407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T22:48:20.750 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":82,"num_osds":12,"num_up_osds":12,"osd_up_since":1707518896,"num_in_osds":12,"osd_in_since":1707518876,"num_remapped_pgs":0} 2024-02-09T22:48:20.751 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-09T22:48:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:20 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/711287407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T22:48:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:20 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/711287407' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-02-09T22:48:20.995 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:48:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:21 smithi003 ceph-mon[32678]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:21.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:21 smithi064 ceph-mon[36638]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:21.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:21 smithi089 ceph-mon[36665]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:22.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 -- 172.21.15.3:0/2681740435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 msgr2=0x7f92d01023b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:22.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 --2- 172.21.15.3:0/2681740435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01023b0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009990 tx=0x7f92cc009ca0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 -- 172.21.15.3:0/2681740435 shutdown_connections 2024-02-09T22:48:22.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 --2- 172.21.15.3:0/2681740435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d01070a0 0x7f92d010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 --2- 172.21.15.3:0/2681740435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01023b0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 --2- 172.21.15.3:0/2681740435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0102980 0x7f92d01069c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 -- 172.21.15.3:0/2681740435 >> 172.21.15.3:0/2681740435 conn(0x7f92d0078e80 msgr2=0x7f92d0079290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:22.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 -- 172.21.15.3:0/2681740435 shutdown_connections 2024-02-09T22:48:22.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.609+0000 7f92d787f700 1 -- 172.21.15.3:0/2681740435 wait complete. 2024-02-09T22:48:22.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 Processor -- start 2024-02-09T22:48:22.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 -- start start 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0102980 0x7f92d01987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d01070a0 0x7f92d019cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d019d180 con 0x7f92d0101fd0 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d019d2f0 con 0x7f92d01070a0 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d787f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d019d4b0 con 0x7f92d0102980 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60756/0 (socket says 172.21.15.3:60756) 2024-02-09T22:48:22.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 -- 172.21.15.3:0/3979607286 learned_addr learned my addr 172.21.15.3:0/3979607286 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d5e1c700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d01070a0 0x7f92d019cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d4e1a700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0102980 0x7f92d01987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 -- 172.21.15.3:0/3979607286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0102980 msgr2=0x7f92d01987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0102980 0x7f92d01987e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 -- 172.21.15.3:0/3979607286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d01070a0 msgr2=0x7f92d019cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d01070a0 0x7f92d019cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 -- 172.21.15.3:0/3979607286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0095f0 con 0x7f92d0101fd0 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.610+0000 7f92d561b700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01982a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000c00 tx=0x7f92cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:22.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.611+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc01d070 con 0x7f92d0101fd0 2024-02-09T22:48:22.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.611+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d019d730 con 0x7f92d0101fd0 2024-02-09T22:48:22.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.611+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92cc022470 con 0x7f92d0101fd0 2024-02-09T22:48:22.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.611+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92cc017650 con 0x7f92d0101fd0 2024-02-09T22:48:22.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.611+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d01a4ff0 con 0x7f92d0101fd0 2024-02-09T22:48:22.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.612+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d010a2d0 con 0x7f92d0101fd0 2024-02-09T22:48:22.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.613+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f92cc0225e0 con 0x7f92d0101fd0 2024-02-09T22:48:22.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.613+0000 7f92c67fc700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f92bc0a2150 0x7f92bc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:22.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.614+0000 7f92d4e1a700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f92bc0a2150 0x7f92bc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:22.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.614+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f92cc0ceff0 con 0x7f92d0101fd0 2024-02-09T22:48:22.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.614+0000 7f92d4e1a700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f92bc0a2150 0x7f92bc0a4610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f92d0101990 tx=0x7f92c0009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:22.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.617+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f92cc0984f0 con 0x7f92d0101fd0 2024-02-09T22:48:22.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.821+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92d00689e0 con 0x7f92d0101fd0 2024-02-09T22:48:22.827 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:22 smithi003 ceph-mon[32678]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:22.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.822+0000 7f92c67fc700 1 -- 172.21.15.3:0/3979607286 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18265 (secure 0 0 0) 0x7f92cc09bbb0 con 0x7f92d0101fd0 2024-02-09T22:48:22.828 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:22.828 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":82,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:48:17.465165+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f92bc0a2150 msgr2=0x7f92bc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f92bc0a2150 0x7f92bc0a4610 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f92d0101990 tx=0x7f92c0009260 comp rx=0 tx=0).stop 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 msgr2=0x7f92d01982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01982a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f92cc000c00 tx=0x7f92cc004730 comp rx=0 tx=0).stop 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 shutdown_connections 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f92bc0a2150 0x7f92bc0a4610 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d01070a0 0x7f92d019cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0101fd0 0x7f92d01982a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 --2- 172.21.15.3:0/3979607286 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0102980 0x7f92d01987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.824+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 >> 172.21.15.3:0/3979607286 conn(0x7f92d0078e80 msgr2=0x7f92d0106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.825+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 shutdown_connections 2024-02-09T22:48:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:22.825+0000 7f92d787f700 1 -- 172.21.15.3:0/3979607286 wait complete. 2024-02-09T22:48:22.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:22 smithi064 ceph-mon[36638]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:22.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:22 smithi089 ceph-mon[36665]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:23.415 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:48:23.416 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-02-09T22:48:23.656 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:48:23.765 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:23 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3979607286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:48:23.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:23 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3979607286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:48:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:23 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3979607286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 -- 172.21.15.3:0/2220078747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 msgr2=0x7f9dfc10eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 --2- 172.21.15.3:0/2220078747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc10eb70 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9df400b0d0 tx=0x7f9df400b3e0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 -- 172.21.15.3:0/2220078747 shutdown_connections 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 --2- 172.21.15.3:0/2220078747 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc10eb70 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 --2- 172.21.15.3:0/2220078747 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dfc107d50 0x7f9dfc108130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 --2- 172.21.15.3:0/2220078747 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 0x7f9dfc1021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 -- 172.21.15.3:0/2220078747 >> 172.21.15.3:0/2220078747 conn(0x7f9dfc078e50 msgr2=0x7f9dfc079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 -- 172.21.15.3:0/2220078747 shutdown_connections 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.201+0000 7f9e019da700 1 -- 172.21.15.3:0/2220078747 wait complete. 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 Processor -- start 2024-02-09T22:48:25.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 -- start start 2024-02-09T22:48:25.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 0x7f9dfc1982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:25.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:25.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dfc107d50 0x7f9dfc19cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:25.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc19d170 con 0x7f9dfc102730 2024-02-09T22:48:25.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc19d2e0 con 0x7f9dfc107d50 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9e019da700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dfc19d4a0 con 0x7f9dfc101d70 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9dfa7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9dfa7fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc198820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60788/0 (socket says 172.21.15.3:60788) 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9dfa7fc700 1 -- 172.21.15.3:0/3036692231 learned_addr learned my addr 172.21.15.3:0/3036692231 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9dfb7fe700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dfc107d50 0x7f9dfc19cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.202+0000 7f9dfaffd700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 0x7f9dfc1982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfa7fc700 1 -- 172.21.15.3:0/3036692231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 msgr2=0x7f9dfc1982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfa7fc700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 0x7f9dfc1982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfa7fc700 1 -- 172.21.15.3:0/3036692231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dfc107d50 msgr2=0x7f9dfc19cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfa7fc700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dfc107d50 0x7f9dfc19cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfa7fc700 1 -- 172.21.15.3:0/3036692231 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df4009d00 con 0x7f9dfc102730 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfaffd700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 0x7f9dfc1982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:48:25.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9dfa7fc700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc198820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9df000b530 tx=0x7f9df000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0011830 con 0x7f9dfc102730 2024-02-09T22:48:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc19d780 con 0x7f9dfc102730 2024-02-09T22:48:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9df0011e70 con 0x7f9dfc102730 2024-02-09T22:48:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df000f340 con 0x7f9dfc102730 2024-02-09T22:48:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.203+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc1a4fe0 con 0x7f9dfc102730 2024-02-09T22:48:25.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.204+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc10a230 con 0x7f9dfc102730 2024-02-09T22:48:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.206+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f9df00106d0 con 0x7f9dfc102730 2024-02-09T22:48:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.206+0000 7f9e009d8700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9de40a2150 0x7f9de40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.207+0000 7f9dfaffd700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9de40a2150 0x7f9de40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.207+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9df00cd7d0 con 0x7f9dfc102730 2024-02-09T22:48:25.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.207+0000 7f9dfaffd700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9de40a2150 0x7f9de40a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005fd0 tx=0x7f9dec005e20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:25.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.210+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9df0096c40 con 0x7f9dfc102730 2024-02-09T22:48:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:25 smithi003 ceph-mon[32678]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:25.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.415+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9dfc111dd0 con 0x7f9dfc102730 2024-02-09T22:48:25.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.415+0000 7f9e009d8700 1 -- 172.21.15.3:0/3036692231 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9df009a300 con 0x7f9dfc102730 2024-02-09T22:48:25.419 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:48:25.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9de40a2150 msgr2=0x7f9de40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9de40a2150 0x7f9de40a4610 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9dec005fd0 tx=0x7f9dec005e20 comp rx=0 tx=0).stop 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 msgr2=0x7f9dfc198820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc198820 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9df000b530 tx=0x7f9df000b840 comp rx=0 tx=0).stop 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 shutdown_connections 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9de40a2150 0x7f9de40a4610 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9dfc101d70 0x7f9dfc1982e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9dfc107d50 0x7f9dfc19cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.417+0000 7f9e019da700 1 --2- 172.21.15.3:0/3036692231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9dfc102730 0x7f9dfc198820 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.418+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 >> 172.21.15.3:0/3036692231 conn(0x7f9dfc078e50 msgr2=0x7f9dfc1007f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.418+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 shutdown_connections 2024-02-09T22:48:25.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:25.418+0000 7f9e019da700 1 -- 172.21.15.3:0/3036692231 wait complete. 2024-02-09T22:48:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:25 smithi064 ceph-mon[36638]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:25 smithi089 ceph-mon[36665]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:26.084 INFO:tasks.cephadm:Adding mds.a on smithi003 2024-02-09T22:48:26.084 INFO:tasks.cephadm:Adding mds.d on smithi003 2024-02-09T22:48:26.084 INFO:tasks.cephadm:Adding mds.g on smithi003 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.j on smithi003 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.b on smithi064 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.e on smithi064 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.h on smithi064 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.k on smithi064 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.c on smithi089 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.f on smithi089 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.i on smithi089 2024-02-09T22:48:26.085 INFO:tasks.cephadm:Adding mds.l on smithi089 2024-02-09T22:48:26.085 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph orch apply mds all '12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l' 2024-02-09T22:48:26.335 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T22:48:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:26 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3036692231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:48:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:26 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3036692231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:48:26.457 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:26 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3036692231' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:48:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:27 smithi003 ceph-mon[32678]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:27.419 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:27 smithi089 ceph-mon[36665]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:27.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:27 smithi064 ceph-mon[36638]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:27.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 -- 172.21.15.89:0/2677616112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 msgr2=0x7f65001075a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:27.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 --2- 172.21.15.89:0/2677616112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f65001075a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f64f4009990 tx=0x7f64f4009ca0 comp rx=0 tx=0).stop 2024-02-09T22:48:27.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 -- 172.21.15.89:0/2677616112 shutdown_connections 2024-02-09T22:48:27.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 --2- 172.21.15.89:0/2677616112 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500107ae0 0x7f6500109ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:27.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 --2- 172.21.15.89:0/2677616112 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6500102880 0x7f6500104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:27.941 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 --2- 172.21.15.89:0/2677616112 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f65001075a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 -- 172.21.15.89:0/2677616112 >> 172.21.15.89:0/2677616112 conn(0x7f65000fc1f0 msgr2=0x7f65000fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 -- 172.21.15.89:0/2677616112 shutdown_connections 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.935+0000 7f6505a2a700 1 -- 172.21.15.89:0/2677616112 wait complete. 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 Processor -- start 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 -- start start 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500102880 0x7f6500197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f6500198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6500107ae0 0x7f650019f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650019f8c0 con 0x7f6500107ae0 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650019fa00 con 0x7f6500102880 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f6505a2a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f650019fb90 con 0x7f65001051b0 2024-02-09T22:48:27.942 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f64fe7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f6500198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f64fe7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f6500198270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.89:46206/0 (socket says 172.21.15.89:46206) 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.936+0000 7f64fe7fc700 1 -- 172.21.15.89:0/4246712485 learned_addr learned my addr 172.21.15.89:0/4246712485 (peer_addr_for_me v2:172.21.15.89:0/0) 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64feffd700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500102880 0x7f6500197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64ff7fe700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6500107ae0 0x7f650019f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64fe7fc700 1 -- 172.21.15.89:0/4246712485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500102880 msgr2=0x7f6500197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64fe7fc700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500102880 0x7f6500197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64fe7fc700 1 -- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6500107ae0 msgr2=0x7f650019f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64fe7fc700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6500107ae0 0x7f650019f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64fe7fc700 1 -- 172.21.15.89:0/4246712485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40095f0 con 0x7f65001051b0 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64feffd700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500102880 0x7f6500197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f64fe7fc700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f6500198270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f64f400b5c0 tx=0x7f64f400f690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:27.943 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f401d070 con 0x7f65001051b0 2024-02-09T22:48:27.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64f400fb40 con 0x7f65001051b0 2024-02-09T22:48:27.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f650019fe10 con 0x7f65001051b0 2024-02-09T22:48:27.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f4017650 con 0x7f65001051b0 2024-02-09T22:48:27.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.937+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65001a01c0 con 0x7f65001051b0 2024-02-09T22:48:27.944 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.938+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65000689e0 con 0x7f65001051b0 2024-02-09T22:48:27.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.939+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f64f400fcb0 con 0x7f65001051b0 2024-02-09T22:48:27.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.940+0000 7f6504a28700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64e80a2150 0x7f64e80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:27.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.940+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f64f40cf0e0 con 0x7f65001051b0 2024-02-09T22:48:27.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.940+0000 7f64feffd700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64e80a2150 0x7f64e80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:27.945 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.940+0000 7f64feffd700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64e80a2150 0x7f64e80a4610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f64f0005950 tx=0x7f64f0005f50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:27.946 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:27.943+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f64f40985e0 con 0x7f65001051b0 2024-02-09T22:48:28.148 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.145+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f650002d960 con 0x7f64e80a2150 2024-02-09T22:48:28.156 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.153+0000 7f6504a28700 1 -- 172.21.15.89:0/4246712485 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f650002d960 con 0x7f64e80a2150 2024-02-09T22:48:28.156 INFO:teuthology.orchestra.run.smithi089.stdout:Scheduled mds.all update... 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64e80a2150 msgr2=0x7f64e80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64e80a2150 0x7f64e80a4610 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f64f0005950 tx=0x7f64f0005f50 comp rx=0 tx=0).stop 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 msgr2=0x7f6500198270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f6500198270 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f64f400b5c0 tx=0x7f64f400f690 comp rx=0 tx=0).stop 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 shutdown_connections 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64e80a2150 0x7f64e80a4610 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6500107ae0 0x7f650019f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6500102880 0x7f6500197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 --2- 172.21.15.89:0/4246712485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f65001051b0 0x7f6500198270 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:28.158 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 >> 172.21.15.89:0/4246712485 conn(0x7f65000fc1f0 msgr2=0x7f65001032c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 shutdown_connections 2024-02-09T22:48:28.159 INFO:teuthology.orchestra.run.smithi089.stderr:2024-02-09T22:48:28.155+0000 7f6505a2a700 1 -- 172.21.15.89:0/4246712485 wait complete. 2024-02-09T22:48:28.768 DEBUG:teuthology.orchestra.run.smithi003:mds.a> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.a.service 2024-02-09T22:48:28.771 DEBUG:teuthology.orchestra.run.smithi003:mds.d> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.d.service 2024-02-09T22:48:28.814 DEBUG:teuthology.orchestra.run.smithi003:mds.g> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.g.service 2024-02-09T22:48:28.856 DEBUG:teuthology.orchestra.run.smithi003:mds.j> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.j.service 2024-02-09T22:48:28.903 DEBUG:teuthology.orchestra.run.smithi064:mds.b> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.b.service 2024-02-09T22:48:28.905 DEBUG:teuthology.orchestra.run.smithi064:mds.e> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.e.service 2024-02-09T22:48:28.948 DEBUG:teuthology.orchestra.run.smithi064:mds.h> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.h.service 2024-02-09T22:48:28.992 DEBUG:teuthology.orchestra.run.smithi064:mds.k> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.k.service 2024-02-09T22:48:29.035 DEBUG:teuthology.orchestra.run.smithi089:mds.c> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.c.service 2024-02-09T22:48:29.037 DEBUG:teuthology.orchestra.run.smithi089:mds.f> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.f.service 2024-02-09T22:48:29.080 DEBUG:teuthology.orchestra.run.smithi089:mds.i> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.i.service 2024-02-09T22:48:29.123 DEBUG:teuthology.orchestra.run.smithi089:mds.l> sudo journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.l.service 2024-02-09T22:48:29.166 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-02-09T22:48:29.166 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:48:29.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:48:29.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l;count:12 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:29.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:29 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 -- 172.21.15.3:0/3218704704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b741001c0 msgr2=0x7f7b741005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 --2- 172.21.15.3:0/3218704704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b741001c0 0x7f7b741005e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009900 tx=0x7f7b6c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 -- 172.21.15.3:0/3218704704 shutdown_connections 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 --2- 172.21.15.3:0/3218704704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b74101d70 0x7f7b74108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 --2- 172.21.15.3:0/3218704704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b741013b0 0x7f7b74101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 --2- 172.21.15.3:0/3218704704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b741001c0 0x7f7b741005e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 -- 172.21.15.3:0/3218704704 >> 172.21.15.3:0/3218704704 conn(0x7f7b740fb710 msgr2=0x7f7b740fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 -- 172.21.15.3:0/3218704704 shutdown_connections 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.277+0000 7f7b79935640 1 -- 172.21.15.3:0/3218704704 wait complete. 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 Processor -- start 2024-02-09T22:48:29.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 -- start start 2024-02-09T22:48:29.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b741001c0 0x7f7b741961c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b741013b0 0x7f7b74196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 0x7f7b7419d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7419dcb0 con 0x7f7b74101d70 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7419de20 con 0x7f7b741001c0 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b79935640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b7419dfe0 con 0x7f7b741013b0 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 0x7f7b7419d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 0x7f7b7419d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38874/0 (socket says 172.21.15.3:38874) 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b737fe640 1 -- 172.21.15.3:0/1250929028 learned_addr learned my addr 172.21.15.3:0/1250929028 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.278+0000 7f7b727fc640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b741013b0 0x7f7b74196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b72ffd640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b741001c0 0x7f7b741961c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b737fe640 1 -- 172.21.15.3:0/1250929028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b741013b0 msgr2=0x7f7b74196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b737fe640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b741013b0 0x7f7b74196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b737fe640 1 -- 172.21.15.3:0/1250929028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b741001c0 msgr2=0x7f7b741961c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b737fe640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b741001c0 0x7f7b741961c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b737fe640 1 -- 172.21.15.3:0/1250929028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c009560 con 0x7f7b74101d70 2024-02-09T22:48:29.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b737fe640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 0x7f7b7419d770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400eb10 tx=0x7f7b6400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64004b20 con 0x7f7b74101d70 2024-02-09T22:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b7419e2c0 con 0x7f7b74101d70 2024-02-09T22:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b64004c80 con 0x7f7b74101d70 2024-02-09T22:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b64010650 con 0x7f7b74101d70 2024-02-09T22:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.279+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b7419e780 con 0x7f7b74101d70 2024-02-09T22:48:29.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.280+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b741074e0 con 0x7f7b74101d70 2024-02-09T22:48:29.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.281+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f7b64005020 con 0x7f7b74101d70 2024-02-09T22:48:29.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.282+0000 7f7b78933640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7b5c0a20c0 0x7f7b5c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.282+0000 7f7b72ffd640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7b5c0a20c0 0x7f7b5c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.282+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f7b64014070 con 0x7f7b74101d70 2024-02-09T22:48:29.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.282+0000 7f7b72ffd640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7b5c0a20c0 0x7f7b5c0a4580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c006010 tx=0x7f7b6c000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:29.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.286+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7b640972e0 con 0x7f7b74101d70 2024-02-09T22:48:29.454 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l;count:12 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:29.455 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:29 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:29.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.470+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b7402ce50 con 0x7f7b74101d70 2024-02-09T22:48:29.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.471+0000 7f7b78933640 1 -- 172.21.15.3:0/1250929028 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18265 (secure 0 0 0) 0x7f7b6409a9a0 con 0x7f7b74101d70 2024-02-09T22:48:29.476 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:29.477 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":82,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:48:17.465165+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:48:29.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7b5c0a20c0 msgr2=0x7f7b5c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7b5c0a20c0 0x7f7b5c0a4580 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c006010 tx=0x7f7b6c000bc0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 msgr2=0x7f7b7419d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 0x7f7b7419d770 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7b6400eb10 tx=0x7f7b6400c2d0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 shutdown_connections 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7b5c0a20c0 0x7f7b5c0a4580 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7b74101d70 0x7f7b7419d770 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7b741013b0 0x7f7b74196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 --2- 172.21.15.3:0/1250929028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7b741001c0 0x7f7b741961c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 >> 172.21.15.3:0/1250929028 conn(0x7f7b740fb710 msgr2=0x7f7b741046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 shutdown_connections 2024-02-09T22:48:29.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.473+0000 7f7b79935640 1 -- 172.21.15.3:0/1250929028 wait complete. 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: Saving service mds.all spec with placement smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l;count:12 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:29.486 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:29 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:29.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:48:29.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:48:29.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 -- 172.21.15.3:0/1375585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101d00 msgr2=0x7f24f8108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/1375585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101d00 0x7f24f8108590 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f24f0009ca0 tx=0x7f24f002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 -- 172.21.15.3:0/1375585609 shutdown_connections 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/1375585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101d00 0x7f24f8108590 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/1375585609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24f8101340 0x7f24f81017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/1375585609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 0x7f24f8100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 -- 172.21.15.3:0/1375585609 >> 172.21.15.3:0/1375585609 conn(0x7f24f80fb6c0 msgr2=0x7f24f80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 -- 172.21.15.3:0/1375585609 shutdown_connections 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.590+0000 7f24fd8ff640 1 -- 172.21.15.3:0/1375585609 wait complete. 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 Processor -- start 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 -- start start 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 0x7f24f8197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 0x7f24f81977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24f8101d00 0x7f24f819e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f819ed70 con 0x7f24f8101340 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f819eee0 con 0x7f24f8100150 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24fd8ff640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f819f0a0 con 0x7f24f8101d00 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24f67fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 0x7f24f81977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24f67fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 0x7f24f81977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38884/0 (socket says 172.21.15.3:38884) 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24f67fc640 1 -- 172.21.15.3:0/4259828998 learned_addr learned my addr 172.21.15.3:0/4259828998 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.591+0000 7f24f77fe640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24f8101d00 0x7f24f819e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f6ffd640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 0x7f24f8197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f67fc640 1 -- 172.21.15.3:0/4259828998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24f8101d00 msgr2=0x7f24f819e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f67fc640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24f8101d00 0x7f24f819e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f67fc640 1 -- 172.21.15.3:0/4259828998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 msgr2=0x7f24f8197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f67fc640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 0x7f24f8197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f67fc640 1 -- 172.21.15.3:0/4259828998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f0009900 con 0x7f24f8101340 2024-02-09T22:48:29.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f6ffd640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 0x7f24f8197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:48:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24f67fc640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 0x7f24f81977c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00e9f0 tx=0x7f24ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec009720 con 0x7f24f8101340 2024-02-09T22:48:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24ec00cd80 con 0x7f24f8101340 2024-02-09T22:48:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f819f380 con 0x7f24f8101340 2024-02-09T22:48:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec010690 con 0x7f24f8101340 2024-02-09T22:48:29.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.592+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f819f840 con 0x7f24f8101340 2024-02-09T22:48:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.593+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8107470 con 0x7f24f8101340 2024-02-09T22:48:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.595+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f24ec0046c0 con 0x7f24f8101340 2024-02-09T22:48:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.595+0000 7f24fc8fd640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f24e00a20c0 0x7f24e00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.595+0000 7f24f6ffd640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f24e00a20c0 0x7f24e00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.595+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f24ec014070 con 0x7f24f8101340 2024-02-09T22:48:29.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.596+0000 7f24f6ffd640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f24e00a20c0 0x7f24e00a4580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f24f81011a0 tx=0x7f24e8005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:29.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.599+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f24ec0972b0 con 0x7f24f8101340 2024-02-09T22:48:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.780+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f24f810cdf0 con 0x7f24f8101340 2024-02-09T22:48:29.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.780+0000 7f24fc8fd640 1 -- 172.21.15.3:0/4259828998 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f24f810cdf0 con 0x7f24f8101340 2024-02-09T22:48:29.783 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f24e00a20c0 msgr2=0x7f24e00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f24e00a20c0 0x7f24e00a4580 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f24f81011a0 tx=0x7f24e8005e70 comp rx=0 tx=0).stop 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 msgr2=0x7f24f81977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 0x7f24f81977c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f24ec00e9f0 tx=0x7f24ec00edf0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 shutdown_connections 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f24e00a20c0 0x7f24e00a4580 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24f8101d00 0x7f24f819e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24f8101340 0x7f24f81977c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 --2- 172.21.15.3:0/4259828998 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24f8100150 0x7f24f8197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.782+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 >> 172.21.15.3:0/4259828998 conn(0x7f24f80fb6c0 msgr2=0x7f24f8104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:29.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.783+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 shutdown_connections 2024-02-09T22:48:29.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.783+0000 7f24fd8ff640 1 -- 172.21.15.3:0/4259828998 wait complete. 2024-02-09T22:48:29.797 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:29.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 -- 172.21.15.3:0/1995197453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101024b0 msgr2=0x7fd710102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 --2- 172.21.15.3:0/1995197453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101024b0 0x7fd710102930 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd700009900 tx=0x7fd70002bc30 comp rx=0 tx=0).stop 2024-02-09T22:48:29.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 -- 172.21.15.3:0/1995197453 shutdown_connections 2024-02-09T22:48:29.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 --2- 172.21.15.3:0/1995197453 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd710102e70 0x7fd710109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 --2- 172.21.15.3:0/1995197453 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101024b0 0x7fd710102930 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 --2- 172.21.15.3:0/1995197453 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd7101012c0 0x7fd7101016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 -- 172.21.15.3:0/1995197453 >> 172.21.15.3:0/1995197453 conn(0x7fd7100fc810 msgr2=0x7fd7100fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 -- 172.21.15.3:0/1995197453 shutdown_connections 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 -- 172.21.15.3:0/1995197453 wait complete. 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.896+0000 7fd717b4b640 1 Processor -- start 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 -- start start 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 0x7fd7101972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7101024b0 0x7fd7101977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd710102e70 0x7fd71019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71019ed90 con 0x7fd7101012c0 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71019ef00 con 0x7fd7101024b0 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71019f0c0 con 0x7fd710102e70 2024-02-09T22:48:29.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 0x7fd7101972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 0x7fd7101972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38900/0 (socket says 172.21.15.3:38900) 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 -- 172.21.15.3:0/3000540782 learned_addr learned my addr 172.21.15.3:0/3000540782 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7150e6640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7101024b0 0x7fd7101977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7160e8640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd710102e70 0x7fd71019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 -- 172.21.15.3:0/3000540782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd710102e70 msgr2=0x7fd71019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd710102e70 0x7fd71019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 -- 172.21.15.3:0/3000540782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7101024b0 msgr2=0x7fd7101977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7101024b0 0x7fd7101977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 -- 172.21.15.3:0/3000540782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd700009560 con 0x7fd7101012c0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7150e6640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7101024b0 0x7fd7101977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd7158e7640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 0x7fd7101972a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd70800c970 tx=0x7fd70800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd708004220 con 0x7fd7101012c0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd708007e90 con 0x7fd7101012c0 2024-02-09T22:48:29.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd71019f3a0 con 0x7fd7101012c0 2024-02-09T22:48:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd70800f450 con 0x7fd7101012c0 2024-02-09T22:48:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.897+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd71019f900 con 0x7fd7101012c0 2024-02-09T22:48:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.898+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7101085e0 con 0x7fd7101012c0 2024-02-09T22:48:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.899+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fd708003610 con 0x7fd7101012c0 2024-02-09T22:48:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.900+0000 7fd706ffd640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd6fc0a2140 0x7fd6fc0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:29.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.900+0000 7fd7150e6640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd6fc0a2140 0x7fd6fc0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:29.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.900+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd7080cd500 con 0x7fd7101012c0 2024-02-09T22:48:29.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.900+0000 7fd7150e6640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd6fc0a2140 0x7fd6fc0a4600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd7101987b0 tx=0x7fd700005be0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:29.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:29.904+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fd708096970 con 0x7fd7101012c0 2024-02-09T22:48:30.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.121+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd71010cc60 con 0x7fd7101012c0 2024-02-09T22:48:30.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.121+0000 7fd706ffd640 1 -- 172.21.15.3:0/3000540782 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fd70809a030 con 0x7fd7101012c0 2024-02-09T22:48:30.124 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:30.124 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-02-09T22:48:30.124 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 1 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd6fc0a2140 msgr2=0x7fd6fc0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd6fc0a2140 0x7fd6fc0a4600 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd7101987b0 tx=0x7fd700005be0 comp rx=0 tx=0).stop 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 msgr2=0x7fd7101972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 0x7fd7101972a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd70800c970 tx=0x7fd70800ce30 comp rx=0 tx=0).stop 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 shutdown_connections 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd6fc0a2140 0x7fd6fc0a4600 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:30.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd7101024b0 0x7fd7101977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd710102e70 0x7fd71019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 --2- 172.21.15.3:0/3000540782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd7101012c0 0x7fd7101972a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 >> 172.21.15.3:0/3000540782 conn(0x7fd7100fc810 msgr2=0x7fd7101057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 shutdown_connections 2024-02-09T22:48:30.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:30.123+0000 7fd717b4b640 1 -- 172.21.15.3:0/3000540782 wait complete. 2024-02-09T22:48:30.403 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:30 smithi003 ceph-mon[32678]: Deploying daemon mds.l on smithi089 2024-02-09T22:48:30.403 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1250929028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:48:30.403 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4259828998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:48:30.403 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3000540782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:30 smithi064 ceph-mon[36638]: Deploying daemon mds.l on smithi089 2024-02-09T22:48:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1250929028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:48:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4259828998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:48:30.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3000540782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:30 smithi089 ceph-mon[36665]: Deploying daemon mds.l on smithi089 2024-02-09T22:48:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1250929028' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:48:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4259828998' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:48:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3000540782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:30.978 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:30 smithi089 systemd[1]: Starting Ceph mds.l for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:48:31.238 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:31 smithi089 ceph-mon[36665]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:31.239 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:30 smithi089 podman[66386]: 2024-02-09 22:48:30.881356086 +0000 UTC m=+0.017579842 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:48:31.239 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:30 smithi089 podman[66386]: 2024-02-09 22:48:30.991315398 +0000 UTC m=+0.127539152 container create d6648ce98dd6b452b38ef29bd6a940a54e38eae42f527b539968cff245673c39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-09T22:48:31.239 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:31 smithi089 podman[66386]: 2024-02-09 22:48:31.210546237 +0000 UTC m=+0.346769978 container init d6648ce98dd6b452b38ef29bd6a940a54e38eae42f527b539968cff245673c39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T22:48:31.239 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:31 smithi089 podman[66386]: 2024-02-09 22:48:31.21324794 +0000 UTC m=+0.349471693 container start d6648ce98dd6b452b38ef29bd6a940a54e38eae42f527b539968cff245673c39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-02-09T22:48:31.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:31 smithi064 ceph-mon[36638]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:31 smithi003 ceph-mon[32678]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:31.727 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:31 smithi089 ceph-mds[66404]: -- 172.21.15.89:0/130611052 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fce68b01a0 con 0x55fce6886800 2024-02-09T22:48:31.728 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l[66400]: starting mds.l at 2024-02-09T22:48:31.728 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:31 smithi089 bash[66386]: d6648ce98dd6b452b38ef29bd6a940a54e38eae42f527b539968cff245673c39 2024-02-09T22:48:31.728 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 22:48:31 smithi089 systemd[1]: Started Ceph mds.l for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:32.139 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 -- 172.21.15.3:0/2223293370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 msgr2=0x7f6828109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 --2- 172.21.15.3:0/2223293370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828109400 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6820009ca0 tx=0x7f682002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 -- 172.21.15.3:0/2223293370 shutdown_connections 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 --2- 172.21.15.3:0/2223293370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6828104680 0x7f6828106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 --2- 172.21.15.3:0/2223293370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828109400 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 --2- 172.21.15.3:0/2223293370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6828101d50 0x7f6828104140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 -- 172.21.15.3:0/2223293370 >> 172.21.15.3:0/2223293370 conn(0x7f68280fb760 msgr2=0x7f68280fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 -- 172.21.15.3:0/2223293370 shutdown_connections 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.252+0000 7f682dc64640 1 -- 172.21.15.3:0/2223293370 wait complete. 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 Processor -- start 2024-02-09T22:48:32.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 -- start start 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6828101d50 0x7f68281016c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6828104680 0x7f68280ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828100240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281009a0 con 0x7f6828106fb0 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6828100b10 con 0x7f6828101d50 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f682dc64640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68281a12e0 con 0x7f6828104680 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.253+0000 7f6827fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828100240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f68277fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6828101d50 0x7f68281016c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828100240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38908/0 (socket says 172.21.15.3:38908) 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 -- 172.21.15.3:0/4093351341 learned_addr learned my addr 172.21.15.3:0/4093351341 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6826ffd640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6828104680 0x7f68280ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 -- 172.21.15.3:0/4093351341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6828104680 msgr2=0x7f68280ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6828104680 0x7f68280ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 -- 172.21.15.3:0/4093351341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6828101d50 msgr2=0x7f68281016c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:32.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6828101d50 0x7f68281016c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 -- 172.21.15.3:0/4093351341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6820009900 con 0x7f6828106fb0 2024-02-09T22:48:32.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.254+0000 7f6827fff640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828100240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6820033000 tx=0x7f6820009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:32.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.257+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68200329b0 con 0x7f6828106fb0 2024-02-09T22:48:32.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.257+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6820003e50 con 0x7f6828106fb0 2024-02-09T22:48:32.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.257+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68281a1530 con 0x7f6828106fb0 2024-02-09T22:48:32.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.257+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6820040450 con 0x7f6828106fb0 2024-02-09T22:48:32.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.257+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68281a1a30 con 0x7f6828106fb0 2024-02-09T22:48:32.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.258+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68280fd250 con 0x7f6828106fb0 2024-02-09T22:48:32.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.259+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f682000a040 con 0x7f6828106fb0 2024-02-09T22:48:32.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.260+0000 7f6824ff9640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f68100a2140 0x7f68100a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:32.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.260+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f68200f0510 con 0x7f6828106fb0 2024-02-09T22:48:32.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.260+0000 7f68277fe640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f68100a2140 0x7f68100a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:32.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.260+0000 7f68277fe640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f68100a2140 0x7f68100a4600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6818009bd0 tx=0x7f6818005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:32.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.264+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f68200b9980 con 0x7f6828106fb0 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6832/2685783987,v1:172.21.15.89:6833/2685783987] up:boot 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: fsmap 1 up:standby 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:32.334 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:32 smithi064 ceph-mon[36638]: Deploying daemon mds.k on smithi064 2024-02-09T22:48:32.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.484+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f682802cdb0 con 0x7f6828106fb0 2024-02-09T22:48:32.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.484+0000 7f6824ff9640 1 -- 172.21.15.3:0/4093351341 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1325 (secure 0 0 0) 0x7f68200bd040 con 0x7f6828106fb0 2024-02-09T22:48:32.488 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:32.488 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-02-09T22:48:32.488 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 2 2024-02-09T22:48:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f68100a2140 msgr2=0x7f68100a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f68100a2140 0x7f68100a4600 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f6818009bd0 tx=0x7f6818005f90 comp rx=0 tx=0).stop 2024-02-09T22:48:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 msgr2=0x7f6828100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828100240 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6820033000 tx=0x7f6820009360 comp rx=0 tx=0).stop 2024-02-09T22:48:32.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 shutdown_connections 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f68100a2140 0x7f68100a4600 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6828106fb0 0x7f6828100240 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6828104680 0x7f68280ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 --2- 172.21.15.3:0/4093351341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6828101d50 0x7f68281016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 >> 172.21.15.3:0/4093351341 conn(0x7f68280fb760 msgr2=0x7f6828102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 shutdown_connections 2024-02-09T22:48:32.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:32.486+0000 7f682dc64640 1 -- 172.21.15.3:0/4093351341 wait complete. 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6832/2685783987,v1:172.21.15.89:6833/2685783987] up:boot 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: fsmap 1 up:standby 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:32.499 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:32 smithi003 ceph-mon[32678]: Deploying daemon mds.k on smithi064 2024-02-09T22:48:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6832/2685783987,v1:172.21.15.89:6833/2685783987] up:boot 2024-02-09T22:48:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: fsmap 1 up:standby 2024-02-09T22:48:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:32.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:32 smithi089 ceph-mon[36665]: Deploying daemon mds.k on smithi064 2024-02-09T22:48:33.384 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:33 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4093351341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:33.384 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:33 smithi064 ceph-mon[36638]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:33 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4093351341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:33 smithi003 ceph-mon[32678]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:33.685 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:33 smithi064 systemd[1]: Starting Ceph mds.k for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:48:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:33 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4093351341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:33 smithi089 ceph-mon[36665]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:34.043 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:33 smithi064 podman[66266]: 2024-02-09 22:48:33.778546046 +0000 UTC m=+0.113480972 container create 661c32850638bb216cd0a13efcd93cbddd896d1ab7169fa3ff2d8bbd279fd7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-02-09T22:48:34.044 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:33 smithi064 podman[66266]: 2024-02-09 22:48:33.683050334 +0000 UTC m=+0.017985265 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:48:34.044 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:34 smithi064 podman[66266]: 2024-02-09 22:48:34.014161067 +0000 UTC m=+0.349095983 container init 661c32850638bb216cd0a13efcd93cbddd896d1ab7169fa3ff2d8bbd279fd7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-09T22:48:34.044 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:34 smithi064 podman[66266]: 2024-02-09 22:48:34.016887313 +0000 UTC m=+0.351822235 container start 661c32850638bb216cd0a13efcd93cbddd896d1ab7169fa3ff2d8bbd279fd7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-02-09T22:48:34.435 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:34 smithi064 ceph-mds[66285]: -- 172.21.15.64:0/240743093 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e5fdf441a0 con 0x55e5fdf1b000 2024-02-09T22:48:34.435 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:34 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k[66281]: starting mds.k at 2024-02-09T22:48:34.435 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:34 smithi064 bash[66266]: 661c32850638bb216cd0a13efcd93cbddd896d1ab7169fa3ff2d8bbd279fd7f2 2024-02-09T22:48:34.435 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 22:48:34 smithi064 systemd[1]: Started Ceph mds.k for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:34.501 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 -- 172.21.15.3:0/3154636178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101d10 msgr2=0x7f44bc1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 --2- 172.21.15.3:0/3154636178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101d10 0x7f44bc1085a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f44b4009ca0 tx=0x7f44b402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 -- 172.21.15.3:0/3154636178 shutdown_connections 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 --2- 172.21.15.3:0/3154636178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101d10 0x7f44bc1085a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 --2- 172.21.15.3:0/3154636178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44bc101350 0x7f44bc1017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 --2- 172.21.15.3:0/3154636178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44bc100160 0x7f44bc100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 -- 172.21.15.3:0/3154636178 >> 172.21.15.3:0/3154636178 conn(0x7f44bc0fb6d0 msgr2=0x7f44bc0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 -- 172.21.15.3:0/3154636178 shutdown_connections 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.602+0000 7f44c311e640 1 -- 172.21.15.3:0/3154636178 wait complete. 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 Processor -- start 2024-02-09T22:48:34.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 -- start start 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44bc100160 0x7f44bc069090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 0x7f44bc19bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44bc101d10 0x7f44bc19e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc19ed90 con 0x7f44bc101350 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc19ef00 con 0x7f44bc101d10 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.603+0000 7f44c311e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44bc19f0c0 con 0x7f44bc100160 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 0x7f44bc19bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 0x7f44bc19bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38930/0 (socket says 172.21.15.3:38930) 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 -- 172.21.15.3:0/2999091776 learned_addr learned my addr 172.21.15.3:0/2999091776 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44c16bb640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44bc101d10 0x7f44bc19e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44c0eba640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44bc100160 0x7f44bc069090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 -- 172.21.15.3:0/2999091776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44bc100160 msgr2=0x7f44bc069090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:34.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44bc100160 0x7f44bc069090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 -- 172.21.15.3:0/2999091776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44bc101d10 msgr2=0x7f44bc19e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44bc101d10 0x7f44bc19e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 -- 172.21.15.3:0/2999091776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b4009900 con 0x7f44bc101350 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44bbfff640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 0x7f44bc19bfc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44a800d770 tx=0x7f44a800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a8009c30 con 0x7f44bc101350 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44a800de80 con 0x7f44bc101350 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a8004e80 con 0x7f44bc101350 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44bc19f3a0 con 0x7f44bc101350 2024-02-09T22:48:34.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.604+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44bc19f8b0 con 0x7f44bc101350 2024-02-09T22:48:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.605+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44bc107420 con 0x7f44bc101350 2024-02-09T22:48:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.606+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f44a8005020 con 0x7f44bc101350 2024-02-09T22:48:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.607+0000 7f44b9ffb640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f44ac0a2140 0x7f44ac0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.607+0000 7f44c0eba640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f44ac0a2140 0x7f44ac0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.607+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f44a80cd1f0 con 0x7f44bc101350 2024-02-09T22:48:34.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.608+0000 7f44c0eba640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f44ac0a2140 0x7f44ac0a4600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f44bc1011b0 tx=0x7f44b0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:34.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.612+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44a8096670 con 0x7f44bc101350 2024-02-09T22:48:34.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.838+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44bc02ce50 con 0x7f44bc101350 2024-02-09T22:48:34.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.838+0000 7f44b9ffb640 1 -- 172.21.15.3:0/2999091776 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7f44a8099d30 con 0x7f44bc101350 2024-02-09T22:48:34.841 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:34.841 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-09T22:48:34.841 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 3 2024-02-09T22:48:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f44ac0a2140 msgr2=0x7f44ac0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f44ac0a2140 0x7f44ac0a4600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f44bc1011b0 tx=0x7f44b0005f90 comp rx=0 tx=0).stop 2024-02-09T22:48:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 msgr2=0x7f44bc19bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 0x7f44bc19bfc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f44a800d770 tx=0x7f44a800dc30 comp rx=0 tx=0).stop 2024-02-09T22:48:34.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 shutdown_connections 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f44ac0a2140 0x7f44ac0a4600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44bc101d10 0x7f44bc19e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44bc101350 0x7f44bc19bfc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 --2- 172.21.15.3:0/2999091776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f44bc100160 0x7f44bc069090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 >> 172.21.15.3:0/2999091776 conn(0x7f44bc0fb6d0 msgr2=0x7f44bc1045f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 shutdown_connections 2024-02-09T22:48:34.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:34.840+0000 7f44c311e640 1 -- 172.21.15.3:0/2999091776 wait complete. 2024-02-09T22:48:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: Deploying daemon mds.j on smithi003 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6832/1710343231,v1:172.21.15.64:6833/1710343231] up:boot 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: fsmap 2 up:standby 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:35.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:35 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2999091776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: Deploying daemon mds.j on smithi003 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6832/1710343231,v1:172.21.15.64:6833/1710343231] up:boot 2024-02-09T22:48:35.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: fsmap 2 up:standby 2024-02-09T22:48:35.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T22:48:35.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:35.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:35 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2999091776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:35.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:35.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:35.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: Deploying daemon mds.j on smithi003 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6832/1710343231,v1:172.21.15.64:6833/1710343231] up:boot 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: fsmap 2 up:standby 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:35.519 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:35 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2999091776' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:36.542 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 systemd[1]: Starting Ceph mds.j for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:48:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:36 smithi064 ceph-mon[36638]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:36.855 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:36.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:36 smithi003 ceph-mon[32678]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:36.881 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 podman[72306]: 2024-02-09 22:48:36.44245844 +0000 UTC m=+0.018348697 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:48:36.881 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 podman[72306]: 2024-02-09 22:48:36.622902789 +0000 UTC m=+0.198793029 container create b486efb1188fab0324028bd08a9b45cf4e7ad6743b44bd6dd82dc7552db96fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131) 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 -- 172.21.15.3:0/1021346680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941024b0 msgr2=0x7f6994102930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1021346680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941024b0 0x7f6994102930 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6984009900 tx=0x7f698402bc30 comp rx=0 tx=0).stop 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 -- 172.21.15.3:0/1021346680 shutdown_connections 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1021346680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6994102e70 0x7f6994109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1021346680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941024b0 0x7f6994102930 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1021346680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69941012c0 0x7f69941016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 -- 172.21.15.3:0/1021346680 >> 172.21.15.3:0/1021346680 conn(0x7f69940fc850 msgr2=0x7f69940fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 -- 172.21.15.3:0/1021346680 shutdown_connections 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.961+0000 7f699bfbc640 1 -- 172.21.15.3:0/1021346680 wait complete. 2024-02-09T22:48:36.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 Processor -- start 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 -- start start 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 0x7f6994074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69941024b0 0x7f6994073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6994102e70 0x7f6994073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940742c0 con 0x7f69941012c0 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6994074430 con 0x7f6994102e70 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f699bfbc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69941a12a0 con 0x7f69941024b0 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f6999d58640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 0x7f6994074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f6999d58640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 0x7f6994074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60092/0 (socket says 172.21.15.3:60092) 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f6999d58640 1 -- 172.21.15.3:0/1509717200 learned_addr learned my addr 172.21.15.3:0/1509717200 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.962+0000 7f6999557640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69941024b0 0x7f6994073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f699a559640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6994102e70 0x7f6994073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f6999d58640 1 -- 172.21.15.3:0/1509717200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69941024b0 msgr2=0x7f6994073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:36.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f6999d58640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69941024b0 0x7f6994073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f6999d58640 1 -- 172.21.15.3:0/1509717200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6994102e70 msgr2=0x7f6994073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f6999d58640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6994102e70 0x7f6994073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f6999d58640 1 -- 172.21.15.3:0/1509717200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6984009560 con 0x7f69941012c0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f699a559640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6994102e70 0x7f6994073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f6999d58640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 0x7f6994074fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f699000c9a0 tx=0x7f699000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6990007c10 con 0x7f69941012c0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6990007d70 con 0x7f69941012c0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69941a1550 con 0x7f69941012c0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699000f910 con 0x7f69941012c0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.963+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69941a1a80 con 0x7f69941012c0 2024-02-09T22:48:36.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.964+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6994108580 con 0x7f69941012c0 2024-02-09T22:48:36.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.965+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f699000fa70 con 0x7f69941012c0 2024-02-09T22:48:36.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.965+0000 7f698affd640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f69800a2140 0x7f69800a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:36.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.966+0000 7f6999557640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f69800a2140 0x7f69800a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:36.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.966+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6990013070 con 0x7f69941012c0 2024-02-09T22:48:36.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.966+0000 7f6999557640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f69800a2140 0x7f69800a4600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6984006010 tx=0x7f6984005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:36.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:36.970+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6990097000 con 0x7f69941012c0 2024-02-09T22:48:36.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:36 smithi089 ceph-mon[36665]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:37.132 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 podman[72306]: 2024-02-09 22:48:36.880142187 +0000 UTC m=+0.456032431 container init b486efb1188fab0324028bd08a9b45cf4e7ad6743b44bd6dd82dc7552db96fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j, RELEASE=main-290212a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True) 2024-02-09T22:48:37.132 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 podman[72306]: 2024-02-09 22:48:36.882806696 +0000 UTC m=+0.458696937 container start b486efb1188fab0324028bd08a9b45cf4e7ad6743b44bd6dd82dc7552db96fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=) 2024-02-09T22:48:37.132 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 ceph-mds[72345]: -- 172.21.15.3:0/1385158822 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f22edac1a0 con 0x55f22ed82c00 2024-02-09T22:48:37.132 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j[72322]: starting mds.j at 2024-02-09T22:48:37.132 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 bash[72306]: b486efb1188fab0324028bd08a9b45cf4e7ad6743b44bd6dd82dc7552db96fd5 2024-02-09T22:48:37.132 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 22:48:36 smithi003 systemd[1]: Started Ceph mds.j for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.214+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f699410cc60 con 0x7f69941012c0 2024-02-09T22:48:37.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.215+0000 7f698affd640 1 -- 172.21.15.3:0/1509717200 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2116 (secure 0 0 0) 0x7f699009a6c0 con 0x7f69941012c0 2024-02-09T22:48:37.218 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:37.218 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-09T22:48:37.218 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 3 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.216+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f69800a2140 msgr2=0x7f69800a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f69800a2140 0x7f69800a4600 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6984006010 tx=0x7f6984005d90 comp rx=0 tx=0).stop 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 msgr2=0x7f6994074fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 0x7f6994074fe0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f699000c9a0 tx=0x7f699000ce60 comp rx=0 tx=0).stop 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 shutdown_connections 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f69800a2140 0x7f69800a4600 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6994102e70 0x7f6994073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69941024b0 0x7f6994073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 --2- 172.21.15.3:0/1509717200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69941012c0 0x7f6994074fe0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 >> 172.21.15.3:0/1509717200 conn(0x7f69940fc850 msgr2=0x7f6994105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:37.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 shutdown_connections 2024-02-09T22:48:37.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:37.217+0000 7f699bfbc640 1 -- 172.21.15.3:0/1509717200 wait complete. 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: Deploying daemon mds.d on smithi003 2024-02-09T22:48:38.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1509717200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:38.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.3:6834/977042251,v1:172.21.15.3:6835/977042251] up:boot 2024-02-09T22:48:38.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: fsmap 3 up:standby 2024-02-09T22:48:38.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:38 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-09T22:48:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: Deploying daemon mds.d on smithi003 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1509717200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.3:6834/977042251,v1:172.21.15.3:6835/977042251] up:boot 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: fsmap 3 up:standby 2024-02-09T22:48:38.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:38 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-09T22:48:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: Deploying daemon mds.d on smithi003 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1509717200' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.3:6834/977042251,v1:172.21.15.3:6835/977042251] up:boot 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: fsmap 3 up:standby 2024-02-09T22:48:38.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:38 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-09T22:48:39.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:39.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 -- 172.21.15.3:0/1550592822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 msgr2=0x7f98dc1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:39.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1550592822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc1085a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f98d4009ca0 tx=0x7f98d402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 -- 172.21.15.3:0/1550592822 shutdown_connections 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1550592822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f98dc101350 0x7f98dc1017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1550592822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc1085a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1550592822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98dc100160 0x7f98dc100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 -- 172.21.15.3:0/1550592822 >> 172.21.15.3:0/1550592822 conn(0x7f98dc0fb6d0 msgr2=0x7f98dc0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 -- 172.21.15.3:0/1550592822 shutdown_connections 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.350+0000 7f98e30c9640 1 -- 172.21.15.3:0/1550592822 wait complete. 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 Processor -- start 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 -- start start 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f98dc100160 0x7f98dc197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98dc101350 0x7f98dc1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:39.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19ed70 con 0x7f98dc101d10 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19eee0 con 0x7f98dc100160 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e30c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98dc19f0a0 con 0x7f98dc101350 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc19e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60096/0 (socket says 172.21.15.3:60096) 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 -- 172.21.15.3:0/1178876746 learned_addr learned my addr 172.21.15.3:0/1178876746 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e0e65640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f98dc100160 0x7f98dc197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98dbfff640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98dc101350 0x7f98dc1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 -- 172.21.15.3:0/1178876746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98dc101350 msgr2=0x7f98dc1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98dc101350 0x7f98dc1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 -- 172.21.15.3:0/1178876746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f98dc100160 msgr2=0x7f98dc197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f98dc100160 0x7f98dc197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.351+0000 7f98e1666640 1 -- 172.21.15.3:0/1178876746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d4009900 con 0x7f98dc101d10 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.352+0000 7f98e1666640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc19e830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f98d4037040 tx=0x7f98d4034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:39.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.352+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4032eb0 con 0x7f98dc101d10 2024-02-09T22:48:39.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.352+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f98d4003e50 con 0x7f98dc101d10 2024-02-09T22:48:39.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.352+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98dc19f320 con 0x7f98dc101d10 2024-02-09T22:48:39.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.352+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4040450 con 0x7f98dc101d10 2024-02-09T22:48:39.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.352+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98dc19f820 con 0x7f98dc101d10 2024-02-09T22:48:39.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.353+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98dc10bb70 con 0x7f98dc101d10 2024-02-09T22:48:39.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.354+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f98d400a040 con 0x7f98dc101d10 2024-02-09T22:48:39.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.354+0000 7f98d9ffb640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f98c40a2140 0x7f98c40a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:39.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.354+0000 7f98e0e65640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f98c40a2140 0x7f98c40a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:39.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.354+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f98d40f0a50 con 0x7f98dc101d10 2024-02-09T22:48:39.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.355+0000 7f98e0e65640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f98c40a2140 0x7f98c40a4600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f98cc005fd0 tx=0x7f98cc00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:39.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.358+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f98d40b9f40 con 0x7f98dc101d10 2024-02-09T22:48:39.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:39 smithi003 ceph-mon[32678]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:39.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:39 smithi064 ceph-mon[36638]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:39.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:39 smithi089 ceph-mon[36665]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:39.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.580+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98dc10bd00 con 0x7f98dc101d10 2024-02-09T22:48:39.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.580+0000 7f98d9ffb640 1 -- 172.21.15.3:0/1178876746 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2901 (secure 0 0 0) 0x7f98d4039b70 con 0x7f98dc101d10 2024-02-09T22:48:39.584 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:39.584 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-09T22:48:39.584 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 4 2024-02-09T22:48:39.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f98c40a2140 msgr2=0x7f98c40a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:39.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f98c40a2140 0x7f98c40a4600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f98cc005fd0 tx=0x7f98cc00b390 comp rx=0 tx=0).stop 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 msgr2=0x7f98dc19e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc19e830 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f98d4037040 tx=0x7f98d4034d40 comp rx=0 tx=0).stop 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 shutdown_connections 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f98c40a2140 0x7f98c40a4600 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f98dc101d10 0x7f98dc19e830 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f98dc101350 0x7f98dc1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 --2- 172.21.15.3:0/1178876746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f98dc100160 0x7f98dc197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 >> 172.21.15.3:0/1178876746 conn(0x7f98dc0fb6d0 msgr2=0x7f98dc0fd130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.582+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 shutdown_connections 2024-02-09T22:48:39.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:39.583+0000 7f98e30c9640 1 -- 172.21.15.3:0/1178876746 wait complete. 2024-02-09T22:48:39.703 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 22:48:39 smithi003 podman[72669]: 2024-02-09 22:48:39.700947852 +0000 UTC m=+0.397281197 container init a32f1d446916f8f37426cbcf06d41e3731265cb8c593876813e5e767889eb2d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:48:40.079 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 22:48:39 smithi003 podman[72669]: 2024-02-09 22:48:39.704133403 +0000 UTC m=+0.400466753 container start a32f1d446916f8f37426cbcf06d41e3731265cb8c593876813e5e767889eb2d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-09T22:48:40.079 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 22:48:39 smithi003 ceph-mds[72709]: -- 172.21.15.3:0/4252351457 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55952bb721a0 con 0x55952bb49000 2024-02-09T22:48:40.079 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 22:48:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d[72703]: starting mds.d at 2024-02-09T22:48:40.079 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 22:48:39 smithi003 bash[72669]: a32f1d446916f8f37426cbcf06d41e3731265cb8c593876813e5e767889eb2d6 2024-02-09T22:48:40.079 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 22:48:39 smithi003 systemd[1]: Started Ceph mds.d for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1178876746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:40 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:40.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1178876746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:40 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1178876746' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:40.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:40.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:40.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:40 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:41.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:41 smithi003 ceph-mon[32678]: Deploying daemon mds.e on smithi064 2024-02-09T22:48:41.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:41 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.3:6836/1424466335,v1:172.21.15.3:6837/1424466335] up:boot 2024-02-09T22:48:41.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:41 smithi003 ceph-mon[32678]: fsmap 4 up:standby 2024-02-09T22:48:41.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:41 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-09T22:48:41.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:41 smithi003 ceph-mon[32678]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:41.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:41 smithi064 ceph-mon[36638]: Deploying daemon mds.e on smithi064 2024-02-09T22:48:41.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:41 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.3:6836/1424466335,v1:172.21.15.3:6837/1424466335] up:boot 2024-02-09T22:48:41.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:41 smithi064 ceph-mon[36638]: fsmap 4 up:standby 2024-02-09T22:48:41.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:41 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-09T22:48:41.390 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:41 smithi064 ceph-mon[36638]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:41.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:41 smithi089 ceph-mon[36665]: Deploying daemon mds.e on smithi064 2024-02-09T22:48:41.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:41 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.3:6836/1424466335,v1:172.21.15.3:6837/1424466335] up:boot 2024-02-09T22:48:41.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:41 smithi089 ceph-mon[36665]: fsmap 4 up:standby 2024-02-09T22:48:41.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:41 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-09T22:48:41.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:41 smithi089 ceph-mon[36665]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:41.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 -- 172.21.15.3:0/2336604200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c101d80 msgr2=0x7fe72c108610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:41.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 --2- 172.21.15.3:0/2336604200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c101d80 0x7fe72c108610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe724009ca0 tx=0x7fe72402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 -- 172.21.15.3:0/2336604200 shutdown_connections 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 --2- 172.21.15.3:0/2336604200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c101d80 0x7fe72c108610 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 --2- 172.21.15.3:0/2336604200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe72c1013c0 0x7fe72c101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 --2- 172.21.15.3:0/2336604200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe72c1001d0 0x7fe72c1005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 -- 172.21.15.3:0/2336604200 >> 172.21.15.3:0/2336604200 conn(0x7fe72c0fb760 msgr2=0x7fe72c0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 -- 172.21.15.3:0/2336604200 shutdown_connections 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.706+0000 7fe733a23640 1 -- 172.21.15.3:0/2336604200 wait complete. 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 Processor -- start 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 -- start start 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 0x7fe72c104160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe72c1013c0 0x7fe72c1027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe72c101d80 0x7fe72c102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:41.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c103440 con 0x7fe72c1001d0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c1035b0 con 0x7fe72c1013c0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe733a23640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe72c1a1260 con 0x7fe72c101d80 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 0x7fe72c104160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 0x7fe72c104160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60130/0 (socket says 172.21.15.3:60130) 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 -- 172.21.15.3:0/2301263483 learned_addr learned my addr 172.21.15.3:0/2301263483 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe730fbe640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe72c1013c0 0x7fe72c1027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe731fc0640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe72c101d80 0x7fe72c102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 -- 172.21.15.3:0/2301263483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe72c101d80 msgr2=0x7fe72c102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe72c101d80 0x7fe72c102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 -- 172.21.15.3:0/2301263483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe72c1013c0 msgr2=0x7fe72c1027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe72c1013c0 0x7fe72c1027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 -- 172.21.15.3:0/2301263483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe724009900 con 0x7fe72c1001d0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.707+0000 7fe7317bf640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 0x7fe72c104160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c9a0 tx=0x7fe72800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.708+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe728007c10 con 0x7fe72c1001d0 2024-02-09T22:48:41.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.708+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe728007d70 con 0x7fe72c1001d0 2024-02-09T22:48:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.708+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe72800f910 con 0x7fe72c1001d0 2024-02-09T22:48:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.708+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe72c1a1510 con 0x7fe72c1001d0 2024-02-09T22:48:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.708+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe72c1a1a70 con 0x7fe72c1001d0 2024-02-09T22:48:41.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.709+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe72c10bb70 con 0x7fe72c1001d0 2024-02-09T22:48:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.710+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fe72800fa70 con 0x7fe72c1001d0 2024-02-09T22:48:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.710+0000 7fe7227fc640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe7180a2140 0x7fe7180a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.710+0000 7fe730fbe640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe7180a2140 0x7fe7180a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.711+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe728013070 con 0x7fe72c1001d0 2024-02-09T22:48:41.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.711+0000 7fe730fbe640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe7180a2140 0x7fe7180a4600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe72c103ed0 tx=0x7fe71c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:41.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.714+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe7280972b0 con 0x7fe72c1001d0 2024-02-09T22:48:41.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.933+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe72c1a1cf0 con 0x7fe72c1001d0 2024-02-09T22:48:41.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.934+0000 7fe7227fc640 1 -- 172.21.15.3:0/2301263483 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3689 (secure 0 0 0) 0x7fe72c1a1cf0 con 0x7fe72c1001d0 2024-02-09T22:48:41.937 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:41.937 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-02-09T22:48:41.937 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 5 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe7180a2140 msgr2=0x7fe7180a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe7180a2140 0x7fe7180a4600 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe72c103ed0 tx=0x7fe71c00b390 comp rx=0 tx=0).stop 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 msgr2=0x7fe72c104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 0x7fe72c104160 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe72800c9a0 tx=0x7fe72800ce60 comp rx=0 tx=0).stop 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 shutdown_connections 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe7180a2140 0x7fe7180a4600 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe72c1013c0 0x7fe72c1027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe72c101d80 0x7fe72c102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 --2- 172.21.15.3:0/2301263483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe72c1001d0 0x7fe72c104160 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:41.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 >> 172.21.15.3:0/2301263483 conn(0x7fe72c0fb760 msgr2=0x7fe72c0fd0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:41.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 shutdown_connections 2024-02-09T22:48:41.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:41.936+0000 7fe733a23640 1 -- 172.21.15.3:0/2301263483 wait complete. 2024-02-09T22:48:42.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:42 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2301263483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:42.186 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 podman[66588]: 2024-02-09 22:48:42.153449273 +0000 UTC m=+0.134814555 container create d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True) 2024-02-09T22:48:42.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:42 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2301263483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:42.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:42 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2301263483' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:42.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 podman[66588]: 2024-02-09 22:48:42.36872542 +0000 UTC m=+0.350090712 container init d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-09T22:48:42.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 podman[66588]: 2024-02-09 22:48:42.371638537 +0000 UTC m=+0.353003815 container start d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:48:42.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 ceph-mds[66607]: -- 172.21.15.64:0/4112794355 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563223f1e1a0 con 0x563223ef4800 2024-02-09T22:48:42.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: starting mds.e at 2024-02-09T22:48:42.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 bash[66588]: d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 2024-02-09T22:48:42.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:48:42 smithi064 systemd[1]: Started Ceph mds.e for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: Deploying daemon mds.f on smithi089 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6834/3082808593,v1:172.21.15.64:6835/3082808593] up:boot 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: fsmap 5 up:standby 2024-02-09T22:48:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:43 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:48:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:43.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:43.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:43.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: Deploying daemon mds.f on smithi089 2024-02-09T22:48:43.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6834/3082808593,v1:172.21.15.64:6835/3082808593] up:boot 2024-02-09T22:48:43.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: fsmap 5 up:standby 2024-02-09T22:48:43.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:43 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:48:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: Deploying daemon mds.f on smithi089 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6834/3082808593,v1:172.21.15.64:6835/3082808593] up:boot 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: fsmap 5 up:standby 2024-02-09T22:48:43.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:43 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:48:43.951 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:44.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 -- 172.21.15.3:0/1721649803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 msgr2=0x7fb704109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 --2- 172.21.15.3:0/1721649803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704109700 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009ca0 tx=0x7fb6fc02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 -- 172.21.15.3:0/1721649803 shutdown_connections 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 --2- 172.21.15.3:0/1721649803 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704109700 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 --2- 172.21.15.3:0/1721649803 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7041024b0 0x7fb704102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 --2- 172.21.15.3:0/1721649803 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7041012c0 0x7fb7041016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 -- 172.21.15.3:0/1721649803 >> 172.21.15.3:0/1721649803 conn(0x7fb7040fc850 msgr2=0x7fb7040fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 -- 172.21.15.3:0/1721649803 shutdown_connections 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.063+0000 7fb709935640 1 -- 172.21.15.3:0/1721649803 wait complete. 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 Processor -- start 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 -- start start 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7041012c0 0x7fb704074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7041024b0 0x7fb704073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:44.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7040742c0 con 0x7fb704102e70 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb704074430 con 0x7fb7041012c0 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb709935640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7041a1310 con 0x7fb7041024b0 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704073b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60150/0 (socket says 172.21.15.3:60150) 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 -- 172.21.15.3:0/1699028807 learned_addr learned my addr 172.21.15.3:0/1699028807 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb6fbfff640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7041024b0 0x7fb704073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb702ffd640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7041012c0 0x7fb704074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 -- 172.21.15.3:0/1699028807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7041024b0 msgr2=0x7fb704073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7041024b0 0x7fb704073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 -- 172.21.15.3:0/1699028807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7041012c0 msgr2=0x7fb704074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7041012c0 0x7fb704074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.064+0000 7fb7037fe640 1 -- 172.21.15.3:0/1699028807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6fc009900 con 0x7fb704102e70 2024-02-09T22:48:44.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.065+0000 7fb7037fe640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704073b60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc033000 tx=0x7fb6fc034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.065+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc0329b0 con 0x7fb704102e70 2024-02-09T22:48:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.065+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6fc003e50 con 0x7fb704102e70 2024-02-09T22:48:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.065+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7041a1560 con 0x7fb704102e70 2024-02-09T22:48:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.065+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc040450 con 0x7fb704102e70 2024-02-09T22:48:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.065+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7041a1a60 con 0x7fb704102e70 2024-02-09T22:48:44.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.066+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7041085e0 con 0x7fb704102e70 2024-02-09T22:48:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.068+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb6fc00a040 con 0x7fb704102e70 2024-02-09T22:48:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.068+0000 7fb708933640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb6f00a2140 0x7fb6f00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.068+0000 7fb702ffd640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb6f00a2140 0x7fb6f00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.068+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb6fc0f0980 con 0x7fb704102e70 2024-02-09T22:48:44.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.069+0000 7fb702ffd640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb6f00a2140 0x7fb6f00a4600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb704102310 tx=0x7fb6f4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:44.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.072+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb6fc0b9e70 con 0x7fb704102e70 2024-02-09T22:48:44.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.290+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb70402ce50 con 0x7fb704102e70 2024-02-09T22:48:44.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.291+0000 7fb708933640 1 -- 172.21.15.3:0/1699028807 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4480 (secure 0 0 0) 0x7fb6fc0bd530 con 0x7fb704102e70 2024-02-09T22:48:44.294 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:44.294 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-02-09T22:48:44.294 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 6 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb6f00a2140 msgr2=0x7fb6f00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb6f00a2140 0x7fb6f00a4600 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb704102310 tx=0x7fb6f4005f90 comp rx=0 tx=0).stop 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 msgr2=0x7fb704073b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704073b60 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc033000 tx=0x7fb6fc034a90 comp rx=0 tx=0).stop 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 shutdown_connections 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb6f00a2140 0x7fb6f00a4600 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb704102e70 0x7fb704073b60 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7041024b0 0x7fb704073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 --2- 172.21.15.3:0/1699028807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb7041012c0 0x7fb704074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 >> 172.21.15.3:0/1699028807 conn(0x7fb7040fc850 msgr2=0x7fb7041057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:44.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 shutdown_connections 2024-02-09T22:48:44.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:44.293+0000 7fb709935640 1 -- 172.21.15.3:0/1699028807 wait complete. 2024-02-09T22:48:44.728 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:44 smithi089 systemd[1]: Starting Ceph mds.f for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:48:45.042 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:44 smithi089 podman[66691]: 2024-02-09 22:48:44.690132917 +0000 UTC m=+0.018401751 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:48:45.043 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:44 smithi089 podman[66691]: 2024-02-09 22:48:44.818469962 +0000 UTC m=+0.146738775 container create 60bbd2e371b33bdf9435055a32905b787354e6f2eef19e681ca41b2bab74b336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131) 2024-02-09T22:48:45.043 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:45 smithi089 podman[66691]: 2024-02-09 22:48:45.037727801 +0000 UTC m=+0.365996624 container init 60bbd2e371b33bdf9435055a32905b787354e6f2eef19e681ca41b2bab74b336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-09T22:48:45.043 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:45 smithi089 podman[66691]: 2024-02-09 22:48:45.040406205 +0000 UTC m=+0.368675034 container start 60bbd2e371b33bdf9435055a32905b787354e6f2eef19e681ca41b2bab74b336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131) 2024-02-09T22:48:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:45 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1699028807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:45 smithi003 ceph-mon[32678]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:45.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:45 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1699028807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:45.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:45 smithi064 ceph-mon[36638]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:45 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1699028807' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:45 smithi089 ceph-mon[36665]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:45.477 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:45 smithi089 ceph-mds[66710]: -- 172.21.15.89:0/756705883 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563055f121a0 con 0x563055ee8800 2024-02-09T22:48:45.477 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:45 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f[66706]: starting mds.f at 2024-02-09T22:48:45.478 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:45 smithi089 bash[66691]: 60bbd2e371b33bdf9435055a32905b787354e6f2eef19e681ca41b2bab74b336 2024-02-09T22:48:45.478 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 22:48:45 smithi089 systemd[1]: Started Ceph mds.f for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:46.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:46.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 -- 172.21.15.3:0/2635583643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc101d10 msgr2=0x7feafc1085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 --2- 172.21.15.3:0/2635583643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc101d10 0x7feafc1085a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7feaf4009ca0 tx=0x7feaf402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 -- 172.21.15.3:0/2635583643 shutdown_connections 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 --2- 172.21.15.3:0/2635583643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc101d10 0x7feafc1085a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 --2- 172.21.15.3:0/2635583643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 0x7feafc1017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 --2- 172.21.15.3:0/2635583643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc100160 0x7feafc100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 -- 172.21.15.3:0/2635583643 >> 172.21.15.3:0/2635583643 conn(0x7feafc0fb6d0 msgr2=0x7feafc0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 -- 172.21.15.3:0/2635583643 shutdown_connections 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.420+0000 7feb033ac640 1 -- 172.21.15.3:0/2635583643 wait complete. 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 Processor -- start 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 -- start start 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 0x7feafc197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:46.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 0x7feafc197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc101d10 0x7feafc19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19ee10 con 0x7feafc100160 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19ef80 con 0x7feafc101d10 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb033ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feafc19f140 con 0x7feafc101350 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 0x7feafc197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 0x7feafc197320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60170/0 (socket says 172.21.15.3:60170) 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 -- 172.21.15.3:0/2592748274 learned_addr learned my addr 172.21.15.3:0/2592748274 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01949640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc101d10 0x7feafc19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb00947640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 0x7feafc197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 -- 172.21.15.3:0/2592748274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 msgr2=0x7feafc197860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 0x7feafc197860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 -- 172.21.15.3:0/2592748274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc101d10 msgr2=0x7feafc19e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc101d10 0x7feafc19e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01148640 1 -- 172.21.15.3:0/2592748274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaf4009900 con 0x7feafc100160 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb01949640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc101d10 0x7feafc19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.421+0000 7feb00947640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 0x7feafc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.422+0000 7feb01148640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 0x7feafc197320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feaf800d770 tx=0x7feaf800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.422+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8009c30 con 0x7feafc100160 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.422+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feafc19f420 con 0x7feafc100160 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.422+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feaf800de80 con 0x7feafc100160 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.422+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaf8010460 con 0x7feafc100160 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.422+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feafc19f980 con 0x7feafc100160 2024-02-09T22:48:46.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.423+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feafc10bbb0 con 0x7feafc100160 2024-02-09T22:48:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.424+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7feaf8003f60 con 0x7feafc100160 2024-02-09T22:48:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.425+0000 7feaf27fc640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feae80a20c0 0x7feae80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.425+0000 7feb00947640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feae80a20c0 0x7feae80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.425+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feaf80cd5e0 con 0x7feafc100160 2024-02-09T22:48:46.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.425+0000 7feb00947640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feae80a20c0 0x7feae80a4580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feaec006fd0 tx=0x7feaec008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:46.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.428+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feaf8096a50 con 0x7feafc100160 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: Deploying daemon mds.c on smithi089 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] up:boot 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: fsmap 6 up:standby 2024-02-09T22:48:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:46 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T22:48:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: Deploying daemon mds.c on smithi089 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] up:boot 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: fsmap 6 up:standby 2024-02-09T22:48:46.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:46 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: Deploying daemon mds.c on smithi089 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] up:boot 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: fsmap 6 up:standby 2024-02-09T22:48:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:46 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T22:48:46.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.648+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feafc10bd40 con 0x7feafc100160 2024-02-09T22:48:46.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.649+0000 7feaf27fc640 1 -- 172.21.15.3:0/2592748274 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5271 (secure 0 0 0) 0x7feaf809a110 con 0x7feafc100160 2024-02-09T22:48:46.653 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:46.653 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-02-09T22:48:46.654 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 7 2024-02-09T22:48:46.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feae80a20c0 msgr2=0x7feae80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:46.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feae80a20c0 0x7feae80a4580 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7feaec006fd0 tx=0x7feaec008040 comp rx=0 tx=0).stop 2024-02-09T22:48:46.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 msgr2=0x7feafc197320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 0x7feafc197320 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7feaf800d770 tx=0x7feaf800dc30 comp rx=0 tx=0).stop 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 shutdown_connections 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feae80a20c0 0x7feae80a4580 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feafc101d10 0x7feafc19e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feafc101350 0x7feafc197860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 --2- 172.21.15.3:0/2592748274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feafc100160 0x7feafc197320 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 >> 172.21.15.3:0/2592748274 conn(0x7feafc0fb6d0 msgr2=0x7feafc068500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:46.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 shutdown_connections 2024-02-09T22:48:46.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:46.651+0000 7feb033ac640 1 -- 172.21.15.3:0/2592748274 wait complete. 2024-02-09T22:48:47.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:47 smithi089 ceph-mon[36665]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:47.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:47 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2592748274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:47.478 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 systemd[1]: Starting Ceph mds.c for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:48:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:47 smithi003 ceph-mon[32678]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:47 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2592748274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:47.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:47 smithi064 ceph-mon[36638]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:47.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:47 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2592748274' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:47.827 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 podman[67000]: 2024-02-09 22:48:47.411072649 +0000 UTC m=+0.017476624 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:48:47.828 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 podman[67000]: 2024-02-09 22:48:47.510439699 +0000 UTC m=+0.116843657 container create ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:48:47.828 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 podman[67000]: 2024-02-09 22:48:47.724981418 +0000 UTC m=+0.331385382 container init ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-09T22:48:47.828 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 podman[67000]: 2024-02-09 22:48:47.727489165 +0000 UTC m=+0.333893140 container start ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c, ceph=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-09T22:48:47.828 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 ceph-mds[67019]: -- 172.21.15.89:0/1411767054 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5565ac1e81a0 con 0x5565ac1bec00 2024-02-09T22:48:47.828 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c[67015]: starting mds.c at 2024-02-09T22:48:47.828 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 bash[67000]: ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 2024-02-09T22:48:48.227 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 22:48:47 smithi089 systemd[1]: Started Ceph mds.c for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:48.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:48.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/2921155174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681012c0 msgr2=0x7f3f681016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:48.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/2921155174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681012c0 0x7f3f681016e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3f64009900 tx=0x7f3f6402bc30 comp rx=0 tx=0).stop 2024-02-09T22:48:48.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/2921155174 shutdown_connections 2024-02-09T22:48:48.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/2921155174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f681024b0 0x7f3f68102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/2921155174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f68102e70 0x7f3f68109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/2921155174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681012c0 0x7f3f681016e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/2921155174 >> 172.21.15.3:0/2921155174 conn(0x7f3f680fc850 msgr2=0x7f3f680fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/2921155174 shutdown_connections 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.782+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/2921155174 wait complete. 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 Processor -- start 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 -- start start 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f681012c0 0x7f3f68074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 0x7f3f68073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102e70 0x7f3f68073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:48.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f680742c0 con 0x7f3f681024b0 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f68074430 con 0x7f3f68102e70 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6fc0e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f681a1320 con 0x7f3f681012c0 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 0x7f3f68073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 0x7f3f68073620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44770/0 (socket says 172.21.15.3:44770) 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 -- 172.21.15.3:0/4076624946 learned_addr learned my addr 172.21.15.3:0/4076624946 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6e1ab640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102e70 0x7f3f68073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d9aa640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f681012c0 0x7f3f68074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 -- 172.21.15.3:0/4076624946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f681012c0 msgr2=0x7f3f68074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f681012c0 0x7f3f68074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 -- 172.21.15.3:0/4076624946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102e70 msgr2=0x7f3f68073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102e70 0x7f3f68073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:48.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 -- 172.21.15.3:0/4076624946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64009560 con 0x7f3f681024b0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.783+0000 7f3f6d1a9640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 0x7f3f68073620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800ea20 tx=0x7f3f5800eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.784+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58009700 con 0x7f3f681024b0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.784+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f5800cd80 con 0x7f3f681024b0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.784+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f681a15d0 con 0x7f3f681024b0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.784+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58018840 con 0x7f3f681024b0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.784+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f681a1ae0 con 0x7f3f681024b0 2024-02-09T22:48:48.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.785+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f681085e0 con 0x7f3f681024b0 2024-02-09T22:48:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.786+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3f58010a80 con 0x7f3f681024b0 2024-02-09T22:48:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.786+0000 7f3f5effd640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f540a20c0 0x7f3f540a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.786+0000 7f3f6d9aa640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f540a20c0 0x7f3f540a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.787+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f58014070 con 0x7f3f681024b0 2024-02-09T22:48:48.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.787+0000 7f3f6d9aa640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f540a20c0 0x7f3f540a4580 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f64005ab0 tx=0x7f3f64005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:48.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:48.790+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f580974b0 con 0x7f3f681024b0 2024-02-09T22:48:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.010+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f6810cc60 con 0x7f3f681024b0 2024-02-09T22:48:49.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.011+0000 7f3f5effd640 1 -- 172.21.15.3:0/4076624946 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6062 (secure 0 0 0) 0x7f3f5809ab70 con 0x7f3f681024b0 2024-02-09T22:48:49.014 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:49.014 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-02-09T22:48:49.015 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 8 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f540a20c0 msgr2=0x7f3f540a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f540a20c0 0x7f3f540a4580 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f64005ab0 tx=0x7f3f64005fd0 comp rx=0 tx=0).stop 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 msgr2=0x7f3f68073620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 0x7f3f68073620 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800ea20 tx=0x7f3f5800eee0 comp rx=0 tx=0).stop 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 shutdown_connections 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f540a20c0 0x7f3f540a4580 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f68102e70 0x7f3f68073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:49.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f681024b0 0x7f3f68073620 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:49.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 --2- 172.21.15.3:0/4076624946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f681012c0 0x7f3f68074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:49.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 >> 172.21.15.3:0/4076624946 conn(0x7f3f680fc850 msgr2=0x7f3f681057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:49.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 shutdown_connections 2024-02-09T22:48:49.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:49.013+0000 7f3f6fc0e640 1 -- 172.21.15.3:0/4076624946 wait complete. 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: Deploying daemon mds.h on smithi064 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] up:boot 2024-02-09T22:48:49.026 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: fsmap 7 up:standby 2024-02-09T22:48:49.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T22:48:49.027 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:48 smithi003 ceph-mon[32678]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: Deploying daemon mds.h on smithi064 2024-02-09T22:48:49.149 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] up:boot 2024-02-09T22:48:49.150 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: fsmap 7 up:standby 2024-02-09T22:48:49.150 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T22:48:49.150 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:48 smithi089 ceph-mon[36665]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:49.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: Deploying daemon mds.h on smithi064 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] up:boot 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: fsmap 7 up:standby 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T22:48:49.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:48 smithi064 ceph-mon[36638]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:50.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:49 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4076624946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:50.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:49 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4076624946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:50.185 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:49 smithi064 systemd[1]: Starting Ceph mds.h for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:48:50.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:49 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4076624946' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:50.435 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 podman[66899]: 2024-02-09 22:48:50.084777245 +0000 UTC m=+0.018157559 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:48:50.436 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 podman[66899]: 2024-02-09 22:48:50.193759118 +0000 UTC m=+0.127139459 container create be2ffb82557bbdcf887e1592d83ca3b5087884a8d2c93295f55d2ffcadb65141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-02-09T22:48:50.436 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 podman[66899]: 2024-02-09 22:48:50.404428749 +0000 UTC m=+0.337809053 container init be2ffb82557bbdcf887e1592d83ca3b5087884a8d2c93295f55d2ffcadb65141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131) 2024-02-09T22:48:50.436 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 podman[66899]: 2024-02-09 22:48:50.407235426 +0000 UTC m=+0.340615726 container start be2ffb82557bbdcf887e1592d83ca3b5087884a8d2c93295f55d2ffcadb65141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:48:50.436 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 ceph-mds[66918]: -- 172.21.15.64:0/275875761 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a45c541a0 con 0x556a45c2a800 2024-02-09T22:48:50.436 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h[66914]: starting mds.h at 2024-02-09T22:48:50.935 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 bash[66899]: be2ffb82557bbdcf887e1592d83ca3b5087884a8d2c93295f55d2ffcadb65141 2024-02-09T22:48:50.935 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 22:48:50 smithi064 systemd[1]: Started Ceph mds.h for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:51.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 -- 172.21.15.3:0/1729868855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85101012c0 msgr2=0x7f85101016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 --2- 172.21.15.3:0/1729868855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85101012c0 0x7f85101016e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f850c009900 tx=0x7f850c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:48:51.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 -- 172.21.15.3:0/1729868855 shutdown_connections 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 --2- 172.21.15.3:0/1729868855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85101024b0 0x7f8510102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 --2- 172.21.15.3:0/1729868855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8510102e70 0x7f8510109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 --2- 172.21.15.3:0/1729868855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f85101012c0 0x7f85101016e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 -- 172.21.15.3:0/1729868855 >> 172.21.15.3:0/1729868855 conn(0x7f85100fc850 msgr2=0x7f85100fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 -- 172.21.15.3:0/1729868855 shutdown_connections 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.140+0000 7f851739a640 1 -- 172.21.15.3:0/1729868855 wait complete. 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 Processor -- start 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 -- start start 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85101012c0 0x7f8510074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85101024b0 0x7f8510073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 0x7f8510073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85100742c0 con 0x7f8510102e70 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8510074430 con 0x7f85101012c0 2024-02-09T22:48:51.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f851739a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85101a13b0 con 0x7f85101024b0 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 0x7f8510073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 0x7f8510073b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44788/0 (socket says 172.21.15.3:44788) 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515136640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85101012c0 0x7f8510074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 -- 172.21.15.3:0/1534503832 learned_addr learned my addr 172.21.15.3:0/1534503832 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8514935640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85101024b0 0x7f8510073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 -- 172.21.15.3:0/1534503832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85101024b0 msgr2=0x7f8510073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85101024b0 0x7f8510073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 -- 172.21.15.3:0/1534503832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85101012c0 msgr2=0x7f8510074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85101012c0 0x7f8510074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8515937640 1 -- 172.21.15.3:0/1534503832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c009560 con 0x7f8510102e70 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.141+0000 7f8514935640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85101024b0 0x7f8510073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.142+0000 7f8515937640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 0x7f8510073b60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f850800b9a0 tx=0x7f850800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.142+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f850800cba0 con 0x7f8510102e70 2024-02-09T22:48:51.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.142+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f850800cd00 con 0x7f8510102e70 2024-02-09T22:48:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.142+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85080076f0 con 0x7f8510102e70 2024-02-09T22:48:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.142+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85101a1660 con 0x7f8510102e70 2024-02-09T22:48:51.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.142+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85101a1b20 con 0x7f8510102e70 2024-02-09T22:48:51.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.143+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8510108600 con 0x7f8510102e70 2024-02-09T22:48:51.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.144+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f8508014a20 con 0x7f8510102e70 2024-02-09T22:48:51.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.145+0000 7f85067fc640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f84fc0a20c0 0x7f84fc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:51.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.145+0000 7f8515136640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f84fc0a20c0 0x7f84fc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:51.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.145+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f85080cd150 con 0x7f8510102e70 2024-02-09T22:48:51.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.145+0000 7f8515136640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f84fc0a20c0 0x7f84fc0a4580 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f850c005ab0 tx=0x7f850c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:51.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.149+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f850804fd20 con 0x7f8510102e70 2024-02-09T22:48:51.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.370+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f851002ce50 con 0x7f8510102e70 2024-02-09T22:48:51.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.370+0000 7f85067fc640 1 -- 172.21.15.3:0/1534503832 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6853 (secure 0 0 0) 0x7f85080965d0 con 0x7f8510102e70 2024-02-09T22:48:51.373 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:51.373 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-02-09T22:48:51.374 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 9 2024-02-09T22:48:51.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f84fc0a20c0 msgr2=0x7f84fc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:51.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f84fc0a20c0 0x7f84fc0a4580 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f850c005ab0 tx=0x7f850c005fd0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 msgr2=0x7f8510073b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:51.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 0x7f8510073b60 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f850800b9a0 tx=0x7f850800be60 comp rx=0 tx=0).stop 2024-02-09T22:48:51.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 shutdown_connections 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f84fc0a20c0 0x7f84fc0a4580 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8510102e70 0x7f8510073b60 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f85101024b0 0x7f8510073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 --2- 172.21.15.3:0/1534503832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f85101012c0 0x7f8510074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 >> 172.21.15.3:0/1534503832 conn(0x7f85100fc850 msgr2=0x7f85101057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 shutdown_connections 2024-02-09T22:48:51.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:51.372+0000 7f851739a640 1 -- 172.21.15.3:0/1534503832 wait complete. 2024-02-09T22:48:51.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: Deploying daemon mds.i on smithi089 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] up:boot 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: fsmap 8 up:standby 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T22:48:51.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1534503832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:51.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:51.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:51.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: Deploying daemon mds.i on smithi089 2024-02-09T22:48:51.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] up:boot 2024-02-09T22:48:51.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: fsmap 8 up:standby 2024-02-09T22:48:51.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T22:48:51.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1534503832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:51.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: Deploying daemon mds.i on smithi089 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] up:boot 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: fsmap 8 up:standby 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T22:48:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1534503832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:52.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:52 smithi089 ceph-mon[36665]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:52 smithi003 ceph-mon[32678]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:52.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:52 smithi064 ceph-mon[36638]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:53.097 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 22:48:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i[67337]: starting mds.i at 2024-02-09T22:48:53.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:53.477 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 22:48:53 smithi089 bash[67322]: d2f4556cecd8ef3db90ab689eb03dbc2463547c6e3703f05e940332eaf3ce48f 2024-02-09T22:48:53.477 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 22:48:53 smithi089 systemd[1]: Started Ceph mds.i for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2653338152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4100160 msgr2=0x7f12e4100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2653338152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4100160 0x7f12e4100580 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f12d4009900 tx=0x7f12d402bc30 comp rx=0 tx=0).stop 2024-02-09T22:48:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2653338152 shutdown_connections 2024-02-09T22:48:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2653338152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e4101d10 0x7f12e41085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2653338152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e4101350 0x7f12e41017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2653338152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4100160 0x7f12e4100580 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2653338152 >> 172.21.15.3:0/2653338152 conn(0x7f12e40fb6d0 msgr2=0x7f12e40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2653338152 shutdown_connections 2024-02-09T22:48:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.488+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2653338152 wait complete. 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 Processor -- start 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 -- start start 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e4100160 0x7f12e4196170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 0x7f12e41966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e4101d10 0x7f12e419d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e419dc60 con 0x7f12e4101350 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e419ddd0 con 0x7f12e4101d10 2024-02-09T22:48:53.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ec4c3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e419df90 con 0x7f12e4100160 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 0x7f12e41966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 0x7f12e41966b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44806/0 (socket says 172.21.15.3:44806) 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 -- 172.21.15.3:0/2458114782 learned_addr learned my addr 172.21.15.3:0/2458114782 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12eaa60640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e4101d10 0x7f12e419d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12ea25f640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e4100160 0x7f12e4196170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 -- 172.21.15.3:0/2458114782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e4100160 msgr2=0x7f12e4196170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:53.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e4100160 0x7f12e4196170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 -- 172.21.15.3:0/2458114782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e4101d10 msgr2=0x7f12e419d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e4101d10 0x7f12e419d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.489+0000 7f12e9a5e640 1 -- 172.21.15.3:0/2458114782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12d4009560 con 0x7f12e4101350 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.490+0000 7f12e9a5e640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 0x7f12e41966b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12e000d950 tx=0x7f12e000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.490+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e0009c30 con 0x7f12e4101350 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.490+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e419e270 con 0x7f12e4101350 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.490+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12e0009d90 con 0x7f12e4101350 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.490+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e000f7e0 con 0x7f12e4101350 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.490+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e419e7d0 con 0x7f12e4101350 2024-02-09T22:48:53.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.491+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e4107500 con 0x7f12e4101350 2024-02-09T22:48:53.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.492+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f12e0010ad0 con 0x7f12e4101350 2024-02-09T22:48:53.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.493+0000 7f12db7fe640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f12d00a20c0 0x7f12d00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:53.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.493+0000 7f12ea25f640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f12d00a20c0 0x7f12d00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:53.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.493+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12e00cd590 con 0x7f12e4101350 2024-02-09T22:48:53.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.493+0000 7f12ea25f640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f12d00a20c0 0x7f12d00a4580 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f12d402c6a0 tx=0x7f12d4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:53.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.497+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f12e00969e0 con 0x7f12e4101350 2024-02-09T22:48:53.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.727+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12e419e400 con 0x7f12e4101350 2024-02-09T22:48:53.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.727+0000 7f12db7fe640 1 -- 172.21.15.3:0/2458114782 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7643 (secure 0 0 0) 0x7f12e419e400 con 0x7f12e4101350 2024-02-09T22:48:53.731 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:53.731 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-02-09T22:48:53.732 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 10 2024-02-09T22:48:53.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f12d00a20c0 msgr2=0x7f12d00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:53.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f12d00a20c0 0x7f12d00a4580 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f12d402c6a0 tx=0x7f12d4005fd0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 msgr2=0x7f12e41966b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:53.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 0x7f12e41966b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f12e000d950 tx=0x7f12e000de10 comp rx=0 tx=0).stop 2024-02-09T22:48:53.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 shutdown_connections 2024-02-09T22:48:53.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f12d00a20c0 0x7f12d00a4580 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f12e4101d10 0x7f12e419d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f12e4101350 0x7f12e41966b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 --2- 172.21.15.3:0/2458114782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f12e4100160 0x7f12e4196170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:53.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 >> 172.21.15.3:0/2458114782 conn(0x7f12e40fb6d0 msgr2=0x7f12e41046d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:53.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 shutdown_connections 2024-02-09T22:48:53.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:53.730+0000 7f12ec4c3640 1 -- 172.21.15.3:0/2458114782 wait complete. 2024-02-09T22:48:54.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:54.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:54.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:54.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: Deploying daemon mds.g on smithi003 2024-02-09T22:48:54.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] up:boot 2024-02-09T22:48:54.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: fsmap 9 up:standby 2024-02-09T22:48:54.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T22:48:54.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:54 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2458114782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:54.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: Deploying daemon mds.g on smithi003 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] up:boot 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: fsmap 9 up:standby 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T22:48:54.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:54 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2458114782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: Deploying daemon mds.g on smithi003 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] up:boot 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: fsmap 9 up:standby 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T22:48:54.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:54 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2458114782' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:55.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:55 smithi064 ceph-mon[36638]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:55.443 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:55 smithi003 ceph-mon[32678]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:55.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:55 smithi089 ceph-mon[36665]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:55.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:55.771 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:55 smithi003 podman[73244]: 2024-02-09 22:48:55.685821832 +0000 UTC m=+0.142567583 container create 1ddc91dd56b1fffc8d7f1a0b9491125d8abbe9312377b529e2b24fa0f28c8e71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=main-290212a, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:48:55.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 -- 172.21.15.3:0/633697633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430106fa0 msgr2=0x7f9430109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 --2- 172.21.15.3:0/633697633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430106fa0 0x7f9430109390 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f9428009ca0 tx=0x7f942802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 -- 172.21.15.3:0/633697633 shutdown_connections 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 --2- 172.21.15.3:0/633697633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430106fa0 0x7f9430109390 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 --2- 172.21.15.3:0/633697633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430104670 0x7f9430106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 --2- 172.21.15.3:0/633697633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9430101d40 0x7f9430104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 -- 172.21.15.3:0/633697633 >> 172.21.15.3:0/633697633 conn(0x7f94300fb6d0 msgr2=0x7f94300fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 -- 172.21.15.3:0/633697633 shutdown_connections 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 -- 172.21.15.3:0/633697633 wait complete. 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 Processor -- start 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.852+0000 7f9436415640 1 -- start start 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 0x7f9430197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9430104670 0x7f94301977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:55.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430106fa0 0x7f943019e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943019ed80 con 0x7f9430101d40 2024-02-09T22:48:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943019eef0 con 0x7f9430104670 2024-02-09T22:48:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943019f0b0 con 0x7f9430106fa0 2024-02-09T22:48:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 0x7f9430197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 0x7f9430197290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44822/0 (socket says 172.21.15.3:44822) 2024-02-09T22:48:55.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 -- 172.21.15.3:0/3206484177 learned_addr learned my addr 172.21.15.3:0/3206484177 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942f7fe640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9430104670 0x7f94301977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f94349b2640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430106fa0 0x7f943019e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 -- 172.21.15.3:0/3206484177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430106fa0 msgr2=0x7f943019e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430106fa0 0x7f943019e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 -- 172.21.15.3:0/3206484177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9430104670 msgr2=0x7f94301977d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9430104670 0x7f94301977d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 -- 172.21.15.3:0/3206484177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9428009900 con 0x7f9430101d40 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f94349b2640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430106fa0 0x7f943019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942ffff640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 0x7f9430197290 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f942000c890 tx=0x7f942000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420004220 con 0x7f9430101d40 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f943019f390 con 0x7f9430101d40 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9420007e90 con 0x7f9430101d40 2024-02-09T22:48:55.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9420004a00 con 0x7f9430101d40 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.853+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f943019f8a0 con 0x7f9430101d40 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.854+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94300fd1c0 con 0x7f9430101d40 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.856+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f94200079b0 con 0x7f9430101d40 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.856+0000 7f942d7fa640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f94180a20c0 0x7f94180a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.856+0000 7f942f7fe640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f94180a20c0 0x7f94180a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.856+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f94200cd8b0 con 0x7f9430101d40 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.857+0000 7f942f7fe640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f94180a20c0 0x7f94180a4580 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f94301987a0 tx=0x7f942400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:55.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:55.860+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9420096d20 con 0x7f9430101d40 2024-02-09T22:48:56.041 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:55 smithi003 podman[73244]: 2024-02-09 22:48:55.938436336 +0000 UTC m=+0.395182098 container init 1ddc91dd56b1fffc8d7f1a0b9491125d8abbe9312377b529e2b24fa0f28c8e71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:48:56.041 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:55 smithi003 podman[73244]: 2024-02-09 22:48:55.941853744 +0000 UTC m=+0.398599496 container start 1ddc91dd56b1fffc8d7f1a0b9491125d8abbe9312377b529e2b24fa0f28c8e71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a) 2024-02-09T22:48:56.041 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:55 smithi003 ceph-mds[73303]: -- 172.21.15.3:0/4134821209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cf181321a0 con 0x55cf18109000 2024-02-09T22:48:56.041 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:55 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g[73280]: starting mds.g at 2024-02-09T22:48:56.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.083+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f943010cc70 con 0x7f9430101d40 2024-02-09T22:48:56.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.084+0000 7f942d7fa640 1 -- 172.21.15.3:0/3206484177 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7643 (secure 0 0 0) 0x7f942009a3e0 con 0x7f9430101d40 2024-02-09T22:48:56.087 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:56.087 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-02-09T22:48:56.087 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 10 2024-02-09T22:48:56.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f94180a20c0 msgr2=0x7f94180a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f94180a20c0 0x7f94180a4580 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f94301987a0 tx=0x7f942400b390 comp rx=0 tx=0).stop 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 msgr2=0x7f9430197290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 0x7f9430197290 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f942000c890 tx=0x7f942000cd50 comp rx=0 tx=0).stop 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 shutdown_connections 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f94180a20c0 0x7f94180a4580 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9430104670 0x7f94301977d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9430106fa0 0x7f943019e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 --2- 172.21.15.3:0/3206484177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9430101d40 0x7f9430197290 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 >> 172.21.15.3:0/3206484177 conn(0x7f94300fb6d0 msgr2=0x7f9430102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 shutdown_connections 2024-02-09T22:48:56.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:56.086+0000 7f9436415640 1 -- 172.21.15.3:0/3206484177 wait complete. 2024-02-09T22:48:56.383 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:56 smithi003 bash[73244]: 1ddc91dd56b1fffc8d7f1a0b9491125d8abbe9312377b529e2b24fa0f28c8e71 2024-02-09T22:48:56.383 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 22:48:56 smithi003 systemd[1]: Started Ceph mds.g for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:56.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3206484177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:56.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:56.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:56.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:56 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:56.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3206484177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:56.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:56.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:56.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:56 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3206484177' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:56.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:56.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:56.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:56 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:57.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:57 smithi003 ceph-mon[32678]: Deploying daemon mds.a on smithi003 2024-02-09T22:48:57.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:57 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.3:6838/3695104270,v1:172.21.15.3:6839/3695104270] up:boot 2024-02-09T22:48:57.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:57 smithi003 ceph-mon[32678]: fsmap 10 up:standby 2024-02-09T22:48:57.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:57 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T22:48:57.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:57 smithi003 ceph-mon[32678]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:57.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:57 smithi064 ceph-mon[36638]: Deploying daemon mds.a on smithi003 2024-02-09T22:48:57.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:57 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.3:6838/3695104270,v1:172.21.15.3:6839/3695104270] up:boot 2024-02-09T22:48:57.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:57 smithi064 ceph-mon[36638]: fsmap 10 up:standby 2024-02-09T22:48:57.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:57 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T22:48:57.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:57 smithi064 ceph-mon[36638]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:57 smithi089 ceph-mon[36665]: Deploying daemon mds.a on smithi003 2024-02-09T22:48:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:57 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.3:6838/3695104270,v1:172.21.15.3:6839/3695104270] up:boot 2024-02-09T22:48:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:57 smithi089 ceph-mon[36665]: fsmap 10 up:standby 2024-02-09T22:48:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:57 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T22:48:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:57 smithi089 ceph-mon[36665]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:58.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:48:58.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 -- 172.21.15.3:0/1400010328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c106f40 msgr2=0x7fdc3c109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 --2- 172.21.15.3:0/1400010328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c106f40 0x7fdc3c109390 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdc34009ca0 tx=0x7fdc3402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 -- 172.21.15.3:0/1400010328 shutdown_connections 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 --2- 172.21.15.3:0/1400010328 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c106f40 0x7fdc3c109390 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 --2- 172.21.15.3:0/1400010328 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3c104610 0x7fdc3c106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 --2- 172.21.15.3:0/1400010328 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c101ce0 0x7fdc3c1040d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 -- 172.21.15.3:0/1400010328 >> 172.21.15.3:0/1400010328 conn(0x7fdc3c0fb6f0 msgr2=0x7fdc3c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.252+0000 7fdc42993640 1 -- 172.21.15.3:0/1400010328 shutdown_connections 2024-02-09T22:48:58.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.253+0000 7fdc42993640 1 -- 172.21.15.3:0/1400010328 wait complete. 2024-02-09T22:48:58.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.253+0000 7fdc42993640 1 Processor -- start 2024-02-09T22:48:58.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.253+0000 7fdc42993640 1 -- start start 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc42993640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3c101ce0 0x7fdc3c197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc42993640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 0x7fdc3c1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc42993640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c106f40 0x7fdc3c19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc42993640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c19ed70 con 0x7fdc3c104610 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc42993640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c19eee0 con 0x7fdc3c106f40 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc42993640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc3c19f0a0 con 0x7fdc3c101ce0 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc3b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 0x7fdc3c1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc3b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 0x7fdc3c1977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51760/0 (socket says 172.21.15.3:51760) 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc3b7fe640 1 -- 172.21.15.3:0/925956770 learned_addr learned my addr 172.21.15.3:0/925956770 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc40f30640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c106f40 0x7fdc3c19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.254+0000 7fdc3bfff640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3c101ce0 0x7fdc3c197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc40f30640 1 -- 172.21.15.3:0/925956770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3c101ce0 msgr2=0x7fdc3c197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc40f30640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3c101ce0 0x7fdc3c197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc40f30640 1 -- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 msgr2=0x7fdc3c1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc40f30640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 0x7fdc3c1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc40f30640 1 -- 172.21.15.3:0/925956770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc34009900 con 0x7fdc3c106f40 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc3b7fe640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 0x7fdc3c1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc40f30640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c106f40 0x7fdc3c19e830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc3402c630 tx=0x7fdc34034a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc340329b0 con 0x7fdc3c106f40 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc3c19f320 con 0x7fdc3c106f40 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc34003e50 con 0x7fdc3c106f40 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc34040450 con 0x7fdc3c106f40 2024-02-09T22:48:58.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.255+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc3c19f7f0 con 0x7fdc3c106f40 2024-02-09T22:48:58.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.256+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc3c0fd1e0 con 0x7fdc3c106f40 2024-02-09T22:48:58.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.257+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fdc3400a040 con 0x7fdc3c106f40 2024-02-09T22:48:58.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.258+0000 7fdc397fa640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdc240a20c0 0x7fdc240a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:48:58.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.258+0000 7fdc3bfff640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdc240a20c0 0x7fdc240a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:48:58.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.258+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fdc340f0250 con 0x7fdc3c106f40 2024-02-09T22:48:58.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.259+0000 7fdc3bfff640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdc240a20c0 0x7fdc240a4580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009bd0 tx=0x7fdc2c005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:48:58.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.262+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdc340b96c0 con 0x7fdc3c106f40 2024-02-09T22:48:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.507+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc3c02cde0 con 0x7fdc3c106f40 2024-02-09T22:48:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.508+0000 7fdc397fa640 1 -- 172.21.15.3:0/925956770 <== mon.2 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8432 (secure 0 0 0) 0x7fdc340bcd80 con 0x7fdc3c106f40 2024-02-09T22:48:58.511 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:48:58.512 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-02-09T22:48:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 11 2024-02-09T22:48:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdc240a20c0 msgr2=0x7fdc240a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdc240a20c0 0x7fdc240a4580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fdc2c009bd0 tx=0x7fdc2c005f90 comp rx=0 tx=0).stop 2024-02-09T22:48:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c106f40 msgr2=0x7fdc3c19e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:48:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c106f40 0x7fdc3c19e830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fdc3402c630 tx=0x7fdc34034a90 comp rx=0 tx=0).stop 2024-02-09T22:48:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 shutdown_connections 2024-02-09T22:48:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdc240a20c0 0x7fdc240a4580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdc3c106f40 0x7fdc3c19e830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdc3c104610 0x7fdc3c1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 --2- 172.21.15.3:0/925956770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdc3c101ce0 0x7fdc3c197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:48:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 >> 172.21.15.3:0/925956770 conn(0x7fdc3c0fb6f0 msgr2=0x7fdc3c1025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:48:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 shutdown_connections 2024-02-09T22:48:58.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:48:58.510+0000 7fdc42993640 1 -- 172.21.15.3:0/925956770 wait complete. 2024-02-09T22:48:58.528 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 podman[73631]: 2024-02-09 22:48:58.422764991 +0000 UTC m=+0.124425998 container create 6792ec0ee9495e58d4fa68a5ef37c5584a2e3d8ad546bb997954700fb090a00e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-09T22:48:58.828 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:58 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/925956770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:58.829 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 podman[73631]: 2024-02-09 22:48:58.734166059 +0000 UTC m=+0.435827091 container init 6792ec0ee9495e58d4fa68a5ef37c5584a2e3d8ad546bb997954700fb090a00e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-02-09T22:48:58.829 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 podman[73631]: 2024-02-09 22:48:58.737127455 +0000 UTC m=+0.438788465 container start 6792ec0ee9495e58d4fa68a5ef37c5584a2e3d8ad546bb997954700fb090a00e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:48:58.829 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 ceph-mds[73662]: -- 172.21.15.3:0/2206212891 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564634c861a0 con 0x564634c5d000 2024-02-09T22:48:58.829 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a[73656]: starting mds.a at 2024-02-09T22:48:58.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:58 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/925956770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:58.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:58 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/925956770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:48:59.132 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 bash[73631]: 6792ec0ee9495e58d4fa68a5ef37c5584a2e3d8ad546bb997954700fb090a00e 2024-02-09T22:48:59.132 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 22:48:58 smithi003 systemd[1]: Started Ceph mds.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:59.827 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:48:59 smithi064 ceph-mon[36638]: Deploying daemon mds.b on smithi064 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:48:59 smithi003 ceph-mon[32678]: Deploying daemon mds.b on smithi064 2024-02-09T22:48:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:48:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:48:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-02-09T22:48:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-02-09T22:48:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:48:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:48:59 smithi089 ceph-mon[36665]: Deploying daemon mds.b on smithi064 2024-02-09T22:49:00.533 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:49:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 -- 172.21.15.3:0/352531610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101d10 msgr2=0x7fa2301085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 --2- 172.21.15.3:0/352531610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101d10 0x7fa2301085a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa228009900 tx=0x7fa22802bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 -- 172.21.15.3:0/352531610 shutdown_connections 2024-02-09T22:49:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 --2- 172.21.15.3:0/352531610 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230101350 0x7fa2301017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 --2- 172.21.15.3:0/352531610 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101d10 0x7fa2301085a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 --2- 172.21.15.3:0/352531610 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230100160 0x7fa230100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 -- 172.21.15.3:0/352531610 >> 172.21.15.3:0/352531610 conn(0x7fa2300fb6d0 msgr2=0x7fa2300fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 -- 172.21.15.3:0/352531610 shutdown_connections 2024-02-09T22:49:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 -- 172.21.15.3:0/352531610 wait complete. 2024-02-09T22:49:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.650+0000 7fa234e0e640 1 Processor -- start 2024-02-09T22:49:00.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 -- start start 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230100160 0x7fa230199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 0x7fa2301999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230101d10 0x7fa2301a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2301a0f50 con 0x7fa230101350 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2301a10c0 con 0x7fa230100160 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa234e0e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2301a1280 con 0x7fa230101d10 2024-02-09T22:49:00.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 0x7fa2301999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 0x7fa2301999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51812/0 (socket says 172.21.15.3:51812) 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 -- 172.21.15.3:0/2677424970 learned_addr learned my addr 172.21.15.3:0/2677424970 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22ed9d640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230101d10 0x7fa2301a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22e59c640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230100160 0x7fa230199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 -- 172.21.15.3:0/2677424970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230101d10 msgr2=0x7fa2301a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230101d10 0x7fa2301a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 -- 172.21.15.3:0/2677424970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230100160 msgr2=0x7fa230199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230100160 0x7fa230199460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 -- 172.21.15.3:0/2677424970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa228009560 con 0x7fa230101350 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22ed9d640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230101d10 0x7fa2301a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22e59c640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230100160 0x7fa230199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa22dd9b640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 0x7fa2301999a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa21800b4a0 tx=0x7fa21800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.651+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218010610 con 0x7fa230101350 2024-02-09T22:49:00.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.652+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa218010c50 con 0x7fa230101350 2024-02-09T22:49:00.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.652+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2301a1560 con 0x7fa230101350 2024-02-09T22:49:00.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.652+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa218017e20 con 0x7fa230101350 2024-02-09T22:49:00.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.652+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2301a1af0 con 0x7fa230101350 2024-02-09T22:49:00.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.653+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa23010bbb0 con 0x7fa230101350 2024-02-09T22:49:00.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.654+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa218017480 con 0x7fa230101350 2024-02-09T22:49:00.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.654+0000 7fa2277fe640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa21c0a20c0 0x7fa21c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:00.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.655+0000 7fa22e59c640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa21c0a20c0 0x7fa21c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:00.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.655+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2180cd310 con 0x7fa230101350 2024-02-09T22:49:00.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.655+0000 7fa22e59c640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa21c0a20c0 0x7fa21c0a4580 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2301011b0 tx=0x7fa220008370 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:00.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.658+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa218096780 con 0x7fa230101350 2024-02-09T22:49:00.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.878+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2301a1d70 con 0x7fa230101350 2024-02-09T22:49:00.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.879+0000 7fa2277fe640 1 -- 172.21.15.3:0/2677424970 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9221 (secure 0 0 0) 0x7fa2180092b0 con 0x7fa230101350 2024-02-09T22:49:00.882 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:00.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:00 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.3:6840/3916323155,v1:172.21.15.3:6841/3916323155] up:boot 2024-02-09T22:49:00.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:00 smithi003 ceph-mon[32678]: fsmap 11 up:standby 2024-02-09T22:49:00.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:00 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T22:49:00.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:00 smithi003 ceph-mon[32678]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:00.883 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-02-09T22:49:00.883 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 12 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa21c0a20c0 msgr2=0x7fa21c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa21c0a20c0 0x7fa21c0a4580 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa2301011b0 tx=0x7fa220008370 comp rx=0 tx=0).stop 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 msgr2=0x7fa2301999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 0x7fa2301999a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa21800b4a0 tx=0x7fa21800b960 comp rx=0 tx=0).stop 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 shutdown_connections 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa21c0a20c0 0x7fa21c0a4580 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa230101d10 0x7fa2301a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa230101350 0x7fa2301999a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 --2- 172.21.15.3:0/2677424970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa230100160 0x7fa230199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:00.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 >> 172.21.15.3:0/2677424970 conn(0x7fa2300fb6d0 msgr2=0x7fa2300fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:00.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 shutdown_connections 2024-02-09T22:49:00.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:00.881+0000 7fa234e0e640 1 -- 172.21.15.3:0/2677424970 wait complete. 2024-02-09T22:49:00.931 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:00 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.3:6840/3916323155,v1:172.21.15.3:6841/3916323155] up:boot 2024-02-09T22:49:00.932 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:00 smithi064 ceph-mon[36638]: fsmap 11 up:standby 2024-02-09T22:49:00.932 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:00 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T22:49:00.932 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:00 smithi064 ceph-mon[36638]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:00 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.3:6840/3916323155,v1:172.21.15.3:6841/3916323155] up:boot 2024-02-09T22:49:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:00 smithi089 ceph-mon[36665]: fsmap 11 up:standby 2024-02-09T22:49:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:00 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T22:49:00.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:00 smithi089 ceph-mon[36665]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:01.187 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 systemd[1]: Starting Ceph mds.b for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T22:49:01.603 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 podman[67218]: 2024-02-09 22:49:01.248750295 +0000 UTC m=+0.018432653 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T22:49:01.603 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 podman[67218]: 2024-02-09 22:49:01.379808066 +0000 UTC m=+0.149490417 container create 85309a727c7612cf340e4cf2807b2236a725dab6aa31c7f1f9eadf28d1adeb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T22:49:01.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2677424970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:01.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:01 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2677424970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:01.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 podman[67218]: 2024-02-09 22:49:01.624458084 +0000 UTC m=+0.394140435 container init 85309a727c7612cf340e4cf2807b2236a725dab6aa31c7f1f9eadf28d1adeb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T22:49:01.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 podman[67218]: 2024-02-09 22:49:01.627585932 +0000 UTC m=+0.397268281 container start 85309a727c7612cf340e4cf2807b2236a725dab6aa31c7f1f9eadf28d1adeb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True) 2024-02-09T22:49:01.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 ceph-mds[67237]: -- 172.21.15.64:0/2135364498 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5651d3f741a0 con 0x5651d3f4a800 2024-02-09T22:49:01.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: starting mds.b at 2024-02-09T22:49:01.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 bash[67218]: 85309a727c7612cf340e4cf2807b2236a725dab6aa31c7f1f9eadf28d1adeb57 2024-02-09T22:49:01.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:01 smithi064 systemd[1]: Started Ceph mds.b for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T22:49:01.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:01 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2677424970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:02.897 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- 172.21.15.3:0/820032197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 msgr2=0x7f9ef8104ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- 172.21.15.3:0/820032197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8104ae0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009900 tx=0x7f9eec02bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- 172.21.15.3:0/820032197 shutdown_connections 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- 172.21.15.3:0/820032197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ef8105020 0x7f9ef810b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- 172.21.15.3:0/820032197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8104ae0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- 172.21.15.3:0/820032197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef8103470 0x7f9ef8103890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- 172.21.15.3:0/820032197 >> 172.21.15.3:0/820032197 conn(0x7f9ef80fe9e0 msgr2=0x7f9ef8100e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- 172.21.15.3:0/820032197 shutdown_connections 2024-02-09T22:49:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- 172.21.15.3:0/820032197 wait complete. 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 Processor -- start 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- start start 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ef8103470 0x7f9ef8199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef8105020 0x7f9ef81a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef81a1020 con 0x7f9ef8104660 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef81a1190 con 0x7f9ef8103470 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.002+0000 7f9f0000f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef81a1350 con 0x7f9ef8105020 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51830/0 (socket says 172.21.15.3:51830) 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 -- 172.21.15.3:0/2341629336 learned_addr learned my addr 172.21.15.3:0/2341629336 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 -- 172.21.15.3:0/2341629336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef8105020 msgr2=0x7f9ef81a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef8105020 0x7f9ef81a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 -- 172.21.15.3:0/2341629336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ef8103470 msgr2=0x7f9ef8199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efddab640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ef8103470 0x7f9ef8199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ef8103470 0x7f9ef8199530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 -- 172.21.15.3:0/2341629336 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec009560 con 0x7f9ef8104660 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9efd5aa640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8199a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9eec006010 tx=0x7f9eec0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec0044c0 con 0x7f9ef8104660 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9eec0052d0 con 0x7f9ef8104660 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec003e10 con 0x7f9ef8104660 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef81a15d0 con 0x7f9ef8104660 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.003+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef81a1bb0 con 0x7f9ef8104660 2024-02-09T22:49:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.004+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef810a790 con 0x7f9ef8104660 2024-02-09T22:49:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.005+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f9eec005440 con 0x7f9ef8104660 2024-02-09T22:49:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.006+0000 7f9ef6ffd640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9ee00a2140 0x7f9ee00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.006+0000 7f9efddab640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9ee00a2140 0x7f9ee00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.006+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9eec0efc50 con 0x7f9ef8104660 2024-02-09T22:49:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.006+0000 7f9efddab640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9ee00a2140 0x7f9ee00a4600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9ef81044c0 tx=0x7f9ef0009500 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:03.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.010+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9eec0b9020 con 0x7f9ef8104660 2024-02-09T22:49:03.025 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.025 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.025 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.025 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.025 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.026 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:49:03.026 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:03.026 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] up:boot 2024-02-09T22:49:03.026 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: fsmap 12 up:standby 2024-02-09T22:49:03.026 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:02 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:49:03.031 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:03.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] up:boot 2024-02-09T22:49:03.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: fsmap 12 up:standby 2024-02-09T22:49:03.032 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:02 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T22:49:03.038 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.038 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.038 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.038 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.039 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.039 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:49:03.039 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:03.040 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] up:boot 2024-02-09T22:49:03.040 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: fsmap 12 up:standby 2024-02-09T22:49:03.040 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:02 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T22:49:03.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.233+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ef81a1760 con 0x7f9ef8104660 2024-02-09T22:49:03.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.234+0000 7f9ef6ffd640 1 -- 172.21.15.3:0/2341629336 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10013 (secure 0 0 0) 0x7f9ef81a1760 con 0x7f9ef8104660 2024-02-09T22:49:03.237 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:03.237 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14634,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14646,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24427,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24437,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24470,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24475,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24481,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24496,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24503,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24533,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-02-09T22:49:03.237 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 13 2024-02-09T22:49:03.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9ee00a2140 msgr2=0x7f9ee00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9ee00a2140 0x7f9ee00a4600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f9ef81044c0 tx=0x7f9ef0009500 comp rx=0 tx=0).stop 2024-02-09T22:49:03.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 msgr2=0x7f9ef8199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8199a70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9eec006010 tx=0x7f9eec0040c0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 shutdown_connections 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9ee00a2140 0x7f9ee00a4600 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ef8105020 0x7f9ef81a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ef8104660 0x7f9ef8199a70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 --2- 172.21.15.3:0/2341629336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ef8103470 0x7f9ef8199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 >> 172.21.15.3:0/2341629336 conn(0x7f9ef80fe9e0 msgr2=0x7f9ef8107960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 shutdown_connections 2024-02-09T22:49:03.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.236+0000 7f9f0000f640 1 -- 172.21.15.3:0/2341629336 wait complete. 2024-02-09T22:49:03.254 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:49:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 -- 172.21.15.3:0/2691602789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 msgr2=0x7fa250109800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 --2- 172.21.15.3:0/2691602789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa250109800 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fa248009ca0 tx=0x7fa24802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 -- 172.21.15.3:0/2691602789 shutdown_connections 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 --2- 172.21.15.3:0/2691602789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2501025b0 0x7fa250102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 --2- 172.21.15.3:0/2691602789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa250109800 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 --2- 172.21.15.3:0/2691602789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2500ff540 0x7fa2500ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 -- 172.21.15.3:0/2691602789 >> 172.21.15.3:0/2691602789 conn(0x7fa2500fae50 msgr2=0x7fa2500fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 -- 172.21.15.3:0/2691602789 shutdown_connections 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 -- 172.21.15.3:0/2691602789 wait complete. 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 Processor -- start 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.364+0000 7fa256348640 1 -- start start 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa256348640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2500ff540 0x7fa250197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa256348640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2501025b0 0x7fa2501977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa256348640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa25019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa256348640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25019ed60 con 0x7fa250102f70 2024-02-09T22:49:03.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa256348640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25019eed0 con 0x7fa2500ff540 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa256348640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa25019f090 con 0x7fa2501025b0 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa25019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa25019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51852/0 (socket says 172.21.15.3:51852) 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 -- 172.21.15.3:0/626499352 learned_addr learned my addr 172.21.15.3:0/626499352 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa24f7fe640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2501025b0 0x7fa2501977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa24ffff640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2500ff540 0x7fa250197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 -- 172.21.15.3:0/626499352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2501025b0 msgr2=0x7fa2501977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2501025b0 0x7fa2501977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 -- 172.21.15.3:0/626499352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2500ff540 msgr2=0x7fa250197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2500ff540 0x7fa250197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 -- 172.21.15.3:0/626499352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248009900 con 0x7fa250102f70 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa24f7fe640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2501025b0 0x7fa2501977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.365+0000 7fa2548e5640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa25019e820 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa24802c630 tx=0x7fa248034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.366+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248032eb0 con 0x7fa250102f70 2024-02-09T22:49:03.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.366+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa25019f310 con 0x7fa250102f70 2024-02-09T22:49:03.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.366+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa248003e50 con 0x7fa250102f70 2024-02-09T22:49:03.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.366+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa25019f810 con 0x7fa250102f70 2024-02-09T22:49:03.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.366+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa248040450 con 0x7fa250102f70 2024-02-09T22:49:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.367+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa250108680 con 0x7fa250102f70 2024-02-09T22:49:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.369+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa24800a040 con 0x7fa250102f70 2024-02-09T22:49:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.369+0000 7fa24d7fa640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa2380a20c0 0x7fa2380a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.369+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa2480f0b90 con 0x7fa250102f70 2024-02-09T22:49:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.369+0000 7fa24ffff640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa2380a20c0 0x7fa2380a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.370+0000 7fa24ffff640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa2380a20c0 0x7fa2380a4580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa250068240 tx=0x7fa240005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:03.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.372+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa2480ba000 con 0x7fa250102f70 2024-02-09T22:49:03.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.583+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa25010ce80 con 0x7fa250102f70 2024-02-09T22:49:03.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.584+0000 7fa24d7fa640 1 -- 172.21.15.3:0/626499352 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18265 (secure 0 0 0) 0x7fa2480bd6c0 con 0x7fa250102f70 2024-02-09T22:49:03.587 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:03.588 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":82,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:48:17.465165+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:49:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa2380a20c0 msgr2=0x7fa2380a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa2380a20c0 0x7fa2380a4580 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fa250068240 tx=0x7fa240005f90 comp rx=0 tx=0).stop 2024-02-09T22:49:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 msgr2=0x7fa25019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa25019e820 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fa24802c630 tx=0x7fa248034cc0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 shutdown_connections 2024-02-09T22:49:03.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa2380a20c0 0x7fa2380a4580 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa250102f70 0x7fa25019e820 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2501025b0 0x7fa2501977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 --2- 172.21.15.3:0/626499352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2500ff540 0x7fa250197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 >> 172.21.15.3:0/626499352 conn(0x7fa2500fae50 msgr2=0x7fa250105850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:03.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 shutdown_connections 2024-02-09T22:49:03.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.587+0000 7fa256348640 1 -- 172.21.15.3:0/626499352 wait complete. 2024-02-09T22:49:03.605 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:49:03.605 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:49:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/3834627716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4100150 msgr2=0x7fb1b4100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/3834627716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4100150 0x7fb1b4100570 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009900 tx=0x7fb1b002bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/3834627716 shutdown_connections 2024-02-09T22:49:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/3834627716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4101340 0x7fb1b41017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/3834627716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1b4101d00 0x7fb1b4108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/3834627716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4100150 0x7fb1b4100570 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/3834627716 >> 172.21.15.3:0/3834627716 conn(0x7fb1b40fb6c0 msgr2=0x7fb1b40fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/3834627716 shutdown_connections 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/3834627716 wait complete. 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 Processor -- start 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- start start 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1b4100150 0x7fb1b4197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 0x7fb1b4197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4101d00 0x7fb1b419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b419ee20 con 0x7fb1b4101340 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b419ef90 con 0x7fb1b4101d00 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.715+0000 7fb1bbd6c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1b419f150 con 0x7fb1b4100150 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 0x7fb1b4197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 0x7fb1b4197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51862/0 (socket says 172.21.15.3:51862) 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1ba309640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4101d00 0x7fb1b419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 -- 172.21.15.3:0/2929703202 learned_addr learned my addr 172.21.15.3:0/2929703202 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9b08640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1b4100150 0x7fb1b4197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 -- 172.21.15.3:0/2929703202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1b4100150 msgr2=0x7fb1b4197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1b4100150 0x7fb1b4197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 -- 172.21.15.3:0/2929703202 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4101d00 msgr2=0x7fb1b419e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4101d00 0x7fb1b419e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 -- 172.21.15.3:0/2929703202 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1b0009560 con 0x7fb1b4101340 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1b9307640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 0x7fb1b4197870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400e9f0 tx=0x7fb1a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4009700 con 0x7fb1b4101340 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1a400cd80 con 0x7fb1b4101340 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1b419f430 con 0x7fb1b4101340 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1a4018920 con 0x7fb1b4101340 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.716+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1b419f940 con 0x7fb1b4101340 2024-02-09T22:49:03.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.717+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1b410bb60 con 0x7fb1b4101340 2024-02-09T22:49:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.718+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb1a4018a80 con 0x7fb1b4101340 2024-02-09T22:49:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.719+0000 7fb1aaffd640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb1a00a20c0 0x7fb1a00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.719+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb1a4014070 con 0x7fb1b4101340 2024-02-09T22:49:03.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.719+0000 7fb1b9b08640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb1a00a20c0 0x7fb1a00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:03.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.719+0000 7fb1b9b08640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb1a00a20c0 0x7fb1a00a4580 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005ab0 tx=0x7fb1b0005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:03.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.722+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb1a4097530 con 0x7fb1b4101340 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2341629336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/626499352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:03 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:03.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.905+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb1b419fdd0 con 0x7fb1b4101340 2024-02-09T22:49:03.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.905+0000 7fb1aaffd640 1 -- 172.21.15.3:0/2929703202 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb1a409abf0 con 0x7fb1b4101340 2024-02-09T22:49:03.909 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.907+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb1a00a20c0 msgr2=0x7fb1a00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.907+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb1a00a20c0 0x7fb1a00a4580 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0005ab0 tx=0x7fb1b0005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.907+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 msgr2=0x7fb1b4197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.907+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 0x7fb1b4197870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb1a400e9f0 tx=0x7fb1a400eeb0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 shutdown_connections 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb1a00a20c0 0x7fb1a00a4580 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb1b4101d00 0x7fb1b419e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb1b4101340 0x7fb1b4197870 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 --2- 172.21.15.3:0/2929703202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb1b4100150 0x7fb1b4197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 >> 172.21.15.3:0/2929703202 conn(0x7fb1b40fb6c0 msgr2=0x7fb1b40fd1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:03.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 shutdown_connections 2024-02-09T22:49:03.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:03.908+0000 7fb1bbd6c640 1 -- 172.21.15.3:0/2929703202 wait complete. 2024-02-09T22:49:03.922 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-02-09T22:49:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 -- 172.21.15.3:0/129613323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c101d50 msgr2=0x7fa72c104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 --2- 172.21.15.3:0/129613323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c101d50 0x7fa72c104140 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009900 tx=0x7fa71c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 -- 172.21.15.3:0/129613323 shutdown_connections 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 --2- 172.21.15.3:0/129613323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa72c106fb0 0x7fa72c1093a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 --2- 172.21.15.3:0/129613323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72c104680 0x7fa72c106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 --2- 172.21.15.3:0/129613323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c101d50 0x7fa72c104140 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 -- 172.21.15.3:0/129613323 >> 172.21.15.3:0/129613323 conn(0x7fa72c0fb6a0 msgr2=0x7fa72c0fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 -- 172.21.15.3:0/129613323 shutdown_connections 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.022+0000 7fa731655640 1 -- 172.21.15.3:0/129613323 wait complete. 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 Processor -- start 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 -- start start 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa72c101d50 0x7fa72c1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72c104680 0x7fa72c197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 0x7fa72c19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c19edc0 con 0x7fa72c106fb0 2024-02-09T22:49:04.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c19ef30 con 0x7fa72c101d50 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa731655640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa72c19f0f0 con 0x7fa72c104680 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 0x7fa72c19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 0x7fa72c19e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51882/0 (socket says 172.21.15.3:51882) 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 -- 172.21.15.3:0/1998402304 learned_addr learned my addr 172.21.15.3:0/1998402304 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72affd640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa72c101d50 0x7fa72c1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72a7fc640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72c104680 0x7fa72c197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 -- 172.21.15.3:0/1998402304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72c104680 msgr2=0x7fa72c197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72c104680 0x7fa72c197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 -- 172.21.15.3:0/1998402304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa72c101d50 msgr2=0x7fa72c1972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa72c101d50 0x7fa72c1972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 -- 172.21.15.3:0/1998402304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa71c009560 con 0x7fa72c106fb0 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.023+0000 7fa72b7fe640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 0x7fa72c19e880 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa72400cad0 tx=0x7fa72400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.024+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72400ae20 con 0x7fa72c106fb0 2024-02-09T22:49:04.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.024+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa724005020 con 0x7fa72c106fb0 2024-02-09T22:49:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.024+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa72c19f3d0 con 0x7fa72c106fb0 2024-02-09T22:49:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.024+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72400f6b0 con 0x7fa72c106fb0 2024-02-09T22:49:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.024+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa72c19f890 con 0x7fa72c106fb0 2024-02-09T22:49:04.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.025+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa72c0fd190 con 0x7fa72c106fb0 2024-02-09T22:49:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.026+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa724005520 con 0x7fa72c106fb0 2024-02-09T22:49:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.026+0000 7fa713fff640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7140a20c0 0x7fa7140a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.026+0000 7fa72affd640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7140a20c0 0x7fa7140a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.026+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa7240cd770 con 0x7fa72c106fb0 2024-02-09T22:49:04.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.027+0000 7fa72affd640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7140a20c0 0x7fa7140a4580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009230 tx=0x7fa71c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:04.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.030+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa724096c90 con 0x7fa72c106fb0 2024-02-09T22:49:04.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2341629336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:04.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/626499352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:04.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:03 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.217+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa72c02ce50 con 0x7fa72c106fb0 2024-02-09T22:49:04.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.219+0000 7fa713fff640 1 -- 172.21.15.3:0/1998402304 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa72409a350 con 0x7fa72c106fb0 2024-02-09T22:49:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:04.222 INFO:teuthology.orchestra.run.smithi003.stdout:[ 2024-02-09T22:49:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T22:49:04.222 INFO:teuthology.orchestra.run.smithi003.stdout: "poolnum": 1, 2024-02-09T22:49:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: "poolname": ".mgr" 2024-02-09T22:49:04.223 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:49:04.223 INFO:teuthology.orchestra.run.smithi003.stdout:] 2024-02-09T22:49:04.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7140a20c0 msgr2=0x7fa7140a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7140a20c0 0x7fa7140a4580 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa71c009230 tx=0x7fa71c005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 msgr2=0x7fa72c19e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 0x7fa72c19e880 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa72400cad0 tx=0x7fa72400cf90 comp rx=0 tx=0).stop 2024-02-09T22:49:04.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 shutdown_connections 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7140a20c0 0x7fa7140a4580 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa72c106fb0 0x7fa72c19e880 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa72c104680 0x7fa72c197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 --2- 172.21.15.3:0/1998402304 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa72c101d50 0x7fa72c1972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 >> 172.21.15.3:0/1998402304 conn(0x7fa72c0fb6a0 msgr2=0x7fa72c102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 shutdown_connections 2024-02-09T22:49:04.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.222+0000 7fa731655640 1 -- 172.21.15.3:0/1998402304 wait complete. 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2341629336' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/626499352' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:03 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:04.237 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-02-09T22:49:04.237 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-02-09T22:49:04.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 -- 172.21.15.3:0/3389466033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0401045e0 msgr2=0x7fb0401069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 --2- 172.21.15.3:0/3389466033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0401045e0 0x7fb0401069d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb034009900 tx=0x7fb03402bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:04.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 -- 172.21.15.3:0/3389466033 shutdown_connections 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 --2- 172.21.15.3:0/3389466033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb040106f10 0x7fb040109360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 --2- 172.21.15.3:0/3389466033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0401045e0 0x7fb0401069d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 --2- 172.21.15.3:0/3389466033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb040101cb0 0x7fb0401040a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 -- 172.21.15.3:0/3389466033 >> 172.21.15.3:0/3389466033 conn(0x7fb0400fb660 msgr2=0x7fb0400fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 -- 172.21.15.3:0/3389466033 shutdown_connections 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.342+0000 7fb045adf640 1 -- 172.21.15.3:0/3389466033 wait complete. 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 Processor -- start 2024-02-09T22:49:04.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 -- start start 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 0x7fb040101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0401045e0 0x7fb0400ffc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb040106f10 0x7fb0401001a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040100900 con 0x7fb040101cb0 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb040100a70 con 0x7fb040106f10 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb045adf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0401a1320 con 0x7fb0401045e0 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb03f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 0x7fb040101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb03f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 0x7fb040101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51898/0 (socket says 172.21.15.3:51898) 2024-02-09T22:49:04.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb03f7fe640 1 -- 172.21.15.3:0/1813337280 learned_addr learned my addr 172.21.15.3:0/1813337280 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.343+0000 7fb03effd640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0401045e0 0x7fb0400ffc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03ffff640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb040106f10 0x7fb0401001a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03f7fe640 1 -- 172.21.15.3:0/1813337280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0401045e0 msgr2=0x7fb0400ffc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03f7fe640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0401045e0 0x7fb0400ffc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03f7fe640 1 -- 172.21.15.3:0/1813337280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb040106f10 msgr2=0x7fb0401001a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03f7fe640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb040106f10 0x7fb0401001a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03f7fe640 1 -- 172.21.15.3:0/1813337280 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009560 con 0x7fb040101cb0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03f7fe640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 0x7fb040101620 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb03000d980 tx=0x7fb03000de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb030009c30 con 0x7fb040101cb0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb03000b660 con 0x7fb040101cb0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03000f6b0 con 0x7fb040101cb0 2024-02-09T22:49:04.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0401a15d0 con 0x7fb040101cb0 2024-02-09T22:49:04.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.344+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0401a1ab0 con 0x7fb040101cb0 2024-02-09T22:49:04.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.345+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0400fd150 con 0x7fb040101cb0 2024-02-09T22:49:04.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.347+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb030010460 con 0x7fb040101cb0 2024-02-09T22:49:04.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.347+0000 7fb03cff9640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb0280a20c0 0x7fb0280a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:04.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.348+0000 7fb03effd640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb0280a20c0 0x7fb0280a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:04.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.348+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb0300cd550 con 0x7fb040101cb0 2024-02-09T22:49:04.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.348+0000 7fb03effd640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb0280a20c0 0x7fb0280a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb034005ab0 tx=0x7fb034005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:04.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.351+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb030096920 con 0x7fb040101cb0 2024-02-09T22:49:04.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:04.547+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb04010cdc0 con 0x7fb040101cb0 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2929703202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1998402304' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:05.126 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:04 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1813337280' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-02-09T22:49:05.144 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2929703202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1998402304' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:05.145 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:04 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1813337280' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-02-09T22:49:05.148 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2929703202' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1998402304' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:05.149 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:04 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1813337280' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-02-09T22:49:05.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.225+0000 7fb03cff9640 1 -- 172.21.15.3:0/1813337280 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb030099fe0 con 0x7fb040101cb0 2024-02-09T22:49:05.228 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' created 2024-02-09T22:49:05.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb0280a20c0 msgr2=0x7fb0280a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:05.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb0280a20c0 0x7fb0280a4580 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fb034005ab0 tx=0x7fb034005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 msgr2=0x7fb040101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 0x7fb040101620 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb03000d980 tx=0x7fb03000de40 comp rx=0 tx=0).stop 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 shutdown_connections 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb0280a20c0 0x7fb0280a4580 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb040106f10 0x7fb0401001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0401045e0 0x7fb0400ffc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 --2- 172.21.15.3:0/1813337280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb040101cb0 0x7fb040101620 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 >> 172.21.15.3:0/1813337280 conn(0x7fb0400fb660 msgr2=0x7fb040102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 shutdown_connections 2024-02-09T22:49:05.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.228+0000 7fb045adf640 1 -- 172.21.15.3:0/1813337280 wait complete. 2024-02-09T22:49:05.247 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-02-09T22:49:05.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 -- 172.21.15.3:0/1582256576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e300ff8e0 msgr2=0x7f3e300ffd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:05.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 --2- 172.21.15.3:0/1582256576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e300ff8e0 0x7f3e300ffd00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009900 tx=0x7f3e2002bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:05.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 -- 172.21.15.3:0/1582256576 shutdown_connections 2024-02-09T22:49:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 --2- 172.21.15.3:0/1582256576 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e30101490 0x7f3e30107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 --2- 172.21.15.3:0/1582256576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100ad0 0x7f3e30100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 --2- 172.21.15.3:0/1582256576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e300ff8e0 0x7f3e300ffd00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 -- 172.21.15.3:0/1582256576 >> 172.21.15.3:0/1582256576 conn(0x7f3e300fae50 msgr2=0x7f3e300fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 -- 172.21.15.3:0/1582256576 shutdown_connections 2024-02-09T22:49:05.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.355+0000 7f3e3498c640 1 -- 172.21.15.3:0/1582256576 wait complete. 2024-02-09T22:49:05.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 Processor -- start 2024-02-09T22:49:05.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 -- start start 2024-02-09T22:49:05.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e300ff8e0 0x7f3e30196260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:05.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100ad0 0x7f3e301967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:05.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 0x7f3e3019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:05.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3019dcb0 con 0x7f3e30101490 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3019de20 con 0x7f3e300ff8e0 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e3498c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3019dfe0 con 0x7f3e30100ad0 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 0x7f3e3019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 0x7f3e3019d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51918/0 (socket says 172.21.15.3:51918) 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 -- 172.21.15.3:0/3418861687 learned_addr learned my addr 172.21.15.3:0/3418861687 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2e59c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e300ff8e0 0x7f3e30196260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2dd9b640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100ad0 0x7f3e301967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 -- 172.21.15.3:0/3418861687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100ad0 msgr2=0x7f3e301967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100ad0 0x7f3e301967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 -- 172.21.15.3:0/3418861687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e300ff8e0 msgr2=0x7f3e30196260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e300ff8e0 0x7f3e30196260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.356+0000 7f3e2ed9d640 1 -- 172.21.15.3:0/3418861687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20009560 con 0x7f3e30101490 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.357+0000 7f3e2ed9d640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 0x7f3e3019d770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800eaa0 tx=0x7f3e2800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:05.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.357+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2800cdb0 con 0x7f3e30101490 2024-02-09T22:49:05.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.357+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e28005020 con 0x7f3e30101490 2024-02-09T22:49:05.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.357+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3019e2c0 con 0x7f3e30101490 2024-02-09T22:49:05.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.357+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e280116b0 con 0x7f3e30101490 2024-02-09T22:49:05.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.357+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3019e780 con 0x7f3e30101490 2024-02-09T22:49:05.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.358+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e30106c00 con 0x7f3e30101490 2024-02-09T22:49:05.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.359+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3e28005520 con 0x7f3e30101490 2024-02-09T22:49:05.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.360+0000 7f3e1f7fe640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e180a20c0 0x7f3e180a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:05.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.360+0000 7f3e2e59c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e180a20c0 0x7f3e180a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:05.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.360+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f3e280cdfa0 con 0x7f3e30101490 2024-02-09T22:49:05.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.360+0000 7f3e2e59c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e180a20c0 0x7f3e180a4580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e20005ab0 tx=0x7f3e20005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:05.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.364+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e280972b0 con 0x7f3e30101490 2024-02-09T22:49:05.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:05.548+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f3e3010bb50 con 0x7f3e30101490 2024-02-09T22:49:06.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.237+0000 7f3e1f7fe640 1 -- 172.21.15.3:0/3418861687 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7f3e2801f0d0 con 0x7f3e30101490 2024-02-09T22:49:06.240 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' created 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e180a20c0 msgr2=0x7f3e180a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e180a20c0 0x7f3e180a4580 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3e20005ab0 tx=0x7f3e20005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 msgr2=0x7f3e3019d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 0x7f3e3019d770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3e2800eaa0 tx=0x7f3e2800ef60 comp rx=0 tx=0).stop 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 shutdown_connections 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e180a20c0 0x7f3e180a4580 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e30101490 0x7f3e3019d770 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30100ad0 0x7f3e301967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 --2- 172.21.15.3:0/3418861687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e300ff8e0 0x7f3e30196260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 >> 172.21.15.3:0/3418861687 conn(0x7f3e300fae50 msgr2=0x7f3e30103dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:06.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 shutdown_connections 2024-02-09T22:49:06.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.239+0000 7f3e3498c640 1 -- 172.21.15.3:0/3418861687 wait complete. 2024-02-09T22:49:06.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 -- 172.21.15.3:0/1133200083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 msgr2=0x7feb34100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 --2- 172.21.15.3:0/1133200083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34100570 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feb2c009900 tx=0x7feb2c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 -- 172.21.15.3:0/1133200083 shutdown_connections 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 --2- 172.21.15.3:0/1133200083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 0x7feb341017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 --2- 172.21.15.3:0/1133200083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb34101d00 0x7feb34108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 --2- 172.21.15.3:0/1133200083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34100570 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 -- 172.21.15.3:0/1133200083 >> 172.21.15.3:0/1133200083 conn(0x7feb340fb6c0 msgr2=0x7feb340fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 -- 172.21.15.3:0/1133200083 shutdown_connections 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.369+0000 7feb391c9640 1 -- 172.21.15.3:0/1133200083 wait complete. 2024-02-09T22:49:06.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 Processor -- start 2024-02-09T22:49:06.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 -- start start 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 0x7feb34197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb34101d00 0x7feb3419e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3419ee30 con 0x7feb34100150 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3419efa0 con 0x7feb34101340 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb391c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3419f160 con 0x7feb34101d00 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb32d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.370+0000 7feb32d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51936/0 (socket says 172.21.15.3:51936) 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 -- 172.21.15.3:0/581109456 learned_addr learned my addr 172.21.15.3:0/581109456 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb3359e640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb34101d00 0x7feb3419e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb3259c640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 0x7feb34197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:06.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 -- 172.21.15.3:0/581109456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb34101d00 msgr2=0x7feb3419e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb34101d00 0x7feb3419e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 -- 172.21.15.3:0/581109456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 msgr2=0x7feb34197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 0x7feb34197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 -- 172.21.15.3:0/581109456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2c009560 con 0x7feb34100150 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb3259c640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 0x7feb34197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb32d9d640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34197340 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feb2c005ab0 tx=0x7feb2c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c004aa0 con 0x7feb34100150 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb2c0052d0 con 0x7feb34100150 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3419f3e0 con 0x7feb34100150 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb2c003e80 con 0x7feb34100150 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.371+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3419f8f0 con 0x7feb34100150 2024-02-09T22:49:06.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.372+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb341074e0 con 0x7feb34100150 2024-02-09T22:49:06.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.374+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7feb2c005000 con 0x7feb34100150 2024-02-09T22:49:06.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.374+0000 7feb2bfff640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feb200a20c0 0x7feb200a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:06.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.375+0000 7feb3259c640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feb200a20c0 0x7feb200a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:06.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.375+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7feb2c0f0660 con 0x7feb34100150 2024-02-09T22:49:06.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.375+0000 7feb3259c640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feb200a20c0 0x7feb200a4580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb34198850 tx=0x7feb1c005d30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:06.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.378+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7feb2c0b9740 con 0x7feb34100150 2024-02-09T22:49:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:06.608+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feb3410cc60 con 0x7feb34100150 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1813337280' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: osdmap e83: 12 total, 12 up, 12 in 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3418861687' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:49:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:49:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1813337280' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-02-09T22:49:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: osdmap e83: 12 total, 12 up, 12 in 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3418861687' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:49:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1813337280' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-02-09T22:49:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: osdmap e83: 12 total, 12 up, 12 in 2024-02-09T22:49:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3418861687' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:49:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:07.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.536+0000 7feb2bfff640 1 -- 172.21.15.3:0/581109456 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7feb2c0bce00 con 0x7feb34100150 2024-02-09T22:49:07.539 INFO:teuthology.orchestra.run.smithi003.stderr:new fs with metadata pool 2 and data pool 3 2024-02-09T22:49:07.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feb200a20c0 msgr2=0x7feb200a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feb200a20c0 0x7feb200a4580 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feb34198850 tx=0x7feb1c005d30 comp rx=0 tx=0).stop 2024-02-09T22:49:07.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 msgr2=0x7feb34197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34197340 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7feb2c005ab0 tx=0x7feb2c004270 comp rx=0 tx=0).stop 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 shutdown_connections 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feb200a20c0 0x7feb200a4580 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feb34101340 0x7feb34197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feb34101d00 0x7feb3419e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 --2- 172.21.15.3:0/581109456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feb34100150 0x7feb34197340 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 >> 172.21.15.3:0/581109456 conn(0x7feb340fb6c0 msgr2=0x7feb341046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 shutdown_connections 2024-02-09T22:49:07.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.539+0000 7feb391c9640 1 -- 172.21.15.3:0/581109456 wait complete. 2024-02-09T22:49:07.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3418861687' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-02-09T22:49:07.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:07 smithi003 ceph-mon[32678]: osdmap e84: 12 total, 12 up, 12 in 2024-02-09T22:49:07.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:07 smithi003 ceph-mon[32678]: pgmap v284: 97 pgs: 9 creating+peering, 87 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:07.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:07.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/581109456' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-02-09T22:49:07.558 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:07 smithi003 ceph-mon[32678]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-02-09T22:49:07.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:49:07.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 -- 172.21.15.3:0/385253090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc0ffba0 msgr2=0x7fa4fc0fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 --2- 172.21.15.3:0/385253090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc0fffc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4009900 tx=0x7fa4f402bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 -- 172.21.15.3:0/385253090 shutdown_connections 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 --2- 172.21.15.3:0/385253090 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4fc100500 0x7fa4fc100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 --2- 172.21.15.3:0/385253090 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc100ea0 0x7fa4fc101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 --2- 172.21.15.3:0/385253090 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc0fffc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 -- 172.21.15.3:0/385253090 >> 172.21.15.3:0/385253090 conn(0x7fa4fc0fb740 msgr2=0x7fa4fc0fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 -- 172.21.15.3:0/385253090 shutdown_connections 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.665+0000 7fa500857640 1 -- 172.21.15.3:0/385253090 wait complete. 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 Processor -- start 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 -- start start 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc1994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 0x7fa4fc199a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4fc100ea0 0x7fa4fc1a0a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc1a0fb0 con 0x7fa4fc100500 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc1a1120 con 0x7fa4fc100ea0 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa500857640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc1a12e0 con 0x7fa4fc0ffba0 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 0x7fa4fc199a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 0x7fa4fc199a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50148/0 (socket says 172.21.15.3:50148) 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 -- 172.21.15.3:0/1947434556 learned_addr learned my addr 172.21.15.3:0/1947434556 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4fad9d640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4fc100ea0 0x7fa4fc1a0a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4fa59c640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc1994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 -- 172.21.15.3:0/1947434556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc0ffba0 msgr2=0x7fa4fc1994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc1994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 -- 172.21.15.3:0/1947434556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4fc100ea0 msgr2=0x7fa4fc1a0a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4fc100ea0 0x7fa4fc1a0a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 -- 172.21.15.3:0/1947434556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f4009560 con 0x7fa4fc100500 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4fa59c640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc1994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:07.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.666+0000 7fa4f9d9b640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 0x7fa4fc199a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d950 tx=0x7fa4e400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:07.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.667+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e4009c30 con 0x7fa4fc100500 2024-02-09T22:49:07.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.667+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc1a15c0 con 0x7fa4fc100500 2024-02-09T22:49:07.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.667+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4e4009d90 con 0x7fa4fc100500 2024-02-09T22:49:07.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.667+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4e400f7e0 con 0x7fa4fc100500 2024-02-09T22:49:07.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.667+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc1a1b20 con 0x7fa4fc100500 2024-02-09T22:49:07.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.668+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc10aa30 con 0x7fa4fc100500 2024-02-09T22:49:07.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.669+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa4e4010ad0 con 0x7fa4fc100500 2024-02-09T22:49:07.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.669+0000 7fa4f37fe640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa4e80a20c0 0x7fa4e80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.669+0000 7fa4fa59c640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa4e80a20c0 0x7fa4e80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.669+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa4e40cd7f0 con 0x7fa4fc100500 2024-02-09T22:49:07.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.670+0000 7fa4fa59c640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa4e80a20c0 0x7fa4e80a4580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4036040 tx=0x7fa4f4000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:07.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.674+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa4e4096910 con 0x7fa4fc100500 2024-02-09T22:49:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3418861687' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-02-09T22:49:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:07 smithi064 ceph-mon[36638]: osdmap e84: 12 total, 12 up, 12 in 2024-02-09T22:49:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:07 smithi064 ceph-mon[36638]: pgmap v284: 97 pgs: 9 creating+peering, 87 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/581109456' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-02-09T22:49:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:07 smithi064 ceph-mon[36638]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-02-09T22:49:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3418861687' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-02-09T22:49:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:07 smithi089 ceph-mon[36665]: osdmap e84: 12 total, 12 up, 12 in 2024-02-09T22:49:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:07 smithi089 ceph-mon[36665]: pgmap v284: 97 pgs: 9 creating+peering, 87 unknown, 1 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:49:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:49:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/581109456' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-02-09T22:49:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:07 smithi089 ceph-mon[36665]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-02-09T22:49:07.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.863+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa4fc02cdb0 con 0x7fa4fc100500 2024-02-09T22:49:07.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.864+0000 7fa4f37fe640 1 -- 172.21.15.3:0/1947434556 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22041 (secure 0 0 0) 0x7fa4e4099fd0 con 0x7fa4fc100500 2024-02-09T22:49:07.867 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:07.867 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":85,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:49:06.609898+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:49:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa4e80a20c0 msgr2=0x7fa4e80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa4e80a20c0 0x7fa4e80a4580 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fa4f4036040 tx=0x7fa4f4000bc0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 msgr2=0x7fa4fc199a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 0x7fa4fc199a00 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa4e400d950 tx=0x7fa4e400de10 comp rx=0 tx=0).stop 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 shutdown_connections 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa4e80a20c0 0x7fa4e80a4580 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4fc100ea0 0x7fa4fc1a0a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4fc100500 0x7fa4fc199a00 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 --2- 172.21.15.3:0/1947434556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4fc0ffba0 0x7fa4fc1994c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 >> 172.21.15.3:0/1947434556 conn(0x7fa4fc0fb740 msgr2=0x7fa4fc0fcea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 shutdown_connections 2024-02-09T22:49:07.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.866+0000 7fa500857640 1 -- 172.21.15.3:0/1947434556 wait complete. 2024-02-09T22:49:07.880 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 -- 172.21.15.3:0/448259272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b4102e70 msgr2=0x7f09b4109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 --2- 172.21.15.3:0/448259272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b4102e70 0x7f09b4109700 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f09b0009ca0 tx=0x7f09b002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 -- 172.21.15.3:0/448259272 shutdown_connections 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 --2- 172.21.15.3:0/448259272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b41024b0 0x7f09b4102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 --2- 172.21.15.3:0/448259272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b4102e70 0x7f09b4109700 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 --2- 172.21.15.3:0/448259272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b41012c0 0x7f09b41016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.980+0000 7f09bb999640 1 -- 172.21.15.3:0/448259272 >> 172.21.15.3:0/448259272 conn(0x7f09b40fc850 msgr2=0x7f09b40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 -- 172.21.15.3:0/448259272 shutdown_connections 2024-02-09T22:49:07.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 -- 172.21.15.3:0/448259272 wait complete. 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 Processor -- start 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 -- start start 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 0x7f09b41971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b41024b0 0x7f09b41976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b4102e70 0x7f09b419e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b419eca0 con 0x7f09b41012c0 2024-02-09T22:49:07.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b419ee10 con 0x7f09b4102e70 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09bb999640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b419efd0 con 0x7f09b41024b0 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09b9735640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 0x7f09b41971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09b9735640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 0x7f09b41971b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50170/0 (socket says 172.21.15.3:50170) 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09b9735640 1 -- 172.21.15.3:0/4060603208 learned_addr learned my addr 172.21.15.3:0/4060603208 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09b8f34640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b41024b0 0x7f09b41976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.981+0000 7f09b9f36640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b4102e70 0x7f09b419e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9735640 1 -- 172.21.15.3:0/4060603208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b41024b0 msgr2=0x7f09b41976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9735640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b41024b0 0x7f09b41976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9735640 1 -- 172.21.15.3:0/4060603208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b4102e70 msgr2=0x7f09b419e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9735640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b4102e70 0x7f09b419e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9735640 1 -- 172.21.15.3:0/4060603208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0009900 con 0x7f09b41012c0 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9f36640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b4102e70 0x7f09b419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09b9735640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 0x7f09b41971b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c970 tx=0x7f09ac00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:07.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac004220 con 0x7f09b41012c0 2024-02-09T22:49:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b419f2b0 con 0x7f09b41012c0 2024-02-09T22:49:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09ac007e90 con 0x7f09b41012c0 2024-02-09T22:49:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ac004800 con 0x7f09b41012c0 2024-02-09T22:49:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.982+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b419f7c0 con 0x7f09b41012c0 2024-02-09T22:49:07.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.983+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b4108510 con 0x7f09b41012c0 2024-02-09T22:49:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.984+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f09ac0079b0 con 0x7f09b41012c0 2024-02-09T22:49:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.984+0000 7f09aa7fc640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09a00a20c0 0x7f09a00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.985+0000 7f09b8f34640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09a00a20c0 0x7f09a00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.985+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f09ac0cdc70 con 0x7f09b41012c0 2024-02-09T22:49:07.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.985+0000 7f09b8f34640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09a00a20c0 0x7f09a00a4580 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f09b41986c0 tx=0x7f09a400a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:07.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:07.988+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f09ac096d90 con 0x7f09b41012c0 2024-02-09T22:49:08.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.171+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09b410cc60 con 0x7f09b41012c0 2024-02-09T22:49:08.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.172+0000 7f09aa7fc640 1 -- 172.21.15.3:0/4060603208 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22041 (secure 0 0 0) 0x7f09ac01d020 con 0x7f09b41012c0 2024-02-09T22:49:08.175 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:08.176 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":85,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:49:06.609898+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:49:08.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09a00a20c0 msgr2=0x7f09a00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09a00a20c0 0x7f09a00a4580 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f09b41986c0 tx=0x7f09a400a2d0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 msgr2=0x7f09b41971b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 0x7f09b41971b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f09ac00c970 tx=0x7f09ac00cd70 comp rx=0 tx=0).stop 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 shutdown_connections 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09a00a20c0 0x7f09a00a4580 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09b4102e70 0x7f09b419e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09b41024b0 0x7f09b41976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 --2- 172.21.15.3:0/4060603208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09b41012c0 0x7f09b41971b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 >> 172.21.15.3:0/4060603208 conn(0x7f09b40fc850 msgr2=0x7f09b41056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 shutdown_connections 2024-02-09T22:49:08.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.174+0000 7f09bb999640 1 -- 172.21.15.3:0/4060603208 wait complete. 2024-02-09T22:49:08.189 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 -- 172.21.15.3:0/2157802647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284106f90 msgr2=0x7f8284109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 --2- 172.21.15.3:0/2157802647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284106f90 0x7f8284109380 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f827c009ca0 tx=0x7f827c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 -- 172.21.15.3:0/2157802647 shutdown_connections 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 --2- 172.21.15.3:0/2157802647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284106f90 0x7f8284109380 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 --2- 172.21.15.3:0/2157802647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8284104660 0x7f8284106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 --2- 172.21.15.3:0/2157802647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284101d30 0x7f8284104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 -- 172.21.15.3:0/2157802647 >> 172.21.15.3:0/2157802647 conn(0x7f82840fb6c0 msgr2=0x7f82840fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 -- 172.21.15.3:0/2157802647 shutdown_connections 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 -- 172.21.15.3:0/2157802647 wait complete. 2024-02-09T22:49:08.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.293+0000 7f8289d8a640 1 Processor -- start 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 -- start start 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 0x7f82841016a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8284104660 0x7f82840ffce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284106f90 0x7f8284100220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284100980 con 0x7f8284101d30 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8284100af0 con 0x7f8284106f90 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82841a1330 con 0x7f8284104660 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 0x7f82841016a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 0x7f82841016a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50186/0 (socket says 172.21.15.3:50186) 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 -- 172.21.15.3:0/1688133926 learned_addr learned my addr 172.21.15.3:0/1688133926 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:08.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8283fff640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284106f90 0x7f8284100220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8282ffd640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8284104660 0x7f82840ffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 -- 172.21.15.3:0/1688133926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8284104660 msgr2=0x7f82840ffce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8284104660 0x7f82840ffce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 -- 172.21.15.3:0/1688133926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284106f90 msgr2=0x7f8284100220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284106f90 0x7f8284100220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 -- 172.21.15.3:0/1688133926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827c009900 con 0x7f8284101d30 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8283fff640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284106f90 0x7f8284100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f82837fe640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 0x7f82841016a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f827400d980 tx=0x7f827400de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8274009c30 con 0x7f8284101d30 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f827400b660 con 0x7f8284101d30 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82841a15e0 con 0x7f8284101d30 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f827400f6b0 con 0x7f8284101d30 2024-02-09T22:49:08.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.294+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82841a1af0 con 0x7f8284101d30 2024-02-09T22:49:08.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.295+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82840fd1b0 con 0x7f8284101d30 2024-02-09T22:49:08.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.296+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f8274010460 con 0x7f8284101d30 2024-02-09T22:49:08.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.297+0000 7f8280ff9640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f826c0a20c0 0x7f826c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.297+0000 7f8282ffd640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f826c0a20c0 0x7f826c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.297+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f82740cd950 con 0x7f8284101d30 2024-02-09T22:49:08.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.298+0000 7f8282ffd640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f826c0a20c0 0x7f826c0a4580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8284101410 tx=0x7f8278005c60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:08.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.302+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82740969d0 con 0x7f8284101d30 2024-02-09T22:49:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.519+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f828402ce50 con 0x7f8284101d30 2024-02-09T22:49:08.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.544+0000 7f8280ff9640 1 -- 172.21.15.3:0/1688133926 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f827409a090 con 0x7f8284101d30 2024-02-09T22:49:08.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f826c0a20c0 msgr2=0x7f826c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f826c0a20c0 0x7f826c0a4580 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f8284101410 tx=0x7f8278005c60 comp rx=0 tx=0).stop 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 msgr2=0x7f82841016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 0x7f82841016a0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f827400d980 tx=0x7f827400de40 comp rx=0 tx=0).stop 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 shutdown_connections 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f826c0a20c0 0x7f826c0a4580 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8284106f90 0x7f8284100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8284104660 0x7f82840ffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 --2- 172.21.15.3:0/1688133926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8284101d30 0x7f82841016a0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 >> 172.21.15.3:0/1688133926 conn(0x7f82840fb6c0 msgr2=0x7f8284102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 shutdown_connections 2024-02-09T22:49:08.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.546+0000 7f8289d8a640 1 -- 172.21.15.3:0/1688133926 wait complete. 2024-02-09T22:49:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: osdmap e85: 12 total, 12 up, 12 in 2024-02-09T22:49:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T22:49:08.559 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/581109456' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-02-09T22:49:08.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-02-09T22:49:08.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-02-09T22:49:08.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1947434556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:08.560 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:08 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4060603208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:08.561 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-02-09T22:49:08.561 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 -- 172.21.15.3:0/3392786225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520100150 msgr2=0x7f8520100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 --2- 172.21.15.3:0/3392786225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520100150 0x7f8520100570 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8510009900 tx=0x7f851002bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 -- 172.21.15.3:0/3392786225 shutdown_connections 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 --2- 172.21.15.3:0/3392786225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520101d00 0x7f8520108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 --2- 172.21.15.3:0/3392786225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8520101340 0x7f85201017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 --2- 172.21.15.3:0/3392786225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520100150 0x7f8520100570 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 -- 172.21.15.3:0/3392786225 >> 172.21.15.3:0/3392786225 conn(0x7f85200fb6c0 msgr2=0x7f85200fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 -- 172.21.15.3:0/3392786225 shutdown_connections 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 -- 172.21.15.3:0/3392786225 wait complete. 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 Processor -- start 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.663+0000 7f85258b5640 1 -- start start 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8520100150 0x7f852006a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520101340 0x7f852019e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 0x7f85201a09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85201a10b0 con 0x7f8520101d00 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85201a1220 con 0x7f8520101340 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85201a13e0 con 0x7f8520100150 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 0x7f85201a09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 0x7f85201a09f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50190/0 (socket says 172.21.15.3:50190) 2024-02-09T22:49:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 -- 172.21.15.3:0/2365498315 learned_addr learned my addr 172.21.15.3:0/2365498315 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851effd640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8520100150 0x7f852006a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851e7fc640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520101340 0x7f852019e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 -- 172.21.15.3:0/2365498315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8520100150 msgr2=0x7f852006a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8520100150 0x7f852006a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 -- 172.21.15.3:0/2365498315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520101340 msgr2=0x7f852019e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520101340 0x7f852019e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 -- 172.21.15.3:0/2365498315 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8510009560 con 0x7f8520101d00 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f851f7fe640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 0x7f85201a09f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f851800cad0 tx=0x7f851800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851800ae20 con 0x7f8520101d00 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8518005020 con 0x7f8520101d00 2024-02-09T22:49:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85201a1660 con 0x7f8520101d00 2024-02-09T22:49:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f851800f6b0 con 0x7f8520101d00 2024-02-09T22:49:08.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.664+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85201a1b20 con 0x7f8520101d00 2024-02-09T22:49:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.666+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85201074f0 con 0x7f8520101d00 2024-02-09T22:49:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.667+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f8518005520 con 0x7f8520101d00 2024-02-09T22:49:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.667+0000 7f85248b3640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f85080a20f0 0x7f85080a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:08.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.667+0000 7f851effd640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f85080a20f0 0x7f85080a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.667+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f85180cdb40 con 0x7f8520101d00 2024-02-09T22:49:08.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.668+0000 7f851effd640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f85080a20f0 0x7f85080a45b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8510009230 tx=0x7f8510005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:08.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.672+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8518096c60 con 0x7f8520101d00 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: osdmap e85: 12 total, 12 up, 12 in 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/581109456' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1947434556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:08.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:08 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4060603208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: osdmap e85: 12 total, 12 up, 12 in 2024-02-09T22:49:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-02-09T22:49:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/581109456' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-02-09T22:49:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-02-09T22:49:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-02-09T22:49:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1947434556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:08 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4060603208' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:08.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:08.895+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f852010bd60 con 0x7f8520101d00 2024-02-09T22:49:09.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.082+0000 7f85248b3640 1 -- 172.21.15.3:0/2365498315 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v16) v1 ==== 105+0+0 (secure 0 0 0) 0x7f851809a320 con 0x7f8520101d00 2024-02-09T22:49:09.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.084+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f85080a20f0 msgr2=0x7f85080a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.084+0000 7f85258b5640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f85080a20f0 0x7f85080a45b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8510009230 tx=0x7f8510005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.084+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 msgr2=0x7f85201a09f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.084+0000 7f85258b5640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 0x7f85201a09f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f851800cad0 tx=0x7f851800cf90 comp rx=0 tx=0).stop 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 shutdown_connections 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f85080a20f0 0x7f85080a45b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8520101340 0x7f852019e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8520101d00 0x7f85201a09f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 --2- 172.21.15.3:0/2365498315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8520100150 0x7f852006a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 >> 172.21.15.3:0/2365498315 conn(0x7f85200fb6c0 msgr2=0x7f85201046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 shutdown_connections 2024-02-09T22:49:09.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.085+0000 7f85258b5640 1 -- 172.21.15.3:0/2365498315 wait complete. 2024-02-09T22:49:09.103 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-02-09T22:49:09.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 -- 172.21.15.3:0/3875369806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8101340 msgr2=0x7fa8b81017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 --2- 172.21.15.3:0/3875369806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8101340 0x7fa8b81017c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fa8a0009900 tx=0x7fa8a002bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 -- 172.21.15.3:0/3875369806 shutdown_connections 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 --2- 172.21.15.3:0/3875369806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8b8101d00 0x7fa8b8108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 --2- 172.21.15.3:0/3875369806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8101340 0x7fa8b81017c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 --2- 172.21.15.3:0/3875369806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8b8100150 0x7fa8b8100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 -- 172.21.15.3:0/3875369806 >> 172.21.15.3:0/3875369806 conn(0x7fa8b80fb6c0 msgr2=0x7fa8b80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 -- 172.21.15.3:0/3875369806 shutdown_connections 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.206+0000 7fa8bd240640 1 -- 172.21.15.3:0/3875369806 wait complete. 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 Processor -- start 2024-02-09T22:49:09.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 -- start start 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 0x7fa8b80690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8b8101340 0x7fa8b819c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8b8101d00 0x7fa8b819e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b819ee20 con 0x7fa8b8100150 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b819ef90 con 0x7fa8b8101340 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8bd240640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8b819f150 con 0x7fa8b8101d00 2024-02-09T22:49:09.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 0x7fa8b80690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 0x7fa8b80690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50208/0 (socket says 172.21.15.3:50208) 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.207+0000 7fa8b6d9d640 1 -- 172.21.15.3:0/4005297799 learned_addr learned my addr 172.21.15.3:0/4005297799 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8b6d9d640 1 -- 172.21.15.3:0/4005297799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8b8101d00 msgr2=0x7fa8b819e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8b6d9d640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8b8101d00 0x7fa8b819e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8b6d9d640 1 -- 172.21.15.3:0/4005297799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8b8101340 msgr2=0x7fa8b819c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8b6d9d640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8b8101340 0x7fa8b819c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8b6d9d640 1 -- 172.21.15.3:0/4005297799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a0009560 con 0x7fa8b8100150 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8b6d9d640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 0x7fa8b80690d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000b450 tx=0x7fa8b000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0011890 con 0x7fa8b8100150 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b819f430 con 0x7fa8b8100150 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8b00103e0 con 0x7fa8b8100150 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b000f580 con 0x7fa8b8100150 2024-02-09T22:49:09.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.208+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b819fa10 con 0x7fa8b8100150 2024-02-09T22:49:09.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.209+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b8107470 con 0x7fa8b8100150 2024-02-09T22:49:09.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.210+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa8b00108e0 con 0x7fa8b8100150 2024-02-09T22:49:09.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.211+0000 7fa8affff640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8a40a20c0 0x7fa8a40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:09.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.211+0000 7fa8b659c640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8a40a20c0 0x7fa8a40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:09.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.211+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa8b00cd9c0 con 0x7fa8b8100150 2024-02-09T22:49:09.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.211+0000 7fa8b659c640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8a40a20c0 0x7fa8a40a4580 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8a002c6a0 tx=0x7fa8a0000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:09.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.215+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa8b0096ae0 con 0x7fa8b8100150 2024-02-09T22:49:09.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: osdmap e86: 12 total, 12 up, 12 in 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1688133926' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: pgmap v287: 97 pgs: 22 creating+peering, 48 unknown, 27 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1688133926' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] up:active 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2365498315' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2365498315' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-02-09T22:49:09.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:09 smithi003 ceph-mon[32678]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-02-09T22:49:09.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:09.436+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7fa8b810bd90 con 0x7fa8b8100150 2024-02-09T22:49:09.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: osdmap e86: 12 total, 12 up, 12 in 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1688133926' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: pgmap v287: 97 pgs: 22 creating+peering, 48 unknown, 27 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1688133926' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] up:active 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2365498315' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2365498315' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-02-09T22:49:09.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:09 smithi064 ceph-mon[36638]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-02-09T22:49:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: osdmap e86: 12 total, 12 up, 12 in 2024-02-09T22:49:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1688133926' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-02-09T22:49:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: pgmap v287: 97 pgs: 22 creating+peering, 48 unknown, 27 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 1 op/s 2024-02-09T22:49:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1688133926' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] up:active 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2365498315' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: daemon mds.h assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2365498315' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: fsmap cephfs:2 {0=b=up:active,1=h=up:creating} 10 up:standby 2024-02-09T22:49:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:09 smithi089 ceph-mon[36665]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-02-09T22:49:10.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.090+0000 7fa8affff640 1 -- 172.21.15.3:0/4005297799 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v17) v1 ==== 121+0+0 (secure 0 0 0) 0x7fa8b009a1a0 con 0x7fa8b8100150 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8a40a20c0 msgr2=0x7fa8a40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8a40a20c0 0x7fa8a40a4580 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa8a002c6a0 tx=0x7fa8a0000bc0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 msgr2=0x7fa8b80690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 0x7fa8b80690d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa8b000b450 tx=0x7fa8b000b910 comp rx=0 tx=0).stop 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 shutdown_connections 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8a40a20c0 0x7fa8a40a4580 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8b8101340 0x7fa8b819c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8b8101d00 0x7fa8b819e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 --2- 172.21.15.3:0/4005297799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8b8100150 0x7fa8b80690d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 >> 172.21.15.3:0/4005297799 conn(0x7fa8b80fb6c0 msgr2=0x7fa8b8104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 shutdown_connections 2024-02-09T22:49:10.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.093+0000 7fa8bd240640 1 -- 172.21.15.3:0/4005297799 wait complete. 2024-02-09T22:49:10.112 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-02-09T22:49:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 -- 172.21.15.3:0/555919703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 msgr2=0x7f02a0108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 --2- 172.21.15.3:0/555919703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a0108590 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f0298009ca0 tx=0x7f029802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 -- 172.21.15.3:0/555919703 shutdown_connections 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 --2- 172.21.15.3:0/555919703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02a0101340 0x7f02a01017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 --2- 172.21.15.3:0/555919703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a0108590 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 --2- 172.21.15.3:0/555919703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a0100150 0x7f02a0100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.220+0000 7f02a5640640 1 -- 172.21.15.3:0/555919703 >> 172.21.15.3:0/555919703 conn(0x7f02a00fb6c0 msgr2=0x7f02a00fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 -- 172.21.15.3:0/555919703 shutdown_connections 2024-02-09T22:49:10.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 -- 172.21.15.3:0/555919703 wait complete. 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 Processor -- start 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 -- start start 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a0100150 0x7f02a01972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02a0101340 0x7f02a01977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a019e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a019eda0 con 0x7f02a0101d00 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a019ef10 con 0x7f02a0101340 2024-02-09T22:49:10.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f02a5640640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02a019f0d0 con 0x7f02a0100150 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a019e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f029f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a019e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50222/0 (socket says 172.21.15.3:50222) 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f029f7fe640 1 -- 172.21.15.3:0/1168825660 learned_addr learned my addr 172.21.15.3:0/1168825660 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.221+0000 7f029effd640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a0100150 0x7f02a01972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f0297fff640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02a0101340 0x7f02a01977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029f7fe640 1 -- 172.21.15.3:0/1168825660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a0100150 msgr2=0x7f02a01972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029f7fe640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a0100150 0x7f02a01972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029f7fe640 1 -- 172.21.15.3:0/1168825660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02a0101340 msgr2=0x7f02a01977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029f7fe640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02a0101340 0x7f02a01977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029f7fe640 1 -- 172.21.15.3:0/1168825660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0298009900 con 0x7f02a0101d00 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029f7fe640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a019e860 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0298032650 tx=0x7f0298034b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02980329b0 con 0x7f02a0101d00 2024-02-09T22:49:10.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0298003e50 con 0x7f02a0101d00 2024-02-09T22:49:10.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0298040450 con 0x7f02a0101d00 2024-02-09T22:49:10.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a019f350 con 0x7f02a0101d00 2024-02-09T22:49:10.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.222+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a019f780 con 0x7f02a0101d00 2024-02-09T22:49:10.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.223+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a010bb20 con 0x7f02a0101d00 2024-02-09T22:49:10.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.224+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f029800a040 con 0x7f02a0101d00 2024-02-09T22:49:10.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.225+0000 7f029cff9640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f028c0a20c0 0x7f028c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:10.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.225+0000 7f029effd640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f028c0a20c0 0x7f028c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:10.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.225+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f02980f0a60 con 0x7f02a0101d00 2024-02-09T22:49:10.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.225+0000 7f029effd640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f028c0a20c0 0x7f028c0a4580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f02a01011a0 tx=0x7f029000b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:10.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.229+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f02980b9ae0 con 0x7f02a0101d00 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4005297799' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4005297799' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] up:active 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-02-09T22:49:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:10 smithi003 ceph-mon[32678]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-02-09T22:49:10.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:10.453+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f02a019f9a0 con 0x7f02a0101d00 2024-02-09T22:49:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4005297799' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4005297799' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] up:active 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-02-09T22:49:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:10 smithi064 ceph-mon[36638]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-02-09T22:49:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-02-09T22:49:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4005297799' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-02-09T22:49:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4005297799' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-02-09T22:49:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] up:active 2024-02-09T22:49:10.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-02-09T22:49:10.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: fsmap cephfs:2 {0=b=up:active,1=h=up:active} 10 up:standby 2024-02-09T22:49:10.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:creating} 9 up:standby 2024-02-09T22:49:10.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:10 smithi089 ceph-mon[36665]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-02-09T22:49:11.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.099+0000 7f029cff9640 1 -- 172.21.15.3:0/1168825660 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7f02980040d0 con 0x7f02a0101d00 2024-02-09T22:49:11.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f028c0a20c0 msgr2=0x7f028c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:11.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f028c0a20c0 0x7f028c0a4580 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f02a01011a0 tx=0x7f029000b390 comp rx=0 tx=0).stop 2024-02-09T22:49:11.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 msgr2=0x7f02a019e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:11.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a019e860 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f0298032650 tx=0x7f0298034b10 comp rx=0 tx=0).stop 2024-02-09T22:49:11.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 shutdown_connections 2024-02-09T22:49:11.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f028c0a20c0 0x7f028c0a4580 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02a0101340 0x7f02a01977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02a0101d00 0x7f02a019e860 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 --2- 172.21.15.3:0/1168825660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02a0100150 0x7f02a01972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 >> 172.21.15.3:0/1168825660 conn(0x7f02a00fb6c0 msgr2=0x7f02a0069930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 shutdown_connections 2024-02-09T22:49:11.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.102+0000 7f02a5640640 1 -- 172.21.15.3:0/1168825660 wait complete. 2024-02-09T22:49:11.122 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-02-09T22:49:11.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 -- 172.21.15.3:0/3607155026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778101cd0 msgr2=0x7f37781040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 --2- 172.21.15.3:0/3607155026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778101cd0 0x7f37781040c0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3774009900 tx=0x7f377402bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 -- 172.21.15.3:0/3607155026 shutdown_connections 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 --2- 172.21.15.3:0/3607155026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3778104600 0x7f37781069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 --2- 172.21.15.3:0/3607155026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3778106f30 0x7f3778109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 --2- 172.21.15.3:0/3607155026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778101cd0 0x7f37781040c0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 -- 172.21.15.3:0/3607155026 >> 172.21.15.3:0/3607155026 conn(0x7f37780fb6e0 msgr2=0x7f37780fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 -- 172.21.15.3:0/3607155026 shutdown_connections 2024-02-09T22:49:11.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.228+0000 7f377f6f4640 1 -- 172.21.15.3:0/3607155026 wait complete. 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 Processor -- start 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 -- start start 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3778101cd0 0x7f3778197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3778104600 0x7f3778197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 0x7f377819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377819ee00 con 0x7f3778106f30 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377819ef70 con 0x7f3778104600 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377f6f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377819f130 con 0x7f3778101cd0 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 0x7f377819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 0x7f377819e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50244/0 (socket says 172.21.15.3:50244) 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 -- 172.21.15.3:0/2302339214 learned_addr learned my addr 172.21.15.3:0/2302339214 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377cc8f640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3778104600 0x7f3778197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 -- 172.21.15.3:0/2302339214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3778101cd0 msgr2=0x7f3778197310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:11.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377d490640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3778101cd0 0x7f3778197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3778101cd0 0x7f3778197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 -- 172.21.15.3:0/2302339214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3778104600 msgr2=0x7f3778197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3778104600 0x7f3778197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.229+0000 7f377dc91640 1 -- 172.21.15.3:0/2302339214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3774009560 con 0x7f3778106f30 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.230+0000 7f377dc91640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 0x7f377819e8c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3770009fd0 tx=0x7f377000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.230+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f377000bea0 con 0x7f3778106f30 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.230+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3770009940 con 0x7f3778106f30 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.230+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377819f410 con 0x7f3778106f30 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.230+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37700106c0 con 0x7f3778106f30 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.230+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377819f920 con 0x7f3778106f30 2024-02-09T22:49:11.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.231+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37780fd1d0 con 0x7f3778106f30 2024-02-09T22:49:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.232+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3770004240 con 0x7f3778106f30 2024-02-09T22:49:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.232+0000 7f376e7fc640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f37640a2140 0x7f37640a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:11.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.232+0000 7f377d490640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f37640a2140 0x7f37640a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.232+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f37700cd950 con 0x7f3778106f30 2024-02-09T22:49:11.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.233+0000 7f377d490640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f37640a2140 0x7f37640a4600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f377402c6a0 tx=0x7f3774005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:11.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.237+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f37700969d0 con 0x7f3778106f30 2024-02-09T22:49:11.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:11.447+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f377802ce50 con 0x7f37640a2140 2024-02-09T22:49:11.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:49:11.452+0000 7f8969bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:49:11.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:49:11.452+0000 7f8969bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:49:11.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:49:11.452+0000 7f8969bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:49:11.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:49:11.452+0000 7f8969bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:49:11.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:49:11.452+0000 7f8969bb4700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1168825660' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: pgmap v288: 97 pgs: 19 creating+peering, 18 unknown, 60 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 8 op/s 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1168825660' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] up:active 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-02-09T22:49:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:11 smithi003 ceph-mon[32678]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1168825660' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: pgmap v288: 97 pgs: 19 creating+peering, 18 unknown, 60 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 8 op/s 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1168825660' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] up:active 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-02-09T22:49:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:11 smithi064 ceph-mon[36638]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-02-09T22:49:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1168825660' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-02-09T22:49:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: pgmap v288: 97 pgs: 19 creating+peering, 18 unknown, 60 active+clean; 577 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s wr, 8 op/s 2024-02-09T22:49:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1168825660' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-02-09T22:49:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] up:active 2024-02-09T22:49:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: daemon mds.c assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-02-09T22:49:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: fsmap cephfs:3 {0=b=up:active,1=h=up:active,2=i=up:active} 9 up:standby 2024-02-09T22:49:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:creating} 8 up:standby 2024-02-09T22:49:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:11 smithi089 ceph-mon[36665]: daemon mds.c is now active in filesystem cephfs as rank 3 2024-02-09T22:49:12.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.130+0000 7f376e7fc640 1 -- 172.21.15.3:0/2302339214 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f377802ce50 con 0x7f37640a2140 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f37640a2140 msgr2=0x7f37640a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f37640a2140 0x7f37640a4600 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f377402c6a0 tx=0x7f3774005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 msgr2=0x7f377819e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 0x7f377819e8c0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3770009fd0 tx=0x7f377000b260 comp rx=0 tx=0).stop 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 shutdown_connections 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f37640a2140 0x7f37640a4600 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3778104600 0x7f3778197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3778106f30 0x7f377819e8c0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 --2- 172.21.15.3:0/2302339214 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3778101cd0 0x7f3778197310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 >> 172.21.15.3:0/2302339214 conn(0x7f37780fb6e0 msgr2=0x7f3778102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:12.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 shutdown_connections 2024-02-09T22:49:12.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.133+0000 7f377f6f4640 1 -- 172.21.15.3:0/2302339214 wait complete. 2024-02-09T22:49:12.149 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 -- 172.21.15.3:0/426978095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98101cd0 msgr2=0x7fdd981040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 --2- 172.21.15.3:0/426978095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98101cd0 0x7fdd981040c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdd88009900 tx=0x7fdd8802bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 -- 172.21.15.3:0/426978095 shutdown_connections 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 --2- 172.21.15.3:0/426978095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd98104600 0x7fdd981069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 --2- 172.21.15.3:0/426978095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98106f30 0x7fdd98109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 --2- 172.21.15.3:0/426978095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98101cd0 0x7fdd981040c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 -- 172.21.15.3:0/426978095 >> 172.21.15.3:0/426978095 conn(0x7fdd980fb6c0 msgr2=0x7fdd980fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 -- 172.21.15.3:0/426978095 shutdown_connections 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.263+0000 7fdd9df83640 1 -- 172.21.15.3:0/426978095 wait complete. 2024-02-09T22:49:12.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 Processor -- start 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 -- start start 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98101cd0 0x7fdd98197360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd98104600 0x7fdd981978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 0x7fdd9819e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9819ee50 con 0x7fdd98106f30 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9819efc0 con 0x7fdd98104600 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd9df83640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9819f180 con 0x7fdd98101cd0 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 0x7fdd9819e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 0x7fdd9819e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50276/0 (socket says 172.21.15.3:50276) 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 -- 172.21.15.3:0/616523624 learned_addr learned my addr 172.21.15.3:0/616523624 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd977fe640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98101cd0 0x7fdd98197360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd96ffd640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd98104600 0x7fdd981978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 -- 172.21.15.3:0/616523624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98101cd0 msgr2=0x7fdd98197360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98101cd0 0x7fdd98197360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 -- 172.21.15.3:0/616523624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd98104600 msgr2=0x7fdd981978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd98104600 0x7fdd981978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.264+0000 7fdd97fff640 1 -- 172.21.15.3:0/616523624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009560 con 0x7fdd98106f30 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.265+0000 7fdd97fff640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 0x7fdd9819e910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009fd0 tx=0x7fdd9000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.265+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9000bea0 con 0x7fdd98106f30 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.265+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd90009940 con 0x7fdd98106f30 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.265+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd900106c0 con 0x7fdd98106f30 2024-02-09T22:49:12.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.265+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd9819f460 con 0x7fdd98106f30 2024-02-09T22:49:12.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.265+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9819f970 con 0x7fdd98106f30 2024-02-09T22:49:12.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.266+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd980fd1b0 con 0x7fdd98106f30 2024-02-09T22:49:12.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.267+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fdd90004240 con 0x7fdd98106f30 2024-02-09T22:49:12.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.267+0000 7fdd94ff9640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdd800a2140 0x7fdd800a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.268+0000 7fdd977fe640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdd800a2140 0x7fdd800a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.268+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdd900cd750 con 0x7fdd98106f30 2024-02-09T22:49:12.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.268+0000 7fdd977fe640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdd800a2140 0x7fdd800a4600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd88000c00 tx=0x7fdd8802c7e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:12.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.271+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdd900969a0 con 0x7fdd98106f30 2024-02-09T22:49:12.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.481+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd9802cec0 con 0x7fdd800a2140 2024-02-09T22:49:12.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.502+0000 7fdd94ff9640 1 -- 172.21.15.3:0/616523624 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdd9802cec0 con 0x7fdd800a2140 2024-02-09T22:49:12.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdd800a2140 msgr2=0x7fdd800a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdd800a2140 0x7fdd800a4600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdd88000c00 tx=0x7fdd8802c7e0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 msgr2=0x7fdd9819e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 0x7fdd9819e910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdd90009fd0 tx=0x7fdd9000b260 comp rx=0 tx=0).stop 2024-02-09T22:49:12.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 shutdown_connections 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdd800a2140 0x7fdd800a4600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd98104600 0x7fdd981978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd98106f30 0x7fdd9819e910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 --2- 172.21.15.3:0/616523624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd98101cd0 0x7fdd98197360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 >> 172.21.15.3:0/616523624 conn(0x7fdd980fb6c0 msgr2=0x7fdd98102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 shutdown_connections 2024-02-09T22:49:12.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.504+0000 7fdd9df83640 1 -- 172.21.15.3:0/616523624 wait complete. 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] up:active 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: Cluster is now healthy 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-02-09T22:49:12.517 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-02-09T22:49:12.518 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:12 smithi003 ceph-mon[32678]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-02-09T22:49:12.519 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] up:active 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: Cluster is now healthy 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-02-09T22:49:12.612 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-mon[36638]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 -- 172.21.15.3:0/2250087796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0101d60 msgr2=0x7f47f0104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 --2- 172.21.15.3:0/2250087796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0101d60 0x7f47f0104150 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f47e4009900 tx=0x7f47e402bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 -- 172.21.15.3:0/2250087796 shutdown_connections 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 --2- 172.21.15.3:0/2250087796 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47f0104690 0x7f47f0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 --2- 172.21.15.3:0/2250087796 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f0106fc0 0x7f47f01093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 --2- 172.21.15.3:0/2250087796 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0101d60 0x7f47f0104150 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 -- 172.21.15.3:0/2250087796 >> 172.21.15.3:0/2250087796 conn(0x7f47f00fb690 msgr2=0x7f47f00fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 -- 172.21.15.3:0/2250087796 shutdown_connections 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.623+0000 7f47f71b6640 1 -- 172.21.15.3:0/2250087796 wait complete. 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 Processor -- start 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 -- start start 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f0101d60 0x7f47f01016d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 0x7f47f00ffd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47f0106fc0 0x7f47f0100250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01009b0 con 0x7f47f0104690 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f0100b20 con 0x7f47f0106fc0 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f01a1330 con 0x7f47f0101d60 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 0x7f47f00ffd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 0x7f47f00ffd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50300/0 (socket says 172.21.15.3:50300) 2024-02-09T22:49:12.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 -- 172.21.15.3:0/1614979977 learned_addr learned my addr 172.21.15.3:0/1614979977 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f5753640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47f0106fc0 0x7f47f0100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f4f52640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f0101d60 0x7f47f01016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 -- 172.21.15.3:0/1614979977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f0101d60 msgr2=0x7f47f01016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f0101d60 0x7f47f01016d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 -- 172.21.15.3:0/1614979977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47f0106fc0 msgr2=0x7f47f0100250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47f0106fc0 0x7f47f0100250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 -- 172.21.15.3:0/1614979977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e4009560 con 0x7f47f0104690 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47effff640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 0x7f47f00ffd10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00d7a0 tx=0x7f47dc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc009c30 con 0x7f47f0104690 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47dc00deb0 con 0x7f47f0104690 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f01a15e0 con 0x7f47f0104690 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.624+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc00f740 con 0x7f47f0104690 2024-02-09T22:49:12.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.625+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f01a1af0 con 0x7f47f0104690 2024-02-09T22:49:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.626+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f00fd180 con 0x7f47f0104690 2024-02-09T22:49:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.627+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f47dc0109d0 con 0x7f47f0104690 2024-02-09T22:49:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.627+0000 7f47edffb640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f47e00a20c0 0x7f47e00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.627+0000 7f47f4f52640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f47e00a20c0 0x7f47e00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.627+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f47dc0cd950 con 0x7f47f0104690 2024-02-09T22:49:12.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.628+0000 7f47f4f52640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f47e00a20c0 0x7f47e00a4580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f47e402c6a0 tx=0x7f47e4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:12.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.631+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f47dc096a70 con 0x7f47f0104690 2024-02-09T22:49:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: from='client.14742 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] up:active 2024-02-09T22:49:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: daemon mds.f assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-02-09T22:49:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-02-09T22:49:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: Cluster is now healthy 2024-02-09T22:49:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: fsmap cephfs:4 {0=b=up:active,1=h=up:active,2=i=up:active,3=c=up:active} 8 up:standby 2024-02-09T22:49:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-02-09T22:49:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:12 smithi089 ceph-mon[36665]: daemon mds.f is now active in filesystem cephfs as rank 4 2024-02-09T22:49:12.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.843+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f47f002cec0 con 0x7f47e00a20c0 2024-02-09T22:49:12.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.846+0000 7f47edffb640 1 -- 172.21.15.3:0/1614979977 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f47f002cec0 con 0x7f47e00a20c0 2024-02-09T22:49:12.849 INFO:teuthology.orchestra.run.smithi003.stdout:/volumes/qa/sv_0/65be51b6-e2cc-4b78-bd8f-6e2400b2264b 2024-02-09T22:49:12.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f47e00a20c0 msgr2=0x7f47e00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f47e00a20c0 0x7f47e00a4580 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f47e402c6a0 tx=0x7f47e4005fd0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 msgr2=0x7f47f00ffd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 0x7f47f00ffd10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00d7a0 tx=0x7f47dc00dc60 comp rx=0 tx=0).stop 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 shutdown_connections 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f47e00a20c0 0x7f47e00a4580 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f47f0106fc0 0x7f47f0100250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.855 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f47f0104690 0x7f47f00ffd10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 --2- 172.21.15.3:0/1614979977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f47f0101d60 0x7f47f01016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 >> 172.21.15.3:0/1614979977 conn(0x7f47f00fb690 msgr2=0x7f47f0102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 shutdown_connections 2024-02-09T22:49:12.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.849+0000 7f47f71b6640 1 -- 172.21.15.3:0/1614979977 wait complete. 2024-02-09T22:49:12.863 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-02-09T22:49:12.935 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:49:12.607+0000 7efe609de700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-02-09T22:49:12.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:49:12.607+0000 7efe609de700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-02-09T22:49:12.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:49:12.607+0000 7efe609de700 -1 mds.pinger is_rank_lagging: rank=2 was never sent ping request. 2024-02-09T22:49:12.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:49:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:49:12.607+0000 7efe609de700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-02-09T22:49:12.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 -- 172.21.15.3:0/1440825974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08541017d0 msgr2=0x7f0854101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 --2- 172.21.15.3:0/1440825974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08541017d0 0x7f0854101bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f0844009900 tx=0x7f084402bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 -- 172.21.15.3:0/1440825974 shutdown_connections 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 --2- 172.21.15.3:0/1440825974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 0x7f08540ff740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 --2- 172.21.15.3:0/1440825974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08540ffc80 0x7f0854100130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 --2- 172.21.15.3:0/1440825974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08541017d0 0x7f0854101bf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 -- 172.21.15.3:0/1440825974 >> 172.21.15.3:0/1440825974 conn(0x7f08540fae40 msgr2=0x7f08540fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 -- 172.21.15.3:0/1440825974 shutdown_connections 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.967+0000 7f08599d7640 1 -- 172.21.15.3:0/1440825974 wait complete. 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 Processor -- start 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 -- start start 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 0x7f0854199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 0x7f0854199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08541017d0 0x7f08541a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08541a0f40 con 0x7f08540ffc80 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08541a10b0 con 0x7f08540ff2c0 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08599d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08541a1270 con 0x7f08541017d0 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 0x7f0854199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 0x7f0854199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50310/0 (socket says 172.21.15.3:50310) 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 -- 172.21.15.3:0/3769240421 learned_addr learned my addr 172.21.15.3:0/3769240421 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f08537fe640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08541017d0 0x7f08541a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f0852ffd640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 0x7f0854199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 -- 172.21.15.3:0/3769240421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08541017d0 msgr2=0x7f08541a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08541017d0 0x7f08541a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 -- 172.21.15.3:0/3769240421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 msgr2=0x7f0854199450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 0x7f0854199450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:12.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 -- 172.21.15.3:0/3769240421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0844009560 con 0x7f08540ffc80 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f0852ffd640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 0x7f0854199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.968+0000 7f084bfff640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 0x7f0854199990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f083c00c970 tx=0x7f083c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.969+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c007c10 con 0x7f08540ffc80 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.969+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08541a1550 con 0x7f08540ffc80 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.969+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f083c007d70 con 0x7f08540ffc80 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.969+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f083c00f8a0 con 0x7f08540ffc80 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.969+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08541a1a60 con 0x7f08540ffc80 2024-02-09T22:49:12.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.970+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f085410b230 con 0x7f08540ffc80 2024-02-09T22:49:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.971+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 16) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f083c018af0 con 0x7f08540ffc80 2024-02-09T22:49:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.971+0000 7f08589d5640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08400a20c0 0x7f08400a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.971+0000 7f0852ffd640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08400a20c0 0x7f08400a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:12.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.972+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f083c013070 con 0x7f08540ffc80 2024-02-09T22:49:12.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.972+0000 7f0852ffd640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08400a20c0 0x7f08400a4580 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f0844005be0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:12.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:12.976+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f083c097070 con 0x7f08540ffc80 2024-02-09T22:49:13.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.188+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f085402cec0 con 0x7f08400a20c0 2024-02-09T22:49:13.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.210+0000 7f08589d5640 1 -- 172.21.15.3:0/3769240421 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f085402cec0 con 0x7f08400a20c0 2024-02-09T22:49:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08400a20c0 msgr2=0x7f08400a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08400a20c0 0x7f08400a4580 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f0844000c00 tx=0x7f0844005be0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 msgr2=0x7f0854199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 0x7f0854199990 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f083c00c970 tx=0x7f083c00ce30 comp rx=0 tx=0).stop 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 shutdown_connections 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08400a20c0 0x7f08400a4580 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f08541017d0 0x7f08541a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f08540ffc80 0x7f0854199990 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 --2- 172.21.15.3:0/3769240421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f08540ff2c0 0x7f0854199450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 >> 172.21.15.3:0/3769240421 conn(0x7f08540fae40 msgr2=0x7f08540fc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 shutdown_connections 2024-02-09T22:49:13.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.212+0000 7f08599d7640 1 -- 172.21.15.3:0/3769240421 wait complete. 2024-02-09T22:49:13.226 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-02-09T22:49:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.334+0000 7f608398a640 1 -- 172.21.15.3:0/1202393225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c0ff520 msgr2=0x7f607c0ff940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.334+0000 7f608398a640 1 --2- 172.21.15.3:0/1202393225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c0ff520 0x7f607c0ff940 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6078009900 tx=0x7f607802bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- 172.21.15.3:0/1202393225 shutdown_connections 2024-02-09T22:49:13.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 --2- 172.21.15.3:0/1202393225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f607c102f50 0x7f607c1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 --2- 172.21.15.3:0/1202393225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c102590 0x7f607c102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 --2- 172.21.15.3:0/1202393225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c0ff520 0x7f607c0ff940 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- 172.21.15.3:0/1202393225 >> 172.21.15.3:0/1202393225 conn(0x7f607c0fae10 msgr2=0x7f607c0fd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- 172.21.15.3:0/1202393225 shutdown_connections 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- 172.21.15.3:0/1202393225 wait complete. 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 Processor -- start 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- start start 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c0ff520 0x7f607c197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f607c102590 0x7f607c197880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 0x7f607c19e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c19ee30 con 0x7f607c102f50 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c19efa0 con 0x7f607c102590 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f608398a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f607c19f160 con 0x7f607c0ff520 2024-02-09T22:49:13.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.335+0000 7f6081f27640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 0x7f607c19e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 0x7f607c19e8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50326/0 (socket says 172.21.15.3:50326) 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 -- 172.21.15.3:0/3783820349 learned_addr learned my addr 172.21.15.3:0/3783820349 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6080f25640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f607c102590 0x7f607c197880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081726640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c0ff520 0x7f607c197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 -- 172.21.15.3:0/3783820349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c0ff520 msgr2=0x7f607c197340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c0ff520 0x7f607c197340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 -- 172.21.15.3:0/3783820349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f607c102590 msgr2=0x7f607c197880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f607c102590 0x7f607c197880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 -- 172.21.15.3:0/3783820349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6078009560 con 0x7f607c102f50 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081726640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c0ff520 0x7f607c197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f6081f27640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 0x7f607c19e8f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f607400cad0 tx=0x7f607400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607400ae20 con 0x7f607c102f50 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6074005020 con 0x7f607c102f50 2024-02-09T22:49:13.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f607c19f440 con 0x7f607c102f50 2024-02-09T22:49:13.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f607400f6b0 con 0x7f607c102f50 2024-02-09T22:49:13.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.336+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f607c19f980 con 0x7f607c102f50 2024-02-09T22:49:13.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.337+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f607c1086c0 con 0x7f607c102f50 2024-02-09T22:49:13.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.338+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f6074005520 con 0x7f607c102f50 2024-02-09T22:49:13.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.339+0000 7f60727fc640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f60680a2150 0x7f60680a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.339+0000 7f6081726640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f60680a2150 0x7f60680a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.339+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f60740cdd60 con 0x7f607c102f50 2024-02-09T22:49:13.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.339+0000 7f6081726640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f60680a2150 0x7f60680a4610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6078036040 tx=0x7f6078000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:13.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.343+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6074096d80 con 0x7f607c102f50 2024-02-09T22:49:13.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.554+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f607c02cec0 con 0x7f60680a2150 2024-02-09T22:49:13.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.557+0000 7f60727fc640 1 -- 172.21.15.3:0/3783820349 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f607c02cec0 con 0x7f60680a2150 2024-02-09T22:49:13.560 INFO:teuthology.orchestra.run.smithi003.stdout:/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f60680a2150 msgr2=0x7f60680a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f60680a2150 0x7f60680a4610 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f6078036040 tx=0x7f6078000bc0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 msgr2=0x7f607c19e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 0x7f607c19e8f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f607400cad0 tx=0x7f607400cf90 comp rx=0 tx=0).stop 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 shutdown_connections 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f60680a2150 0x7f60680a4610 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f607c102590 0x7f607c197880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f607c102f50 0x7f607c19e8f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 --2- 172.21.15.3:0/3783820349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f607c0ff520 0x7f607c197340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.559+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 >> 172.21.15.3:0/3783820349 conn(0x7f607c0fae10 msgr2=0x7f607c105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.560+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 shutdown_connections 2024-02-09T22:49:13.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.560+0000 7f608398a640 1 -- 172.21.15.3:0/3783820349 wait complete. 2024-02-09T22:49:13.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:13 smithi003 ceph-mon[32678]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:13.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:13 smithi003 ceph-mon[32678]: pgmap v289: 97 pgs: 97 active+clean; 581 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s wr, 15 op/s 2024-02-09T22:49:13.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:13 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] up:active 2024-02-09T22:49:13.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:13 smithi003 ceph-mon[32678]: fsmap cephfs:5 7 up:standby 5 up:active 2024-02-09T22:49:13.573 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:13 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:49:13.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:49:13.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 -- 172.21.15.3:0/1864573962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694100b50 msgr2=0x7f8694100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 --2- 172.21.15.3:0/1864573962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694100b50 0x7f8694100fd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8688009900 tx=0x7f868802bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 -- 172.21.15.3:0/1864573962 shutdown_connections 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 --2- 172.21.15.3:0/1864573962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8694101510 0x7f8694107da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 --2- 172.21.15.3:0/1864573962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694100b50 0x7f8694100fd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 --2- 172.21.15.3:0/1864573962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86940ff960 0x7f86940ffd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 -- 172.21.15.3:0/1864573962 >> 172.21.15.3:0/1864573962 conn(0x7f86940faeb0 msgr2=0x7f86940fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 -- 172.21.15.3:0/1864573962 shutdown_connections 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 -- 172.21.15.3:0/1864573962 wait complete. 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 Processor -- start 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.673+0000 7f869a377640 1 -- start start 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86940ff960 0x7f86941038f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8694100b50 0x7f8694101f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 0x7f8694102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8694102bd0 con 0x7f8694101510 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8694102d40 con 0x7f86940ff960 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86941a1380 con 0x7f8694100b50 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 0x7f8694102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 0x7f8694102470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50344/0 (socket says 172.21.15.3:50344) 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 -- 172.21.15.3:0/52060128 learned_addr learned my addr 172.21.15.3:0/52060128 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f86937fe640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8694100b50 0x7f8694101f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8693fff640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86940ff960 0x7f86941038f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 -- 172.21.15.3:0/52060128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8694100b50 msgr2=0x7f8694101f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8694100b50 0x7f8694101f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 -- 172.21.15.3:0/52060128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86940ff960 msgr2=0x7f86941038f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86940ff960 0x7f86941038f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 -- 172.21.15.3:0/52060128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688009560 con 0x7f8694101510 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8693fff640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86940ff960 0x7f86941038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:13.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f8698914640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 0x7f8694102470 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f868c00cb00 tx=0x7f868c00a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c00ae90 con 0x7f8694101510 2024-02-09T22:49:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f868c005020 con 0x7f8694101510 2024-02-09T22:49:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86941a1630 con 0x7f8694101510 2024-02-09T22:49:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868c00f6b0 con 0x7f8694101510 2024-02-09T22:49:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.674+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86941a1b10 con 0x7f8694101510 2024-02-09T22:49:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.676+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869410bb70 con 0x7f8694101510 2024-02-09T22:49:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.677+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f868c005520 con 0x7f8694101510 2024-02-09T22:49:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.677+0000 7f86917fa640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f867c0a21d0 0x7f867c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.677+0000 7f8693fff640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f867c0a21d0 0x7f867c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.677+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f868c0cdc30 con 0x7f8694101510 2024-02-09T22:49:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.678+0000 7f8693fff640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f867c0a21d0 0x7f867c0a4690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f86941009b0 tx=0x7f868400a560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.682+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f868c096c50 con 0x7f8694101510 2024-02-09T22:49:13.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:13 smithi064 ceph-mon[36638]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:13.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:13 smithi064 ceph-mon[36638]: pgmap v289: 97 pgs: 97 active+clean; 581 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s wr, 15 op/s 2024-02-09T22:49:13.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:13 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] up:active 2024-02-09T22:49:13.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:13 smithi064 ceph-mon[36638]: fsmap cephfs:5 7 up:standby 5 up:active 2024-02-09T22:49:13.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:13 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:49:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:13 smithi089 ceph-mon[36665]: from='client.14754 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:13 smithi089 ceph-mon[36665]: pgmap v289: 97 pgs: 97 active+clean; 581 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s wr, 15 op/s 2024-02-09T22:49:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:13 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] up:active 2024-02-09T22:49:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:13 smithi089 ceph-mon[36665]: fsmap cephfs:5 7 up:standby 5 up:active 2024-02-09T22:49:13.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:13 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:49:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.900+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86941a1d90 con 0x7f8694101510 2024-02-09T22:49:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.900+0000 7f86917fa640 1 -- 172.21.15.3:0/52060128 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f86941a1d90 con 0x7f8694101510 2024-02-09T22:49:13.903 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:13.904 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:49:13.115988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:49:13.904 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:49:13.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f867c0a21d0 msgr2=0x7f867c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f867c0a21d0 0x7f867c0a4690 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f86941009b0 tx=0x7f868400a560 comp rx=0 tx=0).stop 2024-02-09T22:49:13.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 msgr2=0x7f8694102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 0x7f8694102470 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f868c00cb00 tx=0x7f868c00a250 comp rx=0 tx=0).stop 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 shutdown_connections 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f867c0a21d0 0x7f867c0a4690 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8694101510 0x7f8694102470 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8694100b50 0x7f8694101f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 --2- 172.21.15.3:0/52060128 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86940ff960 0x7f86941038f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 >> 172.21.15.3:0/52060128 conn(0x7f86940faeb0 msgr2=0x7f86940fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 shutdown_connections 2024-02-09T22:49:13.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:13.902+0000 7f869a377640 1 -- 172.21.15.3:0/52060128 wait complete. 2024-02-09T22:49:13.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 -- 172.21.15.3:0/876068770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 msgr2=0x7f44001085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 --2- 172.21.15.3:0/876068770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f44001085a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009ca0 tx=0x7f43f802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 -- 172.21.15.3:0/876068770 shutdown_connections 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 --2- 172.21.15.3:0/876068770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f44001085a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 --2- 172.21.15.3:0/876068770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4400101350 0x7f44001017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 --2- 172.21.15.3:0/876068770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400100160 0x7f4400100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 -- 172.21.15.3:0/876068770 >> 172.21.15.3:0/876068770 conn(0x7f44000fb6d0 msgr2=0x7f44000fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 -- 172.21.15.3:0/876068770 shutdown_connections 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.016+0000 7f4405be2640 1 -- 172.21.15.3:0/876068770 wait complete. 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 Processor -- start 2024-02-09T22:49:14.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 -- start start 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4400100160 0x7f44001971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400101350 0x7f44001976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f440019e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019ec90 con 0x7f4400101d10 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019ee00 con 0x7f4400101350 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f4405be2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440019efc0 con 0x7f4400100160 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f43fffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f440019e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f43fffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f440019e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50364/0 (socket says 172.21.15.3:50364) 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f43fffff640 1 -- 172.21.15.3:0/966374054 learned_addr learned my addr 172.21.15.3:0/966374054 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.017+0000 7f43ff7fe640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4400100160 0x7f44001971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fffff640 1 -- 172.21.15.3:0/966374054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4400100160 msgr2=0x7f44001971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fffff640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4400100160 0x7f44001971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fffff640 1 -- 172.21.15.3:0/966374054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400101350 msgr2=0x7f44001976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43feffd640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400101350 0x7f44001976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fffff640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400101350 0x7f44001976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fffff640 1 -- 172.21.15.3:0/966374054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f8009900 con 0x7f4400101d10 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43feffd640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400101350 0x7f44001976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fffff640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f440019e750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f43f8037040 tx=0x7f43f8034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8032eb0 con 0x7f4400101d10 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43f8003e50 con 0x7f4400101d10 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440019f240 con 0x7f4400101d10 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8040450 con 0x7f4400101d10 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.018+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440019f6f0 con 0x7f4400101d10 2024-02-09T22:49:14.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.019+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44001073b0 con 0x7f4400101d10 2024-02-09T22:49:14.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.020+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f43f800a040 con 0x7f4400101d10 2024-02-09T22:49:14.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.020+0000 7f43fcff9640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f43e80a2150 0x7f43e80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.021+0000 7f43ff7fe640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f43e80a2150 0x7f43e80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.021+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f43f80f0cb0 con 0x7f4400101d10 2024-02-09T22:49:14.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.021+0000 7f43ff7fe640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f43e80a2150 0x7f43e80a4610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005fd0 tx=0x7f43f0005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:14.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.025+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f43f80b9ce0 con 0x7f4400101d10 2024-02-09T22:49:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.207+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f440002ce50 con 0x7f4400101d10 2024-02-09T22:49:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.208+0000 7f43fcff9640 1 -- 172.21.15.3:0/966374054 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22028 (secure 0 0 0) 0x7f43f80bd3a0 con 0x7f4400101d10 2024-02-09T22:49:14.212 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:14.212 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:49:08.239545+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:49:14.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f43e80a2150 msgr2=0x7f43e80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f43e80a2150 0x7f43e80a4610 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f43f0005fd0 tx=0x7f43f0005850 comp rx=0 tx=0).stop 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 msgr2=0x7f440019e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f440019e750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f43f8037040 tx=0x7f43f8034d40 comp rx=0 tx=0).stop 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 shutdown_connections 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f43e80a2150 0x7f43e80a4610 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4400101350 0x7f44001976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4400101d10 0x7f440019e750 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 --2- 172.21.15.3:0/966374054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4400100160 0x7f44001971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.210+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 >> 172.21.15.3:0/966374054 conn(0x7f44000fb6d0 msgr2=0x7f4400104580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.211+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 shutdown_connections 2024-02-09T22:49:14.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.211+0000 7f4405be2640 1 -- 172.21.15.3:0/966374054 wait complete. 2024-02-09T22:49:14.225 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-02-09T22:49:14.225 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- 172.21.15.3:0/3603748790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 msgr2=0x7fadd01005f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- 172.21.15.3:0/3603748790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd01005f0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fadc4009900 tx=0x7fadc402bc30 comp rx=0 tx=0).stop 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- 172.21.15.3:0/3603748790 shutdown_connections 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- 172.21.15.3:0/3603748790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd0101d80 0x7fadd0108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- 172.21.15.3:0/3603748790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadd01013c0 0x7fadd0101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- 172.21.15.3:0/3603748790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd01005f0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- 172.21.15.3:0/3603748790 >> 172.21.15.3:0/3603748790 conn(0x7fadd00fb760 msgr2=0x7fadd00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- 172.21.15.3:0/3603748790 shutdown_connections 2024-02-09T22:49:14.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- 172.21.15.3:0/3603748790 wait complete. 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 Processor -- start 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- start start 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd0104160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd01013c0 0x7fadd01027a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadd0101d80 0x7fadd0102ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0103440 con 0x7fadd01001d0 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd01035b0 con 0x7fadd01013c0 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd704e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd01a1320 con 0x7fadd0101d80 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.328+0000 7fadd4dea640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd0104160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd0104160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50378/0 (socket says 172.21.15.3:50378) 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 -- 172.21.15.3:0/215487391 learned_addr learned my addr 172.21.15.3:0/215487391 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadcffff640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd01013c0 0x7fadd01027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd55eb640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadd0101d80 0x7fadd0102ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 -- 172.21.15.3:0/215487391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadd0101d80 msgr2=0x7fadd0102ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadd0101d80 0x7fadd0102ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 -- 172.21.15.3:0/215487391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd01013c0 msgr2=0x7fadd01027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd01013c0 0x7fadd01027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 -- 172.21.15.3:0/215487391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4009560 con 0x7fadd01001d0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadcffff640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd01013c0 0x7fadd01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd4dea640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd0104160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fadc4036040 tx=0x7fadc4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc4004df0 con 0x7fadd01001d0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fadc4005430 con 0x7fadd01001d0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd01a1570 con 0x7fadd01001d0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadc403e450 con 0x7fadd01001d0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.329+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd01a1a80 con 0x7fadd01001d0 2024-02-09T22:49:14.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.330+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd010bb70 con 0x7fadd01001d0 2024-02-09T22:49:14.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.331+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fadc4003680 con 0x7fadd01001d0 2024-02-09T22:49:14.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.332+0000 7fadcdffb640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fadc00a2150 0x7fadc00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.332+0000 7fadcffff640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fadc00a2150 0x7fadc00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.332+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fadc40f0060 con 0x7fadd01001d0 2024-02-09T22:49:14.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.332+0000 7fadcffff640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fadc00a2150 0x7fadc00a4610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0058c0 tx=0x7fadbc005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:14.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.337+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fadc40b9090 con 0x7fadd01001d0 2024-02-09T22:49:14.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.514+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fadd00fd8b0 con 0x7fadc00a2150 2024-02-09T22:49:14.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.525+0000 7fadcdffb640 1 -- 172.21.15.3:0/215487391 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335693 (secure 0 0 0) 0x7fadd00fd8b0 con 0x7fadc00a2150 2024-02-09T22:49:14.532 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:14.534 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:49:14.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fadc00a2150 msgr2=0x7fadc00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fadc00a2150 0x7fadc00a4610 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fadbc0058c0 tx=0x7fadbc005850 comp rx=0 tx=0).stop 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 msgr2=0x7fadd0104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd0104160 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fadc4036040 tx=0x7fadc4004270 comp rx=0 tx=0).stop 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 shutdown_connections 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fadc00a2150 0x7fadc00a4610 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fadd01013c0 0x7fadd01027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fadd0101d80 0x7fadd0102ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 --2- 172.21.15.3:0/215487391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fadd01001d0 0x7fadd0104160 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.528+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 >> 172.21.15.3:0/215487391 conn(0x7fadd00fb760 msgr2=0x7fadd00fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.529+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 shutdown_connections 2024-02-09T22:49:14.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.529+0000 7fadd704e640 1 -- 172.21.15.3:0/215487391 wait complete. 2024-02-09T22:49:14.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:14 smithi003 ceph-mon[32678]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:14 smithi003 ceph-mon[32678]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:14 smithi003 ceph-mon[32678]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-02-09T22:49:14.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:14 smithi003 ceph-mon[32678]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:14 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/52060128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:14.542 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:14 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/966374054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:14.543 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2024-02-09T22:49:12.537511+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594946,"num_objects":47,"num_object_clones":0,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":276,"num_write_kb":2967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":222,"ondisk_log_size":222,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":258,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353648,"kb_used_data":12756,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588176,"statfs":{"total":1151940427776,"available":1151578292224,"internally_reserved":0,"allocated":13062144,"data_stored":7100683,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":4578,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.297334"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804488+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:49:08.804488+0000","last_peered":"2024-02-09T22:49:08.804488+0000","last_clean":"2024-02-09T22:49:08.804488+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:49:08.804488+0000","last_undegraded":"2024-02-09T22:49:08.804488+0000","last_fullsized":"2024-02-09T22:49:08.804488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803272+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:49:08.803272+0000","last_peered":"2024-02-09T22:49:08.803272+0000","last_clean":"2024-02-09T22:49:08.803272+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:49:08.803272+0000","last_undegraded":"2024-02-09T22:49:08.803272+0000","last_fullsized":"2024-02-09T22:49:08.803272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803108+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:49:08.803108+0000","last_peered":"2024-02-09T22:49:08.803108+0000","last_clean":"2024-02-09T22:49:08.803108+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:49:08.803108+0000","last_undegraded":"2024-02-09T22:49:08.803108+0000","last_fullsized":"2024-02-09T22:49:08.803108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.270671+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:49:07.270671+0000","last_peered":"2024-02-09T22:49:07.270671+0000","last_clean":"2024-02-09T22:49:07.270671+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:49:07.270671+0000","last_undegraded":"2024-02-09T22:49:07.270671+0000","last_fullsized":"2024-02-09T22:49:07.270671+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:15:26.997278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302467+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:49:08.302467+0000","last_peered":"2024-02-09T22:49:08.302467+0000","last_clean":"2024-02-09T22:49:08.302467+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:49:08.302467+0000","last_undegraded":"2024-02-09T22:49:08.302467+0000","last_fullsized":"2024-02-09T22:49:08.302467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804771+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:49:08.804771+0000","last_peered":"2024-02-09T22:49:08.804771+0000","last_clean":"2024-02-09T22:49:08.804771+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:49:08.804771+0000","last_undegraded":"2024-02-09T22:49:08.804771+0000","last_fullsized":"2024-02-09T22:49:08.804771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805462+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:49:08.805462+0000","last_peered":"2024-02-09T22:49:08.805462+0000","last_clean":"2024-02-09T22:49:08.805462+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:49:08.805462+0000","last_undegraded":"2024-02-09T22:49:08.805462+0000","last_fullsized":"2024-02-09T22:49:08.805462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248368+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:49:08.248368+0000","last_peered":"2024-02-09T22:49:08.248368+0000","last_clean":"2024-02-09T22:49:08.248368+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:49:08.248368+0000","last_undegraded":"2024-02-09T22:49:08.248368+0000","last_fullsized":"2024-02-09T22:49:08.248368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250252+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:49:08.250252+0000","last_peered":"2024-02-09T22:49:08.250252+0000","last_clean":"2024-02-09T22:49:08.250252+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:49:08.250252+0000","last_undegraded":"2024-02-09T22:49:08.250252+0000","last_fullsized":"2024-02-09T22:49:08.250252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805394+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:49:08.805394+0000","last_peered":"2024-02-09T22:49:08.805394+0000","last_clean":"2024-02-09T22:49:08.805394+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:49:08.805394+0000","last_undegraded":"2024-02-09T22:49:08.805394+0000","last_fullsized":"2024-02-09T22:49:08.805394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805565+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:49:08.805565+0000","last_peered":"2024-02-09T22:49:08.805565+0000","last_clean":"2024-02-09T22:49:08.805565+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:49:08.805565+0000","last_undegraded":"2024-02-09T22:49:08.805565+0000","last_fullsized":"2024-02-09T22:49:08.805565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.270650+0000","last_change":"2024-02-09T22:49:07.268535+0000","last_active":"2024-02-09T22:49:07.270650+0000","last_peered":"2024-02-09T22:49:07.270650+0000","last_clean":"2024-02-09T22:49:07.270650+0000","last_became_active":"2024-02-09T22:49:07.267967+0000","last_became_peered":"2024-02-09T22:49:07.267967+0000","last_unstale":"2024-02-09T22:49:07.270650+0000","last_undegraded":"2024-02-09T22:49:07.270650+0000","last_fullsized":"2024-02-09T22:49:07.270650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:37:59.640969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804141+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:49:08.804141+0000","last_peered":"2024-02-09T22:49:08.804141+0000","last_clean":"2024-02-09T22:49:08.804141+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:49:08.804141+0000","last_undegraded":"2024-02-09T22:49:08.804141+0000","last_fullsized":"2024-02-09T22:49:08.804141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803667+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:49:08.803667+0000","last_peered":"2024-02-09T22:49:08.803667+0000","last_clean":"2024-02-09T22:49:08.803667+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:49:08.803667+0000","last_undegraded":"2024-02-09T22:49:08.803667+0000","last_fullsized":"2024-02-09T22:49:08.803667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.802965+0000","last_change":"2024-02-09T22:49:07.269581+0000","last_active":"2024-02-09T22:49:08.802965+0000","last_peered":"2024-02-09T22:49:08.802965+0000","last_clean":"2024-02-09T22:49:08.802965+0000","last_became_active":"2024-02-09T22:49:07.269228+0000","last_became_peered":"2024-02-09T22:49:07.269228+0000","last_unstale":"2024-02-09T22:49:08.802965+0000","last_undegraded":"2024-02-09T22:49:08.802965+0000","last_fullsized":"2024-02-09T22:49:08.802965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:07:20.365278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.249927+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:49:08.249927+0000","last_peered":"2024-02-09T22:49:08.249927+0000","last_clean":"2024-02-09T22:49:08.249927+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:49:08.249927+0000","last_undegraded":"2024-02-09T22:49:08.249927+0000","last_fullsized":"2024-02-09T22:49:08.249927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804535+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:49:08.804535+0000","last_peered":"2024-02-09T22:49:08.804535+0000","last_clean":"2024-02-09T22:49:08.804535+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:49:08.804535+0000","last_undegraded":"2024-02-09T22:49:08.804535+0000","last_fullsized":"2024-02-09T22:49:08.804535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804230+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:49:08.804230+0000","last_peered":"2024-02-09T22:49:08.804230+0000","last_clean":"2024-02-09T22:49:08.804230+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:49:08.804230+0000","last_undegraded":"2024-02-09T22:49:08.804230+0000","last_fullsized":"2024-02-09T22:49:08.804230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.263035+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:49:07.263035+0000","last_peered":"2024-02-09T22:49:07.263035+0000","last_clean":"2024-02-09T22:49:07.263035+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:49:07.263035+0000","last_undegraded":"2024-02-09T22:49:07.263035+0000","last_fullsized":"2024-02-09T22:49:07.263035+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:28.922194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804351+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:49:08.804351+0000","last_peered":"2024-02-09T22:49:08.804351+0000","last_clean":"2024-02-09T22:49:08.804351+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:49:08.804351+0000","last_undegraded":"2024-02-09T22:49:08.804351+0000","last_fullsized":"2024-02-09T22:49:08.804351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248581+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:49:08.248581+0000","last_peered":"2024-02-09T22:49:08.248581+0000","last_clean":"2024-02-09T22:49:08.248581+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:49:08.248581+0000","last_undegraded":"2024-02-09T22:49:08.248581+0000","last_fullsized":"2024-02-09T22:49:08.248581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248893+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:49:08.248893+0000","last_peered":"2024-02-09T22:49:08.248893+0000","last_clean":"2024-02-09T22:49:08.248893+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:49:08.248893+0000","last_undegraded":"2024-02-09T22:49:08.248893+0000","last_fullsized":"2024-02-09T22:49:08.248893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803726+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:49:08.803726+0000","last_peered":"2024-02-09T22:49:08.803726+0000","last_clean":"2024-02-09T22:49:08.803726+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:49:08.803726+0000","last_undegraded":"2024-02-09T22:49:08.803726+0000","last_fullsized":"2024-02-09T22:49:08.803726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302792+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:49:08.302792+0000","last_peered":"2024-02-09T22:49:08.302792+0000","last_clean":"2024-02-09T22:49:08.302792+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:49:08.302792+0000","last_undegraded":"2024-02-09T22:49:08.302792+0000","last_fullsized":"2024-02-09T22:49:08.302792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250082+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:49:08.250082+0000","last_peered":"2024-02-09T22:49:08.250082+0000","last_clean":"2024-02-09T22:49:08.250082+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:49:08.250082+0000","last_undegraded":"2024-02-09T22:49:08.250082+0000","last_fullsized":"2024-02-09T22:49:08.250082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302621+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:49:08.302621+0000","last_peered":"2024-02-09T22:49:08.302621+0000","last_clean":"2024-02-09T22:49:08.302621+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:49:08.302621+0000","last_undegraded":"2024-02-09T22:49:08.302621+0000","last_fullsized":"2024-02-09T22:49:08.302621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.254360+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:49:08.254360+0000","last_peered":"2024-02-09T22:49:08.254360+0000","last_clean":"2024-02-09T22:49:08.254360+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:49:08.254360+0000","last_undegraded":"2024-02-09T22:49:08.254360+0000","last_fullsized":"2024-02-09T22:49:08.254360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098316+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:49:09.098316+0000","last_peered":"2024-02-09T22:49:09.098316+0000","last_clean":"2024-02-09T22:49:09.098316+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:49:09.098316+0000","last_undegraded":"2024-02-09T22:49:09.098316+0000","last_fullsized":"2024-02-09T22:49:09.098316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302429+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:49:08.302429+0000","last_peered":"2024-02-09T22:49:08.302429+0000","last_clean":"2024-02-09T22:49:08.302429+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:49:08.302429+0000","last_undegraded":"2024-02-09T22:49:08.302429+0000","last_fullsized":"2024-02-09T22:49:08.302429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.107227+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:49:10.107227+0000","last_peered":"2024-02-09T22:49:10.107227+0000","last_clean":"2024-02-09T22:49:10.107227+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:49:10.107227+0000","last_undegraded":"2024-02-09T22:49:10.107227+0000","last_fullsized":"2024-02-09T22:49:10.107227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.254248+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:49:08.254248+0000","last_peered":"2024-02-09T22:49:08.254248+0000","last_clean":"2024-02-09T22:49:08.254248+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:49:08.254248+0000","last_undegraded":"2024-02-09T22:49:08.254248+0000","last_fullsized":"2024-02-09T22:49:08.254248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.109383+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:49:10.109383+0000","last_peered":"2024-02-09T22:49:10.109383+0000","last_clean":"2024-02-09T22:49:10.109383+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:49:10.109383+0000","last_undegraded":"2024-02-09T22:49:10.109383+0000","last_fullsized":"2024-02-09T22:49:10.109383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248444+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:49:08.248444+0000","last_peered":"2024-02-09T22:49:08.248444+0000","last_clean":"2024-02-09T22:49:08.248444+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:49:08.248444+0000","last_undegraded":"2024-02-09T22:49:08.248444+0000","last_fullsized":"2024-02-09T22:49:08.248444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098282+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:49:09.098282+0000","last_peered":"2024-02-09T22:49:09.098282+0000","last_clean":"2024-02-09T22:49:09.098282+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:49:09.098282+0000","last_undegraded":"2024-02-09T22:49:09.098282+0000","last_fullsized":"2024-02-09T22:49:09.098282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803690+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:49:08.803690+0000","last_peered":"2024-02-09T22:49:08.803690+0000","last_clean":"2024-02-09T22:49:08.803690+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:49:08.803690+0000","last_undegraded":"2024-02-09T22:49:08.803690+0000","last_fullsized":"2024-02-09T22:49:08.803690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098225+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:49:09.098225+0000","last_peered":"2024-02-09T22:49:09.098225+0000","last_clean":"2024-02-09T22:49:09.098225+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:49:09.098225+0000","last_undegraded":"2024-02-09T22:49:09.098225+0000","last_fullsized":"2024-02-09T22:49:09.098225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250212+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:49:08.250212+0000","last_peered":"2024-02-09T22:49:08.250212+0000","last_clean":"2024-02-09T22:49:08.250212+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:49:08.250212+0000","last_undegraded":"2024-02-09T22:49:08.250212+0000","last_fullsized":"2024-02-09T22:49:08.250212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.099267+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:49:09.099267+0000","last_peered":"2024-02-09T22:49:09.099267+0000","last_clean":"2024-02-09T22:49:09.099267+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:49:09.099267+0000","last_undegraded":"2024-02-09T22:49:09.099267+0000","last_fullsized":"2024-02-09T22:49:09.099267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803874+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:49:08.803874+0000","last_peered":"2024-02-09T22:49:08.803874+0000","last_clean":"2024-02-09T22:49:08.803874+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:49:08.803874+0000","last_undegraded":"2024-02-09T22:49:08.803874+0000","last_fullsized":"2024-02-09T22:49:08.803874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098019+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:49:09.098019+0000","last_peered":"2024-02-09T22:49:09.098019+0000","last_clean":"2024-02-09T22:49:09.098019+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:49:09.098019+0000","last_undegraded":"2024-02-09T22:49:09.098019+0000","last_fullsized":"2024-02-09T22:49:09.098019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250260+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:49:08.250260+0000","last_peered":"2024-02-09T22:49:08.250260+0000","last_clean":"2024-02-09T22:49:08.250260+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:49:08.250260+0000","last_undegraded":"2024-02-09T22:49:08.250260+0000","last_fullsized":"2024-02-09T22:49:08.250260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098152+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:49:09.098152+0000","last_peered":"2024-02-09T22:49:09.098152+0000","last_clean":"2024-02-09T22:49:09.098152+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:49:09.098152+0000","last_undegraded":"2024-02-09T22:49:09.098152+0000","last_fullsized":"2024-02-09T22:49:09.098152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302677+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:49:08.302677+0000","last_peered":"2024-02-09T22:49:08.302677+0000","last_clean":"2024-02-09T22:49:08.302677+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:49:08.302677+0000","last_undegraded":"2024-02-09T22:49:08.302677+0000","last_fullsized":"2024-02-09T22:49:08.302677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.548860+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:49:07.548860+0000","last_peered":"2024-02-09T22:49:07.548860+0000","last_clean":"2024-02-09T22:49:07.548860+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:49:07.548860+0000","last_undegraded":"2024-02-09T22:49:07.548860+0000","last_fullsized":"2024-02-09T22:49:07.548860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803864+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:49:08.803864+0000","last_peered":"2024-02-09T22:49:08.803864+0000","last_clean":"2024-02-09T22:49:08.803864+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:49:08.803864+0000","last_undegraded":"2024-02-09T22:49:08.803864+0000","last_fullsized":"2024-02-09T22:49:08.803864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098260+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:49:09.098260+0000","last_peered":"2024-02-09T22:49:09.098260+0000","last_clean":"2024-02-09T22:49:09.098260+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:49:09.098260+0000","last_undegraded":"2024-02-09T22:49:09.098260+0000","last_fullsized":"2024-02-09T22:49:09.098260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805731+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:49:08.805731+0000","last_peered":"2024-02-09T22:49:08.805731+0000","last_clean":"2024-02-09T22:49:08.805731+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:49:08.805731+0000","last_undegraded":"2024-02-09T22:49:08.805731+0000","last_fullsized":"2024-02-09T22:49:08.805731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.256485+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:49:07.256485+0000","last_peered":"2024-02-09T22:49:07.256485+0000","last_clean":"2024-02-09T22:49:07.256485+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:49:07.256485+0000","last_undegraded":"2024-02-09T22:49:07.256485+0000","last_fullsized":"2024-02-09T22:49:07.256485+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804017+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:49:08.804017+0000","last_peered":"2024-02-09T22:49:08.804017+0000","last_clean":"2024-02-09T22:49:08.804017+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:49:08.804017+0000","last_undegraded":"2024-02-09T22:49:08.804017+0000","last_fullsized":"2024-02-09T22:49:08.804017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.096177+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:49:09.096177+0000","last_peered":"2024-02-09T22:49:09.096177+0000","last_clean":"2024-02-09T22:49:09.096177+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:49:09.096177+0000","last_undegraded":"2024-02-09T22:49:09.096177+0000","last_fullsized":"2024-02-09T22:49:09.096177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.253927+0000","last_change":"2024-02-09T22:49:07.259557+0000","last_active":"2024-02-09T22:49:08.253927+0000","last_peered":"2024-02-09T22:49:08.253927+0000","last_clean":"2024-02-09T22:49:08.253927+0000","last_became_active":"2024-02-09T22:49:07.259234+0000","last_became_peered":"2024-02-09T22:49:07.259234+0000","last_unstale":"2024-02-09T22:49:08.253927+0000","last_undegraded":"2024-02-09T22:49:08.253927+0000","last_fullsized":"2024-02-09T22:49:08.253927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:00:37.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302747+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:49:08.302747+0000","last_peered":"2024-02-09T22:49:08.302747+0000","last_clean":"2024-02-09T22:49:08.302747+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:49:08.302747+0000","last_undegraded":"2024-02-09T22:49:08.302747+0000","last_fullsized":"2024-02-09T22:49:08.302747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804178+0000","last_change":"2024-02-09T22:49:07.270133+0000","last_active":"2024-02-09T22:49:08.804178+0000","last_peered":"2024-02-09T22:49:08.804178+0000","last_clean":"2024-02-09T22:49:08.804178+0000","last_became_active":"2024-02-09T22:49:07.269741+0000","last_became_peered":"2024-02-09T22:49:07.269741+0000","last_unstale":"2024-02-09T22:49:08.804178+0000","last_undegraded":"2024-02-09T22:49:08.804178+0000","last_fullsized":"2024-02-09T22:49:08.804178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:41:21.489866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804059+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:49:08.804059+0000","last_peered":"2024-02-09T22:49:08.804059+0000","last_clean":"2024-02-09T22:49:08.804059+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:49:08.804059+0000","last_undegraded":"2024-02-09T22:49:08.804059+0000","last_fullsized":"2024-02-09T22:49:08.804059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804709+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:49:08.804709+0000","last_peered":"2024-02-09T22:49:08.804709+0000","last_clean":"2024-02-09T22:49:08.804709+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:49:08.804709+0000","last_undegraded":"2024-02-09T22:49:08.804709+0000","last_fullsized":"2024-02-09T22:49:08.804709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805548+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:49:08.805548+0000","last_peered":"2024-02-09T22:49:08.805548+0000","last_clean":"2024-02-09T22:49:08.805548+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:49:08.805548+0000","last_undegraded":"2024-02-09T22:49:08.805548+0000","last_fullsized":"2024-02-09T22:49:08.805548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804914+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:49:08.804914+0000","last_peered":"2024-02-09T22:49:08.804914+0000","last_clean":"2024-02-09T22:49:08.804914+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:49:08.804914+0000","last_undegraded":"2024-02-09T22:49:08.804914+0000","last_fullsized":"2024-02-09T22:49:08.804914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.253873+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:49:08.253873+0000","last_peered":"2024-02-09T22:49:08.253873+0000","last_clean":"2024-02-09T22:49:08.253873+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:49:08.253873+0000","last_undegraded":"2024-02-09T22:49:08.253873+0000","last_fullsized":"2024-02-09T22:49:08.253873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.109078+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:49:10.109078+0000","last_peered":"2024-02-09T22:49:10.109078+0000","last_clean":"2024-02-09T22:49:10.109078+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:49:10.109078+0000","last_undegraded":"2024-02-09T22:49:10.109078+0000","last_fullsized":"2024-02-09T22:49:10.109078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248615+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:49:08.248615+0000","last_peered":"2024-02-09T22:49:08.248615+0000","last_clean":"2024-02-09T22:49:08.248615+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:49:08.248615+0000","last_undegraded":"2024-02-09T22:49:08.248615+0000","last_fullsized":"2024-02-09T22:49:08.248615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.257011+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:49:07.257011+0000","last_peered":"2024-02-09T22:49:07.257011+0000","last_clean":"2024-02-09T22:49:07.257011+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:49:07.257011+0000","last_undegraded":"2024-02-09T22:49:07.257011+0000","last_fullsized":"2024-02-09T22:49:07.257011+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804025+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:49:08.804025+0000","last_peered":"2024-02-09T22:49:08.804025+0000","last_clean":"2024-02-09T22:49:08.804025+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:49:08.804025+0000","last_undegraded":"2024-02-09T22:49:08.804025+0000","last_fullsized":"2024-02-09T22:49:08.804025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803070+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:49:08.803070+0000","last_peered":"2024-02-09T22:49:08.803070+0000","last_clean":"2024-02-09T22:49:08.803070+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:49:08.803070+0000","last_undegraded":"2024-02-09T22:49:08.803070+0000","last_fullsized":"2024-02-09T22:49:08.803070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.108037+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:49:10.108037+0000","last_peered":"2024-02-09T22:49:10.108037+0000","last_clean":"2024-02-09T22:49:10.108037+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:49:10.108037+0000","last_undegraded":"2024-02-09T22:49:10.108037+0000","last_fullsized":"2024-02-09T22:49:10.108037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.095925+0000","last_change":"2024-02-09T22:49:06.259702+0000","last_active":"2024-02-09T22:49:09.095925+0000","last_peered":"2024-02-09T22:49:09.095925+0000","last_clean":"2024-02-09T22:49:09.095925+0000","last_became_active":"2024-02-09T22:49:06.259352+0000","last_became_peered":"2024-02-09T22:49:06.259352+0000","last_unstale":"2024-02-09T22:49:09.095925+0000","last_undegraded":"2024-02-09T22:49:09.095925+0000","last_fullsized":"2024-02-09T22:49:09.095925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:01:11.212161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803611+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:49:08.803611+0000","last_peered":"2024-02-09T22:49:08.803611+0000","last_clean":"2024-02-09T22:49:08.803611+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:49:08.803611+0000","last_undegraded":"2024-02-09T22:49:08.803611+0000","last_fullsized":"2024-02-09T22:49:08.803611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.543824+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:49:09.543824+0000","last_peered":"2024-02-09T22:49:09.543824+0000","last_clean":"2024-02-09T22:49:09.543824+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:49:09.543824+0000","last_undegraded":"2024-02-09T22:49:09.543824+0000","last_fullsized":"2024-02-09T22:49:09.543824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248940+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:49:08.248940+0000","last_peered":"2024-02-09T22:49:08.248940+0000","last_clean":"2024-02-09T22:49:08.248940+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:49:08.248940+0000","last_undegraded":"2024-02-09T22:49:08.248940+0000","last_fullsized":"2024-02-09T22:49:08.248940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"85'2","reported_seq":15,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.546504+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:49:07.546504+0000","last_peered":"2024-02-09T22:49:07.546504+0000","last_clean":"2024-02-09T22:49:07.546504+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:49:07.546504+0000","last_undegraded":"2024-02-09T22:49:07.546504+0000","last_fullsized":"2024-02-09T22:49:07.546504+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803475+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:49:08.803475+0000","last_peered":"2024-02-09T22:49:08.803475+0000","last_clean":"2024-02-09T22:49:08.803475+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:49:08.803475+0000","last_undegraded":"2024-02-09T22:49:08.803475+0000","last_fullsized":"2024-02-09T22:49:08.803475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.096928+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:49:09.096928+0000","last_peered":"2024-02-09T22:49:09.096928+0000","last_clean":"2024-02-09T22:49:09.096928+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:49:09.096928+0000","last_undegraded":"2024-02-09T22:49:09.096928+0000","last_fullsized":"2024-02-09T22:49:09.096928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302396+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:49:08.302396+0000","last_peered":"2024-02-09T22:49:08.302396+0000","last_clean":"2024-02-09T22:49:08.302396+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:49:08.302396+0000","last_undegraded":"2024-02-09T22:49:08.302396+0000","last_fullsized":"2024-02-09T22:49:08.302396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:11.117105+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:49:11.117105+0000","last_peered":"2024-02-09T22:49:11.117105+0000","last_clean":"2024-02-09T22:49:11.117105+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:49:11.117105+0000","last_undegraded":"2024-02-09T22:49:11.117105+0000","last_fullsized":"2024-02-09T22:49:11.117105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803712+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:49:08.803712+0000","last_peered":"2024-02-09T22:49:08.803712+0000","last_clean":"2024-02-09T22:49:08.803712+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:49:08.803712+0000","last_undegraded":"2024-02-09T22:49:08.803712+0000","last_fullsized":"2024-02-09T22:49:08.803712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.243000+0000","last_change":"2024-02-09T22:49:06.253697+0000","last_active":"2024-02-09T22:49:07.243000+0000","last_peered":"2024-02-09T22:49:07.243000+0000","last_clean":"2024-02-09T22:49:07.243000+0000","last_became_active":"2024-02-09T22:49:06.253036+0000","last_became_peered":"2024-02-09T22:49:06.253036+0000","last_unstale":"2024-02-09T22:49:07.243000+0000","last_undegraded":"2024-02-09T22:49:07.243000+0000","last_fullsized":"2024-02-09T22:49:07.243000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:33:01.928416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248775+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:49:08.248775+0000","last_peered":"2024-02-09T22:49:08.248775+0000","last_clean":"2024-02-09T22:49:08.248775+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:49:08.248775+0000","last_undegraded":"2024-02-09T22:49:08.248775+0000","last_fullsized":"2024-02-09T22:49:08.248775+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098673+0000","last_change":"2024-02-09T22:49:06.252438+0000","last_active":"2024-02-09T22:49:09.098673+0000","last_peered":"2024-02-09T22:49:09.098673+0000","last_clean":"2024-02-09T22:49:09.098673+0000","last_became_active":"2024-02-09T22:49:06.252111+0000","last_became_peered":"2024-02-09T22:49:06.252111+0000","last_unstale":"2024-02-09T22:49:09.098673+0000","last_undegraded":"2024-02-09T22:49:09.098673+0000","last_fullsized":"2024-02-09T22:49:09.098673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:04:16.992194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805720+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:49:08.805720+0000","last_peered":"2024-02-09T22:49:08.805720+0000","last_clean":"2024-02-09T22:49:08.805720+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:49:08.805720+0000","last_undegraded":"2024-02-09T22:49:08.805720+0000","last_fullsized":"2024-02-09T22:49:08.805720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.243152+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:49:07.243152+0000","last_peered":"2024-02-09T22:49:07.243152+0000","last_clean":"2024-02-09T22:49:07.243152+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:49:07.243152+0000","last_undegraded":"2024-02-09T22:49:07.243152+0000","last_fullsized":"2024-02-09T22:49:07.243152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805516+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:49:08.805516+0000","last_peered":"2024-02-09T22:49:08.805516+0000","last_clean":"2024-02-09T22:49:08.805516+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:49:08.805516+0000","last_undegraded":"2024-02-09T22:49:08.805516+0000","last_fullsized":"2024-02-09T22:49:08.805516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.254303+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:49:08.254303+0000","last_peered":"2024-02-09T22:49:08.254303+0000","last_clean":"2024-02-09T22:49:08.254303+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:49:08.254303+0000","last_undegraded":"2024-02-09T22:49:08.254303+0000","last_fullsized":"2024-02-09T22:49:08.254303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805499+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:49:08.805499+0000","last_peered":"2024-02-09T22:49:08.805499+0000","last_clean":"2024-02-09T22:49:08.805499+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:49:08.805499+0000","last_undegraded":"2024-02-09T22:49:08.805499+0000","last_fullsized":"2024-02-09T22:49:08.805499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803684+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:49:08.803684+0000","last_peered":"2024-02-09T22:49:08.803684+0000","last_clean":"2024-02-09T22:49:08.803684+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:49:08.803684+0000","last_undegraded":"2024-02-09T22:49:08.803684+0000","last_fullsized":"2024-02-09T22:49:08.803684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805689+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:49:08.805689+0000","last_peered":"2024-02-09T22:49:08.805689+0000","last_clean":"2024-02-09T22:49:08.805689+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:49:08.805689+0000","last_undegraded":"2024-02-09T22:49:08.805689+0000","last_fullsized":"2024-02-09T22:49:08.805689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803789+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:49:08.803789+0000","last_peered":"2024-02-09T22:49:08.803789+0000","last_clean":"2024-02-09T22:49:08.803789+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:49:08.803789+0000","last_undegraded":"2024-02-09T22:49:08.803789+0000","last_fullsized":"2024-02-09T22:49:08.803789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.263077+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:49:07.263077+0000","last_peered":"2024-02-09T22:49:07.263077+0000","last_clean":"2024-02-09T22:49:07.263077+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:49:07.263077+0000","last_undegraded":"2024-02-09T22:49:07.263077+0000","last_fullsized":"2024-02-09T22:49:07.263077+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:30:58.886143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.263607+0000","last_change":"2024-02-09T22:49:07.263133+0000","last_active":"2024-02-09T22:49:07.263607+0000","last_peered":"2024-02-09T22:49:07.263607+0000","last_clean":"2024-02-09T22:49:07.263607+0000","last_became_active":"2024-02-09T22:49:07.260838+0000","last_became_peered":"2024-02-09T22:49:07.260838+0000","last_unstale":"2024-02-09T22:49:07.263607+0000","last_undegraded":"2024-02-09T22:49:07.263607+0000","last_fullsized":"2024-02-09T22:49:07.263607+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:32:34.312220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803444+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:49:08.803444+0000","last_peered":"2024-02-09T22:49:08.803444+0000","last_clean":"2024-02-09T22:49:08.803444+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:49:08.803444+0000","last_undegraded":"2024-02-09T22:49:08.803444+0000","last_fullsized":"2024-02-09T22:49:08.803444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803443+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:49:08.803443+0000","last_peered":"2024-02-09T22:49:08.803443+0000","last_clean":"2024-02-09T22:49:08.803443+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:49:08.803443+0000","last_undegraded":"2024-02-09T22:49:08.803443+0000","last_fullsized":"2024-02-09T22:49:08.803443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804206+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:49:08.804206+0000","last_peered":"2024-02-09T22:49:08.804206+0000","last_clean":"2024-02-09T22:49:08.804206+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:49:08.804206+0000","last_undegraded":"2024-02-09T22:49:08.804206+0000","last_fullsized":"2024-02-09T22:49:08.804206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":10,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-02-09T22:49:07.264529+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:49:07.264529+0000","last_peered":"2024-02-09T22:49:07.264529+0000","last_clean":"2024-02-09T22:49:07.264529+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:49:07.264529+0000","last_undegraded":"2024-02-09T22:49:07.264529+0000","last_fullsized":"2024-02-09T22:49:07.264529+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:03:44.008235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248841+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:49:08.248841+0000","last_peered":"2024-02-09T22:49:08.248841+0000","last_clean":"2024-02-09T22:49:08.248841+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:49:08.248841+0000","last_undegraded":"2024-02-09T22:49:08.248841+0000","last_fullsized":"2024-02-09T22:49:08.248841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.253844+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:49:08.253844+0000","last_peered":"2024-02-09T22:49:08.253844+0000","last_clean":"2024-02-09T22:49:08.253844+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:49:08.253844+0000","last_undegraded":"2024-02-09T22:49:08.253844+0000","last_fullsized":"2024-02-09T22:49:08.253844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248680+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:49:08.248680+0000","last_peered":"2024-02-09T22:49:08.248680+0000","last_clean":"2024-02-09T22:49:08.248680+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:49:08.248680+0000","last_undegraded":"2024-02-09T22:49:08.248680+0000","last_fullsized":"2024-02-09T22:49:08.248680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.249906+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:49:08.249906+0000","last_peered":"2024-02-09T22:49:08.249906+0000","last_clean":"2024-02-09T22:49:08.249906+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:49:08.249906+0000","last_undegraded":"2024-02-09T22:49:08.249906+0000","last_fullsized":"2024-02-09T22:49:08.249906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804122+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:49:08.804122+0000","last_peered":"2024-02-09T22:49:08.804122+0000","last_clean":"2024-02-09T22:49:08.804122+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:49:08.804122+0000","last_undegraded":"2024-02-09T22:49:08.804122+0000","last_fullsized":"2024-02-09T22:49:08.804122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803875+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:49:08.803875+0000","last_peered":"2024-02-09T22:49:08.803875+0000","last_clean":"2024-02-09T22:49:08.803875+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:49:08.803875+0000","last_undegraded":"2024-02-09T22:49:08.803875+0000","last_fullsized":"2024-02-09T22:49:08.803875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4578,"num_objects":45,"num_object_clones":0,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":95,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":28,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":15100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":446368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]}]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1038734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":1,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.747},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.92600000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.95399999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.879},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77400000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":9,"up_from":67,"seq":287762808855,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":445960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60699999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.86199999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.81999999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.879},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.93000000000000005}]}]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":933888,"data_stored":445780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54700000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77700000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75600000000000001,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.89300000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":1.093},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67700000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.80900000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80000000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85299999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":1.073}]},{"osd":6,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":1.169}]},{"osd":7,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68600000000000005,"15min":0.69599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.876,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":1.2549999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.68100000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93899999999999995,"5min":1.03,"15min":1.03},"last":1.3160000000000001}]},{"osd":9,"last update":"Fri Feb 9 22:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":1.2330000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":1.1919999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":1.371},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":1.3440000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]},{"osd":7,"up_from":51,"seq":219043332128,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1531904,"data_stored":1036748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.372},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.68500000000000005}]},{"osd":2,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64900000000000002,"15min":0.64600000000000002},"min":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63300000000000001,"15min":0.627},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82199999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.626},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79000000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41299999999999998}]},{"osd":5,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70399999999999996,"15min":0.69999999999999996},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.72899999999999998,"15min":0.72199999999999998},"min":{"1min":0.504,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68400000000000005,"15min":0.68300000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.61799999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.73099999999999998,"15min":0.72199999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.70499999999999996,"15min":0.69499999999999995},"min":{"1min":0.54700000000000004,"5min":0.498,"15min":0.498},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.57399999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.66700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":445266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.35099999999999998,"15min":0.33800000000000002},"min":{"1min":0.26800000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.308},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39500000000000002},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.29099999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.45600000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.5}]},{"osd":3,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.41899999999999998,"15min":0.40400000000000003},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.46999999999999997,"15min":0.46300000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.36099999999999999}]},{"osd":4,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.499,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.42899999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.49199999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.65400000000000003,"15min":0.63800000000000001},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Feb 9 22:48:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47699999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78500000000000003,"5min":0.876,"15min":0.876},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.53500000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.40999999999999998}]},{"osd":10,"last update":"Fri Feb 9 22:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.51900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":446788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.94599999999999995,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.524,"5min":0.73499999999999999,"15min":0.58899999999999997},"min":{"1min":0.45800000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":18.152000000000001,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.629}]},{"osd":2,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":0.70299999999999996,"15min":0.57799999999999996},"min":{"1min":0.378,"5min":0.37,"15min":0.37},"max":{"1min":18.18,"5min":18.18,"15min":18.18},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.52500000000000002,"15min":0.49299999999999999},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.92800000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.57799999999999996,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.84199999999999997,"5min":1.206,"15min":1.206},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89900000000000002,"5min":1.139,"15min":1.139},"last":0.82999999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.45400000000000001,"15min":0.442},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.442,"15min":0.41599999999999998},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.747,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.625,"15min":0.59099999999999997},"min":{"1min":0.53000000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.499,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.872,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":1.038}]},{"osd":6,"last update":"Fri Feb 9 22:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64300000000000002,"15min":0.63700000000000001},"min":{"1min":0.41399999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69399999999999995,"15min":0.68799999999999994},"min":{"1min":0.45000000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.97399999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.45400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.875,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.878},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66800000000000004,"15min":0.66000000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93999999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":1.024}]},{"osd":8,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.5169999999999999,"15min":1.6359999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0980000000000001,"5min":18.132000000000001,"15min":18.132000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69099999999999995,"15min":0.68799999999999994},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.95399999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":1.0660000000000001}]},{"osd":10,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":1.099}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":446654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52800000000000002,"15min":0.48199999999999998},"min":{"1min":0.33800000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.47599999999999998},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81699999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.51700000000000002},"min":{"1min":0.41099999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73099999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.53400000000000003,"15min":0.503},"min":{"1min":0.36199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.78700000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.378}]},{"osd":3,"last update":"Fri Feb 9 22:48:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41599999999999998,"15min":0.40100000000000002},"min":{"1min":0.28299999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.25}]},{"osd":4,"last update":"Fri Feb 9 22:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59699999999999998,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60999999999999999,"15min":0.59299999999999997},"min":{"1min":0.443,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59399999999999997,"15min":0.57999999999999996},"min":{"1min":0.42599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80700000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.79700000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.308}]},{"osd":6,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53400000000000003,"15min":0.52200000000000002},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.33400000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.42199999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.504},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.82999999999999996,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.36399999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.41499999999999998}]},{"osd":10,"last update":"Fri Feb 9 22:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.49099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":438252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46000000000000002,"15min":0.45000000000000001},"min":{"1min":0.22500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.59499999999999997,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.47099999999999997},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.224},"max":{"1min":0.67700000000000005,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58699999999999997}]},{"osd":1,"last update":"Fri Feb 9 22:48:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.495,"15min":0.46700000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51600000000000001,"15min":0.47399999999999998},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79700000000000004,"5min":0.874,"15min":0.874},"last":0.41999999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.47999999999999998},"min":{"1min":0.35299999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.48399999999999999},"min":{"1min":0.371,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68200000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57199999999999995}]},{"osd":4,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.37}]},{"osd":5,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71299999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44,"15min":0.436},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.32700000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72699999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.60199999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.39000000000000001}]},{"osd":9,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.622}]},{"osd":10,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.437},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.54400000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53300000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":445584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50900000000000001,"5min":0.62,"15min":0.62},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.28999999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.59299999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.753}]},{"osd":2,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57099999999999995,"15min":0.54900000000000004},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77600000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.998}]},{"osd":4,"last update":"Fri Feb 9 22:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.60399999999999998,"15min":0.57799999999999996},"min":{"1min":0.51600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.61099999999999999,"15min":0.58699999999999997},"min":{"1min":0.505,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Feb 9 22:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.42799999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85099999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.88300000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.873},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.97699999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:48:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.66100000000000003,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.71299999999999997,"15min":0.69599999999999995},"min":{"1min":0.53100000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.0409999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":1.0229999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.93999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920816,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32064,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713088,"statfs":{"total":95995035648,"available":95962202112,"internally_reserved":0,"allocated":901120,"data_stored":427529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.39700000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.72299999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.38900000000000001},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.495}]},{"osd":2,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57399999999999995,"15min":0.55800000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58499999999999996,"15min":0.56399999999999995},"min":{"1min":0.38700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.70799999999999996}]},{"osd":3,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.43099999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.59799999999999998,"15min":0.57299999999999995},"min":{"1min":0.42799999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.61799999999999999,"15min":0.58399999999999996},"min":{"1min":0.46100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.80200000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.40699999999999997,"5min":0.38,"15min":0.38},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.64200000000000002,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.56399999999999995}]},{"osd":7,"last update":"Fri Feb 9 22:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.65500000000000003,"15min":0.65000000000000002},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.746},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.443,"5min":0.434,"15min":0.434},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.63200000000000001}]},{"osd":8,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.079,"5min":1.129,"15min":1.129},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.75,"15min":0.74199999999999999},"min":{"1min":0.51600000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.014,"5min":1.264,"15min":1.264},"last":0.83199999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.81699999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.877},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":0.65400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32688,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712464,"statfs":{"total":95995035648,"available":95961563136,"internally_reserved":0,"allocated":1548288,"data_stored":1037020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.51800000000000002,"15min":0.49199999999999999},"min":{"1min":0.40300000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.74299999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.622},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.50800000000000001},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.73099999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.52500000000000002,"15min":0.50600000000000001},"min":{"1min":0.372,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55900000000000005,"15min":0.53800000000000003},"min":{"1min":0.377,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83699999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.49299999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88900000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.45600000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64600000000000002}]},{"osd":8,"last update":"Fri Feb 9 22:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47399999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.56100000000000005}]},{"osd":10,"last update":"Fri Feb 9 22:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:49:14.551 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-09T22:49:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:14 smithi064 ceph-mon[36638]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:14 smithi064 ceph-mon[36638]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:14 smithi064 ceph-mon[36638]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-02-09T22:49:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:14 smithi064 ceph-mon[36638]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:14 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/52060128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:14.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:14 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/966374054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:14.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.683+0000 7fa7644b5640 1 -- 172.21.15.3:0/588643972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c100170 msgr2=0x7fa75c100590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.683+0000 7fa7644b5640 1 --2- 172.21.15.3:0/588643972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c100170 0x7fa75c100590 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa74c009ca0 tx=0x7fa74c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- 172.21.15.3:0/588643972 shutdown_connections 2024-02-09T22:49:14.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 --2- 172.21.15.3:0/588643972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c101360 0x7fa75c1017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 --2- 172.21.15.3:0/588643972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa75c101d20 0x7fa75c1085b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 --2- 172.21.15.3:0/588643972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c100170 0x7fa75c100590 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- 172.21.15.3:0/588643972 >> 172.21.15.3:0/588643972 conn(0x7fa75c0fb6c0 msgr2=0x7fa75c0fdb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- 172.21.15.3:0/588643972 shutdown_connections 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- 172.21.15.3:0/588643972 wait complete. 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 Processor -- start 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- start start 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c100170 0x7fa75c197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 0x7fa75c1978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa75c101d20 0x7fa75c19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19ee00 con 0x7fa75c101360 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19ef70 con 0x7fa75c100170 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa7644b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c19f130 con 0x7fa75c101d20 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa761a50640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 0x7fa75c1978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa761a50640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 0x7fa75c1978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50404/0 (socket says 172.21.15.3:50404) 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.684+0000 7fa761a50640 1 -- 172.21.15.3:0/3837863136 learned_addr learned my addr 172.21.15.3:0/3837863136 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa762251640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c100170 0x7fa75c197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa761a50640 1 -- 172.21.15.3:0/3837863136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa75c101d20 msgr2=0x7fa75c19e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa762a52640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa75c101d20 0x7fa75c19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa761a50640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa75c101d20 0x7fa75c19e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa761a50640 1 -- 172.21.15.3:0/3837863136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c100170 msgr2=0x7fa75c197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa761a50640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c100170 0x7fa75c197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa761a50640 1 -- 172.21.15.3:0/3837863136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa74c009900 con 0x7fa75c101360 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa762251640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c100170 0x7fa75c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:14.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa761a50640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 0x7fa75c1978c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa75000e9f0 tx=0x7fa75000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750009700 con 0x7fa75c101360 2024-02-09T22:49:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c19f410 con 0x7fa75c101360 2024-02-09T22:49:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa75000cd80 con 0x7fa75c101360 2024-02-09T22:49:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa750018920 con 0x7fa75c101360 2024-02-09T22:49:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.685+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c19f920 con 0x7fa75c101360 2024-02-09T22:49:14.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.686+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa75c10bb80 con 0x7fa75c101360 2024-02-09T22:49:14.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.687+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fa750018a80 con 0x7fa75c101360 2024-02-09T22:49:14.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.688+0000 7fa75b7fe640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7440a2150 0x7fa7440a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:14.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.688+0000 7fa762251640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7440a2150 0x7fa7440a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:14.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.688+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa750014070 con 0x7fa75c101360 2024-02-09T22:49:14.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.688+0000 7fa762251640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7440a2150 0x7fa7440a4610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa74c02c700 tx=0x7fa74c0326d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:14.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.692+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa750052d20 con 0x7fa75c101360 2024-02-09T22:49:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:14 smithi089 ceph-mon[36665]: from='client.14760 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:14 smithi089 ceph-mon[36665]: from='client.14766 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:14 smithi089 ceph-mon[36665]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-02-09T22:49:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:14 smithi089 ceph-mon[36665]: from='client.14772 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:49:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:14 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/52060128' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:49:14.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:14 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/966374054' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:14.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.870+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa75c0fd8c0 con 0x7fa7440a2150 2024-02-09T22:49:14.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.881+0000 7fa75b7fe640 1 -- 172.21.15.3:0/3837863136 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+335746 (secure 0 0 0) 0x7fa75c0fd8c0 con 0x7fa7440a2150 2024-02-09T22:49:14.887 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:14.890 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7440a2150 msgr2=0x7fa7440a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7440a2150 0x7fa7440a4610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa74c02c700 tx=0x7fa74c0326d0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 msgr2=0x7fa75c1978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 0x7fa75c1978c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa75000e9f0 tx=0x7fa75000eeb0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 shutdown_connections 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa7440a2150 0x7fa7440a4610 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa75c101d20 0x7fa75c19e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa75c101360 0x7fa75c1978c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 --2- 172.21.15.3:0/3837863136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa75c100170 0x7fa75c197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 >> 172.21.15.3:0/3837863136 conn(0x7fa75c0fb6c0 msgr2=0x7fa75c0fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 shutdown_connections 2024-02-09T22:49:14.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:14.884+0000 7fa7644b5640 1 -- 172.21.15.3:0/3837863136 wait complete. 2024-02-09T22:49:14.898 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2024-02-09T22:49:14.537965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596828,"num_objects":67,"num_object_clones":0,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":313,"num_write_kb":2980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":253,"ondisk_log_size":253,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":270,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353736,"kb_used_data":12856,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588088,"statfs":{"total":1151940427776,"available":1151578202112,"internally_reserved":0,"allocated":13164544,"data_stored":7149212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6460,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001856"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804488+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:49:08.804488+0000","last_peered":"2024-02-09T22:49:08.804488+0000","last_clean":"2024-02-09T22:49:08.804488+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:49:08.804488+0000","last_undegraded":"2024-02-09T22:49:08.804488+0000","last_fullsized":"2024-02-09T22:49:08.804488+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803272+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:49:08.803272+0000","last_peered":"2024-02-09T22:49:08.803272+0000","last_clean":"2024-02-09T22:49:08.803272+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:49:08.803272+0000","last_undegraded":"2024-02-09T22:49:08.803272+0000","last_fullsized":"2024-02-09T22:49:08.803272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803108+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:49:08.803108+0000","last_peered":"2024-02-09T22:49:08.803108+0000","last_clean":"2024-02-09T22:49:08.803108+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:49:08.803108+0000","last_undegraded":"2024-02-09T22:49:08.803108+0000","last_fullsized":"2024-02-09T22:49:08.803108+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804242+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:49:08.804242+0000","last_peered":"2024-02-09T22:49:08.804242+0000","last_clean":"2024-02-09T22:49:08.804242+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:49:08.804242+0000","last_undegraded":"2024-02-09T22:49:08.804242+0000","last_fullsized":"2024-02-09T22:49:08.804242+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:50.286530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302467+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:49:08.302467+0000","last_peered":"2024-02-09T22:49:08.302467+0000","last_clean":"2024-02-09T22:49:08.302467+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:49:08.302467+0000","last_undegraded":"2024-02-09T22:49:08.302467+0000","last_fullsized":"2024-02-09T22:49:08.302467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804771+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:49:08.804771+0000","last_peered":"2024-02-09T22:49:08.804771+0000","last_clean":"2024-02-09T22:49:08.804771+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:49:08.804771+0000","last_undegraded":"2024-02-09T22:49:08.804771+0000","last_fullsized":"2024-02-09T22:49:08.804771+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805462+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:49:08.805462+0000","last_peered":"2024-02-09T22:49:08.805462+0000","last_clean":"2024-02-09T22:49:08.805462+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:49:08.805462+0000","last_undegraded":"2024-02-09T22:49:08.805462+0000","last_fullsized":"2024-02-09T22:49:08.805462+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248368+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:49:08.248368+0000","last_peered":"2024-02-09T22:49:08.248368+0000","last_clean":"2024-02-09T22:49:08.248368+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:49:08.248368+0000","last_undegraded":"2024-02-09T22:49:08.248368+0000","last_fullsized":"2024-02-09T22:49:08.248368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250252+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:49:08.250252+0000","last_peered":"2024-02-09T22:49:08.250252+0000","last_clean":"2024-02-09T22:49:08.250252+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:49:08.250252+0000","last_undegraded":"2024-02-09T22:49:08.250252+0000","last_fullsized":"2024-02-09T22:49:08.250252+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805394+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:49:08.805394+0000","last_peered":"2024-02-09T22:49:08.805394+0000","last_clean":"2024-02-09T22:49:08.805394+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:49:08.805394+0000","last_undegraded":"2024-02-09T22:49:08.805394+0000","last_fullsized":"2024-02-09T22:49:08.805394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805565+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:49:08.805565+0000","last_peered":"2024-02-09T22:49:08.805565+0000","last_clean":"2024-02-09T22:49:08.805565+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:49:08.805565+0000","last_undegraded":"2024-02-09T22:49:08.805565+0000","last_fullsized":"2024-02-09T22:49:08.805565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804123+0000","last_change":"2024-02-09T22:49:07.268535+0000","last_active":"2024-02-09T22:49:08.804123+0000","last_peered":"2024-02-09T22:49:08.804123+0000","last_clean":"2024-02-09T22:49:08.804123+0000","last_became_active":"2024-02-09T22:49:07.267967+0000","last_became_peered":"2024-02-09T22:49:07.267967+0000","last_unstale":"2024-02-09T22:49:08.804123+0000","last_undegraded":"2024-02-09T22:49:08.804123+0000","last_fullsized":"2024-02-09T22:49:08.804123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:33:14.673189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804141+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:49:08.804141+0000","last_peered":"2024-02-09T22:49:08.804141+0000","last_clean":"2024-02-09T22:49:08.804141+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:49:08.804141+0000","last_undegraded":"2024-02-09T22:49:08.804141+0000","last_fullsized":"2024-02-09T22:49:08.804141+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803667+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:49:08.803667+0000","last_peered":"2024-02-09T22:49:08.803667+0000","last_clean":"2024-02-09T22:49:08.803667+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:49:08.803667+0000","last_undegraded":"2024-02-09T22:49:08.803667+0000","last_fullsized":"2024-02-09T22:49:08.803667+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.802965+0000","last_change":"2024-02-09T22:49:07.269581+0000","last_active":"2024-02-09T22:49:08.802965+0000","last_peered":"2024-02-09T22:49:08.802965+0000","last_clean":"2024-02-09T22:49:08.802965+0000","last_became_active":"2024-02-09T22:49:07.269228+0000","last_became_peered":"2024-02-09T22:49:07.269228+0000","last_unstale":"2024-02-09T22:49:08.802965+0000","last_undegraded":"2024-02-09T22:49:08.802965+0000","last_fullsized":"2024-02-09T22:49:08.802965+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:07:20.365278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.249927+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:49:08.249927+0000","last_peered":"2024-02-09T22:49:08.249927+0000","last_clean":"2024-02-09T22:49:08.249927+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:49:08.249927+0000","last_undegraded":"2024-02-09T22:49:08.249927+0000","last_fullsized":"2024-02-09T22:49:08.249927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804535+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:49:08.804535+0000","last_peered":"2024-02-09T22:49:08.804535+0000","last_clean":"2024-02-09T22:49:08.804535+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:49:08.804535+0000","last_undegraded":"2024-02-09T22:49:08.804535+0000","last_fullsized":"2024-02-09T22:49:08.804535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804230+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:49:08.804230+0000","last_peered":"2024-02-09T22:49:08.804230+0000","last_clean":"2024-02-09T22:49:08.804230+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:49:08.804230+0000","last_undegraded":"2024-02-09T22:49:08.804230+0000","last_fullsized":"2024-02-09T22:49:08.804230+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804574+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:49:08.804574+0000","last_peered":"2024-02-09T22:49:08.804574+0000","last_clean":"2024-02-09T22:49:08.804574+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:49:08.804574+0000","last_undegraded":"2024-02-09T22:49:08.804574+0000","last_fullsized":"2024-02-09T22:49:08.804574+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:39.090835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804351+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:49:08.804351+0000","last_peered":"2024-02-09T22:49:08.804351+0000","last_clean":"2024-02-09T22:49:08.804351+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:49:08.804351+0000","last_undegraded":"2024-02-09T22:49:08.804351+0000","last_fullsized":"2024-02-09T22:49:08.804351+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248581+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:49:08.248581+0000","last_peered":"2024-02-09T22:49:08.248581+0000","last_clean":"2024-02-09T22:49:08.248581+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:49:08.248581+0000","last_undegraded":"2024-02-09T22:49:08.248581+0000","last_fullsized":"2024-02-09T22:49:08.248581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248893+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:49:08.248893+0000","last_peered":"2024-02-09T22:49:08.248893+0000","last_clean":"2024-02-09T22:49:08.248893+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:49:08.248893+0000","last_undegraded":"2024-02-09T22:49:08.248893+0000","last_fullsized":"2024-02-09T22:49:08.248893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803726+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:49:08.803726+0000","last_peered":"2024-02-09T22:49:08.803726+0000","last_clean":"2024-02-09T22:49:08.803726+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:49:08.803726+0000","last_undegraded":"2024-02-09T22:49:08.803726+0000","last_fullsized":"2024-02-09T22:49:08.803726+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302792+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:49:08.302792+0000","last_peered":"2024-02-09T22:49:08.302792+0000","last_clean":"2024-02-09T22:49:08.302792+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:49:08.302792+0000","last_undegraded":"2024-02-09T22:49:08.302792+0000","last_fullsized":"2024-02-09T22:49:08.302792+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250082+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:49:08.250082+0000","last_peered":"2024-02-09T22:49:08.250082+0000","last_clean":"2024-02-09T22:49:08.250082+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:49:08.250082+0000","last_undegraded":"2024-02-09T22:49:08.250082+0000","last_fullsized":"2024-02-09T22:49:08.250082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302621+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:49:08.302621+0000","last_peered":"2024-02-09T22:49:08.302621+0000","last_clean":"2024-02-09T22:49:08.302621+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:49:08.302621+0000","last_undegraded":"2024-02-09T22:49:08.302621+0000","last_fullsized":"2024-02-09T22:49:08.302621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.254360+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:49:08.254360+0000","last_peered":"2024-02-09T22:49:08.254360+0000","last_clean":"2024-02-09T22:49:08.254360+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:49:08.254360+0000","last_undegraded":"2024-02-09T22:49:08.254360+0000","last_fullsized":"2024-02-09T22:49:08.254360+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098316+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:49:09.098316+0000","last_peered":"2024-02-09T22:49:09.098316+0000","last_clean":"2024-02-09T22:49:09.098316+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:49:09.098316+0000","last_undegraded":"2024-02-09T22:49:09.098316+0000","last_fullsized":"2024-02-09T22:49:09.098316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302429+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:49:08.302429+0000","last_peered":"2024-02-09T22:49:08.302429+0000","last_clean":"2024-02-09T22:49:08.302429+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:49:08.302429+0000","last_undegraded":"2024-02-09T22:49:08.302429+0000","last_fullsized":"2024-02-09T22:49:08.302429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.107227+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:49:10.107227+0000","last_peered":"2024-02-09T22:49:10.107227+0000","last_clean":"2024-02-09T22:49:10.107227+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:49:10.107227+0000","last_undegraded":"2024-02-09T22:49:10.107227+0000","last_fullsized":"2024-02-09T22:49:10.107227+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.254248+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:49:08.254248+0000","last_peered":"2024-02-09T22:49:08.254248+0000","last_clean":"2024-02-09T22:49:08.254248+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:49:08.254248+0000","last_undegraded":"2024-02-09T22:49:08.254248+0000","last_fullsized":"2024-02-09T22:49:08.254248+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.109383+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:49:10.109383+0000","last_peered":"2024-02-09T22:49:10.109383+0000","last_clean":"2024-02-09T22:49:10.109383+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:49:10.109383+0000","last_undegraded":"2024-02-09T22:49:10.109383+0000","last_fullsized":"2024-02-09T22:49:10.109383+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248444+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:49:08.248444+0000","last_peered":"2024-02-09T22:49:08.248444+0000","last_clean":"2024-02-09T22:49:08.248444+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:49:08.248444+0000","last_undegraded":"2024-02-09T22:49:08.248444+0000","last_fullsized":"2024-02-09T22:49:08.248444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098282+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:49:09.098282+0000","last_peered":"2024-02-09T22:49:09.098282+0000","last_clean":"2024-02-09T22:49:09.098282+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:49:09.098282+0000","last_undegraded":"2024-02-09T22:49:09.098282+0000","last_fullsized":"2024-02-09T22:49:09.098282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803690+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:49:08.803690+0000","last_peered":"2024-02-09T22:49:08.803690+0000","last_clean":"2024-02-09T22:49:08.803690+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:49:08.803690+0000","last_undegraded":"2024-02-09T22:49:08.803690+0000","last_fullsized":"2024-02-09T22:49:08.803690+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098225+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:49:09.098225+0000","last_peered":"2024-02-09T22:49:09.098225+0000","last_clean":"2024-02-09T22:49:09.098225+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:49:09.098225+0000","last_undegraded":"2024-02-09T22:49:09.098225+0000","last_fullsized":"2024-02-09T22:49:09.098225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250212+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:49:08.250212+0000","last_peered":"2024-02-09T22:49:08.250212+0000","last_clean":"2024-02-09T22:49:08.250212+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:49:08.250212+0000","last_undegraded":"2024-02-09T22:49:08.250212+0000","last_fullsized":"2024-02-09T22:49:08.250212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.099267+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:49:09.099267+0000","last_peered":"2024-02-09T22:49:09.099267+0000","last_clean":"2024-02-09T22:49:09.099267+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:49:09.099267+0000","last_undegraded":"2024-02-09T22:49:09.099267+0000","last_fullsized":"2024-02-09T22:49:09.099267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803874+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:49:08.803874+0000","last_peered":"2024-02-09T22:49:08.803874+0000","last_clean":"2024-02-09T22:49:08.803874+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:49:08.803874+0000","last_undegraded":"2024-02-09T22:49:08.803874+0000","last_fullsized":"2024-02-09T22:49:08.803874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098019+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:49:09.098019+0000","last_peered":"2024-02-09T22:49:09.098019+0000","last_clean":"2024-02-09T22:49:09.098019+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:49:09.098019+0000","last_undegraded":"2024-02-09T22:49:09.098019+0000","last_fullsized":"2024-02-09T22:49:09.098019+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.250260+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:49:08.250260+0000","last_peered":"2024-02-09T22:49:08.250260+0000","last_clean":"2024-02-09T22:49:08.250260+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:49:08.250260+0000","last_undegraded":"2024-02-09T22:49:08.250260+0000","last_fullsized":"2024-02-09T22:49:08.250260+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":23,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098152+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:49:09.098152+0000","last_peered":"2024-02-09T22:49:09.098152+0000","last_clean":"2024-02-09T22:49:09.098152+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:49:09.098152+0000","last_undegraded":"2024-02-09T22:49:09.098152+0000","last_fullsized":"2024-02-09T22:49:09.098152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302677+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:49:08.302677+0000","last_peered":"2024-02-09T22:49:08.302677+0000","last_clean":"2024-02-09T22:49:08.302677+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:49:08.302677+0000","last_undegraded":"2024-02-09T22:49:08.302677+0000","last_fullsized":"2024-02-09T22:49:08.302677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:12.123609+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:49:12.123609+0000","last_peered":"2024-02-09T22:49:12.123609+0000","last_clean":"2024-02-09T22:49:12.123609+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:49:12.123609+0000","last_undegraded":"2024-02-09T22:49:12.123609+0000","last_fullsized":"2024-02-09T22:49:12.123609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803864+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:49:08.803864+0000","last_peered":"2024-02-09T22:49:08.803864+0000","last_clean":"2024-02-09T22:49:08.803864+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:49:08.803864+0000","last_undegraded":"2024-02-09T22:49:08.803864+0000","last_fullsized":"2024-02-09T22:49:08.803864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098260+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:49:09.098260+0000","last_peered":"2024-02-09T22:49:09.098260+0000","last_clean":"2024-02-09T22:49:09.098260+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:49:09.098260+0000","last_undegraded":"2024-02-09T22:49:09.098260+0000","last_fullsized":"2024-02-09T22:49:09.098260+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805731+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:49:08.805731+0000","last_peered":"2024-02-09T22:49:08.805731+0000","last_clean":"2024-02-09T22:49:08.805731+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:49:08.805731+0000","last_undegraded":"2024-02-09T22:49:08.805731+0000","last_fullsized":"2024-02-09T22:49:08.805731+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:11.116098+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:49:11.116098+0000","last_peered":"2024-02-09T22:49:11.116098+0000","last_clean":"2024-02-09T22:49:11.116098+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:49:11.116098+0000","last_undegraded":"2024-02-09T22:49:11.116098+0000","last_fullsized":"2024-02-09T22:49:11.116098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804017+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:49:08.804017+0000","last_peered":"2024-02-09T22:49:08.804017+0000","last_clean":"2024-02-09T22:49:08.804017+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:49:08.804017+0000","last_undegraded":"2024-02-09T22:49:08.804017+0000","last_fullsized":"2024-02-09T22:49:08.804017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.096177+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:49:09.096177+0000","last_peered":"2024-02-09T22:49:09.096177+0000","last_clean":"2024-02-09T22:49:09.096177+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:49:09.096177+0000","last_undegraded":"2024-02-09T22:49:09.096177+0000","last_fullsized":"2024-02-09T22:49:09.096177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.253927+0000","last_change":"2024-02-09T22:49:07.259557+0000","last_active":"2024-02-09T22:49:08.253927+0000","last_peered":"2024-02-09T22:49:08.253927+0000","last_clean":"2024-02-09T22:49:08.253927+0000","last_became_active":"2024-02-09T22:49:07.259234+0000","last_became_peered":"2024-02-09T22:49:07.259234+0000","last_unstale":"2024-02-09T22:49:08.253927+0000","last_undegraded":"2024-02-09T22:49:08.253927+0000","last_fullsized":"2024-02-09T22:49:08.253927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:00:37.372611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302747+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:49:08.302747+0000","last_peered":"2024-02-09T22:49:08.302747+0000","last_clean":"2024-02-09T22:49:08.302747+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:49:08.302747+0000","last_undegraded":"2024-02-09T22:49:08.302747+0000","last_fullsized":"2024-02-09T22:49:08.302747+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804178+0000","last_change":"2024-02-09T22:49:07.270133+0000","last_active":"2024-02-09T22:49:08.804178+0000","last_peered":"2024-02-09T22:49:08.804178+0000","last_clean":"2024-02-09T22:49:08.804178+0000","last_became_active":"2024-02-09T22:49:07.269741+0000","last_became_peered":"2024-02-09T22:49:07.269741+0000","last_unstale":"2024-02-09T22:49:08.804178+0000","last_undegraded":"2024-02-09T22:49:08.804178+0000","last_fullsized":"2024-02-09T22:49:08.804178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:41:21.489866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804059+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:49:08.804059+0000","last_peered":"2024-02-09T22:49:08.804059+0000","last_clean":"2024-02-09T22:49:08.804059+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:49:08.804059+0000","last_undegraded":"2024-02-09T22:49:08.804059+0000","last_fullsized":"2024-02-09T22:49:08.804059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804709+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:49:08.804709+0000","last_peered":"2024-02-09T22:49:08.804709+0000","last_clean":"2024-02-09T22:49:08.804709+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:49:08.804709+0000","last_undegraded":"2024-02-09T22:49:08.804709+0000","last_fullsized":"2024-02-09T22:49:08.804709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805548+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:49:08.805548+0000","last_peered":"2024-02-09T22:49:08.805548+0000","last_clean":"2024-02-09T22:49:08.805548+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:49:08.805548+0000","last_undegraded":"2024-02-09T22:49:08.805548+0000","last_fullsized":"2024-02-09T22:49:08.805548+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804914+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:49:08.804914+0000","last_peered":"2024-02-09T22:49:08.804914+0000","last_clean":"2024-02-09T22:49:08.804914+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:49:08.804914+0000","last_undegraded":"2024-02-09T22:49:08.804914+0000","last_fullsized":"2024-02-09T22:49:08.804914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.253873+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:49:08.253873+0000","last_peered":"2024-02-09T22:49:08.253873+0000","last_clean":"2024-02-09T22:49:08.253873+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:49:08.253873+0000","last_undegraded":"2024-02-09T22:49:08.253873+0000","last_fullsized":"2024-02-09T22:49:08.253873+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.109078+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:49:10.109078+0000","last_peered":"2024-02-09T22:49:10.109078+0000","last_clean":"2024-02-09T22:49:10.109078+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:49:10.109078+0000","last_undegraded":"2024-02-09T22:49:10.109078+0000","last_fullsized":"2024-02-09T22:49:10.109078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248615+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:49:08.248615+0000","last_peered":"2024-02-09T22:49:08.248615+0000","last_clean":"2024-02-09T22:49:08.248615+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:49:08.248615+0000","last_undegraded":"2024-02-09T22:49:08.248615+0000","last_fullsized":"2024-02-09T22:49:08.248615+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:11.117099+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:49:11.117099+0000","last_peered":"2024-02-09T22:49:11.117099+0000","last_clean":"2024-02-09T22:49:11.117099+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:49:11.117099+0000","last_undegraded":"2024-02-09T22:49:11.117099+0000","last_fullsized":"2024-02-09T22:49:11.117099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":349,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804025+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:49:08.804025+0000","last_peered":"2024-02-09T22:49:08.804025+0000","last_clean":"2024-02-09T22:49:08.804025+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:49:08.804025+0000","last_undegraded":"2024-02-09T22:49:08.804025+0000","last_fullsized":"2024-02-09T22:49:08.804025+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803070+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:49:08.803070+0000","last_peered":"2024-02-09T22:49:08.803070+0000","last_clean":"2024-02-09T22:49:08.803070+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:49:08.803070+0000","last_undegraded":"2024-02-09T22:49:08.803070+0000","last_fullsized":"2024-02-09T22:49:08.803070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:10.108037+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:49:10.108037+0000","last_peered":"2024-02-09T22:49:10.108037+0000","last_clean":"2024-02-09T22:49:10.108037+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:49:10.108037+0000","last_undegraded":"2024-02-09T22:49:10.108037+0000","last_fullsized":"2024-02-09T22:49:10.108037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.095925+0000","last_change":"2024-02-09T22:49:06.259702+0000","last_active":"2024-02-09T22:49:09.095925+0000","last_peered":"2024-02-09T22:49:09.095925+0000","last_clean":"2024-02-09T22:49:09.095925+0000","last_became_active":"2024-02-09T22:49:06.259352+0000","last_became_peered":"2024-02-09T22:49:06.259352+0000","last_unstale":"2024-02-09T22:49:09.095925+0000","last_undegraded":"2024-02-09T22:49:09.095925+0000","last_fullsized":"2024-02-09T22:49:09.095925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:01:11.212161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803611+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:49:08.803611+0000","last_peered":"2024-02-09T22:49:08.803611+0000","last_clean":"2024-02-09T22:49:08.803611+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:49:08.803611+0000","last_undegraded":"2024-02-09T22:49:08.803611+0000","last_fullsized":"2024-02-09T22:49:08.803611+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.543824+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:49:09.543824+0000","last_peered":"2024-02-09T22:49:09.543824+0000","last_clean":"2024-02-09T22:49:09.543824+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:49:09.543824+0000","last_undegraded":"2024-02-09T22:49:09.543824+0000","last_fullsized":"2024-02-09T22:49:09.543824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248940+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:49:08.248940+0000","last_peered":"2024-02-09T22:49:08.248940+0000","last_clean":"2024-02-09T22:49:08.248940+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:49:08.248940+0000","last_undegraded":"2024-02-09T22:49:08.248940+0000","last_fullsized":"2024-02-09T22:49:08.248940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:12.121823+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:49:12.121823+0000","last_peered":"2024-02-09T22:49:12.121823+0000","last_clean":"2024-02-09T22:49:12.121823+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:49:12.121823+0000","last_undegraded":"2024-02-09T22:49:12.121823+0000","last_fullsized":"2024-02-09T22:49:12.121823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803475+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:49:08.803475+0000","last_peered":"2024-02-09T22:49:08.803475+0000","last_clean":"2024-02-09T22:49:08.803475+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:49:08.803475+0000","last_undegraded":"2024-02-09T22:49:08.803475+0000","last_fullsized":"2024-02-09T22:49:08.803475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.096928+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:49:09.096928+0000","last_peered":"2024-02-09T22:49:09.096928+0000","last_clean":"2024-02-09T22:49:09.096928+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:49:09.096928+0000","last_undegraded":"2024-02-09T22:49:09.096928+0000","last_fullsized":"2024-02-09T22:49:09.096928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.302396+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:49:08.302396+0000","last_peered":"2024-02-09T22:49:08.302396+0000","last_clean":"2024-02-09T22:49:08.302396+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:49:08.302396+0000","last_undegraded":"2024-02-09T22:49:08.302396+0000","last_fullsized":"2024-02-09T22:49:08.302396+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:11.117105+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:49:11.117105+0000","last_peered":"2024-02-09T22:49:11.117105+0000","last_clean":"2024-02-09T22:49:11.117105+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:49:11.117105+0000","last_undegraded":"2024-02-09T22:49:11.117105+0000","last_fullsized":"2024-02-09T22:49:11.117105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803712+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:49:08.803712+0000","last_peered":"2024-02-09T22:49:08.803712+0000","last_clean":"2024-02-09T22:49:08.803712+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:49:08.803712+0000","last_undegraded":"2024-02-09T22:49:08.803712+0000","last_fullsized":"2024-02-09T22:49:08.803712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:11.117643+0000","last_change":"2024-02-09T22:49:06.253697+0000","last_active":"2024-02-09T22:49:11.117643+0000","last_peered":"2024-02-09T22:49:11.117643+0000","last_clean":"2024-02-09T22:49:11.117643+0000","last_became_active":"2024-02-09T22:49:06.253036+0000","last_became_peered":"2024-02-09T22:49:06.253036+0000","last_unstale":"2024-02-09T22:49:11.117643+0000","last_undegraded":"2024-02-09T22:49:11.117643+0000","last_fullsized":"2024-02-09T22:49:11.117643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:33:01.928416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248775+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:49:08.248775+0000","last_peered":"2024-02-09T22:49:08.248775+0000","last_clean":"2024-02-09T22:49:08.248775+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:49:08.248775+0000","last_undegraded":"2024-02-09T22:49:08.248775+0000","last_fullsized":"2024-02-09T22:49:08.248775+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:09.098673+0000","last_change":"2024-02-09T22:49:06.252438+0000","last_active":"2024-02-09T22:49:09.098673+0000","last_peered":"2024-02-09T22:49:09.098673+0000","last_clean":"2024-02-09T22:49:09.098673+0000","last_became_active":"2024-02-09T22:49:06.252111+0000","last_became_peered":"2024-02-09T22:49:06.252111+0000","last_unstale":"2024-02-09T22:49:09.098673+0000","last_undegraded":"2024-02-09T22:49:09.098673+0000","last_fullsized":"2024-02-09T22:49:09.098673+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:04:16.992194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805720+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:49:08.805720+0000","last_peered":"2024-02-09T22:49:08.805720+0000","last_clean":"2024-02-09T22:49:08.805720+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:49:08.805720+0000","last_undegraded":"2024-02-09T22:49:08.805720+0000","last_fullsized":"2024-02-09T22:49:08.805720+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:11.116353+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:49:11.116353+0000","last_peered":"2024-02-09T22:49:11.116353+0000","last_clean":"2024-02-09T22:49:11.116353+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:49:11.116353+0000","last_undegraded":"2024-02-09T22:49:11.116353+0000","last_fullsized":"2024-02-09T22:49:11.116353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805516+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:49:08.805516+0000","last_peered":"2024-02-09T22:49:08.805516+0000","last_clean":"2024-02-09T22:49:08.805516+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:49:08.805516+0000","last_undegraded":"2024-02-09T22:49:08.805516+0000","last_fullsized":"2024-02-09T22:49:08.805516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.254303+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:49:08.254303+0000","last_peered":"2024-02-09T22:49:08.254303+0000","last_clean":"2024-02-09T22:49:08.254303+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:49:08.254303+0000","last_undegraded":"2024-02-09T22:49:08.254303+0000","last_fullsized":"2024-02-09T22:49:08.254303+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805499+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:49:08.805499+0000","last_peered":"2024-02-09T22:49:08.805499+0000","last_clean":"2024-02-09T22:49:08.805499+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:49:08.805499+0000","last_undegraded":"2024-02-09T22:49:08.805499+0000","last_fullsized":"2024-02-09T22:49:08.805499+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803684+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:49:08.803684+0000","last_peered":"2024-02-09T22:49:08.803684+0000","last_clean":"2024-02-09T22:49:08.803684+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:49:08.803684+0000","last_undegraded":"2024-02-09T22:49:08.803684+0000","last_fullsized":"2024-02-09T22:49:08.803684+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.805689+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:49:08.805689+0000","last_peered":"2024-02-09T22:49:08.805689+0000","last_clean":"2024-02-09T22:49:08.805689+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:49:08.805689+0000","last_undegraded":"2024-02-09T22:49:08.805689+0000","last_fullsized":"2024-02-09T22:49:08.805689+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803789+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:49:08.803789+0000","last_peered":"2024-02-09T22:49:08.803789+0000","last_clean":"2024-02-09T22:49:08.803789+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:49:08.803789+0000","last_undegraded":"2024-02-09T22:49:08.803789+0000","last_fullsized":"2024-02-09T22:49:08.803789+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804318+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:49:08.804318+0000","last_peered":"2024-02-09T22:49:08.804318+0000","last_clean":"2024-02-09T22:49:08.804318+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:49:08.804318+0000","last_undegraded":"2024-02-09T22:49:08.804318+0000","last_fullsized":"2024-02-09T22:49:08.804318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:24:49.099644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804058+0000","last_change":"2024-02-09T22:49:07.263133+0000","last_active":"2024-02-09T22:49:08.804058+0000","last_peered":"2024-02-09T22:49:08.804058+0000","last_clean":"2024-02-09T22:49:08.804058+0000","last_became_active":"2024-02-09T22:49:07.260838+0000","last_became_peered":"2024-02-09T22:49:07.260838+0000","last_unstale":"2024-02-09T22:49:08.804058+0000","last_undegraded":"2024-02-09T22:49:08.804058+0000","last_fullsized":"2024-02-09T22:49:08.804058+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:50:52.748901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803444+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:49:08.803444+0000","last_peered":"2024-02-09T22:49:08.803444+0000","last_clean":"2024-02-09T22:49:08.803444+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:49:08.803444+0000","last_undegraded":"2024-02-09T22:49:08.803444+0000","last_fullsized":"2024-02-09T22:49:08.803444+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803443+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:49:08.803443+0000","last_peered":"2024-02-09T22:49:08.803443+0000","last_clean":"2024-02-09T22:49:08.803443+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:49:08.803443+0000","last_undegraded":"2024-02-09T22:49:08.803443+0000","last_fullsized":"2024-02-09T22:49:08.803443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804206+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:49:08.804206+0000","last_peered":"2024-02-09T22:49:08.804206+0000","last_clean":"2024-02-09T22:49:08.804206+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:49:08.804206+0000","last_undegraded":"2024-02-09T22:49:08.804206+0000","last_fullsized":"2024-02-09T22:49:08.804206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804745+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:49:08.804745+0000","last_peered":"2024-02-09T22:49:08.804745+0000","last_clean":"2024-02-09T22:49:08.804745+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:49:08.804745+0000","last_undegraded":"2024-02-09T22:49:08.804745+0000","last_fullsized":"2024-02-09T22:49:08.804745+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:57:01.921327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248841+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:49:08.248841+0000","last_peered":"2024-02-09T22:49:08.248841+0000","last_clean":"2024-02-09T22:49:08.248841+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:49:08.248841+0000","last_undegraded":"2024-02-09T22:49:08.248841+0000","last_fullsized":"2024-02-09T22:49:08.248841+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.253844+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:49:08.253844+0000","last_peered":"2024-02-09T22:49:08.253844+0000","last_clean":"2024-02-09T22:49:08.253844+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:49:08.253844+0000","last_undegraded":"2024-02-09T22:49:08.253844+0000","last_fullsized":"2024-02-09T22:49:08.253844+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.248680+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:49:08.248680+0000","last_peered":"2024-02-09T22:49:08.248680+0000","last_clean":"2024-02-09T22:49:08.248680+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:49:08.248680+0000","last_undegraded":"2024-02-09T22:49:08.248680+0000","last_fullsized":"2024-02-09T22:49:08.248680+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.249906+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:49:08.249906+0000","last_peered":"2024-02-09T22:49:08.249906+0000","last_clean":"2024-02-09T22:49:08.249906+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:49:08.249906+0000","last_undegraded":"2024-02-09T22:49:08.249906+0000","last_fullsized":"2024-02-09T22:49:08.249906+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.804122+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:49:08.804122+0000","last_peered":"2024-02-09T22:49:08.804122+0000","last_clean":"2024-02-09T22:49:08.804122+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:49:08.804122+0000","last_undegraded":"2024-02-09T22:49:08.804122+0000","last_fullsized":"2024-02-09T22:49:08.804122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-02-09T22:49:08.803875+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:49:08.803875+0000","last_peered":"2024-02-09T22:49:08.803875+0000","last_clean":"2024-02-09T22:49:08.803875+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:49:08.803875+0000","last_undegraded":"2024-02-09T22:49:08.803875+0000","last_fullsized":"2024-02-09T22:49:08.803875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6460,"num_objects":65,"num_object_clones":0,"num_object_copies":195,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":132,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":41,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":299008,"data_stored":37390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":446368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71399999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40699999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.498}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52600000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75800000000000001}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67400000000000004}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70399999999999996}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73899999999999999}]}]},{"osd":10,"up_from":73,"seq":313532612626,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32708,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712444,"statfs":{"total":95995035648,"available":95961542656,"internally_reserved":0,"allocated":1564672,"data_stored":1038734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":1,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.747},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.92600000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56899999999999995}]},{"osd":4,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.66600000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.006,"5min":1.006,"15min":1.006},"last":0.95399999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.02,"5min":1.02,"15min":1.02},"last":0.879},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77400000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.155,"5min":1.155,"15min":1.155},"last":0.84999999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97699999999999998}]}]},{"osd":9,"up_from":67,"seq":287762808855,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":445960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"last":0.54900000000000004}]},{"osd":1,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.60699999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.63,"5min":0.63,"15min":0.63},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.65400000000000003}]},{"osd":4,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.86199999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":1.0960000000000001,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.80100000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.502,"5min":0.502,"15min":0.502},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.71999999999999997}]},{"osd":7,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.177,"5min":1.177,"15min":1.177},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:48:25 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.81999999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.754,"15min":0.754},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.879},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.93000000000000005}]}]},{"osd":8,"up_from":59,"seq":253403070492,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":933888,"data_stored":445780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54300000000000004,"15min":0.54700000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.752,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.70199999999999996}]},{"osd":1,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.57999999999999996,"15min":0.58199999999999996},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.77700000000000002,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.59899999999999998,"15min":0.60099999999999998},"min":{"1min":0.38700000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.75600000000000001,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.89300000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41499999999999998,"15min":0.41399999999999998},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":1.093},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.45100000000000001,"15min":0.45500000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.67700000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.80900000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.53500000000000003,"15min":0.53300000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.55800000000000005,"15min":0.55500000000000005},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.60599999999999998,"15min":0.60999999999999999},"min":{"1min":0.42799999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80000000000000004,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.97299999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.61499999999999999,"15min":0.61799999999999999},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.83199999999999996,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":1.0529999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.53000000000000003,"15min":0.53700000000000003},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.54500000000000004,"15min":0.55000000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85299999999999998,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":1.073}]},{"osd":6,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.66100000000000003,"15min":0.66500000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":1.169}]},{"osd":7,"last update":"Fri Feb 9 22:49:08 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.68600000000000005,"15min":0.69599999999999995},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.876,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":1.2549999999999999},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.68000000000000005,"15min":0.68100000000000005},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.93899999999999995,"5min":1.03,"15min":1.03},"last":1.3160000000000001}]},{"osd":9,"last update":"Fri Feb 9 22:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":1.2330000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":1.1919999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:48:53 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":1.371},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":1.3440000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1259999999999999}]}]},{"osd":7,"up_from":51,"seq":219043332128,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28524,"kb_used_data":1496,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716628,"statfs":{"total":95995035648,"available":95965827072,"internally_reserved":0,"allocated":1531904,"data_stored":1036748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.372},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.68500000000000005}]},{"osd":2,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.38300000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.64900000000000002,"15min":0.64600000000000002},"min":{"1min":0.47499999999999998,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.75900000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.63300000000000001,"15min":0.627},"min":{"1min":0.42899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.111,"5min":1.111,"15min":1.111},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.63500000000000001,"15min":0.63700000000000001},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.82199999999999995,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.624,"15min":0.626},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.53600000000000003,"15min":0.53400000000000003},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79000000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.41299999999999998}]},{"osd":5,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.65100000000000002,"15min":0.65300000000000002},"min":{"1min":0.49199999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70399999999999996,"15min":0.69999999999999996},"min":{"1min":0.435,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.71199999999999997}]},{"osd":6,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.72899999999999998,"15min":0.72199999999999998},"min":{"1min":0.504,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68400000000000005,"15min":0.68300000000000005},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.61799999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.78300000000000003,"5min":0.73099999999999998,"15min":0.72199999999999998},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.70499999999999996,"15min":0.69499999999999995},"min":{"1min":0.54700000000000004,"5min":0.498,"15min":0.498},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.57399999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0649999999999999,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.66700000000000004}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]}]},{"osd":6,"up_from":46,"seq":197568495655,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":929792,"data_stored":445266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.35099999999999998,"15min":0.33800000000000002},"min":{"1min":0.26800000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.308},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.40400000000000003,"15min":0.39500000000000002},"min":{"1min":0.29099999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.29099999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.53700000000000003,"15min":0.52900000000000003},"min":{"1min":0.34999999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.54700000000000004,"15min":0.53900000000000003},"min":{"1min":0.42099999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.46899999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.55500000000000005,"15min":0.53900000000000003},"min":{"1min":0.45600000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.5}]},{"osd":3,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.41899999999999998,"15min":0.40400000000000003},"min":{"1min":0.31900000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.46999999999999997,"15min":0.46300000000000002},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.36099999999999999}]},{"osd":4,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.58399999999999996,"15min":0.57899999999999996},"min":{"1min":0.379,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.499,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.42899999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.59499999999999997,"15min":0.58199999999999996},"min":{"1min":0.49199999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.749,"5min":0.65400000000000003,"15min":0.63800000000000001},"min":{"1min":0.52800000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.56999999999999995}]},{"osd":7,"last update":"Fri Feb 9 22:48:41 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.63500000000000001,"15min":0.621},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64000000000000001,"15min":0.63200000000000001},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47699999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.67400000000000004,"15min":0.67600000000000005},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.78500000000000003,"5min":0.876,"15min":0.876},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.70599999999999996,"15min":0.69999999999999996},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.53500000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.40999999999999998}]},{"osd":10,"last update":"Fri Feb 9 22:48:54 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.77700000000000002},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.51900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574911,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":446788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.54400000000000004,"15min":0.51600000000000001},"min":{"1min":0.34000000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.94599999999999995,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.524,"5min":0.73499999999999999,"15min":0.58899999999999997},"min":{"1min":0.45800000000000002,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":18.152000000000001,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.629}]},{"osd":2,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":0.70299999999999996,"15min":0.57799999999999996},"min":{"1min":0.378,"5min":0.37,"15min":0.37},"max":{"1min":18.18,"5min":18.18,"15min":18.18},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.52500000000000002,"15min":0.49299999999999999},"min":{"1min":0.41899999999999998,"5min":0.372,"15min":0.372},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.92800000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:48:09 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.57799999999999996,"15min":0.55900000000000005},"min":{"1min":0.40300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.84199999999999997,"5min":1.206,"15min":1.206},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.55100000000000005,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.89900000000000002,"5min":1.139,"15min":1.139},"last":0.82999999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.45400000000000001,"15min":0.442},"min":{"1min":0.308,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.442,"15min":0.41599999999999998},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.747,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.66700000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.625,"15min":0.59099999999999997},"min":{"1min":0.53000000000000003,"5min":0.44,"15min":0.44},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.63400000000000001,"15min":0.60599999999999998},"min":{"1min":0.499,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.872,"5min":0.90100000000000002,"15min":0.90100000000000002},"last":1.038}]},{"osd":6,"last update":"Fri Feb 9 22:48:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67500000000000004,"5min":0.64300000000000002,"15min":0.63700000000000001},"min":{"1min":0.41399999999999998,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.69399999999999995,"15min":0.68799999999999994},"min":{"1min":0.45000000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"last":0.97399999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.45400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.875,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.878},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.66800000000000004,"15min":0.66000000000000003},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93999999999999995,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":1.024}]},{"osd":8,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.80300000000000005,"5min":1.5169999999999999,"15min":1.6359999999999999},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.0980000000000001,"5min":18.132000000000001,"15min":18.132000000000001},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.69099999999999995,"15min":0.68799999999999994},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.95399999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:48:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.996},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":1.0660000000000001}]},{"osd":10,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"min":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":1.099}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1619999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":446654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:48:15 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.52800000000000002,"15min":0.48199999999999998},"min":{"1min":0.33800000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.47599999999999998},"min":{"1min":0.35399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.81699999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.51700000000000002},"min":{"1min":0.41099999999999998,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.73099999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.53400000000000003,"15min":0.503},"min":{"1min":0.36199999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.78700000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.378}]},{"osd":3,"last update":"Fri Feb 9 22:48:00 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40100000000000002,"15min":0.38500000000000001},"min":{"1min":0.28899999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.58599999999999997,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.41599999999999998,"15min":0.40100000000000002},"min":{"1min":0.28299999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.25}]},{"osd":4,"last update":"Fri Feb 9 22:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.59699999999999998,"15min":0.58399999999999996},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.60999999999999999,"15min":0.59299999999999997},"min":{"1min":0.443,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56499999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:48:55 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59399999999999997,"15min":0.57999999999999996},"min":{"1min":0.42599999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.80700000000000005,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57499999999999996},"min":{"1min":0.32200000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.79700000000000004,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.308}]},{"osd":6,"last update":"Fri Feb 9 22:48:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.53400000000000003,"15min":0.52200000000000002},"min":{"1min":0.374,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.57699999999999996,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.33400000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.71999999999999997,"15min":0.72899999999999998},"min":{"1min":0.42199999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":1.052,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.504},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.46300000000000002,"5min":0.438,"15min":0.438},"max":{"1min":0.82999999999999996,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.36399999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":0.41499999999999998}]},{"osd":10,"last update":"Fri Feb 9 22:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.137},"last":0.49099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411386,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":468434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:44 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.46000000000000002,"15min":0.45000000000000001},"min":{"1min":0.22500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.59499999999999997,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48899999999999999,"15min":0.47099999999999997},"min":{"1min":0.28000000000000003,"5min":0.224,"15min":0.224},"max":{"1min":0.67700000000000005,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Feb 9 22:48:44 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.495,"15min":0.46700000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.51600000000000001,"15min":0.47399999999999998},"min":{"1min":0.31900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.79700000000000004,"5min":0.874,"15min":0.874},"last":0.748}]},{"osd":3,"last update":"Fri Feb 9 22:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.52600000000000002,"15min":0.47999999999999998},"min":{"1min":0.35299999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51000000000000001,"15min":0.48399999999999999},"min":{"1min":0.371,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.68200000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.504}]},{"osd":4,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.58799999999999997,"15min":0.57299999999999995},"min":{"1min":0.44400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.624,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.436,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.65200000000000002}]},{"osd":5,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.43099999999999999,"15min":0.41299999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.71299999999999997,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.44,"15min":0.436},"min":{"1min":0.28999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58799999999999997}]},{"osd":6,"last update":"Fri Feb 9 22:48:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.59299999999999997,"15min":0.58499999999999996},"min":{"1min":0.39000000000000001,"5min":0.37,"15min":0.37},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.57899999999999996,"15min":0.57599999999999996},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.72699999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.55400000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:49:06 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54600000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.84799999999999998,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.621}]},{"osd":9,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.76200000000000001}]},{"osd":10,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.83399999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.85899999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77700000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182453,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32104,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713048,"statfs":{"total":95995035648,"available":95962161152,"internally_reserved":0,"allocated":950272,"data_stored":445584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26600000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.50900000000000001,"5min":0.62,"15min":0.62},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.28999999999999998,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.59299999999999997,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.55900000000000005,"15min":0.52500000000000002},"min":{"1min":0.34699999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56699999999999995,"15min":0.52900000000000003},"min":{"1min":0.38400000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.753}]},{"osd":2,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.56799999999999995,"15min":0.55800000000000005},"min":{"1min":0.40400000000000003,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.79200000000000004,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.57099999999999995,"15min":0.54900000000000004},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.77600000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.998}]},{"osd":4,"last update":"Fri Feb 9 22:48:30 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.60399999999999998,"15min":0.57799999999999996},"min":{"1min":0.51600000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.61099999999999999,"15min":0.58699999999999997},"min":{"1min":0.505,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.64300000000000002}]},{"osd":5,"last update":"Fri Feb 9 22:48:47 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61399999999999999,"15min":0.622},"min":{"1min":0.42799999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.61299999999999999,"15min":0.61699999999999999},"min":{"1min":0.48199999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.85099999999999998,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.88300000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:49:11 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.33800000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.86799999999999999,"5min":0.873,"15min":0.873},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56799999999999995,"15min":0.55000000000000004},"min":{"1min":0.32500000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.97699999999999998,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.97699999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:48:38 2024","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.66100000000000003,"15min":0.65800000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.71299999999999997,"15min":0.69599999999999995},"min":{"1min":0.53100000000000003,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.0409999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:48:26 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":1.0229999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.26,"5min":1.26,"15min":1.26},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":0.999,"5min":0.999,"15min":0.999},"last":0.93999999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96399999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920817,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":445876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.39700000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.72299999999999998,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.41799999999999998,"15min":0.38900000000000001},"min":{"1min":0.26300000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57399999999999995,"15min":0.55800000000000005},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.64800000000000002,"5min":0.58499999999999996,"15min":0.56399999999999995},"min":{"1min":0.38700000000000001,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.747}]},{"osd":3,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.58699999999999997,"15min":0.56899999999999995},"min":{"1min":0.44900000000000001,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.56599999999999995,"15min":0.56200000000000006},"min":{"1min":0.436,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.61099999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.59799999999999998,"15min":0.57299999999999995},"min":{"1min":0.42799999999999999,"5min":0.38,"15min":0.38},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.61799999999999999,"15min":0.58399999999999996},"min":{"1min":0.46100000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71599999999999997}]},{"osd":6,"last update":"Fri Feb 9 22:48:18 2024","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.60299999999999998,"15min":0.59499999999999997},"min":{"1min":0.40699999999999997,"5min":0.38,"15min":0.38},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.64200000000000002,"15min":0.629},"min":{"1min":0.44900000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.2809999999999999,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.63500000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:48:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.65500000000000003,"15min":0.65000000000000002},"min":{"1min":0.35899999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.097,"5min":1.097,"15min":1.097},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.63300000000000001,"15min":0.626},"min":{"1min":0.443,"5min":0.434,"15min":0.434},"max":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.91800000000000004}]},{"osd":8,"last update":"Fri Feb 9 22:49:01 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.72599999999999998},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":1.079,"5min":1.129,"15min":1.129},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.75,"15min":0.74199999999999999},"min":{"1min":0.51600000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.014,"5min":1.264,"15min":1.264},"last":0.81699999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.3049999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.877}]},{"osd":10,"last update":"Fri Feb 9 22:48:50 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":0.95999999999999996}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98999999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757292,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32688,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712464,"statfs":{"total":95995035648,"available":95961563136,"internally_reserved":0,"allocated":1548288,"data_stored":1037020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.51800000000000002,"15min":0.49199999999999999},"min":{"1min":0.40300000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.74299999999999999,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.49099999999999999,"15min":0.47099999999999997},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74399999999999999,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57799999999999996}]},{"osd":1,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.50800000000000001,"15min":0.496},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.622},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52200000000000002,"15min":0.50800000000000001},"min":{"1min":0.35999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.73099999999999998,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.72199999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.26900000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.73099999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.40600000000000003,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.34100000000000003}]},{"osd":3,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.52500000000000002,"15min":0.50600000000000001},"min":{"1min":0.372,"5min":0.307,"15min":0.307},"max":{"1min":0.77900000000000003,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.54300000000000004,"15min":0.52500000000000002},"min":{"1min":0.35099999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:48:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.55900000000000005,"15min":0.53800000000000003},"min":{"1min":0.377,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:48:14 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.45600000000000002,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.81399999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.83699999999999997,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.49299999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:48:36 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":0.88900000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.45600000000000002,"5min":0.441,"15min":0.441},"max":{"1min":0.92000000000000004,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.64600000000000002}]},{"osd":8,"last update":"Fri Feb 9 22:49:04 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.63,"5min":0.60499999999999998,"15min":0.60099999999999998},"min":{"1min":0.40500000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.47399999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:48:24 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.56100000000000005}]},{"osd":10,"last update":"Fri Feb 9 22:48:51 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.77600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":23258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:49:14.907 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-02-09T22:49:14.907 INFO:tasks.cephadm:Setting up client nodes... 2024-02-09T22:49:14.908 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-02-09T22:49:15.180 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:15.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:15 smithi003 ceph-mon[32678]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:49:15.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:15 smithi003 ceph-mon[32678]: pgmap v290: 97 pgs: 97 active+clean; 583 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s wr, 16 op/s 2024-02-09T22:49:15.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:15 smithi064 ceph-mon[36638]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:49:15.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:15 smithi064 ceph-mon[36638]: pgmap v290: 97 pgs: 97 active+clean; 583 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s wr, 16 op/s 2024-02-09T22:49:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:15 smithi089 ceph-mon[36665]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:49:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:15 smithi089 ceph-mon[36665]: pgmap v290: 97 pgs: 97 active+clean; 583 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s wr, 16 op/s 2024-02-09T22:49:16.568 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:16 smithi003 ceph-mon[32678]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:49:16.629 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:16 smithi089 ceph-mon[36665]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:49:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:16 smithi064 ceph-mon[36638]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:49:16.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 -- 172.21.15.3:0/4038298576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008103930 msgr2=0x7f500810a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 --2- 172.21.15.3:0/4038298576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008103930 0x7f500810a1e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800b0d0 tx=0x7f4ff800b3e0 comp rx=0 tx=0).stop 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 -- 172.21.15.3:0/4038298576 shutdown_connections 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 --2- 172.21.15.3:0/4038298576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5008101d90 0x7f50081021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 --2- 172.21.15.3:0/4038298576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008103930 0x7f500810a1e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 --2- 172.21.15.3:0/4038298576 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5008102f90 0x7f50081033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 -- 172.21.15.3:0/4038298576 >> 172.21.15.3:0/4038298576 conn(0x7f50080fd330 msgr2=0x7f50080ff770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 -- 172.21.15.3:0/4038298576 shutdown_connections 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.792+0000 7f500d588700 1 -- 172.21.15.3:0/4038298576 wait complete. 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 Processor -- start 2024-02-09T22:49:16.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 -- start start 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5008101d90 0x7f5008199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 0x7f500819a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5008103930 0x7f50081a1470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50081a1a40 con 0x7f5008102f90 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50081a1b80 con 0x7f5008103930 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f500d588700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50081a1d10 con 0x7f5008101d90 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 0x7f500819a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 0x7f500819a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45962/0 (socket says 172.21.15.3:45962) 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 -- 172.21.15.3:0/2848777371 learned_addr learned my addr 172.21.15.3:0/2848777371 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:16.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f5006ffd700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5008101d90 0x7f5008199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50077fe700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5008103930 0x7f50081a1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 -- 172.21.15.3:0/2848777371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5008101d90 msgr2=0x7f5008199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5008101d90 0x7f5008199eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 -- 172.21.15.3:0/2848777371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5008103930 msgr2=0x7f50081a1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5008103930 0x7f50081a1470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.793+0000 7f50067fc700 1 -- 172.21.15.3:0/2848777371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff8009d00 con 0x7f5008102f90 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.794+0000 7f50067fc700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 0x7f500819a3f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b560 tx=0x7f4ff000b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.794+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0011830 con 0x7f5008102f90 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.794+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50081a1ff0 con 0x7f5008102f90 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.794+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50081a2530 con 0x7f5008102f90 2024-02-09T22:49:16.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.794+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ff0011e70 con 0x7f5008102f90 2024-02-09T22:49:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.794+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ff000f2e0 con 0x7f5008102f90 2024-02-09T22:49:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.795+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe8005360 con 0x7f5008102f90 2024-02-09T22:49:16.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.796+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f4ff0011990 con 0x7f5008102f90 2024-02-09T22:49:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.796+0000 7f4ffffff700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ff40a2310 0x7f4ff40a47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.797+0000 7f5006ffd700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ff40a2310 0x7f4ff40a47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.797+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f4ff00cd970 con 0x7f5008102f90 2024-02-09T22:49:16.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.797+0000 7f5006ffd700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ff40a2310 0x7f4ff40a47d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5008102df0 tx=0x7f5000005d70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:16.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:16.800+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4ff0096aa0 con 0x7f5008102f90 2024-02-09T22:49:17.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.085+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f4fe8005610 con 0x7f5008102f90 2024-02-09T22:49:17.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.087+0000 7f4ffffff700 1 -- 172.21.15.3:0/2848777371 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f4ff00150f0 con 0x7f5008102f90 2024-02-09T22:49:17.090 INFO:teuthology.orchestra.run.smithi003.stdout:[client.0] 2024-02-09T22:49:17.090 INFO:teuthology.orchestra.run.smithi003.stdout: key = AQDtq8ZlNOwpBRAAkdvYVJkoPO85V/rpKDTMXg== 2024-02-09T22:49:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ff40a2310 msgr2=0x7f4ff40a47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ff40a2310 0x7f4ff40a47d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5008102df0 tx=0x7f5000005d70 comp rx=0 tx=0).stop 2024-02-09T22:49:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 msgr2=0x7f500819a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 0x7f500819a3f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4ff000b560 tx=0x7f4ff000b920 comp rx=0 tx=0).stop 2024-02-09T22:49:17.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 shutdown_connections 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ff40a2310 0x7f4ff40a47d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5008101d90 0x7f5008199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5008103930 0x7f50081a1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 --2- 172.21.15.3:0/2848777371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5008102f90 0x7f500819a3f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.089+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 >> 172.21.15.3:0/2848777371 conn(0x7f50080fd330 msgr2=0x7f50080feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.090+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 shutdown_connections 2024-02-09T22:49:17.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:17.090+0000 7f500d588700 1 -- 172.21.15.3:0/2848777371 wait complete. 2024-02-09T22:49:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:17 smithi003 ceph-mon[32678]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 11 KiB/s wr, 27 op/s 2024-02-09T22:49:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:17 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2848777371' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T22:49:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:17 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2848777371' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T22:49:17.644 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:17 smithi089 ceph-mon[36665]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 11 KiB/s wr, 27 op/s 2024-02-09T22:49:17.644 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:17 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2848777371' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T22:49:17.644 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:17 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2848777371' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T22:49:17.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:17 smithi064 ceph-mon[36638]: pgmap v291: 97 pgs: 97 active+clean; 620 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 11 KiB/s wr, 27 op/s 2024-02-09T22:49:17.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:17 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2848777371' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-02-09T22:49:17.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:17 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2848777371' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-02-09T22:49:17.821 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:49:17.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-02-09T22:49:17.822 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-02-09T22:49:17.865 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-02-09T22:49:17.865 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-02-09T22:49:17.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph mgr dump --format=json 2024-02-09T22:49:18.151 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:18.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:18 smithi003 ceph-mon[32678]: pgmap v292: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 9.5 KiB/s wr, 27 op/s 2024-02-09T22:49:18.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:18 smithi064 ceph-mon[36638]: pgmap v292: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 9.5 KiB/s wr, 27 op/s 2024-02-09T22:49:18.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:18 smithi089 ceph-mon[36665]: pgmap v292: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 9.5 KiB/s wr, 27 op/s 2024-02-09T22:49:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 -- 172.21.15.3:0/4104229488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006a6a0 msgr2=0x7f3e3006aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 --2- 172.21.15.3:0/4104229488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006a6a0 0x7f3e3006aa80 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3e18009990 tx=0x7f3e18009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 -- 172.21.15.3:0/4104229488 shutdown_connections 2024-02-09T22:49:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 --2- 172.21.15.3:0/4104229488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30107060 0x7f3e3010ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:19.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 --2- 172.21.15.3:0/4104229488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006a6a0 0x7f3e3006aa80 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 --2- 172.21.15.3:0/4104229488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006b050 0x7f3e30106980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 -- 172.21.15.3:0/4104229488 >> 172.21.15.3:0/4104229488 conn(0x7f3e30077ae0 msgr2=0x7f3e30077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 -- 172.21.15.3:0/4104229488 shutdown_connections 2024-02-09T22:49:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 -- 172.21.15.3:0/4104229488 wait complete. 2024-02-09T22:49:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 Processor -- start 2024-02-09T22:49:19.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.770+0000 7f3e36d73700 1 -- start start 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006a6a0 0x7f3e30100fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 0x7f3e30101510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30107060 0x7f3e30101a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e301a6600 con 0x7f3e3006b050 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e301a6770 con 0x7f3e3006a6a0 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e301a6930 con 0x7f3e30107060 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 0x7f3e30101510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 0x7f3e30101510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45970/0 (socket says 172.21.15.3:45970) 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 -- 172.21.15.3:0/2333333405 learned_addr learned my addr 172.21.15.3:0/2333333405 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e35310700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30107060 0x7f3e30101a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e34b0f700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006a6a0 0x7f3e30100fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:19.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 -- 172.21.15.3:0/2333333405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30107060 msgr2=0x7f3e30101a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30107060 0x7f3e30101a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 -- 172.21.15.3:0/2333333405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006a6a0 msgr2=0x7f3e30100fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006a6a0 0x7f3e30100fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 -- 172.21.15.3:0/2333333405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e180095f0 con 0x7f3e3006b050 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e34b0f700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006a6a0 0x7f3e30100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2ffff700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 0x7f3e30101510 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000c9e0 tx=0x7f3e2000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20007aa0 con 0x7f3e3006b050 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e301a6c10 con 0x7f3e3006b050 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e20007c00 con 0x7f3e3006b050 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.771+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20018660 con 0x7f3e3006b050 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.772+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e301a7110 con 0x7f3e3006b050 2024-02-09T22:49:19.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.773+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3010a320 con 0x7f3e3006b050 2024-02-09T22:49:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.774+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f3e2000f450 con 0x7f3e3006b050 2024-02-09T22:49:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.774+0000 7f3e2dffb700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e1c0a21e0 0x7f3e1c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.774+0000 7f3e34b0f700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e1c0a21e0 0x7f3e1c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.774+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3e200cdbe0 con 0x7f3e3006b050 2024-02-09T22:49:19.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.775+0000 7f3e34b0f700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e1c0a21e0 0x7f3e1c0a46a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b5c0 tx=0x7f3e18005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:19.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:19.779+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e20096c00 con 0x7f3e3006b050 2024-02-09T22:49:20.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.039+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3e300689e0 con 0x7f3e3006b050 2024-02-09T22:49:20.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.045+0000 7f3e2dffb700 1 -- 172.21.15.3:0/2333333405 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+261473 (secure 0 0 0) 0x7f3e2009a2c0 con 0x7f3e3006b050 2024-02-09T22:49:20.051 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:20.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e1c0a21e0 msgr2=0x7f3e1c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e1c0a21e0 0x7f3e1c0a46a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800b5c0 tx=0x7f3e18005fb0 comp rx=0 tx=0).stop 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 msgr2=0x7f3e30101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 0x7f3e30101510 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f3e2000c9e0 tx=0x7f3e2000ccf0 comp rx=0 tx=0).stop 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 shutdown_connections 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e1c0a21e0 0x7f3e1c0a46a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e30107060 0x7f3e30101a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e3006a6a0 0x7f3e30100fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 --2- 172.21.15.3:0/2333333405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e3006b050 0x7f3e30101510 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 >> 172.21.15.3:0/2333333405 conn(0x7f3e30077ae0 msgr2=0x7f3e300fef10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 shutdown_connections 2024-02-09T22:49:20.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:20.048+0000 7f3e36d73700 1 -- 172.21.15.3:0/2333333405 wait complete. 2024-02-09T22:49:20.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:20 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2333333405' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T22:49:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:20 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2333333405' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T22:49:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:20 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2333333405' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-02-09T22:49:20.723 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":17,"flags":0,"active_gid":14148,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6800","nonce":284621866},{"type":"v1","addr":"172.21.15.3:6801","nonce":284621866}]},"active_addr":"172.21.15.3:6801/284621866","active_change":"2024-02-09T22:42:06.452904+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24106,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24107,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.3:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":3470258541}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2619736972}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2951038011}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":2265484823}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.3:0","nonce":1359016272}]}]} 2024-02-09T22:49:20.728 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-02-09T22:49:20.728 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-02-09T22:49:20.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-09T22:49:20.978 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:21.352 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:21 smithi003 ceph-mon[32678]: pgmap v293: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.2 KiB/s wr, 25 op/s 2024-02-09T22:49:21.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:21 smithi064 ceph-mon[36638]: pgmap v293: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.2 KiB/s wr, 25 op/s 2024-02-09T22:49:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:21 smithi089 ceph-mon[36665]: pgmap v293: 97 pgs: 97 active+clean; 621 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 8.2 KiB/s wr, 25 op/s 2024-02-09T22:49:22.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 -- 172.21.15.3:0/3995843056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560100c20 msgr2=0x7fa560101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:22.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 --2- 172.21.15.3:0/3995843056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560100c20 0x7fa560101080 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fa554009990 tx=0x7fa554009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 -- 172.21.15.3:0/3995843056 shutdown_connections 2024-02-09T22:49:22.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 --2- 172.21.15.3:0/3995843056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5601015c0 0x7fa56010daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 --2- 172.21.15.3:0/3995843056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa560106c60 0x7fa560107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 --2- 172.21.15.3:0/3995843056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560100c20 0x7fa560101080 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 -- 172.21.15.3:0/3995843056 >> 172.21.15.3:0/3995843056 conn(0x7fa560077b50 msgr2=0x7fa560077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 -- 172.21.15.3:0/3995843056 shutdown_connections 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 -- 172.21.15.3:0/3995843056 wait complete. 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.609+0000 7fa566b59700 1 Processor -- start 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 -- start start 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa560100c20 0x7fa56019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5601015c0 0x7fa56019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 0x7fa56019ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56019f320 con 0x7fa560106c60 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56019f490 con 0x7fa5601015c0 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa566b59700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56019f650 con 0x7fa560100c20 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 0x7fa56019ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 0x7fa56019ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45988/0 (socket says 172.21.15.3:45988) 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 -- 172.21.15.3:0/3898932423 learned_addr learned my addr 172.21.15.3:0/3898932423 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:22.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5648f5700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa560100c20 0x7fa56019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa55ffff700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5601015c0 0x7fa56019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 -- 172.21.15.3:0/3898932423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa560100c20 msgr2=0x7fa56019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa560100c20 0x7fa56019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 -- 172.21.15.3:0/3898932423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5601015c0 msgr2=0x7fa56019a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5601015c0 0x7fa56019a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 -- 172.21.15.3:0/3898932423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540095f0 con 0x7fa560106c60 2024-02-09T22:49:22.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.610+0000 7fa5650f6700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 0x7fa56019ed10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa55800ada0 tx=0x7fa55800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.611+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55800f070 con 0x7fa560106c60 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.611+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa558007b30 con 0x7fa560106c60 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.611+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5580052e0 con 0x7fa560106c60 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.611+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56019f930 con 0x7fa560106c60 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.611+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5601a7190 con 0x7fa560106c60 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.612+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5600689e0 con 0x7fa560106c60 2024-02-09T22:49:22.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.613+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fa55800c8d0 con 0x7fa560106c60 2024-02-09T22:49:22.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.614+0000 7fa55dffb700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5480a2260 0x7fa5480a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:22.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.614+0000 7fa5648f5700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5480a2260 0x7fa5480a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:22.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.614+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa5580cd120 con 0x7fa560106c60 2024-02-09T22:49:22.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.614+0000 7fa5648f5700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5480a2260 0x7fa5480a4720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa550005950 tx=0x7fa55000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:22.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.620+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa5580961d0 con 0x7fa560106c60 2024-02-09T22:49:22.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.822+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa560110c60 con 0x7fa560106c60 2024-02-09T22:49:22.826 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:22 smithi003 ceph-mon[32678]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.5 KiB/s wr, 24 op/s 2024-02-09T22:49:22.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.824+0000 7fa55dffb700 1 -- 172.21.15.3:0/3898932423 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22028 (secure 0 0 0) 0x7fa558099890 con 0x7fa560106c60 2024-02-09T22:49:22.828 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:22.828 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:49:08.239545+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5480a2260 msgr2=0x7fa5480a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5480a2260 0x7fa5480a4720 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa550005950 tx=0x7fa55000b410 comp rx=0 tx=0).stop 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 msgr2=0x7fa56019ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 0x7fa56019ed10 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa55800ada0 tx=0x7fa55800c330 comp rx=0 tx=0).stop 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 shutdown_connections 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5480a2260 0x7fa5480a4720 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa560100c20 0x7fa56019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa560106c60 0x7fa56019ed10 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 --2- 172.21.15.3:0/3898932423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5601015c0 0x7fa56019a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:22.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 >> 172.21.15.3:0/3898932423 conn(0x7fa560077b50 msgr2=0x7fa5600699f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:22.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 shutdown_connections 2024-02-09T22:49:22.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:22.826+0000 7fa566b59700 1 -- 172.21.15.3:0/3898932423 wait complete. 2024-02-09T22:49:22.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:22 smithi064 ceph-mon[36638]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.5 KiB/s wr, 24 op/s 2024-02-09T22:49:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:22 smithi089 ceph-mon[36665]: pgmap v294: 97 pgs: 97 active+clean; 626 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 8.5 KiB/s wr, 24 op/s 2024-02-09T22:49:23.481 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-02-09T22:49:23.481 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd dump --format=json 2024-02-09T22:49:23.725 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:23.847 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:23 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3898932423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:23 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3898932423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:23 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3898932423' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:25.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 -- 172.21.15.3:0/378057723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 msgr2=0x7f3ae4101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/378057723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae4101290 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0009990 tx=0x7f3ae0009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 -- 172.21.15.3:0/378057723 shutdown_connections 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/378057723 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 0x7f3ae410db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/378057723 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae4101290 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/378057723 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ae4101860 0x7f3ae4105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 -- 172.21.15.3:0/378057723 >> 172.21.15.3:0/378057723 conn(0x7f3ae4076c40 msgr2=0x7f3ae4077050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 -- 172.21.15.3:0/378057723 shutdown_connections 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 -- 172.21.15.3:0/378057723 wait complete. 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 Processor -- start 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.291+0000 7f3aec0a1700 1 -- start start 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae41990f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ae4101860 0x7f3ae4199630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:25.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 0x7f3ae41a4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae4199ea0 con 0x7f3ae4100eb0 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae41a46b0 con 0x7f3ae4101860 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ae41a47f0 con 0x7f3ae4105f40 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae41990f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae41990f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46000/0 (socket says 172.21.15.3:46000) 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 -- 172.21.15.3:0/1756016739 learned_addr learned my addr 172.21.15.3:0/1756016739 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae963c700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ae4101860 0x7f3ae4199630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aea63e700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 0x7f3ae41a4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 -- 172.21.15.3:0/1756016739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 msgr2=0x7f3ae41a4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 0x7f3ae41a4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 -- 172.21.15.3:0/1756016739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ae4101860 msgr2=0x7f3ae4199630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ae4101860 0x7f3ae4199630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 -- 172.21.15.3:0/1756016739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae00095f0 con 0x7f3ae4100eb0 2024-02-09T22:49:25.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aea63e700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 0x7f3ae41a4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:25.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3ae9e3d700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae41990f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0005910 tx=0x7f3ae000f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:25.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae001d070 con 0x7f3ae4100eb0 2024-02-09T22:49:25.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ae0003680 con 0x7f3ae4100eb0 2024-02-09T22:49:25.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ae41a4a20 con 0x7f3ae4100eb0 2024-02-09T22:49:25.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae0017650 con 0x7f3ae4100eb0 2024-02-09T22:49:25.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.292+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ae41a4e30 con 0x7f3ae4100eb0 2024-02-09T22:49:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.293+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ae41091d0 con 0x7f3ae4100eb0 2024-02-09T22:49:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.294+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f3ae0022470 con 0x7f3ae4100eb0 2024-02-09T22:49:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.295+0000 7f3adaffd700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ad00a21e0 0x7f3ad00a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.295+0000 7f3ae963c700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ad00a21e0 0x7f3ad00a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.295+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f3ae00cf500 con 0x7f3ae4100eb0 2024-02-09T22:49:25.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.296+0000 7f3ae963c700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ad00a21e0 0x7f3ad00a46a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4107400 tx=0x7f3ad400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:25.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.299+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3ae00984b0 con 0x7f3ae4100eb0 2024-02-09T22:49:25.303 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:25 smithi003 ceph-mon[32678]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.9 KiB/s wr, 21 op/s 2024-02-09T22:49:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:25 smithi064 ceph-mon[36638]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.9 KiB/s wr, 21 op/s 2024-02-09T22:49:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:25 smithi089 ceph-mon[36665]: pgmap v295: 97 pgs: 97 active+clean; 626 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 9.5 KiB/s rd, 6.9 KiB/s wr, 21 op/s 2024-02-09T22:49:25.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.500+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3ae4068a10 con 0x7f3ae4100eb0 2024-02-09T22:49:25.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.502+0000 7f3adaffd700 1 -- 172.21.15.3:0/1756016739 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22028 (secure 0 0 0) 0x7f3ae009bb70 con 0x7f3ae4100eb0 2024-02-09T22:49:25.506 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:49:25.506 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":86,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:49:08.239545+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ad00a21e0 msgr2=0x7f3ad00a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ad00a21e0 0x7f3ad00a46a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3ae4107400 tx=0x7f3ad400b410 comp rx=0 tx=0).stop 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 msgr2=0x7f3ae41990f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae41990f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0005910 tx=0x7f3ae000f690 comp rx=0 tx=0).stop 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 shutdown_connections 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ad00a21e0 0x7f3ad00a46a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ae4105f40 0x7f3ae41a4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ae4100eb0 0x7f3ae41990f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 --2- 172.21.15.3:0/1756016739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ae4101860 0x7f3ae4199630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 >> 172.21.15.3:0/1756016739 conn(0x7f3ae4076c40 msgr2=0x7f3ae41003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 shutdown_connections 2024-02-09T22:49:25.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:25.504+0000 7f3aec0a1700 1 -- 172.21.15.3:0/1756016739 wait complete. 2024-02-09T22:49:26.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-02-09T22:49:26.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-02-09T22:49:26.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-02-09T22:49:26.096 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-02-09T22:49:26.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-02-09T22:49:26.097 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-02-09T22:49:26.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-02-09T22:49:26.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-02-09T22:49:26.098 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-02-09T22:49:26.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-02-09T22:49:26.099 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-02-09T22:49:26.100 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-02-09T22:49:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:26 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1756016739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:26 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1756016739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:26 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1756016739' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:49:26.784 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.104 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.106 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.110 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.123 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.126 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.126 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.129 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.254 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:27 smithi003 ceph-mon[32678]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 KiB/s wr, 21 op/s 2024-02-09T22:49:27.258 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:27.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:27 smithi064 ceph-mon[36638]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 KiB/s wr, 21 op/s 2024-02-09T22:49:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:27 smithi089 ceph-mon[36665]: pgmap v296: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 7.1 KiB/s wr, 21 op/s 2024-02-09T22:49:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:28.874 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:28 smithi003 ceph-mon[32678]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-02-09T22:49:28.893 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:28 smithi089 ceph-mon[36665]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-02-09T22:49:28.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:28 smithi064 ceph-mon[36638]: pgmap v297: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-02-09T22:49:30.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:30 smithi003 ceph-mon[32678]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-02-09T22:49:30.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:30 smithi064 ceph-mon[36638]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-02-09T22:49:30.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:30 smithi089 ceph-mon[36665]: pgmap v298: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-02-09T22:49:32.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:32 smithi003 ceph-mon[32678]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-02-09T22:49:32.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:32 smithi064 ceph-mon[36638]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-02-09T22:49:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:32 smithi089 ceph-mon[36665]: pgmap v299: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-02-09T22:49:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/419998582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b4100c20 msgr2=0x7ff3b4101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/419998582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b4100c20 0x7ff3b4101080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009990 tx=0x7ff3a4009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/419998582 shutdown_connections 2024-02-09T22:49:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/419998582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3b41015c0 0x7ff3b410daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/419998582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3b4106c60 0x7ff3b4107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/419998582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b4100c20 0x7ff3b4101080 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/419998582 >> 172.21.15.3:0/419998582 conn(0x7ff3b4077b50 msgr2=0x7ff3b4077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/419998582 shutdown_connections 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/419998582 wait complete. 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 Processor -- start 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.302+0000 7ff3bb1c7700 1 -- start start 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3bb1c7700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3b4100c20 0x7ff3b419a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3bb1c7700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b41015c0 0x7ff3b419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3bb1c7700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3b4106c60 0x7ff3b419ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3bb1c7700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b419f380 con 0x7ff3b41015c0 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3bb1c7700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b419f4f0 con 0x7ff3b4100c20 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3bb1c7700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3b419f6b0 con 0x7ff3b4106c60 2024-02-09T22:49:34.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b41015c0 0x7ff3b419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b41015c0 0x7ff3b419a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44568/0 (socket says 172.21.15.3:44568) 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 -- 172.21.15.3:0/2829339478 learned_addr learned my addr 172.21.15.3:0/2829339478 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b9764700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3b4106c60 0x7ff3b419ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b8f63700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3b4100c20 0x7ff3b419a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 -- 172.21.15.3:0/2829339478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3b4106c60 msgr2=0x7ff3b419ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3b4106c60 0x7ff3b419ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 -- 172.21.15.3:0/2829339478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3b4100c20 msgr2=0x7ff3b419a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3b4100c20 0x7ff3b419a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 -- 172.21.15.3:0/2829339478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a40095f0 con 0x7ff3b41015c0 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.303+0000 7ff3b3fff700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b41015c0 0x7ff3b419a9e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40098e0 tx=0x7ff3a400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:34.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.304+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a401d070 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.304+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3b419f930 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.304+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3a4003680 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.304+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40173f0 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.304+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3b41a71f0 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.305+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff3b4110f30 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.306+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7ff3a4022720 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.306+0000 7ff3b1ffb700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff3a00a2260 0x7ff3a00a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b8f63700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff3a00a2260 0x7ff3a00a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff3a40cf850 con 0x7ff3b41015c0 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b1ffb700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] conn(0x7ff3a00aba40 0x7ff3a00ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:34.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 --> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff3a00ae510 con 0x7ff3a00aba40 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3a40cfc00 con 0x7ff3b41015c0 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b9764700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] conn(0x7ff3a00aba40 0x7ff3a00ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b8f63700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff3a00a2260 0x7ff3a00a4720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005950 tx=0x7ff39c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.307+0000 7ff3b9764700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] conn(0x7ff3a00aba40 0x7ff3a00ade60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.308+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== osd.3 v2:172.21.15.3:6810/3084237347 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff3a00ae510 con 0x7ff3a00aba40 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.325+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 --> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff3b406a360 con 0x7ff3a00aba40 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.325+0000 7ff3b1ffb700 1 -- 172.21.15.3:0/2829339478 <== osd.3 v2:172.21.15.3:6810/3084237347 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff3b406a360 con 0x7ff3a00aba40 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] conn(0x7ff3a00aba40 msgr2=0x7ff3a00ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] conn(0x7ff3a00aba40 0x7ff3a00ade60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff3a00a2260 msgr2=0x7ff3a00a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff3a00a2260 0x7ff3a00a4720 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7ff39c005950 tx=0x7ff39c00b410 comp rx=0 tx=0).stop 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b41015c0 msgr2=0x7ff3b419a9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:34.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 --2- 172.21.15.3:0/2829339478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3b41015c0 0x7ff3b419a9e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40098e0 tx=0x7ff3a400f690 comp rx=0 tx=0).stop 2024-02-09T22:49:34.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3b9764700 1 -- 172.21.15.3:0/2829339478 reap_dead start 2024-02-09T22:49:34.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 shutdown_connections 2024-02-09T22:49:34.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 >> 172.21.15.3:0/2829339478 conn(0x7ff3b4077b50 msgr2=0x7ff3b4069a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:34.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 shutdown_connections 2024-02-09T22:49:34.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:33.326+0000 7ff3bb1c7700 1 -- 172.21.15.3:0/2829339478 wait complete. 2024-02-09T22:49:35.356 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:35 smithi003 ceph-mon[32678]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-02-09T22:49:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:35 smithi064 ceph-mon[36638]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-02-09T22:49:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:35 smithi089 ceph-mon[36665]: pgmap v300: 97 pgs: 97 active+clean; 638 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 9 op/s 2024-02-09T22:49:36.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:36 smithi003 ceph-mon[32678]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-02-09T22:49:36.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:36 smithi064 ceph-mon[36638]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-02-09T22:49:36.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:36 smithi089 ceph-mon[36665]: pgmap v301: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-02-09T22:49:38.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 -- 172.21.15.3:0/1525566157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40101860 msgr2=0x7f7a40105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 --2- 172.21.15.3:0/1525566157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40101860 0x7f7a40105860 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7a34009990 tx=0x7f7a34009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 -- 172.21.15.3:0/1525566157 shutdown_connections 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 --2- 172.21.15.3:0/1525566157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40100eb0 0x7f7a40101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 --2- 172.21.15.3:0/1525566157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40105f40 0x7f7a4010db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 --2- 172.21.15.3:0/1525566157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40101860 0x7f7a40105860 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 -- 172.21.15.3:0/1525566157 >> 172.21.15.3:0/1525566157 conn(0x7f7a40076c40 msgr2=0x7f7a40077050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 -- 172.21.15.3:0/1525566157 shutdown_connections 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.716+0000 7f7a4673d700 1 -- 172.21.15.3:0/1525566157 wait complete. 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.717+0000 7f7a4673d700 1 Processor -- start 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.717+0000 7f7a4673d700 1 -- start start 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.717+0000 7f7a4673d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40100eb0 0x7f7a40197230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101860 0x7f7a40197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:38.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40105f40 0x7f7a4019bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4019c110 con 0x7f7a40105f40 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4019c280 con 0x7f7a40101860 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4019c440 con 0x7f7a40100eb0 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40105f40 0x7f7a4019bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40105f40 0x7f7a4019bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43442/0 (socket says 172.21.15.3:43442) 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 -- 172.21.15.3:0/2225998183 learned_addr learned my addr 172.21.15.3:0/2225998183 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a3ffff700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40100eb0 0x7f7a40197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 -- 172.21.15.3:0/2225998183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40100eb0 msgr2=0x7f7a40197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40100eb0 0x7f7a40197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 -- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101860 msgr2=0x7f7a40197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101860 0x7f7a40197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 -- 172.21.15.3:0/2225998183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a340095f0 con 0x7f7a40105f40 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a44cda700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40105f40 0x7f7a4019bb00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800b970 tx=0x7f7a3800bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:38.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3800eb40 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a3800eca0 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380076f0 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4019c720 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.718+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4006a5f0 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.719+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7a40102340 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.720+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f7a38016440 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3d7fa700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7a280a2260 0x7f7a280a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3ffff700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7a280a2260 0x7f7a280a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f7a38006e40 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3d7fa700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] conn(0x7f7a280aba40 0x7f7a280ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 --> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7a280ae510 con 0x7f7a280aba40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a380cfea0 con 0x7f7a40105f40 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3ffff700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7a280a2260 0x7f7a280a4720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009a10 tx=0x7f7a30009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:38.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.721+0000 7f7a3f7fe700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] conn(0x7f7a280aba40 0x7f7a280ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:38.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.722+0000 7f7a3f7fe700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] conn(0x7f7a280aba40 0x7f7a280ade60 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:38.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.723+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== osd.10 v2:172.21.15.64:6824/451880154 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f7a280ae510 con 0x7f7a280aba40 2024-02-09T22:49:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.740+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 --> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7a40111070 con 0x7f7a280aba40 2024-02-09T22:49:38.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.740+0000 7f7a3d7fa700 1 -- 172.21.15.3:0/2225998183 <== osd.10 v2:172.21.15.64:6824/451880154 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f7a40111070 con 0x7f7a280aba40 2024-02-09T22:49:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] conn(0x7f7a280aba40 msgr2=0x7f7a280ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] conn(0x7f7a280aba40 0x7f7a280ade60 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7a280a2260 msgr2=0x7f7a280a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7a280a2260 0x7f7a280a4720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009a10 tx=0x7f7a30009190 comp rx=0 tx=0).stop 2024-02-09T22:49:38.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40105f40 msgr2=0x7f7a4019bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 --2- 172.21.15.3:0/2225998183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40105f40 0x7f7a4019bb00 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7a3800b970 tx=0x7f7a3800bc80 comp rx=0 tx=0).stop 2024-02-09T22:49:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a44cda700 1 -- 172.21.15.3:0/2225998183 reap_dead start 2024-02-09T22:49:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 shutdown_connections 2024-02-09T22:49:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 >> 172.21.15.3:0/2225998183 conn(0x7f7a40076c40 msgr2=0x7f7a401003d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 shutdown_connections 2024-02-09T22:49:38.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:38.741+0000 7f7a4673d700 1 -- 172.21.15.3:0/2225998183 wait complete. 2024-02-09T22:49:38.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:38 smithi003 ceph-mon[32678]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:49:38.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:38 smithi064 ceph-mon[36638]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:49:38.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:38 smithi089 ceph-mon[36665]: pgmap v302: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:49:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 -- 172.21.15.3:0/653102059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea0107d70 msgr2=0x7f2ea0108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:39.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/653102059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea0107d70 0x7f2ea0108150 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000b0d0 tx=0x7f2e9000b3e0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 -- 172.21.15.3:0/653102059 shutdown_connections 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/653102059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea006a370 0x7f2ea010ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/653102059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea0107d70 0x7f2ea0108150 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/653102059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ea00699d0 0x7f2ea0069e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 -- 172.21.15.3:0/653102059 >> 172.21.15.3:0/653102059 conn(0x7f2ea0077b50 msgr2=0x7f2ea0077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 -- 172.21.15.3:0/653102059 shutdown_connections 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 -- 172.21.15.3:0/653102059 wait complete. 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.276+0000 7f2ea74a8700 1 Processor -- start 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 -- start start 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea00699d0 0x7f2ea0198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea006a370 0x7f2ea01988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ea0107d70 0x7f2ea019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:39.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea019d260 con 0x7f2ea006a370 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea019d3d0 con 0x7f2ea0107d70 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea74a8700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea019d590 con 0x7f2ea00699d0 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea006a370 0x7f2ea01988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea006a370 0x7f2ea01988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43450/0 (socket says 172.21.15.3:43450) 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 -- 172.21.15.3:0/2444824432 learned_addr learned my addr 172.21.15.3:0/2444824432 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea5a45700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ea0107d70 0x7f2ea019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea5244700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea00699d0 0x7f2ea0198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 -- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea00699d0 msgr2=0x7f2ea0198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea00699d0 0x7f2ea0198380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 -- 172.21.15.3:0/2444824432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ea0107d70 msgr2=0x7f2ea019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2ea0107d70 0x7f2ea019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 -- 172.21.15.3:0/2444824432 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90009d00 con 0x7f2ea006a370 2024-02-09T22:49:39.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea5244700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2ea00699d0 0x7f2ea0198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.277+0000 7f2ea4a43700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea006a370 0x7f2ea01988c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400d950 tx=0x7f2e9400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.278+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e94009940 con 0x7f2ea006a370 2024-02-09T22:49:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.278+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e94010460 con 0x7f2ea006a370 2024-02-09T22:49:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.278+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9400f620 con 0x7f2ea006a370 2024-02-09T22:49:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.278+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea019d870 con 0x7f2ea006a370 2024-02-09T22:49:39.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.278+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea01a50d0 con 0x7f2ea006a370 2024-02-09T22:49:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.279+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2ea0112000 con 0x7f2ea006a370 2024-02-09T22:49:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.280+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f2e94010960 con 0x7f2ea006a370 2024-02-09T22:49:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2e9e7fc700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e880a20c0 0x7f2e880a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:39.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2ea5244700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e880a20c0 0x7f2e880a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f2e940ce0b0 con 0x7f2ea006a370 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2e9e7fc700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] conn(0x7f2e880ab730 0x7f2e880adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 --> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e880ae200 con 0x7f2e880ab730 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e940ce550 con 0x7f2ea006a370 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2ea5244700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e880a20c0 0x7f2e880a4580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e9000b040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.281+0000 7f2ea5a45700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] conn(0x7f2e880ab730 0x7f2e880adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.282+0000 7f2ea5a45700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] conn(0x7f2e880ab730 0x7f2e880adb50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:39.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.282+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== osd.11 v2:172.21.15.89:6824/2302784119 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2e880ae200 con 0x7f2e880ab730 2024-02-09T22:49:39.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.299+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 --> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2ea0062430 con 0x7f2e880ab730 2024-02-09T22:49:39.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.299+0000 7f2e9e7fc700 1 -- 172.21.15.3:0/2444824432 <== osd.11 v2:172.21.15.89:6824/2302784119 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2ea0062430 con 0x7f2e880ab730 2024-02-09T22:49:39.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.299+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] conn(0x7f2e880ab730 msgr2=0x7f2e880adb50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:39.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.299+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] conn(0x7f2e880ab730 0x7f2e880adb50 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:39.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e880a20c0 msgr2=0x7f2e880a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:39.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2e880a20c0 0x7f2e880a4580 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f2e90009fd0 tx=0x7f2e9000b040 comp rx=0 tx=0).stop 2024-02-09T22:49:39.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea006a370 msgr2=0x7f2ea01988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 --2- 172.21.15.3:0/2444824432 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2ea006a370 0x7f2ea01988c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f2e9400d950 tx=0x7f2e9400dc60 comp rx=0 tx=0).stop 2024-02-09T22:49:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea5a45700 1 -- 172.21.15.3:0/2444824432 reap_dead start 2024-02-09T22:49:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 shutdown_connections 2024-02-09T22:49:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 >> 172.21.15.3:0/2444824432 conn(0x7f2ea0077b50 msgr2=0x7f2ea00ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 shutdown_connections 2024-02-09T22:49:39.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:39.300+0000 7f2ea74a8700 1 -- 172.21.15.3:0/2444824432 wait complete. 2024-02-09T22:49:40.358 INFO:teuthology.orchestra.run.smithi003.stdout:347892350996 2024-02-09T22:49:40.359 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-02-09T22:49:40.738 INFO:teuthology.orchestra.run.smithi003.stdout:107374182458 2024-02-09T22:49:40.738 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-02-09T22:49:40.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:40 smithi003 ceph-mon[32678]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-02-09T22:49:40.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:40 smithi064 ceph-mon[36638]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-02-09T22:49:40.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:40 smithi089 ceph-mon[36665]: pgmap v303: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 426 B/s wr, 10 op/s 2024-02-09T22:49:42.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 -- 172.21.15.3:0/2533510423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 msgr2=0x7fd4e8101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:42.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 --2- 172.21.15.3:0/2533510423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e8101290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009990 tx=0x7fd4d8009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 -- 172.21.15.3:0/2533510423 shutdown_connections 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 --2- 172.21.15.3:0/2533510423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4e8105f00 0x7fd4e810dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 --2- 172.21.15.3:0/2533510423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e8101290 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 --2- 172.21.15.3:0/2533510423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4e8101860 0x7fd4e8105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 -- 172.21.15.3:0/2533510423 >> 172.21.15.3:0/2533510423 conn(0x7fd4e8076c70 msgr2=0x7fd4e8077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 -- 172.21.15.3:0/2533510423 shutdown_connections 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 -- 172.21.15.3:0/2533510423 wait complete. 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.524+0000 7fd4ed212700 1 Processor -- start 2024-02-09T22:49:42.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 -- start start 2024-02-09T22:49:42.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e81028c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4e8101860 0x7fd4e8102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4e8105f00 0x7fd4e8103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81a6600 con 0x7fd4e8100eb0 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81a6770 con 0x7fd4e8105f00 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4ed212700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e81a6930 con 0x7fd4e8101860 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e81028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e81028c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43460/0 (socket says 172.21.15.3:43460) 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 -- 172.21.15.3:0/308660789 learned_addr learned my addr 172.21.15.3:0/308660789 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e759e700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4e8105f00 0x7fd4e8103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e659c700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4e8101860 0x7fd4e8102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 -- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4e8101860 msgr2=0x7fd4e8102e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:42.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd4e8101860 0x7fd4e8102e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 -- 172.21.15.3:0/308660789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4e8105f00 msgr2=0x7fd4e8103340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd4e8105f00 0x7fd4e8103340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 -- 172.21.15.3:0/308660789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80095f0 con 0x7fd4e8100eb0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.525+0000 7fd4e6d9d700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e81028c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800f6a0 tx=0x7fd4d800f6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.526+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d801d070 con 0x7fd4e8100eb0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.526+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e81a6bb0 con 0x7fd4e8100eb0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.526+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4d8003680 con 0x7fd4e8100eb0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.526+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80173f0 con 0x7fd4e8100eb0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.526+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e81a70a0 con 0x7fd4e8100eb0 2024-02-09T22:49:42.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.526+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4e8110eb0 con 0x7fd4e8100eb0 2024-02-09T22:49:42.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.528+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fd4d8022470 con 0x7fd4e8100eb0 2024-02-09T22:49:42.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.528+0000 7fd4cffff700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd4d00a21e0 0x7fd4d00a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:42.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.528+0000 7fd4e659c700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd4d00a21e0 0x7fd4d00a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:42.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.528+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd4d80cf1f0 con 0x7fd4e8100eb0 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.529+0000 7fd4cffff700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] conn(0x7fd4d00ab9c0 0x7fd4d00adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.529+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 --> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd4d00ae490 con 0x7fd4d00ab9c0 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.529+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d80cf690 con 0x7fd4e8100eb0 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.529+0000 7fd4e659c700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd4d00a21e0 0x7fd4d00a46a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8107410 tx=0x7fd4dc00a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.529+0000 7fd4e759e700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] conn(0x7fd4d00ab9c0 0x7fd4d00adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.529+0000 7fd4e759e700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] conn(0x7fd4d00ab9c0 0x7fd4d00adde0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:42.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.530+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== osd.8 v2:172.21.15.89:6816/4011062821 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fd4d00ae490 con 0x7fd4d00ab9c0 2024-02-09T22:49:42.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.547+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 --> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4e80623e0 con 0x7fd4d00ab9c0 2024-02-09T22:49:42.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.547+0000 7fd4cffff700 1 -- 172.21.15.3:0/308660789 <== osd.8 v2:172.21.15.89:6816/4011062821 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fd4e80623e0 con 0x7fd4d00ab9c0 2024-02-09T22:49:42.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] conn(0x7fd4d00ab9c0 msgr2=0x7fd4d00adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] conn(0x7fd4d00ab9c0 0x7fd4d00adde0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd4d00a21e0 msgr2=0x7fd4d00a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd4d00a21e0 0x7fd4d00a46a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8107410 tx=0x7fd4dc00a300 comp rx=0 tx=0).stop 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 msgr2=0x7fd4e81028c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 --2- 172.21.15.3:0/308660789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd4e8100eb0 0x7fd4e81028c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fd4d800f6a0 tx=0x7fd4d800f6d0 comp rx=0 tx=0).stop 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4e759e700 1 -- 172.21.15.3:0/308660789 reap_dead start 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 shutdown_connections 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 >> 172.21.15.3:0/308660789 conn(0x7fd4e8076c70 msgr2=0x7fd4e806a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 shutdown_connections 2024-02-09T22:49:42.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:42.548+0000 7fd4ed212700 1 -- 172.21.15.3:0/308660789 wait complete. 2024-02-09T22:49:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:42 smithi003 ceph-mon[32678]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:49:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:42 smithi064 ceph-mon[36638]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:49:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:42 smithi089 ceph-mon[36665]: pgmap v304: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:49:43.793 INFO:teuthology.orchestra.run.smithi003.stdout:313532612632 2024-02-09T22:49:43.793 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 -- 172.21.15.3:0/136882692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4101da0 msgr2=0x7f58b4102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 --2- 172.21.15.3:0/136882692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4101da0 0x7f58b4102220 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f58a8009990 tx=0x7f58a8009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 -- 172.21.15.3:0/136882692 shutdown_connections 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 --2- 172.21.15.3:0/136882692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b4102760 0x7f58b410ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 --2- 172.21.15.3:0/136882692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58b4107e80 0x7f58b4108260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 --2- 172.21.15.3:0/136882692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4101da0 0x7f58b4102220 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 -- 172.21.15.3:0/136882692 >> 172.21.15.3:0/136882692 conn(0x7f58b4078de0 msgr2=0x7f58b40791f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 -- 172.21.15.3:0/136882692 shutdown_connections 2024-02-09T22:49:44.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.345+0000 7f58ba320700 1 -- 172.21.15.3:0/136882692 wait complete. 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 Processor -- start 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 -- start start 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58b4101da0 0x7f58b41982e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b4102760 0x7f58b4198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4107e80 0x7f58b419cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b419d170 con 0x7f58b4107e80 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b419d2e0 con 0x7f58b4101da0 2024-02-09T22:49:44.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58ba320700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b419d4a0 con 0x7f58b4102760 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58b88bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4107e80 0x7f58b419cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58b88bd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4107e80 0x7f58b419cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43474/0 (socket says 172.21.15.3:43474) 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58b88bd700 1 -- 172.21.15.3:0/3846434413 learned_addr learned my addr 172.21.15.3:0/3846434413 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58b37fe700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b4102760 0x7f58b4198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.346+0000 7f58b3fff700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58b4101da0 0x7f58b41982e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b88bd700 1 -- 172.21.15.3:0/3846434413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b4102760 msgr2=0x7f58b4198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b88bd700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58b4102760 0x7f58b4198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b88bd700 1 -- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58b4101da0 msgr2=0x7f58b41982e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b88bd700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58b4101da0 0x7f58b41982e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b88bd700 1 -- 172.21.15.3:0/3846434413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a80095f0 con 0x7f58b4107e80 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b88bd700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4107e80 0x7f58b419cb60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f58ac007ee0 tx=0x7f58ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac017070 con 0x7f58b4107e80 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b419d780 con 0x7f58b4107e80 2024-02-09T22:49:44.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58ac00f040 con 0x7f58b4107e80 2024-02-09T22:49:44.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ac005650 con 0x7f58b4107e80 2024-02-09T22:49:44.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.347+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b41a4fe0 con 0x7f58b4107e80 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.348+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f58b4111fc0 con 0x7f58b4107e80 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.349+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f58ac004cd0 con 0x7f58b4107e80 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.349+0000 7f58b17fa700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f589c0a21e0 0x7f589c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b3fff700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f589c0a21e0 0x7f589c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f58ac0cdb70 con 0x7f58b4107e80 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b17fa700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] conn(0x7f589c0ab9c0 0x7f589c0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 --> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f589c0ae490 con 0x7f589c0ab9c0 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58ac0cdee0 con 0x7f58b4107e80 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b3fff700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f589c0a21e0 0x7f589c0a46a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f58a4005950 tx=0x7f58a400a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:44.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.350+0000 7f58b37fe700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] conn(0x7f589c0ab9c0 0x7f589c0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.351+0000 7f58b37fe700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] conn(0x7f589c0ab9c0 0x7f589c0adde0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:44.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.352+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== osd.4 v2:172.21.15.64:6808/285015639 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f589c0ae490 con 0x7f589c0ab9c0 2024-02-09T22:49:44.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.368+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 --> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f58b4062430 con 0x7f589c0ab9c0 2024-02-09T22:49:44.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58b17fa700 1 -- 172.21.15.3:0/3846434413 <== osd.4 v2:172.21.15.64:6808/285015639 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f58b4062430 con 0x7f589c0ab9c0 2024-02-09T22:49:44.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] conn(0x7f589c0ab9c0 msgr2=0x7f589c0adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] conn(0x7f589c0ab9c0 0x7f589c0adde0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f589c0a21e0 msgr2=0x7f589c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f589c0a21e0 0x7f589c0a46a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f58a4005950 tx=0x7f58a400a300 comp rx=0 tx=0).stop 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4107e80 msgr2=0x7f58b419cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 --2- 172.21.15.3:0/3846434413 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58b4107e80 0x7f58b419cb60 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f58ac007ee0 tx=0x7f58ac00d3b0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58b88bd700 1 -- 172.21.15.3:0/3846434413 reap_dead start 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 shutdown_connections 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 >> 172.21.15.3:0/3846434413 conn(0x7f58b4078de0 msgr2=0x7f58b4100ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 shutdown_connections 2024-02-09T22:49:44.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.369+0000 7f58ba320700 1 -- 172.21.15.3:0/3846434413 wait complete. 2024-02-09T22:49:44.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/737206784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0101860 msgr2=0x7ff8b0105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/737206784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0101860 0x7ff8b0105820 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0009990 tx=0x7ff8a0009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/737206784 shutdown_connections 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/737206784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0105f00 0x7ff8b010dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/737206784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8b0100eb0 0x7ff8b0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/737206784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0101860 0x7ff8b0105820 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/737206784 >> 172.21.15.3:0/737206784 conn(0x7ff8b0076c70 msgr2=0x7ff8b0077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/737206784 shutdown_connections 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/737206784 wait complete. 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 Processor -- start 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.679+0000 7ff8b7ac0700 1 -- start start 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b7ac0700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0100eb0 0x7ff8b0199090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b7ac0700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8b0101860 0x7ff8b01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b7ac0700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0105f00 0x7ff8b01a4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b7ac0700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b0199ed0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b7ac0700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b01a4650 con 0x7ff8b0101860 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b7ac0700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b01a47c0 con 0x7ff8b0105f00 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0100eb0 0x7ff8b0199090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0100eb0 0x7ff8b0199090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43496/0 (socket says 172.21.15.3:43496) 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 -- 172.21.15.3:0/3538312779 learned_addr learned my addr 172.21.15.3:0/3538312779 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b505b700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8b0101860 0x7ff8b01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b605d700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0105f00 0x7ff8b01a4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 -- 172.21.15.3:0/3538312779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0105f00 msgr2=0x7ff8b01a4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0105f00 0x7ff8b01a4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 -- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8b0101860 msgr2=0x7ff8b01995d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff8b0101860 0x7ff8b01995d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 -- 172.21.15.3:0/3538312779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00095f0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b605d700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff8b0105f00 0x7ff8b01a4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8b585c700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0100eb0 0x7ff8b0199090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00d950 tx=0x7ff8ac00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.680+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac0098e0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.681+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b01a4aa0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.681+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8ac010460 con 0x7ff8b0100eb0 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.681+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ac00f620 con 0x7ff8b0100eb0 2024-02-09T22:49:44.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.681+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b01a4fc0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.681+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff8b0110f00 con 0x7ff8b0100eb0 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7ff8ac0105d0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8a6ffd700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff89c0a2260 0x7ff89c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff8ac0cdf20 con 0x7ff8b0100eb0 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8b505b700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff89c0a2260 0x7ff89c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8a6ffd700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] conn(0x7ff89c0aba40 0x7ff89c0ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:44.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 --> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff89c0ae510 con 0x7ff89c0aba40 2024-02-09T22:49:44.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.683+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff8ac0ce3c0 con 0x7ff8b0100eb0 2024-02-09T22:49:44.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.684+0000 7ff8b505b700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff89c0a2260 0x7ff89c0a4720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000b5c0 tx=0x7ff8a0005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:44.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.684+0000 7ff8b605d700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] conn(0x7ff89c0aba40 0x7ff89c0ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:44.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.684+0000 7ff8b605d700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] conn(0x7ff89c0aba40 0x7ff89c0ade60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:44.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.685+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== osd.7 v2:172.21.15.64:6816/867242656 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff89c0ae510 con 0x7ff89c0aba40 2024-02-09T22:49:44.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.701+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 --> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff8b0062430 con 0x7ff89c0aba40 2024-02-09T22:49:44.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8a6ffd700 1 -- 172.21.15.3:0/3538312779 <== osd.7 v2:172.21.15.64:6816/867242656 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff8b0062430 con 0x7ff89c0aba40 2024-02-09T22:49:44.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] conn(0x7ff89c0aba40 msgr2=0x7ff89c0ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] conn(0x7ff89c0aba40 0x7ff89c0ade60 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff89c0a2260 msgr2=0x7ff89c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff89c0a2260 0x7ff89c0a4720 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000b5c0 tx=0x7ff8a0005960 comp rx=0 tx=0).stop 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0100eb0 msgr2=0x7ff8b0199090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 --2- 172.21.15.3:0/3538312779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff8b0100eb0 0x7ff8b0199090 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff8ac00d950 tx=0x7ff8ac00dc60 comp rx=0 tx=0).stop 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b605d700 1 -- 172.21.15.3:0/3538312779 reap_dead start 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 shutdown_connections 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 >> 172.21.15.3:0/3538312779 conn(0x7ff8b0076c70 msgr2=0x7ff8b00fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.702+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 shutdown_connections 2024-02-09T22:49:44.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:44.703+0000 7ff8b7ac0700 1 -- 172.21.15.3:0/3538312779 wait complete. 2024-02-09T22:49:45.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 -- 172.21.15.3:0/1214324315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 msgr2=0x7efeec105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:45.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1214324315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec105820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7efed4009990 tx=0x7efed4009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 -- 172.21.15.3:0/1214324315 shutdown_connections 2024-02-09T22:49:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1214324315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec105f00 0x7efeec10dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1214324315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efeec100eb0 0x7efeec101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1214324315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec105820 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 -- 172.21.15.3:0/1214324315 >> 172.21.15.3:0/1214324315 conn(0x7efeec076c70 msgr2=0x7efeec077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 -- 172.21.15.3:0/1214324315 shutdown_connections 2024-02-09T22:49:45.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.021+0000 7efef0cbc700 1 -- 172.21.15.3:0/1214324315 wait complete. 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 Processor -- start 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 -- start start 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efeec100eb0 0x7efeec197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec105f00 0x7efeec19ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec19c040 con 0x7efeec101860 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec19c1b0 con 0x7efeec100eb0 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efef0cbc700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeec19c370 con 0x7efeec105f00 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efee9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efee9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43522/0 (socket says 172.21.15.3:43522) 2024-02-09T22:49:45.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efee9d9b700 1 -- 172.21.15.3:0/1613202801 learned_addr learned my addr 172.21.15.3:0/1613202801 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.022+0000 7efeea59c700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efeec100eb0 0x7efeec197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efeead9d700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec105f00 0x7efeec19ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee9d9b700 1 -- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec105f00 msgr2=0x7efeec19ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee9d9b700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efeec105f00 0x7efeec19ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee9d9b700 1 -- 172.21.15.3:0/1613202801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efeec100eb0 msgr2=0x7efeec197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee9d9b700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efeec100eb0 0x7efeec197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee9d9b700 1 -- 172.21.15.3:0/1613202801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed40095f0 con 0x7efeec101860 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee9d9b700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec1976a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efed40098e0 tx=0x7efed400f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed401d070 con 0x7efeec101860 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efed4003680 con 0x7efeec101860 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeec19c5f0 con 0x7efeec101860 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efed40173f0 con 0x7efeec101860 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.023+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeec1a3eb0 con 0x7efeec101860 2024-02-09T22:49:45.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.024+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efeec110f00 con 0x7efeec101860 2024-02-09T22:49:45.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.025+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7efed4022720 con 0x7efeec101860 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efee37fe700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efed80a2260 0x7efed80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efeea59c700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efed80a2260 0x7efed80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efed40cf130 con 0x7efeec101860 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efee37fe700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] conn(0x7efed80aba40 0x7efed80ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 --> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efed80ae510 con 0x7efed80aba40 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7efed40cf5d0 con 0x7efeec101860 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efeea59c700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efed80a2260 0x7efed80a4720 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efedc005950 tx=0x7efedc00a300 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.026+0000 7efeead9d700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] conn(0x7efed80aba40 0x7efed80ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.027+0000 7efeead9d700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] conn(0x7efed80aba40 0x7efed80ade60 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:45.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.028+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== osd.2 v2:172.21.15.89:6800/2362328563 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7efed80ae510 con 0x7efed80aba40 2024-02-09T22:49:45.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.044+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 --> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efeec0623b0 con 0x7efed80aba40 2024-02-09T22:49:45.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.044+0000 7efee37fe700 1 -- 172.21.15.3:0/1613202801 <== osd.2 v2:172.21.15.89:6800/2362328563 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efeec0623b0 con 0x7efed80aba40 2024-02-09T22:49:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] conn(0x7efed80aba40 msgr2=0x7efed80ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] conn(0x7efed80aba40 0x7efed80ade60 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efed80a2260 msgr2=0x7efed80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efed80a2260 0x7efed80a4720 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efedc005950 tx=0x7efedc00a300 comp rx=0 tx=0).stop 2024-02-09T22:49:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 msgr2=0x7efeec1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:45.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 --2- 172.21.15.3:0/1613202801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efeec101860 0x7efeec1976a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7efed40098e0 tx=0x7efed400f690 comp rx=0 tx=0).stop 2024-02-09T22:49:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efeead9d700 1 -- 172.21.15.3:0/1613202801 reap_dead start 2024-02-09T22:49:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 shutdown_connections 2024-02-09T22:49:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 >> 172.21.15.3:0/1613202801 conn(0x7efeec076c70 msgr2=0x7efeec0fed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 shutdown_connections 2024-02-09T22:49:45.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:45.045+0000 7efef0cbc700 1 -- 172.21.15.3:0/1613202801 wait complete. 2024-02-09T22:49:45.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:45 smithi003 ceph-mon[32678]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:49:45.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:45 smithi064 ceph-mon[36638]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:49:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:45 smithi089 ceph-mon[36665]: pgmap v305: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:49:45.670 INFO:teuthology.orchestra.run.smithi003.stdout:141733920824 2024-02-09T22:49:45.670 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-02-09T22:49:46.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 -- 172.21.15.3:0/77919888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac107d50 msgr2=0x7fdaac108130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:46.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 --2- 172.21.15.3:0/77919888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac107d50 0x7fdaac108130 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fdaa40099c0 tx=0x7fdaa4009cd0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 -- 172.21.15.3:0/77919888 shutdown_connections 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 --2- 172.21.15.3:0/77919888 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac102730 0x7fdaac10eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 --2- 172.21.15.3:0/77919888 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac107d50 0x7fdaac108130 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 --2- 172.21.15.3:0/77919888 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaac101d70 0x7fdaac1021f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 -- 172.21.15.3:0/77919888 >> 172.21.15.3:0/77919888 conn(0x7fdaac078e50 msgr2=0x7fdaac079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 -- 172.21.15.3:0/77919888 shutdown_connections 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.224+0000 7fdab3586700 1 -- 172.21.15.3:0/77919888 wait complete. 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 Processor -- start 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 -- start start 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac101d70 0x7fdaac198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac102730 0x7fdaac198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaac107d50 0x7fdaac19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac19d010 con 0x7fdaac102730 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac19d180 con 0x7fdaac101d70 2024-02-09T22:49:46.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab3586700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaac19d340 con 0x7fdaac107d50 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab0b21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac102730 0x7fdaac198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab0b21700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac102730 0x7fdaac198670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43530/0 (socket says 172.21.15.3:43530) 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab0b21700 1 -- 172.21.15.3:0/3315734021 learned_addr learned my addr 172.21.15.3:0/3315734021 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.225+0000 7fdab1b23700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaac107d50 0x7fdaac19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab1322700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac101d70 0x7fdaac198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab0b21700 1 -- 172.21.15.3:0/3315734021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaac107d50 msgr2=0x7fdaac19ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab0b21700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaac107d50 0x7fdaac19ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab0b21700 1 -- 172.21.15.3:0/3315734021 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac101d70 msgr2=0x7fdaac198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab0b21700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdaac101d70 0x7fdaac198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab0b21700 1 -- 172.21.15.3:0/3315734021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa40095f0 con 0x7fdaac102730 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab1b23700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdaac107d50 0x7fdaac19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:46.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab0b21700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac102730 0x7fdaac198670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000d950 tx=0x7fdaa000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa0009b40 con 0x7fdaac102730 2024-02-09T22:49:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdaa0009ca0 con 0x7fdaac102730 2024-02-09T22:49:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaac19d620 con 0x7fdaac102730 2024-02-09T22:49:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaa000f6b0 con 0x7fdaac102730 2024-02-09T22:49:46.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.226+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaac1a4e80 con 0x7fdaac102730 2024-02-09T22:49:46.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.227+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fdaac112070 con 0x7fdaac102730 2024-02-09T22:49:46.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.228+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fdaa0010460 con 0x7fdaac102730 2024-02-09T22:49:46.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdaaa7fc700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fda940a21e0 0x7fda940a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:46.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fdaa00ce0e0 con 0x7fdaac102730 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdab1322700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fda940a21e0 0x7fda940a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdaaa7fc700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] conn(0x7fda940ab9c0 0x7fda940adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 --> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fda940ae490 con 0x7fda940ab9c0 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdaa00ce580 con 0x7fdaac102730 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.229+0000 7fdab1b23700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] conn(0x7fda940ab9c0 0x7fda940adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.230+0000 7fdab1322700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fda940a21e0 0x7fda940a46a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4006010 tx=0x7fdaa4005a90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.230+0000 7fdab1b23700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] conn(0x7fda940ab9c0 0x7fda940adde0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:46.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.230+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== osd.0 v2:172.21.15.3:6802/103055907 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fda940ae490 con 0x7fda940ab9c0 2024-02-09T22:49:46.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.251+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 --> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fdaac062430 con 0x7fda940ab9c0 2024-02-09T22:49:46.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.251+0000 7fdaaa7fc700 1 -- 172.21.15.3:0/3315734021 <== osd.0 v2:172.21.15.3:6802/103055907 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fdaac062430 con 0x7fda940ab9c0 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.251+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] conn(0x7fda940ab9c0 msgr2=0x7fda940adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.251+0000 7fdab3586700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] conn(0x7fda940ab9c0 0x7fda940adde0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fda940a21e0 msgr2=0x7fda940a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fda940a21e0 0x7fda940a46a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fdaa4006010 tx=0x7fdaa4005a90 comp rx=0 tx=0).stop 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac102730 msgr2=0x7fdaac198670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 --2- 172.21.15.3:0/3315734021 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdaac102730 0x7fdaac198670 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fdaa000d950 tx=0x7fdaa000dd10 comp rx=0 tx=0).stop 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab1b23700 1 -- 172.21.15.3:0/3315734021 reap_dead start 2024-02-09T22:49:46.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 shutdown_connections 2024-02-09T22:49:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 >> 172.21.15.3:0/3315734021 conn(0x7fdaac078e50 msgr2=0x7fdaac100910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 shutdown_connections 2024-02-09T22:49:46.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:46.252+0000 7fdab3586700 1 -- 172.21.15.3:0/3315734021 wait complete. 2024-02-09T22:49:46.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:46 smithi003 ceph-mon[32678]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:49:46.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:46 smithi064 ceph-mon[36638]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:49:46.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:46 smithi089 ceph-mon[36665]: pgmap v306: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:49:47.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 -- 172.21.15.3:0/3040206611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 msgr2=0x7f5068101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:47.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 --2- 172.21.15.3:0/3040206611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068101080 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f505c009990 tx=0x7f505c009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 -- 172.21.15.3:0/3040206611 shutdown_connections 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 --2- 172.21.15.3:0/3040206611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50681015c0 0x7f506810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 --2- 172.21.15.3:0/3040206611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5068106c60 0x7f5068107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 --2- 172.21.15.3:0/3040206611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068101080 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 -- 172.21.15.3:0/3040206611 >> 172.21.15.3:0/3040206611 conn(0x7f5068077b50 msgr2=0x7f5068077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 -- 172.21.15.3:0/3040206611 shutdown_connections 2024-02-09T22:49:47.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.053+0000 7f506cd12700 1 -- 172.21.15.3:0/3040206611 wait complete. 2024-02-09T22:49:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 Processor -- start 2024-02-09T22:49:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 -- start start 2024-02-09T22:49:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068196fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50681015c0 0x7f5068197510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5068106c60 0x7f5068197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:47.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681a6710 con 0x7f5068100c20 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681a6880 con 0x7f5068106c60 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506cd12700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50681a6a40 con 0x7f50681015c0 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068196fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068196fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60802/0 (socket says 172.21.15.3:60802) 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 -- 172.21.15.3:0/3290946773 learned_addr learned my addr 172.21.15.3:0/3290946773 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f5066d9d700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5068106c60 0x7f5068197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f5065d9b700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50681015c0 0x7f5068197510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 -- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50681015c0 msgr2=0x7f5068197510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50681015c0 0x7f5068197510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 -- 172.21.15.3:0/3290946773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5068106c60 msgr2=0x7f5068197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5068106c60 0x7f5068197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f506659c700 1 -- 172.21.15.3:0/3290946773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505c0095f0 con 0x7f5068100c20 2024-02-09T22:49:47.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.054+0000 7f5065d9b700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f50681015c0 0x7f5068197510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.055+0000 7f506659c700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068196fd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f505800b530 tx=0x7f505800b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.055+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058011830 con 0x7f5068100c20 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.055+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5058011e70 con 0x7f5068100c20 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.055+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50681a6d20 con 0x7f5068100c20 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.055+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f505800f2e0 con 0x7f5068100c20 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.055+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50681a7210 con 0x7f5068100c20 2024-02-09T22:49:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.056+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5068110fb0 con 0x7f5068100c20 2024-02-09T22:49:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.057+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f50580105f0 con 0x7f5068100c20 2024-02-09T22:49:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.057+0000 7f50577fe700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50500a2260 0x7f50500a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f5065d9b700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50500a2260 0x7f50500a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50580cdbc0 con 0x7f5068100c20 2024-02-09T22:49:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f50577fe700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] conn(0x7f50500aba40 0x7f50500ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:47.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 --> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50500ae510 con 0x7f50500aba40 2024-02-09T22:49:47.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50580ce060 con 0x7f5068100c20 2024-02-09T22:49:47.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f5065d9b700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50500a2260 0x7f50500a4720 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f505c00b5c0 tx=0x7f505c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:47.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f5066d9d700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] conn(0x7f50500aba40 0x7f50500ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:47.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.058+0000 7f5066d9d700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] conn(0x7f50500aba40 0x7f50500ade60 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:47.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.059+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== osd.5 v2:172.21.15.89:6808/708499709 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f50500ae510 con 0x7f50500aba40 2024-02-09T22:49:47.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.077+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 --> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50680623e0 con 0x7f50500aba40 2024-02-09T22:49:47.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.077+0000 7f50577fe700 1 -- 172.21.15.3:0/3290946773 <== osd.5 v2:172.21.15.89:6808/708499709 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f50680623e0 con 0x7f50500aba40 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] conn(0x7f50500aba40 msgr2=0x7f50500ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] conn(0x7f50500aba40 0x7f50500ade60 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50500a2260 msgr2=0x7f50500a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50500a2260 0x7f50500a4720 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f505c00b5c0 tx=0x7f505c005fb0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 msgr2=0x7f5068196fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 --2- 172.21.15.3:0/3290946773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5068100c20 0x7f5068196fd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f505800b530 tx=0x7f505800b8f0 comp rx=0 tx=0).stop 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f5066d9d700 1 -- 172.21.15.3:0/3290946773 reap_dead start 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 shutdown_connections 2024-02-09T22:49:47.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 >> 172.21.15.3:0/3290946773 conn(0x7f5068077b50 msgr2=0x7f50680ff900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:47.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 shutdown_connections 2024-02-09T22:49:47.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:47.078+0000 7f506cd12700 1 -- 172.21.15.3:0/3290946773 wait complete. 2024-02-09T22:49:48.007 INFO:teuthology.orchestra.run.smithi003.stdout:34359738443 2024-02-09T22:49:48.008 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-02-09T22:49:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 -- 172.21.15.3:0/3071007629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144101860 msgr2=0x7f7144105820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 --2- 172.21.15.3:0/3071007629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144101860 0x7f7144105820 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7140009990 tx=0x7f7140009ca0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 -- 172.21.15.3:0/3071007629 shutdown_connections 2024-02-09T22:49:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 --2- 172.21.15.3:0/3071007629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144105f00 0x7f714410dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 --2- 172.21.15.3:0/3071007629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7144100eb0 0x7f7144101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 --2- 172.21.15.3:0/3071007629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144101860 0x7f7144105820 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 -- 172.21.15.3:0/3071007629 >> 172.21.15.3:0/3071007629 conn(0x7f7144076c70 msgr2=0x7f7144077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 -- 172.21.15.3:0/3071007629 shutdown_connections 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.365+0000 7f714c796700 1 -- 172.21.15.3:0/3071007629 wait complete. 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 Processor -- start 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 -- start start 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144100eb0 0x7f7144198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7144101860 0x7f7144198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144105f00 0x7f714419cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714419d130 con 0x7f7144100eb0 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714419d2a0 con 0x7f7144101860 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714c796700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714419d460 con 0x7f7144105f00 2024-02-09T22:49:48.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144100eb0 0x7f7144198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144100eb0 0x7f7144198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60820/0 (socket says 172.21.15.3:60820) 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 -- 172.21.15.3:0/4046428096 learned_addr learned my addr 172.21.15.3:0/4046428096 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714ad33700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144105f00 0x7f714419cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f7149d31700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7144101860 0x7f7144198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 -- 172.21.15.3:0/4046428096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144105f00 msgr2=0x7f714419cb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144105f00 0x7f714419cb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 -- 172.21.15.3:0/4046428096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7144101860 msgr2=0x7f7144198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7144101860 0x7f7144198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714a532700 1 -- 172.21.15.3:0/4046428096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71400095f0 con 0x7f7144100eb0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.366+0000 7f714ad33700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7144105f00 0x7f714419cb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f714a532700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144100eb0 0x7f7144198250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f713800c920 tx=0x7f713800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7138007b00 con 0x7f7144100eb0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7138004d10 con 0x7f7144100eb0 2024-02-09T22:49:48.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714419d740 con 0x7f7144100eb0 2024-02-09T22:49:48.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71380056c0 con 0x7f7144100eb0 2024-02-09T22:49:48.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71441a4fa0 con 0x7f7144100eb0 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.367+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7144110f00 con 0x7f7144100eb0 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.369+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f7138007c60 con 0x7f7144100eb0 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.369+0000 7f71377fe700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f71300a2260 0x7f71300a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.369+0000 7f7149d31700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f71300a2260 0x7f71300a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f71380cdc60 con 0x7f7144100eb0 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f71377fe700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] conn(0x7f71300aba40 0x7f71300ade60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 --> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f71300ae510 con 0x7f71300aba40 2024-02-09T22:49:48.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71380ce100 con 0x7f7144100eb0 2024-02-09T22:49:48.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f714ad33700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] conn(0x7f71300aba40 0x7f71300ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:48.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f7149d31700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f71300a2260 0x7f71300a4720 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7140006010 tx=0x7f714001a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:48.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.370+0000 7f714ad33700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] conn(0x7f71300aba40 0x7f71300ade60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:48.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.371+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== osd.9 v2:172.21.15.3:6826/2163219868 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f71300ae510 con 0x7f71300aba40 2024-02-09T22:49:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.388+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 --> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f71440623b0 con 0x7f71300aba40 2024-02-09T22:49:48.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f71377fe700 1 -- 172.21.15.3:0/4046428096 <== osd.9 v2:172.21.15.3:6826/2163219868 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f71440623b0 con 0x7f71300aba40 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] conn(0x7f71300aba40 msgr2=0x7f71300ade60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] conn(0x7f71300aba40 0x7f71300ade60 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f71300a2260 msgr2=0x7f71300a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f71300a2260 0x7f71300a4720 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7140006010 tx=0x7f714001a040 comp rx=0 tx=0).stop 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144100eb0 msgr2=0x7f7144198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 --2- 172.21.15.3:0/4046428096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7144100eb0 0x7f7144198250 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f713800c920 tx=0x7f713800cce0 comp rx=0 tx=0).stop 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714ad33700 1 -- 172.21.15.3:0/4046428096 reap_dead start 2024-02-09T22:49:48.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 shutdown_connections 2024-02-09T22:49:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 >> 172.21.15.3:0/4046428096 conn(0x7f7144076c70 msgr2=0x7f714406a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 shutdown_connections 2024-02-09T22:49:48.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:48.389+0000 7f714c796700 1 -- 172.21.15.3:0/4046428096 wait complete. 2024-02-09T22:49:48.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:48 smithi003 ceph-mon[32678]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-02-09T22:49:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:48 smithi064 ceph-mon[36638]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-02-09T22:49:48.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:48 smithi089 ceph-mon[36665]: pgmap v307: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s 2024-02-09T22:49:49.061 INFO:teuthology.orchestra.run.smithi003.stdout:253403070499 2024-02-09T22:49:49.062 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-02-09T22:49:49.472 INFO:teuthology.orchestra.run.smithi003.stdout:287762808863 2024-02-09T22:49:49.472 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-02-09T22:49:49.967 INFO:teuthology.orchestra.run.smithi003.stdout:219043332135 2024-02-09T22:49:49.967 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 -- 172.21.15.3:0/305313201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 msgr2=0x7f5ebc10daa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 --2- 172.21.15.3:0/305313201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc10daa0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5eb400b0a0 tx=0x7f5eb400b3b0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 -- 172.21.15.3:0/305313201 shutdown_connections 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 --2- 172.21.15.3:0/305313201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc10daa0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 --2- 172.21.15.3:0/305313201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ebc106c60 0x7f5ebc107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 --2- 172.21.15.3:0/305313201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc100c20 0x7f5ebc101080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 -- 172.21.15.3:0/305313201 >> 172.21.15.3:0/305313201 conn(0x7f5ebc077b50 msgr2=0x7f5ebc077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 -- 172.21.15.3:0/305313201 shutdown_connections 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.419+0000 7f5ec0977700 1 -- 172.21.15.3:0/305313201 wait complete. 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 Processor -- start 2024-02-09T22:49:50.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 -- start start 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc100c20 0x7f5ebc1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ebc106c60 0x7f5ebc19cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc19d280 con 0x7f5ebc1015c0 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc19d3f0 con 0x7f5ebc106c60 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ec0977700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ebc19d5b0 con 0x7f5ebc100c20 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc198930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60840/0 (socket says 172.21.15.3:60840) 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 -- 172.21.15.3:0/1742799034 learned_addr learned my addr 172.21.15.3:0/1742799034 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5ebad9d700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ebc106c60 0x7f5ebc19cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eba59c700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc100c20 0x7f5ebc1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 -- 172.21.15.3:0/1742799034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc100c20 msgr2=0x7f5ebc1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ebc100c20 0x7f5ebc1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 -- 172.21.15.3:0/1742799034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ebc106c60 msgr2=0x7f5ebc19cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ebc106c60 0x7f5ebc19cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.420+0000 7f5eb9d9b700 1 -- 172.21.15.3:0/1742799034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb4009d00 con 0x7f5ebc1015c0 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.421+0000 7f5eb9d9b700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc198930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000d980 tx=0x7f5eb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.421+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0004a30 con 0x7f5ebc1015c0 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.421+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ebc19d890 con 0x7f5ebc1015c0 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.421+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5eb0005500 con 0x7f5ebc1015c0 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.421+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000fc80 con 0x7f5ebc1015c0 2024-02-09T22:49:50.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.421+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ebc1a50f0 con 0x7f5ebc1015c0 2024-02-09T22:49:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.422+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5ebc110fd0 con 0x7f5ebc1015c0 2024-02-09T22:49:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.423+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f5eb0005000 con 0x7f5ebc1015c0 2024-02-09T22:49:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.423+0000 7f5eab7fe700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ea40a21e0 0x7f5ea40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.423+0000 7f5eba59c700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ea40a21e0 0x7f5ea40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5eb00ce0d0 con 0x7f5ebc1015c0 2024-02-09T22:49:50.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5eab7fe700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] conn(0x7f5ea40ab9c0 0x7f5ea40adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 --> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5ea40ae490 con 0x7f5ea40ab9c0 2024-02-09T22:49:50.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5eb00ce570 con 0x7f5ebc1015c0 2024-02-09T22:49:50.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5ebad9d700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] conn(0x7f5ea40ab9c0 0x7f5ea40adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5eba59c700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ea40a21e0 0x7f5ea40a46a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009a10 tx=0x7f5eac009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:50.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.424+0000 7f5ebad9d700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] conn(0x7f5ea40ab9c0 0x7f5ea40adde0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:50.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.425+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== osd.6 v2:172.21.15.3:6818/1303593428 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f5ea40ae490 con 0x7f5ea40ab9c0 2024-02-09T22:49:50.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.441+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 --> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5ebc104ce0 con 0x7f5ea40ab9c0 2024-02-09T22:49:50.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.441+0000 7f5eab7fe700 1 -- 172.21.15.3:0/1742799034 <== osd.6 v2:172.21.15.3:6818/1303593428 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f5ebc104ce0 con 0x7f5ea40ab9c0 2024-02-09T22:49:50.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] conn(0x7f5ea40ab9c0 msgr2=0x7f5ea40adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] conn(0x7f5ea40ab9c0 0x7f5ea40adde0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ea40a21e0 msgr2=0x7f5ea40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5ea40a21e0 0x7f5ea40a46a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009a10 tx=0x7f5eac009190 comp rx=0 tx=0).stop 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 msgr2=0x7f5ebc198930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 --2- 172.21.15.3:0/1742799034 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ebc1015c0 0x7f5ebc198930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000d980 tx=0x7f5eb000dc90 comp rx=0 tx=0).stop 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ebad9d700 1 -- 172.21.15.3:0/1742799034 reap_dead start 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 shutdown_connections 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 >> 172.21.15.3:0/1742799034 conn(0x7f5ebc077b50 msgr2=0x7f5ebc0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 shutdown_connections 2024-02-09T22:49:50.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.442+0000 7f5ec0977700 1 -- 172.21.15.3:0/1742799034 wait complete. 2024-02-09T22:49:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 -- 172.21.15.3:0/2841498376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 msgr2=0x7fcbb810ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/2841498376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb810ec40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000b0d0 tx=0x7fcbb000b3e0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 -- 172.21.15.3:0/2841498376 shutdown_connections 2024-02-09T22:49:50.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/2841498376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb810ec40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/2841498376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbb8107e70 0x7fcbb8108250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/2841498376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbb8100270 0x7fcbb81006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 -- 172.21.15.3:0/2841498376 >> 172.21.15.3:0/2841498376 conn(0x7fcbb80771f0 msgr2=0x7fcbb80fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.611+0000 7fcbbf7db700 1 -- 172.21.15.3:0/2841498376 shutdown_connections 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 -- 172.21.15.3:0/2841498376 wait complete. 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 Processor -- start 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 -- start start 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbb8100270 0x7fcbb8198490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb81989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbb8107e70 0x7fcbb819cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb819d320 con 0x7fcbb8100c30 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb819d490 con 0x7fcbb8100270 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbf7db700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb819d650 con 0x7fcbb8107e70 2024-02-09T22:49:50.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbcd76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb81989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.612+0000 7fcbbcd76700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb81989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60856/0 (socket says 172.21.15.3:60856) 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 -- 172.21.15.3:0/1293826957 learned_addr learned my addr 172.21.15.3:0/1293826957 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbd577700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbb8100270 0x7fcbb8198490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbdd78700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbb8107e70 0x7fcbb819cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 -- 172.21.15.3:0/1293826957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbb8107e70 msgr2=0x7fcbb819cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbb8107e70 0x7fcbb819cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 -- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbb8100270 msgr2=0x7fcbb8198490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbb8100270 0x7fcbb8198490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 -- 172.21.15.3:0/1293826957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb0009d00 con 0x7fcbb8100c30 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbdd78700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcbb8107e70 0x7fcbb819cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbd577700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcbb8100270 0x7fcbb8198490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbcd76700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb81989d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcba800d950 tx=0x7fcba800dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba8009b40 con 0x7fcbb8100c30 2024-02-09T22:49:50.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb819d930 con 0x7fcbb8100c30 2024-02-09T22:49:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcba8009ca0 con 0x7fcbb8100c30 2024-02-09T22:49:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba800f6b0 con 0x7fcbb8100c30 2024-02-09T22:49:50.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.613+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb81a5190 con 0x7fcbb8100c30 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.614+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcbb8112130 con 0x7fcbb8100c30 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fcba8010460 con 0x7fcbb8100c30 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbae7fc700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcba40a21e0 0x7fcba40a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcba80ce000 con 0x7fcbb8100c30 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbbd577700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcba40a21e0 0x7fcba40a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbae7fc700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] conn(0x7fcba40ab9c0 0x7fcba40adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 --> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcba40ae490 con 0x7fcba40ab9c0 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.616+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcba80ce4a0 con 0x7fcbb8100c30 2024-02-09T22:49:50.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.617+0000 7fcbbd577700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcba40a21e0 0x7fcba40a46a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005e90 tx=0x7fcbb4005d20 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.617+0000 7fcbbdd78700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] conn(0x7fcba40ab9c0 0x7fcba40adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:49:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.617+0000 7fcbbdd78700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] conn(0x7fcba40ab9c0 0x7fcba40adde0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:49:50.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.618+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== osd.1 v2:172.21.15.64:6800/368704973 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcba40ae490 con 0x7fcba40ab9c0 2024-02-09T22:49:50.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 --> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcbb80623b0 con 0x7fcba40ab9c0 2024-02-09T22:49:50.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbae7fc700 1 -- 172.21.15.3:0/1293826957 <== osd.1 v2:172.21.15.64:6800/368704973 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcbb80623b0 con 0x7fcba40ab9c0 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] conn(0x7fcba40ab9c0 msgr2=0x7fcba40adde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] conn(0x7fcba40ab9c0 0x7fcba40adde0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcba40a21e0 msgr2=0x7fcba40a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcba40a21e0 0x7fcba40a46a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4005e90 tx=0x7fcbb4005d20 comp rx=0 tx=0).stop 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 msgr2=0x7fcbb81989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbf7db700 1 --2- 172.21.15.3:0/1293826957 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcbb8100c30 0x7fcbb81989d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fcba800d950 tx=0x7fcba800dd10 comp rx=0 tx=0).stop 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.635+0000 7fcbbdd78700 1 -- 172.21.15.3:0/1293826957 reap_dead start 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.636+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 shutdown_connections 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.636+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 >> 172.21.15.3:0/1293826957 conn(0x7fcbb80771f0 msgr2=0x7fcbb8104af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.636+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 shutdown_connections 2024-02-09T22:49:50.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:49:50.636+0000 7fcbbf7db700 1 -- 172.21.15.3:0/1293826957 wait complete. 2024-02-09T22:49:50.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:50 smithi003 ceph-mon[32678]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-02-09T22:49:50.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:50 smithi064 ceph-mon[36638]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-02-09T22:49:50.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:50 smithi089 ceph-mon[36665]: pgmap v308: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 9 op/s 2024-02-09T22:49:50.984 INFO:teuthology.orchestra.run.smithi003.stdout:163208757300 2024-02-09T22:49:50.984 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-02-09T22:49:51.744 INFO:teuthology.orchestra.run.smithi003.stdout:77309411393 2024-02-09T22:49:51.745 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-02-09T22:49:52.373 INFO:teuthology.orchestra.run.smithi003.stdout:55834574920 2024-02-09T22:49:52.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-02-09T22:49:52.620 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:52 smithi003 ceph-mon[32678]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-02-09T22:49:52.637 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:52.639 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:52.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:52 smithi064 ceph-mon[36638]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-02-09T22:49:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:52 smithi089 ceph-mon[36665]: pgmap v309: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-02-09T22:49:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:53.348 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:53.348 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:53.349 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:53.349 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:53.833 INFO:teuthology.orchestra.run.smithi003.stdout:197568495663 2024-02-09T22:49:53.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-02-09T22:49:54.106 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:49:55.368 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:55 smithi003 ceph-mon[32678]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:49:55.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:55 smithi064 ceph-mon[36638]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:49:55.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:55 smithi089 ceph-mon[36665]: pgmap v310: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:49:56.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:56 smithi003 ceph-mon[32678]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:49:56.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:56 smithi064 ceph-mon[36638]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:49:56.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:56 smithi089 ceph-mon[36665]: pgmap v311: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:49:58.814 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:49:58 smithi003 ceph-mon[32678]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:49:58.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:49:58 smithi064 ceph-mon[36638]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:49:58.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:49:58 smithi089 ceph-mon[36665]: pgmap v312: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 -- 172.21.15.3:0/1029727682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 msgr2=0x7efd84101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 --2- 172.21.15.3:0/1029727682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd84101080 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7efd78009990 tx=0x7efd78009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 -- 172.21.15.3:0/1029727682 shutdown_connections 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 --2- 172.21.15.3:0/1029727682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd84106cb0 0x7efd84107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 --2- 172.21.15.3:0/1029727682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd841015c0 0x7efd8410da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 --2- 172.21.15.3:0/1029727682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd84101080 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 -- 172.21.15.3:0/1029727682 >> 172.21.15.3:0/1029727682 conn(0x7efd84077c30 msgr2=0x7efd84078040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:00.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 -- 172.21.15.3:0/1029727682 shutdown_connections 2024-02-09T22:50:00.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.039+0000 7efd8973f700 1 -- 172.21.15.3:0/1029727682 wait complete. 2024-02-09T22:50:00.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 Processor -- start 2024-02-09T22:50:00.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 -- start start 2024-02-09T22:50:00.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd841982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:00.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd841015c0 0x7efd841987e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:00.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd84106cb0 0x7efd8419cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419d180 con 0x7efd84100c00 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419d2f0 con 0x7efd841015c0 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd8973f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419d4b0 con 0x7efd84106cb0 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd82ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd841982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.040+0000 7efd82ffd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd841982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38158/0 (socket says 172.21.15.3:38158) 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 -- 172.21.15.3:0/2952148988 learned_addr learned my addr 172.21.15.3:0/2952148988 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd837fe700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd84106cb0 0x7efd8419cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 -- 172.21.15.3:0/2952148988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd84106cb0 msgr2=0x7efd8419cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd84106cb0 0x7efd8419cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 -- 172.21.15.3:0/2952148988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd841015c0 msgr2=0x7efd841987e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd841015c0 0x7efd841987e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 -- 172.21.15.3:0/2952148988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd780095f0 con 0x7efd84100c00 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd82ffd700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd841982a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd7400b530 tx=0x7efd7400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:00.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74011830 con 0x7efd84100c00 2024-02-09T22:50:00.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8419d790 con 0x7efd84100c00 2024-02-09T22:50:00.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd74011e70 con 0x7efd84100c00 2024-02-09T22:50:00.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7400f2e0 con 0x7efd84100c00 2024-02-09T22:50:00.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.041+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd841a4ff0 con 0x7efd84100c00 2024-02-09T22:50:00.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.042+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd840689e0 con 0x7efd84100c00 2024-02-09T22:50:00.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.043+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7efd740105f0 con 0x7efd84100c00 2024-02-09T22:50:00.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.044+0000 7efd6bfff700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd6c0a21e0 0x7efd6c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:00.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.044+0000 7efd827fc700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd6c0a21e0 0x7efd6c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:00.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.044+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efd740cdc10 con 0x7efd84100c00 2024-02-09T22:50:00.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.044+0000 7efd827fc700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd6c0a21e0 0x7efd6c0a46a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd780098e0 tx=0x7efd78005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:00.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.048+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efd74096c40 con 0x7efd84100c00 2024-02-09T22:50:00.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.253+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7efd84110da0 con 0x7efd84100c00 2024-02-09T22:50:00.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.254+0000 7efd6bfff700 1 -- 172.21.15.3:0/2952148988 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7efd74010380 con 0x7efd84100c00 2024-02-09T22:50:00.257 INFO:teuthology.orchestra.run.smithi003.stdout:347892351000 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd6c0a21e0 msgr2=0x7efd6c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd6c0a21e0 0x7efd6c0a46a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd780098e0 tx=0x7efd78005fb0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 msgr2=0x7efd841982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd841982a0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7efd7400b530 tx=0x7efd7400b8f0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 shutdown_connections 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efd6c0a21e0 0x7efd6c0a46a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd84106cb0 0x7efd8419cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd84100c00 0x7efd841982a0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 --2- 172.21.15.3:0/2952148988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd841015c0 0x7efd841987e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:00.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 >> 172.21.15.3:0/2952148988 conn(0x7efd84077c30 msgr2=0x7efd840ffef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:00.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 shutdown_connections 2024-02-09T22:50:00.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:00.256+0000 7efd8973f700 1 -- 172.21.15.3:0/2952148988 wait complete. 2024-02-09T22:50:00.288 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:00 smithi003 ceph-mon[32678]: overall HEALTH_OK 2024-02-09T22:50:00.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:00 smithi064 ceph-mon[36638]: overall HEALTH_OK 2024-02-09T22:50:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:00 smithi089 ceph-mon[36665]: overall HEALTH_OK 2024-02-09T22:50:01.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2952148988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T22:50:01.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:01 smithi003 ceph-mon[32678]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-02-09T22:50:01.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:01 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2952148988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T22:50:01.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:01 smithi064 ceph-mon[36638]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-02-09T22:50:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:01 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2952148988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-02-09T22:50:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:01 smithi089 ceph-mon[36665]: pgmap v313: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-02-09T22:50:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 -- 172.21.15.3:0/3175838362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc074490 msgr2=0x7fddbc074910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 --2- 172.21.15.3:0/3175838362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc074490 0x7fddbc074910 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fddac009990 tx=0x7fddac009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 -- 172.21.15.3:0/3175838362 shutdown_connections 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 --2- 172.21.15.3:0/3175838362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddbc1129b0 0x7fddbc114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 --2- 172.21.15.3:0/3175838362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc075d50 0x7fddbc073f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 --2- 172.21.15.3:0/3175838362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc074490 0x7fddbc074910 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 -- 172.21.15.3:0/3175838362 >> 172.21.15.3:0/3175838362 conn(0x7fddbc078e50 msgr2=0x7fddbc079260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 -- 172.21.15.3:0/3175838362 shutdown_connections 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 -- 172.21.15.3:0/3175838362 wait complete. 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 Processor -- start 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.973+0000 7fddc2cec700 1 -- start start 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddbc074490 0x7fddbc1a4650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 0x7fddbc1a4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc1129b0 0x7fddbc19e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc19eed0 con 0x7fddbc075d50 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc19f040 con 0x7fddbc1129b0 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddbc19f200 con 0x7fddbc074490 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 0x7fddbc1a4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 0x7fddbc1a4b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38166/0 (socket says 172.21.15.3:38166) 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 -- 172.21.15.3:0/1580408993 learned_addr learned my addr 172.21.15.3:0/1580408993 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc0a88700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddbc074490 0x7fddbc1a4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc1289700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc1129b0 0x7fddbc19e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 -- 172.21.15.3:0/1580408993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddbc074490 msgr2=0x7fddbc1a4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddbc074490 0x7fddbc1a4650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 -- 172.21.15.3:0/1580408993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc1129b0 msgr2=0x7fddbc19e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc1129b0 0x7fddbc19e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 -- 172.21.15.3:0/1580408993 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddac0095f0 con 0x7fddbc075d50 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc1289700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc1129b0 0x7fddbc19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb3fff700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 0x7fddbc1a4b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fddac0098e0 tx=0x7fddac00f690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac01d070 con 0x7fddbc075d50 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddac003680 con 0x7fddbc075d50 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddbc19f480 con 0x7fddbc075d50 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.974+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddac017650 con 0x7fddbc075d50 2024-02-09T22:50:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.975+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddbc1ab5d0 con 0x7fddbc075d50 2024-02-09T22:50:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.976+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddbc075600 con 0x7fddbc075d50 2024-02-09T22:50:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.977+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fddac022470 con 0x7fddbc075d50 2024-02-09T22:50:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.978+0000 7fddb1ffb700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdda40a2260 0x7fdda40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.978+0000 7fddc0a88700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdda40a2260 0x7fdda40a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.978+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fddac0cf1c0 con 0x7fddbc075d50 2024-02-09T22:50:01.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.979+0000 7fddc0a88700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdda40a2260 0x7fdda40a4720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fddb8007be0 tx=0x7fddb80073d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:01.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:01.982+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fddac098270 con 0x7fddbc075d50 2024-02-09T22:50:02.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.185+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fddbc0689e0 con 0x7fddbc075d50 2024-02-09T22:50:02.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.185+0000 7fddb1ffb700 1 -- 172.21.15.3:0/1580408993 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fddac027090 con 0x7fddbc075d50 2024-02-09T22:50:02.189 INFO:teuthology.orchestra.run.smithi003.stdout:55834574922 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdda40a2260 msgr2=0x7fdda40a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdda40a2260 0x7fdda40a4720 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fddb8007be0 tx=0x7fddb80073d0 comp rx=0 tx=0).stop 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 msgr2=0x7fddbc1a4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 0x7fddbc1a4b90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fddac0098e0 tx=0x7fddac00f690 comp rx=0 tx=0).stop 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 shutdown_connections 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdda40a2260 0x7fdda40a4720 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fddbc074490 0x7fddbc1a4650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fddbc1129b0 0x7fddbc19e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 --2- 172.21.15.3:0/1580408993 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fddbc075d50 0x7fddbc1a4b90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:02.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 >> 172.21.15.3:0/1580408993 conn(0x7fddbc078e50 msgr2=0x7fddbc111770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 shutdown_connections 2024-02-09T22:50:02.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:02.188+0000 7fddc2cec700 1 -- 172.21.15.3:0/1580408993 wait complete. 2024-02-09T22:50:02.321 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:02.329 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:02.441 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:02 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1580408993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T22:50:02.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1580408993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T22:50:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:02 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1580408993' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-02-09T22:50:03.372 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:03 smithi003 ceph-mon[32678]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 -- 172.21.15.3:0/814154985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 msgr2=0x7fce08101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/814154985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce08101080 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fcdf4009990 tx=0x7fcdf4009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 -- 172.21.15.3:0/814154985 shutdown_connections 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/814154985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce081015c0 0x7fce0810daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/814154985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce08106c60 0x7fce08107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/814154985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce08101080 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 -- 172.21.15.3:0/814154985 >> 172.21.15.3:0/814154985 conn(0x7fce08077b50 msgr2=0x7fce08077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 -- 172.21.15.3:0/814154985 shutdown_connections 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 -- 172.21.15.3:0/814154985 wait complete. 2024-02-09T22:50:03.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 Processor -- start 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.388+0000 7fce0ee7b700 1 -- start start 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce081028d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce081015c0 0x7fce08102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08106c60 0x7fce08103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce081a65c0 con 0x7fce08100c20 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce081a6730 con 0x7fce081015c0 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce081a68f0 con 0x7fce08106c60 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce081028d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce081028d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38176/0 (socket says 172.21.15.3:38176) 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 -- 172.21.15.3:0/2103655305 learned_addr learned my addr 172.21.15.3:0/2103655305 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce07fff700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce081015c0 0x7fce08102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0d418700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08106c60 0x7fce08103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 -- 172.21.15.3:0/2103655305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08106c60 msgr2=0x7fce08103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08106c60 0x7fce08103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 -- 172.21.15.3:0/2103655305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce081015c0 msgr2=0x7fce08102e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce081015c0 0x7fce08102e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 -- 172.21.15.3:0/2103655305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40095f0 con 0x7fce08100c20 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0cc17700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce081028d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00d950 tx=0x7fcdfc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00f890 con 0x7fce08100c20 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdfc010460 con 0x7fce08100c20 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce081a6bd0 con 0x7fce08100c20 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.389+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdfc00e610 con 0x7fce08100c20 2024-02-09T22:50:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.390+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce081a70d0 con 0x7fce08100c20 2024-02-09T22:50:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.391+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce080689e0 con 0x7fce08100c20 2024-02-09T22:50:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.392+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fcdfc010960 con 0x7fce08100c20 2024-02-09T22:50:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.392+0000 7fce05ffb700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcdf80a21e0 0x7fcdf80a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.392+0000 7fce07fff700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcdf80a21e0 0x7fcdf80a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.392+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcdfc0cd9a0 con 0x7fce08100c20 2024-02-09T22:50:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.393+0000 7fce07fff700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcdf80a21e0 0x7fcdf80a46a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce080780c0 tx=0x7fcdf400b560 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.397+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fcdfc0969c0 con 0x7fce08100c20 2024-02-09T22:50:03.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.602+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fce08110d40 con 0x7fce08100c20 2024-02-09T22:50:03.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.602+0000 7fce05ffb700 1 -- 172.21.15.3:0/2103655305 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcdfc09a080 con 0x7fce08100c20 2024-02-09T22:50:03.609 INFO:teuthology.orchestra.run.smithi003.stdout:219043332139 2024-02-09T22:50:03.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.604+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcdf80a21e0 msgr2=0x7fcdf80a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:03.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.604+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcdf80a21e0 0x7fcdf80a46a0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fce080780c0 tx=0x7fcdf400b560 comp rx=0 tx=0).stop 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.604+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 msgr2=0x7fce081028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.604+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce081028d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fcdfc00d950 tx=0x7fcdfc00dc60 comp rx=0 tx=0).stop 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 shutdown_connections 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcdf80a21e0 0x7fcdf80a46a0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fce08106c60 0x7fce08103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fce08100c20 0x7fce081028d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 --2- 172.21.15.3:0/2103655305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fce081015c0 0x7fce08102e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:03.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 >> 172.21.15.3:0/2103655305 conn(0x7fce08077b50 msgr2=0x7fce080fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:03.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 shutdown_connections 2024-02-09T22:50:03.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:03.605+0000 7fce0ee7b700 1 -- 172.21.15.3:0/2103655305 wait complete. 2024-02-09T22:50:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:03 smithi064 ceph-mon[36638]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-02-09T22:50:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:03 smithi089 ceph-mon[36665]: pgmap v314: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-02-09T22:50:04.514 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:04 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2103655305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T22:50:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:04 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2103655305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T22:50:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:04 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2103655305' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-02-09T22:50:05.554 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:05 smithi003 ceph-mon[32678]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:05.587 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574920 got 55834574922 for osd.1 2024-02-09T22:50:05.587 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:05 smithi064 ceph-mon[36638]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:05 smithi089 ceph-mon[36665]: pgmap v315: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:06.495 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:06 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:50:06.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.517+0000 7ff081460700 1 -- 172.21.15.3:0/3436520592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c101da0 msgr2=0x7ff07c102220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.517+0000 7ff081460700 1 --2- 172.21.15.3:0/3436520592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c101da0 0x7ff07c102220 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ff0700099c0 tx=0x7ff070009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 -- 172.21.15.3:0/3436520592 shutdown_connections 2024-02-09T22:50:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 --2- 172.21.15.3:0/3436520592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c10ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 --2- 172.21.15.3:0/3436520592 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff07c107db0 0x7ff07c108190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 --2- 172.21.15.3:0/3436520592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c101da0 0x7ff07c102220 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 -- 172.21.15.3:0/3436520592 >> 172.21.15.3:0/3436520592 conn(0x7ff07c078e80 msgr2=0x7ff07c079290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 -- 172.21.15.3:0/3436520592 shutdown_connections 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 -- 172.21.15.3:0/3436520592 wait complete. 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 Processor -- start 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.518+0000 7ff081460700 1 -- start start 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff081460700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff07c101da0 0x7ff07c1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff081460700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff081460700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c107db0 0x7ff07c19cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff081460700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c19d150 con 0x7ff07c107db0 2024-02-09T22:50:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff081460700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c19d2c0 con 0x7ff07c101da0 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff081460700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff07c19d480 con 0x7ff07c102760 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c107db0 0x7ff07c19cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c198800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:47002/0 (socket says 172.21.15.3:47002) 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 -- 172.21.15.3:0/1844235487 learned_addr learned my addr 172.21.15.3:0/1844235487 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07b7fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c107db0 0x7ff07c19cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49908/0 (socket says 172.21.15.3:49908) 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07affd700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff07c101da0 0x7ff07c1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 -- 172.21.15.3:0/1844235487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff07c101da0 msgr2=0x7ff07c1982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff07c101da0 0x7ff07c1982c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 -- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c107db0 msgr2=0x7ff07c19cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c107db0 0x7ff07c19cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 -- 172.21.15.3:0/1844235487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff074009c40 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff07a7fc700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c198800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff070006010 tx=0x7ff070005280 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.519+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07001d070 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.520+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0700095f0 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.520+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff070003680 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.520+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000fbe0 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.520+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07c1a4df0 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.521+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff07c10a290 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.522+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7ff07000b730 con 0x7ff07c102760 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.522+0000 7ff063fff700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0640a21e0 0x7ff0640a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.523+0000 7ff07affd700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0640a21e0 0x7ff0640a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.523+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff0700cefe0 con 0x7ff07c102760 2024-02-09T22:50:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.523+0000 7ff07affd700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0640a21e0 0x7ff0640a46a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009be0 tx=0x7ff06c009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.531+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff070097fb0 con 0x7ff07c102760 2024-02-09T22:50:06.661 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:06 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:50:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:06 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:50:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 -- 172.21.15.3:0/1708850921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6480699d0 msgr2=0x7fa648069e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 --2- 172.21.15.3:0/1708850921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6480699d0 0x7fa648069e30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fa63c0099c0 tx=0x7fa63c009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 -- 172.21.15.3:0/1708850921 shutdown_connections 2024-02-09T22:50:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 --2- 172.21.15.3:0/1708850921 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa64806a370 0x7fa64810ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 --2- 172.21.15.3:0/1708850921 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa648107d70 0x7fa648108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 --2- 172.21.15.3:0/1708850921 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6480699d0 0x7fa648069e30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 -- 172.21.15.3:0/1708850921 >> 172.21.15.3:0/1708850921 conn(0x7fa648077b50 msgr2=0x7fa648077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 -- 172.21.15.3:0/1708850921 shutdown_connections 2024-02-09T22:50:06.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.602+0000 7fa64d68b700 1 -- 172.21.15.3:0/1708850921 wait complete. 2024-02-09T22:50:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 Processor -- start 2024-02-09T22:50:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 -- start start 2024-02-09T22:50:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 0x7fa6481981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64806a370 0x7fa648198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa648107d70 0x7fa64819cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64819d0c0 con 0x7fa64806a370 2024-02-09T22:50:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64819d230 con 0x7fa648107d70 2024-02-09T22:50:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa64d68b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa64819d3f0 con 0x7fa6480699d0 2024-02-09T22:50:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa646ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 0x7fa6481981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.603+0000 7fa646ffd700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 0x7fa6481981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:47020/0 (socket says 172.21.15.3:47020) 2024-02-09T22:50:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 -- 172.21.15.3:0/406800088 learned_addr learned my addr 172.21.15.3:0/406800088 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:06.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa6477fe700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa648107d70 0x7fa64819cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa6467fc700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64806a370 0x7fa648198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 -- 172.21.15.3:0/406800088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa648107d70 msgr2=0x7fa64819cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa648107d70 0x7fa64819cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 -- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64806a370 msgr2=0x7fa648198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64806a370 0x7fa648198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 -- 172.21.15.3:0/406800088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa63c0095f0 con 0x7fa6480699d0 2024-02-09T22:50:06.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa6467fc700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64806a370 0x7fa648198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.604+0000 7fa646ffd700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 0x7fa6481981e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa63800e9a0 tx=0x7fa63800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.605+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa63800ca10 con 0x7fa6480699d0 2024-02-09T22:50:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.605+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa64819d6d0 con 0x7fa6480699d0 2024-02-09T22:50:06.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.605+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa63800cb70 con 0x7fa6480699d0 2024-02-09T22:50:06.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.605+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa638018680 con 0x7fa6480699d0 2024-02-09T22:50:06.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.605+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6481a4f30 con 0x7fa6480699d0 2024-02-09T22:50:06.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.606+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa64810a290 con 0x7fa6480699d0 2024-02-09T22:50:06.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.607+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fa63800cce0 con 0x7fa6480699d0 2024-02-09T22:50:06.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.608+0000 7fa62ffff700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa6300a21e0 0x7fa6300a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.608+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fa638014070 con 0x7fa6480699d0 2024-02-09T22:50:06.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.608+0000 7fa6467fc700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa6300a21e0 0x7fa6300a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.608+0000 7fa6467fc700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa6300a21e0 0x7fa6300a46a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000c00 tx=0x7fa63c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:06.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.611+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa638096db0 con 0x7fa6480699d0 2024-02-09T22:50:06.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.740+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7ff07c1995d0 con 0x7ff07c102760 2024-02-09T22:50:06.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.740+0000 7ff063fff700 1 -- 172.21.15.3:0/1844235487 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7ff07009b670 con 0x7ff07c102760 2024-02-09T22:50:06.744 INFO:teuthology.orchestra.run.smithi003.stdout:287762808866 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0640a21e0 msgr2=0x7ff0640a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0640a21e0 0x7ff0640a46a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009be0 tx=0x7ff06c009260 comp rx=0 tx=0).stop 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 msgr2=0x7ff07c198800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c198800 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff070006010 tx=0x7ff070005280 comp rx=0 tx=0).stop 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 shutdown_connections 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0640a21e0 0x7ff0640a46a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff07c107db0 0x7ff07c19cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff07c101da0 0x7ff07c1982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 --2- 172.21.15.3:0/1844235487 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff07c102760 0x7ff07c198800 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.742+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 >> 172.21.15.3:0/1844235487 conn(0x7ff07c078e80 msgr2=0x7ff07c100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:06.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.743+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 shutdown_connections 2024-02-09T22:50:06.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.743+0000 7ff081460700 1 -- 172.21.15.3:0/1844235487 wait complete. 2024-02-09T22:50:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.821+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7fa648199540 con 0x7fa6480699d0 2024-02-09T22:50:06.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.822+0000 7fa62ffff700 1 -- 172.21.15.3:0/406800088 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fa63809a470 con 0x7fa6480699d0 2024-02-09T22:50:06.825 INFO:teuthology.orchestra.run.smithi003.stdout:313532612637 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa6300a21e0 msgr2=0x7fa6300a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa6300a21e0 0x7fa6300a46a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa63c000c00 tx=0x7fa63c005fb0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 msgr2=0x7fa6481981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 0x7fa6481981e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa63800e9a0 tx=0x7fa63800ecb0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 shutdown_connections 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa6300a21e0 0x7fa6300a46a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6480699d0 0x7fa6481981e0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa648107d70 0x7fa64819cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 --2- 172.21.15.3:0/406800088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa64806a370 0x7fa648198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 >> 172.21.15.3:0/406800088 conn(0x7fa648077b50 msgr2=0x7fa6480ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 shutdown_connections 2024-02-09T22:50:06.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.824+0000 7fa64d68b700 1 -- 172.21.15.3:0/406800088 wait complete. 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 -- 172.21.15.3:0/572263875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e90107d60 msgr2=0x7f1e90108140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 --2- 172.21.15.3:0/572263875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e90107d60 0x7f1e90108140 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1e80009990 tx=0x7f1e80009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 -- 172.21.15.3:0/572263875 shutdown_connections 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 --2- 172.21.15.3:0/572263875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e9006ae40 0x7f1e9010eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 --2- 172.21.15.3:0/572263875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e90107d60 0x7f1e90108140 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 --2- 172.21.15.3:0/572263875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 0x7f1e9006a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 -- 172.21.15.3:0/572263875 >> 172.21.15.3:0/572263875 conn(0x7f1e90077bc0 msgr2=0x7f1e90077fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 -- 172.21.15.3:0/572263875 shutdown_connections 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.971+0000 7f1e9608e700 1 -- 172.21.15.3:0/572263875 wait complete. 2024-02-09T22:50:06.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 Processor -- start 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 -- start start 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 0x7f1e90198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 0x7f1e901987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e90107d60 0x7f1e9019cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9019d150 con 0x7f1e9006ae40 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9019d2c0 con 0x7f1e9006a4a0 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e9608e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e9019d480 con 0x7f1e90107d60 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 0x7f1e901987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.972+0000 7f1e8effd700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 0x7f1e901987b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49960/0 (socket says 172.21.15.3:49960) 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 -- 172.21.15.3:0/476238209 learned_addr learned my addr 172.21.15.3:0/476238209 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8f7fe700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 0x7f1e90198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8ffff700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e90107d60 0x7f1e9019cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 -- 172.21.15.3:0/476238209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e90107d60 msgr2=0x7f1e9019cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e90107d60 0x7f1e9019cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 -- 172.21.15.3:0/476238209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 msgr2=0x7f1e90198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 0x7f1e90198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 -- 172.21.15.3:0/476238209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e800095f0 con 0x7f1e9006ae40 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8f7fe700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 0x7f1e90198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8effd700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 0x7f1e901987b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400d980 tx=0x7f1e8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e84004220 con 0x7f1e9006ae40 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e84007640 con 0x7f1e9006ae40 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e9019d760 con 0x7f1e9006ae40 2024-02-09T22:50:06.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e8400f690 con 0x7f1e9006ae40 2024-02-09T22:50:06.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.973+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e901a4fc0 con 0x7f1e9006ae40 2024-02-09T22:50:06.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.974+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e9010a280 con 0x7f1e9006ae40 2024-02-09T22:50:06.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.975+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f1e84004410 con 0x7f1e9006ae40 2024-02-09T22:50:06.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.976+0000 7f1e8cff9700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1e780a21e0 0x7f1e780a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:06.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.976+0000 7f1e8f7fe700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1e780a21e0 0x7f1e780a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:06.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.976+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1e840cdfc0 con 0x7f1e9006ae40 2024-02-09T22:50:06.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.977+0000 7f1e8f7fe700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1e780a21e0 0x7f1e780a46a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000b5c0 tx=0x7f1e80005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:06.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:06.980+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1e84096ee0 con 0x7f1e9006ae40 2024-02-09T22:50:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.181+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f1e901995d0 con 0x7f1e9006ae40 2024-02-09T22:50:07.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.181+0000 7f1e8cff9700 1 -- 172.21.15.3:0/476238209 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1e84016090 con 0x7f1e9006ae40 2024-02-09T22:50:07.184 INFO:teuthology.orchestra.run.smithi003.stdout:34359738446 2024-02-09T22:50:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1e780a21e0 msgr2=0x7f1e780a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1e780a21e0 0x7f1e780a46a0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1e8000b5c0 tx=0x7f1e80005960 comp rx=0 tx=0).stop 2024-02-09T22:50:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 msgr2=0x7f1e901987b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 0x7f1e901987b0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1e8400d980 tx=0x7f1e8400dc90 comp rx=0 tx=0).stop 2024-02-09T22:50:07.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 shutdown_connections 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1e780a21e0 0x7f1e780a46a0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1e90107d60 0x7f1e9019cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1e9006a4a0 0x7f1e90198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 --2- 172.21.15.3:0/476238209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1e9006ae40 0x7f1e901987b0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 >> 172.21.15.3:0/476238209 conn(0x7f1e90077bc0 msgr2=0x7f1e900ff930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 shutdown_connections 2024-02-09T22:50:07.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:07.183+0000 7f1e9608e700 1 -- 172.21.15.3:0/476238209 wait complete. 2024-02-09T22:50:07.264 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-02-09T22:50:07.264 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-02-09T22:50:07.264 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-02-09T22:50:07.264 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-02-09T22:50:07.535 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1844235487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/406800088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T22:50:07.536 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/476238209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T22:50:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-02-09T22:50:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-02-09T22:50:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-02-09T22:50:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-02-09T22:50:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1844235487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T22:50:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/406800088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T22:50:07.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/476238209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T22:50:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-02-09T22:50:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-02-09T22:50:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]: dispatch 2024-02-09T22:50:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: pgmap v316: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1844235487' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/406800088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-02-09T22:50:07.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/476238209' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-02-09T22:50:08.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-02-09T22:50:08.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-02-09T22:50:08.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-02-09T22:50:08.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-02-09T22:50:08.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-02-09T22:50:08.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: osdmap e87: 12 total, 12 up, 12 in 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:08 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.440 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332135 got 219043332139 for osd.7 2024-02-09T22:50:08.441 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: osdmap e87: 12 total, 12 up, 12 in 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:08 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-02-09T22:50:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-02-09T22:50:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.b", "id": [2, 4]}]': finished 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [5, 9]}]': finished 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 6]}]': finished 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: osdmap e87: 12 total, 12 up, 12 in 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:08 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:08.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350996 got 347892351000 for osd.11 2024-02-09T22:50:08.890 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:09.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:09 smithi003 ceph-mon[32678]: osdmap e88: 12 total, 12 up, 12 in 2024-02-09T22:50:09.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:09 smithi003 ceph-mon[32678]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-02-09T22:50:09.431 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808863 got 287762808866 for osd.9 2024-02-09T22:50:09.431 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 -- 172.21.15.3:0/3901594376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 msgr2=0x7f15fc101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 --2- 172.21.15.3:0/3901594376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc101080 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f15e8009990 tx=0x7f15e8009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 -- 172.21.15.3:0/3901594376 shutdown_connections 2024-02-09T22:50:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 --2- 172.21.15.3:0/3901594376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15fc106c60 0x7f15fc107040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 --2- 172.21.15.3:0/3901594376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 0x7f15fc10daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 --2- 172.21.15.3:0/3901594376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc101080 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 -- 172.21.15.3:0/3901594376 >> 172.21.15.3:0/3901594376 conn(0x7f15fc077b50 msgr2=0x7f15fc077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 -- 172.21.15.3:0/3901594376 shutdown_connections 2024-02-09T22:50:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 -- 172.21.15.3:0/3901594376 wait complete. 2024-02-09T22:50:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 Processor -- start 2024-02-09T22:50:09.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.674+0000 7f160317e700 1 -- start start 2024-02-09T22:50:09.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160317e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc1982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:09.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160317e700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 0x7f15fc198800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160317e700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15fc106c60 0x7f15fc19cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160317e700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc19d1a0 con 0x7f15fc100c20 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160317e700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc19d310 con 0x7f15fc1015c0 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160317e700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15fc19d4d0 con 0x7f15fc106c60 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc1982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc1982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49972/0 (socket says 172.21.15.3:49972) 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 -- 172.21.15.3:0/1301535118 learned_addr learned my addr 172.21.15.3:0/1301535118 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f160171b700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15fc106c60 0x7f15fc19cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f15fbfff700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 0x7f15fc198800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 -- 172.21.15.3:0/1301535118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15fc106c60 msgr2=0x7f15fc19cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15fc106c60 0x7f15fc19cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 -- 172.21.15.3:0/1301535118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 msgr2=0x7f15fc198800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:09.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 0x7f15fc198800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 -- 172.21.15.3:0/1301535118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e80095f0 con 0x7f15fc100c20 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f15fbfff700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 0x7f15fc198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f1600f1a700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc1982c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f15f000b530 tx=0x7f15f000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0011830 con 0x7f15fc100c20 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.675+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15f0011e70 con 0x7f15fc100c20 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.676+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15fc19d7b0 con 0x7f15fc100c20 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.676+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f000f340 con 0x7f15fc100c20 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.676+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15fc1a5010 con 0x7f15fc100c20 2024-02-09T22:50:09.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.677+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15fc0689e0 con 0x7f15fc100c20 2024-02-09T22:50:09.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.678+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f15f00106d0 con 0x7f15fc100c20 2024-02-09T22:50:09.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.678+0000 7f15f9ffb700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f15ec0a21e0 0x7f15ec0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:09.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.678+0000 7f15fbfff700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f15ec0a21e0 0x7f15ec0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:09.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.678+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f15f00cd850 con 0x7f15fc100c20 2024-02-09T22:50:09.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.679+0000 7f15fbfff700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f15ec0a21e0 0x7f15ec0a46a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006010 tx=0x7f15e800b540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:09.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:09 smithi064 ceph-mon[36638]: osdmap e88: 12 total, 12 up, 12 in 2024-02-09T22:50:09.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:09 smithi064 ceph-mon[36638]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-02-09T22:50:09.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.683+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f15f00967e0 con 0x7f15fc100c20 2024-02-09T22:50:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:09 smithi089 ceph-mon[36665]: osdmap e88: 12 total, 12 up, 12 in 2024-02-09T22:50:09.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:09 smithi089 ceph-mon[36665]: pgmap v319: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 11 op/s 2024-02-09T22:50:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.886+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f15fc110d20 con 0x7f15fc100c20 2024-02-09T22:50:09.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.887+0000 7f15f9ffb700 1 -- 172.21.15.3:0/1301535118 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f15f0099ea0 con 0x7f15fc100c20 2024-02-09T22:50:09.890 INFO:teuthology.orchestra.run.smithi003.stdout:141733920829 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f15ec0a21e0 msgr2=0x7f15ec0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f15ec0a21e0 0x7f15ec0a46a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f15e8006010 tx=0x7f15e800b540 comp rx=0 tx=0).stop 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 msgr2=0x7f15fc1982c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc1982c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f15f000b530 tx=0x7f15f000b840 comp rx=0 tx=0).stop 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 shutdown_connections 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f15ec0a21e0 0x7f15ec0a46a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f15fc106c60 0x7f15fc19cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f15fc100c20 0x7f15fc1982c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 --2- 172.21.15.3:0/1301535118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f15fc1015c0 0x7f15fc198800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:09.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 >> 172.21.15.3:0/1301535118 conn(0x7f15fc077b50 msgr2=0x7f15fc0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:09.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 shutdown_connections 2024-02-09T22:50:09.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:09.889+0000 7f160317e700 1 -- 172.21.15.3:0/1301535118 wait complete. 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 -- 172.21.15.3:0/1088490794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c100eb0 msgr2=0x7fdf6c101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 --2- 172.21.15.3:0/1088490794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c101290 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fdf68009990 tx=0x7fdf68009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 -- 172.21.15.3:0/1088490794 shutdown_connections 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 --2- 172.21.15.3:0/1088490794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c105f40 0x7fdf6c10db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 --2- 172.21.15.3:0/1088490794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c101290 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 --2- 172.21.15.3:0/1088490794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 0x7fdf6c105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.210+0000 7fdf7358c700 1 -- 172.21.15.3:0/1088490794 >> 172.21.15.3:0/1088490794 conn(0x7fdf6c076c40 msgr2=0x7fdf6c077050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:10.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.211+0000 7fdf7358c700 1 -- 172.21.15.3:0/1088490794 shutdown_connections 2024-02-09T22:50:10.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.211+0000 7fdf7358c700 1 -- 172.21.15.3:0/1088490794 wait complete. 2024-02-09T22:50:10.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.211+0000 7fdf7358c700 1 Processor -- start 2024-02-09T22:50:10.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.211+0000 7fdf7358c700 1 -- start start 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf7358c700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c1971b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf7358c700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 0x7fdf6c1976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf7358c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 0x7fdf6c19ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf7358c700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c19c090 con 0x7fdf6c105f40 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf7358c700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c19c200 con 0x7fdf6c100eb0 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf7358c700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf6c19c3c0 con 0x7fdf6c101860 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 0x7fdf6c19ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 0x7fdf6c19ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49988/0 (socket says 172.21.15.3:49988) 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 -- 172.21.15.3:0/2757000405 learned_addr learned my addr 172.21.15.3:0/2757000405 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:10.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf70b27700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 0x7fdf6c1976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71328700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c1971b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 -- 172.21.15.3:0/2757000405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 msgr2=0x7fdf6c1976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 0x7fdf6c1976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 -- 172.21.15.3:0/2757000405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c100eb0 msgr2=0x7fdf6c1971b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c1971b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 -- 172.21.15.3:0/2757000405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680095f0 con 0x7fdf6c105f40 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71328700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf70b27700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 0x7fdf6c1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf71b29700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 0x7fdf6c19ba80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdf64007ee0 tx=0x7fdf6400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.212+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64017070 con 0x7fdf6c105f40 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.213+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf6400f040 con 0x7fdf6c105f40 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.213+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c19c6a0 con 0x7fdf6c105f40 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.213+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf64005760 con 0x7fdf6c105f40 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.213+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c1a3f00 con 0x7fdf6c105f40 2024-02-09T22:50:10.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.214+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c109160 con 0x7fdf6c105f40 2024-02-09T22:50:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.215+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fdf640058c0 con 0x7fdf6c105f40 2024-02-09T22:50:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.215+0000 7fdf627fc700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdf580a21e0 0x7fdf580a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.215+0000 7fdf71328700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdf580a21e0 0x7fdf580a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.215+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fdf640cd650 con 0x7fdf6c105f40 2024-02-09T22:50:10.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.216+0000 7fdf71328700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdf580a21e0 0x7fdf580a46a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf68006010 tx=0x7fdf6801a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:10.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.220+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fdf640964d0 con 0x7fdf6c105f40 2024-02-09T22:50:10.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1301535118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T22:50:10.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.425+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fdf6c198510 con 0x7fdf6c105f40 2024-02-09T22:50:10.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.425+0000 7fdf627fc700 1 -- 172.21.15.3:0/2757000405 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fdf64099b90 con 0x7fdf6c105f40 2024-02-09T22:50:10.429 INFO:teuthology.orchestra.run.smithi003.stdout:163208757304 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.427+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdf580a21e0 msgr2=0x7fdf580a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.427+0000 7fdf7358c700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdf580a21e0 0x7fdf580a46a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fdf68006010 tx=0x7fdf6801a040 comp rx=0 tx=0).stop 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.427+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 msgr2=0x7fdf6c19ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.427+0000 7fdf7358c700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 0x7fdf6c19ba80 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fdf64007ee0 tx=0x7fdf6400d3b0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 shutdown_connections 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fdf580a21e0 0x7fdf580a46a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdf6c105f40 0x7fdf6c19ba80 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdf6c100eb0 0x7fdf6c1971b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 --2- 172.21.15.3:0/2757000405 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdf6c101860 0x7fdf6c1976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 >> 172.21.15.3:0/2757000405 conn(0x7fdf6c076c40 msgr2=0x7fdf6c100430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 shutdown_connections 2024-02-09T22:50:10.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.428+0000 7fdf7358c700 1 -- 172.21.15.3:0/2757000405 wait complete. 2024-02-09T22:50:10.540 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738443 got 34359738446 for osd.0 2024-02-09T22:50:10.540 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1301535118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T22:50:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 -- 172.21.15.3:0/1073623465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988101e70 msgr2=0x7f7988102250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 --2- 172.21.15.3:0/1073623465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988101e70 0x7f7988102250 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7984009990 tx=0x7f7984009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 -- 172.21.15.3:0/1073623465 shutdown_connections 2024-02-09T22:50:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 --2- 172.21.15.3:0/1073623465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7988106f50 0x7f798810eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 --2- 172.21.15.3:0/1073623465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988101e70 0x7f7988102250 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 --2- 172.21.15.3:0/1073623465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7988102820 0x7f7988106870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 -- 172.21.15.3:0/1073623465 >> 172.21.15.3:0/1073623465 conn(0x7f79880fd720 msgr2=0x7f79880ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 -- 172.21.15.3:0/1073623465 shutdown_connections 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.721+0000 7f798e79d700 1 -- 172.21.15.3:0/1073623465 wait complete. 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 Processor -- start 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 -- start start 2024-02-09T22:50:10.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7988101e70 0x7f7988198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7988102820 0x7f7988198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 0x7f798819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819cfc0 con 0x7f7988106f50 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819d130 con 0x7f7988102820 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798e79d700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798819d2f0 con 0x7f7988101e70 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 0x7f798819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 0x7f798819c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50006/0 (socket says 172.21.15.3:50006) 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 -- 172.21.15.3:0/3594942645 learned_addr learned my addr 172.21.15.3:0/3594942645 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798cf9a700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7988102820 0x7f7988198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798d79b700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7988101e70 0x7f7988198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 -- 172.21.15.3:0/3594942645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7988101e70 msgr2=0x7f7988198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7988101e70 0x7f7988198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 -- 172.21.15.3:0/3594942645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7988102820 msgr2=0x7f7988198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7988102820 0x7f7988198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.722+0000 7f798df9c700 1 -- 172.21.15.3:0/3594942645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79840095f0 con 0x7f7988106f50 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.723+0000 7f798df9c700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 0x7f798819c9b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f798000b970 tx=0x7f798000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.723+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f798000c930 con 0x7f7988106f50 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.723+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f798819d5d0 con 0x7f7988106f50 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.723+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7980014440 con 0x7f7988106f50 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.723+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79800075d0 con 0x7f7988106f50 2024-02-09T22:50:10.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.723+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79881a4e30 con 0x7f7988106f50 2024-02-09T22:50:10.730 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1301535118' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-02-09T22:50:10.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.724+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798810a180 con 0x7f7988106f50 2024-02-09T22:50:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.725+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f79800145b0 con 0x7f7988106f50 2024-02-09T22:50:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.725+0000 7f797e7fc700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f79740a21e0 0x7f79740a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.725+0000 7f798d79b700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f79740a21e0 0x7f79740a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.725+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7980006e40 con 0x7f7988106f50 2024-02-09T22:50:10.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.726+0000 7f798d79b700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f79740a21e0 0x7f79740a46a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7984006010 tx=0x7f798401a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:10.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.730+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7980096380 con 0x7f7988106f50 2024-02-09T22:50:10.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.935+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f7988199440 con 0x7f7988106f50 2024-02-09T22:50:10.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.935+0000 7f797e7fc700 1 -- 172.21.15.3:0/3594942645 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f7980019020 con 0x7f7988106f50 2024-02-09T22:50:10.938 INFO:teuthology.orchestra.run.smithi003.stdout:107374182465 2024-02-09T22:50:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f79740a21e0 msgr2=0x7f79740a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f79740a21e0 0x7f79740a46a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7984006010 tx=0x7f798401a040 comp rx=0 tx=0).stop 2024-02-09T22:50:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 msgr2=0x7f798819c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 0x7f798819c9b0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f798000b970 tx=0x7f798000bc80 comp rx=0 tx=0).stop 2024-02-09T22:50:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 shutdown_connections 2024-02-09T22:50:10.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f79740a21e0 0x7f79740a46a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7988101e70 0x7f7988198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7988106f50 0x7f798819c9b0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 --2- 172.21.15.3:0/3594942645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7988102820 0x7f7988198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 >> 172.21.15.3:0/3594942645 conn(0x7f79880fd720 msgr2=0x7f79880ff3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 shutdown_connections 2024-02-09T22:50:10.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:10.937+0000 7f798e79d700 1 -- 172.21.15.3:0/3594942645 wait complete. 2024-02-09T22:50:11.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:11 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2757000405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T22:50:11.603 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:11 smithi003 ceph-mon[32678]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T22:50:11.604 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:11 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3594942645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T22:50:11.632 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920824 got 141733920829 for osd.4 2024-02-09T22:50:11.632 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:11 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2757000405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T22:50:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:11 smithi064 ceph-mon[36638]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T22:50:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:11 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3594942645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T22:50:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:11 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2757000405' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-02-09T22:50:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:11 smithi089 ceph-mon[36665]: pgmap v320: 97 pgs: 4 peering, 93 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 8 op/s; 0 B/s, 0 objects/s recovering 2024-02-09T22:50:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:11 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3594942645' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-02-09T22:50:12.189 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:12.551 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:12 smithi003 ceph-mon[32678]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-09T22:50:12.580 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612632 got 313532612637 for osd.10 2024-02-09T22:50:12.580 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:12 smithi064 ceph-mon[36638]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-09T22:50:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:12 smithi089 ceph-mon[36665]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-02-09T22:50:13.488 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757300 got 163208757304 for osd.5 2024-02-09T22:50:13.488 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:13.575 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:13 smithi003 ceph-mon[32678]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s; 13 B/s, 0 objects/s recovering 2024-02-09T22:50:13.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 -- 172.21.15.3:0/1816255428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8102130 msgr2=0x7f77f810db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 --2- 172.21.15.3:0/1816255428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8102130 0x7f77f810db00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f77f000b0a0 tx=0x7f77f000b3b0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 -- 172.21.15.3:0/1816255428 shutdown_connections 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 --2- 172.21.15.3:0/1816255428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77f8105c60 0x7f77f8106040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 --2- 172.21.15.3:0/1816255428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8102130 0x7f77f810db00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 --2- 172.21.15.3:0/1816255428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77f8101770 0x7f77f8101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 -- 172.21.15.3:0/1816255428 >> 172.21.15.3:0/1816255428 conn(0x7f77f8075f80 msgr2=0x7f77f8076390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 -- 172.21.15.3:0/1816255428 shutdown_connections 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.678+0000 7f77fd848700 1 -- 172.21.15.3:0/1816255428 wait complete. 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 Processor -- start 2024-02-09T22:50:13.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 -- start start 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77f8101770 0x7f77f8198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77f8102130 0x7f77f8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 0x7f77f819cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f819d310 con 0x7f77f8105c60 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f819d480 con 0x7f77f8101770 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77fd848700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f819d640 con 0x7f77f8102130 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 0x7f77f819cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:13.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 0x7f77f819cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50024/0 (socket says 172.21.15.3:50024) 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 -- 172.21.15.3:0/1182746882 learned_addr learned my addr 172.21.15.3:0/1182746882 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f67fc700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77f8102130 0x7f77f8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f6ffd700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77f8101770 0x7f77f8198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 -- 172.21.15.3:0/1182746882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77f8102130 msgr2=0x7f77f8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77f8102130 0x7f77f8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 -- 172.21.15.3:0/1182746882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77f8101770 msgr2=0x7f77f8198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77f8101770 0x7f77f8198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.679+0000 7f77f77fe700 1 -- 172.21.15.3:0/1182746882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0009d00 con 0x7f77f8105c60 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.680+0000 7f77f77fe700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 0x7f77f819cd00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f77f000b830 tx=0x7f77f000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.680+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f000e040 con 0x7f77f8105c60 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.680+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77f001bb30 con 0x7f77f8105c60 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.680+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f819d8c0 con 0x7f77f8105c60 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.680+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f00058b0 con 0x7f77f8105c60 2024-02-09T22:50:13.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.680+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f81a5180 con 0x7f77f8105c60 2024-02-09T22:50:13.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.681+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f80689e0 con 0x7f77f8105c60 2024-02-09T22:50:13.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:13 smithi064 ceph-mon[36638]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s; 13 B/s, 0 objects/s recovering 2024-02-09T22:50:13.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.682+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f77f001bca0 con 0x7f77f8105c60 2024-02-09T22:50:13.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.682+0000 7f77fc846700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f77e40a2260 0x7f77e40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:13.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.683+0000 7f77f6ffd700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f77e40a2260 0x7f77e40a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:13.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.683+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f77f00cf900 con 0x7f77f8105c60 2024-02-09T22:50:13.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.683+0000 7f77f6ffd700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f77e40a2260 0x7f77e40a4720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f77e0006fd0 tx=0x7f77e0008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:13.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.686+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f77f0098810 con 0x7f77f8105c60 2024-02-09T22:50:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:13 smithi089 ceph-mon[36665]: pgmap v321: 97 pgs: 1 peering, 96 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 11 op/s; 13 B/s, 0 objects/s recovering 2024-02-09T22:50:13.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.897+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f77f8110d60 con 0x7f77f8105c60 2024-02-09T22:50:13.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.897+0000 7f77fc846700 1 -- 172.21.15.3:0/1182746882 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f77f009bed0 con 0x7f77f8105c60 2024-02-09T22:50:13.901 INFO:teuthology.orchestra.run.smithi003.stdout:253403070505 2024-02-09T22:50:13.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.899+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f77e40a2260 msgr2=0x7f77e40a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.899+0000 7f77fd848700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f77e40a2260 0x7f77e40a4720 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f77e0006fd0 tx=0x7f77e0008040 comp rx=0 tx=0).stop 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.899+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 msgr2=0x7f77f819cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.899+0000 7f77fd848700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 0x7f77f819cd00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f77f000b830 tx=0x7f77f000cf90 comp rx=0 tx=0).stop 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.899+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 shutdown_connections 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f77e40a2260 0x7f77e40a4720 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77f8105c60 0x7f77f819cd00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77f8101770 0x7f77f8198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 --2- 172.21.15.3:0/1182746882 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77f8102130 0x7f77f8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 >> 172.21.15.3:0/1182746882 conn(0x7f77f8075f80 msgr2=0x7f77f8100720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 shutdown_connections 2024-02-09T22:50:13.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:13.900+0000 7f77fd848700 1 -- 172.21.15.3:0/1182746882 wait complete. 2024-02-09T22:50:14.411 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182458 got 107374182465 for osd.3 2024-02-09T22:50:14.411 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:14.544 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:14 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1182746882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T22:50:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:14 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1182746882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T22:50:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:14 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1182746882' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-02-09T22:50:15.459 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:15 smithi003 ceph-mon[32678]: pgmap v322: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s; 13 B/s, 0 objects/s recovering 2024-02-09T22:50:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 -- 172.21.15.3:0/3933524201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd806a370 msgr2=0x7f1fd810ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/3933524201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd806a370 0x7f1fd810ebb0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f1fd00099c0 tx=0x7f1fd0009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 -- 172.21.15.3:0/3933524201 shutdown_connections 2024-02-09T22:50:15.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/3933524201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd8107d70 0x7f1fd8108150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/3933524201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd806a370 0x7f1fd810ebb0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/3933524201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd80699d0 0x7f1fd8069e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 -- 172.21.15.3:0/3933524201 >> 172.21.15.3:0/3933524201 conn(0x7f1fd8077b50 msgr2=0x7f1fd8077f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 -- 172.21.15.3:0/3933524201 shutdown_connections 2024-02-09T22:50:15.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.562+0000 7f1fdc92a700 1 -- 172.21.15.3:0/3933524201 wait complete. 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 Processor -- start 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 -- start start 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80699d0 0x7f1fd8100cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd806a370 0x7f1fd8101230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 0x7f1fd8101770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81a6520 con 0x7f1fd8107d70 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81a6690 con 0x7f1fd806a370 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fdc92a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fd81a6850 con 0x7f1fd80699d0 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fd6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 0x7f1fd8101770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fd5d9b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd806a370 0x7f1fd8101230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fd6d9d700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 0x7f1fd8101770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50046/0 (socket says 172.21.15.3:50046) 2024-02-09T22:50:15.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.563+0000 7f1fd6d9d700 1 -- 172.21.15.3:0/458966798 learned_addr learned my addr 172.21.15.3:0/458966798 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd659c700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80699d0 0x7f1fd8100cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd5d9b700 1 -- 172.21.15.3:0/458966798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80699d0 msgr2=0x7f1fd8100cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd5d9b700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80699d0 0x7f1fd8100cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd5d9b700 1 -- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 msgr2=0x7f1fd8101770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd5d9b700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 0x7f1fd8101770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd5d9b700 1 -- 172.21.15.3:0/458966798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd00095f0 con 0x7f1fd806a370 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd659c700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80699d0 0x7f1fd8100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd6d9d700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 0x7f1fd8101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fd5d9b700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd806a370 0x7f1fd8101230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000c970 tx=0x7f1fc000cc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0007b00 con 0x7f1fd806a370 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fd81a6b30 con 0x7f1fd806a370 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fc000ce60 con 0x7f1fd806a370 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fd81a7080 con 0x7f1fd806a370 2024-02-09T22:50:15.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.564+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0005580 con 0x7f1fd806a370 2024-02-09T22:50:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.566+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fb8005330 con 0x7f1fd806a370 2024-02-09T22:50:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.567+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f1fc0007c60 con 0x7f1fd806a370 2024-02-09T22:50:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.567+0000 7f1fcf7fe700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1fc40a2260 0x7f1fc40a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.567+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1fc00cd7d0 con 0x7f1fd806a370 2024-02-09T22:50:15.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.567+0000 7f1fd659c700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1fc40a2260 0x7f1fc40a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:15.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.568+0000 7f1fd659c700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1fc40a2260 0x7f1fc40a4720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d260 tx=0x7f1fc8009040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:15.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.572+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1fc0096650 con 0x7f1fd806a370 2024-02-09T22:50:15.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:15 smithi064 ceph-mon[36638]: pgmap v322: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s; 13 B/s, 0 objects/s recovering 2024-02-09T22:50:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:15 smithi089 ceph-mon[36665]: pgmap v322: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 7 op/s; 13 B/s, 0 objects/s recovering 2024-02-09T22:50:15.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.776+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f1fb80058b0 con 0x7f1fd806a370 2024-02-09T22:50:15.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.777+0000 7f1fcf7fe700 1 -- 172.21.15.3:0/458966798 <== mon.2 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1fc0099d10 con 0x7f1fd806a370 2024-02-09T22:50:15.780 INFO:teuthology.orchestra.run.smithi003.stdout:77309411399 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1fc40a2260 msgr2=0x7f1fc40a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1fc40a2260 0x7f1fc40a4720 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1fc800d260 tx=0x7f1fc8009040 comp rx=0 tx=0).stop 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd806a370 msgr2=0x7f1fd8101230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd806a370 0x7f1fd8101230 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000c970 tx=0x7f1fc000cc80 comp rx=0 tx=0).stop 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 shutdown_connections 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1fc40a2260 0x7f1fc40a4720 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fd80699d0 0x7f1fd8100cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fd8107d70 0x7f1fd8101770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 --2- 172.21.15.3:0/458966798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fd806a370 0x7f1fd8101230 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:15.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 >> 172.21.15.3:0/458966798 conn(0x7f1fd8077b50 msgr2=0x7f1fd80ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 shutdown_connections 2024-02-09T22:50:15.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:15.779+0000 7f1fdc92a700 1 -- 172.21.15.3:0/458966798 wait complete. 2024-02-09T22:50:16.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:16 smithi003 ceph-mon[32678]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T22:50:16.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:16 smithi003 ceph-mon[32678]: Cluster is now healthy 2024-02-09T22:50:16.532 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:16 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/458966798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T22:50:16.565 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411393 got 77309411399 for osd.2 2024-02-09T22:50:16.565 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:16.629 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:16 smithi089 ceph-mon[36665]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T22:50:16.629 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:16 smithi089 ceph-mon[36665]: Cluster is now healthy 2024-02-09T22:50:16.629 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:16 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/458966798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T22:50:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:16 smithi064 ceph-mon[36638]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-02-09T22:50:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:16 smithi064 ceph-mon[36638]: Cluster is now healthy 2024-02-09T22:50:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:16 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/458966798' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 -- 172.21.15.3:0/116589193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0100eb0 msgr2=0x7fa2b0101290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 --2- 172.21.15.3:0/116589193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0100eb0 0x7fa2b0101290 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fa298009990 tx=0x7fa298009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 -- 172.21.15.3:0/116589193 shutdown_connections 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 --2- 172.21.15.3:0/116589193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b0105f40 0x7fa2b010db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 --2- 172.21.15.3:0/116589193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0100eb0 0x7fa2b0101290 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 --2- 172.21.15.3:0/116589193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2b0101860 0x7fa2b0105860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 -- 172.21.15.3:0/116589193 >> 172.21.15.3:0/116589193 conn(0x7fa2b0076c40 msgr2=0x7fa2b0077050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 -- 172.21.15.3:0/116589193 shutdown_connections 2024-02-09T22:50:17.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.995+0000 7fa2b6098700 1 -- 172.21.15.3:0/116589193 wait complete. 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 Processor -- start 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 -- start start 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b0100eb0 0x7fa2b0198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2b0101860 0x7fa2b0198780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 0x7fa2b019cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b019d120 con 0x7fa2b0105f40 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b019d290 con 0x7fa2b0101860 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.996+0000 7fa2b6098700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b019d450 con 0x7fa2b0100eb0 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 0x7fa2b019cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2aeffd700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2b0101860 0x7fa2b0198780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:17.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 0x7fa2b019cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55164/0 (socket says 172.21.15.3:55164) 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 -- 172.21.15.3:0/2116599807 learned_addr learned my addr 172.21.15.3:0/2116599807 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2af7fe700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b0100eb0 0x7fa2b0198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 -- 172.21.15.3:0/2116599807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b0100eb0 msgr2=0x7fa2b0198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b0100eb0 0x7fa2b0198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 -- 172.21.15.3:0/2116599807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2b0101860 msgr2=0x7fa2b0198780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2b0101860 0x7fa2b0198780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 -- 172.21.15.3:0/2116599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2980095f0 con 0x7fa2b0105f40 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2affff700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 0x7fa2b019cb10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8007ee0 tx=0x7fa2a800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:17.003 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8017070 con 0x7fa2b0105f40 2024-02-09T22:50:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2a800f040 con 0x7fa2b0105f40 2024-02-09T22:50:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b019d730 con 0x7fa2b0105f40 2024-02-09T22:50:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8005650 con 0x7fa2b0105f40 2024-02-09T22:50:17.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.997+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b01a4f90 con 0x7fa2b0105f40 2024-02-09T22:50:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:16.998+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b010a1e0 con 0x7fa2b0105f40 2024-02-09T22:50:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.000+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fa2a8004cd0 con 0x7fa2b0105f40 2024-02-09T22:50:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.001+0000 7fa2acff9700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa29c0a21e0 0x7fa29c0a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:17.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.001+0000 7fa2af7fe700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa29c0a21e0 0x7fa29c0a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.001+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa2a80cdb90 con 0x7fa2b0105f40 2024-02-09T22:50:17.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.001+0000 7fa2af7fe700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa29c0a21e0 0x7fa29c0a46a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa29800b5c0 tx=0x7fa298005960 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:17.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.004+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa2a80969b0 con 0x7fa2b0105f40 2024-02-09T22:50:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.211+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fa2b01995a0 con 0x7fa2b0105f40 2024-02-09T22:50:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.211+0000 7fa2acff9700 1 -- 172.21.15.3:0/2116599807 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa2a809a070 con 0x7fa2b0105f40 2024-02-09T22:50:17.214 INFO:teuthology.orchestra.run.smithi003.stdout:197568495668 2024-02-09T22:50:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa29c0a21e0 msgr2=0x7fa29c0a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa29c0a21e0 0x7fa29c0a46a0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa29800b5c0 tx=0x7fa298005960 comp rx=0 tx=0).stop 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 msgr2=0x7fa2b019cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 0x7fa2b019cb10 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa2a8007ee0 tx=0x7fa2a800d3b0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 shutdown_connections 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa29c0a21e0 0x7fa29c0a46a0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa2b0100eb0 0x7fa2b0198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa2b0105f40 0x7fa2b019cb10 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 --2- 172.21.15.3:0/2116599807 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa2b0101860 0x7fa2b0198780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.213+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 >> 172.21.15.3:0/2116599807 conn(0x7fa2b0076c40 msgr2=0x7fa2b006a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.214+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 shutdown_connections 2024-02-09T22:50:17.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:17.214+0000 7fa2b6098700 1 -- 172.21.15.3:0/2116599807 wait complete. 2024-02-09T22:50:17.397 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070499 got 253403070505 for osd.8 2024-02-09T22:50:17.398 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:17 smithi003 ceph-mon[32678]: pgmap v323: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 12 B/s, 0 objects/s recovering 2024-02-09T22:50:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:17 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2116599807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T22:50:17.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:17 smithi064 ceph-mon[36638]: pgmap v323: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 12 B/s, 0 objects/s recovering 2024-02-09T22:50:17.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:17 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2116599807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T22:50:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:17 smithi089 ceph-mon[36665]: pgmap v323: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s; 12 B/s, 0 objects/s recovering 2024-02-09T22:50:17.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:17 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2116599807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-02-09T22:50:17.967 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495663 got 197568495668 for osd.6 2024-02-09T22:50:17.968 DEBUG:teuthology.parallel:result is None 2024-02-09T22:50:17.968 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-02-09T22:50:17.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-09T22:50:18.782 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:18.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:18 smithi003 ceph-mon[32678]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-02-09T22:50:18.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:18 smithi064 ceph-mon[36638]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-02-09T22:50:18.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:18 smithi089 ceph-mon[36665]: pgmap v324: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 10 B/s, 0 objects/s recovering 2024-02-09T22:50:20.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 -- 172.21.15.3:0/46544299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 msgr2=0x7faec010dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 --2- 172.21.15.3:0/46544299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec010dad0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7faeb800b0a0 tx=0x7faeb800b3b0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 -- 172.21.15.3:0/46544299 shutdown_connections 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 --2- 172.21.15.3:0/46544299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faec0100eb0 0x7faec0101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 --2- 172.21.15.3:0/46544299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec010dad0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 --2- 172.21.15.3:0/46544299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 0x7faec0105820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 -- 172.21.15.3:0/46544299 >> 172.21.15.3:0/46544299 conn(0x7faec0076c70 msgr2=0x7faec0077080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 -- 172.21.15.3:0/46544299 shutdown_connections 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 -- 172.21.15.3:0/46544299 wait complete. 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.412+0000 7faec632b700 1 Processor -- start 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 -- start start 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faec0100eb0 0x7faec01028c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 0x7faec0102e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec0103340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec01a68a0 con 0x7faec0105f00 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec01a6a10 con 0x7faec0101860 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec632b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faec01a6bd0 con 0x7faec0100eb0 2024-02-09T22:50:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec0103340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec0103340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55182/0 (socket says 172.21.15.3:55182) 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 -- 172.21.15.3:0/519238070 learned_addr learned my addr 172.21.15.3:0/519238070 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faebffff700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faec0100eb0 0x7faec01028c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faebf7fe700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 0x7faec0102e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 -- 172.21.15.3:0/519238070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faec0100eb0 msgr2=0x7faec01028c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faec0100eb0 0x7faec01028c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 -- 172.21.15.3:0/519238070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 msgr2=0x7faec0102e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 0x7faec0102e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faec48c8700 1 -- 172.21.15.3:0/519238070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb8009d00 con 0x7faec0105f00 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.413+0000 7faebf7fe700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 0x7faec0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.414+0000 7faec48c8700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec0103340 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faeb800b830 tx=0x7faeb800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.414+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb800e040 con 0x7faec0105f00 2024-02-09T22:50:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.414+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faec01a6e50 con 0x7faec0105f00 2024-02-09T22:50:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.414+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faeb801bb30 con 0x7faec0105f00 2024-02-09T22:50:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.414+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeb80058b0 con 0x7faec0105f00 2024-02-09T22:50:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.414+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faec01a7310 con 0x7faec0105f00 2024-02-09T22:50:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.415+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faec010a230 con 0x7faec0105f00 2024-02-09T22:50:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.416+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7faeb801bca0 con 0x7faec0105f00 2024-02-09T22:50:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.416+0000 7faebd7fa700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7faea80a2260 0x7faea80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.416+0000 7faebffff700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7faea80a2260 0x7faea80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.416+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7faeb80cf900 con 0x7faec0105f00 2024-02-09T22:50:20.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.417+0000 7faebffff700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7faea80a2260 0x7faea80a4720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009a10 tx=0x7faeb0009190 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:20.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.420+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7faeb8098810 con 0x7faec0105f00 2024-02-09T22:50:20.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.655+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faec0110d50 con 0x7faea80a2260 2024-02-09T22:50:20.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.665+0000 7faebd7fa700 1 -- 172.21.15.3:0/519238070 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+343842 (secure 0 0 0) 0x7faec0110d50 con 0x7faea80a2260 2024-02-09T22:50:20.674 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:20.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7faea80a2260 msgr2=0x7faea80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7faea80a2260 0x7faea80a4720 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faeb0009a10 tx=0x7faeb0009190 comp rx=0 tx=0).stop 2024-02-09T22:50:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 msgr2=0x7faec0103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec0103340 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7faeb800b830 tx=0x7faeb800cf90 comp rx=0 tx=0).stop 2024-02-09T22:50:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 shutdown_connections 2024-02-09T22:50:20.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7faea80a2260 0x7faea80a4720 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faec0100eb0 0x7faec01028c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faec0105f00 0x7faec0103340 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 --2- 172.21.15.3:0/519238070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faec0101860 0x7faec0102e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 >> 172.21.15.3:0/519238070 conn(0x7faec0076c70 msgr2=0x7faec006a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 shutdown_connections 2024-02-09T22:50:20.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:20.670+0000 7faec632b700 1 -- 172.21.15.3:0/519238070 wait complete. 2024-02-09T22:50:20.680 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:50:20.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:20 smithi003 ceph-mon[32678]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-02-09T22:50:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:20 smithi064 ceph-mon[36638]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-02-09T22:50:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:20 smithi089 ceph-mon[36665]: pgmap v325: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 9 B/s, 0 objects/s recovering 2024-02-09T22:50:21.300 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":325,"stamp":"2024-02-09T22:50:20.555831+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656809,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":815,"num_read_kb":528,"num_write":418,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":343,"ondisk_log_size":343,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354364,"kb_used_data":13452,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587460,"statfs":{"total":1151940427776,"available":1151577559040,"internally_reserved":0,"allocated":13774848,"data_stored":7517667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":109,"num_read_kb":54,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003342"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264215+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:50:08.264215+0000","last_peered":"2024-02-09T22:50:08.264215+0000","last_clean":"2024-02-09T22:50:08.264215+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:50:08.264215+0000","last_undegraded":"2024-02-09T22:50:08.264215+0000","last_fullsized":"2024-02-09T22:50:08.264215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265998+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:50:08.265998+0000","last_peered":"2024-02-09T22:50:08.265998+0000","last_clean":"2024-02-09T22:50:08.265998+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:50:08.265998+0000","last_undegraded":"2024-02-09T22:50:08.265998+0000","last_fullsized":"2024-02-09T22:50:08.265998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266453+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:50:08.266453+0000","last_peered":"2024-02-09T22:50:08.266453+0000","last_clean":"2024-02-09T22:50:08.266453+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:50:08.266453+0000","last_undegraded":"2024-02-09T22:50:08.266453+0000","last_fullsized":"2024-02-09T22:50:08.266453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538092+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:50:08.538092+0000","last_peered":"2024-02-09T22:50:08.538092+0000","last_clean":"2024-02-09T22:50:08.538092+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:50:08.538092+0000","last_undegraded":"2024-02-09T22:50:08.538092+0000","last_fullsized":"2024-02-09T22:50:08.538092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:50.286530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266623+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:50:08.266623+0000","last_peered":"2024-02-09T22:50:08.266623+0000","last_clean":"2024-02-09T22:50:08.266623+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:50:08.266623+0000","last_undegraded":"2024-02-09T22:50:08.266623+0000","last_fullsized":"2024-02-09T22:50:08.266623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515923+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:50:08.515923+0000","last_peered":"2024-02-09T22:50:08.515923+0000","last_clean":"2024-02-09T22:50:08.515923+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:50:08.515923+0000","last_undegraded":"2024-02-09T22:50:08.515923+0000","last_fullsized":"2024-02-09T22:50:08.515923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264736+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:50:08.264736+0000","last_peered":"2024-02-09T22:50:08.264736+0000","last_clean":"2024-02-09T22:50:08.264736+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:50:08.264736+0000","last_undegraded":"2024-02-09T22:50:08.264736+0000","last_fullsized":"2024-02-09T22:50:08.264736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265839+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:50:08.265839+0000","last_peered":"2024-02-09T22:50:08.265839+0000","last_clean":"2024-02-09T22:50:08.265839+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:50:08.265839+0000","last_undegraded":"2024-02-09T22:50:08.265839+0000","last_fullsized":"2024-02-09T22:50:08.265839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538413+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:50:08.538413+0000","last_peered":"2024-02-09T22:50:08.538413+0000","last_clean":"2024-02-09T22:50:08.538413+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:50:08.538413+0000","last_undegraded":"2024-02-09T22:50:08.538413+0000","last_fullsized":"2024-02-09T22:50:08.538413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.516201+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:50:08.516201+0000","last_peered":"2024-02-09T22:50:08.516201+0000","last_clean":"2024-02-09T22:50:08.516201+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:50:08.516201+0000","last_undegraded":"2024-02-09T22:50:08.516201+0000","last_fullsized":"2024-02-09T22:50:08.516201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265144+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:50:08.265144+0000","last_peered":"2024-02-09T22:50:08.265144+0000","last_clean":"2024-02-09T22:50:08.265144+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:50:08.265144+0000","last_undegraded":"2024-02-09T22:50:08.265144+0000","last_fullsized":"2024-02-09T22:50:08.265144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.517473+0000","last_change":"2024-02-09T22:50:08.517050+0000","last_active":"2024-02-09T22:50:08.517473+0000","last_peered":"2024-02-09T22:50:08.517473+0000","last_clean":"2024-02-09T22:50:08.517473+0000","last_became_active":"2024-02-09T22:50:08.516688+0000","last_became_peered":"2024-02-09T22:50:08.516688+0000","last_unstale":"2024-02-09T22:50:08.517473+0000","last_undegraded":"2024-02-09T22:50:08.517473+0000","last_fullsized":"2024-02-09T22:50:08.517473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:44:51.672735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615737+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:50:08.615737+0000","last_peered":"2024-02-09T22:50:08.615737+0000","last_clean":"2024-02-09T22:50:08.615737+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:50:08.615737+0000","last_undegraded":"2024-02-09T22:50:08.615737+0000","last_fullsized":"2024-02-09T22:50:08.615737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266475+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:50:08.266475+0000","last_peered":"2024-02-09T22:50:08.266475+0000","last_clean":"2024-02-09T22:50:08.266475+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:50:08.266475+0000","last_undegraded":"2024-02-09T22:50:08.266475+0000","last_fullsized":"2024-02-09T22:50:08.266475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.616823+0000","last_change":"2024-02-09T22:50:08.616823+0000","last_active":"2024-02-09T22:50:08.616823+0000","last_peered":"2024-02-09T22:50:08.616823+0000","last_clean":"2024-02-09T22:50:08.616823+0000","last_became_active":"2024-02-09T22:50:08.616482+0000","last_became_peered":"2024-02-09T22:50:08.616482+0000","last_unstale":"2024-02-09T22:50:08.616823+0000","last_undegraded":"2024-02-09T22:50:08.616823+0000","last_fullsized":"2024-02-09T22:50:08.616823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:57:44.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537554+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:50:08.537554+0000","last_peered":"2024-02-09T22:50:08.537554+0000","last_clean":"2024-02-09T22:50:08.537554+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:50:08.537554+0000","last_undegraded":"2024-02-09T22:50:08.537554+0000","last_fullsized":"2024-02-09T22:50:08.537554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264328+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:50:08.264328+0000","last_peered":"2024-02-09T22:50:08.264328+0000","last_clean":"2024-02-09T22:50:08.264328+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:50:08.264328+0000","last_undegraded":"2024-02-09T22:50:08.264328+0000","last_fullsized":"2024-02-09T22:50:08.264328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263394+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:50:08.263394+0000","last_peered":"2024-02-09T22:50:08.263394+0000","last_clean":"2024-02-09T22:50:08.263394+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:50:08.263394+0000","last_undegraded":"2024-02-09T22:50:08.263394+0000","last_fullsized":"2024-02-09T22:50:08.263394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266124+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:50:08.266124+0000","last_peered":"2024-02-09T22:50:08.266124+0000","last_clean":"2024-02-09T22:50:08.266124+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:50:08.266124+0000","last_undegraded":"2024-02-09T22:50:08.266124+0000","last_fullsized":"2024-02-09T22:50:08.266124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:39.090835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:18.213108+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:50:18.213108+0000","last_peered":"2024-02-09T22:50:18.213108+0000","last_clean":"2024-02-09T22:50:18.213108+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:50:18.213108+0000","last_undegraded":"2024-02-09T22:50:18.213108+0000","last_fullsized":"2024-02-09T22:50:18.213108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266854+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:50:08.266854+0000","last_peered":"2024-02-09T22:50:08.266854+0000","last_clean":"2024-02-09T22:50:08.266854+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:50:08.266854+0000","last_undegraded":"2024-02-09T22:50:08.266854+0000","last_fullsized":"2024-02-09T22:50:08.266854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267136+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:50:08.267136+0000","last_peered":"2024-02-09T22:50:08.267136+0000","last_clean":"2024-02-09T22:50:08.267136+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:50:08.267136+0000","last_undegraded":"2024-02-09T22:50:08.267136+0000","last_fullsized":"2024-02-09T22:50:08.267136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266068+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:50:08.266068+0000","last_peered":"2024-02-09T22:50:08.266068+0000","last_clean":"2024-02-09T22:50:08.266068+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:50:08.266068+0000","last_undegraded":"2024-02-09T22:50:08.266068+0000","last_fullsized":"2024-02-09T22:50:08.266068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267805+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:50:08.267805+0000","last_peered":"2024-02-09T22:50:08.267805+0000","last_clean":"2024-02-09T22:50:08.267805+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:50:08.267805+0000","last_undegraded":"2024-02-09T22:50:08.267805+0000","last_fullsized":"2024-02-09T22:50:08.267805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537848+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:50:08.537848+0000","last_peered":"2024-02-09T22:50:08.537848+0000","last_clean":"2024-02-09T22:50:08.537848+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:50:08.537848+0000","last_undegraded":"2024-02-09T22:50:08.537848+0000","last_fullsized":"2024-02-09T22:50:08.537848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267118+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:50:08.267118+0000","last_peered":"2024-02-09T22:50:08.267118+0000","last_clean":"2024-02-09T22:50:08.267118+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:50:08.267118+0000","last_undegraded":"2024-02-09T22:50:08.267118+0000","last_fullsized":"2024-02-09T22:50:08.267118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615674+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:50:08.615674+0000","last_peered":"2024-02-09T22:50:08.615674+0000","last_clean":"2024-02-09T22:50:08.615674+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:50:08.615674+0000","last_undegraded":"2024-02-09T22:50:08.615674+0000","last_fullsized":"2024-02-09T22:50:08.615674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:15.211829+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:50:15.211829+0000","last_peered":"2024-02-09T22:50:15.211829+0000","last_clean":"2024-02-09T22:50:15.211829+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:50:15.211829+0000","last_undegraded":"2024-02-09T22:50:15.211829+0000","last_fullsized":"2024-02-09T22:50:15.211829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267696+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:50:08.267696+0000","last_peered":"2024-02-09T22:50:08.267696+0000","last_clean":"2024-02-09T22:50:08.267696+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:50:08.267696+0000","last_undegraded":"2024-02-09T22:50:08.267696+0000","last_fullsized":"2024-02-09T22:50:08.267696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515862+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:50:08.515862+0000","last_peered":"2024-02-09T22:50:08.515862+0000","last_clean":"2024-02-09T22:50:08.515862+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:50:08.515862+0000","last_undegraded":"2024-02-09T22:50:08.515862+0000","last_fullsized":"2024-02-09T22:50:08.515862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615664+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:50:08.615664+0000","last_peered":"2024-02-09T22:50:08.615664+0000","last_clean":"2024-02-09T22:50:08.615664+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:50:08.615664+0000","last_undegraded":"2024-02-09T22:50:08.615664+0000","last_fullsized":"2024-02-09T22:50:08.615664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615818+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:50:08.615818+0000","last_peered":"2024-02-09T22:50:08.615818+0000","last_clean":"2024-02-09T22:50:08.615818+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:50:08.615818+0000","last_undegraded":"2024-02-09T22:50:08.615818+0000","last_fullsized":"2024-02-09T22:50:08.615818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266385+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:50:08.266385+0000","last_peered":"2024-02-09T22:50:08.266385+0000","last_clean":"2024-02-09T22:50:08.266385+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:50:08.266385+0000","last_undegraded":"2024-02-09T22:50:08.266385+0000","last_fullsized":"2024-02-09T22:50:08.266385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615165+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:50:08.615165+0000","last_peered":"2024-02-09T22:50:08.615165+0000","last_clean":"2024-02-09T22:50:08.615165+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:50:08.615165+0000","last_undegraded":"2024-02-09T22:50:08.615165+0000","last_fullsized":"2024-02-09T22:50:08.615165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263175+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:50:08.263175+0000","last_peered":"2024-02-09T22:50:08.263175+0000","last_clean":"2024-02-09T22:50:08.263175+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:50:08.263175+0000","last_undegraded":"2024-02-09T22:50:08.263175+0000","last_fullsized":"2024-02-09T22:50:08.263175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":149,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:15.210942+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:50:15.210942+0000","last_peered":"2024-02-09T22:50:15.210942+0000","last_clean":"2024-02-09T22:50:15.210942+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:50:15.210942+0000","last_undegraded":"2024-02-09T22:50:15.210942+0000","last_fullsized":"2024-02-09T22:50:15.210942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":126,"num_read_kb":126,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538198+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:50:08.538198+0000","last_peered":"2024-02-09T22:50:08.538198+0000","last_clean":"2024-02-09T22:50:08.538198+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:50:08.538198+0000","last_undegraded":"2024-02-09T22:50:08.538198+0000","last_fullsized":"2024-02-09T22:50:08.538198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264958+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:50:08.264958+0000","last_peered":"2024-02-09T22:50:08.264958+0000","last_clean":"2024-02-09T22:50:08.264958+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:50:08.264958+0000","last_undegraded":"2024-02-09T22:50:08.264958+0000","last_fullsized":"2024-02-09T22:50:08.264958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266372+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:50:08.266372+0000","last_peered":"2024-02-09T22:50:08.266372+0000","last_clean":"2024-02-09T22:50:08.266372+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:50:08.266372+0000","last_undegraded":"2024-02-09T22:50:08.266372+0000","last_fullsized":"2024-02-09T22:50:08.266372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615718+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:50:08.615718+0000","last_peered":"2024-02-09T22:50:08.615718+0000","last_clean":"2024-02-09T22:50:08.615718+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:50:08.615718+0000","last_undegraded":"2024-02-09T22:50:08.615718+0000","last_fullsized":"2024-02-09T22:50:08.615718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538017+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:50:08.538017+0000","last_peered":"2024-02-09T22:50:08.538017+0000","last_clean":"2024-02-09T22:50:08.538017+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:50:08.538017+0000","last_undegraded":"2024-02-09T22:50:08.538017+0000","last_fullsized":"2024-02-09T22:50:08.538017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'29","reported_seq":180,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:15.211012+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:50:15.211012+0000","last_peered":"2024-02-09T22:50:15.211012+0000","last_clean":"2024-02-09T22:50:15.211012+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:50:15.211012+0000","last_undegraded":"2024-02-09T22:50:15.211012+0000","last_fullsized":"2024-02-09T22:50:15.211012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59394,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":132,"num_read_kb":124,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266349+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:50:08.266349+0000","last_peered":"2024-02-09T22:50:08.266349+0000","last_clean":"2024-02-09T22:50:08.266349+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:50:08.266349+0000","last_undegraded":"2024-02-09T22:50:08.266349+0000","last_fullsized":"2024-02-09T22:50:08.266349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267676+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:50:08.267676+0000","last_peered":"2024-02-09T22:50:08.267676+0000","last_clean":"2024-02-09T22:50:08.267676+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:50:08.267676+0000","last_undegraded":"2024-02-09T22:50:08.267676+0000","last_fullsized":"2024-02-09T22:50:08.267676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615712+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:50:08.615712+0000","last_peered":"2024-02-09T22:50:08.615712+0000","last_clean":"2024-02-09T22:50:08.615712+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:50:08.615712+0000","last_undegraded":"2024-02-09T22:50:08.615712+0000","last_fullsized":"2024-02-09T22:50:08.615712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615556+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:50:08.615556+0000","last_peered":"2024-02-09T22:50:08.615556+0000","last_clean":"2024-02-09T22:50:08.615556+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:50:08.615556+0000","last_undegraded":"2024-02-09T22:50:08.615556+0000","last_fullsized":"2024-02-09T22:50:08.615556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264889+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:50:08.264889+0000","last_peered":"2024-02-09T22:50:08.264889+0000","last_clean":"2024-02-09T22:50:08.264889+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:50:08.264889+0000","last_undegraded":"2024-02-09T22:50:08.264889+0000","last_fullsized":"2024-02-09T22:50:08.264889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:16.227362+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:50:16.227362+0000","last_peered":"2024-02-09T22:50:16.227362+0000","last_clean":"2024-02-09T22:50:16.227362+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:50:16.227362+0000","last_undegraded":"2024-02-09T22:50:16.227362+0000","last_fullsized":"2024-02-09T22:50:16.227362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264613+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:50:08.264613+0000","last_peered":"2024-02-09T22:50:08.264613+0000","last_clean":"2024-02-09T22:50:08.264613+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:50:08.264613+0000","last_undegraded":"2024-02-09T22:50:08.264613+0000","last_fullsized":"2024-02-09T22:50:08.264613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265893+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:50:08.265893+0000","last_peered":"2024-02-09T22:50:08.265893+0000","last_clean":"2024-02-09T22:50:08.265893+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:50:08.265893+0000","last_undegraded":"2024-02-09T22:50:08.265893+0000","last_fullsized":"2024-02-09T22:50:08.265893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.539717+0000","last_change":"2024-02-09T22:50:08.539326+0000","last_active":"2024-02-09T22:50:08.539717+0000","last_peered":"2024-02-09T22:50:08.539717+0000","last_clean":"2024-02-09T22:50:08.539717+0000","last_became_active":"2024-02-09T22:50:08.539091+0000","last_became_peered":"2024-02-09T22:50:08.539091+0000","last_unstale":"2024-02-09T22:50:08.539717+0000","last_undegraded":"2024-02-09T22:50:08.539717+0000","last_fullsized":"2024-02-09T22:50:08.539717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:31:57.282733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267149+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:50:08.267149+0000","last_peered":"2024-02-09T22:50:08.267149+0000","last_clean":"2024-02-09T22:50:08.267149+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:50:08.267149+0000","last_undegraded":"2024-02-09T22:50:08.267149+0000","last_fullsized":"2024-02-09T22:50:08.267149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.616790+0000","last_change":"2024-02-09T22:50:08.616790+0000","last_active":"2024-02-09T22:50:08.616790+0000","last_peered":"2024-02-09T22:50:08.616790+0000","last_clean":"2024-02-09T22:50:08.616790+0000","last_became_active":"2024-02-09T22:50:08.616404+0000","last_became_peered":"2024-02-09T22:50:08.616404+0000","last_unstale":"2024-02-09T22:50:08.616790+0000","last_undegraded":"2024-02-09T22:50:08.616790+0000","last_fullsized":"2024-02-09T22:50:08.616790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:14:18.902026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263341+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:50:08.263341+0000","last_peered":"2024-02-09T22:50:08.263341+0000","last_clean":"2024-02-09T22:50:08.263341+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:50:08.263341+0000","last_undegraded":"2024-02-09T22:50:08.263341+0000","last_fullsized":"2024-02-09T22:50:08.263341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.516172+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:50:08.516172+0000","last_peered":"2024-02-09T22:50:08.516172+0000","last_clean":"2024-02-09T22:50:08.516172+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:50:08.516172+0000","last_undegraded":"2024-02-09T22:50:08.516172+0000","last_fullsized":"2024-02-09T22:50:08.516172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:19.214867+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:50:19.214867+0000","last_peered":"2024-02-09T22:50:19.214867+0000","last_clean":"2024-02-09T22:50:19.214867+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:50:19.214867+0000","last_undegraded":"2024-02-09T22:50:19.214867+0000","last_fullsized":"2024-02-09T22:50:19.214867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.516128+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:50:08.516128+0000","last_peered":"2024-02-09T22:50:08.516128+0000","last_clean":"2024-02-09T22:50:08.516128+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:50:08.516128+0000","last_undegraded":"2024-02-09T22:50:08.516128+0000","last_fullsized":"2024-02-09T22:50:08.516128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615341+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:50:08.615341+0000","last_peered":"2024-02-09T22:50:08.615341+0000","last_clean":"2024-02-09T22:50:08.615341+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:50:08.615341+0000","last_undegraded":"2024-02-09T22:50:08.615341+0000","last_fullsized":"2024-02-09T22:50:08.615341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":88,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:14.222823+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:50:14.222823+0000","last_peered":"2024-02-09T22:50:14.222823+0000","last_clean":"2024-02-09T22:50:14.222823+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:50:14.222823+0000","last_undegraded":"2024-02-09T22:50:14.222823+0000","last_fullsized":"2024-02-09T22:50:14.222823+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":64,"num_read_kb":64,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266817+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:50:08.266817+0000","last_peered":"2024-02-09T22:50:08.266817+0000","last_clean":"2024-02-09T22:50:08.266817+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:50:08.266817+0000","last_undegraded":"2024-02-09T22:50:08.266817+0000","last_fullsized":"2024-02-09T22:50:08.266817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538316+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:50:08.538316+0000","last_peered":"2024-02-09T22:50:08.538316+0000","last_clean":"2024-02-09T22:50:08.538316+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:50:08.538316+0000","last_undegraded":"2024-02-09T22:50:08.538316+0000","last_fullsized":"2024-02-09T22:50:08.538316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264693+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:50:08.264693+0000","last_peered":"2024-02-09T22:50:08.264693+0000","last_clean":"2024-02-09T22:50:08.264693+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:50:08.264693+0000","last_undegraded":"2024-02-09T22:50:08.264693+0000","last_fullsized":"2024-02-09T22:50:08.264693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615683+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:50:08.615683+0000","last_peered":"2024-02-09T22:50:08.615683+0000","last_clean":"2024-02-09T22:50:08.615683+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:50:08.615683+0000","last_undegraded":"2024-02-09T22:50:08.615683+0000","last_fullsized":"2024-02-09T22:50:08.615683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264784+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:50:08.264784+0000","last_peered":"2024-02-09T22:50:08.264784+0000","last_clean":"2024-02-09T22:50:08.264784+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:50:08.264784+0000","last_undegraded":"2024-02-09T22:50:08.264784+0000","last_fullsized":"2024-02-09T22:50:08.264784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.278066+0000","last_change":"2024-02-09T22:50:08.278066+0000","last_active":"2024-02-09T22:50:08.278066+0000","last_peered":"2024-02-09T22:50:08.278066+0000","last_clean":"2024-02-09T22:50:08.278066+0000","last_became_active":"2024-02-09T22:50:08.271486+0000","last_became_peered":"2024-02-09T22:50:08.271486+0000","last_unstale":"2024-02-09T22:50:08.278066+0000","last_undegraded":"2024-02-09T22:50:08.278066+0000","last_fullsized":"2024-02-09T22:50:08.278066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:54:12.295421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.616063+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:50:08.616063+0000","last_peered":"2024-02-09T22:50:08.616063+0000","last_clean":"2024-02-09T22:50:08.616063+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:50:08.616063+0000","last_undegraded":"2024-02-09T22:50:08.616063+0000","last_fullsized":"2024-02-09T22:50:08.616063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263461+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:50:08.263461+0000","last_peered":"2024-02-09T22:50:08.263461+0000","last_clean":"2024-02-09T22:50:08.263461+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:50:08.263461+0000","last_undegraded":"2024-02-09T22:50:08.263461+0000","last_fullsized":"2024-02-09T22:50:08.263461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267157+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:50:08.267157+0000","last_peered":"2024-02-09T22:50:08.267157+0000","last_clean":"2024-02-09T22:50:08.267157+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:50:08.267157+0000","last_undegraded":"2024-02-09T22:50:08.267157+0000","last_fullsized":"2024-02-09T22:50:08.267157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538049+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:50:08.538049+0000","last_peered":"2024-02-09T22:50:08.538049+0000","last_clean":"2024-02-09T22:50:08.538049+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:50:08.538049+0000","last_undegraded":"2024-02-09T22:50:08.538049+0000","last_fullsized":"2024-02-09T22:50:08.538049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266104+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:50:08.266104+0000","last_peered":"2024-02-09T22:50:08.266104+0000","last_clean":"2024-02-09T22:50:08.266104+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:50:08.266104+0000","last_undegraded":"2024-02-09T22:50:08.266104+0000","last_fullsized":"2024-02-09T22:50:08.266104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537439+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:50:08.537439+0000","last_peered":"2024-02-09T22:50:08.537439+0000","last_clean":"2024-02-09T22:50:08.537439+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:50:08.537439+0000","last_undegraded":"2024-02-09T22:50:08.537439+0000","last_fullsized":"2024-02-09T22:50:08.537439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265754+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:50:08.265754+0000","last_peered":"2024-02-09T22:50:08.265754+0000","last_clean":"2024-02-09T22:50:08.265754+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:50:08.265754+0000","last_undegraded":"2024-02-09T22:50:08.265754+0000","last_fullsized":"2024-02-09T22:50:08.265754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266189+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:50:08.266189+0000","last_peered":"2024-02-09T22:50:08.266189+0000","last_clean":"2024-02-09T22:50:08.266189+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:50:08.266189+0000","last_undegraded":"2024-02-09T22:50:08.266189+0000","last_fullsized":"2024-02-09T22:50:08.266189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263619+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:50:08.263619+0000","last_peered":"2024-02-09T22:50:08.263619+0000","last_clean":"2024-02-09T22:50:08.263619+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:50:08.263619+0000","last_undegraded":"2024-02-09T22:50:08.263619+0000","last_fullsized":"2024-02-09T22:50:08.263619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.273440+0000","last_change":"2024-02-09T22:50:08.273440+0000","last_active":"2024-02-09T22:50:08.273440+0000","last_peered":"2024-02-09T22:50:08.273440+0000","last_clean":"2024-02-09T22:50:08.273440+0000","last_became_active":"2024-02-09T22:50:08.268960+0000","last_became_peered":"2024-02-09T22:50:08.268960+0000","last_unstale":"2024-02-09T22:50:08.273440+0000","last_undegraded":"2024-02-09T22:50:08.273440+0000","last_fullsized":"2024-02-09T22:50:08.273440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:41.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266756+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:50:08.266756+0000","last_peered":"2024-02-09T22:50:08.266756+0000","last_clean":"2024-02-09T22:50:08.266756+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:50:08.266756+0000","last_undegraded":"2024-02-09T22:50:08.266756+0000","last_fullsized":"2024-02-09T22:50:08.266756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.542393+0000","last_change":"2024-02-09T22:50:08.542393+0000","last_active":"2024-02-09T22:50:08.542393+0000","last_peered":"2024-02-09T22:50:08.542393+0000","last_clean":"2024-02-09T22:50:08.542393+0000","last_became_active":"2024-02-09T22:50:08.539084+0000","last_became_peered":"2024-02-09T22:50:08.539084+0000","last_unstale":"2024-02-09T22:50:08.542393+0000","last_undegraded":"2024-02-09T22:50:08.542393+0000","last_fullsized":"2024-02-09T22:50:08.542393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:32:59.713756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264947+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:50:08.264947+0000","last_peered":"2024-02-09T22:50:08.264947+0000","last_clean":"2024-02-09T22:50:08.264947+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:50:08.264947+0000","last_undegraded":"2024-02-09T22:50:08.264947+0000","last_fullsized":"2024-02-09T22:50:08.264947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:16.213228+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:50:16.213228+0000","last_peered":"2024-02-09T22:50:16.213228+0000","last_clean":"2024-02-09T22:50:16.213228+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:50:16.213228+0000","last_undegraded":"2024-02-09T22:50:16.213228+0000","last_fullsized":"2024-02-09T22:50:16.213228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265079+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:50:08.265079+0000","last_peered":"2024-02-09T22:50:08.265079+0000","last_clean":"2024-02-09T22:50:08.265079+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:50:08.265079+0000","last_undegraded":"2024-02-09T22:50:08.265079+0000","last_fullsized":"2024-02-09T22:50:08.265079+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615601+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:50:08.615601+0000","last_peered":"2024-02-09T22:50:08.615601+0000","last_clean":"2024-02-09T22:50:08.615601+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:50:08.615601+0000","last_undegraded":"2024-02-09T22:50:08.615601+0000","last_fullsized":"2024-02-09T22:50:08.615601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264821+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:50:08.264821+0000","last_peered":"2024-02-09T22:50:08.264821+0000","last_clean":"2024-02-09T22:50:08.264821+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:50:08.264821+0000","last_undegraded":"2024-02-09T22:50:08.264821+0000","last_fullsized":"2024-02-09T22:50:08.264821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615940+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:50:08.615940+0000","last_peered":"2024-02-09T22:50:08.615940+0000","last_clean":"2024-02-09T22:50:08.615940+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:50:08.615940+0000","last_undegraded":"2024-02-09T22:50:08.615940+0000","last_fullsized":"2024-02-09T22:50:08.615940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265111+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:50:08.265111+0000","last_peered":"2024-02-09T22:50:08.265111+0000","last_clean":"2024-02-09T22:50:08.265111+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:50:08.265111+0000","last_undegraded":"2024-02-09T22:50:08.265111+0000","last_fullsized":"2024-02-09T22:50:08.265111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263695+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:50:08.263695+0000","last_peered":"2024-02-09T22:50:08.263695+0000","last_clean":"2024-02-09T22:50:08.263695+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:50:08.263695+0000","last_undegraded":"2024-02-09T22:50:08.263695+0000","last_fullsized":"2024-02-09T22:50:08.263695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266089+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:50:08.266089+0000","last_peered":"2024-02-09T22:50:08.266089+0000","last_clean":"2024-02-09T22:50:08.266089+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:50:08.266089+0000","last_undegraded":"2024-02-09T22:50:08.266089+0000","last_fullsized":"2024-02-09T22:50:08.266089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:24:49.099644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.272126+0000","last_change":"2024-02-09T22:50:08.272126+0000","last_active":"2024-02-09T22:50:08.272126+0000","last_peered":"2024-02-09T22:50:08.272126+0000","last_clean":"2024-02-09T22:50:08.272126+0000","last_became_active":"2024-02-09T22:50:08.271722+0000","last_became_peered":"2024-02-09T22:50:08.271722+0000","last_unstale":"2024-02-09T22:50:08.272126+0000","last_undegraded":"2024-02-09T22:50:08.272126+0000","last_fullsized":"2024-02-09T22:50:08.272126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:43:31.304504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266306+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:50:08.266306+0000","last_peered":"2024-02-09T22:50:08.266306+0000","last_clean":"2024-02-09T22:50:08.266306+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:50:08.266306+0000","last_undegraded":"2024-02-09T22:50:08.266306+0000","last_fullsized":"2024-02-09T22:50:08.266306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615962+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:50:08.615962+0000","last_peered":"2024-02-09T22:50:08.615962+0000","last_clean":"2024-02-09T22:50:08.615962+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:50:08.615962+0000","last_undegraded":"2024-02-09T22:50:08.615962+0000","last_fullsized":"2024-02-09T22:50:08.615962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263896+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:50:08.263896+0000","last_peered":"2024-02-09T22:50:08.263896+0000","last_clean":"2024-02-09T22:50:08.263896+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:50:08.263896+0000","last_undegraded":"2024-02-09T22:50:08.263896+0000","last_fullsized":"2024-02-09T22:50:08.263896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266477+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:50:08.266477+0000","last_peered":"2024-02-09T22:50:08.266477+0000","last_clean":"2024-02-09T22:50:08.266477+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:50:08.266477+0000","last_undegraded":"2024-02-09T22:50:08.266477+0000","last_fullsized":"2024-02-09T22:50:08.266477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:57:01.921327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267050+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:50:08.267050+0000","last_peered":"2024-02-09T22:50:08.267050+0000","last_clean":"2024-02-09T22:50:08.267050+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:50:08.267050+0000","last_undegraded":"2024-02-09T22:50:08.267050+0000","last_fullsized":"2024-02-09T22:50:08.267050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615220+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:50:08.615220+0000","last_peered":"2024-02-09T22:50:08.615220+0000","last_clean":"2024-02-09T22:50:08.615220+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:50:08.615220+0000","last_undegraded":"2024-02-09T22:50:08.615220+0000","last_fullsized":"2024-02-09T22:50:08.615220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266093+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:50:08.266093+0000","last_peered":"2024-02-09T22:50:08.266093+0000","last_clean":"2024-02-09T22:50:08.266093+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:50:08.266093+0000","last_undegraded":"2024-02-09T22:50:08.266093+0000","last_fullsized":"2024-02-09T22:50:08.266093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537694+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:50:08.537694+0000","last_peered":"2024-02-09T22:50:08.537694+0000","last_clean":"2024-02-09T22:50:08.537694+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:50:08.537694+0000","last_undegraded":"2024-02-09T22:50:08.537694+0000","last_fullsized":"2024-02-09T22:50:08.537694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615862+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:50:08.615862+0000","last_peered":"2024-02-09T22:50:08.615862+0000","last_clean":"2024-02-09T22:50:08.615862+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:50:08.615862+0000","last_undegraded":"2024-02-09T22:50:08.615862+0000","last_fullsized":"2024-02-09T22:50:08.615862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263746+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:50:08.263746+0000","last_peered":"2024-02-09T22:50:08.263746+0000","last_clean":"2024-02-09T22:50:08.263746+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:50:08.263746+0000","last_undegraded":"2024-02-09T22:50:08.263746+0000","last_fullsized":"2024-02-09T22:50:08.263746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66207,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":679,"num_read_kb":409,"num_write":235,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351004,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.379}]},{"osd":1,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.41599999999999998}]},{"osd":2,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.39000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":4,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.505},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.54500000000000004}]},{"osd":6,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.47999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"last":0.75600000000000001}]},{"osd":10,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.746,"5min":0.746,"15min":0.746},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.59699999999999998}]}]},{"osd":10,"up_from":73,"seq":313532612640,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32804,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712348,"statfs":{"total":95995035648,"available":95961444352,"internally_reserved":0,"allocated":1662976,"data_stored":1113885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.83999999999999997}]},{"osd":1,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.495,"15min":0.499},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.66900000000000004}]},{"osd":2,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72899999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.67300000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.81100000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.94699999999999995},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76000000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.85699999999999998}]},{"osd":4,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.78200000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66400000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.52800000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96599999999999997,"5min":1.006,"15min":1.006},"last":1.0189999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91100000000000003,"5min":1.02,"15min":1.02},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71799999999999997,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78400000000000003,"15min":0.78700000000000003},"min":{"1min":0.58199999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.98199999999999998,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.79900000000000004,"15min":0.80400000000000005},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.95299999999999996,"5min":1.069,"15min":1.069},"last":0.996}]},{"osd":9,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.80900000000000005},"min":{"1min":0.64300000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.024,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":1.0409999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81499999999999995,"15min":0.82299999999999995},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98499999999999999,"5min":1.155,"15min":1.155},"last":0.93100000000000005}]},{"osd":11,"last update":"Fri Feb 9 22:49:16 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":1.0940000000000001}]}]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":463392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51400000000000001,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79300000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84399999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.71899999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.70999999999999996,"15min":0.71699999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93300000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.90700000000000003,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.72399999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99099999999999999,"5min":1.01,"15min":1.01},"last":0.502}]},{"osd":6,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.036,"5min":2.036,"15min":2.036},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79600000000000004,"15min":0.80500000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.04,"5min":1.177,"15min":1.177},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79800000000000004,"15min":0.80800000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.054,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.61799999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78800000000000003,"15min":0.79400000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.82999999999999996,"15min":0.83999999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.101,"5min":1.387,"15min":1.387},"last":0.70999999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.75600000000000001,"15min":0.754},"min":{"1min":0.54000000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.022,"5min":1.089,"15min":1.089},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78000000000000003,"15min":0.78100000000000003},"min":{"1min":0.59299999999999997,"5min":0.504,"15min":0.504},"max":{"1min":1.0309999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66900000000000004}]}]},{"osd":8,"up_from":59,"seq":253403070507,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":463658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.60599999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.749},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89300000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.371,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.46100000000000002},"min":{"1min":0.35799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.81899999999999995}]},{"osd":4,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97299999999999998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.55300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.86299999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.78900000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.63,"15min":0.621},"min":{"1min":0.46600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69899999999999995,"15min":0.69999999999999996},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.70599999999999996,"15min":0.68999999999999995},"min":{"1min":0.58099999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.85399999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.57199999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.67900000000000005,"15min":0.66400000000000003},"min":{"1min":0.61499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.95399999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.71099999999999997,"15min":0.68799999999999994},"min":{"1min":0.66100000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.72299999999999998,"15min":0.70199999999999996},"min":{"1min":0.68999999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.97699999999999998}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.0009999999999999}]}]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1581056,"data_stored":1056288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.47899999999999998,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.997,"15min":0.997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.38,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.749,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.505}]},{"osd":2,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":1.01,"15min":1.01},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64700000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63},"min":{"1min":0.49099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":1.111,"15min":1.111},"last":0.622},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45300000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.627,"15min":0.627},"min":{"1min":0.41399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79300000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65900000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65800000000000003,"15min":0.65600000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.872,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71199999999999997,"15min":0.70299999999999996},"min":{"1min":0.48899999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89500000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.81499999999999995}]},{"osd":6,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.72499999999999998},"min":{"1min":0.628,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.47499999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.79900000000000004}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96599999999999997,"5min":1.016,"15min":1.016},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.73199999999999998,"15min":0.70399999999999996},"min":{"1min":0.60499999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.82899999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.873},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97499999999999998,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.77300000000000002}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82799999999999996,"15min":0.82399999999999995},"min":{"1min":0.65900000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.96899999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83499999999999996,"15min":0.83199999999999996},"min":{"1min":0.66700000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.85799999999999998}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.90700000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495669,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.376},{"interface":"front","average":{"1min":0.373,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40699999999999997}]},{"osd":1,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.42199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.505}]},{"osd":2,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55300000000000005,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.875,"15min":0.875},"last":0.67100000000000004}]},{"osd":3,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.436},{"interface":"front","average":{"1min":0.441,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.57299999999999995}]},{"osd":4,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.443,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.59299999999999997}]},{"osd":5,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.79700000000000004}]},{"osd":7,"last update":"Fri Feb 9 22:49:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92500000000000004,"5min":0.999,"15min":0.999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64800000000000002,"15min":0.63400000000000001},"min":{"1min":0.47699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85499999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.63700000000000001}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84799999999999998,"5min":0.876,"15min":0.876},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.81200000000000006,"5min":0.995,"15min":0.995},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Feb 9 22:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.68000000000000005,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.75},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.67200000000000004,"15min":0.68700000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90400000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.71699999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.56100000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.86799999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.625},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.77300000000000002},"min":{"1min":0.51900000000000002,"5min":0.499,"15min":0.499},"max":{"1min":0.94099999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.76000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":523416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57699999999999996,"15min":0.52700000000000002},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.76300000000000001,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.80600000000000005,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.73499999999999999,"15min":0.58899999999999997},"min":{"1min":0.47899999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.94799999999999995,"5min":18.18,"15min":18.18},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.56899999999999995,"15min":0.50800000000000001},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71099999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.94099999999999995,"5min":1.206,"15min":1.206},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.59199999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.48099999999999998,"15min":0.45100000000000001},"min":{"1min":0.36699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.46899999999999997,"15min":0.42499999999999999},"min":{"1min":0.35999999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.66800000000000004,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.67800000000000005,"15min":0.621},"min":{"1min":0.59199999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.75900000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66600000000000004,"15min":0.64500000000000002},"min":{"1min":0.48999999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Fri Feb 9 22:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63900000000000001,"15min":0.621},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68400000000000005,"15min":0.66500000000000004},"min":{"1min":0.435,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.87}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.353,"15min":1.581},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1439999999999999,"5min":18.132000000000001,"15min":18.132000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69999999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.60499999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.85599999999999998,"15min":0.85399999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.78000000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.746},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.63600000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.84599999999999997}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.80000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":464448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55400000000000005,"15min":0.48999999999999999},"min":{"1min":0.42599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88700000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.378},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52000000000000002,"15min":0.47999999999999998},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.745,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40600000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54600000000000004,"15min":0.52000000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.503},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.378,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.78200000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.745,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59299999999999997},"min":{"1min":0.376,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85899999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81399999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56599999999999995,"15min":0.57299999999999995},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84999999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.48999999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.70499999999999996,"15min":0.73199999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.93999999999999995,"5min":1.081,"15min":1.081},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.72999999999999998,"15min":0.75800000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89700000000000002,"5min":1.202,"15min":1.202},"last":0.71299999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.78200000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.91600000000000004,"5min":1.137,"15min":1.137},"last":0.66500000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":524027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.45600000000000002},"min":{"1min":0.40899999999999997,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.51600000000000001,"15min":0.47999999999999998},"min":{"1min":0.38,"5min":0.224,"15min":0.224},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.53600000000000003}]},{"osd":1,"last update":"Fri Feb 9 22:49:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.52600000000000002,"15min":0.47799999999999998},"min":{"1min":0.377,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.55200000000000005,"15min":0.48599999999999999},"min":{"1min":0.33700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.85299999999999998,"5min":0.874,"15min":0.874},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55900000000000005,"15min":0.49099999999999999},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.48999999999999999},"min":{"1min":0.39900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57499999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.80300000000000005}]},{"osd":5,"last update":"Fri Feb 9 22:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.45800000000000002,"15min":0.42199999999999999},"min":{"1min":0.34399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.66600000000000004}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.874,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90700000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.51900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.625}]},{"osd":9,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.85199999999999998}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.82799999999999996}]},{"osd":11,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.77900000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182467,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":462934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57899999999999996,"5min":0.62,"15min":0.62},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.36199999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.53100000000000003},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78100000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.53500000000000003},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.753,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61099999999999999,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84299999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.66500000000000004,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.9139999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":0.69099999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:49:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.443,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91900000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.435,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.53700000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.747,"5min":0.873,"15min":0.873},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.82799999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.39600000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.433,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.81499999999999995,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.873,"5min":0.75,"15min":0.70799999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.8789999999999996,"5min":5.8789999999999996,"15min":5.8789999999999996},"last":0.59199999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.754,"15min":0.77800000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95099999999999996,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0229999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.442}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0640000000000001,"5min":1.26,"15min":1.26},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.73799999999999999,"15min":0.747},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93999999999999995,"5min":0.999,"15min":0.999},"last":0.72899999999999998}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.71699999999999997}]}]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":462970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.39800000000000002},"min":{"1min":0.28699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.57499999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.433,"15min":0.39400000000000002},"min":{"1min":0.308,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85099999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.379,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.871}]},{"osd":3,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.57399999999999995},"min":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.434},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60899999999999999,"15min":0.57699999999999996},"min":{"1min":0.51300000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.64700000000000002,"15min":0.59399999999999997},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92100000000000004,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.41099999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.79900000000000004,"5min":1.056,"15min":1.056},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64600000000000002,"15min":0.63},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.73499999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65500000000000003,"15min":0.65000000000000002},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88100000000000001,"5min":1.097,"15min":1.097},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63600000000000001,"15min":0.627},"min":{"1min":0.44400000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.621}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93300000000000005,"5min":1.129,"15min":1.129},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76900000000000002,"15min":0.748},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.264,"15min":1.264},"last":0.63500000000000001}]},{"osd":9,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80200000000000005,"15min":0.81100000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96799999999999997,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76900000000000002,"15min":0.77900000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.196,"15min":1.196},"last":0.81899999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.83899999999999997,"15min":0.84099999999999997},"min":{"1min":0.53000000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.026,"5min":1.179,"15min":1.179},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95999999999999996,"5min":1.238,"15min":1.238},"last":0.70499999999999996}]},{"osd":11,"last update":"Fri Feb 9 22:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.77200000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757306,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32740,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712412,"statfs":{"total":95995035648,"available":95961509888,"internally_reserved":0,"allocated":1593344,"data_stored":1055966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.49299999999999999},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70899999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.50600000000000001,"15min":0.47599999999999998},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.53900000000000003,"15min":0.51400000000000001},"min":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.753,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.71499999999999997}]},{"osd":2,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.39600000000000002},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.60799999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.41799999999999998,"15min":0.39800000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71499999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.61399999999999999}]},{"osd":4,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.503,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78600000000000003,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.93500000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:49:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.59299999999999997,"15min":0.58699999999999997},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.90100000000000002,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.58899999999999997,"15min":0.57699999999999996},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.878,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.81299999999999994}]},{"osd":7,"last update":"Fri Feb 9 22:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.878},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.89100000000000001}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.872,"5min":0.996,"15min":0.996},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.85799999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.94699999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73699999999999999,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.91700000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96299999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.97899999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:50:21.307 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph pg dump --format=json 2024-02-09T22:50:21.558 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:21.938 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:21 smithi003 ceph-mon[32678]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:50:21.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:21.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:21.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:50:21.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:50:21.971 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:21 smithi003 ceph-mon[32678]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:21.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:21 smithi089 ceph-mon[36665]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:50:21.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:21.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:21.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:50:21.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:50:21.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:21 smithi089 ceph-mon[36665]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:22.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:21 smithi064 ceph-mon[36638]: from='client.14970 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:50:22.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:50:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:50:22.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:21 smithi064 ceph-mon[36638]: from='mgr.14148 172.21.15.3:0/1645460186' entity='mgr.x' 2024-02-09T22:50:22.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:22 smithi089 ceph-mon[36665]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-02-09T22:50:23.009 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:22 smithi003 ceph-mon[32678]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-02-09T22:50:23.087 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:22 smithi064 ceph-mon[36638]: pgmap v326: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 10 op/s; 9 B/s, 0 objects/s recovering 2024-02-09T22:50:23.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 -- 172.21.15.3:0/2206644513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 msgr2=0x7f3108100f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 --2- 172.21.15.3:0/2206644513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108100f80 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f30fc009990 tx=0x7f30fc009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 -- 172.21.15.3:0/2206644513 shutdown_connections 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 --2- 172.21.15.3:0/2206644513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3108106b90 0x7f3108106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 --2- 172.21.15.3:0/2206644513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31081014c0 0x7f310810d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 --2- 172.21.15.3:0/2206644513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108100f80 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 -- 172.21.15.3:0/2206644513 >> 172.21.15.3:0/2206644513 conn(0x7f31080fc5f0 msgr2=0x7f31080fea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 -- 172.21.15.3:0/2206644513 shutdown_connections 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.110+0000 7f310cba1700 1 -- 172.21.15.3:0/2206644513 wait complete. 2024-02-09T22:50:23.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 Processor -- start 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 -- start start 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108198ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31081014c0 0x7f3108199530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3108106b90 0x7f31081a4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3108199dd0 con 0x7f3108100b00 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31081a45b0 con 0x7f3108106b90 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f310cba1700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31081a46f0 con 0x7f31081014c0 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f31077fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108198ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:23.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f31077fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108198ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55206/0 (socket says 172.21.15.3:55206) 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.111+0000 7f31077fe700 1 -- 172.21.15.3:0/1892335437 learned_addr learned my addr 172.21.15.3:0/1892335437 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f3106ffd700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31081014c0 0x7f3108199530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f31077fe700 1 -- 172.21.15.3:0/1892335437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31081014c0 msgr2=0x7f3108199530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f31077fe700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31081014c0 0x7f3108199530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f31077fe700 1 -- 172.21.15.3:0/1892335437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3108106b90 msgr2=0x7f31081a4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f31077fe700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3108106b90 0x7f31081a4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f31077fe700 1 -- 172.21.15.3:0/1892335437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30fc0095f0 con 0x7f3108100b00 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f31077fe700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108198ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f30f800d9b0 tx=0x7f30f800dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f800bb60 con 0x7f3108100b00 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30f8007e80 con 0x7f3108100b00 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30f8004e20 con 0x7f3108100b00 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31081a49a0 con 0x7f3108100b00 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.112+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31081a4ef0 con 0x7f3108100b00 2024-02-09T22:50:23.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.113+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3108109120 con 0x7f3108100b00 2024-02-09T22:50:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.114+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f30f8004020 con 0x7f3108100b00 2024-02-09T22:50:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.115+0000 7f3104ff9700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f30f00a2260 0x7f30f00a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.115+0000 7f3106ffd700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f30f00a2260 0x7f30f00a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.115+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f30f80cdc00 con 0x7f3108100b00 2024-02-09T22:50:23.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.115+0000 7f3106ffd700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f30f00a2260 0x7f30f00a4720 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f30fc006010 tx=0x7f30fc01a040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:23.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.120+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f30f8096a80 con 0x7f3108100b00 2024-02-09T22:50:23.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.317+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3108110c10 con 0x7f30f00a2260 2024-02-09T22:50:23.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.328+0000 7f3104ff9700 1 -- 172.21.15.3:0/1892335437 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+344040 (secure 0 0 0) 0x7f3108110c10 con 0x7f30f00a2260 2024-02-09T22:50:23.336 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:23.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f30f00a2260 msgr2=0x7f30f00a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:23.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f30f00a2260 0x7f30f00a4720 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f30fc006010 tx=0x7f30fc01a040 comp rx=0 tx=0).stop 2024-02-09T22:50:23.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 msgr2=0x7f3108198ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108198ff0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f30f800d9b0 tx=0x7f30f800dd70 comp rx=0 tx=0).stop 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 shutdown_connections 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f30f00a2260 0x7f30f00a4720 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3108106b90 0x7f31081a4070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3108100b00 0x7f3108198ff0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 --2- 172.21.15.3:0/1892335437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31081014c0 0x7f3108199530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 >> 172.21.15.3:0/1892335437 conn(0x7f31080fc5f0 msgr2=0x7f31080fcb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 shutdown_connections 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:23.331+0000 7f310cba1700 1 -- 172.21.15.3:0/1892335437 wait complete. 2024-02-09T22:50:23.339 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:50:23.966 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:23 smithi003 ceph-mon[32678]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:50:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:23 smithi089 ceph-mon[36665]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:50:23.994 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":326,"stamp":"2024-02-09T22:50:22.556450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":656809,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":845,"num_read_kb":548,"num_write":418,"num_write_kb":3070,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":343,"ondisk_log_size":343,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":354364,"kb_used_data":13452,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124587460,"statfs":{"total":1151940427776,"available":1151577559040,"internally_reserved":0,"allocated":13774848,"data_stored":7517667,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":348894521},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-5,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":129,"num_read_kb":69,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003464"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264215+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:50:08.264215+0000","last_peered":"2024-02-09T22:50:08.264215+0000","last_clean":"2024-02-09T22:50:08.264215+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:50:08.264215+0000","last_undegraded":"2024-02-09T22:50:08.264215+0000","last_fullsized":"2024-02-09T22:50:08.264215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265998+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:50:08.265998+0000","last_peered":"2024-02-09T22:50:08.265998+0000","last_clean":"2024-02-09T22:50:08.265998+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:50:08.265998+0000","last_undegraded":"2024-02-09T22:50:08.265998+0000","last_fullsized":"2024-02-09T22:50:08.265998+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266453+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:50:08.266453+0000","last_peered":"2024-02-09T22:50:08.266453+0000","last_clean":"2024-02-09T22:50:08.266453+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:50:08.266453+0000","last_undegraded":"2024-02-09T22:50:08.266453+0000","last_fullsized":"2024-02-09T22:50:08.266453+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538092+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:50:08.538092+0000","last_peered":"2024-02-09T22:50:08.538092+0000","last_clean":"2024-02-09T22:50:08.538092+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:50:08.538092+0000","last_undegraded":"2024-02-09T22:50:08.538092+0000","last_fullsized":"2024-02-09T22:50:08.538092+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:50.286530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266623+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:50:08.266623+0000","last_peered":"2024-02-09T22:50:08.266623+0000","last_clean":"2024-02-09T22:50:08.266623+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:50:08.266623+0000","last_undegraded":"2024-02-09T22:50:08.266623+0000","last_fullsized":"2024-02-09T22:50:08.266623+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515923+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:50:08.515923+0000","last_peered":"2024-02-09T22:50:08.515923+0000","last_clean":"2024-02-09T22:50:08.515923+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:50:08.515923+0000","last_undegraded":"2024-02-09T22:50:08.515923+0000","last_fullsized":"2024-02-09T22:50:08.515923+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264736+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:50:08.264736+0000","last_peered":"2024-02-09T22:50:08.264736+0000","last_clean":"2024-02-09T22:50:08.264736+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:50:08.264736+0000","last_undegraded":"2024-02-09T22:50:08.264736+0000","last_fullsized":"2024-02-09T22:50:08.264736+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265839+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:50:08.265839+0000","last_peered":"2024-02-09T22:50:08.265839+0000","last_clean":"2024-02-09T22:50:08.265839+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:50:08.265839+0000","last_undegraded":"2024-02-09T22:50:08.265839+0000","last_fullsized":"2024-02-09T22:50:08.265839+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538413+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:50:08.538413+0000","last_peered":"2024-02-09T22:50:08.538413+0000","last_clean":"2024-02-09T22:50:08.538413+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:50:08.538413+0000","last_undegraded":"2024-02-09T22:50:08.538413+0000","last_fullsized":"2024-02-09T22:50:08.538413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.516201+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:50:08.516201+0000","last_peered":"2024-02-09T22:50:08.516201+0000","last_clean":"2024-02-09T22:50:08.516201+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:50:08.516201+0000","last_undegraded":"2024-02-09T22:50:08.516201+0000","last_fullsized":"2024-02-09T22:50:08.516201+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265144+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:50:08.265144+0000","last_peered":"2024-02-09T22:50:08.265144+0000","last_clean":"2024-02-09T22:50:08.265144+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:50:08.265144+0000","last_undegraded":"2024-02-09T22:50:08.265144+0000","last_fullsized":"2024-02-09T22:50:08.265144+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.517473+0000","last_change":"2024-02-09T22:50:08.517050+0000","last_active":"2024-02-09T22:50:08.517473+0000","last_peered":"2024-02-09T22:50:08.517473+0000","last_clean":"2024-02-09T22:50:08.517473+0000","last_became_active":"2024-02-09T22:50:08.516688+0000","last_became_peered":"2024-02-09T22:50:08.516688+0000","last_unstale":"2024-02-09T22:50:08.517473+0000","last_undegraded":"2024-02-09T22:50:08.517473+0000","last_fullsized":"2024-02-09T22:50:08.517473+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:44:51.672735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615737+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:50:08.615737+0000","last_peered":"2024-02-09T22:50:08.615737+0000","last_clean":"2024-02-09T22:50:08.615737+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:50:08.615737+0000","last_undegraded":"2024-02-09T22:50:08.615737+0000","last_fullsized":"2024-02-09T22:50:08.615737+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266475+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:50:08.266475+0000","last_peered":"2024-02-09T22:50:08.266475+0000","last_clean":"2024-02-09T22:50:08.266475+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:50:08.266475+0000","last_undegraded":"2024-02-09T22:50:08.266475+0000","last_fullsized":"2024-02-09T22:50:08.266475+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.616823+0000","last_change":"2024-02-09T22:50:08.616823+0000","last_active":"2024-02-09T22:50:08.616823+0000","last_peered":"2024-02-09T22:50:08.616823+0000","last_clean":"2024-02-09T22:50:08.616823+0000","last_became_active":"2024-02-09T22:50:08.616482+0000","last_became_peered":"2024-02-09T22:50:08.616482+0000","last_unstale":"2024-02-09T22:50:08.616823+0000","last_undegraded":"2024-02-09T22:50:08.616823+0000","last_fullsized":"2024-02-09T22:50:08.616823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:57:44.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537554+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:50:08.537554+0000","last_peered":"2024-02-09T22:50:08.537554+0000","last_clean":"2024-02-09T22:50:08.537554+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:50:08.537554+0000","last_undegraded":"2024-02-09T22:50:08.537554+0000","last_fullsized":"2024-02-09T22:50:08.537554+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264328+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:50:08.264328+0000","last_peered":"2024-02-09T22:50:08.264328+0000","last_clean":"2024-02-09T22:50:08.264328+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:50:08.264328+0000","last_undegraded":"2024-02-09T22:50:08.264328+0000","last_fullsized":"2024-02-09T22:50:08.264328+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263394+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:50:08.263394+0000","last_peered":"2024-02-09T22:50:08.263394+0000","last_clean":"2024-02-09T22:50:08.263394+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:50:08.263394+0000","last_undegraded":"2024-02-09T22:50:08.263394+0000","last_fullsized":"2024-02-09T22:50:08.263394+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266124+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:50:08.266124+0000","last_peered":"2024-02-09T22:50:08.266124+0000","last_clean":"2024-02-09T22:50:08.266124+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:50:08.266124+0000","last_undegraded":"2024-02-09T22:50:08.266124+0000","last_fullsized":"2024-02-09T22:50:08.266124+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:39.090835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":100,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:18.213108+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:50:18.213108+0000","last_peered":"2024-02-09T22:50:18.213108+0000","last_clean":"2024-02-09T22:50:18.213108+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:50:18.213108+0000","last_undegraded":"2024-02-09T22:50:18.213108+0000","last_fullsized":"2024-02-09T22:50:18.213108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266854+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:50:08.266854+0000","last_peered":"2024-02-09T22:50:08.266854+0000","last_clean":"2024-02-09T22:50:08.266854+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:50:08.266854+0000","last_undegraded":"2024-02-09T22:50:08.266854+0000","last_fullsized":"2024-02-09T22:50:08.266854+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267136+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:50:08.267136+0000","last_peered":"2024-02-09T22:50:08.267136+0000","last_clean":"2024-02-09T22:50:08.267136+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:50:08.267136+0000","last_undegraded":"2024-02-09T22:50:08.267136+0000","last_fullsized":"2024-02-09T22:50:08.267136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266068+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:50:08.266068+0000","last_peered":"2024-02-09T22:50:08.266068+0000","last_clean":"2024-02-09T22:50:08.266068+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:50:08.266068+0000","last_undegraded":"2024-02-09T22:50:08.266068+0000","last_fullsized":"2024-02-09T22:50:08.266068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267805+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:50:08.267805+0000","last_peered":"2024-02-09T22:50:08.267805+0000","last_clean":"2024-02-09T22:50:08.267805+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:50:08.267805+0000","last_undegraded":"2024-02-09T22:50:08.267805+0000","last_fullsized":"2024-02-09T22:50:08.267805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537848+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:50:08.537848+0000","last_peered":"2024-02-09T22:50:08.537848+0000","last_clean":"2024-02-09T22:50:08.537848+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:50:08.537848+0000","last_undegraded":"2024-02-09T22:50:08.537848+0000","last_fullsized":"2024-02-09T22:50:08.537848+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267118+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:50:08.267118+0000","last_peered":"2024-02-09T22:50:08.267118+0000","last_clean":"2024-02-09T22:50:08.267118+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:50:08.267118+0000","last_undegraded":"2024-02-09T22:50:08.267118+0000","last_fullsized":"2024-02-09T22:50:08.267118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615674+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:50:08.615674+0000","last_peered":"2024-02-09T22:50:08.615674+0000","last_clean":"2024-02-09T22:50:08.615674+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:50:08.615674+0000","last_undegraded":"2024-02-09T22:50:08.615674+0000","last_fullsized":"2024-02-09T22:50:08.615674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":96,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:20.218580+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:50:20.218580+0000","last_peered":"2024-02-09T22:50:20.218580+0000","last_clean":"2024-02-09T22:50:20.218580+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:50:20.218580+0000","last_undegraded":"2024-02-09T22:50:20.218580+0000","last_fullsized":"2024-02-09T22:50:20.218580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267696+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:50:08.267696+0000","last_peered":"2024-02-09T22:50:08.267696+0000","last_clean":"2024-02-09T22:50:08.267696+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:50:08.267696+0000","last_undegraded":"2024-02-09T22:50:08.267696+0000","last_fullsized":"2024-02-09T22:50:08.267696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515862+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:50:08.515862+0000","last_peered":"2024-02-09T22:50:08.515862+0000","last_clean":"2024-02-09T22:50:08.515862+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:50:08.515862+0000","last_undegraded":"2024-02-09T22:50:08.515862+0000","last_fullsized":"2024-02-09T22:50:08.515862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615664+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:50:08.615664+0000","last_peered":"2024-02-09T22:50:08.615664+0000","last_clean":"2024-02-09T22:50:08.615664+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:50:08.615664+0000","last_undegraded":"2024-02-09T22:50:08.615664+0000","last_fullsized":"2024-02-09T22:50:08.615664+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615818+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:50:08.615818+0000","last_peered":"2024-02-09T22:50:08.615818+0000","last_clean":"2024-02-09T22:50:08.615818+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:50:08.615818+0000","last_undegraded":"2024-02-09T22:50:08.615818+0000","last_fullsized":"2024-02-09T22:50:08.615818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266385+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:50:08.266385+0000","last_peered":"2024-02-09T22:50:08.266385+0000","last_clean":"2024-02-09T22:50:08.266385+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:50:08.266385+0000","last_undegraded":"2024-02-09T22:50:08.266385+0000","last_fullsized":"2024-02-09T22:50:08.266385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615165+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:50:08.615165+0000","last_peered":"2024-02-09T22:50:08.615165+0000","last_clean":"2024-02-09T22:50:08.615165+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:50:08.615165+0000","last_undegraded":"2024-02-09T22:50:08.615165+0000","last_fullsized":"2024-02-09T22:50:08.615165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263175+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:50:08.263175+0000","last_peered":"2024-02-09T22:50:08.263175+0000","last_clean":"2024-02-09T22:50:08.263175+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:50:08.263175+0000","last_undegraded":"2024-02-09T22:50:08.263175+0000","last_fullsized":"2024-02-09T22:50:08.263175+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":159,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:20.217890+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:50:20.217890+0000","last_peered":"2024-02-09T22:50:20.217890+0000","last_clean":"2024-02-09T22:50:20.217890+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:50:20.217890+0000","last_undegraded":"2024-02-09T22:50:20.217890+0000","last_fullsized":"2024-02-09T22:50:20.217890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":136,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538198+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:50:08.538198+0000","last_peered":"2024-02-09T22:50:08.538198+0000","last_clean":"2024-02-09T22:50:08.538198+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:50:08.538198+0000","last_undegraded":"2024-02-09T22:50:08.538198+0000","last_fullsized":"2024-02-09T22:50:08.538198+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264958+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:50:08.264958+0000","last_peered":"2024-02-09T22:50:08.264958+0000","last_clean":"2024-02-09T22:50:08.264958+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:50:08.264958+0000","last_undegraded":"2024-02-09T22:50:08.264958+0000","last_fullsized":"2024-02-09T22:50:08.264958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266372+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:50:08.266372+0000","last_peered":"2024-02-09T22:50:08.266372+0000","last_clean":"2024-02-09T22:50:08.266372+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:50:08.266372+0000","last_undegraded":"2024-02-09T22:50:08.266372+0000","last_fullsized":"2024-02-09T22:50:08.266372+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615718+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:50:08.615718+0000","last_peered":"2024-02-09T22:50:08.615718+0000","last_clean":"2024-02-09T22:50:08.615718+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:50:08.615718+0000","last_undegraded":"2024-02-09T22:50:08.615718+0000","last_fullsized":"2024-02-09T22:50:08.615718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538017+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:50:08.538017+0000","last_peered":"2024-02-09T22:50:08.538017+0000","last_clean":"2024-02-09T22:50:08.538017+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:50:08.538017+0000","last_undegraded":"2024-02-09T22:50:08.538017+0000","last_fullsized":"2024-02-09T22:50:08.538017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'29","reported_seq":190,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:20.217825+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:50:20.217825+0000","last_peered":"2024-02-09T22:50:20.217825+0000","last_clean":"2024-02-09T22:50:20.217825+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:50:20.217825+0000","last_undegraded":"2024-02-09T22:50:20.217825+0000","last_fullsized":"2024-02-09T22:50:20.217825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59394,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":142,"num_read_kb":129,"num_write":31,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266349+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:50:08.266349+0000","last_peered":"2024-02-09T22:50:08.266349+0000","last_clean":"2024-02-09T22:50:08.266349+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:50:08.266349+0000","last_undegraded":"2024-02-09T22:50:08.266349+0000","last_fullsized":"2024-02-09T22:50:08.266349+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267676+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:50:08.267676+0000","last_peered":"2024-02-09T22:50:08.267676+0000","last_clean":"2024-02-09T22:50:08.267676+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:50:08.267676+0000","last_undegraded":"2024-02-09T22:50:08.267676+0000","last_fullsized":"2024-02-09T22:50:08.267676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615712+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:50:08.615712+0000","last_peered":"2024-02-09T22:50:08.615712+0000","last_clean":"2024-02-09T22:50:08.615712+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:50:08.615712+0000","last_undegraded":"2024-02-09T22:50:08.615712+0000","last_fullsized":"2024-02-09T22:50:08.615712+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615556+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:50:08.615556+0000","last_peered":"2024-02-09T22:50:08.615556+0000","last_clean":"2024-02-09T22:50:08.615556+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:50:08.615556+0000","last_undegraded":"2024-02-09T22:50:08.615556+0000","last_fullsized":"2024-02-09T22:50:08.615556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264889+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:50:08.264889+0000","last_peered":"2024-02-09T22:50:08.264889+0000","last_clean":"2024-02-09T22:50:08.264889+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:50:08.264889+0000","last_undegraded":"2024-02-09T22:50:08.264889+0000","last_fullsized":"2024-02-09T22:50:08.264889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:16.227362+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:50:16.227362+0000","last_peered":"2024-02-09T22:50:16.227362+0000","last_clean":"2024-02-09T22:50:16.227362+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:50:16.227362+0000","last_undegraded":"2024-02-09T22:50:16.227362+0000","last_fullsized":"2024-02-09T22:50:16.227362+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":68,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264613+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:50:08.264613+0000","last_peered":"2024-02-09T22:50:08.264613+0000","last_clean":"2024-02-09T22:50:08.264613+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:50:08.264613+0000","last_undegraded":"2024-02-09T22:50:08.264613+0000","last_fullsized":"2024-02-09T22:50:08.264613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265893+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:50:08.265893+0000","last_peered":"2024-02-09T22:50:08.265893+0000","last_clean":"2024-02-09T22:50:08.265893+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:50:08.265893+0000","last_undegraded":"2024-02-09T22:50:08.265893+0000","last_fullsized":"2024-02-09T22:50:08.265893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.539717+0000","last_change":"2024-02-09T22:50:08.539326+0000","last_active":"2024-02-09T22:50:08.539717+0000","last_peered":"2024-02-09T22:50:08.539717+0000","last_clean":"2024-02-09T22:50:08.539717+0000","last_became_active":"2024-02-09T22:50:08.539091+0000","last_became_peered":"2024-02-09T22:50:08.539091+0000","last_unstale":"2024-02-09T22:50:08.539717+0000","last_undegraded":"2024-02-09T22:50:08.539717+0000","last_fullsized":"2024-02-09T22:50:08.539717+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:31:57.282733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267149+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:50:08.267149+0000","last_peered":"2024-02-09T22:50:08.267149+0000","last_clean":"2024-02-09T22:50:08.267149+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:50:08.267149+0000","last_undegraded":"2024-02-09T22:50:08.267149+0000","last_fullsized":"2024-02-09T22:50:08.267149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.616790+0000","last_change":"2024-02-09T22:50:08.616790+0000","last_active":"2024-02-09T22:50:08.616790+0000","last_peered":"2024-02-09T22:50:08.616790+0000","last_clean":"2024-02-09T22:50:08.616790+0000","last_became_active":"2024-02-09T22:50:08.616404+0000","last_became_peered":"2024-02-09T22:50:08.616404+0000","last_unstale":"2024-02-09T22:50:08.616790+0000","last_undegraded":"2024-02-09T22:50:08.616790+0000","last_fullsized":"2024-02-09T22:50:08.616790+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:14:18.902026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263341+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:50:08.263341+0000","last_peered":"2024-02-09T22:50:08.263341+0000","last_clean":"2024-02-09T22:50:08.263341+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:50:08.263341+0000","last_undegraded":"2024-02-09T22:50:08.263341+0000","last_fullsized":"2024-02-09T22:50:08.263341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.516172+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:50:08.516172+0000","last_peered":"2024-02-09T22:50:08.516172+0000","last_clean":"2024-02-09T22:50:08.516172+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:50:08.516172+0000","last_undegraded":"2024-02-09T22:50:08.516172+0000","last_fullsized":"2024-02-09T22:50:08.516172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:19.214867+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:50:19.214867+0000","last_peered":"2024-02-09T22:50:19.214867+0000","last_clean":"2024-02-09T22:50:19.214867+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:50:19.214867+0000","last_undegraded":"2024-02-09T22:50:19.214867+0000","last_fullsized":"2024-02-09T22:50:19.214867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.516128+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:50:08.516128+0000","last_peered":"2024-02-09T22:50:08.516128+0000","last_clean":"2024-02-09T22:50:08.516128+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:50:08.516128+0000","last_undegraded":"2024-02-09T22:50:08.516128+0000","last_fullsized":"2024-02-09T22:50:08.516128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615341+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:50:08.615341+0000","last_peered":"2024-02-09T22:50:08.615341+0000","last_clean":"2024-02-09T22:50:08.615341+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:50:08.615341+0000","last_undegraded":"2024-02-09T22:50:08.615341+0000","last_fullsized":"2024-02-09T22:50:08.615341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":93,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:19.231450+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:50:19.231450+0000","last_peered":"2024-02-09T22:50:19.231450+0000","last_clean":"2024-02-09T22:50:19.231450+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:50:19.231450+0000","last_undegraded":"2024-02-09T22:50:19.231450+0000","last_fullsized":"2024-02-09T22:50:19.231450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":69,"num_read_kb":69,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266817+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:50:08.266817+0000","last_peered":"2024-02-09T22:50:08.266817+0000","last_clean":"2024-02-09T22:50:08.266817+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:50:08.266817+0000","last_undegraded":"2024-02-09T22:50:08.266817+0000","last_fullsized":"2024-02-09T22:50:08.266817+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538316+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:50:08.538316+0000","last_peered":"2024-02-09T22:50:08.538316+0000","last_clean":"2024-02-09T22:50:08.538316+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:50:08.538316+0000","last_undegraded":"2024-02-09T22:50:08.538316+0000","last_fullsized":"2024-02-09T22:50:08.538316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":353,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264693+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:50:08.264693+0000","last_peered":"2024-02-09T22:50:08.264693+0000","last_clean":"2024-02-09T22:50:08.264693+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:50:08.264693+0000","last_undegraded":"2024-02-09T22:50:08.264693+0000","last_fullsized":"2024-02-09T22:50:08.264693+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615683+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:50:08.615683+0000","last_peered":"2024-02-09T22:50:08.615683+0000","last_clean":"2024-02-09T22:50:08.615683+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:50:08.615683+0000","last_undegraded":"2024-02-09T22:50:08.615683+0000","last_fullsized":"2024-02-09T22:50:08.615683+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264784+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:50:08.264784+0000","last_peered":"2024-02-09T22:50:08.264784+0000","last_clean":"2024-02-09T22:50:08.264784+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:50:08.264784+0000","last_undegraded":"2024-02-09T22:50:08.264784+0000","last_fullsized":"2024-02-09T22:50:08.264784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.278066+0000","last_change":"2024-02-09T22:50:08.278066+0000","last_active":"2024-02-09T22:50:08.278066+0000","last_peered":"2024-02-09T22:50:08.278066+0000","last_clean":"2024-02-09T22:50:08.278066+0000","last_became_active":"2024-02-09T22:50:08.271486+0000","last_became_peered":"2024-02-09T22:50:08.271486+0000","last_unstale":"2024-02-09T22:50:08.278066+0000","last_undegraded":"2024-02-09T22:50:08.278066+0000","last_fullsized":"2024-02-09T22:50:08.278066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:54:12.295421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.616063+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:50:08.616063+0000","last_peered":"2024-02-09T22:50:08.616063+0000","last_clean":"2024-02-09T22:50:08.616063+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:50:08.616063+0000","last_undegraded":"2024-02-09T22:50:08.616063+0000","last_fullsized":"2024-02-09T22:50:08.616063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263461+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:50:08.263461+0000","last_peered":"2024-02-09T22:50:08.263461+0000","last_clean":"2024-02-09T22:50:08.263461+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:50:08.263461+0000","last_undegraded":"2024-02-09T22:50:08.263461+0000","last_fullsized":"2024-02-09T22:50:08.263461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267157+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:50:08.267157+0000","last_peered":"2024-02-09T22:50:08.267157+0000","last_clean":"2024-02-09T22:50:08.267157+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:50:08.267157+0000","last_undegraded":"2024-02-09T22:50:08.267157+0000","last_fullsized":"2024-02-09T22:50:08.267157+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538049+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:50:08.538049+0000","last_peered":"2024-02-09T22:50:08.538049+0000","last_clean":"2024-02-09T22:50:08.538049+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:50:08.538049+0000","last_undegraded":"2024-02-09T22:50:08.538049+0000","last_fullsized":"2024-02-09T22:50:08.538049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266104+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:50:08.266104+0000","last_peered":"2024-02-09T22:50:08.266104+0000","last_clean":"2024-02-09T22:50:08.266104+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:50:08.266104+0000","last_undegraded":"2024-02-09T22:50:08.266104+0000","last_fullsized":"2024-02-09T22:50:08.266104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537439+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:50:08.537439+0000","last_peered":"2024-02-09T22:50:08.537439+0000","last_clean":"2024-02-09T22:50:08.537439+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:50:08.537439+0000","last_undegraded":"2024-02-09T22:50:08.537439+0000","last_fullsized":"2024-02-09T22:50:08.537439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265754+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:50:08.265754+0000","last_peered":"2024-02-09T22:50:08.265754+0000","last_clean":"2024-02-09T22:50:08.265754+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:50:08.265754+0000","last_undegraded":"2024-02-09T22:50:08.265754+0000","last_fullsized":"2024-02-09T22:50:08.265754+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266189+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:50:08.266189+0000","last_peered":"2024-02-09T22:50:08.266189+0000","last_clean":"2024-02-09T22:50:08.266189+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:50:08.266189+0000","last_undegraded":"2024-02-09T22:50:08.266189+0000","last_fullsized":"2024-02-09T22:50:08.266189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263619+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:50:08.263619+0000","last_peered":"2024-02-09T22:50:08.263619+0000","last_clean":"2024-02-09T22:50:08.263619+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:50:08.263619+0000","last_undegraded":"2024-02-09T22:50:08.263619+0000","last_fullsized":"2024-02-09T22:50:08.263619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.273440+0000","last_change":"2024-02-09T22:50:08.273440+0000","last_active":"2024-02-09T22:50:08.273440+0000","last_peered":"2024-02-09T22:50:08.273440+0000","last_clean":"2024-02-09T22:50:08.273440+0000","last_became_active":"2024-02-09T22:50:08.268960+0000","last_became_peered":"2024-02-09T22:50:08.268960+0000","last_unstale":"2024-02-09T22:50:08.273440+0000","last_undegraded":"2024-02-09T22:50:08.273440+0000","last_fullsized":"2024-02-09T22:50:08.273440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:41.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266756+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:50:08.266756+0000","last_peered":"2024-02-09T22:50:08.266756+0000","last_clean":"2024-02-09T22:50:08.266756+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:50:08.266756+0000","last_undegraded":"2024-02-09T22:50:08.266756+0000","last_fullsized":"2024-02-09T22:50:08.266756+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.542393+0000","last_change":"2024-02-09T22:50:08.542393+0000","last_active":"2024-02-09T22:50:08.542393+0000","last_peered":"2024-02-09T22:50:08.542393+0000","last_clean":"2024-02-09T22:50:08.542393+0000","last_became_active":"2024-02-09T22:50:08.539084+0000","last_became_peered":"2024-02-09T22:50:08.539084+0000","last_unstale":"2024-02-09T22:50:08.542393+0000","last_undegraded":"2024-02-09T22:50:08.542393+0000","last_fullsized":"2024-02-09T22:50:08.542393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:32:59.713756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264947+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:50:08.264947+0000","last_peered":"2024-02-09T22:50:08.264947+0000","last_clean":"2024-02-09T22:50:08.264947+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:50:08.264947+0000","last_undegraded":"2024-02-09T22:50:08.264947+0000","last_fullsized":"2024-02-09T22:50:08.264947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":91,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:16.213228+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:50:16.213228+0000","last_peered":"2024-02-09T22:50:16.213228+0000","last_clean":"2024-02-09T22:50:16.213228+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:50:16.213228+0000","last_undegraded":"2024-02-09T22:50:16.213228+0000","last_fullsized":"2024-02-09T22:50:16.213228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":67,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265079+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:50:08.265079+0000","last_peered":"2024-02-09T22:50:08.265079+0000","last_clean":"2024-02-09T22:50:08.265079+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:50:08.265079+0000","last_undegraded":"2024-02-09T22:50:08.265079+0000","last_fullsized":"2024-02-09T22:50:08.265079+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615601+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:50:08.615601+0000","last_peered":"2024-02-09T22:50:08.615601+0000","last_clean":"2024-02-09T22:50:08.615601+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:50:08.615601+0000","last_undegraded":"2024-02-09T22:50:08.615601+0000","last_fullsized":"2024-02-09T22:50:08.615601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264821+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:50:08.264821+0000","last_peered":"2024-02-09T22:50:08.264821+0000","last_clean":"2024-02-09T22:50:08.264821+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:50:08.264821+0000","last_undegraded":"2024-02-09T22:50:08.264821+0000","last_fullsized":"2024-02-09T22:50:08.264821+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615940+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:50:08.615940+0000","last_peered":"2024-02-09T22:50:08.615940+0000","last_clean":"2024-02-09T22:50:08.615940+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:50:08.615940+0000","last_undegraded":"2024-02-09T22:50:08.615940+0000","last_fullsized":"2024-02-09T22:50:08.615940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265111+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:50:08.265111+0000","last_peered":"2024-02-09T22:50:08.265111+0000","last_clean":"2024-02-09T22:50:08.265111+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:50:08.265111+0000","last_undegraded":"2024-02-09T22:50:08.265111+0000","last_fullsized":"2024-02-09T22:50:08.265111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263695+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:50:08.263695+0000","last_peered":"2024-02-09T22:50:08.263695+0000","last_clean":"2024-02-09T22:50:08.263695+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:50:08.263695+0000","last_undegraded":"2024-02-09T22:50:08.263695+0000","last_fullsized":"2024-02-09T22:50:08.263695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266089+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:50:08.266089+0000","last_peered":"2024-02-09T22:50:08.266089+0000","last_clean":"2024-02-09T22:50:08.266089+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:50:08.266089+0000","last_undegraded":"2024-02-09T22:50:08.266089+0000","last_fullsized":"2024-02-09T22:50:08.266089+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:24:49.099644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.272126+0000","last_change":"2024-02-09T22:50:08.272126+0000","last_active":"2024-02-09T22:50:08.272126+0000","last_peered":"2024-02-09T22:50:08.272126+0000","last_clean":"2024-02-09T22:50:08.272126+0000","last_became_active":"2024-02-09T22:50:08.271722+0000","last_became_peered":"2024-02-09T22:50:08.271722+0000","last_unstale":"2024-02-09T22:50:08.272126+0000","last_undegraded":"2024-02-09T22:50:08.272126+0000","last_fullsized":"2024-02-09T22:50:08.272126+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:43:31.304504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266306+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:50:08.266306+0000","last_peered":"2024-02-09T22:50:08.266306+0000","last_clean":"2024-02-09T22:50:08.266306+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:50:08.266306+0000","last_undegraded":"2024-02-09T22:50:08.266306+0000","last_fullsized":"2024-02-09T22:50:08.266306+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615962+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:50:08.615962+0000","last_peered":"2024-02-09T22:50:08.615962+0000","last_clean":"2024-02-09T22:50:08.615962+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:50:08.615962+0000","last_undegraded":"2024-02-09T22:50:08.615962+0000","last_fullsized":"2024-02-09T22:50:08.615962+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263896+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:50:08.263896+0000","last_peered":"2024-02-09T22:50:08.263896+0000","last_clean":"2024-02-09T22:50:08.263896+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:50:08.263896+0000","last_undegraded":"2024-02-09T22:50:08.263896+0000","last_fullsized":"2024-02-09T22:50:08.263896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266477+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:50:08.266477+0000","last_peered":"2024-02-09T22:50:08.266477+0000","last_clean":"2024-02-09T22:50:08.266477+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:50:08.266477+0000","last_undegraded":"2024-02-09T22:50:08.266477+0000","last_fullsized":"2024-02-09T22:50:08.266477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:57:01.921327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267050+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:50:08.267050+0000","last_peered":"2024-02-09T22:50:08.267050+0000","last_clean":"2024-02-09T22:50:08.267050+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:50:08.267050+0000","last_undegraded":"2024-02-09T22:50:08.267050+0000","last_fullsized":"2024-02-09T22:50:08.267050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615220+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:50:08.615220+0000","last_peered":"2024-02-09T22:50:08.615220+0000","last_clean":"2024-02-09T22:50:08.615220+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:50:08.615220+0000","last_undegraded":"2024-02-09T22:50:08.615220+0000","last_fullsized":"2024-02-09T22:50:08.615220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266093+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:50:08.266093+0000","last_peered":"2024-02-09T22:50:08.266093+0000","last_clean":"2024-02-09T22:50:08.266093+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:50:08.266093+0000","last_undegraded":"2024-02-09T22:50:08.266093+0000","last_fullsized":"2024-02-09T22:50:08.266093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537694+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:50:08.537694+0000","last_peered":"2024-02-09T22:50:08.537694+0000","last_clean":"2024-02-09T22:50:08.537694+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:50:08.537694+0000","last_undegraded":"2024-02-09T22:50:08.537694+0000","last_fullsized":"2024-02-09T22:50:08.537694+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615862+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:50:08.615862+0000","last_peered":"2024-02-09T22:50:08.615862+0000","last_clean":"2024-02-09T22:50:08.615862+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:50:08.615862+0000","last_undegraded":"2024-02-09T22:50:08.615862+0000","last_fullsized":"2024-02-09T22:50:08.615862+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263746+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:50:08.263746+0000","last_peered":"2024-02-09T22:50:08.263746+0000","last_clean":"2024-02-09T22:50:08.263746+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:50:08.263746+0000","last_undegraded":"2024-02-09T22:50:08.263746+0000","last_fullsized":"2024-02-09T22:50:08.263746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":66207,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":709,"num_read_kb":429,"num_write":235,"num_write_kb":130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":198621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":204,"ondisk_log_size":204,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351005,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.68600000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63500000000000001,"15min":0.626},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.65100000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.44600000000000001},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.498}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.51500000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.77800000000000002}]},{"osd":5,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.755},{"interface":"front","average":{"1min":0.62,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.70699999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.66600000000000004},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.72299999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.84599999999999997},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.81299999999999994}]},{"osd":9,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.747,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.89500000000000002}]},{"osd":10,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92800000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.59699999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.91000000000000003}]}]},{"osd":10,"up_from":73,"seq":313532612641,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32804,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712348,"statfs":{"total":95995035648,"available":95961444352,"internally_reserved":0,"allocated":1662976,"data_stored":1113885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.73899999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.495,"15min":0.499},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.52500000000000002,"15min":0.52900000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.5}]},{"osd":2,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.65400000000000003,"15min":0.65800000000000003},"min":{"1min":0.42199999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.81000000000000005,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.65500000000000003,"15min":0.65400000000000003},"min":{"1min":0.442,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.69199999999999995}]},{"osd":3,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.66400000000000003,"15min":0.67300000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.81100000000000005,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.63100000000000001,"15min":0.64200000000000002},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.76000000000000001,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.78600000000000003}]},{"osd":4,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58099999999999996,"15min":0.58299999999999996},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.79300000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.41499999999999998}]},{"osd":5,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66400000000000003,"15min":0.66300000000000003},"min":{"1min":0.46400000000000002,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.85099999999999998},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.75,"15min":0.754},"min":{"1min":0.52800000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.96599999999999997,"5min":1.006,"15min":1.006},"last":0.76400000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.91100000000000003,"5min":1.02,"15min":1.02},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.68000000000000005,"5min":0.71799999999999997,"15min":0.72399999999999998},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.92000000000000004}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.76700000000000002,"5min":0.78400000000000003,"15min":0.78700000000000003},"min":{"1min":0.58199999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.98199999999999998,"5min":1.0840000000000001,"15min":1.0840000000000001},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.79900000000000004,"15min":0.80400000000000005},"min":{"1min":0.59999999999999998,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":0.95299999999999996,"5min":1.069,"15min":1.069},"last":0.81100000000000005}]},{"osd":9,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.80900000000000005},"min":{"1min":0.64300000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.024,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.81499999999999995,"15min":0.82299999999999995},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98499999999999999,"5min":1.155,"15min":1.155},"last":0.56000000000000005}]},{"osd":11,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.97899999999999998}]}]},{"osd":9,"up_from":67,"seq":287762808870,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":463392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.47399999999999998,"15min":0.46999999999999997},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.6040000000000001,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51400000000000001,"15min":0.51300000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.5629999999999999,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.51900000000000002}]},{"osd":1,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.63,"15min":0.63500000000000001},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.629,"15min":0.63500000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.69199999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.62,"15min":0.61799999999999999},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.625,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.79300000000000004,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.58999999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.55600000000000005,"15min":0.54800000000000004},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.246,"5min":2.246,"15min":2.246},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.57099999999999995,"15min":0.57899999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.84399999999999997,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.71899999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.68300000000000005},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.70999999999999996,"15min":0.71699999999999997},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.93300000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.70599999999999996,"15min":0.71199999999999997},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":0.90700000000000003,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.69999999999999996,"5min":0.71999999999999997,"15min":0.72399999999999998},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.99099999999999999,"5min":1.01,"15min":1.01},"last":0.502}]},{"osd":6,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.69199999999999995,"15min":0.70199999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.73699999999999999,"15min":0.746},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":2.036,"5min":2.036,"15min":2.036},"last":0.63400000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.73999999999999999,"5min":0.79600000000000004,"15min":0.80500000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":1.04,"5min":1.177,"15min":1.177},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.79800000000000004,"15min":0.80800000000000005},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.054,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.61799999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.78800000000000003,"15min":0.79400000000000004},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.173,"5min":1.173,"15min":1.173},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.77000000000000002,"5min":0.82999999999999996,"15min":0.83999999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.101,"5min":1.387,"15min":1.387},"last":0.70999999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.75600000000000001,"15min":0.754},"min":{"1min":0.54000000000000004,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.022,"5min":1.089,"15min":1.089},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.78000000000000003,"15min":0.78100000000000003},"min":{"1min":0.59299999999999997,"5min":0.504,"15min":0.504},"max":{"1min":1.0309999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.66900000000000004}]}]},{"osd":8,"up_from":59,"seq":253403070507,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":463658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58699999999999997}]},{"osd":1,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.60599999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.749},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89300000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.69499999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.371,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.46100000000000002},"min":{"1min":0.35799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.56100000000000005}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.81899999999999995}]},{"osd":4,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97299999999999998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.748,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.55300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.86299999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.78900000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.63,"15min":0.621},"min":{"1min":0.46600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69899999999999995,"15min":0.69999999999999996},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.70599999999999996,"15min":0.68999999999999995},"min":{"1min":0.58099999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.85399999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.67200000000000004,"15min":0.65400000000000003},"min":{"1min":0.57199999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.2330000000000001,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.77100000000000002,"5min":0.67900000000000005,"15min":0.66400000000000003},"min":{"1min":0.61499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.95399999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.71099999999999997,"15min":0.68799999999999994},"min":{"1min":0.66100000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.371,"5min":1.371,"15min":1.371},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.72299999999999998,"15min":0.70199999999999996},"min":{"1min":0.68999999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.97699999999999998}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.0009999999999999}]}]},{"osd":7,"up_from":51,"seq":219043332143,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28568,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716584,"statfs":{"total":95995035648,"available":95965782016,"internally_reserved":0,"allocated":1581056,"data_stored":1056288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.47899999999999998,"15min":0.46700000000000003},"min":{"1min":0.32400000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.997,"15min":0.997},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.51700000000000002,"15min":0.501},"min":{"1min":0.38,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.749,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.505}]},{"osd":2,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59599999999999997,"15min":0.60399999999999998},"min":{"1min":0.42099999999999999,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88,"5min":1.01,"15min":1.01},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.65100000000000002,"15min":0.64700000000000002},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.71199999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64200000000000002,"15min":0.63},"min":{"1min":0.49099999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.89700000000000002,"5min":1.111,"15min":1.111},"last":0.622},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.45300000000000001,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.68600000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.627,"15min":0.627},"min":{"1min":0.41399999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88200000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.54500000000000004,"15min":0.53700000000000003},"min":{"1min":0.36199999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.79300000000000004,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.65900000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65800000000000003,"15min":0.65600000000000003},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.872,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.76000000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.71199999999999997,"15min":0.70299999999999996},"min":{"1min":0.48899999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89500000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.81499999999999995}]},{"osd":6,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.73699999999999999,"15min":0.72499999999999998},"min":{"1min":0.628,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.91800000000000004,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.69099999999999995,"15min":0.68500000000000005},"min":{"1min":0.47499999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.79900000000000004}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96599999999999997,"5min":1.016,"15min":1.016},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.73199999999999998,"15min":0.70399999999999996},"min":{"1min":0.60499999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.82899999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.77300000000000002,"15min":0.77500000000000002},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.004,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.873},{"interface":"front","average":{"1min":0.78300000000000003,"5min":0.79800000000000004,"15min":0.80000000000000004},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.97499999999999998,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.77300000000000002}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.82799999999999996,"15min":0.82399999999999995},"min":{"1min":0.65900000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.96899999999999997,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.83499999999999996,"15min":0.83199999999999996},"min":{"1min":0.66700000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.85799999999999998}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.90700000000000003}]}]},{"osd":6,"up_from":46,"seq":197568495670,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463109,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.28799999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.42199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55300000000000005,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.875,"15min":0.875},"last":0.314}]},{"osd":3,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.378},{"interface":"front","average":{"1min":0.441,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.32700000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.443,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.57299999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.52000000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:49:44 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63900000000000001,"15min":0.623},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.92500000000000004,"5min":0.999,"15min":0.999},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.64800000000000002,"15min":0.63400000000000001},"min":{"1min":0.47699999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.85499999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.55400000000000005}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84799999999999998,"5min":0.876,"15min":0.876},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.81200000000000006,"5min":0.995,"15min":0.995},"last":0.59999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.68000000000000005,"15min":0.69599999999999995},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.67200000000000004,"15min":0.68700000000000006},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.90400000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.434}]},{"osd":10,"last update":"Fri Feb 9 22:49:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73299999999999998,"15min":0.73799999999999999},"min":{"1min":0.56100000000000005,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.86799999999999999,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.76700000000000002,"15min":0.77300000000000002},"min":{"1min":0.51900000000000002,"5min":0.499,"15min":0.499},"max":{"1min":0.94099999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59099999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574926,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1032192,"data_stored":523416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57699999999999996,"15min":0.52700000000000002},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.76300000000000001,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.80600000000000005,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.73499999999999999,"15min":0.58899999999999997},"min":{"1min":0.47899999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.94799999999999995,"5min":18.18,"15min":18.18},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.56899999999999995,"15min":0.50800000000000001},"min":{"1min":0.46800000000000003,"5min":0.372,"15min":0.372},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.71099999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.94099999999999995,"5min":1.206,"15min":1.206},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.59199999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.48099999999999998,"15min":0.45100000000000001},"min":{"1min":0.36699999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.46899999999999997,"15min":0.42499999999999999},"min":{"1min":0.35999999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.46899999999999997}]},{"osd":5,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79100000000000004,"5min":0.66800000000000004,"15min":0.60599999999999998},"min":{"1min":0.47199999999999998,"5min":0.44,"15min":0.44},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.91200000000000003},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.67800000000000005,"15min":0.621},"min":{"1min":0.59199999999999997,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.75900000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66600000000000004,"15min":0.64500000000000002},"min":{"1min":0.48999999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.53800000000000003}]},{"osd":7,"last update":"Fri Feb 9 22:49:42 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63900000000000001,"15min":0.621},"min":{"1min":0.439,"5min":0.377,"15min":0.377},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.73599999999999999,"5min":0.68400000000000005,"15min":0.66500000000000004},"min":{"1min":0.435,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.87}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.353,"15min":1.581},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1439999999999999,"5min":18.132000000000001,"15min":18.132000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69999999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.81200000000000006,"15min":0.81000000000000005},"min":{"1min":0.60499999999999998,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.85599999999999998,"15min":0.85399999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.78000000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.78200000000000003,"15min":0.78000000000000003},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.002,"5min":1.002,"15min":1.002},"last":0.746},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.85699999999999998,"15min":0.84899999999999998},"min":{"1min":0.63600000000000001,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.84599999999999997}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.95299999999999996},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.80000000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738450,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":464448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:49:15 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.55400000000000005,"15min":0.48999999999999999},"min":{"1min":0.42599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.88700000000000001,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.378},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52000000000000002,"15min":0.47999999999999998},"min":{"1min":0.34999999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.745,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.40600000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:49:41 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.54600000000000004,"15min":0.52000000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.84199999999999997,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.503},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.55300000000000005,"15min":0.50900000000000001},"min":{"1min":0.378,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.78200000000000003,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.61599999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.45300000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:49:31 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.58899999999999997,"15min":0.58099999999999996},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.745,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.60999999999999999,"15min":0.59299999999999997},"min":{"1min":0.376,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.85899999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.53000000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.60599999999999998,"15min":0.58399999999999996},"min":{"1min":0.439,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.81399999999999995,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.56599999999999995,"15min":0.57299999999999995},"min":{"1min":0.308,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84999999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.65200000000000002}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.48999999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:49:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.70499999999999996,"15min":0.73199999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.93999999999999995,"5min":1.081,"15min":1.081},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.72999999999999998,"15min":0.75800000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89700000000000002,"5min":1.202,"15min":1.202},"last":0.71299999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:49:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.69499999999999995,"15min":0.71099999999999997},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.78200000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.72999999999999998,"15min":0.74199999999999999},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.91600000000000004,"5min":1.137,"15min":1.137},"last":0.66500000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68600000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":524027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:46 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47899999999999998,"15min":0.45600000000000002},"min":{"1min":0.40899999999999997,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.51600000000000001,"15min":0.47999999999999998},"min":{"1min":0.38,"5min":0.224,"15min":0.224},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.53600000000000003}]},{"osd":1,"last update":"Fri Feb 9 22:49:46 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.52600000000000002,"15min":0.47799999999999998},"min":{"1min":0.377,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.55200000000000005,"15min":0.48599999999999999},"min":{"1min":0.33700000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.85299999999999998,"5min":0.874,"15min":0.874},"last":0.81499999999999995}]},{"osd":3,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55900000000000005,"15min":0.49099999999999999},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.48999999999999999},"min":{"1min":0.39900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.57499999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.80300000000000005}]},{"osd":5,"last update":"Fri Feb 9 22:49:51 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.45800000000000002,"15min":0.42199999999999999},"min":{"1min":0.34399999999999997,"5min":0.251,"15min":0.251},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.45700000000000002,"15min":0.442},"min":{"1min":0.31,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.66600000000000004}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.874,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90700000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.51900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.625}]},{"osd":9,"last update":"Fri Feb 9 22:49:26 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.64600000000000002,"15min":0.64800000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.63800000000000001,"15min":0.63700000000000001},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.85199999999999998}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.82799999999999996}]},{"osd":11,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.77900000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182468,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":987136,"data_stored":462934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57899999999999996,"5min":0.62,"15min":0.62},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.64600000000000002}]},{"osd":1,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.53100000000000003},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78100000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.504},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.53500000000000003},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.753,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.439}]},{"osd":2,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.621}]},{"osd":4,"last update":"Fri Feb 9 22:49:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.61099999999999999,"15min":0.57999999999999996},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.84299999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.66500000000000004,"15min":0.60499999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":5.9139999999999997,"5min":5.9139999999999997,"15min":5.9139999999999997},"last":0.45500000000000002}]},{"osd":5,"last update":"Fri Feb 9 22:49:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.443,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.91900000000000004,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60699999999999998,"15min":0.61499999999999999},"min":{"1min":0.435,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.88300000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.75900000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.747,"5min":0.873,"15min":0.873},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.82799999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.48099999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:49:40 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.65400000000000003,"15min":0.65600000000000003},"min":{"1min":0.433,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.81499999999999995,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.873,"5min":0.75,"15min":0.70799999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":5.8789999999999996,"5min":5.8789999999999996,"15min":5.8789999999999996},"last":0.58899999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.754,"15min":0.77800000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.95099999999999996,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.78700000000000003,"15min":0.80600000000000005},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.0229999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.57399999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.81999999999999995,"15min":0.83799999999999997},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.0640000000000001,"5min":1.26,"15min":1.26},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.73799999999999999,"15min":0.747},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.93999999999999995,"5min":0.999,"15min":0.999},"last":0.94199999999999995}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.64100000000000001}]}]},{"osd":4,"up_from":33,"seq":141733920831,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":462970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.41899999999999998,"15min":0.39800000000000002},"min":{"1min":0.28699999999999998,"5min":0.255,"15min":0.255},"max":{"1min":0.57499999999999996,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.433,"15min":0.39400000000000002},"min":{"1min":0.308,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.61599999999999999,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.50600000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.46999999999999997,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85099999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.56899999999999995},"min":{"1min":0.379,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.80000000000000004,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.871}]},{"osd":3,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.60199999999999998,"15min":0.57399999999999995},"min":{"1min":0.41399999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.874,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.434},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56299999999999994,"15min":0.56100000000000005},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.60899999999999999,"15min":0.57699999999999996},"min":{"1min":0.51300000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.69599999999999995,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.64700000000000002,"15min":0.59399999999999997},"min":{"1min":0.46200000000000002,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.92100000000000004,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.55800000000000005}]},{"osd":6,"last update":"Fri Feb 9 22:49:19 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.60899999999999999,"15min":0.59699999999999998},"min":{"1min":0.41099999999999998,"5min":0.38,"15min":0.38},"max":{"1min":0.79900000000000004,"5min":1.056,"15min":1.056},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.64600000000000002,"15min":0.63},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.91100000000000003,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.73499999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.64900000000000002,"5min":0.65500000000000003,"15min":0.65000000000000002},"min":{"1min":0.37,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88100000000000001,"5min":1.097,"15min":1.097},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.63600000000000001,"15min":0.627},"min":{"1min":0.44400000000000001,"5min":0.434,"15min":0.434},"max":{"1min":0.91800000000000004,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.621}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93300000000000005,"5min":1.129,"15min":1.129},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76900000000000002,"15min":0.748},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.264,"15min":1.264},"last":0.63500000000000001}]},{"osd":9,"last update":"Fri Feb 9 22:49:29 2024","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.80200000000000005,"15min":0.81100000000000005},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.96799999999999997,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.76900000000000002,"15min":0.77900000000000003},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.95399999999999996,"5min":1.196,"15min":1.196},"last":0.81899999999999995}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.83899999999999997,"15min":0.84099999999999997},"min":{"1min":0.53000000000000003,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.026,"5min":1.179,"15min":1.179},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.76500000000000001,"15min":0.76900000000000002},"min":{"1min":0.56999999999999995,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.95999999999999996,"5min":1.238,"15min":1.238},"last":0.70499999999999996}]},{"osd":11,"last update":"Fri Feb 9 22:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.77200000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757307,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32740,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712412,"statfs":{"total":95995035648,"available":95961509888,"internally_reserved":0,"allocated":1593344,"data_stored":1055966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.52300000000000002,"15min":0.49299999999999999},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.70899999999999996,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.498},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.50600000000000001,"15min":0.47599999999999998},"min":{"1min":0.41499999999999998,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.47699999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.52900000000000003,"15min":0.503},"min":{"1min":0.379,"5min":0.31,"15min":0.31},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.53900000000000003,"15min":0.51400000000000001},"min":{"1min":0.33100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.753,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.628}]},{"osd":2,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.39600000000000002},"min":{"1min":0.27400000000000002,"5min":0.251,"15min":0.251},"max":{"1min":0.60799999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.41799999999999998,"15min":0.39800000000000002},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":0.70799999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.59399999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.52700000000000002,"15min":0.50700000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71499999999999997,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.55200000000000005,"15min":0.52800000000000002},"min":{"1min":0.35499999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.64000000000000001}]},{"osd":4,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.503,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78600000000000003,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.70499999999999996},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.56699999999999995,"15min":0.53900000000000003},"min":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.61299999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60899999999999999,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85299999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60599999999999998,"15min":0.58299999999999996},"min":{"1min":0.47499999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.81299999999999994,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74099999999999999}]},{"osd":7,"last update":"Fri Feb 9 22:49:39 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.64900000000000002,"15min":0.64500000000000002},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.86599999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.64800000000000002,"15min":0.65100000000000002},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.93600000000000005,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.80100000000000005}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.872,"5min":0.996,"15min":0.996},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.82699999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:49:27 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.71099999999999997,"15min":0.71199999999999997},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.71099999999999997,"15min":0.71599999999999997},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.83899999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73699999999999999,"15min":0.73399999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.91700000000000004,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.72799999999999998,"15min":0.72199999999999998},"min":{"1min":0.499,"5min":0.499,"15min":0.499},"max":{"1min":0.96299999999999997,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.86299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61757,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61130,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:50:24.000 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-02-09T22:50:24.000 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-02-09T22:50:24.000 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-02-09T22:50:24.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph health --format=json 2024-02-09T22:50:24.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:23 smithi064 ceph-mon[36638]: from='client.14976 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:50:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:25.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:25 smithi003 ceph-mon[32678]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-02-09T22:50:25.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:25 smithi064 ceph-mon[36638]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-02-09T22:50:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:25 smithi089 ceph-mon[36665]: pgmap v327: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 9 op/s 2024-02-09T22:50:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.916+0000 7f088f2c9700 1 -- 172.21.15.3:0/1577307617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f088806a420 msgr2=0x7f088806a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.916+0000 7f088f2c9700 1 --2- 172.21.15.3:0/1577307617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f088806a420 0x7f088806a880 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f08840099c0 tx=0x7f0884009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 -- 172.21.15.3:0/1577307617 shutdown_connections 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 --2- 172.21.15.3:0/1577307617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0888107da0 0x7f0888108180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 --2- 172.21.15.3:0/1577307617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 0x7f088810ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 --2- 172.21.15.3:0/1577307617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f088806a420 0x7f088806a880 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 -- 172.21.15.3:0/1577307617 >> 172.21.15.3:0/1577307617 conn(0x7f0888076c40 msgr2=0x7f0888077050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 -- 172.21.15.3:0/1577307617 shutdown_connections 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 -- 172.21.15.3:0/1577307617 wait complete. 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 Processor -- start 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.917+0000 7f088f2c9700 1 -- start start 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f088806a420 0x7f0888198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 0x7f0888198790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 0x7f088819cb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088819d130 con 0x7f0888107da0 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088819d2a0 con 0x7f088806adc0 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f088819d460 con 0x7f088806a420 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 0x7f088819cb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d065700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f088806a420 0x7f0888198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 0x7f088819cb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55210/0 (socket says 172.21.15.3:55210) 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 -- 172.21.15.3:0/3410014731 learned_addr learned my addr 172.21.15.3:0/3410014731 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088c864700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 0x7f0888198790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 -- 172.21.15.3:0/3410014731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f088806a420 msgr2=0x7f0888198250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f088806a420 0x7f0888198250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 -- 172.21.15.3:0/3410014731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 msgr2=0x7f0888198790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 0x7f0888198790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 -- 172.21.15.3:0/3410014731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08840095f0 con 0x7f0888107da0 2024-02-09T22:50:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d065700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f088806a420 0x7f0888198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088c864700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 0x7f0888198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088d866700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 0x7f088819cb20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f088000ada0 tx=0x7f088000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088000f070 con 0x7f0888107da0 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f088819d740 con 0x7f0888107da0 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0880007b30 con 0x7f0888107da0 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.918+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08800053c0 con 0x7f0888107da0 2024-02-09T22:50:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.919+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08881a4fa0 con 0x7f0888107da0 2024-02-09T22:50:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.920+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f088810a2c0 con 0x7f0888107da0 2024-02-09T22:50:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.921+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f088000c980 con 0x7f0888107da0 2024-02-09T22:50:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.921+0000 7f087e7fc700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08740a21e0 0x7f08740a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.921+0000 7f088d065700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08740a21e0 0x7f08740a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.921+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f08800cd430 con 0x7f0888107da0 2024-02-09T22:50:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.922+0000 7f088d065700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08740a21e0 0x7f08740a46a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0878009fd0 tx=0x7f0878009260 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:25.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:25.926+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0880096340 con 0x7f0888107da0 2024-02-09T22:50:26.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.174+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f08881995e0 con 0x7f0888107da0 2024-02-09T22:50:26.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.175+0000 7f087e7fc700 1 -- 172.21.15.3:0/3410014731 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f0880099a00 con 0x7f0888107da0 2024-02-09T22:50:26.178 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:26.178 INFO:teuthology.orchestra.run.smithi003.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08740a21e0 msgr2=0x7f08740a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08740a21e0 0x7f08740a46a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f0878009fd0 tx=0x7f0878009260 comp rx=0 tx=0).stop 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 msgr2=0x7f088819cb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 0x7f088819cb20 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f088000ada0 tx=0x7f088000c330 comp rx=0 tx=0).stop 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 shutdown_connections 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f08740a21e0 0x7f08740a46a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f088806a420 0x7f0888198250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0888107da0 0x7f088819cb20 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 --2- 172.21.15.3:0/3410014731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f088806adc0 0x7f0888198790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 >> 172.21.15.3:0/3410014731 conn(0x7f0888076c40 msgr2=0x7f08880ffa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:26.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 shutdown_connections 2024-02-09T22:50:26.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.177+0000 7f088f2c9700 1 -- 172.21.15.3:0/3410014731 wait complete. 2024-02-09T22:50:26.497 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:26 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3410014731' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T22:50:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:26 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3410014731' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T22:50:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:26 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3410014731' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-02-09T22:50:26.807 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-02-09T22:50:26.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:50:26.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 -- 172.21.15.3:0/4049333594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344101d60 msgr2=0x7f9344104150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:26.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 --2- 172.21.15.3:0/4049333594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344101d60 0x7f9344104150 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f9340009900 tx=0x7f934002bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:26.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 -- 172.21.15.3:0/4049333594 shutdown_connections 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 --2- 172.21.15.3:0/4049333594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9344106fc0 0x7f93441093b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 --2- 172.21.15.3:0/4049333594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9344104690 0x7f9344106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 --2- 172.21.15.3:0/4049333594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344101d60 0x7f9344104150 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 -- 172.21.15.3:0/4049333594 >> 172.21.15.3:0/4049333594 conn(0x7f93440fb6d0 msgr2=0x7f93440fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 -- 172.21.15.3:0/4049333594 shutdown_connections 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.910+0000 7f934c52c640 1 -- 172.21.15.3:0/4049333594 wait complete. 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 Processor -- start 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 -- start start 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9344101d60 0x7f93441972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9344104690 0x7f93441977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 0x7f934419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:26.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934419ed90 con 0x7f9344106fc0 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934419ef00 con 0x7f9344101d60 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f934419f0c0 con 0x7f9344104690 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 0x7f934419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 0x7f934419e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38756/0 (socket says 172.21.15.3:38756) 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 -- 172.21.15.3:0/1412793683 learned_addr learned my addr 172.21.15.3:0/1412793683 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934a2c8640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9344101d60 0x7f93441972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f9349ac7640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9344104690 0x7f93441977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 -- 172.21.15.3:0/1412793683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9344104690 msgr2=0x7f93441977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9344104690 0x7f93441977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 -- 172.21.15.3:0/1412793683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9344101d60 msgr2=0x7f93441972a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9344101d60 0x7f93441972a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 -- 172.21.15.3:0/1412793683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9340009560 con 0x7f9344106fc0 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934aac9640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 0x7f934419e850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f933c00b970 tx=0x7f933c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:26.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c00cb40 con 0x7f9344106fc0 2024-02-09T22:50:26.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f933c00cca0 con 0x7f9344106fc0 2024-02-09T22:50:26.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f934419f3a0 con 0x7f9344106fc0 2024-02-09T22:50:26.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.911+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f933c0077d0 con 0x7f9344106fc0 2024-02-09T22:50:26.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.912+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f934419f930 con 0x7f9344106fc0 2024-02-09T22:50:26.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.912+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93440fd1c0 con 0x7f9344106fc0 2024-02-09T22:50:26.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.913+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f933c014440 con 0x7f9344106fc0 2024-02-09T22:50:26.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.914+0000 7f933b7fe640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f93300a21d0 0x7f93300a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:26.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.914+0000 7f934a2c8640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f93300a21d0 0x7f93300a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:26.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.914+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f933c0cd9d0 con 0x7f9344106fc0 2024-02-09T22:50:26.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.915+0000 7f934a2c8640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f93300a21d0 0x7f93300a4690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9340006010 tx=0x7f9340005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:26.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:26.918+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f933c096a10 con 0x7f9344106fc0 2024-02-09T22:50:27.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.102+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f934410ccd0 con 0x7f9344106fc0 2024-02-09T22:50:27.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.104+0000 7f933b7fe640 1 -- 172.21.15.3:0/1412793683 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f933c023090 con 0x7f9344106fc0 2024-02-09T22:50:27.107 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:27.107 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f93300a21d0 msgr2=0x7f93300a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f93300a21d0 0x7f93300a4690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f9340006010 tx=0x7f9340005d90 comp rx=0 tx=0).stop 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 msgr2=0x7f934419e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 0x7f934419e850 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f933c00b970 tx=0x7f933c00be30 comp rx=0 tx=0).stop 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 shutdown_connections 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f93300a21d0 0x7f93300a4690 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9344106fc0 0x7f934419e850 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9344104690 0x7f93441977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 --2- 172.21.15.3:0/1412793683 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9344101d60 0x7f93441972a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 >> 172.21.15.3:0/1412793683 conn(0x7f93440fb6d0 msgr2=0x7f93441026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:27.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 shutdown_connections 2024-02-09T22:50:27.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.106+0000 7f934c52c640 1 -- 172.21.15.3:0/1412793683 wait complete. 2024-02-09T22:50:27.121 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:50:27.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:50:27.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 -- 172.21.15.3:0/633605990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 msgr2=0x7f4968104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 --2- 172.21.15.3:0/633605990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968104160 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4958009900 tx=0x7f495802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 -- 172.21.15.3:0/633605990 shutdown_connections 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 --2- 172.21.15.3:0/633605990 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 0x7f4968106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 --2- 172.21.15.3:0/633605990 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 0x7f49681093c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 --2- 172.21.15.3:0/633605990 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968104160 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 -- 172.21.15.3:0/633605990 >> 172.21.15.3:0/633605990 conn(0x7f49680fb6e0 msgr2=0x7f49680fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 -- 172.21.15.3:0/633605990 shutdown_connections 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.225+0000 7f496de2a640 1 -- 172.21.15.3:0/633605990 wait complete. 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 Processor -- start 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 -- start start 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968197300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 0x7f4968197840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 0x7f496819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496819edf0 con 0x7f4968101d70 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496819ef60 con 0x7f49681046a0 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f496de2a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f496819f120 con 0x7f4968106fd0 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968197300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968197300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38776/0 (socket says 172.21.15.3:38776) 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 -- 172.21.15.3:0/2514127517 learned_addr learned my addr 172.21.15.3:0/2514127517 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f4966ffd640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 0x7f4968197840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f4967fff640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 0x7f496819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 -- 172.21.15.3:0/2514127517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 msgr2=0x7f496819e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 0x7f496819e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 -- 172.21.15.3:0/2514127517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 msgr2=0x7f4968197840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 0x7f4968197840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f49677fe640 1 -- 172.21.15.3:0/2514127517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4958009560 con 0x7f4968101d70 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f4966ffd640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 0x7f4968197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.226+0000 7f4967fff640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 0x7f496819e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.227+0000 7f49677fe640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968197300 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4958006010 tx=0x7f49580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.227+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49580044e0 con 0x7f4968101d70 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.227+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f496819f3a0 con 0x7f4968101d70 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.227+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f496819f900 con 0x7f4968101d70 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.227+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49580054e0 con 0x7f4968101d70 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.227+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f495803e450 con 0x7f4968101d70 2024-02-09T22:50:27.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.228+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4954005290 con 0x7f4968101d70 2024-02-09T22:50:27.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.229+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f49580035f0 con 0x7f4968101d70 2024-02-09T22:50:27.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.230+0000 7f4964ff9640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f49500a2150 0x7f49500a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.230+0000 7f4966ffd640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f49500a2150 0x7f49500a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.230+0000 7f4966ffd640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f49500a2150 0x7f49500a4610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f495c0058c0 tx=0x7f495c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:27.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.230+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f495803ce30 con 0x7f4968101d70 2024-02-09T22:50:27.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.235+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f49580b9400 con 0x7f4968101d70 2024-02-09T22:50:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:27 smithi003 ceph-mon[32678]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:27.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:27 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1412793683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.420+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4954005ac0 con 0x7f4968101d70 2024-02-09T22:50:27.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.421+0000 7f4964ff9640 1 -- 172.21.15.3:0/2514127517 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49580bcac0 con 0x7f4968101d70 2024-02-09T22:50:27.424 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f49500a2150 msgr2=0x7f49500a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f49500a2150 0x7f49500a4610 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f495c0058c0 tx=0x7f495c005850 comp rx=0 tx=0).stop 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 msgr2=0x7f4968197300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968197300 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f4958006010 tx=0x7f49580042f0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 shutdown_connections 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f49500a2150 0x7f49500a4610 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f49681046a0 0x7f4968197840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4968106fd0 0x7f496819e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 --2- 172.21.15.3:0/2514127517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4968101d70 0x7f4968197300 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 >> 172.21.15.3:0/2514127517 conn(0x7f49680fb6e0 msgr2=0x7f4968102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:27.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 shutdown_connections 2024-02-09T22:50:27.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.423+0000 7f496de2a640 1 -- 172.21.15.3:0/2514127517 wait complete. 2024-02-09T22:50:27.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 -- 172.21.15.3:0/3121707606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20102430 msgr2=0x7f6c201028b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 --2- 172.21.15.3:0/3121707606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20102430 0x7f6c201028b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f6c14009900 tx=0x7f6c1402bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 -- 172.21.15.3:0/3121707606 shutdown_connections 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 --2- 172.21.15.3:0/3121707606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c20102df0 0x7f6c20109680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 --2- 172.21.15.3:0/3121707606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20102430 0x7f6c201028b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 --2- 172.21.15.3:0/3121707606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20101240 0x7f6c20101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 -- 172.21.15.3:0/3121707606 >> 172.21.15.3:0/3121707606 conn(0x7f6c200fc7b0 msgr2=0x7f6c200febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 -- 172.21.15.3:0/3121707606 shutdown_connections 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.542+0000 7f6c2678c640 1 -- 172.21.15.3:0/3121707606 wait complete. 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 Processor -- start 2024-02-09T22:50:27.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 -- start start 2024-02-09T22:50:27.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 0x7f6c20074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20102430 0x7f6c20073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c20102df0 0x7f6c20073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c200742c0 con 0x7f6c20101240 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c20074430 con 0x7f6c20102430 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c2678c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c201a1310 con 0x7f6c20102df0 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 0x7f6c20074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 0x7f6c20074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38788/0 (socket says 172.21.15.3:38788) 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 -- 172.21.15.3:0/2999469451 learned_addr learned my addr 172.21.15.3:0/2999469451 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1f7fe640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20102430 0x7f6c20073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c24d29640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c20102df0 0x7f6c20073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 -- 172.21.15.3:0/2999469451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c20102df0 msgr2=0x7f6c20073b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c20102df0 0x7f6c20073b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 -- 172.21.15.3:0/2999469451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20102430 msgr2=0x7f6c20073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20102430 0x7f6c20073620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.543+0000 7f6c1ffff640 1 -- 172.21.15.3:0/2999469451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c14009560 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c1f7fe640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20102430 0x7f6c20073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c1ffff640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 0x7f6c20074fe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000d7a0 tx=0x7f6c1000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10009c30 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c1000deb0 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c10010460 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c201a15c0 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.544+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c201a1b20 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.545+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c201085e0 con 0x7f6c20101240 2024-02-09T22:50:27.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.546+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f6c10003ef0 con 0x7f6c20101240 2024-02-09T22:50:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.546+0000 7f6c1d7fa640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6c080a21d0 0x7f6c080a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.547+0000 7f6c1f7fe640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6c080a21d0 0x7f6c080a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.547+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f6c100cd7a0 con 0x7f6c20101240 2024-02-09T22:50:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.547+0000 7f6c1f7fe640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6c080a21d0 0x7f6c080a4690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6c20074d50 tx=0x7f6c14005be0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:27.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.551+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6c10096620 con 0x7f6c20101240 2024-02-09T22:50:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:27 smithi064 ceph-mon[36638]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:27.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:27 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1412793683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:27.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:27 smithi089 ceph-mon[36665]: pgmap v328: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:27.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:27 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1412793683' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:27.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.731+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6c2002ce50 con 0x7f6c20101240 2024-02-09T22:50:27.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.731+0000 7f6c1d7fa640 1 -- 172.21.15.3:0/2999469451 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6c10099ce0 con 0x7f6c20101240 2024-02-09T22:50:27.735 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:50:27.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6c080a21d0 msgr2=0x7f6c080a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6c080a21d0 0x7f6c080a4690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f6c20074d50 tx=0x7f6c14005be0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 msgr2=0x7f6c20074fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 0x7f6c20074fe0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000d7a0 tx=0x7f6c1000dc60 comp rx=0 tx=0).stop 2024-02-09T22:50:27.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 shutdown_connections 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f6c080a21d0 0x7f6c080a4690 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c20102430 0x7f6c20073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c20102df0 0x7f6c20073b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 --2- 172.21.15.3:0/2999469451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c20101240 0x7f6c20074fe0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 >> 172.21.15.3:0/2999469451 conn(0x7f6c200fc7b0 msgr2=0x7f6c201057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 shutdown_connections 2024-02-09T22:50:27.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.733+0000 7f6c2678c640 1 -- 172.21.15.3:0/2999469451 wait complete. 2024-02-09T22:50:27.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:50:27.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 -- 172.21.15.3:0/1787674079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 msgr2=0x7f7c6c1028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/1787674079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1028a0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f7c60009900 tx=0x7f7c6002bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 -- 172.21.15.3:0/1787674079 shutdown_connections 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/1787674079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c6c102de0 0x7f7c6c109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/1787674079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1028a0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/1787674079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 0x7f7c6c101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 -- 172.21.15.3:0/1787674079 >> 172.21.15.3:0/1787674079 conn(0x7f7c6c0fc760 msgr2=0x7f7c6c0febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 -- 172.21.15.3:0/1787674079 shutdown_connections 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.850+0000 7f7c71c9d640 1 -- 172.21.15.3:0/1787674079 wait complete. 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 Processor -- start 2024-02-09T22:50:27.854 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 -- start start 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 0x7f7c6c197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1977d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c6c102de0 0x7f7c6c19e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c19ed80 con 0x7f7c6c102420 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c19eef0 con 0x7f7c6c101230 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c71c9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c6c19f0b0 con 0x7f7c6c102de0 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1977d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1977d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38802/0 (socket says 172.21.15.3:38802) 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 -- 172.21.15.3:0/288839955 learned_addr learned my addr 172.21.15.3:0/288839955 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6bfff640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c6c102de0 0x7f7c6c19e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6b7fe640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 0x7f7c6c197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 -- 172.21.15.3:0/288839955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c6c102de0 msgr2=0x7f7c6c19e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c6c102de0 0x7f7c6c19e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 -- 172.21.15.3:0/288839955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 msgr2=0x7f7c6c197290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 0x7f7c6c197290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6affd640 1 -- 172.21.15.3:0/288839955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c60009560 con 0x7f7c6c102420 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.851+0000 7f7c6b7fe640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 0x7f7c6c197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.852+0000 7f7c6affd640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1977d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c60006010 tx=0x7f7c600042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.852+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c600044e0 con 0x7f7c6c102420 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.852+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c600054e0 con 0x7f7c6c102420 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.852+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c6c19f330 con 0x7f7c6c102420 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.852+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6003e450 con 0x7f7c6c102420 2024-02-09T22:50:27.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.852+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c6c19f890 con 0x7f7c6c102420 2024-02-09T22:50:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.853+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c6c108550 con 0x7f7c6c102420 2024-02-09T22:50:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.854+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f7c600035f0 con 0x7f7c6c102420 2024-02-09T22:50:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.854+0000 7f7c68ff9640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7c540a21d0 0x7f7c540a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.855+0000 7f7c6b7fe640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7c540a21d0 0x7f7c540a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:27.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.855+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7c600effc0 con 0x7f7c6c102420 2024-02-09T22:50:27.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.855+0000 7f7c6b7fe640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7c540a21d0 0x7f7c540a4690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0058c0 tx=0x7f7c5c005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:27.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:27.859+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7c600b8ff0 con 0x7f7c6c102420 2024-02-09T22:50:28.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.038+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c6c10ce90 con 0x7f7c6c102420 2024-02-09T22:50:28.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.039+0000 7f7c68ff9640 1 -- 172.21.15.3:0/288839955 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c600bc6b0 con 0x7f7c6c102420 2024-02-09T22:50:28.042 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:50:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.040+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7c540a21d0 msgr2=0x7f7c540a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.040+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7c540a21d0 0x7f7c540a4690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f7c5c0058c0 tx=0x7f7c5c005850 comp rx=0 tx=0).stop 2024-02-09T22:50:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.040+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 msgr2=0x7f7c6c1977d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.040+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1977d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7c60006010 tx=0x7f7c600042f0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 shutdown_connections 2024-02-09T22:50:28.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7c540a21d0 0x7f7c540a4690 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c6c102de0 0x7f7c6c19e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c6c102420 0x7f7c6c1977d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 --2- 172.21.15.3:0/288839955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c6c101230 0x7f7c6c197290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 >> 172.21.15.3:0/288839955 conn(0x7f7c6c0fc760 msgr2=0x7f7c6c105720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:28.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 shutdown_connections 2024-02-09T22:50:28.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.041+0000 7f7c71c9d640 1 -- 172.21.15.3:0/288839955 wait complete. 2024-02-09T22:50:28.055 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:50:28.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 -- 172.21.15.3:0/3644969680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 msgr2=0x7fb604109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/3644969680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604109390 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc009ca0 tx=0x7fb5fc02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 -- 172.21.15.3:0/3644969680 shutdown_connections 2024-02-09T22:50:28.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/3644969680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb604104670 0x7fb604106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/3644969680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604109390 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/3644969680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb604101d40 0x7fb604104130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 -- 172.21.15.3:0/3644969680 >> 172.21.15.3:0/3644969680 conn(0x7fb6040fb6d0 msgr2=0x7fb6040fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:28.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 -- 172.21.15.3:0/3644969680 shutdown_connections 2024-02-09T22:50:28.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.154+0000 7fb60bb0f640 1 -- 172.21.15.3:0/3644969680 wait complete. 2024-02-09T22:50:28.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 Processor -- start 2024-02-09T22:50:28.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 -- start start 2024-02-09T22:50:28.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb604101d40 0x7fb6041016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb604104670 0x7fb6040ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604100990 con 0x7fb604106fa0 2024-02-09T22:50:28.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb604100b00 con 0x7fb604101d40 2024-02-09T22:50:28.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60bb0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6041a1310 con 0x7fb604104670 2024-02-09T22:50:28.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60a0ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60a0ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604100230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38820/0 (socket says 172.21.15.3:38820) 2024-02-09T22:50:28.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb60a0ac640 1 -- 172.21.15.3:0/388074341 learned_addr learned my addr 172.21.15.3:0/388074341 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:28.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb6090aa640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb604104670 0x7fb6040ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.155+0000 7fb6098ab640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb604101d40 0x7fb6041016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60a0ac640 1 -- 172.21.15.3:0/388074341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb604104670 msgr2=0x7fb6040ffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60a0ac640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb604104670 0x7fb6040ffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60a0ac640 1 -- 172.21.15.3:0/388074341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb604101d40 msgr2=0x7fb6041016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60a0ac640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb604101d40 0x7fb6041016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60a0ac640 1 -- 172.21.15.3:0/388074341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5fc009900 con 0x7fb604106fa0 2024-02-09T22:50:28.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60a0ac640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604100230 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc033000 tx=0x7fb5fc034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:28.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc0329b0 con 0x7fb604106fa0 2024-02-09T22:50:28.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5fc003e50 con 0x7fb604106fa0 2024-02-09T22:50:28.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6041a1560 con 0x7fb604106fa0 2024-02-09T22:50:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5fc040450 con 0x7fb604106fa0 2024-02-09T22:50:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.156+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6041a1a10 con 0x7fb604106fa0 2024-02-09T22:50:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.157+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6040fd1c0 con 0x7fb604106fa0 2024-02-09T22:50:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.159+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fb5fc00a040 con 0x7fb604106fa0 2024-02-09T22:50:28.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.159+0000 7fb5faffd640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5f00a2150 0x7fb5f00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.159+0000 7fb6098ab640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5f00a2150 0x7fb5f00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.159+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb5fc0f0fd0 con 0x7fb604106fa0 2024-02-09T22:50:28.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.160+0000 7fb6098ab640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5f00a2150 0x7fb5f00a4610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb600005fd0 tx=0x7fb600005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:28.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.163+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb5fc0b9f60 con 0x7fb604106fa0 2024-02-09T22:50:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:28 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2514127517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:28 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2999469451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:28 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/288839955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.382+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb60410cc70 con 0x7fb604106fa0 2024-02-09T22:50:28.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.383+0000 7fb5faffd640 1 -- 172.21.15.3:0/388074341 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11361 (secure 0 0 0) 0x7fb5fc0bd620 con 0x7fb604106fa0 2024-02-09T22:50:28.386 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:28.387 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:49:13.115988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:50:28.387 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:50:28.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5f00a2150 msgr2=0x7fb5f00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5f00a2150 0x7fb5f00a4610 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb600005fd0 tx=0x7fb600005850 comp rx=0 tx=0).stop 2024-02-09T22:50:28.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 msgr2=0x7fb604100230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604100230 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb5fc033000 tx=0x7fb5fc034a90 comp rx=0 tx=0).stop 2024-02-09T22:50:28.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 shutdown_connections 2024-02-09T22:50:28.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb5f00a2150 0x7fb5f00a4610 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb604106fa0 0x7fb604100230 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb604104670 0x7fb6040ffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 --2- 172.21.15.3:0/388074341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb604101d40 0x7fb6041016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 >> 172.21.15.3:0/388074341 conn(0x7fb6040fb6d0 msgr2=0x7fb604102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:28.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 shutdown_connections 2024-02-09T22:50:28.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.385+0000 7fb60bb0f640 1 -- 172.21.15.3:0/388074341 wait complete. 2024-02-09T22:50:28.400 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:50:28.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.499+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/3927072443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 msgr2=0x7ff7d41040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.499+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/3927072443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d41040d0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc009900 tx=0x7ff7cc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:28.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/3927072443 shutdown_connections 2024-02-09T22:50:28.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/3927072443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7d4104610 0x7ff7d4106a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/3927072443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7d4106f40 0x7ff7d4109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/3927072443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d41040d0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/3927072443 >> 172.21.15.3:0/3927072443 conn(0x7ff7d40fb6d0 msgr2=0x7ff7d40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/3927072443 shutdown_connections 2024-02-09T22:50:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/3927072443 wait complete. 2024-02-09T22:50:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 Processor -- start 2024-02-09T22:50:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- start start 2024-02-09T22:50:28.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d4197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7d4104610 0x7ff7d4197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7d4106f40 0x7ff7d419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d419ed30 con 0x7ff7d4101ce0 2024-02-09T22:50:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d419eea0 con 0x7ff7d4106f40 2024-02-09T22:50:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.500+0000 7ff7d8a4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7d419f060 con 0x7ff7d4104610 2024-02-09T22:50:28.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d4197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d4197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38828/0 (socket says 172.21.15.3:38828) 2024-02-09T22:50:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 -- 172.21.15.3:0/1141191458 learned_addr learned my addr 172.21.15.3:0/1141191458 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d1d9b640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7d4104610 0x7ff7d4197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 -- 172.21.15.3:0/1141191458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7d4104610 msgr2=0x7ff7d4197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7d4104610 0x7ff7d4197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 -- 172.21.15.3:0/1141191458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7d4106f40 msgr2=0x7ff7d419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7d4106f40 0x7ff7d419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 -- 172.21.15.3:0/1141191458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7cc009560 con 0x7ff7d4101ce0 2024-02-09T22:50:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d259c640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d4197240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc006010 tx=0x7ff7cc0040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0044c0 con 0x7ff7d4101ce0 2024-02-09T22:50:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d419f2e0 con 0x7ff7d4101ce0 2024-02-09T22:50:28.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7cc0052d0 con 0x7ff7d4101ce0 2024-02-09T22:50:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc003e10 con 0x7ff7d4101ce0 2024-02-09T22:50:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.501+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d419f7f0 con 0x7ff7d4101ce0 2024-02-09T22:50:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.502+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7d40fd1c0 con 0x7ff7d4101ce0 2024-02-09T22:50:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.503+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7ff7cc005440 con 0x7ff7d4101ce0 2024-02-09T22:50:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.504+0000 7ff7c37fe640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff7bc0a2150 0x7ff7bc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.504+0000 7ff7d1d9b640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff7bc0a2150 0x7ff7bc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.504+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff7cc0f0770 con 0x7ff7d4101ce0 2024-02-09T22:50:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.504+0000 7ff7d1d9b640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff7bc0a2150 0x7ff7bc0a4610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8005e10 tx=0x7ff7c8005da0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:28.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.507+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff7cc0b9700 con 0x7ff7d4101ce0 2024-02-09T22:50:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:28 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2514127517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:28 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2999469451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:28 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/288839955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.690+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7d410cc70 con 0x7ff7d4101ce0 2024-02-09T22:50:28.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.691+0000 7ff7c37fe640 1 -- 172.21.15.3:0/1141191458 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7ff7cc0bcdc0 con 0x7ff7d4101ce0 2024-02-09T22:50:28.695 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:28.695 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff7bc0a2150 msgr2=0x7ff7bc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff7bc0a2150 0x7ff7bc0a4610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8005e10 tx=0x7ff7c8005da0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 msgr2=0x7ff7d4197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d4197240 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc006010 tx=0x7ff7cc0040c0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 shutdown_connections 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff7bc0a2150 0x7ff7bc0a4610 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff7d4106f40 0x7ff7d419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff7d4104610 0x7ff7d4197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 --2- 172.21.15.3:0/1141191458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff7d4101ce0 0x7ff7d4197240 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 >> 172.21.15.3:0/1141191458 conn(0x7ff7d40fb6d0 msgr2=0x7ff7d41025a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 shutdown_connections 2024-02-09T22:50:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.694+0000 7ff7d8a4e640 1 -- 172.21.15.3:0/1141191458 wait complete. 2024-02-09T22:50:28.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:50:28.709 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:50:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:28 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2514127517' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:28 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2999469451' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:28 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/288839955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 -- 172.21.15.3:0/1232812653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34100150 msgr2=0x7fec34100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 --2- 172.21.15.3:0/1232812653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34100150 0x7fec34100570 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec24009ca0 tx=0x7fec2402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 -- 172.21.15.3:0/1232812653 shutdown_connections 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 --2- 172.21.15.3:0/1232812653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec34101340 0x7fec341017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 --2- 172.21.15.3:0/1232812653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 0x7fec34108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 --2- 172.21.15.3:0/1232812653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34100150 0x7fec34100570 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 -- 172.21.15.3:0/1232812653 >> 172.21.15.3:0/1232812653 conn(0x7fec340fb6c0 msgr2=0x7fec340fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 -- 172.21.15.3:0/1232812653 shutdown_connections 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.811+0000 7fec3a165640 1 -- 172.21.15.3:0/1232812653 wait complete. 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 Processor -- start 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 -- start start 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec34100150 0x7fec341974d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 0x7fec34197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 0x7fec3419e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419ef20 con 0x7fec34101340 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419f090 con 0x7fec34100150 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec3a165640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec3419f250 con 0x7fec34101d00 2024-02-09T22:50:28.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec32ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 0x7fec34197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec32ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 0x7fec34197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38848/0 (socket says 172.21.15.3:38848) 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec32ffd640 1 -- 172.21.15.3:0/1308596504 learned_addr learned my addr 172.21.15.3:0/1308596504 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.812+0000 7fec337fe640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec34100150 0x7fec341974d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec33fff640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 0x7fec3419e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec32ffd640 1 -- 172.21.15.3:0/1308596504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 msgr2=0x7fec3419e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec32ffd640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 0x7fec3419e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec32ffd640 1 -- 172.21.15.3:0/1308596504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec34100150 msgr2=0x7fec341974d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec32ffd640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec34100150 0x7fec341974d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec32ffd640 1 -- 172.21.15.3:0/1308596504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec24009900 con 0x7fec34101340 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec33fff640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 0x7fec3419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec32ffd640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 0x7fec34197a10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00b4a0 tx=0x7fec1c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c010610 con 0x7fec34101340 2024-02-09T22:50:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec3419f530 con 0x7fec34101340 2024-02-09T22:50:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec1c010c50 con 0x7fec34101340 2024-02-09T22:50:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c017e20 con 0x7fec34101340 2024-02-09T22:50:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.813+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3419fac0 con 0x7fec34101340 2024-02-09T22:50:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.814+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3410bc20 con 0x7fec34101340 2024-02-09T22:50:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.816+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fec1c017480 con 0x7fec34101340 2024-02-09T22:50:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.816+0000 7fec30ff9640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec200a2150 0x7fec200a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.816+0000 7fec337fe640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec200a2150 0x7fec200a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.816+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fec1c0cd740 con 0x7fec34101340 2024-02-09T22:50:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.817+0000 7fec337fe640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec200a2150 0x7fec200a4610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec2402c700 tx=0x7fec240326b0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:28.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:28.819+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec1c0966d0 con 0x7fec34101340 2024-02-09T22:50:29.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.001+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fec3419ff90 con 0x7fec34101340 2024-02-09T22:50:29.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.001+0000 7fec30ff9640 1 -- 172.21.15.3:0/1308596504 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fec1c0092b0 con 0x7fec34101340 2024-02-09T22:50:29.005 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:50:29.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec200a2150 msgr2=0x7fec200a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec200a2150 0x7fec200a4610 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fec2402c700 tx=0x7fec240326b0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 msgr2=0x7fec34197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 0x7fec34197a10 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00b4a0 tx=0x7fec1c00b960 comp rx=0 tx=0).stop 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 shutdown_connections 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec200a2150 0x7fec200a4610 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec34101d00 0x7fec3419e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec34101340 0x7fec34197a10 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 --2- 172.21.15.3:0/1308596504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec34100150 0x7fec341974d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.003+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 >> 172.21.15.3:0/1308596504 conn(0x7fec340fb6c0 msgr2=0x7fec340fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.004+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 shutdown_connections 2024-02-09T22:50:29.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.004+0000 7fec3a165640 1 -- 172.21.15.3:0/1308596504 wait complete. 2024-02-09T22:50:29.018 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:50:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 -- 172.21.15.3:0/175985562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 msgr2=0x7f62fc100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 --2- 172.21.15.3:0/175985562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc100570 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f62ec009900 tx=0x7f62ec02bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 -- 172.21.15.3:0/175985562 shutdown_connections 2024-02-09T22:50:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 --2- 172.21.15.3:0/175985562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101d00 0x7f62fc108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 --2- 172.21.15.3:0/175985562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101340 0x7f62fc1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 --2- 172.21.15.3:0/175985562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc100570 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 -- 172.21.15.3:0/175985562 >> 172.21.15.3:0/175985562 conn(0x7f62fc0fb6c0 msgr2=0x7f62fc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 -- 172.21.15.3:0/175985562 shutdown_connections 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.122+0000 7f6301502640 1 -- 172.21.15.3:0/175985562 wait complete. 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 Processor -- start 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 -- start start 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc06a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101340 0x7f62fc19c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101d00 0x7f62fc19e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc19ee40 con 0x7f62fc100150 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc19efb0 con 0x7f62fc101340 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f6301502640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62fc19f170 con 0x7f62fc101d00 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc06a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc06a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38864/0 (socket says 172.21.15.3:38864) 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 -- 172.21.15.3:0/3122043539 learned_addr learned my addr 172.21.15.3:0/3122043539 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62f3fff640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101340 0x7f62fc19c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62fb7fe640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101d00 0x7f62fc19e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 -- 172.21.15.3:0/3122043539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101d00 msgr2=0x7f62fc19e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101d00 0x7f62fc19e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 -- 172.21.15.3:0/3122043539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101340 msgr2=0x7f62fc19c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101340 0x7f62fc19c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 -- 172.21.15.3:0/3122043539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62ec009560 con 0x7f62fc100150 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62f3fff640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101340 0x7f62fc19c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62fb7fe640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101d00 0x7f62fc19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.123+0000 7f62faffd640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc06a460 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f62ec005ab0 tx=0x7f62ec005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.124+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0043d0 con 0x7f62fc100150 2024-02-09T22:50:29.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.124+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62ec0052d0 con 0x7f62fc100150 2024-02-09T22:50:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.124+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62fc19f3f0 con 0x7f62fc100150 2024-02-09T22:50:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.124+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62ec003d40 con 0x7f62fc100150 2024-02-09T22:50:29.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.124+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62fc19f900 con 0x7f62fc100150 2024-02-09T22:50:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.125+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62fc107470 con 0x7f62fc100150 2024-02-09T22:50:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.126+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f62ec005440 con 0x7f62fc100150 2024-02-09T22:50:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.126+0000 7f62f8ff9640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f62e80a2150 0x7f62e80a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.126+0000 7f62f3fff640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f62e80a2150 0x7f62e80a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.127+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f62ec0effc0 con 0x7f62fc100150 2024-02-09T22:50:29.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.127+0000 7f62f3fff640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f62e80a2150 0x7f62e80a4610 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f62fc06a1d0 tx=0x7f62e4005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:29.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.131+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f62ec0b8f50 con 0x7f62fc100150 2024-02-09T22:50:29.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.313+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f62fc10bdb0 con 0x7f62fc100150 2024-02-09T22:50:29.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.313+0000 7f62f8ff9640 1 -- 172.21.15.3:0/3122043539 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f62ec0bc610 con 0x7f62fc100150 2024-02-09T22:50:29.316 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.315+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f62e80a2150 msgr2=0x7f62e80a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.315+0000 7f6301502640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f62e80a2150 0x7f62e80a4610 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f62fc06a1d0 tx=0x7f62e4005f90 comp rx=0 tx=0).stop 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.315+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 msgr2=0x7f62fc06a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc06a460 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f62ec005ab0 tx=0x7f62ec005e50 comp rx=0 tx=0).stop 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 shutdown_connections 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f62e80a2150 0x7f62e80a4610 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62fc101340 0x7f62fc19c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62fc101d00 0x7f62fc19e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 --2- 172.21.15.3:0/3122043539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62fc100150 0x7f62fc06a460 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 >> 172.21.15.3:0/3122043539 conn(0x7f62fc0fb6c0 msgr2=0x7f62fc104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 shutdown_connections 2024-02-09T22:50:29.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.316+0000 7f6301502640 1 -- 172.21.15.3:0/3122043539 wait complete. 2024-02-09T22:50:29.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:29 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/388074341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:29.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:29 smithi003 ceph-mon[32678]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:29.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:29 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1141191458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:29.329 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:29 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1308596504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:29.330 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:50:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 -- 172.21.15.3:0/3292340160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101330 msgr2=0x7f2c801017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 --2- 172.21.15.3:0/3292340160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101330 0x7f2c801017b0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f2c70009900 tx=0x7f2c7002bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:29.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 -- 172.21.15.3:0/3292340160 shutdown_connections 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 --2- 172.21.15.3:0/3292340160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c80101cf0 0x7f2c80108580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 --2- 172.21.15.3:0/3292340160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101330 0x7f2c801017b0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 --2- 172.21.15.3:0/3292340160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c80100140 0x7f2c80100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 -- 172.21.15.3:0/3292340160 >> 172.21.15.3:0/3292340160 conn(0x7f2c800fb690 msgr2=0x7f2c800fdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 -- 172.21.15.3:0/3292340160 shutdown_connections 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 -- 172.21.15.3:0/3292340160 wait complete. 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 Processor -- start 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.433+0000 7f2c88577640 1 -- start start 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c80100140 0x7f2c8010aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c80101330 0x7f2c80109040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 0x7f2c80109580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80109ce0 con 0x7f2c80101cf0 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80109e50 con 0x7f2c80100140 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c801a1320 con 0x7f2c80101330 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 0x7f2c80109580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 0x7f2c80109580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38872/0 (socket says 172.21.15.3:38872) 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 -- 172.21.15.3:0/2802859975 learned_addr learned my addr 172.21.15.3:0/2802859975 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86313640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c80100140 0x7f2c8010aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c85b12640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c80101330 0x7f2c80109040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 -- 172.21.15.3:0/2802859975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c80101330 msgr2=0x7f2c80109040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c80101330 0x7f2c80109040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 -- 172.21.15.3:0/2802859975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c80100140 msgr2=0x7f2c8010aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c80100140 0x7f2c8010aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 -- 172.21.15.3:0/2802859975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c70009560 con 0x7f2c80101cf0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c86b14640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 0x7f2c80109580 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800ead0 tx=0x7f2c7800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7800cde0 con 0x7f2c80101cf0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c78005020 con 0x7f2c80101cf0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c801a15d0 con 0x7f2c80101cf0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c780116b0 con 0x7f2c80101cf0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.434+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c801a1a60 con 0x7f2c80101cf0 2024-02-09T22:50:29.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.435+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c80107460 con 0x7f2c80101cf0 2024-02-09T22:50:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.436+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f2c78005520 con 0x7f2c80101cf0 2024-02-09T22:50:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.437+0000 7f2c777fe640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2c6c0a2150 0x7f2c6c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.437+0000 7f2c86313640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2c6c0a2150 0x7f2c6c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.437+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2c780cdfe0 con 0x7f2c80101cf0 2024-02-09T22:50:29.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.437+0000 7f2c86313640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2c6c0a2150 0x7f2c6c0a4610 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2c80101190 tx=0x7f2c7c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:29.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.441+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2c78101d20 con 0x7f2c80101cf0 2024-02-09T22:50:29.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.622+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2c8002ce50 con 0x7f2c80101cf0 2024-02-09T22:50:29.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.622+0000 7f2c777fe640 1 -- 172.21.15.3:0/2802859975 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2c7801f020 con 0x7f2c80101cf0 2024-02-09T22:50:29.626 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:50:29.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2c6c0a2150 msgr2=0x7f2c6c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2c6c0a2150 0x7f2c6c0a4610 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2c80101190 tx=0x7f2c7c00b390 comp rx=0 tx=0).stop 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 msgr2=0x7f2c80109580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 0x7f2c80109580 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f2c7800ead0 tx=0x7f2c7800ef90 comp rx=0 tx=0).stop 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 shutdown_connections 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2c6c0a2150 0x7f2c6c0a4610 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2c80101cf0 0x7f2c80109580 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2c80101330 0x7f2c80109040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 --2- 172.21.15.3:0/2802859975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2c80100140 0x7f2c8010aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.624+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 >> 172.21.15.3:0/2802859975 conn(0x7f2c800fb690 msgr2=0x7f2c80104630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.625+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 shutdown_connections 2024-02-09T22:50:29.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.625+0000 7f2c88577640 1 -- 172.21.15.3:0/2802859975 wait complete. 2024-02-09T22:50:29.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:50:29.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:29 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/388074341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:29.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:29 smithi064 ceph-mon[36638]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:29.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:29 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1141191458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:29.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:29 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1308596504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:29.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:29 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/388074341' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:29.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:29 smithi089 ceph-mon[36665]: pgmap v329: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:29.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:29 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1141191458' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:29.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:29 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1308596504' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:29.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 -- 172.21.15.3:0/3269529862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f341028e0 msgr2=0x7f4f34102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/3269529862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f341028e0 0x7f4f34102d00 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4f30009900 tx=0x7f4f3002bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 -- 172.21.15.3:0/3269529862 shutdown_connections 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/3269529862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f340ff350 0x7f4f340ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/3269529862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f340ffd10 0x7f4f341098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/3269529862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f341028e0 0x7f4f34102d00 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 -- 172.21.15.3:0/3269529862 >> 172.21.15.3:0/3269529862 conn(0x7f4f340faeb0 msgr2=0x7f4f340fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 -- 172.21.15.3:0/3269529862 shutdown_connections 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.738+0000 7f4f3c70c640 1 -- 172.21.15.3:0/3269529862 wait complete. 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 Processor -- start 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 -- start start 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 0x7f4f341972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f340ffd10 0x7f4f341977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f341028e0 0x7f4f3419e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3419ed90 con 0x7f4f340ff350 2024-02-09T22:50:29.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3419ef00 con 0x7f4f341028e0 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3c70c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f3419f0c0 con 0x7f4f340ffd10 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 0x7f4f341972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 0x7f4f341972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38896/0 (socket says 172.21.15.3:38896) 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 -- 172.21.15.3:0/1054702197 learned_addr learned my addr 172.21.15.3:0/1054702197 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3aca9640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f341028e0 0x7f4f3419e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 -- 172.21.15.3:0/1054702197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f340ffd10 msgr2=0x7f4f341977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f340ffd10 0x7f4f341977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 -- 172.21.15.3:0/1054702197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f341028e0 msgr2=0x7f4f3419e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f341028e0 0x7f4f3419e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 -- 172.21.15.3:0/1054702197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f30009560 con 0x7f4f340ff350 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3aca9640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f341028e0 0x7f4f3419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.739+0000 7f4f3a4a8640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 0x7f4f341972a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4f30036040 tx=0x7f4f300042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.740+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f300044e0 con 0x7f4f340ff350 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.740+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f300054e0 con 0x7f4f340ff350 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.740+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f3003e450 con 0x7f4f340ff350 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.740+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f3419f340 con 0x7f4f340ff350 2024-02-09T22:50:29.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.740+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f3419f920 con 0x7f4f340ff350 2024-02-09T22:50:29.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.741+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f34108740 con 0x7f4f340ff350 2024-02-09T22:50:29.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.742+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f4f30003710 con 0x7f4f340ff350 2024-02-09T22:50:29.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.743+0000 7f4f2b7fe640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4f200a21d0 0x7f4f200a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:29.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.743+0000 7f4f39ca7640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4f200a21d0 0x7f4f200a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:29.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.743+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4f300f07f0 con 0x7f4f340ff350 2024-02-09T22:50:29.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.743+0000 7f4f39ca7640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4f200a21d0 0x7f4f200a4690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4f341987b0 tx=0x7f4f24005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:29.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.746+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4f300b9780 con 0x7f4f340ff350 2024-02-09T22:50:29.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.965+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4f3410ccd0 con 0x7f4f340ff350 2024-02-09T22:50:29.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.965+0000 7f4f2b7fe640 1 -- 172.21.15.3:0/1054702197 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f4f300bce40 con 0x7f4f340ff350 2024-02-09T22:50:29.969 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:29.969 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:49:13.115988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:50:29.969 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:50:29.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4f200a21d0 msgr2=0x7f4f200a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4f200a21d0 0x7f4f200a4690 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4f341987b0 tx=0x7f4f24005c70 comp rx=0 tx=0).stop 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 msgr2=0x7f4f341972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 0x7f4f341972a0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f4f30036040 tx=0x7f4f300042f0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 shutdown_connections 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4f200a21d0 0x7f4f200a4690 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f341028e0 0x7f4f3419e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f340ffd10 0x7f4f341977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 --2- 172.21.15.3:0/1054702197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f340ff350 0x7f4f341972a0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 >> 172.21.15.3:0/1054702197 conn(0x7f4f340faeb0 msgr2=0x7f4f34105910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.967+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 shutdown_connections 2024-02-09T22:50:29.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:29.968+0000 7f4f3c70c640 1 -- 172.21.15.3:0/1054702197 wait complete. 2024-02-09T22:50:29.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:50:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 -- 172.21.15.3:0/946952516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 msgr2=0x7f00e41016e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 --2- 172.21.15.3:0/946952516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e41016e0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009900 tx=0x7f00d402bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 -- 172.21.15.3:0/946952516 shutdown_connections 2024-02-09T22:50:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 --2- 172.21.15.3:0/946952516 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e41024b0 0x7f00e4102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 --2- 172.21.15.3:0/946952516 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00e4102e70 0x7f00e4109700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 --2- 172.21.15.3:0/946952516 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e41016e0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 -- 172.21.15.3:0/946952516 >> 172.21.15.3:0/946952516 conn(0x7f00e40fc850 msgr2=0x7f00e40fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 -- 172.21.15.3:0/946952516 shutdown_connections 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 -- 172.21.15.3:0/946952516 wait complete. 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.081+0000 7f00e980f640 1 Processor -- start 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 -- start start 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e4199450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00e41024b0 0x7f00e4199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e4102e70 0x7f00e41a0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41a0f40 con 0x7f00e41012c0 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41a10b0 con 0x7f00e4102e70 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e980f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00e41a1270 con 0x7f00e41024b0 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e4199450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e4199450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38908/0 (socket says 172.21.15.3:38908) 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 -- 172.21.15.3:0/3397642539 learned_addr learned my addr 172.21.15.3:0/3397642539 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e37fe640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e4102e70 0x7f00e41a0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 -- 172.21.15.3:0/3397642539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00e41024b0 msgr2=0x7f00e4199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00e41024b0 0x7f00e4199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 -- 172.21.15.3:0/3397642539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e4102e70 msgr2=0x7f00e41a0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e4102e70 0x7f00e41a0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 -- 172.21.15.3:0/3397642539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d4009560 con 0x7f00e41012c0 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e37fe640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e4102e70 0x7f00e41a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.082+0000 7f00e2ffd640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e4199450 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f00d4000c00 tx=0x7f00d4004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.083+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d4004df0 con 0x7f00e41012c0 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.083+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00d4005430 con 0x7f00e41012c0 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.083+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00e41a14f0 con 0x7f00e41012c0 2024-02-09T22:50:30.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.083+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00d403e450 con 0x7f00e41012c0 2024-02-09T22:50:30.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.083+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00e41a1a00 con 0x7f00e41012c0 2024-02-09T22:50:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.084+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00e410cc60 con 0x7f00e41012c0 2024-02-09T22:50:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.085+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f00d40055a0 con 0x7f00e41012c0 2024-02-09T22:50:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.086+0000 7f00e880d640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f00cc0a2150 0x7f00cc0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.086+0000 7f00e27fc640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f00cc0a2150 0x7f00cc0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.086+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f00d40f00d0 con 0x7f00e41012c0 2024-02-09T22:50:30.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.086+0000 7f00e27fc640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f00cc0a2150 0x7f00cc0a4610 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00e419a960 tx=0x7f00d8005c70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.090+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f00d40b9060 con 0x7f00e41012c0 2024-02-09T22:50:30.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.277+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00e410cdf0 con 0x7f00e41012c0 2024-02-09T22:50:30.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.280+0000 7f00e880d640 1 -- 172.21.15.3:0/3397642539 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f00d40bc720 con 0x7f00e41012c0 2024-02-09T22:50:30.283 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:30.283 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:50:30.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f00cc0a2150 msgr2=0x7f00cc0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f00cc0a2150 0x7f00cc0a4610 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f00e419a960 tx=0x7f00d8005c70 comp rx=0 tx=0).stop 2024-02-09T22:50:30.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 msgr2=0x7f00e4199450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e4199450 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f00d4000c00 tx=0x7f00d4004270 comp rx=0 tx=0).stop 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 shutdown_connections 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f00cc0a2150 0x7f00cc0a4610 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f00e4102e70 0x7f00e41a0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f00e41024b0 0x7f00e4199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 --2- 172.21.15.3:0/3397642539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f00e41012c0 0x7f00e4199450 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 >> 172.21.15.3:0/3397642539 conn(0x7f00e40fc850 msgr2=0x7f00e4073460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 shutdown_connections 2024-02-09T22:50:30.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.283+0000 7f00e980f640 1 -- 172.21.15.3:0/3397642539 wait complete. 2024-02-09T22:50:30.302 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.402+0000 7f504c9c2640 1 -- 172.21.15.3:0/4488734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 msgr2=0x7f50481096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.402+0000 7f504c9c2640 1 --2- 172.21.15.3:0/4488734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f50481096c0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f5040009ca0 tx=0x7f504002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- 172.21.15.3:0/4488734 shutdown_connections 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 --2- 172.21.15.3:0/4488734 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f50481096c0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 --2- 172.21.15.3:0/4488734 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5048102470 0x7f50481028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 --2- 172.21.15.3:0/4488734 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5048101280 0x7f50481016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- 172.21.15.3:0/4488734 >> 172.21.15.3:0/4488734 conn(0x7f50480fc7f0 msgr2=0x7f50480fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- 172.21.15.3:0/4488734 shutdown_connections 2024-02-09T22:50:30.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- 172.21.15.3:0/4488734 wait complete. 2024-02-09T22:50:30.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 Processor -- start 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- start start 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5048101280 0x7f5048074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5048102470 0x7f5048073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f5048073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50480742c0 con 0x7f5048102e30 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.403+0000 7f504c9c2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5048074430 con 0x7f5048102470 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f504c9c2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50481a12c0 con 0x7f5048101280 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f5048073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f5048073b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38922/0 (socket says 172.21.15.3:38922) 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 -- 172.21.15.3:0/1073462450 learned_addr learned my addr 172.21.15.3:0/1073462450 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f504659c640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5048101280 0x7f5048074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 -- 172.21.15.3:0/1073462450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5048101280 msgr2=0x7f5048074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5048101280 0x7f5048074fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 -- 172.21.15.3:0/1073462450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5048102470 msgr2=0x7f5048073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5045d9b640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5048102470 0x7f5048073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5048102470 0x7f5048073620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 -- 172.21.15.3:0/1073462450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5040009900 con 0x7f5048102e30 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f5046d9d640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f5048073b60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f50400098d0 tx=0x7f5040034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040032eb0 con 0x7f5048102e30 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5040003e50 con 0x7f5048102e30 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50481a1510 con 0x7f5048102e30 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50400406b0 con 0x7f5048102e30 2024-02-09T22:50:30.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.404+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50481a19c0 con 0x7f5048102e30 2024-02-09T22:50:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.405+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50481085a0 con 0x7f5048102e30 2024-02-09T22:50:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.406+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f504000a040 con 0x7f5048102e30 2024-02-09T22:50:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.407+0000 7f503f7fe640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50340a2150 0x7f50340a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.407+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f50400f1270 con 0x7f5048102e30 2024-02-09T22:50:30.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.407+0000 7f504659c640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50340a2150 0x7f50340a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.407+0000 7f504659c640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50340a2150 0x7f50340a4610 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5030005fd0 tx=0x7f5030005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.411+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f50400ba3b0 con 0x7f5048102e30 2024-02-09T22:50:30.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3122043539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:30.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2802859975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:30.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:30 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1054702197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:30.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.634+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f504810cc90 con 0x7f5048102e30 2024-02-09T22:50:30.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.635+0000 7f503f7fe640 1 -- 172.21.15.3:0/1073462450 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f50400bda70 con 0x7f5048102e30 2024-02-09T22:50:30.638 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:30.638 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:49:13.115988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:50:30.638 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50340a2150 msgr2=0x7f50340a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50340a2150 0x7f50340a4610 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5030005fd0 tx=0x7f5030005850 comp rx=0 tx=0).stop 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 msgr2=0x7f5048073b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f5048073b60 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f50400098d0 tx=0x7f5040034d40 comp rx=0 tx=0).stop 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 shutdown_connections 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f50340a2150 0x7f50340a4610 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5048102470 0x7f5048073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5048102e30 0x7f5048073b60 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 --2- 172.21.15.3:0/1073462450 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5048101280 0x7f5048074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 >> 172.21.15.3:0/1073462450 conn(0x7f50480fc7f0 msgr2=0x7f5048105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 shutdown_connections 2024-02-09T22:50:30.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.637+0000 7f504c9c2640 1 -- 172.21.15.3:0/1073462450 wait complete. 2024-02-09T22:50:30.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-02-09T22:49:06.610270+0000', 'modified': '2024-02-09T22:49:13.115988+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24533, 'mds_1': 24503, 'mds_2': 24496, 'mds_3': 24481, 'mds_4': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.3:6839/3695104270', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6838', 'nonce': 3695104270}, {'type': 'v1', 'addr': '172.21.15.3:6839', 'nonce': 3695104270}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.3:6841/3916323155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6840', 'nonce': 3916323155}, {'type': 'v1', 'addr': '172.21.15.3:6841', 'nonce': 3916323155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6833/2685783987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6832', 'nonce': 2685783987}, {'type': 'v1', 'addr': '172.21.15.89:6833', 'nonce': 2685783987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24437': {'gid': 24437, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6833/1710343231', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6832', 'nonce': 1710343231}, {'type': 'v1', 'addr': '172.21.15.64:6833', 'nonce': 1710343231}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/3082808593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 3082808593}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 3082808593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.89:6835/4241363743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 4241363743}, {'type': 'v1', 'addr': '172.21.15.89:6835', 'nonce': 4241363743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.89:6837/1935794058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 1935794058}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 1935794058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.89:6839/305807401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 305807401}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 305807401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.64:6837/3217313563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 3217313563}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 3217313563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.64:6839/4207075446', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 4207075446}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 4207075446}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-09T22:50:30.652 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-02-09T22:50:30.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-02-09T22:50:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3122043539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2802859975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:30 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1054702197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3122043539' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2802859975' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:30 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1054702197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.754+0000 7fd23db4f640 1 -- 172.21.15.3:0/3203460017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 msgr2=0x7fd238100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.754+0000 7fd23db4f640 1 --2- 172.21.15.3:0/3203460017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238100570 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fd228009900 tx=0x7fd22802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 -- 172.21.15.3:0/3203460017 shutdown_connections 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 --2- 172.21.15.3:0/3203460017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd238101d00 0x7fd238108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 --2- 172.21.15.3:0/3203460017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd238101340 0x7fd2381017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 --2- 172.21.15.3:0/3203460017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238100570 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 -- 172.21.15.3:0/3203460017 >> 172.21.15.3:0/3203460017 conn(0x7fd2380fb6c0 msgr2=0x7fd2380fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 -- 172.21.15.3:0/3203460017 shutdown_connections 2024-02-09T22:50:30.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 -- 172.21.15.3:0/3203460017 wait complete. 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 Processor -- start 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 -- start start 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238069070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.755+0000 7fd23db4f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd238101340 0x7fd23819c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd23db4f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd238101d00 0x7fd23819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd23db4f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23819ee90 con 0x7fd238100150 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd23db4f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23819f000 con 0x7fd238101d00 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd23db4f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23819f1c0 con 0x7fd238101340 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238069070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238069070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38946/0 (socket says 172.21.15.3:38946) 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 -- 172.21.15.3:0/1659964016 learned_addr learned my addr 172.21.15.3:0/1659964016 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd236ffd640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd238101340 0x7fd23819c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 -- 172.21.15.3:0/1659964016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd238101340 msgr2=0x7fd23819c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd238101340 0x7fd23819c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 -- 172.21.15.3:0/1659964016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd238101d00 msgr2=0x7fd23819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd238101d00 0x7fd23819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 -- 172.21.15.3:0/1659964016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd228009560 con 0x7fd238100150 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd2377fe640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238069070 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd228000c00 tx=0x7fd228004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd234ff9640 1 -- 172.21.15.3:0/1659964016 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd228004df0 con 0x7fd238100150 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd234ff9640 1 -- 172.21.15.3:0/1659964016 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd228005430 con 0x7fd238100150 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd23819f3e0 con 0x7fd238100150 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd234ff9640 1 -- 172.21.15.3:0/1659964016 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22803e450 con 0x7fd238100150 2024-02-09T22:50:30.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.756+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd23819f8f0 con 0x7fd238100150 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.758+0000 7fd234ff9640 1 -- 172.21.15.3:0/1659964016 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fd228003680 con 0x7fd238100150 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 Processor -- start 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 -- start start 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd234ff9640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd2200a2150 0x7fd2200a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eefd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef021b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef04cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ee6cb30 con 0x55d92eefd6b0 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ee6a060 con 0x55d92ef02820 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd240804740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ee65840 con 0x55d92eefccc0 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd237fff640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd2200a2150 0x7fd2200a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd234ff9640 1 -- 172.21.15.3:0/1659964016 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd2280f0240 con 0x7fd238100150 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef021b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38950/0 (socket says 172.21.15.3:38950) 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 -- 172.21.15.3:0/3440332423 learned_addr learned my addr 172.21.15.3:0/3440332423 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd237fff640 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef04cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eefd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 -- 172.21.15.3:0/3440332423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 msgr2=0x55d92eefd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eefd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 -- 172.21.15.3:0/3440332423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 msgr2=0x55d92ef04cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef04cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd236ffd640 1 -- 172.21.15.3:0/3440332423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d92ee2b620 con 0x55d92eefd6b0 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.759+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eefd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd237fff640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd2200a2150 0x7fd2200a4610 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd230005fd0 tx=0x7fd230009c30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef021b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00b740 tx=0x7fd22c00bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1543e2006245ed1c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd21dffb640 1 -- 172.21.15.3:0/3440332423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c003d50 con 0x55d92eefd6b0 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd21dffb640 1 -- 172.21.15.3:0/3440332423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd22c00be50 con 0x55d92eefd6b0 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd21dffb640 1 -- 172.21.15.3:0/3440332423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c011440 con 0x55d92eefd6b0 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 -- 172.21.15.3:0/3440332423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 msgr2=0x55d92ef021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef021b0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fd22c00b740 tx=0x7fd22c00bc00 comp rx=0 tx=0).stop 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 -- 172.21.15.3:0/3440332423 shutdown_connections 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef04cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef021b0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 --2- 172.21.15.3:0/3440332423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eefd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.760+0000 7fd240804740 1 -- 172.21.15.3:0/3440332423 >> 172.21.15.3:0/3440332423 conn(0x55d92eef8230 msgr2=0x55d92eef8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 -- 172.21.15.3:0/3440332423 shutdown_connections 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 -- 172.21.15.3:0/3440332423 wait complete. 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 -- start start 2024-02-09T22:50:30.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 Processor -- start 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92ef0afb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef12530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ee0acd0 con 0x55d92eefd6b0 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92eb39310 con 0x55d92ef02820 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92edeb490 con 0x55d92eefccc0 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd240804740 10 client.0 _ll_register_callbacks cb 0x55d92eb76ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd236ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd236ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38962/0 (socket says 172.21.15.3:38962) 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.761+0000 7fd236ffd640 1 -- 172.21.15.3:0/3684416038 learned_addr learned my addr 172.21.15.3:0/3684416038 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92ef0afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd237fff640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef12530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd236ffd640 1 -- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 msgr2=0x55d92ef0afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92ef0afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd236ffd640 1 -- 172.21.15.3:0/3684416038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 msgr2=0x55d92ef12530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef12530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd236ffd640 1 -- 172.21.15.3:0/3684416038 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd22c004310 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd237fff640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef12530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0b4f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0039c0 tx=0x7fd22c003d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=982b67bab2a314b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd216ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd216ffd640 1 -- 172.21.15.3:0/3684416038 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c0050a0 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd216ffd640 1 -- 172.21.15.3:0/3684416038 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd22c003e70 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d92eb656c0 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd216ffd640 1 -- 172.21.15.3:0/3684416038 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd22c005200 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd240804740 10 client.15066 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d92ee63ec0 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.762+0000 7fd216ffd640 1 -- 172.21.15.3:0/3684416038 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d92eb656c0 con 0x55d92eefd6b0 2024-02-09T22:50:30.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd216ffd640 1 -- 172.21.15.3:0/3684416038 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd22c019420 con 0x55d92eefd6b0 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd240804740 10 client.15066 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd240804740 10 client.15066 resolve_mds: resolved name 'f' to daemon mds.f 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd240804740 20 client.15066 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd240804740 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef17800 0x55d92ef1bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd240804740 4 client.15066 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d92ebb6270 con 0x55d92ef17800 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef17800 0x55d92ef1bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd216ffd640 10 client.15066 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.763+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef17800 0x55d92ef1bc80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c8016fa70c9018eb server_cookie=2bd86e906d24978b in_seq=0 out_seq=0 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd216ffd640 1 -- 172.21.15.3:0/3684416038 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d92ebb6270 con 0x55d92ef17800 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd216ffd640 10 client.15066 handle_command_reply: tid=0 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 client.15066 shutdown 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 msgr2=0x55d92ef0b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0b4f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0039c0 tx=0x7fd22c003d80 comp rx=0 tx=0).stop 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 shutdown_connections 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef17800 0x55d92ef1bc80 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92ef02820 0x55d92ef12530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0b4f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 --2- 172.21.15.3:0/3684416038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92ef0afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd216ffd640 0 client.15066 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 >> 172.21.15.3:0/3684416038 conn(0x55d92eef8230 msgr2=0x55d92eef7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 shutdown_connections 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.764+0000 7fd240804740 1 -- 172.21.15.3:0/3684416038 wait complete. 2024-02-09T22:50:30.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.765+0000 7fd240804740 20 client.15066 trim_cache size 0 max 16384 2024-02-09T22:50:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.780+0000 7fd240804740 1 Processor -- start 2024-02-09T22:50:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.780+0000 7fd240804740 1 -- start start 2024-02-09T22:50:30.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.780+0000 7fd240804740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eef8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd240804740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefd6b0 0x55d92ef124f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd240804740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 0x55d92ef04c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd240804740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ef18090 con 0x55d92ef02820 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd240804740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ef133e0 con 0x55d92eefd6b0 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd240804740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ef0bfa0 con 0x55d92eefccc0 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 0x55d92ef04c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 0x55d92ef04c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38972/0 (socket says 172.21.15.3:38972) 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 -- 172.21.15.3:0/4112567078 learned_addr learned my addr 172.21.15.3:0/4112567078 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd2377fe640 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eef8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd236ffd640 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefd6b0 0x55d92ef124f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 -- 172.21.15.3:0/4112567078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 msgr2=0x55d92eef8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eef8610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 -- 172.21.15.3:0/4112567078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefd6b0 msgr2=0x55d92ef124f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefd6b0 0x55d92ef124f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 -- 172.21.15.3:0/4112567078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d92ef16e90 con 0x55d92ef02820 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd237fff640 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 0x55d92ef04c10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd230009740 tx=0x7fd23001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6adaf860f0a0d7d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd21d7fa640 1 -- 172.21.15.3:0/4112567078 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd23001a070 con 0x55d92ef02820 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd21d7fa640 1 -- 172.21.15.3:0/4112567078 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd230003e00 con 0x55d92ef02820 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.781+0000 7fd21d7fa640 1 -- 172.21.15.3:0/4112567078 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd230020400 con 0x55d92ef02820 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 -- 172.21.15.3:0/4112567078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 msgr2=0x55d92ef04c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 0x55d92ef04c10 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd230009740 tx=0x7fd23001d000 comp rx=0 tx=0).stop 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 -- 172.21.15.3:0/4112567078 shutdown_connections 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefd6b0 0x55d92ef124f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92ef02820 0x55d92ef04c10 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 --2- 172.21.15.3:0/4112567078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92eefccc0 0x55d92eef8610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 -- 172.21.15.3:0/4112567078 >> 172.21.15.3:0/4112567078 conn(0x55d92eef8230 msgr2=0x55d92eef7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 -- 172.21.15.3:0/4112567078 shutdown_connections 2024-02-09T22:50:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 -- 172.21.15.3:0/4112567078 wait complete. 2024-02-09T22:50:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.782+0000 7fd240804740 1 -- start start 2024-02-09T22:50:30.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 Processor -- start 2024-02-09T22:50:30.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefccc0 0x55d92ef12480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92ef02820 0x55d92ef0f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ef16d50 con 0x55d92eefd6b0 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ef6d940 con 0x55d92eefccc0 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d92ef6da80 con 0x55d92ef02820 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd240804740 10 client.0 _ll_register_callbacks cb 0x7fd22c004120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38974/0 (socket says 172.21.15.3:38974) 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 -- 172.21.15.3:0/3670517288 learned_addr learned my addr 172.21.15.3:0/3670517288 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefccc0 0x55d92ef12480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd237fff640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92ef02820 0x55d92ef0f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 -- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92ef02820 msgr2=0x55d92ef0f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92ef02820 0x55d92ef0f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 -- 172.21.15.3:0/3670517288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefccc0 msgr2=0x55d92ef12480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefccc0 0x55d92ef12480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.783+0000 7fd236ffd640 1 -- 172.21.15.3:0/3670517288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2300095b0 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd236ffd640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0ef70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0039c0 tx=0x7fd22c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e09eaa301f105a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd216ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd216ffd640 1 -- 172.21.15.3:0/3670517288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c004b30 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd216ffd640 1 -- 172.21.15.3:0/3670517288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fd22c000e40 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d92ef181d0 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd216ffd640 1 -- 172.21.15.3:0/3670517288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd22c0199f0 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 10 client.15072 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d92ee00a90 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd216ffd640 1 -- 172.21.15.3:0/3670517288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d92ef181d0 con 0x55d92eefd6b0 2024-02-09T22:50:30.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd216ffd640 1 -- 172.21.15.3:0/3670517288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd22c043450 con 0x55d92eefd6b0 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 10 client.15072 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 10 client.15072 resolve_mds: resolved name 'f' to daemon mds.f 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 20 client.15072 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef72a20 0x55d92ef6ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 4 client.15072 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.784+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7fd22c00be50 con 0x55d92ef72a20 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef72a20 0x55d92ef6ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd2377fe640 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef72a20 0x55d92ef6ee40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ec01c1989e96c3cd server_cookie=8fe338e15dcc34d2 in_seq=0 out_seq=0 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd216ffd640 10 client.15072 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd216ffd640 1 -- 172.21.15.3:0/3670517288 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x7fd22c00be50 con 0x55d92ef72a20 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd216ffd640 10 client.15072 handle_command_reply: tid=0 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 client.15072 shutdown 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 msgr2=0x55d92ef0ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0ef70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd22c0039c0 tx=0x7fd22c00b480 comp rx=0 tx=0).stop 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 shutdown_connections 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d92ef72a20 0x55d92ef6ee40 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d92ef02820 0x55d92ef0f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d92eefd6b0 0x55d92ef0ef70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd216ffd640 0 client.15072 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 --2- 172.21.15.3:0/3670517288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d92eefccc0 0x55d92ef12480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 >> 172.21.15.3:0/3670517288 conn(0x55d92eef8230 msgr2=0x55d92eef6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 shutdown_connections 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.785+0000 7fd240804740 1 -- 172.21.15.3:0/3670517288 wait complete. 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd240804740 20 client.15072 trim_cache size 0 max 16384 2024-02-09T22:50:30.795 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 4, 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24475, 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 78.673319327000002, 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:30.796 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 105.720635835 2024-02-09T22:50:30.799 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd2200a2150 msgr2=0x7fd2200a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd2200a2150 0x7fd2200a4610 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd230005fd0 tx=0x7fd230009c30 comp rx=0 tx=0).stop 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 msgr2=0x7fd238069070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238069070 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fd228000c00 tx=0x7fd228004270 comp rx=0 tx=0).stop 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 shutdown_connections 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fd2200a2150 0x7fd2200a4610 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd238101d00 0x7fd23819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd238101340 0x7fd23819c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 --2- 172.21.15.3:0/1659964016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd238100150 0x7fd238069070 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 >> 172.21.15.3:0/1659964016 conn(0x7fd2380fb6c0 msgr2=0x7fd2381046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 shutdown_connections 2024-02-09T22:50:30.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.786+0000 7fd23db4f640 1 -- 172.21.15.3:0/1659964016 wait complete. 2024-02-09T22:50:30.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-09T22:50:30.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 -- 172.21.15.3:0/2175493262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634101340 msgr2=0x7fa6341017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 --2- 172.21.15.3:0/2175493262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634101340 0x7fa6341017c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa628009900 tx=0x7fa62802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:30.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 -- 172.21.15.3:0/2175493262 shutdown_connections 2024-02-09T22:50:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 --2- 172.21.15.3:0/2175493262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 0x7fa634108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 --2- 172.21.15.3:0/2175493262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634101340 0x7fa6341017c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 --2- 172.21.15.3:0/2175493262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634100150 0x7fa634100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 -- 172.21.15.3:0/2175493262 >> 172.21.15.3:0/2175493262 conn(0x7fa6340fb6c0 msgr2=0x7fa6340fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 -- 172.21.15.3:0/2175493262 shutdown_connections 2024-02-09T22:50:30.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.904+0000 7fa638dd7640 1 -- 172.21.15.3:0/2175493262 wait complete. 2024-02-09T22:50:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 Processor -- start 2024-02-09T22:50:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 -- start start 2024-02-09T22:50:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 0x7fa6341994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634101340 0x7fa634199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 0x7fa6341a0a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6341a0fd0 con 0x7fa634100150 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6341a1140 con 0x7fa634101d00 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa638dd7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6341a1300 con 0x7fa634101340 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa63259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 0x7fa6341994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa63259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 0x7fa6341994e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38980/0 (socket says 172.21.15.3:38980) 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa63259c640 1 -- 172.21.15.3:0/1932344161 learned_addr learned my addr 172.21.15.3:0/1932344161 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa632d9d640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 0x7fa6341a0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.905+0000 7fa631d9b640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634101340 0x7fa634199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa63259c640 1 -- 172.21.15.3:0/1932344161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634101340 msgr2=0x7fa634199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa63259c640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634101340 0x7fa634199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa63259c640 1 -- 172.21.15.3:0/1932344161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 msgr2=0x7fa6341a0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa63259c640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 0x7fa6341a0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa63259c640 1 -- 172.21.15.3:0/1932344161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa628009560 con 0x7fa634100150 2024-02-09T22:50:30.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa632d9d640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 0x7fa6341a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa631d9b640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634101340 0x7fa634199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa63259c640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 0x7fa6341994e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa62400d950 tx=0x7fa62400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa6237fe640 1 -- 172.21.15.3:0/1932344161 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624009c30 con 0x7fa634100150 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa6237fe640 1 -- 172.21.15.3:0/1932344161 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa624009d90 con 0x7fa634100150 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6341a15e0 con 0x7fa634100150 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa6237fe640 1 -- 172.21.15.3:0/1932344161 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62400f720 con 0x7fa634100150 2024-02-09T22:50:30.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.906+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6341a1aa0 con 0x7fa634100150 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.908+0000 7fa6237fe640 1 -- 172.21.15.3:0/1932344161 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fa624010a60 con 0x7fa634100150 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 Processor -- start 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 -- start start 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa6237fe640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa61c0a2150 0x7fa61c0a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a5763460e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a57634b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a57634dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5762b5b30 con 0x55a5763466b0 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5762b3060 con 0x55a57634b820 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5762ae840 con 0x55a576345cc0 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa632d9d640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa61c0a2150 0x7fa61c0a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a57634b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa6237fe640 1 -- 172.21.15.3:0/1932344161 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa6240cdc90 con 0x7fa634100150 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a57634b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38984/0 (socket says 172.21.15.3:38984) 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 -- 172.21.15.3:0/182621530 learned_addr learned my addr 172.21.15.3:0/182621530 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa63259c640 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a5763460e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 -- 172.21.15.3:0/182621530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 msgr2=0x55a5763460e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a5763460e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 -- 172.21.15.3:0/182621530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 msgr2=0x55a57634dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a57634dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.909+0000 7fa631d9b640 1 -- 172.21.15.3:0/182621530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a576274620 con 0x55a5763466b0 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa632d9d640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa61c0a2150 0x7fa61c0a4610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa62c005fd0 tx=0x7fa62c00d710 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa631d9b640 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a57634b1b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa628006010 tx=0x7fa62802c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253e2c9714cad226 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa620ff9640 1 -- 172.21.15.3:0/182621530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6280046c0 con 0x55a5763466b0 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa620ff9640 1 -- 172.21.15.3:0/182621530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa628004510 con 0x55a5763466b0 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa620ff9640 1 -- 172.21.15.3:0/182621530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6280054b0 con 0x55a5763466b0 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa63b9aa740 1 -- 172.21.15.3:0/182621530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 msgr2=0x55a57634b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.910+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a57634b1b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa628006010 tx=0x7fa62802c960 comp rx=0 tx=0).stop 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- 172.21.15.3:0/182621530 shutdown_connections 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a57634dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a57634b1b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/182621530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a5763460e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- 172.21.15.3:0/182621530 >> 172.21.15.3:0/182621530 conn(0x55a576341230 msgr2=0x55a576341630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- 172.21.15.3:0/182621530 shutdown_connections 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- 172.21.15.3:0/182621530 wait complete. 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- start start 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 Processor -- start 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a57634fb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a576350050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a576357090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a576253cd0 con 0x55a5763466b0 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a575f82310 con 0x55a57634b820 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a576234490 con 0x55a576345cc0 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63b9aa740 10 client.0 _ll_register_callbacks cb 0x55a575fbfac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa631d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a576350050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa631d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a576350050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38988/0 (socket says 172.21.15.3:38988) 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa631d9b640 1 -- 172.21.15.3:0/3644844609 learned_addr learned my addr 172.21.15.3:0/3644844609 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa63259c640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a57634fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.911+0000 7fa632d9d640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a576357090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa631d9b640 1 -- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 msgr2=0x55a57634fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa631d9b640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a57634fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa631d9b640 1 -- 172.21.15.3:0/3644844609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 msgr2=0x55a576357090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa631d9b640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a576357090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa631d9b640 1 -- 172.21.15.3:0/3644844609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa634108e90 con 0x55a5763466b0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa632d9d640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a576357090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa631d9b640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a576350050 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa628009900 tx=0x7fa62802c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e43df0fbac57463 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa619ffb640 1 -- 172.21.15.3:0/3644844609 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa628005620 con 0x55a5763466b0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a575fae6c0 con 0x55a5763466b0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa619ffb640 1 -- 172.21.15.3:0/3644844609 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa628034440 con 0x55a5763466b0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa619ffb640 1 -- 172.21.15.3:0/3644844609 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa628055030 con 0x55a5763466b0 2024-02-09T22:50:30.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa63b9aa740 10 client.15084 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a5762acec0 con 0x55a5763466b0 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa619ffb640 1 -- 172.21.15.3:0/3644844609 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a575fae6c0 con 0x55a5763466b0 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa619ffb640 1 -- 172.21.15.3:0/3644844609 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa62803c040 con 0x55a5763466b0 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.912+0000 7fa63b9aa740 10 client.15084 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63b9aa740 10 client.15084 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63b9aa740 20 client.15084 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57635ba20 0x55a5763604e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63b9aa740 4 client.15084 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a575fff270 con 0x55a57635ba20 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63259c640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57635ba20 0x55a5763604e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa63259c640 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57635ba20 0x55a5763604e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c0160c284e9b5115 server_cookie=1d6de7f43f778a04 in_seq=0 out_seq=0 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.913+0000 7fa619ffb640 10 client.15084 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:30.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa619ffb640 1 -- 172.21.15.3:0/3644844609 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a575fff270 con 0x55a57635ba20 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa619ffb640 10 client.15084 handle_command_reply: tid=0 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 client.15084 shutdown 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 msgr2=0x55a576350050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a576350050 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa628009900 tx=0x7fa62802c660 comp rx=0 tx=0).stop 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 shutdown_connections 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57635ba20 0x55a5763604e0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a57634b820 0x55a576357090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa619ffb640 0 client.15084 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a5763466b0 0x55a576350050 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3644844609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a576345cc0 0x55a57634fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 >> 172.21.15.3:0/3644844609 conn(0x55a576341230 msgr2=0x55a576340510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 shutdown_connections 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3644844609 wait complete. 2024-02-09T22:50:30.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.914+0000 7fa63b9aa740 20 client.15084 trim_cache size 0 max 16384 2024-02-09T22:50:30.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 Processor -- start 2024-02-09T22:50:30.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 -- start start 2024-02-09T22:50:30.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576340390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576357050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576364500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a57634e370 con 0x55a576341230 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a575fff270 con 0x55a576345cc0 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5763421a0 con 0x55a5763466b0 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576340390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576340390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:38990/0 (socket says 172.21.15.3:38990) 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.930+0000 7fa63259c640 1 -- 172.21.15.3:0/1862853887 learned_addr learned my addr 172.21.15.3:0/1862853887 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa631d9b640 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576357050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa632d9d640 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576364500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa63259c640 1 -- 172.21.15.3:0/1862853887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 msgr2=0x55a576364500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa63259c640 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576364500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa63259c640 1 -- 172.21.15.3:0/1862853887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 msgr2=0x55a576357050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa63259c640 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576357050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa63259c640 1 -- 172.21.15.3:0/1862853887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a576249a90 con 0x55a576341230 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa63259c640 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576340390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa62409a800 tx=0x7fa624010e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0f530407e93a670 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa61bfff640 1 -- 172.21.15.3:0/1862853887 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa62409a870 con 0x55a576341230 2024-02-09T22:50:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa61bfff640 1 -- 172.21.15.3:0/1862853887 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa62409e9c0 con 0x55a576341230 2024-02-09T22:50:30.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.931+0000 7fa61bfff640 1 -- 172.21.15.3:0/1862853887 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624098c00 con 0x55a576341230 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 -- 172.21.15.3:0/1862853887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 msgr2=0x55a576340390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576340390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa62409a800 tx=0x7fa624010e00 comp rx=0 tx=0).stop 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 -- 172.21.15.3:0/1862853887 shutdown_connections 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576357050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576364500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/1862853887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576340390 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 -- 172.21.15.3:0/1862853887 >> 172.21.15.3:0/1862853887 conn(0x55a576341980 msgr2=0x55a57633fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 -- 172.21.15.3:0/1862853887 shutdown_connections 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 -- 172.21.15.3:0/1862853887 wait complete. 2024-02-09T22:50:30.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 -- start start 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.932+0000 7fa63b9aa740 1 Processor -- start 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576357090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576353b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576354140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5763b21a0 con 0x55a576341230 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5763b22e0 con 0x55a576345cc0 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5763b2420 con 0x55a5763466b0 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 10 client.0 _ll_register_callbacks cb 0x7fa628009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576357090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576357090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39002/0 (socket says 172.21.15.3:39002) 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 -- 172.21.15.3:0/3180873274 learned_addr learned my addr 172.21.15.3:0/3180873274 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa631d9b640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576353b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa632d9d640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576354140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 -- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 msgr2=0x55a576354140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576354140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 -- 172.21.15.3:0/3180873274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 msgr2=0x55a576353b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576353b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 -- 172.21.15.3:0/3180873274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6341a1aa0 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63259c640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576357090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa624009610 tx=0x7fa62409a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d814bae111a39cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa619ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa619ffb640 1 -- 172.21.15.3:0/3180873274 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa624098910 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa619ffb640 1 -- 172.21.15.3:0/3180873274 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa6240cb8a0 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a57635d6f0 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa619ffb640 1 -- 172.21.15.3:0/3180873274 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa624065d50 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 10 client.15090 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.933+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55a57626f930 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa619ffb640 1 -- 172.21.15.3:0/3180873274 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a57635d6f0 con 0x55a576341230 2024-02-09T22:50:30.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa619ffb640 1 -- 172.21.15.3:0/3180873274 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa62409bba0 con 0x55a576341230 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa63b9aa740 10 client.15090 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa63b9aa740 10 client.15090 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa63b9aa740 20 client.15090 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57634b820 0x55a5763b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa63b9aa740 4 client.15090 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a57635c3d0 con 0x55a57634b820 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa631d9b640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57634b820 0x55a5763b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa631d9b640 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57634b820 0x55a5763b3740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d10a33f4a58bc43 server_cookie=5061e38153b860fc in_seq=0 out_seq=0 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.934+0000 7fa619ffb640 10 client.15090 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa619ffb640 1 -- 172.21.15.3:0/3180873274 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a57635c3d0 con 0x55a57634b820 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa619ffb640 10 client.15090 handle_command_reply: tid=0 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 client.15090 shutdown 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 msgr2=0x55a576357090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576357090 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fa624009610 tx=0x7fa62409a250 comp rx=0 tx=0).stop 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 shutdown_connections 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55a57634b820 0x55a5763b3740 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a576345cc0 0x55a576353b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa619ffb640 0 client.15090 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a5763466b0 0x55a576354140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 --2- 172.21.15.3:0/3180873274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a576341230 0x55a576357090 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 >> 172.21.15.3:0/3180873274 conn(0x55a576341980 msgr2=0x55a576364d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 shutdown_connections 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 1 -- 172.21.15.3:0/3180873274 wait complete. 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.935+0000 7fa63b9aa740 20 client.15090 trim_cache size 0 max 16384 2024-02-09T22:50:30.944 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 3, 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24481, 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 79.827780778999994, 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 103.183565753 2024-02-09T22:50:30.945 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:30.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa61c0a2150 msgr2=0x7fa61c0a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa61c0a2150 0x7fa61c0a4610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa62c005fd0 tx=0x7fa62c00d710 comp rx=0 tx=0).stop 2024-02-09T22:50:30.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 msgr2=0x7fa6341994e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:30.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 0x7fa6341994e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fa62400d950 tx=0x7fa62400de10 comp rx=0 tx=0).stop 2024-02-09T22:50:30.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 shutdown_connections 2024-02-09T22:50:30.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa61c0a2150 0x7fa61c0a4610 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa634101d00 0x7fa6341a0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa634101340 0x7fa634199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 --2- 172.21.15.3:0/1932344161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa634100150 0x7fa6341994e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 >> 172.21.15.3:0/1932344161 conn(0x7fa6340fb6c0 msgr2=0x7fa6340fd160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 shutdown_connections 2024-02-09T22:50:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:30.936+0000 7fa638dd7640 1 -- 172.21.15.3:0/1932344161 wait complete. 2024-02-09T22:50:30.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 -- 172.21.15.3:0/2875272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 msgr2=0x7ff0a8100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/2875272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a8100570 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff098009ca0 tx=0x7ff09802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 -- 172.21.15.3:0/2875272165 shutdown_connections 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/2875272165 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0a8101340 0x7ff0a81017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/2875272165 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0a8101d00 0x7ff0a8108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/2875272165 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a8100570 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 -- 172.21.15.3:0/2875272165 >> 172.21.15.3:0/2875272165 conn(0x7ff0a80fb6c0 msgr2=0x7ff0a80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 -- 172.21.15.3:0/2875272165 shutdown_connections 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.050+0000 7ff0b06f4640 1 -- 172.21.15.3:0/2875272165 wait complete. 2024-02-09T22:50:31.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 Processor -- start 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 -- start start 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a80690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0a8101340 0x7ff0a819c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0a8101d00 0x7ff0a819e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a819ee30 con 0x7ff0a8100150 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a819efa0 con 0x7ff0a8101340 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0b06f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a819f160 con 0x7ff0a8101d00 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a80690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a80690d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39016/0 (socket says 172.21.15.3:39016) 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 -- 172.21.15.3:0/786880475 learned_addr learned my addr 172.21.15.3:0/786880475 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0a8101340 0x7ff0a819c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0aec91640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0a8101d00 0x7ff0a819e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 -- 172.21.15.3:0/786880475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0a8101d00 msgr2=0x7ff0a819e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0a8101d00 0x7ff0a819e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 -- 172.21.15.3:0/786880475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0a8101340 msgr2=0x7ff0a819c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0a8101340 0x7ff0a819c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 -- 172.21.15.3:0/786880475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff098009900 con 0x7ff0a8100150 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.051+0000 7ff0ae490640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a80690d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff098037040 tx=0x7ff098034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.052+0000 7ff0a77fe640 1 -- 172.21.15.3:0/786880475 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098032eb0 con 0x7ff0a8100150 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.052+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a819f3e0 con 0x7ff0a8100150 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.052+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a819f8e0 con 0x7ff0a8100150 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.052+0000 7ff0a77fe640 1 -- 172.21.15.3:0/786880475 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff098003e50 con 0x7ff0a8100150 2024-02-09T22:50:31.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.052+0000 7ff0a77fe640 1 -- 172.21.15.3:0/786880475 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff098040470 con 0x7ff0a8100150 2024-02-09T22:50:31.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0a77fe640 1 -- 172.21.15.3:0/786880475 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7ff09800a040 con 0x7ff0a8100150 2024-02-09T22:50:31.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 Processor -- start 2024-02-09T22:50:31.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 -- start start 2024-02-09T22:50:31.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf932f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf93341b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf9336cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf929eb30 con 0x560cf932f6b0 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf929c060 con 0x560cf932ecc0 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0b339c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf9297840 con 0x560cf9334820 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0ae490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf93341b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0a77fe640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0900a2150 0x7ff0900a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0ae490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf93341b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39032/0 (socket says 172.21.15.3:39032) 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.054+0000 7ff0ae490640 1 -- 172.21.15.3:0/708984938 learned_addr learned my addr 172.21.15.3:0/708984938 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0aec91640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0900a2150 0x7ff0900a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0aec91640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf932f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf9336cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0a77fe640 1 -- 172.21.15.3:0/786880475 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff0980f08d0 con 0x7ff0a8100150 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0ae490640 1 -- 172.21.15.3:0/708984938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 msgr2=0x560cf9336cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0ae490640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf9336cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0ae490640 1 -- 172.21.15.3:0/708984938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 msgr2=0x560cf932f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0ae490640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf932f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0ae490640 1 -- 172.21.15.3:0/708984938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cf925d620 con 0x560cf932f6b0 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0aec91640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf932f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf9336cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0ae490640 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf93341b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff0980b9f90 tx=0x7ff0980095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cca5cd07ffdbbeb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0aec91640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0900a2150 0x7ff0900a4610 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff0980ee3d0 tx=0x7ff0a0005e50 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0a4ff9640 1 -- 172.21.15.3:0/708984938 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980bcf20 con 0x560cf932f6b0 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0a4ff9640 1 -- 172.21.15.3:0/708984938 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0980bdcc0 con 0x560cf932f6b0 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.055+0000 7ff0a4ff9640 1 -- 172.21.15.3:0/708984938 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0980ee600 con 0x560cf932f6b0 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 -- 172.21.15.3:0/708984938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 msgr2=0x560cf93341b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf93341b0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7ff0980b9f90 tx=0x7ff0980095c0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 -- 172.21.15.3:0/708984938 shutdown_connections 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf9336cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf93341b0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 --2- 172.21.15.3:0/708984938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf932f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 -- 172.21.15.3:0/708984938 >> 172.21.15.3:0/708984938 conn(0x560cf932a230 msgr2=0x560cf932a630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 -- 172.21.15.3:0/708984938 shutdown_connections 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 -- 172.21.15.3:0/708984938 wait complete. 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 -- start start 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.056+0000 7ff0b339c740 1 Processor -- start 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf9338940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf9338e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf933fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf923ccd0 con 0x560cf932f6b0 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf8f6b310 con 0x560cf932ecc0 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf921d490 con 0x560cf9334820 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 10 client.0 _ll_register_callbacks cb 0x560cf8fa8ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf9338e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf9338e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39042/0 (socket says 172.21.15.3:39042) 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 -- 172.21.15.3:0/4241889509 learned_addr learned my addr 172.21.15.3:0/4241889509 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0aec91640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf933fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0ae490640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf9338940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 -- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 msgr2=0x560cf933fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf933fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 -- 172.21.15.3:0/4241889509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 msgr2=0x560cf9338940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf9338940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 -- 172.21.15.3:0/4241889509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a819f8e0 con 0x560cf932f6b0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf9338e80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00bf30 tx=0x7ff09c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d608fb97f75aedf9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff08dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff08dffb640 1 -- 172.21.15.3:0/4241889509 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff09c004220 con 0x560cf932f6b0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff08dffb640 1 -- 172.21.15.3:0/4241889509 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff09c004380 con 0x560cf932f6b0 2024-02-09T22:50:31.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.057+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560cf8f976c0 con 0x560cf932f6b0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff08dffb640 1 -- 172.21.15.3:0/4241889509 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff09c0106a0 con 0x560cf932f6b0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 10 client.15102 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560cf9295ec0 con 0x560cf932f6b0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff08dffb640 1 -- 172.21.15.3:0/4241889509 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560cf8f976c0 con 0x560cf932f6b0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff08dffb640 1 -- 172.21.15.3:0/4241889509 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff09c003b40 con 0x560cf932f6b0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 10 client.15102 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 20 client.15102 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf9344cf0 0x560cf93445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 4 client.15102 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cf8fe8270 con 0x560cf9344cf0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.058+0000 7ff0ae490640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf9344cf0 0x560cf93445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.059+0000 7ff08dffb640 10 client.15102 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:31.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.059+0000 7ff0ae490640 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf9344cf0 0x560cf93445a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2d3c8f3390c97116 server_cookie=ea63cfec5443a0c7 in_seq=0 out_seq=0 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff08dffb640 1 -- 172.21.15.3:0/4241889509 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560cf8fe8270 con 0x560cf9344cf0 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff08dffb640 10 client.15102 handle_command_reply: tid=0 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 client.15102 shutdown 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 msgr2=0x560cf9338e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf9338e80 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff09c00bf30 tx=0x7ff09c0128e0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 shutdown_connections 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf9344cf0 0x560cf93445a0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf9334820 0x560cf933fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff08dffb640 0 client.15102 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf932f6b0 0x560cf9338e80 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 --2- 172.21.15.3:0/4241889509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932ecc0 0x560cf9338940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 >> 172.21.15.3:0/4241889509 conn(0x560cf932a230 msgr2=0x560cf9329510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 shutdown_connections 2024-02-09T22:50:31.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 1 -- 172.21.15.3:0/4241889509 wait complete. 2024-02-09T22:50:31.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.060+0000 7ff0b339c740 20 client.15102 trim_cache size 0 max 16384 2024-02-09T22:50:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 Processor -- start 2024-02-09T22:50:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 -- start start 2024-02-09T22:50:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf9328c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf93291c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf932fda0 con 0x560cf9334820 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf9338770 con 0x560cf932f6b0 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0b339c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf9290540 con 0x560cf932ecc0 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf93291c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf93291c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39050/0 (socket says 172.21.15.3:39050) 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 -- 172.21.15.3:0/3929028505 learned_addr learned my addr 172.21.15.3:0/3929028505 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0ae490640 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf9328c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 -- 172.21.15.3:0/3929028505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 msgr2=0x560cf9328c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf9328c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 -- 172.21.15.3:0/3929028505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 msgr2=0x560cf933fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 -- 172.21.15.3:0/3929028505 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cf9232a90 con 0x560cf9334820 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.076+0000 7ff0aec91640 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf93291c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000be60 tx=0x7ff0a0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb78f8f898633710 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff08ffff640 1 -- 172.21.15.3:0/3929028505 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a0028680 con 0x560cf9334820 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff08ffff640 1 -- 172.21.15.3:0/3929028505 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0a0028cc0 con 0x560cf9334820 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff08ffff640 1 -- 172.21.15.3:0/3929028505 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a001fbc0 con 0x560cf9334820 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 -- 172.21.15.3:0/3929028505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 msgr2=0x560cf93291c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf93291c0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000be60 tx=0x7ff0a0012000 comp rx=0 tx=0).stop 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 -- 172.21.15.3:0/3929028505 shutdown_connections 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf93291c0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 --2- 172.21.15.3:0/3929028505 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf9328c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 -- 172.21.15.3:0/3929028505 >> 172.21.15.3:0/3929028505 conn(0x560cf932a230 msgr2=0x560cf9338940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 -- 172.21.15.3:0/3929028505 shutdown_connections 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.077+0000 7ff0b339c740 1 -- 172.21.15.3:0/3929028505 wait complete. 2024-02-09T22:50:31.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 -- start start 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 Processor -- start 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf933d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf933e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf933eed0 con 0x560cf9334820 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf933f040 con 0x560cf932f6b0 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cf933f200 con 0x560cf932ecc0 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0b339c740 10 client.0 _ll_register_callbacks cb 0x7ff0980bd390 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0aec91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf933e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0aec91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf933e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39062/0 (socket says 172.21.15.3:39062) 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0aec91640 1 -- 172.21.15.3:0/2362049241 learned_addr learned my addr 172.21.15.3:0/2362049241 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0ae490640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf933d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.078+0000 7ff0aec91640 1 -- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 msgr2=0x560cf933d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0aec91640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf933d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0aec91640 1 -- 172.21.15.3:0/2362049241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 msgr2=0x560cf933d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0aec91640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0aec91640 1 -- 172.21.15.3:0/2362049241 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a0018070 con 0x560cf9334820 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0adc8f640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0aec91640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf933e990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff0a001f700 tx=0x7ff0a000e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e827c056f2edde6b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff08dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff08dffb640 1 -- 172.21.15.3:0/2362049241 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a001fc90 con 0x560cf9334820 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff08dffb640 1 -- 172.21.15.3:0/2362049241 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff0a001fdf0 con 0x560cf9334820 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff0980bdcc0 con 0x560cf9334820 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff08dffb640 1 -- 172.21.15.3:0/2362049241 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0a0028480 con 0x560cf9334820 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0b339c740 10 client.15108 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x560cf9258930 con 0x560cf9334820 2024-02-09T22:50:31.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff08dffb640 1 -- 172.21.15.3:0/2362049241 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0980bdcc0 con 0x560cf9334820 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff08dffb640 1 -- 172.21.15.3:0/2362049241 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0a0011bd0 con 0x560cf9334820 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0b339c740 10 client.15108 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0b339c740 10 client.15108 resolve_mds: resolved name 'i' to daemon mds.i 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.079+0000 7ff0b339c740 20 client.15108 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0b339c740 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf939f760 0x560cf939d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0b339c740 4 client.15108 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "status"}) v1 -- 0x560cf8fd4610 con 0x560cf939f760 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0ae490640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf939f760 0x560cf939d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0ae490640 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf939f760 0x560cf939d3c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1ecb10add03c3d89 server_cookie=650f38df80827f03 in_seq=0 out_seq=0 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff08dffb640 10 client.15108 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff08dffb640 1 -- 172.21.15.3:0/2362049241 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x560cf8fd4610 con 0x560cf939f760 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff08dffb640 10 client.15108 handle_command_reply: tid=0 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0b339c740 1 client.15108 shutdown 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 msgr2=0x560cf933e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.080+0000 7ff0b339c740 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf933e990 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff0a001f700 tx=0x7ff0a000e9b0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 shutdown_connections 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x560cf939f760 0x560cf939d3c0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560cf932f6b0 0x560cf933d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560cf9334820 0x560cf933e990 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 --2- 172.21.15.3:0/2362049241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560cf932ecc0 0x560cf933d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 >> 172.21.15.3:0/2362049241 conn(0x560cf932a230 msgr2=0x560cf932d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff08dffb640 0 client.15108 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 shutdown_connections 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 1 -- 172.21.15.3:0/2362049241 wait complete. 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b339c740 20 client.15108 trim_cache size 0 max 16384 2024-02-09T22:50:31.090 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 2, 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24496, 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 80.982223946000005, 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 98.059582367999994 2024-02-09T22:50:31.091 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:31.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0900a2150 msgr2=0x7ff0900a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0900a2150 0x7ff0900a4610 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7ff0980ee3d0 tx=0x7ff0a0005e50 comp rx=0 tx=0).stop 2024-02-09T22:50:31.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 msgr2=0x7ff0a80690d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a80690d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff098037040 tx=0x7ff098034d40 comp rx=0 tx=0).stop 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 shutdown_connections 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0900a2150 0x7ff0900a4610 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0a8101340 0x7ff0a819c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0a8101d00 0x7ff0a819e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 --2- 172.21.15.3:0/786880475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0a8100150 0x7ff0a80690d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 >> 172.21.15.3:0/786880475 conn(0x7ff0a80fb6c0 msgr2=0x7ff0a8104760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 shutdown_connections 2024-02-09T22:50:31.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.081+0000 7ff0b06f4640 1 -- 172.21.15.3:0/786880475 wait complete. 2024-02-09T22:50:31.095 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-02-09T22:50:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 -- 172.21.15.3:0/3927425047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc101d90 msgr2=0x7f09cc104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 --2- 172.21.15.3:0/3927425047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc101d90 0x7f09cc104180 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009900 tx=0x7f09c802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 -- 172.21.15.3:0/3927425047 shutdown_connections 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 --2- 172.21.15.3:0/3927425047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc106ff0 0x7f09cc1093e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 --2- 172.21.15.3:0/3927425047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09cc1046c0 0x7f09cc106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 --2- 172.21.15.3:0/3927425047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc101d90 0x7f09cc104180 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 -- 172.21.15.3:0/3927425047 >> 172.21.15.3:0/3927425047 conn(0x7f09cc0fb6c0 msgr2=0x7f09cc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 -- 172.21.15.3:0/3927425047 shutdown_connections 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.195+0000 7f09d44a7640 1 -- 172.21.15.3:0/3927425047 wait complete. 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 Processor -- start 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 -- start start 2024-02-09T22:50:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09cc101d90 0x7f09cc1972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc1046c0 0x7f09cc197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 0x7f09cc19e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc19edc0 con 0x7f09cc106ff0 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc19ef30 con 0x7f09cc1046c0 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d44a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09cc19f0f0 con 0x7f09cc101d90 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 0x7f09cc19e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 0x7f09cc19e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39080/0 (socket says 172.21.15.3:39080) 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 -- 172.21.15.3:0/4274428642 learned_addr learned my addr 172.21.15.3:0/4274428642 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2243640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09cc101d90 0x7f09cc1972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d1a42640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc1046c0 0x7f09cc197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 -- 172.21.15.3:0/4274428642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09cc101d90 msgr2=0x7f09cc1972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09cc101d90 0x7f09cc1972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 -- 172.21.15.3:0/4274428642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc1046c0 msgr2=0x7f09cc197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc1046c0 0x7f09cc197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d2a44640 1 -- 172.21.15.3:0/4274428642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09c8009560 con 0x7f09cc106ff0 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.196+0000 7f09d1a42640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc1046c0 0x7f09cc197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.197+0000 7f09d2a44640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 0x7f09cc19e880 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f09c400eb10 tx=0x7f09c400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.197+0000 7f09c37fe640 1 -- 172.21.15.3:0/4274428642 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4004b20 con 0x7f09cc106ff0 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.197+0000 7f09c37fe640 1 -- 172.21.15.3:0/4274428642 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09c4004c80 con 0x7f09cc106ff0 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.197+0000 7f09c37fe640 1 -- 172.21.15.3:0/4274428642 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c4010650 con 0x7f09cc106ff0 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.197+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09cc19f3d0 con 0x7f09cc106ff0 2024-02-09T22:50:31.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.197+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc19f8e0 con 0x7f09cc106ff0 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.199+0000 7f09d7084740 1 Processor -- start 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09c37fe640 1 -- 172.21.15.3:0/4274428642 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f09c4005020 con 0x7f09cc106ff0 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 -- start start 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb5820 0x55d8fcdb7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcd1fb30 con 0x55d8fcdb06b0 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcd1d060 con 0x55d8fcdafcc0 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d7084740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcd18840 con 0x55d8fcdb5820 2024-02-09T22:50:31.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d2243640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d2243640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb51b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39096/0 (socket says 172.21.15.3:39096) 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d2243640 1 -- 172.21.15.3:0/1391074096 learned_addr learned my addr 172.21.15.3:0/1391074096 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.200+0000 7f09d1a42640 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09c37fe640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09b80a21d0 0x7f09b80a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d1a42640 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb5820 0x55d8fcdb7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2243640 1 -- 172.21.15.3:0/1391074096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb5820 msgr2=0x55d8fcdb7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2243640 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb5820 0x55d8fcdb7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2243640 1 -- 172.21.15.3:0/1391074096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 msgr2=0x55d8fcdb00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2243640 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2243640 1 -- 172.21.15.3:0/1391074096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8fccde620 con 0x55d8fcdb06b0 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2a44640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09b80a21d0 0x7f09b80a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09c37fe640 1 -- 172.21.15.3:0/4274428642 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f09c4014070 con 0x7f09cc106ff0 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2243640 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb51b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009230 tx=0x7f09c802c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddffc1840c1d611 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09c0bfe640 1 -- 172.21.15.3:0/1391074096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c802cc10 con 0x55d8fcdb06b0 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09d2a44640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09b80a21d0 0x7f09b80a4690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f09c409afb0 tx=0x7f09c406aed0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09c0bfe640 1 -- 172.21.15.3:0/1391074096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09c8004040 con 0x55d8fcdb06b0 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.201+0000 7f09c0bfe640 1 -- 172.21.15.3:0/1391074096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c8005330 con 0x55d8fcdb06b0 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 -- 172.21.15.3:0/1391074096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 msgr2=0x55d8fcdb51b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb51b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f09c8009230 tx=0x7f09c802c960 comp rx=0 tx=0).stop 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 -- 172.21.15.3:0/1391074096 shutdown_connections 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb5820 0x55d8fcdb7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb51b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 --2- 172.21.15.3:0/1391074096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 -- 172.21.15.3:0/1391074096 >> 172.21.15.3:0/1391074096 conn(0x55d8fcdab230 msgr2=0x55d8fcdab630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 -- 172.21.15.3:0/1391074096 shutdown_connections 2024-02-09T22:50:31.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 -- 172.21.15.3:0/1391074096 wait complete. 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 -- start start 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.202+0000 7f09d7084740 1 Processor -- start 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdba030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdb5820 0x55d8fcdc1070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fccbdcd0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fc9ec310 con 0x55d8fcdb5820 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcc9e490 con 0x55d8fcdb06b0 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 10 client.0 _ll_register_callbacks cb 0x55d8fca29ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb9af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39108/0 (socket says 172.21.15.3:39108) 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 -- 172.21.15.3:0/2659505215 learned_addr learned my addr 172.21.15.3:0/2659505215 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdb5820 0x55d8fcdc1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d2243640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdba030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 -- 172.21.15.3:0/2659505215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 msgr2=0x55d8fcdba030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdba030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 -- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdb5820 msgr2=0x55d8fcdc1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdb5820 0x55d8fcdc1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 -- 172.21.15.3:0/2659505215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09cc109f70 con 0x55d8fcdafcc0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d1a42640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb9af0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f09bc013b40 tx=0x7f09bc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee30ec286300c3ab server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09b5ffb640 1 -- 172.21.15.3:0/2659505215 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc01f3f0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09b5ffb640 1 -- 172.21.15.3:0/2659505215 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f09bc01f550 con 0x55d8fcdafcc0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.203+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8fca186c0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09b5ffb640 1 -- 172.21.15.3:0/2659505215 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09bc01f710 con 0x55d8fcdafcc0 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 10 client.15120 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:31.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d8fcd16ec0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09b5ffb640 1 -- 172.21.15.3:0/2659505215 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d8fca186c0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09b5ffb640 1 -- 172.21.15.3:0/2659505215 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f09bc028070 con 0x55d8fcdafcc0 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 10 client.15120 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 10 client.15120 resolve_mds: resolved name 'h' to daemon mds.h 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 20 client.15120 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdc6540 0x55d8fcdc57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 4 client.15120 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d8fca69270 con 0x55d8fcdc6540 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.204+0000 7f09d2243640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdc6540 0x55d8fcdc57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.205+0000 7f09b5ffb640 10 client.15120 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.205+0000 7f09d2243640 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdc6540 0x55d8fcdc57a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ff8e0c350f52c03d server_cookie=a5d06cc295e9aeff in_seq=0 out_seq=0 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.205+0000 7f09b5ffb640 1 -- 172.21.15.3:0/2659505215 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d8fca69270 con 0x55d8fcdc6540 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.205+0000 7f09b5ffb640 10 client.15120 handle_command_reply: tid=0 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.205+0000 7f09d7084740 1 client.15120 shutdown 2024-02-09T22:50:31.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 msgr2=0x55d8fcdb9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb9af0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f09bc013b40 tx=0x7f09bc004730 comp rx=0 tx=0).stop 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 shutdown_connections 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdc6540 0x55d8fcdc57a0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdb5820 0x55d8fcdc1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdba030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 --2- 172.21.15.3:0/2659505215 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdb9af0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09b5ffb640 0 client.15120 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 >> 172.21.15.3:0/2659505215 conn(0x55d8fcdab230 msgr2=0x55d8fcdaa510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 shutdown_connections 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 1 -- 172.21.15.3:0/2659505215 wait complete. 2024-02-09T22:50:31.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.206+0000 7f09d7084740 20 client.15120 trim_cache size 0 max 16384 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 Processor -- start 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 -- start start 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 0x55d8fcdb9700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdc1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcdaf880 con 0x55d8fcdab230 2024-02-09T22:50:31.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcda6af0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d7084740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fca69270 con 0x55d8fcdb06b0 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 0x55d8fcdb9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 0x55d8fcdb9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39112/0 (socket says 172.21.15.3:39112) 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 -- 172.21.15.3:0/1128215502 learned_addr learned my addr 172.21.15.3:0/1128215502 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d2243640 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdc1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 -- 172.21.15.3:0/1128215502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 msgr2=0x55d8fcdb9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 -- 172.21.15.3:0/1128215502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 msgr2=0x55d8fcdc1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdc1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d1a42640 1 -- 172.21.15.3:0/1128215502 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d8fccb3a90 con 0x55d8fcdab230 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.222+0000 7f09d2243640 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdc1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.223+0000 7f09d1a42640 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 0x55d8fcdb9700 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0137a0 tx=0x7f09bc00df40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76827416831ed3cf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.223+0000 7f09b7fff640 1 -- 172.21.15.3:0/1128215502 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc00fb00 con 0x55d8fcdab230 2024-02-09T22:50:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.223+0000 7f09b7fff640 1 -- 172.21.15.3:0/1128215502 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09bc025070 con 0x55d8fcdab230 2024-02-09T22:50:31.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.223+0000 7f09b7fff640 1 -- 172.21.15.3:0/1128215502 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09bc004c60 con 0x55d8fcdab230 2024-02-09T22:50:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 -- 172.21.15.3:0/1128215502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 msgr2=0x55d8fcdb9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 0x55d8fcdb9700 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f09bc0137a0 tx=0x7f09bc00df40 comp rx=0 tx=0).stop 2024-02-09T22:50:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 -- 172.21.15.3:0/1128215502 shutdown_connections 2024-02-09T22:50:31.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdc1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdb9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 --2- 172.21.15.3:0/1128215502 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdab230 0x55d8fcdb9700 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 -- 172.21.15.3:0/1128215502 >> 172.21.15.3:0/1128215502 conn(0x55d8fcdab980 msgr2=0x55d8fcdce9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 -- 172.21.15.3:0/1128215502 shutdown_connections 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 -- 172.21.15.3:0/1128215502 wait complete. 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 -- start start 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.224+0000 7f09d7084740 1 Processor -- start 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdab230 0x55d8fcdbe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdbe9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdbfa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcdbff40 con 0x55d8fcdb06b0 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcdc00b0 con 0x55d8fcdab230 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d8fcdc0270 con 0x55d8fcdafcc0 2024-02-09T22:50:31.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d7084740 10 client.0 _ll_register_callbacks cb 0x7f09c8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d1a42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdbfa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d1a42640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdbfa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39120/0 (socket says 172.21.15.3:39120) 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d1a42640 1 -- 172.21.15.3:0/3494654237 learned_addr learned my addr 172.21.15.3:0/3494654237 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdbe9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d1a42640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdab230 0x55d8fcdbe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 -- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdab230 msgr2=0x55d8fcdbe4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdab230 0x55d8fcdbe4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 -- 172.21.15.3:0/3494654237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 msgr2=0x55d8fcdbfa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdbfa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 -- 172.21.15.3:0/3494654237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d8fcd16ec0 con 0x55d8fcdafcc0 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d1a42640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdbfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09d2243640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdbe9f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55d8fcdac1a0 tx=0x7f09c802c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92ba8e3660b5614e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.225+0000 7f09b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09b5ffb640 1 -- 172.21.15.3:0/3494654237 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09c805a070 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d8fcdc6250 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09b5ffb640 1 -- 172.21.15.3:0/3494654237 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f09c804f040 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09b5ffb640 1 -- 172.21.15.3:0/3494654237 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f09c802cb50 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 10 client.24844 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55d8fccd9930 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09b5ffb640 1 -- 172.21.15.3:0/3494654237 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09c8065020 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09b5ffb640 1 -- 172.21.15.3:0/3494654237 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f09c805d020 con 0x55d8fcdafcc0 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 10 client.24844 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 10 client.24844 resolve_mds: resolved name 'h' to daemon mds.h 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 20 client.24844 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdb5820 0x55d8fce1e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 4 client.24844 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d8fcdc7570 con 0x55d8fcdb5820 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.226+0000 7f09d1a42640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdb5820 0x55d8fce1e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d1a42640 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdb5820 0x55d8fce1e150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82fca2f9246ed8e1 server_cookie=d5eec392851bebc7 in_seq=0 out_seq=0 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09b5ffb640 10 client.24844 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09b5ffb640 1 -- 172.21.15.3:0/3494654237 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55d8fcdc7570 con 0x55d8fcdb5820 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09b5ffb640 10 client.24844 handle_command_reply: tid=0 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 client.24844 shutdown 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdafcc0 msgr2=0x55d8fcdbe9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdbe9f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x55d8fcdac1a0 tx=0x7f09c802c8e0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 shutdown_connections 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55d8fcdb5820 0x55d8fce1e150 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d8fcdb06b0 0x55d8fcdbfa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d8fcdafcc0 0x55d8fcdbe9f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 --2- 172.21.15.3:0/3494654237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d8fcdab230 0x55d8fcdbe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09b5ffb640 0 client.24844 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 >> 172.21.15.3:0/3494654237 conn(0x55d8fcdab980 msgr2=0x55d8fcdb9700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 shutdown_connections 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.227+0000 7f09d7084740 1 -- 172.21.15.3:0/3494654237 wait complete. 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d7084740 20 client.24844 trim_cache size 0 max 16384 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 1, 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24503, 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:31.237 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 82.143498953999995, 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 100.795482757 2024-02-09T22:50:31.238 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09b80a21d0 msgr2=0x7f09b80a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09b80a21d0 0x7f09b80a4690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f09c409afb0 tx=0x7f09c406aed0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 msgr2=0x7f09cc19e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 0x7f09cc19e880 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f09c400eb10 tx=0x7f09c400c2d0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 shutdown_connections 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f09b80a21d0 0x7f09b80a4690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f09cc1046c0 0x7f09cc197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f09cc106ff0 0x7f09cc19e880 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 --2- 172.21.15.3:0/4274428642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f09cc101d90 0x7f09cc1972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 >> 172.21.15.3:0/4274428642 conn(0x7f09cc0fb6c0 msgr2=0x7f09cc1026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 shutdown_connections 2024-02-09T22:50:31.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.228+0000 7f09d44a7640 1 -- 172.21.15.3:0/4274428642 wait complete. 2024-02-09T22:50:31.245 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-09T22:50:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 -- 172.21.15.3:0/922207074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8100150 msgr2=0x7f4af8100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 --2- 172.21.15.3:0/922207074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8100150 0x7f4af8100570 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009900 tx=0x7f4af402bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 -- 172.21.15.3:0/922207074 shutdown_connections 2024-02-09T22:50:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 --2- 172.21.15.3:0/922207074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4af8101d00 0x7f4af8108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 --2- 172.21.15.3:0/922207074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101340 0x7f4af81017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 --2- 172.21.15.3:0/922207074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8100150 0x7f4af8100570 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 -- 172.21.15.3:0/922207074 >> 172.21.15.3:0/922207074 conn(0x7f4af80fb6c0 msgr2=0x7f4af80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 -- 172.21.15.3:0/922207074 shutdown_connections 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.347+0000 7f4b00348640 1 -- 172.21.15.3:0/922207074 wait complete. 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 Processor -- start 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 -- start start 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4af8100150 0x7f4af80690d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 0x7f4af819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101d00 0x7f4af819e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af819ee10 con 0x7f4af8101340 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af819ef80 con 0x7f4af8100150 2024-02-09T22:50:31.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4b00348640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4af819f140 con 0x7f4af8101d00 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 0x7f4af819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 0x7f4af819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39148/0 (socket says 172.21.15.3:39148) 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 -- 172.21.15.3:0/1078642895 learned_addr learned my addr 172.21.15.3:0/1078642895 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4af8100150 0x7f4af80690d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101d00 0x7f4af819e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 -- 172.21.15.3:0/1078642895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101d00 msgr2=0x7f4af819e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101d00 0x7f4af819e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 -- 172.21.15.3:0/1078642895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4af8100150 msgr2=0x7f4af80690d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4af8100150 0x7f4af80690d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afd8e3640 1 -- 172.21.15.3:0/1078642895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af4009560 con 0x7f4af8101340 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.348+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101d00 0x7f4af819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:31.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.349+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 0x7f4af819c010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800b4a0 tx=0x7f4ae800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.349+0000 7f4aef7fe640 1 -- 172.21.15.3:0/1078642895 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8010610 con 0x7f4af8101340 2024-02-09T22:50:31.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.349+0000 7f4aef7fe640 1 -- 172.21.15.3:0/1078642895 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ae8010c50 con 0x7f4af8101340 2024-02-09T22:50:31.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.349+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4af819f420 con 0x7f4af8101340 2024-02-09T22:50:31.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.349+0000 7f4aef7fe640 1 -- 172.21.15.3:0/1078642895 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ae8017e20 con 0x7f4af8101340 2024-02-09T22:50:31.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.349+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af819fa00 con 0x7f4af8101340 2024-02-09T22:50:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.351+0000 7f4b02ff2740 1 Processor -- start 2024-02-09T22:50:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.351+0000 7f4b02ff2740 1 -- start start 2024-02-09T22:50:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 0x564951e590e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e5e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e5e820 0x564951e60cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951dc8b30 con 0x564951e5e820 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951dc6060 con 0x564951e596b0 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951dc1840 con 0x564951e58cc0 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 0x564951e590e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 0x564951e590e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57622/0 (socket says 172.21.15.3:57622) 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 -- 172.21.15.3:0/2301704645 learned_addr learned my addr 172.21.15.3:0/2301704645 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e5e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4aef7fe640 1 -- 172.21.15.3:0/1078642895 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f4ae8017480 con 0x7f4af8101340 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 -- 172.21.15.3:0/2301704645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 msgr2=0x564951e5e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e5e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 -- 172.21.15.3:0/2301704645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e5e820 msgr2=0x564951e60cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e5e820 0x564951e60cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe8e5640 1 -- 172.21.15.3:0/2301704645 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564951d87620 con 0x564951e58cc0 2024-02-09T22:50:31.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.352+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e5e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 0x564951e590e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4af0008e60 tx=0x7f4af000d710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de63782daf43bbe7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4aef7fe640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ae40a21d0 0x7f4ae40a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4aecbfe640 1 -- 172.21.15.3:0/2301704645 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af000f830 con 0x564951e58cc0 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4aecbfe640 1 -- 172.21.15.3:0/2301704645 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4af000fe70 con 0x564951e58cc0 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4aecbfe640 1 -- 172.21.15.3:0/2301704645 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af000e3b0 con 0x564951e58cc0 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ae40a21d0 0x7f4ae40a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.353+0000 7f4aef7fe640 1 -- 172.21.15.3:0/1078642895 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4ae80cd300 con 0x7f4af8101340 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2301704645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 msgr2=0x564951e590e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ae40a21d0 0x7f4ae40a4690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009230 tx=0x7f4af4005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 0x564951e590e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4af0008e60 tx=0x7f4af000d710 comp rx=0 tx=0).stop 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2301704645 shutdown_connections 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e5e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e5e820 0x564951e60cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2301704645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e58cc0 0x564951e590e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2301704645 >> 172.21.15.3:0/2301704645 conn(0x564951e54230 msgr2=0x564951e54630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2301704645 shutdown_connections 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2301704645 wait complete. 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- start start 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 Processor -- start 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e62c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e6a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951d66cd0 con 0x564951e58cc0 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951a95310 con 0x564951e596b0 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951d47490 con 0x564951e5e820 2024-02-09T22:50:31.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4b02ff2740 10 client.0 _ll_register_callbacks cb 0x564951ad2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4afe0e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e62c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4afe0e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e62c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39172/0 (socket says 172.21.15.3:39172) 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.354+0000 7f4afe0e4640 1 -- 172.21.15.3:0/3374493603 learned_addr learned my addr 172.21.15.3:0/3374493603 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e6a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe0e4640 1 -- 172.21.15.3:0/3374493603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 msgr2=0x564951e6a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e6a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe0e4640 1 -- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 msgr2=0x564951e631a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e631a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe0e4640 1 -- 172.21.15.3:0/3374493603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4af0008a90 con 0x564951e58cc0 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e62c60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009900 tx=0x7f4af402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40f1b8f67786a26 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4ae1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/3374493603 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af40052c0 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/3374493603 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4af4005420 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564951ac16c0 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/3374493603 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4af40055e0 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4b02ff2740 10 client.15135 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564951dbfec0 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.355+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/3374493603 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564951ac16c0 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/3374493603 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4af405e070 con 0x564951e58cc0 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4b02ff2740 10 client.15135 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4b02ff2740 10 client.15135 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4b02ff2740 20 client.15135 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951e6f630 0x564951e6e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4b02ff2740 4 client.15135 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564951b12270 con 0x564951e6f630 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951e6f630 0x564951e6e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4ae1ffb640 10 client.15135 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.356+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951e6f630 0x564951e6e8b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8f92afb59079811 server_cookie=a4d0cce6fd382606 in_seq=0 out_seq=0 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/3374493603 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564951b12270 con 0x564951e6f630 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4ae1ffb640 10 client.15135 handle_command_reply: tid=0 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 client.15135 shutdown 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 msgr2=0x564951e62c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e62c60 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009900 tx=0x7f4af402bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 shutdown_connections 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951e6f630 0x564951e6e8b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e631a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4ae1ffb640 0 client.15135 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e6a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/3374493603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e62c60 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 >> 172.21.15.3:0/3374493603 conn(0x564951e54230 msgr2=0x564951e53510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 shutdown_connections 2024-02-09T22:50:31.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 1 -- 172.21.15.3:0/3374493603 wait complete. 2024-02-09T22:50:31.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.357+0000 7f4b02ff2740 20 client.15135 trim_cache size 0 max 16384 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 Processor -- start 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 -- start start 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e63100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e54610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e52c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951e71610 con 0x564951e58cc0 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951e6d460 con 0x564951e596b0 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951b12270 con 0x564951e5e820 2024-02-09T22:50:31.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e63100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e63100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39184/0 (socket says 172.21.15.3:39184) 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 -- 172.21.15.3:0/1778621127 learned_addr learned my addr 172.21.15.3:0/1778621127 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e52c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e54610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 -- 172.21.15.3:0/1778621127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 msgr2=0x564951e52c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e52c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 -- 172.21.15.3:0/1778621127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 msgr2=0x564951e54610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e54610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.373+0000 7f4afe0e4640 1 -- 172.21.15.3:0/1778621127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564951e61d60 con 0x564951e58cc0 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e52c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e54610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e63100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4af4000c00 tx=0x7f4af402cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=218640c7e3f575da server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4ae3fff640 1 -- 172.21.15.3:0/1778621127 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af404cb80 con 0x564951e58cc0 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4ae3fff640 1 -- 172.21.15.3:0/1778621127 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4af405b070 con 0x564951e58cc0 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4ae3fff640 1 -- 172.21.15.3:0/1778621127 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af4003770 con 0x564951e58cc0 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 -- 172.21.15.3:0/1778621127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 msgr2=0x564951e63100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e63100 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f4af4000c00 tx=0x7f4af402cc10 comp rx=0 tx=0).stop 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 -- 172.21.15.3:0/1778621127 shutdown_connections 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e596b0 0x564951e54610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e5e820 0x564951e52c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/1778621127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951e63100 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 -- 172.21.15.3:0/1778621127 >> 172.21.15.3:0/1778621127 conn(0x564951e54230 msgr2=0x564951e5df80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.374+0000 7f4b02ff2740 1 -- 172.21.15.3:0/1778621127 shutdown_connections 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 -- 172.21.15.3:0/1778621127 wait complete. 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 -- start start 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 Processor -- start 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951ec1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e596b0 0x564951ec20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e5e820 0x564951ec9090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951ec95d0 con 0x564951e58cc0 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951ec9710 con 0x564951e5e820 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564951ec9880 con 0x564951e596b0 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4b02ff2740 10 client.0 _ll_register_callbacks cb 0x7f4af0008600 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951ec1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951ec1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39200/0 (socket says 172.21.15.3:39200) 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 -- 172.21.15.3:0/2520447645 learned_addr learned my addr 172.21.15.3:0/2520447645 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe8e5640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e5e820 0x564951ec9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e596b0 0x564951ec20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 -- 172.21.15.3:0/2520447645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e596b0 msgr2=0x564951ec20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e596b0 0x564951ec20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 -- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e5e820 msgr2=0x564951ec9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e5e820 0x564951ec9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 -- 172.21.15.3:0/2520447645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564951dbfec0 con 0x564951e58cc0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.375+0000 7f4afe0e4640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951ec1b80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4af4004c50 tx=0x7f4af4005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ac63a832d6171e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4ae1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/2520447645 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4af404c510 con 0x564951e58cc0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/2520447645 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4af40036f0 con 0x564951e58cc0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564951e70000 con 0x564951e58cc0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/2520447645 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4af40038b0 con 0x564951e58cc0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 10 client.15141 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564951d5ca90 con 0x564951e58cc0 2024-02-09T22:50:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/2520447645 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564951e70000 con 0x564951e58cc0 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/2520447645 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4af405e040 con 0x564951e58cc0 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 10 client.15141 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 10 client.15141 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 20 client.15141 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951ece610 0x564951ecaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 4 client.15141 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4af000fe70 con 0x564951ece610 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.376+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951ece610 0x564951ecaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4ae1ffb640 10 client.15141 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:31.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4afd8e3640 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951ece610 0x564951ecaae0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd191ef3041c1a66 server_cookie=1731b1f8a26ff4c6 in_seq=0 out_seq=0 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4ae1ffb640 1 -- 172.21.15.3:0/2520447645 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x7f4af000fe70 con 0x564951ece610 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4ae1ffb640 10 client.15141 handle_command_reply: tid=0 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 client.15141 shutdown 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 msgr2=0x564951ec1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951ec1b80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4af4004c50 tx=0x7f4af4005740 comp rx=0 tx=0).stop 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 shutdown_connections 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564951ece610 0x564951ecaae0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564951e5e820 0x564951ec9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564951e596b0 0x564951ec20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4ae1ffb640 0 client.15141 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 --2- 172.21.15.3:0/2520447645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564951e58cc0 0x564951ec1b80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 >> 172.21.15.3:0/2520447645 conn(0x564951e54230 msgr2=0x564951e62850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 shutdown_connections 2024-02-09T22:50:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.377+0000 7f4b02ff2740 1 -- 172.21.15.3:0/2520447645 wait complete. 2024-02-09T22:50:31.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b02ff2740 20 client.15141 trim_cache size 0 max 16384 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24533, 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 83.839634636, 2024-02-09T22:50:31.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:31.393 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:31.393 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:31.393 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 89.725020611999994 2024-02-09T22:50:31.393 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ae40a21d0 msgr2=0x7f4ae40a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ae40a21d0 0x7f4ae40a4690 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009230 tx=0x7f4af4005fd0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 msgr2=0x7f4af819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 0x7f4af819c010 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4ae800b4a0 tx=0x7f4ae800b960 comp rx=0 tx=0).stop 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 shutdown_connections 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4ae40a21d0 0x7f4ae40a4690 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4af8101d00 0x7f4af819e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4af8101340 0x7f4af819c010 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 --2- 172.21.15.3:0/1078642895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4af8100150 0x7f4af80690d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 >> 172.21.15.3:0/1078642895 conn(0x7f4af80fb6c0 msgr2=0x7f4af81046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 shutdown_connections 2024-02-09T22:50:31.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:31.378+0000 7f4b00348640 1 -- 172.21.15.3:0/1078642895 wait complete. 2024-02-09T22:50:31.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:31 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3397642539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:31.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:31 smithi003 ceph-mon[32678]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-02-09T22:50:31.395 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:31 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1073462450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:31.396 INFO:tasks.cephadm:Setup complete, yielding 2024-02-09T22:50:31.396 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T22:50:31.406 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T22:50:31.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph orch status' 2024-02-09T22:50:31.651 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:31 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3397642539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:31 smithi064 ceph-mon[36638]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-02-09T22:50:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:31 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1073462450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:31 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3397642539' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:31 smithi089 ceph-mon[36665]: pgmap v330: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-02-09T22:50:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:31 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1073462450' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:32 smithi003 ceph-mon[32678]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-02-09T22:50:32.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:32 smithi064 ceph-mon[36638]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-02-09T22:50:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:32 smithi089 ceph-mon[36665]: pgmap v331: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.444+0000 7f9ccbf47700 1 -- 172.21.15.3:0/1966685728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 msgr2=0x7f9cc41033b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.444+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/1966685728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41033b0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f9cb40099c0 tx=0x7f9cb4009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- 172.21.15.3:0/1966685728 shutdown_connections 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/1966685728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 0x7f9cc4102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/1966685728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9cc41038f0 0x7f9cc410a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/1966685728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41033b0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- 172.21.15.3:0/1966685728 >> 172.21.15.3:0/1966685728 conn(0x7f9cc40fd330 msgr2=0x7f9cc40ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- 172.21.15.3:0/1966685728 shutdown_connections 2024-02-09T22:50:33.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- 172.21.15.3:0/1966685728 wait complete. 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 Processor -- start 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- start start 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 0x7f9cc4197d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9cc41038f0 0x7f9cc419f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc419f8f0 con 0x7f9cc4102f50 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc419fa30 con 0x7f9cc41038f0 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.445+0000 7f9ccbf47700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc419fbc0 con 0x7f9cc4101d50 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39228/0 (socket says 172.21.15.3:39228) 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 -- 172.21.15.3:0/2337809931 learned_addr learned my addr 172.21.15.3:0/2337809931 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc9ce3700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 0x7f9cc4197d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cca4e4700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9cc41038f0 0x7f9cc419f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:33.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 -- 172.21.15.3:0/2337809931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 msgr2=0x7f9cc4197d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 0x7f9cc4197d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 -- 172.21.15.3:0/2337809931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9cc41038f0 msgr2=0x7f9cc419f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9cc41038f0 0x7f9cc419f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 -- 172.21.15.3:0/2337809931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40095f0 con 0x7f9cc4102f50 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc9ce3700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 0x7f9cc4197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cc94e2700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41982a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4005910 tx=0x7f9cb4005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb401d070 con 0x7f9cc4102f50 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cb4022470 con 0x7f9cc4102f50 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc419fe40 con 0x7f9cc4102f50 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb400f6c0 con 0x7f9cc4102f50 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.446+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc41a0250 con 0x7f9cc4102f50 2024-02-09T22:50:33.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.447+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc40689e0 con 0x7f9cc4102f50 2024-02-09T22:50:33.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.448+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7f9cb40225e0 con 0x7f9cc4102f50 2024-02-09T22:50:33.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.449+0000 7f9cbaffd700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9cb00a21e0 0x7f9cb00a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:33.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.449+0000 7f9cc9ce3700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9cb00a21e0 0x7f9cb00a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:33.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.449+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9cb40cf9d0 con 0x7f9cc4102f50 2024-02-09T22:50:33.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.449+0000 7f9cc9ce3700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9cb00a21e0 0x7f9cb00a46a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4102db0 tx=0x7f9cc0005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:33.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.452+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9cb4098880 con 0x7f9cc4102f50 2024-02-09T22:50:33.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.673+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f9cc402d920 con 0x7f9cb00a21e0 2024-02-09T22:50:33.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.674+0000 7f9cbaffd700 1 -- 172.21.15.3:0/2337809931 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f9cc402d920 con 0x7f9cb00a21e0 2024-02-09T22:50:33.677 INFO:teuthology.orchestra.run.smithi003.stdout:Backend: cephadm 2024-02-09T22:50:33.677 INFO:teuthology.orchestra.run.smithi003.stdout:Available: Yes 2024-02-09T22:50:33.677 INFO:teuthology.orchestra.run.smithi003.stdout:Paused: No 2024-02-09T22:50:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9cb00a21e0 msgr2=0x7f9cb00a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9cb00a21e0 0x7f9cb00a46a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4102db0 tx=0x7f9cc0005f90 comp rx=0 tx=0).stop 2024-02-09T22:50:33.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 msgr2=0x7f9cc41982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41982a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9cb4005910 tx=0x7f9cb4005e80 comp rx=0 tx=0).stop 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 shutdown_connections 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9cb00a21e0 0x7f9cb00a46a0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9cc4101d50 0x7f9cc4197d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9cc41038f0 0x7f9cc419f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 --2- 172.21.15.3:0/2337809931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9cc4102f50 0x7f9cc41982a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 >> 172.21.15.3:0/2337809931 conn(0x7f9cc40fd330 msgr2=0x7f9cc4106320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 shutdown_connections 2024-02-09T22:50:33.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:33.676+0000 7f9ccbf47700 1 -- 172.21.15.3:0/2337809931 wait complete. 2024-02-09T22:50:34.314 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ps' 2024-02-09T22:50:34.558 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:35.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:35 smithi003 ceph-mon[32678]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:35.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:35 smithi003 ceph-mon[32678]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:35 smithi064 ceph-mon[36638]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:35.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:35 smithi064 ceph-mon[36638]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:35 smithi089 ceph-mon[36665]: from='client.15147 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:35.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:35 smithi089 ceph-mon[36665]: pgmap v332: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.244+0000 7fc130456700 1 -- 172.21.15.3:0/472881989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128103930 msgr2=0x7fc12810a1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.244+0000 7fc130456700 1 --2- 172.21.15.3:0/472881989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128103930 0x7fc12810a1e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc12000b0d0 tx=0x7fc12000b3e0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 -- 172.21.15.3:0/472881989 shutdown_connections 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 --2- 172.21.15.3:0/472881989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc128101d90 0x7fc1281021b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 --2- 172.21.15.3:0/472881989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128103930 0x7fc12810a1e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 --2- 172.21.15.3:0/472881989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc128102f90 0x7fc1281033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 -- 172.21.15.3:0/472881989 >> 172.21.15.3:0/472881989 conn(0x7fc1280fd370 msgr2=0x7fc1280ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 -- 172.21.15.3:0/472881989 shutdown_connections 2024-02-09T22:50:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 -- 172.21.15.3:0/472881989 wait complete. 2024-02-09T22:50:36.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 Processor -- start 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.245+0000 7fc130456700 1 -- start start 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc128101d90 0x7fc128197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 0x7fc128198300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc128103930 0x7fc12819f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12819f950 con 0x7fc128102f90 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12819fa90 con 0x7fc128103930 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12819fc20 con 0x7fc128101d90 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 0x7fc128198300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 0x7fc128198300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39234/0 (socket says 172.21.15.3:39234) 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 -- 172.21.15.3:0/1795226153 learned_addr learned my addr 172.21.15.3:0/1795226153 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12e1f2700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc128101d90 0x7fc128197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:36.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12e9f3700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc128103930 0x7fc12819f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 -- 172.21.15.3:0/1795226153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc128101d90 msgr2=0x7fc128197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc128101d90 0x7fc128197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 -- 172.21.15.3:0/1795226153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc128103930 msgr2=0x7fc12819f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc128103930 0x7fc12819f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 -- 172.21.15.3:0/1795226153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc120009d00 con 0x7fc128102f90 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc12d9f1700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 0x7fc128198300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc11800c950 tx=0x7fc11800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11800cea0 con 0x7fc128102f90 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc118004830 con 0x7fc128102f90 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12819ff00 con 0x7fc128102f90 2024-02-09T22:50:36.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1180054f0 con 0x7fc128102f90 2024-02-09T22:50:36.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.246+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1281a0370 con 0x7fc128102f90 2024-02-09T22:50:36.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.247+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1281090f0 con 0x7fc128102f90 2024-02-09T22:50:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.248+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 17) v1 ==== 136818+0+0 (secure 0 0 0) 0x7fc1180049a0 con 0x7fc128102f90 2024-02-09T22:50:36.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.249+0000 7fc11f7fe700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc1140a21e0 0x7fc1140a46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.249+0000 7fc12e1f2700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc1140a21e0 0x7fc1140a46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.249+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc1180cdd30 con 0x7fc128102f90 2024-02-09T22:50:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.250+0000 7fc12e1f2700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc1140a21e0 0x7fc1140a46a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc128102df0 tx=0x7fc12400a380 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.253+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc118096bb0 con 0x7fc128102f90 2024-02-09T22:50:36.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.465+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc12802d960 con 0x7fc1140a21e0 2024-02-09T22:50:36.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.479+0000 7fc11f7fe700 1 -- 172.21.15.3:0/1795226153 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7fc12802d960 con 0x7fc1140a21e0 2024-02-09T22:50:36.482 INFO:teuthology.orchestra.run.smithi003.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-02-09T22:50:36.482 INFO:teuthology.orchestra.run.smithi003.stdout:mds.a smithi003 running (98s) 15s ago 97s 18.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 6792ec0ee949 2024-02-09T22:50:36.482 INFO:teuthology.orchestra.run.smithi003.stdout:mds.b smithi064 running (95s) 28s ago 94s 26.2M - 19.0.0-1361-g290212a1 c328ab9dcfc7 85309a727c76 2024-02-09T22:50:36.482 INFO:teuthology.orchestra.run.smithi003.stdout:mds.c smithi089 running (109s) 28s ago 108s 19.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ba0092832da0 2024-02-09T22:50:36.482 INFO:teuthology.orchestra.run.smithi003.stdout:mds.d smithi003 running (117s) 15s ago 116s 13.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 a32f1d446916 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.e smithi064 running (114s) 28s ago 114s 16.8M - 19.0.0-1361-g290212a1 c328ab9dcfc7 d178c1235776 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.f smithi089 running (111s) 28s ago 111s 16.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 60bbd2e371b3 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.g smithi003 running (100s) 15s ago 100s 17.1M - 19.0.0-1361-g290212a1 c328ab9dcfc7 1ddc91dd56b1 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.h smithi064 running (106s) 28s ago 105s 17.5M - 19.0.0-1361-g290212a1 c328ab9dcfc7 be2ffb82557b 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.i smithi089 running (103s) 28s ago 103s 17.4M - 19.0.0-1361-g290212a1 c328ab9dcfc7 d2f4556cecd8 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.j smithi003 running (2m) 15s ago 119s 14.0M - 19.0.0-1361-g290212a1 c328ab9dcfc7 b486efb1188f 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.k smithi064 running (2m) 28s ago 2m 16.9M - 19.0.0-1361-g290212a1 c328ab9dcfc7 661c32850638 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mds.l smithi089 running (2m) 28s ago 2m 18.2M - 19.0.0-1361-g290212a1 c328ab9dcfc7 d6648ce98dd6 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.x smithi003 *:9283,8765 running (9m) 15s ago 9m 513M - 19.0.0-1361-g290212a1 c328ab9dcfc7 0ae12abd5b18 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.y smithi064 *:8443,8765 running (7m) 28s ago 7m 453M - 19.0.0-1361-g290212a1 c328ab9dcfc7 ea86b2fc3cf5 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mgr.z smithi089 *:8443,8765 running (7m) 28s ago 7m 450M - 19.0.0-1361-g290212a1 c328ab9dcfc7 bfdaf36675bf 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mon.a smithi003 running (9m) 15s ago 9m 52.7M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 1d1b447cf16c 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mon.b smithi064 running (7m) 28s ago 7m 39.1M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 aea879413355 2024-02-09T22:50:36.483 INFO:teuthology.orchestra.run.smithi003.stdout:mon.c smithi089 running (7m) 28s ago 7m 43.4M 2048M 19.0.0-1361-g290212a1 c328ab9dcfc7 c1741322896a 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 smithi003 running (6m) 15s ago 6m 51.0M 4273M 19.0.0-1361-g290212a1 c328ab9dcfc7 4e68f37a564f 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 smithi064 running (6m) 28s ago 6m 49.0M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 28c3e92bda39 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 smithi089 running (6m) 28s ago 6m 49.2M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 6360825648cb 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 smithi003 running (5m) 15s ago 5m 57.0M 4273M 19.0.0-1361-g290212a1 c328ab9dcfc7 7b8ce729ef9a 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 smithi064 running (5m) 28s ago 5m 52.6M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 ddec16151a00 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 smithi089 running (4m) 28s ago 4m 56.5M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 f98239c4bbf7 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 smithi003 running (4m) 15s ago 4m 45.0M 4273M 19.0.0-1361-g290212a1 c328ab9dcfc7 0f90e2b1d7e3 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 smithi064 running (4m) 28s ago 4m 46.3M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 cf4ff0c8317f 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8 smithi089 running (3m) 28s ago 3m 45.3M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 1f0102b68da6 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9 smithi003 running (3m) 15s ago 3m 45.5M 4273M 19.0.0-1361-g290212a1 c328ab9dcfc7 196d67ecb6db 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10 smithi064 running (2m) 28s ago 2m 52.7M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 5ba7c6fb049a 2024-02-09T22:50:36.484 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11 smithi089 running (2m) 28s ago 2m 44.3M 4270M 19.0.0-1361-g290212a1 c328ab9dcfc7 d92859fb52e2 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc1140a21e0 msgr2=0x7fc1140a46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc1140a21e0 0x7fc1140a46a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc128102df0 tx=0x7fc12400a380 comp rx=0 tx=0).stop 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 msgr2=0x7fc128198300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 0x7fc128198300 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc11800c950 tx=0x7fc11800cc60 comp rx=0 tx=0).stop 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 shutdown_connections 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc1140a21e0 0x7fc1140a46a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc128101d90 0x7fc128197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc128103930 0x7fc12819f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 --2- 172.21.15.3:0/1795226153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc128102f90 0x7fc128198300 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 >> 172.21.15.3:0/1795226153 conn(0x7fc1280fd370 msgr2=0x7fc128106230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 shutdown_connections 2024-02-09T22:50:36.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:36.481+0000 7fc130456700 1 -- 172.21.15.3:0/1795226153 wait complete. 2024-02-09T22:50:36.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:36 smithi003 ceph-mon[32678]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:36.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:36 smithi003 ceph-mon[32678]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:36.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:36 smithi064 ceph-mon[36638]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:36.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:36 smithi064 ceph-mon[36638]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:36.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:36 smithi089 ceph-mon[36665]: from='client.15153 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:36.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:36 smithi089 ceph-mon[36665]: pgmap v333: 97 pgs: 97 active+clean; 641 KiB data, 346 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:37.114 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph orch ls' 2024-02-09T22:50:37.358 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:38.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:38 smithi003 ceph-mon[32678]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-02-09T22:50:38.711 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:38 smithi003 ceph-mon[32678]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-02-09T22:50:38.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:38 smithi064 ceph-mon[36638]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-02-09T22:50:38.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:38 smithi064 ceph-mon[36638]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 -- 172.21.15.3:0/1320782673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d41038f0 msgr2=0x7fa5d410a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1320782673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d41038f0 0x7fa5d410a1a0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00b0d0 tx=0x7fa5cc00b3e0 comp rx=0 tx=0).stop 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 -- 172.21.15.3:0/1320782673 shutdown_connections 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1320782673 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d4101d50 0x7fa5d4102170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1320782673 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d41038f0 0x7fa5d410a1a0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1320782673 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4102f50 0x7fa5d41033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.916+0000 7fa5db29a700 1 -- 172.21.15.3:0/1320782673 >> 172.21.15.3:0/1320782673 conn(0x7fa5d40fd330 msgr2=0x7fa5d40ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 -- 172.21.15.3:0/1320782673 shutdown_connections 2024-02-09T22:50:38.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 -- 172.21.15.3:0/1320782673 wait complete. 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 Processor -- start 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 -- start start 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 0x7fa5d4075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4102f50 0x7fa5d4074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d41038f0 0x7fa5d4074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d4074dd0 con 0x7fa5d4101d50 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d4074f40 con 0x7fa5d4102f50 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.917+0000 7fa5db29a700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d41a1e50 con 0x7fa5d41038f0 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 0x7fa5d4075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 0x7fa5d4075ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35010/0 (socket says 172.21.15.3:35010) 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 -- 172.21.15.3:0/1545043717 learned_addr learned my addr 172.21.15.3:0/1545043717 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d8835700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4102f50 0x7fa5d4074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9837700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d41038f0 0x7fa5d4074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:38.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 -- 172.21.15.3:0/1545043717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d41038f0 msgr2=0x7fa5d4074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d41038f0 0x7fa5d4074650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 -- 172.21.15.3:0/1545043717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4102f50 msgr2=0x7fa5d4074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4102f50 0x7fa5d4074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 -- 172.21.15.3:0/1545043717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5cc009d00 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5d9036700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 0x7fa5d4075ac0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400c970 tx=0x7fa5c400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4004220 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5c4007d80 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d41a2130 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4003d00 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.918+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d41a2630 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.919+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d41090b0 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.920+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa5c40078a0 con 0x7fa5d4101d50 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.921+0000 7fa5ca7fc700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5c00a2150 0x7fa5c00a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.921+0000 7fa5d8835700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5c00a2150 0x7fa5c00a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:38.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.921+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa5c40cdc00 con 0x7fa5d4101d50 2024-02-09T22:50:38.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.921+0000 7fa5d8835700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5c00a2150 0x7fa5c00a4610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0005950 tx=0x7fa5d000b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:38.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:38.924+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa5c4096be0 con 0x7fa5d4101d50 2024-02-09T22:50:38.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:38 smithi089 ceph-mon[36665]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-02-09T22:50:38.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:38 smithi089 ceph-mon[36665]: pgmap v334: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 9 op/s 2024-02-09T22:50:39.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.137+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5d402d8e0 con 0x7fa5c00a2150 2024-02-09T22:50:39.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.143+0000 7fa5ca7fc700 1 -- 172.21.15.3:0/1545043717 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7fa5d402d8e0 con 0x7fa5c00a2150 2024-02-09T22:50:39.146 INFO:teuthology.orchestra.run.smithi003.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-02-09T22:50:39.147 INFO:teuthology.orchestra.run.smithi003.stdout:mds.all 12/12 31s ago 2m smithi003=a;smithi003=d;smithi003=g;smithi003=j;smithi064=b;smithi064=e;smithi064=h;smithi064=k;smithi089=c;smithi089=f;smithi089=i;smithi089=l;count:12 2024-02-09T22:50:39.147 INFO:teuthology.orchestra.run.smithi003.stdout:mgr 3/3 31s ago 7m smithi003=x;smithi064=y;smithi089=z;count:3 2024-02-09T22:50:39.147 INFO:teuthology.orchestra.run.smithi003.stdout:mon 3/3 31s ago 7m smithi003:172.21.15.3=a;smithi064:172.21.15.64=b;smithi089:172.21.15.89=c;count:3 2024-02-09T22:50:39.147 INFO:teuthology.orchestra.run.smithi003.stdout:osd 12 31s ago - 2024-02-09T22:50:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5c00a2150 msgr2=0x7fa5c00a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5c00a2150 0x7fa5c00a4610 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa5d0005950 tx=0x7fa5d000b410 comp rx=0 tx=0).stop 2024-02-09T22:50:39.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 msgr2=0x7fa5d4075ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 0x7fa5d4075ac0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fa5c400c970 tx=0x7fa5c400cc80 comp rx=0 tx=0).stop 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 shutdown_connections 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa5c00a2150 0x7fa5c00a4610 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5d41038f0 0x7fa5d4074650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5d4101d50 0x7fa5d4075ac0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 --2- 172.21.15.3:0/1545043717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5d4102f50 0x7fa5d4074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.145+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 >> 172.21.15.3:0/1545043717 conn(0x7fa5d40fd330 msgr2=0x7fa5d41061f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.146+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 shutdown_connections 2024-02-09T22:50:39.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:39.146+0000 7fa5db29a700 1 -- 172.21.15.3:0/1545043717 wait complete. 2024-02-09T22:50:39.823 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph orch host ls' 2024-02-09T22:50:40.072 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:40.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:40 smithi003 ceph-mon[32678]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:40.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:40 smithi064 ceph-mon[36638]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:40 smithi089 ceph-mon[36665]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:41.374 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:41 smithi003 ceph-mon[32678]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-02-09T22:50:41.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:41 smithi064 ceph-mon[36638]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-02-09T22:50:41.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:41 smithi089 ceph-mon[36665]: pgmap v335: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 85 B/s wr, 9 op/s 2024-02-09T22:50:41.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 -- 172.21.15.3:0/1814566240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c98101d50 msgr2=0x7f9c98102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:41.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/1814566240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c98101d50 0x7f9c98102170 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9c880099c0 tx=0x7f9c88009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 -- 172.21.15.3:0/1814566240 shutdown_connections 2024-02-09T22:50:41.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/1814566240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c981038f0 0x7f9c9810a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/1814566240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c98101d50 0x7f9c98102170 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/1814566240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98102f50 0x7f9c981033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 -- 172.21.15.3:0/1814566240 >> 172.21.15.3:0/1814566240 conn(0x7f9c980fd330 msgr2=0x7f9c980ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 -- 172.21.15.3:0/1814566240 shutdown_connections 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.754+0000 7f9c9da8f700 1 -- 172.21.15.3:0/1814566240 wait complete. 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 Processor -- start 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 -- start start 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98101d50 0x7f9c98197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c98102f50 0x7f9c981981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 0x7f9c9819f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9819f810 con 0x7f9c981038f0 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9819f950 con 0x7f9c98101d50 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c9da8f700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9819fb10 con 0x7f9c98102f50 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c97fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 0x7f9c9819f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c97fff700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 0x7f9c9819f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35030/0 (socket says 172.21.15.3:35030) 2024-02-09T22:50:41.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.755+0000 7f9c97fff700 1 -- 172.21.15.3:0/3545267621 learned_addr learned my addr 172.21.15.3:0/3545267621 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c977fe700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98101d50 0x7f9c98197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c96ffd700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c98102f50 0x7f9c981981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c97fff700 1 -- 172.21.15.3:0/3545267621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c98102f50 msgr2=0x7f9c981981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c97fff700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c98102f50 0x7f9c981981d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c97fff700 1 -- 172.21.15.3:0/3545267621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98101d50 msgr2=0x7f9c98197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c97fff700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98101d50 0x7f9c98197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c97fff700 1 -- 172.21.15.3:0/3545267621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c880095f0 con 0x7f9c981038f0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c977fe700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98101d50 0x7f9c98197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c97fff700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 0x7f9c9819f240 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9c90007d70 tx=0x7f9c9000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000ad40 con 0x7f9c981038f0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c90012900 con 0x7f9c981038f0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9819fdf0 con 0x7f9c981038f0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90011880 con 0x7f9c981038f0 2024-02-09T22:50:41.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.756+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c981a0210 con 0x7f9c981038f0 2024-02-09T22:50:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.757+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c98109110 con 0x7f9c981038f0 2024-02-09T22:50:41.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.758+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f9c90012a70 con 0x7f9c981038f0 2024-02-09T22:50:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.759+0000 7f9c94ff9700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9c800a2150 0x7f9c800a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.759+0000 7f9c977fe700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9c800a2150 0x7f9c800a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.759+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9c900cdfa0 con 0x7f9c981038f0 2024-02-09T22:50:41.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.759+0000 7f9c977fe700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9c800a2150 0x7f9c800a4610 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9c880038c0 tx=0x7f9c88005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:41.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.763+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9c90101d20 con 0x7f9c981038f0 2024-02-09T22:50:41.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.973+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c9802d8e0 con 0x7f9c800a2150 2024-02-09T22:50:41.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.974+0000 7f9c94ff9700 1 -- 172.21.15.3:0/3545267621 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+187 (secure 0 0 0) 0x7f9c9802d8e0 con 0x7f9c800a2150 2024-02-09T22:50:41.977 INFO:teuthology.orchestra.run.smithi003.stdout:HOST ADDR LABELS STATUS 2024-02-09T22:50:41.978 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 172.21.15.3 2024-02-09T22:50:41.978 INFO:teuthology.orchestra.run.smithi003.stdout:smithi064 172.21.15.64 2024-02-09T22:50:41.978 INFO:teuthology.orchestra.run.smithi003.stdout:smithi089 172.21.15.89 2024-02-09T22:50:41.978 INFO:teuthology.orchestra.run.smithi003.stdout:3 hosts in cluster 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9c800a2150 msgr2=0x7f9c800a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9c800a2150 0x7f9c800a4610 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9c880038c0 tx=0x7f9c88005fb0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 msgr2=0x7f9c9819f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 0x7f9c9819f240 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9c90007d70 tx=0x7f9c9000a340 comp rx=0 tx=0).stop 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 shutdown_connections 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f9c800a2150 0x7f9c800a4610 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c981038f0 0x7f9c9819f240 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c98101d50 0x7f9c98197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 --2- 172.21.15.3:0/3545267621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c98102f50 0x7f9c981981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:41.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 >> 172.21.15.3:0/3545267621 conn(0x7f9c980fd330 msgr2=0x7f9c98106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:41.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 shutdown_connections 2024-02-09T22:50:41.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:41.977+0000 7f9c9da8f700 1 -- 172.21.15.3:0/3545267621 wait complete. 2024-02-09T22:50:42.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph orch device ls' 2024-02-09T22:50:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:42 smithi003 ceph-mon[32678]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:42 smithi003 ceph-mon[32678]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:50:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:42 smithi064 ceph-mon[36638]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:42.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:42 smithi064 ceph-mon[36638]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:50:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:42 smithi089 ceph-mon[36665]: from='client.15165 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:42 smithi089 ceph-mon[36665]: pgmap v336: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:50:44.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 -- 172.21.15.3:0/1090527932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb750107ae0 msgr2=0x7fb750109ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:44.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 --2- 172.21.15.3:0/1090527932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb750107ae0 0x7fb750109ed0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fb74800b0a0 tx=0x7fb74800b3b0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 -- 172.21.15.3:0/1090527932 shutdown_connections 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 --2- 172.21.15.3:0/1090527932 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb750107ae0 0x7fb750109ed0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 --2- 172.21.15.3:0/1090527932 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb750102880 0x7fb750104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 --2- 172.21.15.3:0/1090527932 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb7501051b0 0x7fb7501075a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.550+0000 7fb757a6b700 1 -- 172.21.15.3:0/1090527932 >> 172.21.15.3:0/1090527932 conn(0x7fb7500fc1f0 msgr2=0x7fb7500fe630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 -- 172.21.15.3:0/1090527932 shutdown_connections 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 -- 172.21.15.3:0/1090527932 wait complete. 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 Processor -- start 2024-02-09T22:50:44.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 -- start start 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb750102880 0x7fb750197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 0x7fb7501983b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb750107ae0 0x7fb75019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75019fa00 con 0x7fb7501051b0 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75019fb40 con 0x7fb750107ae0 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb757a6b700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb75019fcd0 con 0x7fb750102880 2024-02-09T22:50:44.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb755006700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 0x7fb7501983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.551+0000 7fb755006700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 0x7fb7501983b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35056/0 (socket says 172.21.15.3:35056) 2024-02-09T22:50:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 -- 172.21.15.3:0/332274393 learned_addr learned my addr 172.21.15.3:0/332274393 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb756008700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb750107ae0 0x7fb75019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755807700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb750102880 0x7fb750197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:44.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 -- 172.21.15.3:0/332274393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb750102880 msgr2=0x7fb750197e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:44.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb750102880 0x7fb750197e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 -- 172.21.15.3:0/332274393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb750107ae0 msgr2=0x7fb75019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:44.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb750107ae0 0x7fb75019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 -- 172.21.15.3:0/332274393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb748009d00 con 0x7fb7501051b0 2024-02-09T22:50:44.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb755006700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 0x7fb7501983b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d980 tx=0x7fb74000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:44.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb740009b40 con 0x7fb7501051b0 2024-02-09T22:50:44.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb75019ffb0 con 0x7fb7501051b0 2024-02-09T22:50:44.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb74000b6e0 con 0x7fb7501051b0 2024-02-09T22:50:44.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb74000f620 con 0x7fb7501051b0 2024-02-09T22:50:44.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.552+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7501a04f0 con 0x7fb7501051b0 2024-02-09T22:50:44.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.553+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7500689e0 con 0x7fb7501051b0 2024-02-09T22:50:44.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.554+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb740010460 con 0x7fb7501051b0 2024-02-09T22:50:44.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.555+0000 7fb746ffd700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb73c0a21d0 0x7fb73c0a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:44.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.555+0000 7fb755807700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb73c0a21d0 0x7fb73c0a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:44.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.555+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb7400cdd80 con 0x7fb7501051b0 2024-02-09T22:50:44.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.555+0000 7fb755807700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb73c0a21d0 0x7fb73c0a4690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006fd0 tx=0x7fb74c008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:44.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.559+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb740096d60 con 0x7fb7501051b0 2024-02-09T22:50:44.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.770+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb75002d920 con 0x7fb73c0a21d0 2024-02-09T22:50:44.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.772+0000 7fb746ffd700 1 -- 172.21.15.3:0/332274393 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7fb75002d920 con 0x7fb73c0a21d0 2024-02-09T22:50:44.776 INFO:teuthology.orchestra.run.smithi003.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-02-09T22:50:44.776 INFO:teuthology.orchestra.run.smithi003.stdout:smithi003 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT5330009Z400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-02-09T22:50:44.776 INFO:teuthology.orchestra.run.smithi003.stdout:smithi064 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT534400C2400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-09T22:50:44.776 INFO:teuthology.orchestra.run.smithi003.stdout:smithi089 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040126400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb73c0a21d0 msgr2=0x7fb73c0a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb73c0a21d0 0x7fb73c0a4690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb74c006fd0 tx=0x7fb74c008040 comp rx=0 tx=0).stop 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 msgr2=0x7fb7501983b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 0x7fb7501983b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fb74000d980 tx=0x7fb74000dc90 comp rx=0 tx=0).stop 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 shutdown_connections 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb73c0a21d0 0x7fb73c0a4690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb750102880 0x7fb750197e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb750107ae0 0x7fb75019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 --2- 172.21.15.3:0/332274393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb7501051b0 0x7fb7501983b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 >> 172.21.15.3:0/332274393 conn(0x7fb7500fc1f0 msgr2=0x7fb7501031c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:44.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 shutdown_connections 2024-02-09T22:50:44.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:44.774+0000 7fb757a6b700 1 -- 172.21.15.3:0/332274393 wait complete. 2024-02-09T22:50:45.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:45 smithi003 ceph-mon[32678]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:50:45.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:45 smithi064 ceph-mon[36638]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:50:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:45 smithi089 ceph-mon[36665]: pgmap v337: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 9 op/s 2024-02-09T22:50:45.610 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-02-09T22:50:45.620 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T22:50:45.621 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph fs dump' 2024-02-09T22:50:45.870 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:46.333 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:46 smithi003 ceph-mon[32678]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:46.379 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:46 smithi089 ceph-mon[36665]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:46.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:46 smithi064 ceph-mon[36638]: from='client.15171 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:50:47.325 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:47 smithi003 ceph-mon[32678]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:50:47.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:47 smithi064 ceph-mon[36638]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:50:47.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:47 smithi089 ceph-mon[36665]: pgmap v338: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 426 B/s wr, 11 op/s 2024-02-09T22:50:47.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/522062473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8103910 msgr2=0x7f4db810a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:47.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/522062473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8103910 0x7f4db810a1c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f4db000b0d0 tx=0x7f4db000b3e0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/522062473 shutdown_connections 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/522062473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8101d70 0x7f4db8102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/522062473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8103910 0x7f4db810a1c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/522062473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4db8102f70 0x7f4db81033d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/522062473 >> 172.21.15.3:0/522062473 conn(0x7f4db80fd330 msgr2=0x7f4db80ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/522062473 shutdown_connections 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.605+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/522062473 wait complete. 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 Processor -- start 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 -- start start 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 0x7f4db8075ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8102f70 0x7f4db8074110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:47.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4db8103910 0x7f4db8074650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8074dd0 con 0x7f4db8101d70 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db8074f40 con 0x7f4db8103910 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbfdb2700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db81a1e60 con 0x7f4db8102f70 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbdb4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 0x7f4db8075ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbdb4e700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 0x7f4db8075ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41652/0 (socket says 172.21.15.3:41652) 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbdb4e700 1 -- 172.21.15.3:0/363145521 learned_addr learned my addr 172.21.15.3:0/363145521 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbd34d700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8102f70 0x7f4db8074110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.606+0000 7f4dbe34f700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4db8103910 0x7f4db8074650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbdb4e700 1 -- 172.21.15.3:0/363145521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8102f70 msgr2=0x7f4db8074110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbdb4e700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8102f70 0x7f4db8074110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbdb4e700 1 -- 172.21.15.3:0/363145521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4db8103910 msgr2=0x7f4db8074650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:47.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbdb4e700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4db8103910 0x7f4db8074650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbdb4e700 1 -- 172.21.15.3:0/363145521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db0009d00 con 0x7f4db8101d70 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbdb4e700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 0x7f4db8075ac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4da800d980 tx=0x7f4da800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8004020 con 0x7f4db8101d70 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db81a2140 con 0x7f4db8101d70 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4da800f690 con 0x7f4db8101d70 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4da8003b40 con 0x7f4db8101d70 2024-02-09T22:50:47.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.607+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db81a2690 con 0x7f4db8101d70 2024-02-09T22:50:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.608+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db8109060 con 0x7f4db8101d70 2024-02-09T22:50:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.609+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f4da8003ca0 con 0x7f4db8101d70 2024-02-09T22:50:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.609+0000 7f4daeffd700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4da40a2150 0x7f4da40a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.610+0000 7f4dbd34d700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4da40a2150 0x7f4da40a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.610+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f4da80cd860 con 0x7f4db8101d70 2024-02-09T22:50:47.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.610+0000 7f4dbd34d700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4da40a2150 0x7f4da40a4610 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4db4005950 tx=0x7f4db400b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:47.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.614+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f4da8096840 con 0x7f4db8101d70 2024-02-09T22:50:47.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.860+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f4db80689e0 con 0x7f4db8101d70 2024-02-09T22:50:47.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.860+0000 7f4daeffd700 1 -- 172.21.15.3:0/363145521 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) v1 ==== 76+0+2949 (secure 0 0 0) 0x7f4da8099f00 con 0x7f4db8101d70 2024-02-09T22:50:47.864 INFO:teuthology.orchestra.run.smithi003.stdout:e24 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:legacy client fscid: 1 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:Filesystem 'cephfs' (1) 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:fs_name cephfs 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 24 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-02-09T22:49:06.610270+0000 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2024-02-09T22:49:13.115988+0000 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:tableserver 0 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:root 0 2024-02-09T22:50:47.865 INFO:teuthology.orchestra.run.smithi003.stdout:session_timeout 300 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:session_autoclose 300 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:max_file_size 1099511627776 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:max_xattr_size 65536 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:required_client_features {} 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:last_failure 0 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:last_failure_osd_epoch 0 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:max_mds 5 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:in 0,1,2,3,4 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:up {0=24533,1=24503,2=24496,3=24481,4=24475} 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:failed 2024-02-09T22:50:47.866 INFO:teuthology.orchestra.run.smithi003.stdout:damaged 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:stopped 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:data_pools [3] 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:metadata_pool 2 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:inline_data disabled 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:balancer 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:bal_rank_mask -1 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:standby_count_wanted 0 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.b{0:24533} state up:active seq 3 addr [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.e{0:24470} state up:standby-replay seq 1 addr [v2:172.21.15.64:6834/3082808593,v1:172.21.15.64:6835/3082808593] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.h{1:24503} state up:active seq 6 addr [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.k{1:24437} state up:standby-replay seq 1 addr [v2:172.21.15.64:6832/1710343231,v1:172.21.15.64:6833/1710343231] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.867 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.i{2:24496} state up:active seq 6 addr [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.l{2:24427} state up:standby-replay seq 1 addr [v2:172.21.15.89:6832/2685783987,v1:172.21.15.89:6833/2685783987] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.c{3:24481} state up:active seq 7 addr [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.a{3:14646} state up:standby-replay seq 1 addr [v2:172.21.15.3:6840/3916323155,v1:172.21.15.3:6841/3916323155] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.f{4:24475} state up:active seq 8 addr [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.g{4:14634} state up:standby-replay seq 1 addr [v2:172.21.15.3:6838/3695104270,v1:172.21.15.3:6839/3695104270] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:Standby daemons: 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.j{-1:14550} state up:standby seq 1 addr [v2:172.21.15.3:6834/977042251,v1:172.21.15.3:6835/977042251] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.868 INFO:teuthology.orchestra.run.smithi003.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.3:6836/1424466335,v1:172.21.15.3:6837/1424466335] compat {c=[1],r=[1],i=[fff]}] 2024-02-09T22:50:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4da40a2150 msgr2=0x7f4da40a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4da40a2150 0x7f4da40a4610 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f4db4005950 tx=0x7f4db400b410 comp rx=0 tx=0).stop 2024-02-09T22:50:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 msgr2=0x7f4db8075ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 0x7f4db8075ac0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f4da800d980 tx=0x7f4da800dc90 comp rx=0 tx=0).stop 2024-02-09T22:50:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 shutdown_connections 2024-02-09T22:50:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f4da40a2150 0x7f4da40a4610 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4db8103910 0x7f4db8074650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4db8101d70 0x7f4db8075ac0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 --2- 172.21.15.3:0/363145521 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4db8102f70 0x7f4db8074110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 >> 172.21.15.3:0/363145521 conn(0x7f4db80fd330 msgr2=0x7f4db81061a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 shutdown_connections 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:47.863+0000 7f4dbfdb2700 1 -- 172.21.15.3:0/363145521 wait complete. 2024-02-09T22:50:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:50:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:48 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/363145521' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T22:50:48.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:48 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/363145521' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T22:50:48.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:48 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/363145521' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-02-09T22:50:48.539 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- bash -c 'ceph osd dump' 2024-02-09T22:50:48.785 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:49.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:49 smithi003 ceph-mon[32678]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-02-09T22:50:49.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:49 smithi064 ceph-mon[36638]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-02-09T22:50:49.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:49 smithi089 ceph-mon[36665]: pgmap v339: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 426 B/s wr, 10 op/s 2024-02-09T22:50:50.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.328+0000 7f8096832700 1 -- 172.21.15.3:0/482585355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090101d50 msgr2=0x7f8090102170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.328+0000 7f8096832700 1 --2- 172.21.15.3:0/482585355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090101d50 0x7f8090102170 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f80800099c0 tx=0x7f8080009cd0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 -- 172.21.15.3:0/482585355 shutdown_connections 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 --2- 172.21.15.3:0/482585355 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80901038f0 0x7f809010a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 --2- 172.21.15.3:0/482585355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8090101d50 0x7f8090102170 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 --2- 172.21.15.3:0/482585355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8090102f50 0x7f80901033b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 -- 172.21.15.3:0/482585355 >> 172.21.15.3:0/482585355 conn(0x7f80900fd330 msgr2=0x7f80900ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 -- 172.21.15.3:0/482585355 shutdown_connections 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 -- 172.21.15.3:0/482585355 wait complete. 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 Processor -- start 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.329+0000 7f8096832700 1 -- start start 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8090101d50 0x7f8090197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8090102f50 0x7f80901981d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:50.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 0x7f809019f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809019f810 con 0x7f80901038f0 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809019f950 con 0x7f8090102f50 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f809019fb10 con 0x7f8090101d50 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 0x7f809019f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 0x7f809019f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41682/0 (socket says 172.21.15.3:41682) 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 -- 172.21.15.3:0/4166504062 learned_addr learned my addr 172.21.15.3:0/4166504062 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f808ffff700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8090101d50 0x7f8090197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f808f7fe700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8090102f50 0x7f80901981d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 -- 172.21.15.3:0/4166504062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8090101d50 msgr2=0x7f8090197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8090101d50 0x7f8090197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 -- 172.21.15.3:0/4166504062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8090102f50 msgr2=0x7f80901981d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8090102f50 0x7f80901981d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 -- 172.21.15.3:0/4166504062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80800095f0 con 0x7f80901038f0 2024-02-09T22:50:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8094dcf700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 0x7f809019f240 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8088007d70 tx=0x7f808800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f808800ad40 con 0x7f80901038f0 2024-02-09T22:50:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8088012900 con 0x7f80901038f0 2024-02-09T22:50:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f809019fdf0 con 0x7f80901038f0 2024-02-09T22:50:50.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.330+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8088011780 con 0x7f80901038f0 2024-02-09T22:50:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.331+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80901a0210 con 0x7f80901038f0 2024-02-09T22:50:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.332+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8090109110 con 0x7f80901038f0 2024-02-09T22:50:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.333+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f8088012a70 con 0x7f80901038f0 2024-02-09T22:50:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.333+0000 7f808d7fa700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f80780a2150 0x7f80780a4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:50.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.333+0000 7f808ffff700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f80780a2150 0x7f80780a4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.334+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f80880cdf90 con 0x7f80901038f0 2024-02-09T22:50:50.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.334+0000 7f808ffff700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f80780a2150 0x7f80780a4610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f80800036c0 tx=0x7f8080005ad0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:50.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.338+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8088101d20 con 0x7f80901038f0 2024-02-09T22:50:50.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.540+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f809002d870 con 0x7f80901038f0 2024-02-09T22:50:50.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.541+0000 7f808d7fa700 1 -- 172.21.15.3:0/4166504062 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v88) v1 ==== 56+0+5211 (secure 0 0 0) 0x7f8088096f70 con 0x7f80901038f0 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:epoch 88 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:created 2024-02-09T22:41:08.917818+0000 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:modified 2024-02-09T22:50:08.260275+0000 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:crush_version 26 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:full_ratio 0.95 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:backfillfull_ratio 0.9 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:nearfull_ratio 0.85 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:require_min_compat_client luminous 2024-02-09T22:50:50.546 INFO:teuthology.orchestra.run.smithi003.stdout:min_compat_client luminous 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:require_osd_release squid 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:stretch_mode_enabled false 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:max_osd 12 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6802/103055907,v1:172.21.15.3:6803/103055907] [v2:172.21.15.3:6804/103055907,v1:172.21.15.3:6805/103055907] exists,up c1313805-be20-4bd1-a375-4d8c740e1ff2 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.1 up in weight 1 up_from 13 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6800/368704973,v1:172.21.15.64:6801/368704973] [v2:172.21.15.64:6802/368704973,v1:172.21.15.64:6803/368704973] exists,up b4780ed6-656c-4d81-9520-d56cc32e39f9 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.2 up in weight 1 up_from 18 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6800/2362328563,v1:172.21.15.89:6801/2362328563] [v2:172.21.15.89:6802/2362328563,v1:172.21.15.89:6803/2362328563] exists,up d1c14346-6703-4597-8214-860019453569 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6810/3084237347,v1:172.21.15.3:6811/3084237347] [v2:172.21.15.3:6812/3084237347,v1:172.21.15.3:6813/3084237347] exists,up c7129351-e0c4-4563-8e98-d818d30d891a 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.4 up in weight 1 up_from 33 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6808/285015639,v1:172.21.15.64:6809/285015639] [v2:172.21.15.64:6810/285015639,v1:172.21.15.64:6811/285015639] exists,up bef8649b-bcc5-40c9-8df2-b59957282a8d 2024-02-09T22:50:50.547 INFO:teuthology.orchestra.run.smithi003.stdout:osd.5 up in weight 1 up_from 38 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6808/708499709,v1:172.21.15.89:6809/708499709] [v2:172.21.15.89:6810/708499709,v1:172.21.15.89:6811/708499709] exists,up 245a8e5f-2535-4346-8457-dd94e211574d 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.6 up in weight 1 up_from 46 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6818/1303593428,v1:172.21.15.3:6819/1303593428] [v2:172.21.15.3:6820/1303593428,v1:172.21.15.3:6821/1303593428] exists,up 392d6f64-4fbc-472e-96ef-dfb2b5291327 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.7 up in weight 1 up_from 51 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6816/867242656,v1:172.21.15.64:6817/867242656] [v2:172.21.15.64:6818/867242656,v1:172.21.15.64:6819/867242656] exists,up c96f9dfa-d559-4d7f-9fde-00d3fbbe856a 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.8 up in weight 1 up_from 59 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6816/4011062821,v1:172.21.15.89:6817/4011062821] [v2:172.21.15.89:6818/4011062821,v1:172.21.15.89:6819/4011062821] exists,up 49491de9-7b30-46a5-9869-b3ff6d2285f7 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.9 up in weight 1 up_from 67 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.3:6826/2163219868,v1:172.21.15.3:6827/2163219868] [v2:172.21.15.3:6828/2163219868,v1:172.21.15.3:6829/2163219868] exists,up 46d134d1-6b63-4909-8a0a-c5c431c9dfdb 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.64:6824/451880154,v1:172.21.15.64:6825/451880154] [v2:172.21.15.64:6826/451880154,v1:172.21.15.64:6827/451880154] exists,up c065623e-ebf8-4100-8d19-26bcc2a837aa 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6824/2302784119,v1:172.21.15.89:6825/2302784119] [v2:172.21.15.89:6826/2302784119,v1:172.21.15.89:6827/2302784119] exists,up 33affb4f-8f05-46cd-b347-fe905769c48c 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.5 [8,11] 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.a [2,11] 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 2.b [2,4] 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.e [4,8] 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.11 [5,9] 2024-02-09T22:50:50.548 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.22 [3,9] 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.31 [4,6] 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:pg_upmap_items 3.34 [4,9] 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/2045974484 expires 2024-02-10T22:42:06.452795+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/2045974484 expires 2024-02-10T22:42:06.452795+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1324827090 expires 2024-02-10T22:42:06.452795+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/858232672 expires 2024-02-10T22:41:39.084868+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/715011297 expires 2024-02-10T22:42:06.452795+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6800/2830848370 expires 2024-02-10T22:41:39.084868+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:6801/2830848370 expires 2024-02-10T22:41:39.084868+0000 2024-02-09T22:50:50.549 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1807309503 expires 2024-02-10T22:41:39.084868+0000 2024-02-09T22:50:50.550 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/222016476 expires 2024-02-10T22:41:39.084868+0000 2024-02-09T22:50:50.550 INFO:teuthology.orchestra.run.smithi003.stdout:blocklist 172.21.15.3:0/1462820865 expires 2024-02-10T22:42:06.452795+0000 2024-02-09T22:50:50.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.543+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f80780a2150 msgr2=0x7f80780a4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.543+0000 7f8096832700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f80780a2150 0x7f80780a4610 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f80800036c0 tx=0x7f8080005ad0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.543+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 msgr2=0x7f809019f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.543+0000 7f8096832700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 0x7f809019f240 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f8088007d70 tx=0x7f808800a340 comp rx=0 tx=0).stop 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 shutdown_connections 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f80780a2150 0x7f80780a4610 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8090101d50 0x7f8090197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80901038f0 0x7f809019f240 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 --2- 172.21.15.3:0/4166504062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8090102f50 0x7f80901981d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 >> 172.21.15.3:0/4166504062 conn(0x7f80900fd330 msgr2=0x7f8090106250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 shutdown_connections 2024-02-09T22:50:50.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:50.544+0000 7f8096832700 1 -- 172.21.15.3:0/4166504062 wait complete. 2024-02-09T22:50:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4166504062' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-09T22:50:50.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:50 smithi003 ceph-mon[32678]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-02-09T22:50:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4166504062' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-09T22:50:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:50 smithi064 ceph-mon[36638]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-02-09T22:50:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4166504062' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-02-09T22:50:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:50 smithi089 ceph-mon[36665]: pgmap v340: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 341 B/s wr, 9 op/s 2024-02-09T22:50:51.111 INFO:teuthology.run_tasks:Running task fs.ready... 2024-02-09T22:50:51.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 -- 172.21.15.3:0/4158000677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094101350 msgr2=0x7f50941017d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 --2- 172.21.15.3:0/4158000677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094101350 0x7f50941017d0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f5088009900 tx=0x7f508802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 -- 172.21.15.3:0/4158000677 shutdown_connections 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 --2- 172.21.15.3:0/4158000677 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5094101d10 0x7f50941085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 --2- 172.21.15.3:0/4158000677 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094101350 0x7f50941017d0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 --2- 172.21.15.3:0/4158000677 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5094100160 0x7f5094100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 -- 172.21.15.3:0/4158000677 >> 172.21.15.3:0/4158000677 conn(0x7f50940fb6d0 msgr2=0x7f50940fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:51.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 -- 172.21.15.3:0/4158000677 shutdown_connections 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 -- 172.21.15.3:0/4158000677 wait complete. 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.226+0000 7f5098979640 1 Processor -- start 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 -- start start 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 0x7f50941973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5094101350 0x7f50941978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5094101d10 0x7f509419e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509419ee50 con 0x7f5094100160 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509419efc0 con 0x7f5094101d10 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f509419f180 con 0x7f5094101350 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 0x7f50941973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 0x7f50941973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41702/0 (socket says 172.21.15.3:41702) 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 -- 172.21.15.3:0/4219666475 learned_addr learned my addr 172.21.15.3:0/4219666475 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5091d9b640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5094101350 0x7f50941978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5092d9d640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5094101d10 0x7f509419e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 -- 172.21.15.3:0/4219666475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5094101350 msgr2=0x7f50941978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5094101350 0x7f50941978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 -- 172.21.15.3:0/4219666475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5094101d10 msgr2=0x7f509419e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5094101d10 0x7f509419e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 -- 172.21.15.3:0/4219666475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5088009560 con 0x7f5094100160 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f509259c640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 0x7f50941973b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f508400b450 tx=0x7f508400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084011890 con 0x7f5094100160 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50840103e0 con 0x7f5094100160 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400f580 con 0x7f5094100160 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f509419f460 con 0x7f5094100160 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.227+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f509419f970 con 0x7f5094100160 2024-02-09T22:50:51.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.229+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509410bbf0 con 0x7f5094100160 2024-02-09T22:50:51.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.230+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f50840108e0 con 0x7f5094100160 2024-02-09T22:50:51.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.230+0000 7f50837fe640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f507c0a20c0 0x7f507c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.230+0000 7f5091d9b640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f507c0a20c0 0x7f507c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.230+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f50840cd990 con 0x7f5094100160 2024-02-09T22:50:51.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.231+0000 7f5091d9b640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f507c0a20c0 0x7f507c0a4580 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5088006010 tx=0x7f5088005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:51.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.235+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5084096970 con 0x7f5094100160 2024-02-09T22:50:51.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.420+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f509410bdd0 con 0x7f5094100160 2024-02-09T22:50:51.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.421+0000 7f50837fe640 1 -- 172.21.15.3:0/4219666475 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f508409a030 con 0x7f5094100160 2024-02-09T22:50:51.425 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:51.426 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:50:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f507c0a20c0 msgr2=0x7f507c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f507c0a20c0 0x7f507c0a4580 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5088006010 tx=0x7f5088005d90 comp rx=0 tx=0).stop 2024-02-09T22:50:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 msgr2=0x7f50941973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 0x7f50941973b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f508400b450 tx=0x7f508400b910 comp rx=0 tx=0).stop 2024-02-09T22:50:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 shutdown_connections 2024-02-09T22:50:51.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f507c0a20c0 0x7f507c0a4580 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5094101d10 0x7f509419e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5094101350 0x7f50941978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 --2- 172.21.15.3:0/4219666475 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5094100160 0x7f50941973b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 >> 172.21.15.3:0/4219666475 conn(0x7f50940fb6d0 msgr2=0x7f50940fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:51.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 shutdown_connections 2024-02-09T22:50:51.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.423+0000 7f5098979640 1 -- 172.21.15.3:0/4219666475 wait complete. 2024-02-09T22:50:51.438 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:50:51.439 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/629347724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0102df0 msgr2=0x7f7ad0109680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/629347724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0102df0 0x7f7ad0109680 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f7ac8009ca0 tx=0x7f7ac802c100 comp rx=0 tx=0).stop 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/629347724 shutdown_connections 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/629347724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ad0102430 0x7f7ad01028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/629347724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0102df0 0x7f7ad0109680 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/629347724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0101240 0x7f7ad0101660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/629347724 >> 172.21.15.3:0/629347724 conn(0x7f7ad00fc7b0 msgr2=0x7f7ad00febf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/629347724 shutdown_connections 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.540+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/629347724 wait complete. 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 Processor -- start 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 -- start start 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 0x7f7ad01972a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0102430 0x7f7ad01977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ad0102df0 0x7f7ad019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad019ed90 con 0x7f7ad0101240 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad019ef00 con 0x7f7ad0102df0 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad6fd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ad019f0c0 con 0x7f7ad0102430 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 0x7f7ad01972a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 0x7f7ad01972a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41714/0 (socket says 172.21.15.3:41714) 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 -- 172.21.15.3:0/498428624 learned_addr learned my addr 172.21.15.3:0/498428624 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:51.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad556d640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ad0102df0 0x7f7ad019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7acffff640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0102430 0x7f7ad01977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 -- 172.21.15.3:0/498428624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0102430 msgr2=0x7f7ad01977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0102430 0x7f7ad01977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 -- 172.21.15.3:0/498428624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ad0102df0 msgr2=0x7f7ad019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ad0102df0 0x7f7ad019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 -- 172.21.15.3:0/498428624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ac8009900 con 0x7f7ad0101240 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7acffff640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0102430 0x7f7ad01977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.541+0000 7f7ad4d6c640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 0x7f7ad01972a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c970 tx=0x7f7ac400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.542+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4004220 con 0x7f7ad0101240 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.542+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ac4007e90 con 0x7f7ad0101240 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.542+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad019f3a0 con 0x7f7ad0101240 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.542+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ac4004a00 con 0x7f7ad0101240 2024-02-09T22:50:51.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.542+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad019f860 con 0x7f7ad0101240 2024-02-09T22:50:51.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.543+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad0108500 con 0x7f7ad0101240 2024-02-09T22:50:51.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.544+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f7ac40079b0 con 0x7f7ad0101240 2024-02-09T22:50:51.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.544+0000 7f7acdffb640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7ac00a20c0 0x7f7ac00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.544+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f7ac40cdd60 con 0x7f7ad0101240 2024-02-09T22:50:51.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.544+0000 7f7acffff640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7ac00a20c0 0x7f7ac00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.545+0000 7f7acffff640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7ac00a20c0 0x7f7ac00a4580 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7ad01987b0 tx=0x7f7abc005d30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:51.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.548+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7ac4096d40 con 0x7f7ad0101240 2024-02-09T22:50:51.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.733+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ad002cdb0 con 0x7f7ad0101240 2024-02-09T22:50:51.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.733+0000 7f7acdffb640 1 -- 172.21.15.3:0/498428624 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ac401d020 con 0x7f7ad0101240 2024-02-09T22:50:51.737 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:50:51.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7ac00a20c0 msgr2=0x7f7ac00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7ac00a20c0 0x7f7ac00a4580 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7ad01987b0 tx=0x7f7abc005d30 comp rx=0 tx=0).stop 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 msgr2=0x7f7ad01972a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 0x7f7ad01972a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f7ac400c970 tx=0x7f7ac400ce30 comp rx=0 tx=0).stop 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 shutdown_connections 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f7ac00a20c0 0x7f7ac00a4580 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7ad0102df0 0x7f7ad019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7ad0102430 0x7f7ad01977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 --2- 172.21.15.3:0/498428624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7ad0101240 0x7f7ad01972a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 >> 172.21.15.3:0/498428624 conn(0x7f7ad00fc7b0 msgr2=0x7f7ad01056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 shutdown_connections 2024-02-09T22:50:51.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.735+0000 7f7ad6fd0640 1 -- 172.21.15.3:0/498428624 wait complete. 2024-02-09T22:50:51.749 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4219666475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:51.750 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:50:51.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.851+0000 7f588cc37640 1 -- 172.21.15.3:0/2897225906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101d00 msgr2=0x7f5888108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.851+0000 7f588cc37640 1 --2- 172.21.15.3:0/2897225906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101d00 0x7f5888108590 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5878009ca0 tx=0x7f587802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 -- 172.21.15.3:0/2897225906 shutdown_connections 2024-02-09T22:50:51.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 --2- 172.21.15.3:0/2897225906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888101340 0x7f58881017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 --2- 172.21.15.3:0/2897225906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101d00 0x7f5888108590 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 --2- 172.21.15.3:0/2897225906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5888100150 0x7f5888100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 -- 172.21.15.3:0/2897225906 >> 172.21.15.3:0/2897225906 conn(0x7f58880fb6c0 msgr2=0x7f58880fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 -- 172.21.15.3:0/2897225906 shutdown_connections 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 -- 172.21.15.3:0/2897225906 wait complete. 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 Processor -- start 2024-02-09T22:50:51.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.852+0000 7f588cc37640 1 -- start start 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588cc37640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888100150 0x7f5888197200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588cc37640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 0x7f5888197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588cc37640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5888101d00 0x7f588819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588cc37640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588819ecf0 con 0x7f5888101340 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588cc37640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588819ee60 con 0x7f5888100150 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588cc37640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588819f020 con 0x7f5888101d00 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 0x7f5888197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888100150 0x7f5888197200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 0x7f5888197740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41726/0 (socket says 172.21.15.3:41726) 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f588659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888100150 0x7f5888197200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53332/0 (socket says 172.21.15.3:53332) 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 -- 172.21.15.3:0/4001338657 learned_addr learned my addr 172.21.15.3:0/4001338657 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5886d9d640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5888101d00 0x7f588819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 -- 172.21.15.3:0/4001338657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5888101d00 msgr2=0x7f588819e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5888101d00 0x7f588819e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 -- 172.21.15.3:0/4001338657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888100150 msgr2=0x7f5888197200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888100150 0x7f5888197200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 -- 172.21.15.3:0/4001338657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5878009900 con 0x7f5888101340 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.853+0000 7f5885d9b640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 0x7f5888197740 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f587000c920 tx=0x7f587000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.854+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870007c70 con 0x7f5888101340 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.854+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5870007dd0 con 0x7f5888101340 2024-02-09T22:50:51.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.854+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588819f300 con 0x7f5888101340 2024-02-09T22:50:51.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.854+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5870018860 con 0x7f5888101340 2024-02-09T22:50:51.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.854+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588819f810 con 0x7f5888101340 2024-02-09T22:50:51.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.855+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f588810bb60 con 0x7f5888101340 2024-02-09T22:50:51.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.856+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f587000fac0 con 0x7f5888101340 2024-02-09T22:50:51.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.856+0000 7f587f7fe640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f58740a20c0 0x7f58740a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:51.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.856+0000 7f588659c640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f58740a20c0 0x7f58740a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:51.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.856+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f58700cddf0 con 0x7f5888101340 2024-02-09T22:50:51.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.857+0000 7f588659c640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f58740a20c0 0x7f58740a4580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f58881011a0 tx=0x7f5880005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:51.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:51.861+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5870096dd0 con 0x7f5888101340 2024-02-09T22:50:51.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4219666475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:51.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4219666475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.042+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f588819fd30 con 0x7f5888101340 2024-02-09T22:50:52.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.043+0000 7f587f7fe640 1 -- 172.21.15.3:0/4001338657 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f587009a490 con 0x7f5888101340 2024-02-09T22:50:52.048 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:50:52.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f58740a20c0 msgr2=0x7f58740a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f58740a20c0 0x7f58740a4580 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f58881011a0 tx=0x7f5880005f90 comp rx=0 tx=0).stop 2024-02-09T22:50:52.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 msgr2=0x7f5888197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 0x7f5888197740 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f587000c920 tx=0x7f587000cde0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 shutdown_connections 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f58740a20c0 0x7f58740a4580 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5888101d00 0x7f588819e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5888101340 0x7f5888197740 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 --2- 172.21.15.3:0/4001338657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5888100150 0x7f5888197200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 >> 172.21.15.3:0/4001338657 conn(0x7f58880fb6c0 msgr2=0x7f58880fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 shutdown_connections 2024-02-09T22:50:52.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.045+0000 7f588cc37640 1 -- 172.21.15.3:0/4001338657 wait complete. 2024-02-09T22:50:52.061 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 -- 172.21.15.3:0/1382821678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 msgr2=0x7f54fc1017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 --2- 172.21.15.3:0/1382821678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc1017c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009900 tx=0x7f54f002bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 -- 172.21.15.3:0/1382821678 shutdown_connections 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 --2- 172.21.15.3:0/1382821678 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 0x7f54fc108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 --2- 172.21.15.3:0/1382821678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc1017c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 --2- 172.21.15.3:0/1382821678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 0x7f54fc100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 -- 172.21.15.3:0/1382821678 >> 172.21.15.3:0/1382821678 conn(0x7f54fc0fb6c0 msgr2=0x7f54fc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 -- 172.21.15.3:0/1382821678 shutdown_connections 2024-02-09T22:50:52.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 -- 172.21.15.3:0/1382821678 wait complete. 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.163+0000 7f5502064640 1 Processor -- start 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 -- start start 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 0x7f54fc069f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 0x7f54fc1a0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1a11d0 con 0x7f54fc101340 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1a1340 con 0x7f54fc101d00 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f5502064640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54fc1a1500 con 0x7f54fc100150 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc19e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41748/0 (socket says 172.21.15.3:41748) 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 -- 172.21.15.3:0/2645287436 learned_addr learned my addr 172.21.15.3:0/2645287436 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54fb7fe640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 0x7f54fc069f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54fbfff640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 0x7f54fc1a0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 -- 172.21.15.3:0/2645287436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 msgr2=0x7f54fc069f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 0x7f54fc069f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 -- 172.21.15.3:0/2645287436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 msgr2=0x7f54fc1a0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 0x7f54fc1a0b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 -- 172.21.15.3:0/2645287436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0009560 con 0x7f54fc101340 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54fb7fe640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 0x7f54fc069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54fbfff640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 0x7f54fc1a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54faffd640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc19e5c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f54f0005ab0 tx=0x7f54f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.164+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f00044e0 con 0x7f54fc101340 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.165+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54fc1a1720 con 0x7f54fc101340 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.165+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54f00054e0 con 0x7f54fc101340 2024-02-09T22:50:52.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.165+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f003e450 con 0x7f54fc101340 2024-02-09T22:50:52.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.165+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54fc1a1cb0 con 0x7f54fc101340 2024-02-09T22:50:52.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.165+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54fc1075b0 con 0x7f54fc101340 2024-02-09T22:50:52.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.167+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f54f00035f0 con 0x7f54fc101340 2024-02-09T22:50:52.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.167+0000 7f54f8ff9640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f54e40a20f0 0x7f54e40a45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.167+0000 7f54fb7fe640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f54e40a20f0 0x7f54e40a45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.167+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f54f00f0270 con 0x7f54fc101340 2024-02-09T22:50:52.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.168+0000 7f54fb7fe640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f54e40a20f0 0x7f54e40a45b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f54fc1011a0 tx=0x7f54ec008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:52.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.171+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f54f00b9250 con 0x7f54fc101340 2024-02-09T22:50:52.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.352+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f54fc02cdb0 con 0x7f54fc101340 2024-02-09T22:50:52.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.352+0000 7f54f8ff9640 1 -- 172.21.15.3:0/2645287436 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f54f00bc910 con 0x7f54fc101340 2024-02-09T22:50:52.356 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:50:52.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f54e40a20f0 msgr2=0x7f54e40a45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f54e40a20f0 0x7f54e40a45b0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f54fc1011a0 tx=0x7f54ec008040 comp rx=0 tx=0).stop 2024-02-09T22:50:52.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 msgr2=0x7f54fc19e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc19e5c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f54f0005ab0 tx=0x7f54f00042f0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 shutdown_connections 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f54e40a20f0 0x7f54e40a45b0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f54fc101d00 0x7f54fc1a0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f54fc101340 0x7f54fc19e5c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 --2- 172.21.15.3:0/2645287436 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f54fc100150 0x7f54fc069f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 >> 172.21.15.3:0/2645287436 conn(0x7f54fc0fb6c0 msgr2=0x7f54fc104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 shutdown_connections 2024-02-09T22:50:52.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.354+0000 7f5502064640 1 -- 172.21.15.3:0/2645287436 wait complete. 2024-02-09T22:50:52.369 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:50:52.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 -- 172.21.15.3:0/2877062580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0101d50 msgr2=0x7f35a0104140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 --2- 172.21.15.3:0/2877062580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0101d50 0x7f35a0104140 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f359c009900 tx=0x7f359c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 -- 172.21.15.3:0/2877062580 shutdown_connections 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 --2- 172.21.15.3:0/2877062580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a0104680 0x7f35a0106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 --2- 172.21.15.3:0/2877062580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a0106fb0 0x7f35a0109400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 --2- 172.21.15.3:0/2877062580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0101d50 0x7f35a0104140 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 -- 172.21.15.3:0/2877062580 >> 172.21.15.3:0/2877062580 conn(0x7f35a00fb760 msgr2=0x7f35a00fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 -- 172.21.15.3:0/2877062580 shutdown_connections 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.467+0000 7f35a7840640 1 -- 172.21.15.3:0/2877062580 wait complete. 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 Processor -- start 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 -- start start 2024-02-09T22:50:52.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a0101d50 0x7f35a0101670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a0104680 0x7f35a00ffd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 0x7f35a0100240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a01009d0 con 0x7f35a0106fb0 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0100b40 con 0x7f35a0101d50 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a7840640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a01a1370 con 0x7f35a0104680 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a5ddd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 0x7f35a0100240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a5ddd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 0x7f35a0100240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41764/0 (socket says 172.21.15.3:41764) 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a5ddd640 1 -- 172.21.15.3:0/911639067 learned_addr learned my addr 172.21.15.3:0/911639067 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a55dc640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a0101d50 0x7f35a0101670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.468+0000 7f35a4ddb640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a0104680 0x7f35a00ffd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a5ddd640 1 -- 172.21.15.3:0/911639067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a0104680 msgr2=0x7f35a00ffd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a5ddd640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a0104680 0x7f35a00ffd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a5ddd640 1 -- 172.21.15.3:0/911639067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a0101d50 msgr2=0x7f35a0101670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a5ddd640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a0101d50 0x7f35a0101670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a5ddd640 1 -- 172.21.15.3:0/911639067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c009560 con 0x7f35a0106fb0 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a5ddd640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 0x7f35a0100240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f359800ead0 tx=0x7f359800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800cde0 con 0x7f35a0106fb0 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3598005020 con 0x7f35a0106fb0 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a01a15d0 con 0x7f35a0106fb0 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35980116b0 con 0x7f35a0106fb0 2024-02-09T22:50:52.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.469+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a01a1ae0 con 0x7f35a0106fb0 2024-02-09T22:50:52.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.470+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a00fd250 con 0x7f35a0106fb0 2024-02-09T22:50:52.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.471+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3598005520 con 0x7f35a0106fb0 2024-02-09T22:50:52.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.471+0000 7f35967fc640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f358c0a20c0 0x7f358c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.472+0000 7f35a55dc640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f358c0a20c0 0x7f358c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.472+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f35980cde40 con 0x7f35a0106fb0 2024-02-09T22:50:52.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.472+0000 7f35a55dc640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f358c0a20c0 0x7f358c0a4580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f359c009230 tx=0x7f359c005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:52.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.476+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3598096e70 con 0x7f35a0106fb0 2024-02-09T22:50:52.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.695+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35a002cdb0 con 0x7f35a0106fb0 2024-02-09T22:50:52.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.696+0000 7f35967fc640 1 -- 172.21.15.3:0/911639067 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f359801f020 con 0x7f35a0106fb0 2024-02-09T22:50:52.700 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:52.700 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:49:13.115988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:50:52.700 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.697+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f358c0a20c0 msgr2=0x7f358c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.697+0000 7f35a7840640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f358c0a20c0 0x7f358c0a4580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f359c009230 tx=0x7f359c005fd0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.697+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 msgr2=0x7f35a0100240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.697+0000 7f35a7840640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 0x7f35a0100240 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f359800ead0 tx=0x7f359800ef90 comp rx=0 tx=0).stop 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 shutdown_connections 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f358c0a20c0 0x7f358c0a4580 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a0106fb0 0x7f35a0100240 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a0104680 0x7f35a00ffd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 --2- 172.21.15.3:0/911639067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a0101d50 0x7f35a0101670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 >> 172.21.15.3:0/911639067 conn(0x7f35a00fb760 msgr2=0x7f35a0102690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 shutdown_connections 2024-02-09T22:50:52.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.698+0000 7f35a7840640 1 -- 172.21.15.3:0/911639067 wait complete. 2024-02-09T22:50:52.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:50:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 -- 172.21.15.3:0/380224903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc101d70 msgr2=0x7f96cc108600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 --2- 172.21.15.3:0/380224903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc101d70 0x7f96cc108600 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009ca0 tx=0x7f96c402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 -- 172.21.15.3:0/380224903 shutdown_connections 2024-02-09T22:50:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 --2- 172.21.15.3:0/380224903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc1013b0 0x7f96cc101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 --2- 172.21.15.3:0/380224903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc101d70 0x7f96cc108600 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 --2- 172.21.15.3:0/380224903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1001c0 0x7f96cc1005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 -- 172.21.15.3:0/380224903 >> 172.21.15.3:0/380224903 conn(0x7f96cc0fb710 msgr2=0x7f96cc0fdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 -- 172.21.15.3:0/380224903 shutdown_connections 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 -- 172.21.15.3:0/380224903 wait complete. 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.812+0000 7f96d2193640 1 Processor -- start 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 -- start start 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 0x7f96cc1971f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1013b0 0x7f96cc197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc101d70 0x7f96cc19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc19ece0 con 0x7f96cc1001c0 2024-02-09T22:50:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc19ee50 con 0x7f96cc101d70 2024-02-09T22:50:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc19f010 con 0x7f96cc1013b0 2024-02-09T22:50:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 0x7f96cc1971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 0x7f96cc1971f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41778/0 (socket says 172.21.15.3:41778) 2024-02-09T22:50:52.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 -- 172.21.15.3:0/3349781671 learned_addr learned my addr 172.21.15.3:0/3349781671 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cbfff640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc101d70 0x7f96cc19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96caffd640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1013b0 0x7f96cc197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 -- 172.21.15.3:0/3349781671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1013b0 msgr2=0x7f96cc197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1013b0 0x7f96cc197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 -- 172.21.15.3:0/3349781671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc101d70 msgr2=0x7f96cc19e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc101d70 0x7f96cc19e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:52.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 -- 172.21.15.3:0/3349781671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4009900 con 0x7f96cc1001c0 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96caffd640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1013b0 0x7f96cc197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cbfff640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc101d70 0x7f96cc19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96cb7fe640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 0x7f96cc1971f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00d950 tx=0x7f96bc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc009c30 con 0x7f96cc1001c0 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96bc009d90 con 0x7f96cc1001c0 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc19f2f0 con 0x7f96cc1001c0 2024-02-09T22:50:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.813+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96bc00f720 con 0x7f96cc1001c0 2024-02-09T22:50:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.814+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc19f7b0 con 0x7f96cc1001c0 2024-02-09T22:50:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.814+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc107410 con 0x7f96cc1001c0 2024-02-09T22:50:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.815+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f96bc010a60 con 0x7f96cc1001c0 2024-02-09T22:50:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.816+0000 7f96c8ff9640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96b40a20c0 0x7f96b40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.816+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f96bc0cdaf0 con 0x7f96cc1001c0 2024-02-09T22:50:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.816+0000 7f96caffd640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96b40a20c0 0x7f96b40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:52.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.816+0000 7f96caffd640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96b40a20c0 0x7f96b40a4580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96cc198700 tx=0x7f96c0005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:52.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:52.820+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f96bc096ad0 con 0x7f96cc1001c0 2024-02-09T22:50:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:52 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/498428624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:52 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4001338657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:52 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2645287436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:52 smithi003 ceph-mon[32678]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-02-09T22:50:52.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:52 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/498428624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:52 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4001338657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:52 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2645287436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:52 smithi064 ceph-mon[36638]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-02-09T22:50:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:52 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/498428624' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:52 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4001338657' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:52 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2645287436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:52.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:52 smithi089 ceph-mon[36665]: pgmap v341: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-02-09T22:50:53.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.004+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96cc02ce50 con 0x7f96cc1001c0 2024-02-09T22:50:53.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.005+0000 7f96c8ff9640 1 -- 172.21.15.3:0/3349781671 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f96bc09a190 con 0x7f96cc1001c0 2024-02-09T22:50:53.010 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:53.010 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96b40a20c0 msgr2=0x7f96b40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96b40a20c0 0x7f96b40a4580 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f96cc198700 tx=0x7f96c0005d90 comp rx=0 tx=0).stop 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 msgr2=0x7f96cc1971f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 0x7f96cc1971f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f96bc00d950 tx=0x7f96bc00de10 comp rx=0 tx=0).stop 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 shutdown_connections 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f96b40a20c0 0x7f96b40a4580 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96cc101d70 0x7f96cc19e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96cc1013b0 0x7f96cc197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 --2- 172.21.15.3:0/3349781671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96cc1001c0 0x7f96cc1971f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 >> 172.21.15.3:0/3349781671 conn(0x7f96cc0fb710 msgr2=0x7f96cc1045e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 shutdown_connections 2024-02-09T22:50:53.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.008+0000 7f96d2193640 1 -- 172.21.15.3:0/3349781671 wait complete. 2024-02-09T22:50:53.023 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:50:53.024 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:50:53.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.125+0000 7fb235e81640 1 -- 172.21.15.3:0/166167811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230101340 msgr2=0x7fb2301017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.125+0000 7fb235e81640 1 --2- 172.21.15.3:0/166167811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230101340 0x7fb2301017c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb224009900 tx=0x7fb22402bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.125+0000 7fb235e81640 1 -- 172.21.15.3:0/166167811 shutdown_connections 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.125+0000 7fb235e81640 1 --2- 172.21.15.3:0/166167811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb230101d00 0x7fb230108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.125+0000 7fb235e81640 1 --2- 172.21.15.3:0/166167811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230101340 0x7fb2301017c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 --2- 172.21.15.3:0/166167811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb230100150 0x7fb230100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- 172.21.15.3:0/166167811 >> 172.21.15.3:0/166167811 conn(0x7fb2300fb6c0 msgr2=0x7fb2300fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- 172.21.15.3:0/166167811 shutdown_connections 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- 172.21.15.3:0/166167811 wait complete. 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 Processor -- start 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- start start 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 0x7fb2301961e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb230101340 0x7fb230196720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb230101d00 0x7fb23019d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019dcd0 con 0x7fb230100150 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019de40 con 0x7fb230101d00 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.126+0000 7fb235e81640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23019e000 con 0x7fb230101340 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 0x7fb2301961e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 0x7fb2301961e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41808/0 (socket says 172.21.15.3:41808) 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 -- 172.21.15.3:0/3464949978 learned_addr learned my addr 172.21.15.3:0/3464949978 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22effd640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb230101340 0x7fb230196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 -- 172.21.15.3:0/3464949978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb230101340 msgr2=0x7fb230196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb230101340 0x7fb230196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 -- 172.21.15.3:0/3464949978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb230101d00 msgr2=0x7fb23019d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb230101d00 0x7fb23019d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 -- 172.21.15.3:0/3464949978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb224009560 con 0x7fb230100150 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22f7fe640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 0x7fb2301961e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c970 tx=0x7fb22000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb220004220 con 0x7fb230100150 2024-02-09T22:50:53.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb23019e2e0 con 0x7fb230100150 2024-02-09T22:50:53.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb220007e90 con 0x7fb230100150 2024-02-09T22:50:53.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb22000f450 con 0x7fb230100150 2024-02-09T22:50:53.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.127+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb23019e810 con 0x7fb230100150 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.128+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb230107490 con 0x7fb230100150 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.129+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb220003610 con 0x7fb230100150 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.130+0000 7fb22cff9640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb2180a20c0 0x7fb2180a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.130+0000 7fb22effd640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb2180a20c0 0x7fb2180a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.130+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb2200cd980 con 0x7fb230100150 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.130+0000 7fb22effd640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb2180a20c0 0x7fb2180a4580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb224009230 tx=0x7fb224005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:53.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.134+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb220096960 con 0x7fb230100150 2024-02-09T22:50:53.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.314+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb23010bb60 con 0x7fb230100150 2024-02-09T22:50:53.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.315+0000 7fb22cff9640 1 -- 172.21.15.3:0/3464949978 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb23010bb60 con 0x7fb230100150 2024-02-09T22:50:53.320 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:50:53.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb2180a20c0 msgr2=0x7fb2180a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb2180a20c0 0x7fb2180a4580 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fb224009230 tx=0x7fb224005fd0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 msgr2=0x7fb2301961e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 0x7fb2301961e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb22000c970 tx=0x7fb22000ce30 comp rx=0 tx=0).stop 2024-02-09T22:50:53.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 shutdown_connections 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb2180a20c0 0x7fb2180a4580 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb230101d00 0x7fb23019d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb230101340 0x7fb230196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 --2- 172.21.15.3:0/3464949978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb230100150 0x7fb2301961e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 >> 172.21.15.3:0/3464949978 conn(0x7fb2300fb6c0 msgr2=0x7fb230104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 shutdown_connections 2024-02-09T22:50:53.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.317+0000 7fb235e81640 1 -- 172.21.15.3:0/3464949978 wait complete. 2024-02-09T22:50:53.332 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:50:53.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 -- 172.21.15.3:0/1253886184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 msgr2=0x7fec88102a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/1253886184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88102a30 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fec7c009900 tx=0x7fec7c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:53.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 -- 172.21.15.3:0/1253886184 shutdown_connections 2024-02-09T22:50:53.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/1253886184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 0x7fec88109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/1253886184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88102a30 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/1253886184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 0x7fec880ff960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 -- 172.21.15.3:0/1253886184 >> 172.21.15.3:0/1253886184 conn(0x7fec880fae50 msgr2=0x7fec880fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 -- 172.21.15.3:0/1253886184 shutdown_connections 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 -- 172.21.15.3:0/1253886184 wait complete. 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.434+0000 7fec8f5bc640 1 Processor -- start 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 -- start start 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 0x7fec881972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 0x7fec8819e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8819edd0 con 0x7fec881025b0 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8819ef40 con 0x7fec880ff540 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8819f100 con 0x7fec88102f70 2024-02-09T22:50:53.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88197820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41832/0 (socket says 172.21.15.3:41832) 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 -- 172.21.15.3:0/599564791 learned_addr learned my addr 172.21.15.3:0/599564791 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8d358640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 0x7fec881972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8db59640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 0x7fec8819e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 -- 172.21.15.3:0/599564791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 msgr2=0x7fec8819e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 0x7fec8819e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 -- 172.21.15.3:0/599564791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 msgr2=0x7fec881972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 0x7fec881972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 -- 172.21.15.3:0/599564791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c009560 con 0x7fec881025b0 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8d358640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 0x7fec881972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8db59640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 0x7fec8819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8cb57640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88197820 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006010 tx=0x7fec7c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c0044e0 con 0x7fec881025b0 2024-02-09T22:50:53.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec8819f380 con 0x7fec881025b0 2024-02-09T22:50:53.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec7c0054e0 con 0x7fec881025b0 2024-02-09T22:50:53.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c03e450 con 0x7fec881025b0 2024-02-09T22:50:53.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.435+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8819f890 con 0x7fec881025b0 2024-02-09T22:50:53.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.436+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec881086e0 con 0x7fec881025b0 2024-02-09T22:50:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.437+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fec7c0035f0 con 0x7fec881025b0 2024-02-09T22:50:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.438+0000 7fec867fc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec700a20c0 0x7fec700a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.438+0000 7fec8d358640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec700a20c0 0x7fec700a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.438+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fec7c0f0150 con 0x7fec881025b0 2024-02-09T22:50:53.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.439+0000 7fec8d358640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec700a20c0 0x7fec700a4580 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fec88069630 tx=0x7fec80009540 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:53.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.443+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fec7c0b90d0 con 0x7fec881025b0 2024-02-09T22:50:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.623+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fec8802ce50 con 0x7fec881025b0 2024-02-09T22:50:53.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.625+0000 7fec867fc640 1 -- 172.21.15.3:0/599564791 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fec7c0bc790 con 0x7fec881025b0 2024-02-09T22:50:53.629 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:50:53.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec700a20c0 msgr2=0x7fec700a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec700a20c0 0x7fec700a4580 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fec88069630 tx=0x7fec80009540 comp rx=0 tx=0).stop 2024-02-09T22:50:53.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 msgr2=0x7fec88197820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88197820 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fec7c006010 tx=0x7fec7c0042f0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 shutdown_connections 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fec700a20c0 0x7fec700a4580 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fec88102f70 0x7fec8819e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fec881025b0 0x7fec88197820 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 --2- 172.21.15.3:0/599564791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fec880ff540 0x7fec881972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 >> 172.21.15.3:0/599564791 conn(0x7fec880fae50 msgr2=0x7fec881058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 shutdown_connections 2024-02-09T22:50:53.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.627+0000 7fec8f5bc640 1 -- 172.21.15.3:0/599564791 wait complete. 2024-02-09T22:50:53.642 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:50:53.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- 172.21.15.3:0/353961446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484101230 msgr2=0x7ff484101650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/353961446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484101230 0x7ff484101650 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7ff478009900 tx=0x7ff47802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- 172.21.15.3:0/353961446 shutdown_connections 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/353961446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484102de0 0x7ff484109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/353961446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff484102420 0x7ff4841028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/353961446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484101230 0x7ff484101650 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- 172.21.15.3:0/353961446 >> 172.21.15.3:0/353961446 conn(0x7ff4840fc760 msgr2=0x7ff4840febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- 172.21.15.3:0/353961446 shutdown_connections 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- 172.21.15.3:0/353961446 wait complete. 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 Processor -- start 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- start start 2024-02-09T22:50:53.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff484101230 0x7ff4841972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484102420 0x7ff484197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 0x7ff48419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48419edc0 con 0x7ff484102de0 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48419ef30 con 0x7ff484102420 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff48ac4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff48419f0f0 con 0x7ff484101230 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff4891eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 0x7ff48419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.745+0000 7ff4891eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 0x7ff48419e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41850/0 (socket says 172.21.15.3:41850) 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 -- 172.21.15.3:0/3541204736 learned_addr learned my addr 172.21.15.3:0/3541204736 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff483fff640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484102420 0x7ff484197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 -- 172.21.15.3:0/3541204736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff484101230 msgr2=0x7ff4841972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff484101230 0x7ff4841972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 -- 172.21.15.3:0/3541204736 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484102420 msgr2=0x7ff484197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484102420 0x7ff484197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 -- 172.21.15.3:0/3541204736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478009560 con 0x7ff484102de0 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff4891eb640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 0x7ff48419e880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00eaa0 tx=0x7ff47c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c00cdb0 con 0x7ff484102de0 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff47c005020 con 0x7ff484102de0 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48419f3d0 con 0x7ff484102de0 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47c0116b0 con 0x7ff484102de0 2024-02-09T22:50:53.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.746+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff48419f890 con 0x7ff484102de0 2024-02-09T22:50:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.747+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff484108570 con 0x7ff484102de0 2024-02-09T22:50:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.748+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7ff47c005520 con 0x7ff484102de0 2024-02-09T22:50:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.748+0000 7ff481ffb640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff4740a20c0 0x7ff4740a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.749+0000 7ff4889ea640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff4740a20c0 0x7ff4740a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.749+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff47c0cdfd0 con 0x7ff484102de0 2024-02-09T22:50:53.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.749+0000 7ff4889ea640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff4740a20c0 0x7ff4740a4580 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff47802c6a0 tx=0x7ff478000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:53.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.753+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff47c096f50 con 0x7ff484102de0 2024-02-09T22:50:53.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:53 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/911639067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:53.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:53 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3349781671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:53.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:53 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3464949978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:53.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:53 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/911639067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:53.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:53 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3349781671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:53.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:53 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3464949978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:53.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.935+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff48410cc40 con 0x7ff484102de0 2024-02-09T22:50:53.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.936+0000 7ff481ffb640 1 -- 172.21.15.3:0/3541204736 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff48410cc40 con 0x7ff484102de0 2024-02-09T22:50:53.941 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:50:53.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff4740a20c0 msgr2=0x7ff4740a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff4740a20c0 0x7ff4740a4580 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff47802c6a0 tx=0x7ff478000bc0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 msgr2=0x7ff48419e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:53.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 0x7ff48419e880 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff47c00eaa0 tx=0x7ff47c00ef60 comp rx=0 tx=0).stop 2024-02-09T22:50:53.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 shutdown_connections 2024-02-09T22:50:53.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff4740a20c0 0x7ff4740a4580 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff484102420 0x7ff484197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff484102de0 0x7ff48419e880 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 --2- 172.21.15.3:0/3541204736 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff484101230 0x7ff4841972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:53.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 >> 172.21.15.3:0/3541204736 conn(0x7ff4840fc760 msgr2=0x7ff484105740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:53.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 shutdown_connections 2024-02-09T22:50:53.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:53.938+0000 7ff48ac4e640 1 -- 172.21.15.3:0/3541204736 wait complete. 2024-02-09T22:50:53.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:50:53.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:53 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/911639067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:53.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:53 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3349781671' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:53.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:53 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3464949978' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1748028455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 msgr2=0x7f3bb0109700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1748028455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb0109700 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009ca0 tx=0x7f3ba802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1748028455 shutdown_connections 2024-02-09T22:50:54.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1748028455 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bb01024b0 0x7f3bb0102930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1748028455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb0109700 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1748028455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bb01012c0 0x7f3bb01016e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1748028455 >> 172.21.15.3:0/1748028455 conn(0x7f3bb00fc850 msgr2=0x7f3bb00fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1748028455 shutdown_connections 2024-02-09T22:50:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1748028455 wait complete. 2024-02-09T22:50:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 Processor -- start 2024-02-09T22:50:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.053+0000 7f3bb6dd7640 1 -- start start 2024-02-09T22:50:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bb01012c0 0x7f3bb0197270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bb01024b0 0x7f3bb01977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb019e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb019ed60 con 0x7f3bb0102e70 2024-02-09T22:50:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb019eed0 con 0x7f3bb01024b0 2024-02-09T22:50:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb019f090 con 0x7f3bb01012c0 2024-02-09T22:50:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb019e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb019e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41870/0 (socket says 172.21.15.3:41870) 2024-02-09T22:50:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 -- 172.21.15.3:0/1314864101 learned_addr learned my addr 172.21.15.3:0/1314864101 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb4b73640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bb01012c0 0x7f3bb0197270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3baffff640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bb01024b0 0x7f3bb01977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 -- 172.21.15.3:0/1314864101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bb01012c0 msgr2=0x7f3bb0197270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bb01012c0 0x7f3bb0197270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 -- 172.21.15.3:0/1314864101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bb01024b0 msgr2=0x7f3bb01977b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bb01024b0 0x7f3bb01977b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 -- 172.21.15.3:0/1314864101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba8009900 con 0x7f3bb0102e70 2024-02-09T22:50:54.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb5374640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb019e820 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8037040 tx=0x7f3ba8034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8032eb0 con 0x7f3bb0102e70 2024-02-09T22:50:54.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ba8003e50 con 0x7f3bb0102e70 2024-02-09T22:50:54.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb019f310 con 0x7f3bb0102e70 2024-02-09T22:50:54.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.054+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8040450 con 0x7f3bb0102e70 2024-02-09T22:50:54.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.055+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb019f810 con 0x7f3bb0102e70 2024-02-09T22:50:54.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.055+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb01085e0 con 0x7f3bb0102e70 2024-02-09T22:50:54.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.057+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3ba800a040 con 0x7f3bb0102e70 2024-02-09T22:50:54.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.057+0000 7f3badffb640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ba00a2140 0x7f3ba00a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.057+0000 7f3bb4b73640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ba00a2140 0x7f3ba00a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.057+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ba80f0ee0 con 0x7f3bb0102e70 2024-02-09T22:50:54.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.058+0000 7f3bb4b73640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ba00a2140 0x7f3ba00a4600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4005fd0 tx=0x7f3ba400b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.061+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3ba80b9f40 con 0x7f3bb0102e70 2024-02-09T22:50:54.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.281+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb010cc60 con 0x7f3bb0102e70 2024-02-09T22:50:54.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.282+0000 7f3badffb640 1 -- 172.21.15.3:0/1314864101 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f3ba8044090 con 0x7f3bb0102e70 2024-02-09T22:50:54.286 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:54.286 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:49:13.115988+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:50:54.287 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 24 2024-02-09T22:50:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ba00a2140 msgr2=0x7f3ba00a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ba00a2140 0x7f3ba00a4600 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4005fd0 tx=0x7f3ba400b390 comp rx=0 tx=0).stop 2024-02-09T22:50:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 msgr2=0x7f3bb019e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb019e820 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8037040 tx=0x7f3ba8034d40 comp rx=0 tx=0).stop 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 shutdown_connections 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ba00a2140 0x7f3ba00a4600 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3bb01024b0 0x7f3bb01977b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3bb0102e70 0x7f3bb019e820 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 --2- 172.21.15.3:0/1314864101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3bb01012c0 0x7f3bb0197270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 >> 172.21.15.3:0/1314864101 conn(0x7f3bb00fc850 msgr2=0x7f3bb01057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 shutdown_connections 2024-02-09T22:50:54.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.284+0000 7f3bb6dd7640 1 -- 172.21.15.3:0/1314864101 wait complete. 2024-02-09T22:50:54.299 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:50:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 -- 172.21.15.3:0/3302632228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102500 msgr2=0x7fa8f4102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 --2- 172.21.15.3:0/3302632228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102500 0x7fa8f4102980 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fa8e4009900 tx=0x7fa8e402bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 -- 172.21.15.3:0/3302632228 shutdown_connections 2024-02-09T22:50:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 --2- 172.21.15.3:0/3302632228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f4102ec0 0x7fa8f4109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 --2- 172.21.15.3:0/3302632228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102500 0x7fa8f4102980 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 --2- 172.21.15.3:0/3302632228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f40ff530 0x7fa8f40ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 -- 172.21.15.3:0/3302632228 >> 172.21.15.3:0/3302632228 conn(0x7fa8f40fae60 msgr2=0x7fa8f40fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 -- 172.21.15.3:0/3302632228 shutdown_connections 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 -- 172.21.15.3:0/3302632228 wait complete. 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 Processor -- start 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.398+0000 7fa8fa481640 1 -- start start 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f40ff530 0x7fa8f41972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f4102500 0x7fa8f4197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 0x7fa8f419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f419edc0 con 0x7fa8f4102ec0 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f419ef30 con 0x7fa8f4102500 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f419f0f0 con 0x7fa8f40ff530 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 0x7fa8f419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 0x7fa8f419e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41890/0 (socket says 172.21.15.3:41890) 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 -- 172.21.15.3:0/2616558058 learned_addr learned my addr 172.21.15.3:0/2616558058 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f3fff640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f40ff530 0x7fa8f41972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f37fe640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f4102500 0x7fa8f4197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 -- 172.21.15.3:0/2616558058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f40ff530 msgr2=0x7fa8f41972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f40ff530 0x7fa8f41972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 -- 172.21.15.3:0/2616558058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f4102500 msgr2=0x7fa8f4197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f4102500 0x7fa8f4197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 -- 172.21.15.3:0/2616558058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4009560 con 0x7fa8f4102ec0 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f3fff640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f40ff530 0x7fa8f41972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f37fe640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f4102500 0x7fa8f4197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f8a1e640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 0x7fa8f419e880 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009fd0 tx=0x7fa8ec00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec00bea0 con 0x7fa8f4102ec0 2024-02-09T22:50:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec009940 con 0x7fa8f4102ec0 2024-02-09T22:50:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f419f3d0 con 0x7fa8f4102ec0 2024-02-09T22:50:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0106c0 con 0x7fa8f4102ec0 2024-02-09T22:50:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.399+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f419f890 con 0x7fa8f4102ec0 2024-02-09T22:50:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.400+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f4108630 con 0x7fa8f4102ec0 2024-02-09T22:50:54.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.401+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fa8ec004240 con 0x7fa8f4102ec0 2024-02-09T22:50:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.402+0000 7fa8f17fa640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8e00a20c0 0x7fa8e00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.402+0000 7fa8f3fff640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8e00a20c0 0x7fa8e00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.402+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fa8ec0cdac0 con 0x7fa8f4102ec0 2024-02-09T22:50:54.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.402+0000 7fa8f3fff640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8e00a20c0 0x7fa8e00a4580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4069730 tx=0x7fa8dc005e70 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.407+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa8ec096a40 con 0x7fa8f4102ec0 2024-02-09T22:50:54.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.592+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8f402ce50 con 0x7fa8f4102ec0 2024-02-09T22:50:54.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.594+0000 7fa8f17fa640 1 -- 172.21.15.3:0/2616558058 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7fa8ec09a100 con 0x7fa8f4102ec0 2024-02-09T22:50:54.598 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:50:54.598 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:50:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8e00a20c0 msgr2=0x7fa8e00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8e00a20c0 0x7fa8e00a4580 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa8f4069730 tx=0x7fa8dc005e70 comp rx=0 tx=0).stop 2024-02-09T22:50:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 msgr2=0x7fa8f419e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 0x7fa8f419e880 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009fd0 tx=0x7fa8ec00b260 comp rx=0 tx=0).stop 2024-02-09T22:50:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 shutdown_connections 2024-02-09T22:50:54.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fa8e00a20c0 0x7fa8e00a4580 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8f4102500 0x7fa8f4197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8f4102ec0 0x7fa8f419e880 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 --2- 172.21.15.3:0/2616558058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8f40ff530 0x7fa8f41972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 >> 172.21.15.3:0/2616558058 conn(0x7fa8f40fae60 msgr2=0x7fa8f4105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 shutdown_connections 2024-02-09T22:50:54.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.596+0000 7fa8fa481640 1 -- 172.21.15.3:0/2616558058 wait complete. 2024-02-09T22:50:54.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-02-09T22:49:06.610270+0000', 'modified': '2024-02-09T22:49:13.115988+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24533, 'mds_1': 24503, 'mds_2': 24496, 'mds_3': 24481, 'mds_4': 24475}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14634': {'gid': 14634, 'name': 'g', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.3:6839/3695104270', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6838', 'nonce': 3695104270}, {'type': 'v1', 'addr': '172.21.15.3:6839', 'nonce': 3695104270}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_14646': {'gid': 14646, 'name': 'a', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.3:6841/3916323155', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.3:6840', 'nonce': 3916323155}, {'type': 'v1', 'addr': '172.21.15.3:6841', 'nonce': 3916323155}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24427': {'gid': 24427, 'name': 'l', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.89:6833/2685783987', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6832', 'nonce': 2685783987}, {'type': 'v1', 'addr': '172.21.15.89:6833', 'nonce': 2685783987}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24437': {'gid': 24437, 'name': 'k', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6833/1710343231', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6832', 'nonce': 1710343231}, {'type': 'v1', 'addr': '172.21.15.64:6833', 'nonce': 1710343231}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24470': {'gid': 24470, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.64:6835/3082808593', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6834', 'nonce': 3082808593}, {'type': 'v1', 'addr': '172.21.15.64:6835', 'nonce': 3082808593}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24475': {'gid': 24475, 'name': 'f', 'rank': 4, 'incarnation': 22, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.89:6835/4241363743', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6834', 'nonce': 4241363743}, {'type': 'v1', 'addr': '172.21.15.89:6835', 'nonce': 4241363743}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24481': {'gid': 24481, 'name': 'c', 'rank': 3, 'incarnation': 20, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.89:6837/1935794058', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6836', 'nonce': 1935794058}, {'type': 'v1', 'addr': '172.21.15.89:6837', 'nonce': 1935794058}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24496': {'gid': 24496, 'name': 'i', 'rank': 2, 'incarnation': 18, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.89:6839/305807401', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6838', 'nonce': 305807401}, {'type': 'v1', 'addr': '172.21.15.89:6839', 'nonce': 305807401}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24503': {'gid': 24503, 'name': 'h', 'rank': 1, 'incarnation': 16, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.64:6837/3217313563', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6836', 'nonce': 3217313563}, {'type': 'v1', 'addr': '172.21.15.64:6837', 'nonce': 3217313563}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24533': {'gid': 24533, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.64:6839/4207075446', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.64:6838', 'nonce': 4207075446}, {'type': 'v1', 'addr': '172.21.15.64:6839', 'nonce': 4207075446}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-02-09T22:50:54.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-02-09T22:50:54.613 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 -- 172.21.15.3:0/125448760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc106f30 msgr2=0x7efdfc109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 --2- 172.21.15.3:0/125448760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc106f30 0x7efdfc109380 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009ca0 tx=0x7efdf402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 -- 172.21.15.3:0/125448760 shutdown_connections 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 --2- 172.21.15.3:0/125448760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc104600 0x7efdfc1069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 --2- 172.21.15.3:0/125448760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc106f30 0x7efdfc109380 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 --2- 172.21.15.3:0/125448760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdfc101cd0 0x7efdfc1040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 -- 172.21.15.3:0/125448760 >> 172.21.15.3:0/125448760 conn(0x7efdfc0fb6c0 msgr2=0x7efdfc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 -- 172.21.15.3:0/125448760 shutdown_connections 2024-02-09T22:50:54.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.714+0000 7efe027e2640 1 -- 172.21.15.3:0/125448760 wait complete. 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 Processor -- start 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 -- start start 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdfc101cd0 0x7efdfc197280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 0x7efdfc1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc106f30 0x7efdfc19e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc19ed70 con 0x7efdfc104600 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc19eee0 con 0x7efdfc106f30 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efe027e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc19f0a0 con 0x7efdfc101cd0 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 0x7efdfc1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 0x7efdfc1977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41900/0 (socket says 172.21.15.3:41900) 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.715+0000 7efdfb7fe640 1 -- 172.21.15.3:0/1088939784 learned_addr learned my addr 172.21.15.3:0/1088939784 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efe00d7f640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc106f30 0x7efdfc19e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfbfff640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdfc101cd0 0x7efdfc197280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfb7fe640 1 -- 172.21.15.3:0/1088939784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdfc101cd0 msgr2=0x7efdfc197280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdfc101cd0 0x7efdfc197280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfb7fe640 1 -- 172.21.15.3:0/1088939784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc106f30 msgr2=0x7efdfc19e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc106f30 0x7efdfc19e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfb7fe640 1 -- 172.21.15.3:0/1088939784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4009900 con 0x7efdfc104600 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efe00d7f640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc106f30 0x7efdfc19e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 0x7efdfc1977c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efdf000c970 tx=0x7efdf000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdf97fa640 1 -- 172.21.15.3:0/1088939784 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0007bb0 con 0x7efdfc104600 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdf97fa640 1 -- 172.21.15.3:0/1088939784 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf0007d10 con 0x7efdfc104600 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc19f380 con 0x7efdfc104600 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efdf97fa640 1 -- 172.21.15.3:0/1088939784 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf000f450 con 0x7efdfc104600 2024-02-09T22:50:54.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.716+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc19f890 con 0x7efdfc104600 2024-02-09T22:50:54.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.718+0000 7efdf97fa640 1 -- 172.21.15.3:0/1088939784 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7efdf0004f20 con 0x7efdfc104600 2024-02-09T22:50:54.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 Processor -- start 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 -- start start 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4dd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4e4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d44cb30 con 0x55c49d4e2820 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d44a060 con 0x55c49d4dd6b0 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe053ba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d445840 con 0x55c49d4dccc0 2024-02-09T22:50:54.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdf97fa640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efde40a20c0 0x7efde40a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4e4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe00d7f640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efde40a20c0 0x7efde40a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4e4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41912/0 (socket says 172.21.15.3:41912) 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 -- 172.21.15.3:0/944023579 learned_addr learned my addr 172.21.15.3:0/944023579 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe00d7f640 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4dd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdf97fa640 1 -- 172.21.15.3:0/1088939784 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efdf00cdd70 con 0x7efdfc104600 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfbfff640 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 -- 172.21.15.3:0/944023579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 msgr2=0x55c49d4dd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4dd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 -- 172.21.15.3:0/944023579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 msgr2=0x55c49d4e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 -- 172.21.15.3:0/944023579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c49d40b620 con 0x55c49d4e2820 2024-02-09T22:50:54.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efe00d7f640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efde40a20c0 0x7efde40a4580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdf4006010 tx=0x7efdf4032e80 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.719+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4e4cb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efdf000afd0 tx=0x7efdf006db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53c6abece195f4e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efde27fc640 1 -- 172.21.15.3:0/944023579 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0005600 con 0x55c49d4e2820 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efde27fc640 1 -- 172.21.15.3:0/944023579 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf009b1b0 con 0x55c49d4e2820 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efde27fc640 1 -- 172.21.15.3:0/944023579 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0098db0 con 0x55c49d4e2820 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 -- 172.21.15.3:0/944023579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 msgr2=0x55c49d4e4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4e4cb0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7efdf000afd0 tx=0x7efdf006db70 comp rx=0 tx=0).stop 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 -- 172.21.15.3:0/944023579 shutdown_connections 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4e4cb0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 --2- 172.21.15.3:0/944023579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4dd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 -- 172.21.15.3:0/944023579 >> 172.21.15.3:0/944023579 conn(0x55c49d4d8230 msgr2=0x55c49d4d8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 -- 172.21.15.3:0/944023579 shutdown_connections 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.720+0000 7efe053ba740 1 -- 172.21.15.3:0/944023579 wait complete. 2024-02-09T22:50:54.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 -- start start 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 Processor -- start 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d3eacd0 con 0x55c49d4e2820 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d119310 con 0x55c49d4dccc0 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d3cb490 con 0x55c49d4dd6b0 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe053ba740 10 client.0 _ll_register_callbacks cb 0x55c49d156ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4ee0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41914/0 (socket says 172.21.15.3:41914) 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 -- 172.21.15.3:0/2273740137 learned_addr learned my addr 172.21.15.3:0/2273740137 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efdfbfff640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 -- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dd6b0 msgr2=0x55c49d4e7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 -- 172.21.15.3:0/2273740137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dccc0 msgr2=0x55c49d4e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efe00d7f640 1 -- 172.21.15.3:0/2273740137 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc19f890 con 0x55c49d4e2820 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.721+0000 7efdfbfff640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe00d7f640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4ee0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009ca0 tx=0x7efdf4046040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b6afd938cde7ee server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efddb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efddb7fe640 1 -- 172.21.15.3:0/2273740137 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4034660 con 0x55c49d4e2820 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efddb7fe640 1 -- 172.21.15.3:0/2273740137 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efdf4004450 con 0x55c49d4e2820 2024-02-09T22:50:54.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c49d1456c0 con 0x55c49d4e2820 2024-02-09T22:50:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efddb7fe640 1 -- 172.21.15.3:0/2273740137 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf40323f0 con 0x55c49d4e2820 2024-02-09T22:50:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 10 client.15261 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c49d443ec0 con 0x55c49d4e2820 2024-02-09T22:50:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efddb7fe640 1 -- 172.21.15.3:0/2273740137 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c49d1456c0 con 0x55c49d4e2820 2024-02-09T22:50:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efddb7fe640 1 -- 172.21.15.3:0/2273740137 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdf404e070 con 0x55c49d4e2820 2024-02-09T22:50:54.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 10 client.15261 resolve_mds: resolved name 'f' to daemon mds.f 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 20 client.15261 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d4f3540 0x55c49d4f2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 4 client.15261 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.722+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c49d196270 con 0x55c49d4f3540 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.723+0000 7efdfbfff640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d4f3540 0x55c49d4f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.723+0000 7efddb7fe640 10 client.15261 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.723+0000 7efdfbfff640 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d4f3540 0x55c49d4f2790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=119a1956f1836d9 server_cookie=9a40f9ed56af30ec in_seq=0 out_seq=0 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efddb7fe640 1 -- 172.21.15.3:0/2273740137 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c49d196270 con 0x55c49d4f3540 2024-02-09T22:50:54.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efddb7fe640 10 client.15261 handle_command_reply: tid=0 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 client.15261 shutdown 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 msgr2=0x55c49d4ee0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4ee0c0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009ca0 tx=0x7efdf4046040 comp rx=0 tx=0).stop 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 shutdown_connections 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d4f3540 0x55c49d4f2790 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d4ee0c0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efddb7fe640 0 client.15261 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 --2- 172.21.15.3:0/2273740137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 >> 172.21.15.3:0/2273740137 conn(0x55c49d4d8230 msgr2=0x55c49d4d7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 shutdown_connections 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 1 -- 172.21.15.3:0/2273740137 wait complete. 2024-02-09T22:50:54.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.724+0000 7efe053ba740 20 client.15261 trim_cache size 0 max 16384 2024-02-09T22:50:54.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 Processor -- start 2024-02-09T22:50:54.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 -- start start 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4e2820 0x55c49d4ee080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d4d3b00 con 0x55c49d4dd6b0 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d196270 con 0x55c49d4e2820 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efe053ba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d1456c0 con 0x55c49d4dccc0 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41918/0 (socket says 172.21.15.3:41918) 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 -- 172.21.15.3:0/632333760 learned_addr learned my addr 172.21.15.3:0/632333760 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 -- 172.21.15.3:0/632333760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 msgr2=0x55c49d4e6b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfbfff640 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 -- 172.21.15.3:0/632333760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4e2820 msgr2=0x55c49d4ee080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4e2820 0x55c49d4ee080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 -- 172.21.15.3:0/632333760 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c49d3e0a90 con 0x55c49d4dd6b0 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfbfff640 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efdf00ce0e0 tx=0x7efdf00a3560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=465e624ba3965b1a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efde1ffb640 1 -- 172.21.15.3:0/632333760 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0007e80 con 0x55c49d4dd6b0 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.740+0000 7efde1ffb640 1 -- 172.21.15.3:0/632333760 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf0005600 con 0x55c49d4dd6b0 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efde1ffb640 1 -- 172.21.15.3:0/632333760 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0097120 con 0x55c49d4dd6b0 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 -- 172.21.15.3:0/632333760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 msgr2=0x55c49d4e7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efdf00ce0e0 tx=0x7efdf00a3560 comp rx=0 tx=0).stop 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 -- 172.21.15.3:0/632333760 shutdown_connections 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4e2820 0x55c49d4ee080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4dd6b0 0x55c49d4e7080 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 --2- 172.21.15.3:0/632333760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4e6b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 -- 172.21.15.3:0/632333760 >> 172.21.15.3:0/632333760 conn(0x55c49d4d8230 msgr2=0x55c49d4e5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 -- 172.21.15.3:0/632333760 shutdown_connections 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 -- 172.21.15.3:0/632333760 wait complete. 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.741+0000 7efe053ba740 1 -- start start 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 Processor -- start 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4eaff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4eb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d549140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d4ebb40 con 0x55c49d4e2820 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d549680 con 0x55c49d4dd6b0 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49d5497c0 con 0x55c49d4dccc0 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 10 client.0 _ll_register_callbacks cb 0x7efdf00046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d549140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d549140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41930/0 (socket says 172.21.15.3:41930) 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 -- 172.21.15.3:0/4039679395 learned_addr learned my addr 172.21.15.3:0/4039679395 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 -- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 msgr2=0x55c49d4eaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efdfbfff640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4eaff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4eaff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 -- 172.21.15.3:0/4039679395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 msgr2=0x55c49d4eb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4eb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 -- 172.21.15.3:0/4039679395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c49d40b620 con 0x55c49d4e2820 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efdfb7fe640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe00d7f640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d549140 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efdf40348f0 tx=0x7efdf4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff01c33c9d6e7e58 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efddb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efddb7fe640 1 -- 172.21.15.3:0/4039679395 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4003d40 con 0x55c49d4e2820 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efddb7fe640 1 -- 172.21.15.3:0/4039679395 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7efdf4034660 con 0x55c49d4e2820 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.742+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c49d4f3250 con 0x55c49d4e2820 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efddb7fe640 1 -- 172.21.15.3:0/4039679395 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf4034da0 con 0x55c49d4e2820 2024-02-09T22:50:54.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 10 client.15267 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x55c49d406930 con 0x55c49d4e2820 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efddb7fe640 1 -- 172.21.15.3:0/4039679395 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c49d4f3250 con 0x55c49d4e2820 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efddb7fe640 1 -- 172.21.15.3:0/4039679395 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdf403e040 con 0x55c49d4e2820 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 10 client.15267 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 10 client.15267 resolve_mds: resolved name 'f' to daemon mds.f 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 20 client.15267 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d54e710 0x55c49d54abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 4 client.15267 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55c49d4f4570 con 0x55c49d54e710 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efdfbfff640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d54e710 0x55c49d54abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efddb7fe640 10 client.15267 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.743+0000 7efdfbfff640 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d54e710 0x55c49d54abb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=39fdb70998018407 server_cookie=28d6ddefc1ae62eb in_seq=0 out_seq=0 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efddb7fe640 1 -- 172.21.15.3:0/4039679395 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55c49d4f4570 con 0x55c49d54e710 2024-02-09T22:50:54.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efddb7fe640 10 client.15267 handle_command_reply: tid=0 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 client.15267 shutdown 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 msgr2=0x55c49d549140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d549140 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7efdf40348f0 tx=0x7efdf4004a40 comp rx=0 tx=0).stop 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 shutdown_connections 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55c49d54e710 0x55c49d54abb0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c49d4dd6b0 0x55c49d4eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c49d4e2820 0x55c49d549140 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 --2- 172.21.15.3:0/4039679395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c49d4dccc0 0x55c49d4eaff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 >> 172.21.15.3:0/4039679395 conn(0x55c49d4d8230 msgr2=0x55c49d4e6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efddb7fe640 0 client.15267 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 shutdown_connections 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 1 -- 172.21.15.3:0/4039679395 wait complete. 2024-02-09T22:50:54.757 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe053ba740 20 client.15267 trim_cache size 0 max 16384 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 4, 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24475, 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 102.63296808699999, 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:54.758 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:54.759 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:54.759 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 129.680285008 2024-02-09T22:50:54.759 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efde40a20c0 msgr2=0x7efde40a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe027e2640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efde40a20c0 0x7efde40a4580 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7efdf4006010 tx=0x7efdf4032e80 comp rx=0 tx=0).stop 2024-02-09T22:50:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 msgr2=0x7efdfc1977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.744+0000 7efe027e2640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 0x7efdfc1977c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7efdf000c970 tx=0x7efdf000ce30 comp rx=0 tx=0).stop 2024-02-09T22:50:54.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 shutdown_connections 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7efde40a20c0 0x7efde40a4580 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdfc106f30 0x7efdfc19e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdfc104600 0x7efdfc1977c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 --2- 172.21.15.3:0/1088939784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdfc101cd0 0x7efdfc197280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 >> 172.21.15.3:0/1088939784 conn(0x7efdfc0fb6c0 msgr2=0x7efdfc102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 shutdown_connections 2024-02-09T22:50:54.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.745+0000 7efe027e2640 1 -- 172.21.15.3:0/1088939784 wait complete. 2024-02-09T22:50:54.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-02-09T22:50:54.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 -- 172.21.15.3:0/4137049533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 msgr2=0x7feeb4108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 --2- 172.21.15.3:0/4137049533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb4108590 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7feeac009ca0 tx=0x7feeac02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 -- 172.21.15.3:0/4137049533 shutdown_connections 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 --2- 172.21.15.3:0/4137049533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4101340 0x7feeb41017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 --2- 172.21.15.3:0/4137049533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb4108590 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 --2- 172.21.15.3:0/4137049533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb4100150 0x7feeb4100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 -- 172.21.15.3:0/4137049533 >> 172.21.15.3:0/4137049533 conn(0x7feeb40fb6c0 msgr2=0x7feeb40fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 -- 172.21.15.3:0/4137049533 shutdown_connections 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.861+0000 7feebbe9d640 1 -- 172.21.15.3:0/4137049533 wait complete. 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 Processor -- start 2024-02-09T22:50:54.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 -- start start 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4100150 0x7feeb406a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb4101340 0x7feeb419bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb419e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb419edf0 con 0x7feeb4101d00 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb419ef60 con 0x7feeb4100150 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feebbe9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb419f120 con 0x7feeb4101340 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb419e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb419e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41950/0 (socket says 172.21.15.3:41950) 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 -- 172.21.15.3:0/2688677196 learned_addr learned my addr 172.21.15.3:0/2688677196 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeb9c39640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4100150 0x7feeb406a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 -- 172.21.15.3:0/2688677196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb4101340 msgr2=0x7feeb419bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb4101340 0x7feeb419bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 -- 172.21.15.3:0/2688677196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4100150 msgr2=0x7feeb406a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4100150 0x7feeb406a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 -- 172.21.15.3:0/2688677196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac009900 con 0x7feeb4101d00 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeb9c39640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4100150 0x7feeb406a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.862+0000 7feeba43a640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb419e610 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feeac037040 tx=0x7feeac034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.863+0000 7feeaaffd640 1 -- 172.21.15.3:0/2688677196 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac0329b0 con 0x7feeb4101d00 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.863+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb419f340 con 0x7feeb4101d00 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.863+0000 7feeaaffd640 1 -- 172.21.15.3:0/2688677196 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feeac003e50 con 0x7feeb4101d00 2024-02-09T22:50:54.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.863+0000 7feeaaffd640 1 -- 172.21.15.3:0/2688677196 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeac040450 con 0x7feeb4101d00 2024-02-09T22:50:54.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.863+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb419f840 con 0x7feeb4101d00 2024-02-09T22:50:54.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.865+0000 7feeaaffd640 1 -- 172.21.15.3:0/2688677196 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7feeac00a040 con 0x7feeb4101d00 2024-02-09T22:50:54.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.865+0000 7feebea73740 1 Processor -- start 2024-02-09T22:50:54.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.865+0000 7feebea73740 1 -- start start 2024-02-09T22:50:54.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feebea73740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332dd00e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feebea73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd06b0 0x564332dd51b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeaaffd640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feea00a20c0 0x7feea00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feebea73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332dd7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feebea73740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332d3fb30 con 0x564332dcfcc0 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feebea73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332d3d060 con 0x564332dd06b0 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feebea73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332d38840 con 0x564332dd5820 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332dd00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeba43a640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feea00a20c0 0x7feea00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332dd00e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41952/0 (socket says 172.21.15.3:41952) 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 -- 172.21.15.3:0/1563622874 learned_addr learned my addr 172.21.15.3:0/1563622874 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9c39640 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd06b0 0x564332dd51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeaaffd640 1 -- 172.21.15.3:0/2688677196 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feeac0f0cb0 con 0x7feeb4101d00 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332dd7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 -- 172.21.15.3:0/1563622874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 msgr2=0x564332dd7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332dd7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 -- 172.21.15.3:0/1563622874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd06b0 msgr2=0x564332dd51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd06b0 0x564332dd51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 -- 172.21.15.3:0/1563622874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564332cfe620 con 0x564332dcfcc0 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeba43a640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feea00a20c0 0x7feea00a4580 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feeac0c5c00 tx=0x7feeac088a30 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7feeb9438640 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332dd00e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7feea4009fc0 tx=0x7feea40094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=826eaad8556c3bc5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.866+0000 7fee9ffff640 1 -- 172.21.15.3:0/1563622874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4005220 con 0x564332dcfcc0 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7fee9ffff640 1 -- 172.21.15.3:0/1563622874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea4018030 con 0x564332dcfcc0 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7fee9ffff640 1 -- 172.21.15.3:0/1563622874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4012650 con 0x564332dcfcc0 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 -- 172.21.15.3:0/1563622874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 msgr2=0x564332dd00e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332dd00e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7feea4009fc0 tx=0x7feea40094e0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 -- 172.21.15.3:0/1563622874 shutdown_connections 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd06b0 0x564332dd51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332dd7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 --2- 172.21.15.3:0/1563622874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332dd00e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 -- 172.21.15.3:0/1563622874 >> 172.21.15.3:0/1563622874 conn(0x564332dcb230 msgr2=0x564332dcb630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 -- 172.21.15.3:0/1563622874 shutdown_connections 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.867+0000 7feebea73740 1 -- 172.21.15.3:0/1563622874 wait complete. 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 -- start start 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 Processor -- start 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dcfcc0 0x564332dd9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dda120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 0x564332de1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332cddcd0 con 0x564332dd06b0 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332a0c310 con 0x564332dd5820 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332cbe490 con 0x564332dcfcc0 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feebea73740 10 client.0 _ll_register_callbacks cb 0x564332a49ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dda120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dda120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41966/0 (socket says 172.21.15.3:41966) 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 -- 172.21.15.3:0/2027741307 learned_addr learned my addr 172.21.15.3:0/2027741307 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9438640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dcfcc0 0x564332dd9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 -- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dcfcc0 msgr2=0x564332dd9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dcfcc0 0x564332dd9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 -- 172.21.15.3:0/2027741307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 msgr2=0x564332de1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 0x564332de1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9c39640 1 -- 172.21.15.3:0/2027741307 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea4009bf0 con 0x564332dd06b0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.868+0000 7feeb9438640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dcfcc0 0x564332dd9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feeb9c39640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dda120 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feeb00126b0 tx=0x7feeb0012b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2012c7fd61c4ea54 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7fee9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7fee9d7fa640 1 -- 172.21.15.3:0/2027741307 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0017070 con 0x564332dd06b0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7fee9d7fa640 1 -- 172.21.15.3:0/2027741307 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feeb0004320 con 0x564332dd06b0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564332a386c0 con 0x564332dd06b0 2024-02-09T22:50:54.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7fee9d7fa640 1 -- 172.21.15.3:0/2027741307 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb00044e0 con 0x564332dd06b0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 10 client.15279 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564332d36ec0 con 0x564332dd06b0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7fee9d7fa640 1 -- 172.21.15.3:0/2027741307 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564332a386c0 con 0x564332dd06b0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7fee9d7fa640 1 -- 172.21.15.3:0/2027741307 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feeb0025070 con 0x564332dd06b0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 10 client.15279 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 10 client.15279 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 20 client.15279 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332de65f0 0x564332de5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 4 client.15279 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.869+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564332a89270 con 0x564332de65f0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.870+0000 7feeb9438640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332de65f0 0x564332de5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.870+0000 7feeb9438640 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332de65f0 0x564332de5890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9fd6a01307f6afa3 server_cookie=17937798c6d43710 in_seq=0 out_seq=0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.870+0000 7fee9d7fa640 10 client.15279 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7fee9d7fa640 1 -- 172.21.15.3:0/2027741307 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564332a89270 con 0x564332de65f0 2024-02-09T22:50:54.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7fee9d7fa640 10 client.15279 handle_command_reply: tid=0 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 client.15279 shutdown 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 msgr2=0x564332dda120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dda120 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7feeb00126b0 tx=0x7feeb0012b70 comp rx=0 tx=0).stop 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 shutdown_connections 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332de65f0 0x564332de5890 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 0x564332de1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dda120 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7fee9d7fa640 0 client.15279 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 --2- 172.21.15.3:0/2027741307 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dcfcc0 0x564332dd9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 >> 172.21.15.3:0/2027741307 conn(0x564332dcb230 msgr2=0x564332dca510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 shutdown_connections 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 1 -- 172.21.15.3:0/2027741307 wait complete. 2024-02-09T22:50:54.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.871+0000 7feebea73740 20 client.15279 trim_cache size 0 max 16384 2024-02-09T22:50:54.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:54 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/599564791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:54 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3541204736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:54 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1314864101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:54.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:54 smithi003 ceph-mon[32678]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:54.884 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:54 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2616558058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:54.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 Processor -- start 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 -- start start 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332deea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd06b0 0x564332de1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 0x564332dc9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332dcbd20 con 0x564332dcfcc0 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332dd8370 con 0x564332dd5820 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feebea73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332a89270 con 0x564332dd06b0 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332deea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332deea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41978/0 (socket says 172.21.15.3:41978) 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 -- 172.21.15.3:0/3990432627 learned_addr learned my addr 172.21.15.3:0/3990432627 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 -- 172.21.15.3:0/3990432627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd06b0 msgr2=0x564332de1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd06b0 0x564332de1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 -- 172.21.15.3:0/3990432627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 msgr2=0x564332dc9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 0x564332dc9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 -- 172.21.15.3:0/3990432627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564332cd3a90 con 0x564332dcfcc0 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7feeb9438640 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332deea90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feea4009350 tx=0x7feea4009f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8a2a39898f0264 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7fee9f7fe640 1 -- 172.21.15.3:0/3990432627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4025d70 con 0x564332dcfcc0 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7fee9f7fe640 1 -- 172.21.15.3:0/3990432627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea4024040 con 0x564332dcfcc0 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.887+0000 7fee9f7fe640 1 -- 172.21.15.3:0/3990432627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea401d910 con 0x564332dcfcc0 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 -- 172.21.15.3:0/3990432627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 msgr2=0x564332deea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332deea90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7feea4009350 tx=0x7feea4009f50 comp rx=0 tx=0).stop 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 -- 172.21.15.3:0/3990432627 shutdown_connections 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dd5820 0x564332dc9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd06b0 0x564332de1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 --2- 172.21.15.3:0/3990432627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dcfcc0 0x564332deea90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 -- 172.21.15.3:0/3990432627 >> 172.21.15.3:0/3990432627 conn(0x564332dcb230 msgr2=0x564332dcb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 -- 172.21.15.3:0/3990432627 shutdown_connections 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 -- 172.21.15.3:0/3990432627 wait complete. 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 -- start start 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.888+0000 7feebea73740 1 Processor -- start 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dcfcc0 0x564332dde020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dde560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332e3c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332ddeba0 con 0x564332dd06b0 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332e3c730 con 0x564332dcfcc0 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564332e3c870 con 0x564332dd5820 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 10 client.0 _ll_register_callbacks cb 0x7feea4009380 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dde560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dde560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41988/0 (socket says 172.21.15.3:41988) 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 -- 172.21.15.3:0/1620585609 learned_addr learned my addr 172.21.15.3:0/1620585609 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:54.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9438640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dcfcc0 0x564332dde020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9438640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332e3c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 -- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 msgr2=0x564332e3c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332e3c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 -- 172.21.15.3:0/1620585609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dcfcc0 msgr2=0x564332dde020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dcfcc0 0x564332dde020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 -- 172.21.15.3:0/1620585609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564332cfe620 con 0x564332dd06b0 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9438640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dcfcc0 0x564332dde020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feeb9c39640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dde560 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feeb0004020 tx=0x7feeb0005570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf45de280ae58e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7fee9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:54.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7fee9d7fa640 1 -- 172.21.15.3:0/1620585609 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb0017040 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7fee9d7fa640 1 -- 172.21.15.3:0/1620585609 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7feeb0000e40 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.889+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564332de7620 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7fee9d7fa640 1 -- 172.21.15.3:0/1620585609 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7feeb0003ac0 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 10 client.15285 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x564332cf9930 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7fee9d7fa640 1 -- 172.21.15.3:0/1620585609 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564332de7620 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7fee9d7fa640 1 -- 172.21.15.3:0/1620585609 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7feeb0025070 con 0x564332dd06b0 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 10 client.15285 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 10 client.15285 resolve_mds: resolved name 'c' to daemon mds.c 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 20 client.15285 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332e41b00 0x564332e3dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 4 client.15285 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7feea4018030 con 0x564332e41b00 2024-02-09T22:50:54.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.890+0000 7feeb9438640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332e41b00 0x564332e3dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7fee9d7fa640 10 client.15285 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feeb9438640 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332e41b00 0x564332e3dfc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3cc3165fe4d85960 server_cookie=f0e671748eaf326a in_seq=0 out_seq=0 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7fee9d7fa640 1 -- 172.21.15.3:0/1620585609 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x7feea4018030 con 0x564332e41b00 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7fee9d7fa640 10 client.15285 handle_command_reply: tid=0 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 client.15285 shutdown 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 msgr2=0x564332dde560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dde560 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7feeb0004020 tx=0x7feeb0005570 comp rx=0 tx=0).stop 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 shutdown_connections 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564332e41b00 0x564332e3dfc0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564332dd5820 0x564332e3c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564332dd06b0 0x564332dde560 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 --2- 172.21.15.3:0/1620585609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564332dcfcc0 0x564332dde020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7fee9d7fa640 0 client.15285 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 >> 172.21.15.3:0/1620585609 conn(0x564332dcb230 msgr2=0x564332deea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 shutdown_connections 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 1 -- 172.21.15.3:0/1620585609 wait complete. 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.891+0000 7feebea73740 20 client.15285 trim_cache size 0 max 16384 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 3, 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24481, 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 103.784888258, 2024-02-09T22:50:54.903 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:54.904 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 127.140673189 2024-02-09T22:50:54.904 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feea00a20c0 msgr2=0x7feea00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feea00a20c0 0x7feea00a4580 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feeac0c5c00 tx=0x7feeac088a30 comp rx=0 tx=0).stop 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 msgr2=0x7feeb419e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb419e610 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7feeac037040 tx=0x7feeac034a90 comp rx=0 tx=0).stop 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 shutdown_connections 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7feea00a20c0 0x7feea00a4580 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb4101d00 0x7feeb419e610 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb4101340 0x7feeb419bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 --2- 172.21.15.3:0/2688677196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb4100150 0x7feeb406a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:54.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 >> 172.21.15.3:0/2688677196 conn(0x7feeb40fb6c0 msgr2=0x7feeb4104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:54.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 shutdown_connections 2024-02-09T22:50:54.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:54.892+0000 7feebbe9d640 1 -- 172.21.15.3:0/2688677196 wait complete. 2024-02-09T22:50:54.907 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-02-09T22:50:54.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:54 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/599564791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:54 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3541204736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:54 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1314864101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:54.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:54 smithi064 ceph-mon[36638]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:54.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:54 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2616558058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:54.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:54 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/599564791' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:54 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3541204736' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:50:54.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:54 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1314864101' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:50:54.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:54 smithi089 ceph-mon[36665]: pgmap v342: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:54.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:54 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2616558058' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:50:55.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 -- 172.21.15.3:0/2824714565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ff2b0 msgr2=0x7f99780ff730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 --2- 172.21.15.3:0/2824714565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ff2b0 0x7f99780ff730 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f9968009900 tx=0x7f996802bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:55.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 -- 172.21.15.3:0/2824714565 shutdown_connections 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 --2- 172.21.15.3:0/2824714565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ffc70 0x7f9978109820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 --2- 172.21.15.3:0/2824714565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ff2b0 0x7f99780ff730 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 --2- 172.21.15.3:0/2824714565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9978102840 0x7f9978102c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 -- 172.21.15.3:0/2824714565 >> 172.21.15.3:0/2824714565 conn(0x7f99780fae50 msgr2=0x7f99780fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 -- 172.21.15.3:0/2824714565 shutdown_connections 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.006+0000 7f997f4c0640 1 -- 172.21.15.3:0/2824714565 wait complete. 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 Processor -- start 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 -- start start 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ff2b0 0x7f99781972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 0x7f9978197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9978102840 0x7f997819e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997819ede0 con 0x7f99780ffc70 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997819ef50 con 0x7f99780ff2b0 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997f4c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997819f110 con 0x7f9978102840 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 0x7f9978197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 0x7f9978197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42000/0 (socket says 172.21.15.3:42000) 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 -- 172.21.15.3:0/1945927226 learned_addr learned my addr 172.21.15.3:0/1945927226 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997da5d640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9978102840 0x7f997819e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997d25c640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ff2b0 0x7f99781972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 -- 172.21.15.3:0/1945927226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9978102840 msgr2=0x7f997819e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9978102840 0x7f997819e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 -- 172.21.15.3:0/1945927226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ff2b0 msgr2=0x7f99781972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ff2b0 0x7f99781972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 -- 172.21.15.3:0/1945927226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968009560 con 0x7f99780ffc70 2024-02-09T22:50:55.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997d25c640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ff2b0 0x7f99781972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:55.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 0x7f9978197830 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9968006010 tx=0x7f99680042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.007+0000 7f996e7fc640 1 -- 172.21.15.3:0/1945927226 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680044e0 con 0x7f99780ffc70 2024-02-09T22:50:55.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.008+0000 7f996e7fc640 1 -- 172.21.15.3:0/1945927226 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99680054e0 con 0x7f99780ffc70 2024-02-09T22:50:55.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.008+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997819f390 con 0x7f99780ffc70 2024-02-09T22:50:55.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.008+0000 7f996e7fc640 1 -- 172.21.15.3:0/1945927226 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996803e450 con 0x7f99780ffc70 2024-02-09T22:50:55.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.008+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997819f8a0 con 0x7f99780ffc70 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f996e7fc640 1 -- 172.21.15.3:0/1945927226 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f9968003680 con 0x7f99780ffc70 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 Processor -- start 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 -- start start 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f996e7fc640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f99640a20c0 0x7f99640a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9afecc0 0x5599b9aff0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b041b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b06cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9a6eb30 con 0x5599b9aff6b0 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9a6c060 con 0x5599b9afecc0 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.010+0000 7f9982169740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9a67840 con 0x5599b9b04820 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997da5d640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f99640a20c0 0x7f99640a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b041b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42010/0 (socket says 172.21.15.3:42010) 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 -- 172.21.15.3:0/4224975058 learned_addr learned my addr 172.21.15.3:0/4224975058 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f996e7fc640 1 -- 172.21.15.3:0/1945927226 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f99680f0140 con 0x7f99780ffc70 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997da5d640 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b06cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997d25c640 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9afecc0 0x5599b9aff0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 -- 172.21.15.3:0/4224975058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 msgr2=0x5599b9b06cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b06cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 -- 172.21.15.3:0/4224975058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9afecc0 msgr2=0x5599b9aff0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9afecc0 0x5599b9aff0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 -- 172.21.15.3:0/4224975058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599b9a2d620 con 0x5599b9aff6b0 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997da5d640 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b06cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997da5d640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f99640a20c0 0x7f99640a4580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9970005c40 tx=0x7f99700129e0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f997ca5b640 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b041b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f996804eee0 tx=0x7f996808ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67c812516169fc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f99637fe640 1 -- 172.21.15.3:0/4224975058 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680bc7e0 con 0x5599b9aff6b0 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f99637fe640 1 -- 172.21.15.3:0/4224975058 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99680bd580 con 0x5599b9aff6b0 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.011+0000 7f99637fe640 1 -- 172.21.15.3:0/4224975058 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680c0fd0 con 0x5599b9aff6b0 2024-02-09T22:50:55.024 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 -- 172.21.15.3:0/4224975058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 msgr2=0x5599b9b041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b041b0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f996804eee0 tx=0x7f996808ee90 comp rx=0 tx=0).stop 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 -- 172.21.15.3:0/4224975058 shutdown_connections 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b06cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b041b0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 --2- 172.21.15.3:0/4224975058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9afecc0 0x5599b9aff0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 -- 172.21.15.3:0/4224975058 >> 172.21.15.3:0/4224975058 conn(0x5599b9afa230 msgr2=0x5599b9afa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 -- 172.21.15.3:0/4224975058 shutdown_connections 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 -- 172.21.15.3:0/4224975058 wait complete. 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 -- start start 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.012+0000 7f9982169740 1 Processor -- start 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 0x5599b9b087d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b08d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b0fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9a0ccd0 con 0x5599b9afecc0 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b973b310 con 0x5599b9aff6b0 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b99ed490 con 0x5599b9b04820 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9982169740 10 client.0 _ll_register_callbacks cb 0x5599b9778ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 0x5599b9b087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 0x5599b9b087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42020/0 (socket says 172.21.15.3:42020) 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 -- 172.21.15.3:0/1453485527 learned_addr learned my addr 172.21.15.3:0/1453485527 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b08d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 -- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 msgr2=0x5599b9b0fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b0fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 -- 172.21.15.3:0/1453485527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 msgr2=0x5599b9b08d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b08d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 -- 172.21.15.3:0/1453485527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997819f8a0 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b08d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f997d25c640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 0x5599b9b087d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9974010670 tx=0x7f9974010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e3a51fd94d31b45 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9960ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.013+0000 7f9960ff9640 1 -- 172.21.15.3:0/1453485527 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974005220 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599b97676c0 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9960ff9640 1 -- 172.21.15.3:0/1453485527 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9974010e70 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9960ff9640 1 -- 172.21.15.3:0/1453485527 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9974005380 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 10 client.15297 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5599b9a65ec0 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9960ff9640 1 -- 172.21.15.3:0/1453485527 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5599b97676c0 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9960ff9640 1 -- 172.21.15.3:0/1453485527 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9974025070 con 0x5599b9afecc0 2024-02-09T22:50:55.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 10 client.15297 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 10 client.15297 resolve_mds: resolved name 'i' to daemon mds.i 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 20 client.15297 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b152b0 0x5599b9b144d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 4 client.15297 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599b97b8270 con 0x5599b9b152b0 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.014+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b152b0 0x5599b9b144d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.015+0000 7f9960ff9640 10 client.15297 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.015+0000 7f997ca5b640 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b152b0 0x5599b9b144d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1151e7826b2506a4 server_cookie=5a097e7462758e21 in_seq=0 out_seq=0 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.015+0000 7f9960ff9640 1 -- 172.21.15.3:0/1453485527 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5599b97b8270 con 0x5599b9b152b0 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.015+0000 7f9960ff9640 10 client.15297 handle_command_reply: tid=0 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.015+0000 7f9982169740 1 client.15297 shutdown 2024-02-09T22:50:55.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 msgr2=0x5599b9b087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 0x5599b9b087d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9974010670 tx=0x7f9974010b30 comp rx=0 tx=0).stop 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 shutdown_connections 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b152b0 0x5599b9b144d0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b08d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9b04820 0x5599b9b0fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9960ff9640 0 client.15297 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 --2- 172.21.15.3:0/1453485527 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9afecc0 0x5599b9b087d0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 >> 172.21.15.3:0/1453485527 conn(0x5599b9afa230 msgr2=0x5599b9af9510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 shutdown_connections 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 1 -- 172.21.15.3:0/1453485527 wait complete. 2024-02-09T22:50:55.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.016+0000 7f9982169740 20 client.15297 trim_cache size 0 max 16384 2024-02-09T22:50:55.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.031+0000 7f9982169740 1 Processor -- start 2024-02-09T22:50:55.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.031+0000 7f9982169740 1 -- start start 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9982169740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b1d750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9982169740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b0fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9982169740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9b04820 0x5599b9af8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9982169740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9b045a0 con 0x5599b9aff6b0 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9982169740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b97b8270 con 0x5599b9b04820 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9982169740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b97676c0 con 0x5599b9afecc0 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b0fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b0fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42030/0 (socket says 172.21.15.3:42030) 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 -- 172.21.15.3:0/909334821 learned_addr learned my addr 172.21.15.3:0/909334821 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997d25c640 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b1d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997da5d640 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9b04820 0x5599b9af8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 -- 172.21.15.3:0/909334821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 msgr2=0x5599b9b1d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b1d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 -- 172.21.15.3:0/909334821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9b04820 msgr2=0x5599b9af8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9b04820 0x5599b9af8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 -- 172.21.15.3:0/909334821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599b9a02a90 con 0x5599b9aff6b0 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997da5d640 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9b04820 0x5599b9af8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f997ca5b640 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b0fc70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f99680c1190 tx=0x7f99680bcd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6585462e8b9d5d9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9962ffd640 1 -- 172.21.15.3:0/909334821 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680bce80 con 0x5599b9aff6b0 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9962ffd640 1 -- 172.21.15.3:0/909334821 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9968005650 con 0x5599b9aff6b0 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.032+0000 7f9962ffd640 1 -- 172.21.15.3:0/909334821 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99680cc510 con 0x5599b9aff6b0 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 -- 172.21.15.3:0/909334821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 msgr2=0x5599b9b0fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b0fc70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f99680c1190 tx=0x7f99680bcd50 comp rx=0 tx=0).stop 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 -- 172.21.15.3:0/909334821 shutdown_connections 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9b04820 0x5599b9af8c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9aff6b0 0x5599b9b0fc70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 --2- 172.21.15.3:0/909334821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b1d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 -- 172.21.15.3:0/909334821 >> 172.21.15.3:0/909334821 conn(0x5599b9afa230 msgr2=0x5599b9b03e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 -- 172.21.15.3:0/909334821 shutdown_connections 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 -- 172.21.15.3:0/909334821 wait complete. 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 -- start start 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.033+0000 7f9982169740 1 Processor -- start 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b0ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b0cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 0x5599b9b6aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9b0ff40 con 0x5599b9b04820 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9b100b0 con 0x5599b9aff6b0 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599b9b6b3f0 con 0x5599b9afecc0 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 10 client.0 _ll_register_callbacks cb 0x7f9968003f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 0x5599b9b6aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 0x5599b9b6aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42036/0 (socket says 172.21.15.3:42036) 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 -- 172.21.15.3:0/861838718 learned_addr learned my addr 172.21.15.3:0/861838718 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997d25c640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b0ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997ca5b640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b0cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 -- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 msgr2=0x5599b9b0ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b0ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 -- 172.21.15.3:0/861838718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 msgr2=0x5599b9b0cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b0cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 -- 172.21.15.3:0/861838718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599b9a2d620 con 0x5599b9b04820 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f997da5d640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 0x5599b9b6aeb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9970010d90 tx=0x7f9970004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f86e4d46429593 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9960ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9960ff9640 1 -- 172.21.15.3:0/861838718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f997000ab00 con 0x5599b9b04820 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9960ff9640 1 -- 172.21.15.3:0/861838718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9970027400 con 0x5599b9b04820 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.034+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99680bd580 con 0x5599b9b04820 2024-02-09T22:50:55.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9960ff9640 1 -- 172.21.15.3:0/861838718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f997002d030 con 0x5599b9b04820 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 10 client.15303 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5599b9a28930 con 0x5599b9b04820 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9960ff9640 1 -- 172.21.15.3:0/861838718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99680bd580 con 0x5599b9b04820 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9960ff9640 1 -- 172.21.15.3:0/861838718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9970025020 con 0x5599b9b04820 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 10 client.15303 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 20 client.15303 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b70610 0x5599b9b6ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 4 client.15303 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5599b97a4610 con 0x5599b9b70610 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.035+0000 7f997d25c640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b70610 0x5599b9b6ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9960ff9640 10 client.15303 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f997d25c640 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b70610 0x5599b9b6ca70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65a0d6ed2e7b17ff server_cookie=4dc1a271901a5faa in_seq=0 out_seq=0 2024-02-09T22:50:55.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9960ff9640 1 -- 172.21.15.3:0/861838718 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5599b97a4610 con 0x5599b9b70610 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9960ff9640 10 client.15303 handle_command_reply: tid=0 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 client.15303 shutdown 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 msgr2=0x5599b9b6aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 0x5599b9b6aeb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f9970010d90 tx=0x7f9970004a40 comp rx=0 tx=0).stop 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 shutdown_connections 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5599b9b70610 0x5599b9b6ca70 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599b9aff6b0 0x5599b9b0cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599b9b04820 0x5599b9b6aeb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9960ff9640 0 client.15303 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 --2- 172.21.15.3:0/861838718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599b9afecc0 0x5599b9b0ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 >> 172.21.15.3:0/861838718 conn(0x5599b9afa230 msgr2=0x5599b9b03a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 shutdown_connections 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 1 -- 172.21.15.3:0/861838718 wait complete. 2024-02-09T22:50:55.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.036+0000 7f9982169740 20 client.15303 trim_cache size 0 max 16384 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 2, 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24496, 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 104.938698614, 2024-02-09T22:50:55.048 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:55.049 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:55.049 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:55.049 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 122.01605698900001 2024-02-09T22:50:55.049 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f99640a20c0 msgr2=0x7f99640a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f99640a20c0 0x7f99640a4580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9970005c40 tx=0x7f99700129e0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 msgr2=0x7f9978197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 0x7f9978197830 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9968006010 tx=0x7f99680042f0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 shutdown_connections 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f99640a20c0 0x7f99640a4580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9978102840 0x7f997819e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f99780ffc70 0x7f9978197830 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 --2- 172.21.15.3:0/1945927226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f99780ff2b0 0x7f99781972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 >> 172.21.15.3:0/1945927226 conn(0x7f99780fae50 msgr2=0x7f99781058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 shutdown_connections 2024-02-09T22:50:55.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.037+0000 7f997f4c0640 1 -- 172.21.15.3:0/1945927226 wait complete. 2024-02-09T22:50:55.051 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-02-09T22:50:55.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 -- 172.21.15.3:0/1937923814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198101340 msgr2=0x7f51981017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1937923814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198101340 0x7f51981017c0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f5180009900 tx=0x7f518002bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 -- 172.21.15.3:0/1937923814 shutdown_connections 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1937923814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101d00 0x7f5198108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1937923814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198101340 0x7f51981017c0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1937923814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198100150 0x7f5198100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 -- 172.21.15.3:0/1937923814 >> 172.21.15.3:0/1937923814 conn(0x7f51980fb6e0 msgr2=0x7f51980fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 -- 172.21.15.3:0/1937923814 shutdown_connections 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.151+0000 7f519cc0b640 1 -- 172.21.15.3:0/1937923814 wait complete. 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 Processor -- start 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 -- start start 2024-02-09T22:50:55.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 0x7f519806a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101340 0x7f519819c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198101d00 0x7f519819e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519819ee30 con 0x7f5198100150 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519819efa0 con 0x7f5198101d00 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519cc0b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f519819f160 con 0x7f5198101340 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 0x7f519806a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 0x7f519806a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42064/0 (socket says 172.21.15.3:42064) 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 -- 172.21.15.3:0/1144217546 learned_addr learned my addr 172.21.15.3:0/1144217546 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f5196d9d640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198101d00 0x7f519819e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f518ffff640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101340 0x7f519819c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 -- 172.21.15.3:0/1144217546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101340 msgr2=0x7f519819c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101340 0x7f519819c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 -- 172.21.15.3:0/1144217546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198101d00 msgr2=0x7f519819e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198101d00 0x7f519819e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f519659c640 1 -- 172.21.15.3:0/1144217546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5180009560 con 0x7f5198100150 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f5196d9d640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198101d00 0x7f519819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.152+0000 7f518ffff640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101340 0x7f519819c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.153+0000 7f519659c640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 0x7f519806a440 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f518800d770 tx=0x7f518800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.153+0000 7f518f7fe640 1 -- 172.21.15.3:0/1144217546 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188009c30 con 0x7f5198100150 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.153+0000 7f518f7fe640 1 -- 172.21.15.3:0/1144217546 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f518800de80 con 0x7f5198100150 2024-02-09T22:50:55.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.153+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f519819f3e0 con 0x7f5198100150 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.153+0000 7f518f7fe640 1 -- 172.21.15.3:0/1144217546 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188010460 con 0x7f5198100150 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.153+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f519819f8f0 con 0x7f5198100150 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f518f7fe640 1 -- 172.21.15.3:0/1144217546 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f5188003f20 con 0x7f5198100150 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 Processor -- start 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 -- start start 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f518f7fe640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51840a20c0 0x7f51840a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed699cc0 0x5607ed69a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 0x5607ed69f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6a1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed609b30 con 0x5607ed69a6b0 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed607060 con 0x5607ed69f820 2024-02-09T22:50:55.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.155+0000 7f519f7d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed602840 con 0x5607ed699cc0 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 0x5607ed69f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518f7fe640 1 -- 172.21.15.3:0/1144217546 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51880cd710 con 0x7f5198100150 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f5196d9d640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51840a20c0 0x7f51840a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 0x5607ed69f1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42074/0 (socket says 172.21.15.3:42074) 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 -- 172.21.15.3:0/2761476150 learned_addr learned my addr 172.21.15.3:0/2761476150 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f519659c640 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed699cc0 0x5607ed69a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f5196d9d640 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6a1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 -- 172.21.15.3:0/2761476150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed699cc0 msgr2=0x5607ed69a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed699cc0 0x5607ed69a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 -- 172.21.15.3:0/2761476150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 msgr2=0x5607ed6a1cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6a1cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 -- 172.21.15.3:0/2761476150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ed5c8620 con 0x5607ed69a6b0 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f5196d9d640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51840a20c0 0x7f51840a4580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f51900126c0 tx=0x7f5190012ac0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518ffff640 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 0x5607ed69f1b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5180006010 tx=0x7f518002c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c3114a463e332a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518cff9640 1 -- 172.21.15.3:0/2761476150 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51800046c0 con 0x5607ed69a6b0 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518cff9640 1 -- 172.21.15.3:0/2761476150 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5180004510 con 0x5607ed69a6b0 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.156+0000 7f518cff9640 1 -- 172.21.15.3:0/2761476150 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51800054b0 con 0x5607ed69a6b0 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 -- 172.21.15.3:0/2761476150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 msgr2=0x5607ed69f1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 0x5607ed69f1b0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5180006010 tx=0x7f518002c960 comp rx=0 tx=0).stop 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 -- 172.21.15.3:0/2761476150 shutdown_connections 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6a1cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed69a6b0 0x5607ed69f1b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2761476150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed699cc0 0x5607ed69a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 -- 172.21.15.3:0/2761476150 >> 172.21.15.3:0/2761476150 conn(0x5607ed695230 msgr2=0x5607ed695630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 -- 172.21.15.3:0/2761476150 shutdown_connections 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 -- 172.21.15.3:0/2761476150 wait complete. 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.157+0000 7f519f7d6740 1 -- start start 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 Processor -- start 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a4090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6ab060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed5a7cd0 con 0x5607ed699cc0 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed2d6310 con 0x5607ed69a6b0 2024-02-09T22:50:55.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed588490 con 0x5607ed69f820 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 10 client.0 _ll_register_callbacks cb 0x5607ed313ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42084/0 (socket says 172.21.15.3:42084) 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 -- 172.21.15.3:0/2357231239 learned_addr learned my addr 172.21.15.3:0/2357231239 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f5196d9d640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6ab060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 -- 172.21.15.3:0/2357231239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 msgr2=0x5607ed6ab060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6ab060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 -- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 msgr2=0x5607ed6a4090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f518ffff640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 -- 172.21.15.3:0/2357231239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5198108e90 con 0x5607ed699cc0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519659c640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3b50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5188096130 tx=0x7f518809ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bc0d07fb06e231f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f517dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f517dffb640 1 -- 172.21.15.3:0/2357231239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188098750 con 0x5607ed699cc0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f517dffb640 1 -- 172.21.15.3:0/2357231239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51880035f0 con 0x5607ed699cc0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.158+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607ed3026c0 con 0x5607ed699cc0 2024-02-09T22:50:55.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f517dffb640 1 -- 172.21.15.3:0/2357231239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51880bb3b0 con 0x5607ed699cc0 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 10 client.15315 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5607ed600ec0 con 0x5607ed699cc0 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f517dffb640 1 -- 172.21.15.3:0/2357231239 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5607ed3026c0 con 0x5607ed699cc0 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f517dffb640 1 -- 172.21.15.3:0/2357231239 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f518809e160 con 0x5607ed699cc0 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 10 client.15315 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 10 client.15315 resolve_mds: resolved name 'h' to daemon mds.h 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 20 client.15315 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed6b04b0 0x5607ed6af6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 4 client.15315 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607ed353270 con 0x5607ed6b04b0 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.159+0000 7f518ffff640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed6b04b0 0x5607ed6af6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.160+0000 7f517dffb640 10 client.15315 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.160+0000 7f518ffff640 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed6b04b0 0x5607ed6af6e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=df33ea829606484f server_cookie=9a6e553a99843e02 in_seq=0 out_seq=0 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.160+0000 7f517dffb640 1 -- 172.21.15.3:0/2357231239 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5607ed353270 con 0x5607ed6b04b0 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.160+0000 7f517dffb640 10 client.15315 handle_command_reply: tid=0 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.160+0000 7f519f7d6740 1 client.15315 shutdown 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 msgr2=0x5607ed6a3b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3b50 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5188096130 tx=0x7f518809ac90 comp rx=0 tx=0).stop 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 shutdown_connections 2024-02-09T22:50:55.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed6b04b0 0x5607ed6af6e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a4090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f517dffb640 0 client.15315 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6ab060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 --2- 172.21.15.3:0/2357231239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3b50 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 >> 172.21.15.3:0/2357231239 conn(0x5607ed695230 msgr2=0x5607ed694510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 shutdown_connections 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 1 -- 172.21.15.3:0/2357231239 wait complete. 2024-02-09T22:50:55.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.161+0000 7f519f7d6740 20 client.15315 trim_cache size 0 max 16384 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 Processor -- start 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 -- start start 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69a6b0 0x5607ed6ab020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6b8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed6b2490 con 0x5607ed699cc0 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed695d20 con 0x5607ed69f820 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519f7d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed6ae2e0 con 0x5607ed69a6b0 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42088/0 (socket says 172.21.15.3:42088) 2024-02-09T22:50:55.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 -- 172.21.15.3:0/326817484 learned_addr learned my addr 172.21.15.3:0/326817484 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 -- 172.21.15.3:0/326817484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69a6b0 msgr2=0x5607ed6ab020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69a6b0 0x5607ed6ab020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 -- 172.21.15.3:0/326817484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 msgr2=0x5607ed6b8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6b8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 -- 172.21.15.3:0/326817484 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ed59da90 con 0x5607ed699cc0 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f519659c640 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f518809a3e0 tx=0x7f5188094e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1746fb790d33f581 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f517ffff640 1 -- 172.21.15.3:0/326817484 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51880ba0c0 con 0x5607ed699cc0 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f517ffff640 1 -- 172.21.15.3:0/326817484 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51880a2a10 con 0x5607ed699cc0 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.177+0000 7f517ffff640 1 -- 172.21.15.3:0/326817484 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518809e160 con 0x5607ed699cc0 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 -- 172.21.15.3:0/326817484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 msgr2=0x5607ed6a3170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3170 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f518809a3e0 tx=0x7f5188094e80 comp rx=0 tx=0).stop 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 -- 172.21.15.3:0/326817484 shutdown_connections 2024-02-09T22:50:55.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69f820 0x5607ed6b8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69a6b0 0x5607ed6ab020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 --2- 172.21.15.3:0/326817484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a3170 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 -- 172.21.15.3:0/326817484 >> 172.21.15.3:0/326817484 conn(0x5607ed695230 msgr2=0x5607ed695610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 -- 172.21.15.3:0/326817484 shutdown_connections 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 -- 172.21.15.3:0/326817484 wait complete. 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 -- start start 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.178+0000 7f519f7d6740 1 Processor -- start 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a8b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6a9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed6aa0e0 con 0x5607ed699cc0 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed6aa250 con 0x5607ed69a6b0 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ed6aa410 con 0x5607ed69f820 2024-02-09T22:50:55.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 10 client.0 _ll_register_callbacks cb 0x7f5180009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a8650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42102/0 (socket says 172.21.15.3:42102) 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 -- 172.21.15.3:0/1017737458 learned_addr learned my addr 172.21.15.3:0/1017737458 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f5196d9d640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6a9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f518ffff640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 -- 172.21.15.3:0/1017737458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 msgr2=0x5607ed6a9ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6a9ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 -- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 msgr2=0x5607ed6a8b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 -- 172.21.15.3:0/1017737458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607ed600ec0 con 0x5607ed699cc0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519659c640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a8650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5188009fd0 tx=0x7f5188096020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=158bb40ab78403ef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f517dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:55.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f517dffb640 1 -- 172.21.15.3:0/1017737458 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51880ba0c0 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f517dffb640 1 -- 172.21.15.3:0/1017737458 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f518809e160 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607ed6b0e80 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f517dffb640 1 -- 172.21.15.3:0/1017737458 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51880a2a10 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 10 client.15321 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.179+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x5607ed5c3930 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f517dffb640 1 -- 172.21.15.3:0/1017737458 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5607ed6b0e80 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f517dffb640 1 -- 172.21.15.3:0/1017737458 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f51880994a0 con 0x5607ed699cc0 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f519f7d6740 10 client.15321 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f519f7d6740 10 client.15321 resolve_mds: resolved name 'h' to daemon mds.h 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f519f7d6740 20 client.15321 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f519f7d6740 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed70bb30 0x5607ed7081d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f519f7d6740 4 client.15321 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f5180004510 con 0x5607ed70bb30 2024-02-09T22:50:55.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f518ffff640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed70bb30 0x5607ed7081d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f518ffff640 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed70bb30 0x5607ed7081d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=647466cd9a428f30 server_cookie=9d2a1a2d79934668 in_seq=0 out_seq=0 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.180+0000 7f517dffb640 10 client.15321 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f517dffb640 1 -- 172.21.15.3:0/1017737458 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x7f5180004510 con 0x5607ed70bb30 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f517dffb640 10 client.15321 handle_command_reply: tid=0 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 client.15321 shutdown 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 msgr2=0x5607ed6a8650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a8650 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5188009fd0 tx=0x7f5188096020 comp rx=0 tx=0).stop 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 shutdown_connections 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5607ed70bb30 0x5607ed7081d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607ed69a6b0 0x5607ed6a8b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607ed69f820 0x5607ed6a9ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f517dffb640 0 client.15321 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 --2- 172.21.15.3:0/1017737458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607ed699cc0 0x5607ed6a8650 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 >> 172.21.15.3:0/1017737458 conn(0x5607ed695230 msgr2=0x5607ed6b8950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 shutdown_connections 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 1 -- 172.21.15.3:0/1017737458 wait complete. 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519f7d6740 20 client.15321 trim_cache size 0 max 16384 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 1, 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24503, 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:55.192 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 106.09789324099999, 2024-02-09T22:50:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:55.193 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 124.74987378 2024-02-09T22:50:55.193 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51840a20c0 msgr2=0x7f51840a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51840a20c0 0x7f51840a4580 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f51900126c0 tx=0x7f5190012ac0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 msgr2=0x7f519806a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 0x7f519806a440 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f518800d770 tx=0x7f518800dc30 comp rx=0 tx=0).stop 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 shutdown_connections 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51840a20c0 0x7f51840a4580 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5198101d00 0x7f519819e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5198101340 0x7f519819c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 --2- 172.21.15.3:0/1144217546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5198100150 0x7f519806a440 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 >> 172.21.15.3:0/1144217546 conn(0x7f51980fb6e0 msgr2=0x7f51981046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.181+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 shutdown_connections 2024-02-09T22:50:55.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.182+0000 7f519cc0b640 1 -- 172.21.15.3:0/1144217546 wait complete. 2024-02-09T22:50:55.196 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-02-09T22:50:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 -- 172.21.15.3:0/1191488104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4101d30 msgr2=0x7fcfe4104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 --2- 172.21.15.3:0/1191488104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4101d30 0x7fcfe4104120 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009900 tx=0x7fcfcc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:50:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 -- 172.21.15.3:0/1191488104 shutdown_connections 2024-02-09T22:50:55.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 --2- 172.21.15.3:0/1191488104 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe4104660 0x7fcfe4106a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 --2- 172.21.15.3:0/1191488104 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4106f90 0x7fcfe4109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 --2- 172.21.15.3:0/1191488104 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4101d30 0x7fcfe4104120 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 -- 172.21.15.3:0/1191488104 >> 172.21.15.3:0/1191488104 conn(0x7fcfe40fb6c0 msgr2=0x7fcfe40fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 -- 172.21.15.3:0/1191488104 shutdown_connections 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 -- 172.21.15.3:0/1191488104 wait complete. 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.296+0000 7fcfea488640 1 Processor -- start 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 -- start start 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4101d30 0x7fcfe41974b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 0x7fcfe41979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe4106f90 0x7fcfe419ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419efa0 con 0x7fcfe4104660 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419f110 con 0x7fcfe4106f90 2024-02-09T22:50:55.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfea488640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe419f2d0 con 0x7fcfe4101d30 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 0x7fcfe41979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 0x7fcfe41979f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42130/0 (socket says 172.21.15.3:42130) 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 -- 172.21.15.3:0/2583978199 learned_addr learned my addr 172.21.15.3:0/2583978199 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe4106f90 0x7fcfe419ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4101d30 0x7fcfe41974b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 -- 172.21.15.3:0/2583978199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4101d30 msgr2=0x7fcfe41974b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4101d30 0x7fcfe41974b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 -- 172.21.15.3:0/2583978199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe4106f90 msgr2=0x7fcfe419ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe4106f90 0x7fcfe419ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 -- 172.21.15.3:0/2583978199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfcc009560 con 0x7fcfe4104660 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4101d30 0x7fcfe41974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.297+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 0x7fcfe41979f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b450 tx=0x7fcfd400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.298+0000 7fcfe17fa640 1 -- 172.21.15.3:0/2583978199 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4011890 con 0x7fcfe4104660 2024-02-09T22:50:55.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.298+0000 7fcfe17fa640 1 -- 172.21.15.3:0/2583978199 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd40103e0 con 0x7fcfe4104660 2024-02-09T22:50:55.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.298+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe419f5b0 con 0x7fcfe4104660 2024-02-09T22:50:55.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.298+0000 7fcfe17fa640 1 -- 172.21.15.3:0/2583978199 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400f580 con 0x7fcfe4104660 2024-02-09T22:50:55.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.298+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe419fb40 con 0x7fcfe4104660 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfe17fa640 1 -- 172.21.15.3:0/2583978199 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fcfd40108e0 con 0x7fcfe4104660 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 Processor -- start 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfe17fa640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcfd00a20c0 0x7fcfd00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 -- start start 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcfd00a20c0 0x7fcfd00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e374f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e3c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfe17fa640 1 -- 172.21.15.3:0/2583978199 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcfd40cda40 con 0x7fcfe4104660 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 0x561935e3f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935da2b30 con 0x561935e3cc30 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935da0060 con 0x561935e370d0 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfed069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935d9b840 con 0x561935e37ac0 2024-02-09T22:50:55.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.300+0000 7fcfe8a25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 0x561935e3f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 0x561935e3f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42136/0 (socket says 172.21.15.3:42136) 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 -- 172.21.15.3:0/3681533242 learned_addr learned my addr 172.21.15.3:0/3681533242 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e3c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e374f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcfd00a20c0 0x7fcfd00a4580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00a860 tx=0x7fcfdc00ac60 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 -- 172.21.15.3:0/3681533242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 msgr2=0x561935e3c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e3c5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 -- 172.21.15.3:0/3681533242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 msgr2=0x561935e374f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e374f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 -- 172.21.15.3:0/3681533242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561935d61620 con 0x561935e3cc30 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e3c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 0x561935e3f0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009a60 tx=0x7fcfdc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7360da0a69942521 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfda7fc640 1 -- 172.21.15.3:0/3681533242 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc015e80 con 0x561935e3cc30 2024-02-09T22:50:55.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfda7fc640 1 -- 172.21.15.3:0/3681533242 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfdc022400 con 0x561935e3cc30 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.301+0000 7fcfda7fc640 1 -- 172.21.15.3:0/3681533242 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfdc01b640 con 0x561935e3cc30 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 -- 172.21.15.3:0/3681533242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 msgr2=0x561935e3f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 0x561935e3f0c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009a60 tx=0x7fcfdc01d000 comp rx=0 tx=0).stop 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 -- 172.21.15.3:0/3681533242 shutdown_connections 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e3cc30 0x561935e3f0c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e3c5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 --2- 172.21.15.3:0/3681533242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e374f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 -- 172.21.15.3:0/3681533242 >> 172.21.15.3:0/3681533242 conn(0x561935e2e230 msgr2=0x561935e2e630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 -- 172.21.15.3:0/3681533242 shutdown_connections 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 -- 172.21.15.3:0/3681533242 wait complete. 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 -- start start 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.302+0000 7fcfed069740 1 Processor -- start 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e41460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935e484a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935d40cd0 con 0x561935e37ac0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935a6f310 con 0x561935e370d0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935d21490 con 0x561935e3cc30 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 10 client.0 _ll_register_callbacks cb 0x561935aacac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e41460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e41460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42144/0 (socket says 172.21.15.3:42144) 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 -- 172.21.15.3:0/962701152 learned_addr learned my addr 172.21.15.3:0/962701152 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935e484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 -- 172.21.15.3:0/962701152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 msgr2=0x561935e484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935e484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 -- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 msgr2=0x561935e40f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 -- 172.21.15.3:0/962701152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfdc013040 con 0x561935e37ac0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e41460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009fd0 tx=0x7fcfd409afd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c2808259fbf321d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:55.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfc37fe640 1 -- 172.21.15.3:0/962701152 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd4010550 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfc37fe640 1 -- 172.21.15.3:0/962701152 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcfd40106b0 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561935a9b6c0 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfc37fe640 1 -- 172.21.15.3:0/962701152 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd40b3020 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 10 client.15333 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.303+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561935d99ec0 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfc37fe640 1 -- 172.21.15.3:0/962701152 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfd40b7070 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfc37fe640 1 -- 172.21.15.3:0/962701152 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcfd4099bd0 con 0x561935e37ac0 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfed069740 10 client.15333 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfed069740 10 client.15333 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfed069740 20 client.15333 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfed069740 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935e4d860 0x561935e51ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfed069740 4 client.15333 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:50:55.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561935aec270 con 0x561935e4d860 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935e4d860 0x561935e51ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfc37fe640 10 client.15333 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.304+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935e4d860 0x561935e51ce0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=496663be9ca4a274 server_cookie=37abcdd301fd3465 in_seq=0 out_seq=0 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfc37fe640 1 -- 172.21.15.3:0/962701152 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561935aec270 con 0x561935e4d860 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfc37fe640 10 client.15333 handle_command_reply: tid=0 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 client.15333 shutdown 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 msgr2=0x561935e41460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e41460 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009fd0 tx=0x7fcfd409afd0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 shutdown_connections 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935e4d860 0x561935e51ce0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935e484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e41460 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 --2- 172.21.15.3:0/962701152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfc37fe640 0 client.15333 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:55.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 >> 172.21.15.3:0/962701152 conn(0x561935e2e230 msgr2=0x561935e2d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 shutdown_connections 2024-02-09T22:50:55.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 1 -- 172.21.15.3:0/962701152 wait complete. 2024-02-09T22:50:55.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.305+0000 7fcfed069740 20 client.15333 trim_cache size 0 max 16384 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 Processor -- start 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 -- start start 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e48460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e3cc30 0x561935e41460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935e2f230 con 0x561935e370d0 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935e3c9b0 con 0x561935e3cc30 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfed069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935aec270 con 0x561935e37ac0 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfe3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42146/0 (socket says 172.21.15.3:42146) 2024-02-09T22:50:55.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.321+0000 7fcfe3fff640 1 -- 172.21.15.3:0/765498576 learned_addr learned my addr 172.21.15.3:0/765498576 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfe3fff640 1 -- 172.21.15.3:0/765498576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 msgr2=0x561935e48460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e48460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfe3fff640 1 -- 172.21.15.3:0/765498576 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e3cc30 msgr2=0x561935e41460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e3cc30 0x561935e41460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfe3fff640 1 -- 172.21.15.3:0/765498576 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561935d36a90 con 0x561935e370d0 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 0x561935e40f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009900 tx=0x7fcfcc005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d852f746fca1e2c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfd9ffb640 1 -- 172.21.15.3:0/765498576 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc04b070 con 0x561935e370d0 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfd9ffb640 1 -- 172.21.15.3:0/765498576 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfcc047b80 con 0x561935e370d0 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfd9ffb640 1 -- 172.21.15.3:0/765498576 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfcc047270 con 0x561935e370d0 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfed069740 1 -- 172.21.15.3:0/765498576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 msgr2=0x561935e40f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.322+0000 7fcfed069740 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 0x561935e40f20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009900 tx=0x7fcfcc005de0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- 172.21.15.3:0/765498576 shutdown_connections 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e3cc30 0x561935e41460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e37ac0 0x561935e48460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 --2- 172.21.15.3:0/765498576 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e370d0 0x561935e40f20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- 172.21.15.3:0/765498576 >> 172.21.15.3:0/765498576 conn(0x561935e2e230 msgr2=0x561935e3c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- 172.21.15.3:0/765498576 shutdown_connections 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- 172.21.15.3:0/765498576 wait complete. 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- start start 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 Processor -- start 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e450c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e45600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935ea39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935e48620 con 0x561935e37ac0 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935e48790 con 0x561935e370d0 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561935ea3ee0 con 0x561935e3cc30 2024-02-09T22:50:55.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfed069740 10 client.0 _ll_register_callbacks cb 0x7fcfdc009990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e45600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfe37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e45600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42160/0 (socket says 172.21.15.3:42160) 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.323+0000 7fcfe37fe640 1 -- 172.21.15.3:0/1770040416 learned_addr learned my addr 172.21.15.3:0/1770040416 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935ea39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe37fe640 1 -- 172.21.15.3:0/1770040416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 msgr2=0x561935ea39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935ea39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe37fe640 1 -- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 msgr2=0x561935e450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe37fe640 1 -- 172.21.15.3:0/1770040416 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe4109f60 con 0x561935e37ac0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe8a25640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935ea39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfe37fe640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e45600 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfd409a710 tx=0x7fcfd4010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b56206bb97d6362 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfc37fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfc37fe640 1 -- 172.21.15.3:0/1770040416 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd409ae80 con 0x561935e37ac0 2024-02-09T22:50:55.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561935e4d570 con 0x561935e37ac0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfc37fe640 1 -- 172.21.15.3:0/1770040416 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fcfd409e8d0 con 0x561935e37ac0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfc37fe640 1 -- 172.21.15.3:0/1770040416 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfd40b3290 con 0x561935e37ac0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 10 client.15339 fetch_fsmap learned FSMap version 24 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) v3 -- 0x561935d5c930 con 0x561935e37ac0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfc37fe640 1 -- 172.21.15.3:0/1770040416 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561935e4d570 con 0x561935e37ac0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfc37fe640 1 -- 172.21.15.3:0/1770040416 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 24) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcfd4099640 con 0x561935e37ac0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 10 client.15339 fetch_fsmap finished waiting for FSMap version 24 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 10 client.15339 resolve_mds: resolved name 'b' to daemon mds.b 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 20 client.15339 populate_metadata read hostname 'smithi003' 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935ea8de0 0x561935ea52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 4 client.15339 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.324+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "status"}) v1 -- 0x561935e4e890 con 0x561935ea8de0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935ea8de0 0x561935ea52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:55.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfe3fff640 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935ea8de0 0x561935ea52f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb23161eec7bd0e server_cookie=eb88a5d98262011b in_seq=0 out_seq=0 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfc37fe640 10 client.15339 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfc37fe640 1 -- 172.21.15.3:0/1770040416 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+321 (crc 0 0 0) 0x561935e4e890 con 0x561935ea8de0 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfc37fe640 10 client.15339 handle_command_reply: tid=0 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 client.15339 shutdown 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 msgr2=0x561935e45600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e45600 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fcfd409a710 tx=0x7fcfd4010c20 comp rx=0 tx=0).stop 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 shutdown_connections 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561935ea8de0 0x561935ea52f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561935e3cc30 0x561935ea39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561935e37ac0 0x561935e45600 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfc37fe640 0 client.15339 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 --2- 172.21.15.3:0/1770040416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561935e370d0 0x561935e450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.325+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 >> 172.21.15.3:0/1770040416 conn(0x561935e2e230 msgr2=0x561935e40ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 shutdown_connections 2024-02-09T22:50:55.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfed069740 1 -- 172.21.15.3:0/1770040416 wait complete. 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfed069740 20 client.15339 trim_cache size 0 max 16384 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "cluster_fsid": "2bf8a73a-c79c-11ee-95b8-87774f69a715", 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "whoami": 0, 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "id": 24533, 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "want_state": "up:active", 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "state": "up:active", 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "fs_name": "cephfs", 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "rank_uptime": 107.78867581, 2024-02-09T22:50:55.338 INFO:teuthology.orchestra.run.smithi003.stdout: "mdsmap_epoch": 24, 2024-02-09T22:50:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch": 88, 2024-02-09T22:50:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_epoch_barrier": 86, 2024-02-09T22:50:55.339 INFO:teuthology.orchestra.run.smithi003.stdout: "uptime": 113.674063096 2024-02-09T22:50:55.339 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcfd00a20c0 msgr2=0x7fcfd00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcfd00a20c0 0x7fcfd00a4580 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc00a860 tx=0x7fcfdc00ac60 comp rx=0 tx=0).stop 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 msgr2=0x7fcfe41979f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 0x7fcfe41979f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fcfd400b450 tx=0x7fcfd400b910 comp rx=0 tx=0).stop 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 shutdown_connections 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fcfd00a20c0 0x7fcfd00a4580 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcfe4106f90 0x7fcfe419ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcfe4104660 0x7fcfe41979f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 --2- 172.21.15.3:0/2583978199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcfe4101d30 0x7fcfe41974b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 >> 172.21.15.3:0/2583978199 conn(0x7fcfe40fb6c0 msgr2=0x7fcfe4102790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 shutdown_connections 2024-02-09T22:50:55.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:55.326+0000 7fcfea488640 1 -- 172.21.15.3:0/2583978199 wait complete. 2024-02-09T22:50:55.341 INFO:teuthology.run_tasks:Running task kclient... 2024-02-09T22:50:55.371 INFO:tasks.kclient:Mounting kernel clients... 2024-02-09T22:50:55.371 INFO:tasks.kclient:config is {'client.0': {}} 2024-02-09T22:50:55.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:55.372 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-02-09T22:50:55.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:55.387 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-02-09T22:50:55.455 INFO:teuthology.orchestra.run.smithi003.stderr:Cannot find device "ceph-brx" 2024-02-09T22:50:55.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T22:50:55.457 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-02-09T22:50:55.457 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-02-09T22:50:55.457 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-09T22:50:55.458 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph fs ls 2024-02-09T22:50:55.741 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T22:50:56.628 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:56 smithi003 ceph-mon[32678]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:56.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:56 smithi064 ceph-mon[36638]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:56.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:56 smithi089 ceph-mon[36665]: pgmap v343: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 -- 172.21.15.3:0/2100674942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58101860 msgr2=0x7fac58105860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 --2- 172.21.15.3:0/2100674942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58101860 0x7fac58105860 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fac4c009990 tx=0x7fac4c009ca0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 -- 172.21.15.3:0/2100674942 shutdown_connections 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 --2- 172.21.15.3:0/2100674942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58105f40 0x7fac5810db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 --2- 172.21.15.3:0/2100674942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac58100eb0 0x7fac58101290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 --2- 172.21.15.3:0/2100674942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58101860 0x7fac58105860 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 -- 172.21.15.3:0/2100674942 >> 172.21.15.3:0/2100674942 conn(0x7fac58076c40 msgr2=0x7fac58077050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:57.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 -- 172.21.15.3:0/2100674942 shutdown_connections 2024-02-09T22:50:57.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 -- 172.21.15.3:0/2100674942 wait complete. 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.423+0000 7fac5dd67700 1 Processor -- start 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 -- start start 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 0x7fac5810e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58101860 0x7fac5810e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac58105f40 0x7fac5810ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac581a65c0 con 0x7fac58100eb0 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac581a6730 con 0x7fac58105f40 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac5dd67700 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac581a68f0 con 0x7fac58101860 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 0x7fac5810e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:57.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 0x7fac5810e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39920/0 (socket says 172.21.15.3:39920) 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 -- 172.21.15.3:0/188346098 learned_addr learned my addr 172.21.15.3:0/188346098 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac57fff700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac58105f40 0x7fac5810ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac56ffd700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58101860 0x7fac5810e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 -- 172.21.15.3:0/188346098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58101860 msgr2=0x7fac5810e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58101860 0x7fac5810e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 -- 172.21.15.3:0/188346098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac58105f40 msgr2=0x7fac5810ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac58105f40 0x7fac5810ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac577fe700 1 -- 172.21.15.3:0/188346098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4c0095f0 con 0x7fac58100eb0 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.424+0000 7fac56ffd700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58101860 0x7fac5810e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.425+0000 7fac577fe700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 0x7fac5810e3f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac4800d950 tx=0x7fac4800dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.425+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac48009940 con 0x7fac58100eb0 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.425+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac48010460 con 0x7fac58100eb0 2024-02-09T22:50:57.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.425+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac581a6bd0 con 0x7fac58100eb0 2024-02-09T22:50:57.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.425+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac4800f620 con 0x7fac58100eb0 2024-02-09T22:50:57.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.425+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac581a7120 con 0x7fac58100eb0 2024-02-09T22:50:57.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.426+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac58109120 con 0x7fac58100eb0 2024-02-09T22:50:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.427+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fac480105d0 con 0x7fac58100eb0 2024-02-09T22:50:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.427+0000 7fac54ff9700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fac400a21d0 0x7fac400a4690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:50:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.428+0000 7fac56ffd700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fac400a21d0 0x7fac400a4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:50:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.428+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fac480cdad0 con 0x7fac58100eb0 2024-02-09T22:50:57.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.428+0000 7fac56ffd700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fac400a21d0 0x7fac400a4690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00b5c0 tx=0x7fac4c005fb0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:50:57.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.432+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fac4809a5e0 con 0x7fac58100eb0 2024-02-09T22:50:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.682+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fac58111dd0 con 0x7fac58100eb0 2024-02-09T22:50:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.683+0000 7fac54ff9700 1 -- 172.21.15.3:0/188346098 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7fac4809a170 con 0x7fac58100eb0 2024-02-09T22:50:57.687 INFO:teuthology.orchestra.run.smithi003.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fac400a21d0 msgr2=0x7fac400a4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fac400a21d0 0x7fac400a4690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fac4c00b5c0 tx=0x7fac4c005fb0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 msgr2=0x7fac5810e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 0x7fac5810e3f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7fac4800d950 tx=0x7fac4800dc60 comp rx=0 tx=0).stop 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 shutdown_connections 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fac400a21d0 0x7fac400a4690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fac58105f40 0x7fac5810ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fac58100eb0 0x7fac5810e3f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 --2- 172.21.15.3:0/188346098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fac58101860 0x7fac5810e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 >> 172.21.15.3:0/188346098 conn(0x7fac58076c40 msgr2=0x7fac581050d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 shutdown_connections 2024-02-09T22:50:57.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:50:57.685+0000 7fac5dd67700 1 -- 172.21.15.3:0/188346098 wait complete. 2024-02-09T22:50:58.041 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:57 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/188346098' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T22:50:58.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:57 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/188346098' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T22:50:58.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:57 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/188346098' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-02-09T22:50:58.416 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-02-09T22:50:58.416 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi003.front.sepia.ceph.com 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-02-09T22:50:58.417 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-02-09T22:50:58.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:58.417 DEBUG:teuthology.orchestra.run.smithi003:> ip addr 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: inet 127.0.0.1/8 scope host lo 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 ::1/128 scope host 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ac brd ff:ff:ff:ff:ff:ff 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f0 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T22:50:58.442 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:6b:fd:ad brd ff:ff:ff:ff:ff:ff 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp7s0f1 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dc brd ff:ff:ff:ff:ff:ff 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f0 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: inet 172.21.15.3/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft 41764sec preferred_lft 41764sec 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: inet6 fe80::ec4:7aff:febd:15dc/64 scope link noprefixroute 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: valid_lft forever preferred_lft forever 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: link/ether 0c:c4:7a:bd:15:dd brd ff:ff:ff:ff:ff:ff 2024-02-09T22:50:58.443 INFO:teuthology.orchestra.run.smithi003.stdout: altname enp1s0f1 2024-02-09T22:50:58.444 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-02-09T22:50:58.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add name ceph-brx type bridge 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr flush dev ceph-brx 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set ceph-brx up 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-02-09T22:50:58.444 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-02-09T22:50:58.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:58.617 DEBUG:teuthology.orchestra.run.smithi003:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-02-09T22:50:58.646 INFO:teuthology.orchestra.run.smithi003.stdout:1 2024-02-09T22:50:58.649 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:58.649 DEBUG:teuthology.orchestra.run.smithi003:> route 2024-02-09T22:50:58.724 INFO:teuthology.orchestra.run.smithi003.stdout:Kernel IP routing table 2024-02-09T22:50:58.724 INFO:teuthology.orchestra.run.smithi003.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-02-09T22:50:58.724 INFO:teuthology.orchestra.run.smithi003.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-02-09T22:50:58.724 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-02-09T22:50:58.725 INFO:teuthology.orchestra.run.smithi003.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-02-09T22:50:58.725 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:58.725 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-02-09T22:50:58.725 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-02-09T22:50:58.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-02-09T22:50:58.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-02-09T22:50:58.725 DEBUG:teuthology.orchestra.run.smithi003:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-02-09T22:50:58.725 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-02-09T22:50:58.904 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:50:58 smithi003 ceph-mon[32678]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:59.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.039 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-02-09T22:50:59.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.055 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list-id 2024-02-09T22:50:59.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.110 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-02-09T22:50:59.110 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-02-09T22:50:59.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-09T22:50:59.110 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-02-09T22:50:59.110 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-02-09T22:50:59.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:50:58 smithi064 ceph-mon[36638]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:59.192 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-02-09T22:50:59.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-02-09T22:50:59.193 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-02-09T22:50:59.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:50:58 smithi089 ceph-mon[36665]: pgmap v344: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-02-09T22:50:59.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.333 DEBUG:teuthology.orchestra.run.smithi003:> (cd / && exec bash -c ' 2024-02-09T22:50:59.333 DEBUG:teuthology.orchestra.run.smithi003:> set -e 2024-02-09T22:50:59.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set brx.0 up 2024-02-09T22:50:59.333 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link set dev brx.0 master ceph-brx 2024-02-09T22:50:59.333 DEBUG:teuthology.orchestra.run.smithi003:> ') 2024-02-09T22:50:59.375 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-09T22:50:59.375 DEBUG:teuthology.orchestra.run.smithi003:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-02-09T22:50:59.429 INFO:teuthology.orchestra.run.smithi003.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-02-09T22:50:59.429 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-02-09T22:50:59.430 DEBUG:teuthology.orchestra.run.smithi003:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-02-09T22:50:59.484 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-02-09T22:50:59.484 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-02-09T22:50:59.484 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 2024-02-09T22:50:59.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.485 DEBUG:teuthology.orchestra.run.smithi003:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-02-09T22:50:59.568 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-02-09T22:50:59.569 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-02-09T22:50:59.569 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-02-09T22:50:59.697 INFO:teuthology.orchestra.run.smithi003.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync 2024-02-09T22:50:59.697 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync". 2024-02-09T22:50:59.697 INFO:teuthology.orchestra.run.smithi003.stdout:invalid new device string format 2024-02-09T22:50:59.697 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: resolved to: "172.21.15.3:3300,172.21.15.64:3300,172.21.15.89:3300" 2024-02-09T22:50:59.697 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.3:3300,172.21.15.64:3300,172.21.15.89:3300:/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 2024-02-09T22:50:59.697 INFO:teuthology.orchestra.run.smithi003.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=crc,wsync,key=0,fsid=2bf8a73a-c79c-11ee-95b8-87774f69a715" will pass to kernel 2024-02-09T22:50:59.698 INFO:teuthology.orchestra.run.smithi003.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-02-09T22:50:59.698 INFO:teuthology.orchestra.run.smithi003.stdout: You just mounted a file system that supports labels which does not 2024-02-09T22:50:59.698 INFO:teuthology.orchestra.run.smithi003.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-02-09T22:50:59.698 INFO:teuthology.orchestra.run.smithi003.stdout: applications will generate AVC messages and not be allowed access to 2024-02-09T22:50:59.698 INFO:teuthology.orchestra.run.smithi003.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-02-09T22:50:59.699 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-02-09T22:50:59.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-02-09T22:50:59.731 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.731 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T22:50:59.787 INFO:teuthology.orchestra.run.smithi003.stdout:client24979 2024-02-09T22:50:59.787 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:50:59.787 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T22:50:59.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.844 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T22:50:59.900 INFO:teuthology.orchestra.run.smithi003.stdout:client24979 2024-02-09T22:50:59.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.900 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-02-09T22:50:59.957 INFO:teuthology.orchestra.run.smithi003.stdout:client24979 2024-02-09T22:50:59.957 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:50:59.958 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715.client24979/status') 2024-02-09T22:51:00.024 INFO:teuthology.orchestra.run.smithi003.stdout:instance: client.24979 (3)172.21.15.3:0/1339955193 2024-02-09T22:51:00.026 INFO:teuthology.orchestra.run.smithi003.stdout:blocklisted: false 2024-02-09T22:51:00.027 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records in 2024-02-09T22:51:00.027 INFO:teuthology.orchestra.run.smithi003.stderr:0+1 records out 2024-02-09T22:51:00.027 INFO:teuthology.orchestra.run.smithi003.stderr:70 bytes copied, 2.0958e-05 s, 3.3 MB/s 2024-02-09T22:51:00.027 INFO:teuthology.run_tasks:Running task exec... 2024-02-09T22:51:00.042 INFO:teuthology.task.exec:Executing custom commands... 2024-02-09T22:51:00.042 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T22:51:00.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/2560866556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8101d10 msgr2=0x7ff0b81085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/2560866556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8101d10 0x7ff0b81085a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0009ca0 tx=0x7ff0b002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/2560866556 shutdown_connections 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/2560866556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0b8101350 0x7ff0b81017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/2560866556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8101d10 0x7ff0b81085a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/2560866556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0b8100160 0x7ff0b8100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/2560866556 >> 172.21.15.3:0/2560866556 conn(0x7ff0b80fb6d0 msgr2=0x7ff0b80fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/2560866556 shutdown_connections 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.168+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/2560866556 wait complete. 2024-02-09T22:51:00.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 Processor -- start 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 -- start start 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 0x7ff0b81040f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0b8101350 0x7ff0b8102730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0b8101d10 0x7ff0b8102c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81033d0 con 0x7ff0b8100160 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8103540 con 0x7ff0b8101350 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81a1340 con 0x7ff0b8101d10 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 0x7ff0b81040f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 0x7ff0b81040f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39950/0 (socket says 172.21.15.3:39950) 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 -- 172.21.15.3:0/1837446821 learned_addr learned my addr 172.21.15.3:0/1837446821 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b759e640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0b8101d10 0x7ff0b8102c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0affff640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0b8101350 0x7ff0b8102730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 -- 172.21.15.3:0/1837446821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0b8101d10 msgr2=0x7ff0b8102c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0b8101d10 0x7ff0b8102c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 -- 172.21.15.3:0/1837446821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0b8101350 msgr2=0x7ff0b8102730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0b8101350 0x7ff0b8102730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 -- 172.21.15.3:0/1837446821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0009900 con 0x7ff0b8100160 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b6d9d640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 0x7ff0b81040f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800d980 tx=0x7ff0a800de40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8009c30 con 0x7ff0b8100160 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b81a15f0 con 0x7ff0b8100160 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0a8009d90 con 0x7ff0b8100160 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.169+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800f7a0 con 0x7ff0b8100160 2024-02-09T22:51:00.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.170+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b81a1b00 con 0x7ff0b8100160 2024-02-09T22:51:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.171+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b810bb70 con 0x7ff0b8100160 2024-02-09T22:51:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.171+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7ff0a8010ad0 con 0x7ff0b8100160 2024-02-09T22:51:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.172+0000 7ff0b4d99640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0a40a20b0 0x7ff0a40a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.172+0000 7ff0affff640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0a40a20b0 0x7ff0a40a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.172+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7ff0a80cd9a0 con 0x7ff0b8100160 2024-02-09T22:51:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.172+0000 7ff0affff640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0a40a20b0 0x7ff0a40a4570 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8103e60 tx=0x7ff0a0005ed0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:00.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.178+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0a8096920 con 0x7ff0b8100160 2024-02-09T22:51:00.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.396+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) v1 -- 0x7ff0b81a1d80 con 0x7ff0b8100160 2024-02-09T22:51:00.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.448+0000 7ff0b4d99640 1 -- 172.21.15.3:0/1837446821 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) v1 ==== 117+0+0 (secure 0 0 0) 0x7ff0a8099fe0 con 0x7ff0b8100160 2024-02-09T22:51:00.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.450+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0a40a20b0 msgr2=0x7ff0a40a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.450+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0a40a20b0 0x7ff0a40a4570 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0b8103e60 tx=0x7ff0a0005ed0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.450+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 msgr2=0x7ff0b81040f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.450+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 0x7ff0b81040f0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7ff0a800d980 tx=0x7ff0a800de40 comp rx=0 tx=0).stop 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 shutdown_connections 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7ff0a40a20b0 0x7ff0a40a4570 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0b8101350 0x7ff0b8102730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0b8101d10 0x7ff0b8102c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 --2- 172.21.15.3:0/1837446821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0b8100160 0x7ff0b81040f0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 >> 172.21.15.3:0/1837446821 conn(0x7ff0b80fb6d0 msgr2=0x7ff0b80fd180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 shutdown_connections 2024-02-09T22:51:00.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.451+0000 7ff0bd1d0640 1 -- 172.21.15.3:0/1837446821 wait complete. 2024-02-09T22:51:00.467 INFO:teuthology.run_tasks:Running task check-counter... 2024-02-09T22:51:00.478 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-02-09T22:51:00.492 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 -- 172.21.15.3:0/782641124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50101d10 msgr2=0x7f3f501085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 --2- 172.21.15.3:0/782641124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50101d10 0x7f3f501085a0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f3f48009ca0 tx=0x7f3f4802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 -- 172.21.15.3:0/782641124 shutdown_connections 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 --2- 172.21.15.3:0/782641124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101350 0x7f3f501017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 --2- 172.21.15.3:0/782641124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50101d10 0x7f3f501085a0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 --2- 172.21.15.3:0/782641124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f50100160 0x7f3f50100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.592+0000 7f3f586eb640 1 -- 172.21.15.3:0/782641124 >> 172.21.15.3:0/782641124 conn(0x7f3f500fb6d0 msgr2=0x7f3f500fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 -- 172.21.15.3:0/782641124 shutdown_connections 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 -- 172.21.15.3:0/782641124 wait complete. 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 Processor -- start 2024-02-09T22:51:00.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 -- start start 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 0x7f3f50199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f50101350 0x7f3f501999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101d10 0x7f3f501a0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f501a0f50 con 0x7f3f50100160 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f501a10c0 con 0x7f3f50101d10 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f586eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f501a1280 con 0x7f3f50101350 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f56487640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 0x7f3f50199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f56487640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 0x7f3f50199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39982/0 (socket says 172.21.15.3:39982) 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.593+0000 7f3f56487640 1 -- 172.21.15.3:0/2329221855 learned_addr learned my addr 172.21.15.3:0/2329221855 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56487640 1 -- 172.21.15.3:0/2329221855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f50101350 msgr2=0x7f3f501999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56487640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f50101350 0x7f3f501999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56487640 1 -- 172.21.15.3:0/2329221855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101d10 msgr2=0x7f3f501a0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56c88640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101d10 0x7f3f501a0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56487640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101d10 0x7f3f501a0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56487640 1 -- 172.21.15.3:0/2329221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48009900 con 0x7f3f50100160 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56c88640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101d10 0x7f3f501a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f56487640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 0x7f3f50199460 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00d950 tx=0x7f3f4c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c009c30 con 0x7f3f50100160 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f4c009d90 con 0x7f3f50100160 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f501a1560 con 0x7f3f50100160 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c00f880 con 0x7f3f50100160 2024-02-09T22:51:00.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.594+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f501a1a70 con 0x7f3f50100160 2024-02-09T22:51:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.595+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f5010cc30 con 0x7f3f50100160 2024-02-09T22:51:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.596+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3f4c00f9e0 con 0x7f3f50100160 2024-02-09T22:51:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.597+0000 7f3f477fe640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f3c0a20c0 0x7f3f3c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.597+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3f4c0cdfa0 con 0x7f3f50100160 2024-02-09T22:51:00.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.597+0000 7f3f55c86640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f3c0a20c0 0x7f3f3c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.597+0000 7f3f55c86640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f3c0a20c0 0x7f3f3c0a4580 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f5019a970 tx=0x7f3f40009590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:00.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.601+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3f4c101d20 con 0x7f3f50100160 2024-02-09T22:51:00.658 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:00 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1837446821' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-02-09T22:51:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:00 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1837446821' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-02-09T22:51:00.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.786+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f501a1f90 con 0x7f3f50100160 2024-02-09T22:51:00.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.788+0000 7f3f477fe640 1 -- 172.21.15.3:0/2329221855 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f3f4c096f80 con 0x7f3f50100160 2024-02-09T22:51:00.792 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:00.793 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:51:00.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f3c0a20c0 msgr2=0x7f3f3c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f3c0a20c0 0x7f3f3c0a4580 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f3f5019a970 tx=0x7f3f40009590 comp rx=0 tx=0).stop 2024-02-09T22:51:00.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 msgr2=0x7f3f50199460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 0x7f3f50199460 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c00d950 tx=0x7f3f4c00de10 comp rx=0 tx=0).stop 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 shutdown_connections 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3f3c0a20c0 0x7f3f3c0a4580 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3f50101d10 0x7f3f501a0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3f50101350 0x7f3f501999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 --2- 172.21.15.3:0/2329221855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3f50100160 0x7f3f50199460 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 >> 172.21.15.3:0/2329221855 conn(0x7f3f500fb6d0 msgr2=0x7f3f50108ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 shutdown_connections 2024-02-09T22:51:00.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.791+0000 7f3f586eb640 1 -- 172.21.15.3:0/2329221855 wait complete. 2024-02-09T22:51:00.806 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:51:00.807 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:51:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- 172.21.15.3:0/3252774929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650106f30 msgr2=0x7f9650109380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- 172.21.15.3:0/3252774929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650106f30 0x7f9650109380 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f9648009ca0 tx=0x7f964802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- 172.21.15.3:0/3252774929 shutdown_connections 2024-02-09T22:51:00.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- 172.21.15.3:0/3252774929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650106f30 0x7f9650109380 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- 172.21.15.3:0/3252774929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650104600 0x7f96501069f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- 172.21.15.3:0/3252774929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9650101cd0 0x7f96501040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- 172.21.15.3:0/3252774929 >> 172.21.15.3:0/3252774929 conn(0x7f96500fb6c0 msgr2=0x7f96500fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:00.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- 172.21.15.3:0/3252774929 shutdown_connections 2024-02-09T22:51:00.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- 172.21.15.3:0/3252774929 wait complete. 2024-02-09T22:51:00.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 Processor -- start 2024-02-09T22:51:00.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- start start 2024-02-09T22:51:00.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9650101cd0 0x7f9650101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 0x7f96500ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650106f30 0x7f96501001c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650100920 con 0x7f9650104600 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9650100a90 con 0x7f9650101cd0 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f9657415640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96501a1320 con 0x7f9650106f30 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 0x7f96500ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 0x7f96500ffc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40000/0 (socket says 172.21.15.3:40000) 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 -- 172.21.15.3:0/2122047490 learned_addr learned my addr 172.21.15.3:0/2122047490 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96551b1640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9650101cd0 0x7f9650101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96559b2640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650106f30 0x7f96501001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 -- 172.21.15.3:0/2122047490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650106f30 msgr2=0x7f96501001c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650106f30 0x7f96501001c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 -- 172.21.15.3:0/2122047490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9650101cd0 msgr2=0x7f9650101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9650101cd0 0x7f9650101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96549b0640 1 -- 172.21.15.3:0/2122047490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9648009900 con 0x7f9650104600 2024-02-09T22:51:00.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.915+0000 7f96559b2640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650106f30 0x7f96501001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.916+0000 7f96549b0640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 0x7f96500ffc80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f964000ea20 tx=0x7f964000eee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.916+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96400096c0 con 0x7f9650104600 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.916+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9640005500 con 0x7f9650104600 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.916+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96501a15d0 con 0x7f9650104600 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.916+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9640010430 con 0x7f9650104600 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.916+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96501a1a90 con 0x7f9650104600 2024-02-09T22:51:00.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.917+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96500fd1b0 con 0x7f9650104600 2024-02-09T22:51:00.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.918+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f9640005000 con 0x7f9650104600 2024-02-09T22:51:00.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.919+0000 7f96467fc640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f963c0a20c0 0x7f963c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:00.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.919+0000 7f96551b1640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f963c0a20c0 0x7f963c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:00.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.919+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f9640014070 con 0x7f9650104600 2024-02-09T22:51:00.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.919+0000 7f96551b1640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f963c0a20c0 0x7f963c0a4580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f964c00bbe0 tx=0x7f964c00b390 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:00.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:00.923+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f9640097240 con 0x7f9650104600 2024-02-09T22:51:00.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:00 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1837446821' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-02-09T22:51:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.105+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f965010cdf0 con 0x7f9650104600 2024-02-09T22:51:01.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.105+0000 7f96467fc640 1 -- 172.21.15.3:0/2122047490 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f965010cdf0 con 0x7f9650104600 2024-02-09T22:51:01.109 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:51:01.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f963c0a20c0 msgr2=0x7f963c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f963c0a20c0 0x7f963c0a4580 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f964c00bbe0 tx=0x7f964c00b390 comp rx=0 tx=0).stop 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 msgr2=0x7f96500ffc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 0x7f96500ffc80 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f964000ea20 tx=0x7f964000eee0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 shutdown_connections 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f963c0a20c0 0x7f963c0a4580 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9650106f30 0x7f96501001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9650104600 0x7f96500ffc80 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 --2- 172.21.15.3:0/2122047490 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9650101cd0 0x7f9650101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.107+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 >> 172.21.15.3:0/2122047490 conn(0x7f96500fb6c0 msgr2=0x7f9650102630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.108+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 shutdown_connections 2024-02-09T22:51:01.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.108+0000 7f9657415640 1 -- 172.21.15.3:0/2122047490 wait complete. 2024-02-09T22:51:01.122 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 -- 172.21.15.3:0/2544162458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101d00 msgr2=0x7f8388108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 --2- 172.21.15.3:0/2544162458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101d00 0x7f8388108590 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f8380009ca0 tx=0x7f838002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 -- 172.21.15.3:0/2544162458 shutdown_connections 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 --2- 172.21.15.3:0/2544162458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101340 0x7f83881017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 --2- 172.21.15.3:0/2544162458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101d00 0x7f8388108590 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 --2- 172.21.15.3:0/2544162458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 0x7f8388100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 -- 172.21.15.3:0/2544162458 >> 172.21.15.3:0/2544162458 conn(0x7f83880fb6c0 msgr2=0x7f83880fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 -- 172.21.15.3:0/2544162458 shutdown_connections 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.226+0000 7f838d2a7640 1 -- 172.21.15.3:0/2544162458 wait complete. 2024-02-09T22:51:01.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 Processor -- start 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 -- start start 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 0x7f8388197330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 0x7f8388197870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101d00 0x7f838819e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838819ee20 con 0x7f8388101340 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838819ef90 con 0x7f8388101d00 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f838d2a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838819f150 con 0x7f8388100150 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 0x7f8388197870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 0x7f8388197870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40012/0 (socket says 172.21.15.3:40012) 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 -- 172.21.15.3:0/2609457172 learned_addr learned my addr 172.21.15.3:0/2609457172 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f83877fe640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101d00 0x7f838819e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f8386ffd640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 0x7f8388197330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 -- 172.21.15.3:0/2609457172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 msgr2=0x7f8388197330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 0x7f8388197330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 -- 172.21.15.3:0/2609457172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101d00 msgr2=0x7f838819e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101d00 0x7f838819e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 -- 172.21.15.3:0/2609457172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8380009900 con 0x7f8388101340 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f83877fe640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101d00 0x7f838819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f8386ffd640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 0x7f8388197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.227+0000 7f837ffff640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 0x7f8388197870 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f837000d950 tx=0x7f837000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.228+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370009c30 con 0x7f8388101340 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.228+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8370009d90 con 0x7f8388101340 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.228+0000 7f838d2a7640 1 -- 172.21.15.3:0/2609457172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838819f430 con 0x7f8388101340 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.228+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837000f7e0 con 0x7f8388101340 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.228+0000 7f838d2a7640 1 -- 172.21.15.3:0/2609457172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838819f990 con 0x7f8388101340 2024-02-09T22:51:01.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.229+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f836c005290 con 0x7f8388101340 2024-02-09T22:51:01.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.230+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f8370010ad0 con 0x7f8388101340 2024-02-09T22:51:01.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.231+0000 7f8384ff9640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f83740a2140 0x7f83740a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.231+0000 7f8386ffd640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f83740a2140 0x7f83740a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.231+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f83700cd910 con 0x7f8388101340 2024-02-09T22:51:01.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.231+0000 7f8386ffd640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f83740a2140 0x7f83740a4600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f83881011a0 tx=0x7f8378005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:01.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.234+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8370096950 con 0x7f8388101340 2024-02-09T22:51:01.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.415+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f836c005ac0 con 0x7f8388101340 2024-02-09T22:51:01.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.416+0000 7f8384ff9640 1 -- 172.21.15.3:0/2609457172 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f83700170a0 con 0x7f8388101340 2024-02-09T22:51:01.419 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f83740a2140 msgr2=0x7f83740a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f83740a2140 0x7f83740a4600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f83881011a0 tx=0x7f8378005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 msgr2=0x7f8388197870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 0x7f8388197870 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f837000d950 tx=0x7f837000de10 comp rx=0 tx=0).stop 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 shutdown_connections 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f83740a2140 0x7f83740a4600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8388101d00 0x7f838819e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8388101340 0x7f8388197870 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 --2- 172.21.15.3:0/2609457172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8388100150 0x7f8388197330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 >> 172.21.15.3:0/2609457172 conn(0x7f83880fb6c0 msgr2=0x7f83880fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:01.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 shutdown_connections 2024-02-09T22:51:01.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.418+0000 7f837dffb640 1 -- 172.21.15.3:0/2609457172 wait complete. 2024-02-09T22:51:01.432 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 -- 172.21.15.3:0/3702940125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 msgr2=0x7f5d88100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 --2- 172.21.15.3:0/3702940125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d88100550 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009900 tx=0x7f5d7002bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 -- 172.21.15.3:0/3702940125 shutdown_connections 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 --2- 172.21.15.3:0/3702940125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 0x7f5d88108570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 --2- 172.21.15.3:0/3702940125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88101320 0x7f5d881017a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 --2- 172.21.15.3:0/3702940125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d88100550 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.535+0000 7f5d8d193640 1 -- 172.21.15.3:0/3702940125 >> 172.21.15.3:0/3702940125 conn(0x7f5d880fb660 msgr2=0x7f5d880fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.536+0000 7f5d8d193640 1 -- 172.21.15.3:0/3702940125 shutdown_connections 2024-02-09T22:51:01.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.536+0000 7f5d8d193640 1 -- 172.21.15.3:0/3702940125 wait complete. 2024-02-09T22:51:01.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.536+0000 7f5d8d193640 1 Processor -- start 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.536+0000 7f5d8d193640 1 -- start start 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d881972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88101320 0x7f5d88197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 0x7f5d8819e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8819edc0 con 0x7f5d88100130 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8819ef30 con 0x7f5d88101ce0 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8819f0f0 con 0x7f5d88101320 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d881972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d881972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40040/0 (socket says 172.21.15.3:40040) 2024-02-09T22:51:01.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 -- 172.21.15.3:0/1376681119 learned_addr learned my addr 172.21.15.3:0/1376681119 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8659c640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88101320 0x7f5d88197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8759e640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 0x7f5d8819e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 -- 172.21.15.3:0/1376681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88101320 msgr2=0x7f5d88197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88101320 0x7f5d88197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 -- 172.21.15.3:0/1376681119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 msgr2=0x7f5d8819e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 0x7f5d8819e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 -- 172.21.15.3:0/1376681119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70009560 con 0x7f5d88100130 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8759e640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 0x7f5d8819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d86d9d640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d881972d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5d70036040 tx=0x7f5d700042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d700044e0 con 0x7f5d88100130 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8819f370 con 0x7f5d88100130 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8819f8d0 con 0x7f5d88100130 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d700054e0 con 0x7f5d88100130 2024-02-09T22:51:01.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.537+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d7003e450 con 0x7f5d88100130 2024-02-09T22:51:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.539+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8810bb00 con 0x7f5d88100130 2024-02-09T22:51:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.540+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f5d70003710 con 0x7f5d88100130 2024-02-09T22:51:01.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.541+0000 7f5d7ffff640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5d740a20c0 0x7f5d740a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.541+0000 7f5d8659c640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5d740a20c0 0x7f5d740a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.541+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5d700f0190 con 0x7f5d88100130 2024-02-09T22:51:01.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.541+0000 7f5d8659c640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5d740a20c0 0x7f5d740a4580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5d881987e0 tx=0x7f5d7800a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:01.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.549+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5d700b9110 con 0x7f5d88100130 2024-02-09T22:51:01.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:01 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1837446821' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-02-09T22:51:01.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:01 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:51:01.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:01 smithi089 ceph-mon[36665]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-02-09T22:51:01.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:01 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2329221855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:01.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:01 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2122047490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:01.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:01 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2609457172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:01.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.732+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5d8819f500 con 0x7f5d88100130 2024-02-09T22:51:01.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.733+0000 7f5d7ffff640 1 -- 172.21.15.3:0/1376681119 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5d8819f500 con 0x7f5d88100130 2024-02-09T22:51:01.736 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:51:01.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5d740a20c0 msgr2=0x7f5d740a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5d740a20c0 0x7f5d740a4580 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5d881987e0 tx=0x7f5d7800a2d0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 msgr2=0x7f5d881972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d881972d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5d70036040 tx=0x7f5d700042f0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 shutdown_connections 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5d740a20c0 0x7f5d740a4580 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d88101ce0 0x7f5d8819e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d88101320 0x7f5d88197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.734+0000 7f5d8d193640 1 --2- 172.21.15.3:0/1376681119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d88100130 0x7f5d881972d0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.735+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 >> 172.21.15.3:0/1376681119 conn(0x7f5d880fb660 msgr2=0x7f5d88068500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.735+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 shutdown_connections 2024-02-09T22:51:01.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.735+0000 7f5d8d193640 1 -- 172.21.15.3:0/1376681119 wait complete. 2024-02-09T22:51:01.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1837446821' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-02-09T22:51:01.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:01 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:51:01.747 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:01 smithi003 ceph-mon[32678]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-02-09T22:51:01.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2329221855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:01.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2122047490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:01.748 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:01 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2609457172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:01.749 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:51:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- 172.21.15.3:0/3314698739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4102e30 msgr2=0x7f2bd41096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- 172.21.15.3:0/3314698739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4102e30 0x7f2bd41096c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc009ca0 tx=0x7f2bcc02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- 172.21.15.3:0/3314698739 shutdown_connections 2024-02-09T22:51:01.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- 172.21.15.3:0/3314698739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4102e30 0x7f2bd41096c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- 172.21.15.3:0/3314698739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102470 0x7f2bd41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- 172.21.15.3:0/3314698739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2bd4101280 0x7f2bd41016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- 172.21.15.3:0/3314698739 >> 172.21.15.3:0/3314698739 conn(0x7f2bd40fc7f0 msgr2=0x7f2bd40fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- 172.21.15.3:0/3314698739 shutdown_connections 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- 172.21.15.3:0/3314698739 wait complete. 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 Processor -- start 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- start start 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 0x7f2bd4197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2bd4102470 0x7f2bd4197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102e30 0x7f2bd419e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd419ed30 con 0x7f2bd4101280 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd419eea0 con 0x7f2bd4102470 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.849+0000 7f2bda881640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bd419f060 con 0x7f2bd4102e30 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 0x7f2bd4197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 0x7f2bd4197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40050/0 (socket says 172.21.15.3:40050) 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 -- 172.21.15.3:0/1324402071 learned_addr learned my addr 172.21.15.3:0/1324402071 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd8e1e640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102e30 0x7f2bd419e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd37fe640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2bd4102470 0x7f2bd4197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 -- 172.21.15.3:0/1324402071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102e30 msgr2=0x7f2bd419e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102e30 0x7f2bd419e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 -- 172.21.15.3:0/1324402071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2bd4102470 msgr2=0x7f2bd4197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2bd4102470 0x7f2bd4197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 -- 172.21.15.3:0/1324402071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bcc009900 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd8e1e640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102e30 0x7f2bd419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd3fff640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 0x7f2bd4197240 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d770 tx=0x7f2bc400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4009c30 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bd419f340 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2bc400de80 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4010460 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.850+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bd419f8a0 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.851+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bd41085a0 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.852+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f2bc4003f60 con 0x7f2bd4101280 2024-02-09T22:51:01.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.853+0000 7f2bd17fa640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2bc00a20c0 0x7f2bc00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.853+0000 7f2bd37fe640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2bc00a20c0 0x7f2bc00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.853+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f2bc40cda90 con 0x7f2bd4101280 2024-02-09T22:51:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.854+0000 7f2bd37fe640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2bc00a20c0 0x7f2bc00a4580 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4198750 tx=0x7f2bbc00b590 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:01.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:01.858+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f2bc4096a70 con 0x7f2bd4101280 2024-02-09T22:51:01.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:01 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1837446821' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-02-09T22:51:01.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:01 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:51:01.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:01 smithi064 ceph-mon[36638]: pgmap v345: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 170 B/s wr, 9 op/s 2024-02-09T22:51:01.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:01 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2329221855' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:01.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:01 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2122047490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:01.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:01 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2609457172' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.041+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2bd410cc90 con 0x7f2bd4101280 2024-02-09T22:51:02.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.043+0000 7f2bd17fa640 1 -- 172.21.15.3:0/1324402071 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f2bc409a130 con 0x7f2bd4101280 2024-02-09T22:51:02.046 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:02.046 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2bc00a20c0 msgr2=0x7f2bc00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2bc00a20c0 0x7f2bc00a4580 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2bd4198750 tx=0x7f2bbc00b590 comp rx=0 tx=0).stop 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 msgr2=0x7f2bd4197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 0x7f2bd4197240 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f2bc400d770 tx=0x7f2bc400dc30 comp rx=0 tx=0).stop 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 shutdown_connections 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f2bc00a20c0 0x7f2bc00a4580 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2bd4102470 0x7f2bd4197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2bd4102e30 0x7f2bd419e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 --2- 172.21.15.3:0/1324402071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2bd4101280 0x7f2bd4197240 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 >> 172.21.15.3:0/1324402071 conn(0x7f2bd40fc7f0 msgr2=0x7f2bd4105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 shutdown_connections 2024-02-09T22:51:02.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.045+0000 7f2bda881640 1 -- 172.21.15.3:0/1324402071 wait complete. 2024-02-09T22:51:02.059 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:51:02.060 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:51:02.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 -- 172.21.15.3:0/2529910127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c100150 msgr2=0x7fab3c100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 --2- 172.21.15.3:0/2529910127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c100150 0x7fab3c100570 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fab34009900 tx=0x7fab3402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 -- 172.21.15.3:0/2529910127 shutdown_connections 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 --2- 172.21.15.3:0/2529910127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab3c101d00 0x7fab3c108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 --2- 172.21.15.3:0/2529910127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab3c101340 0x7fab3c1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 --2- 172.21.15.3:0/2529910127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c100150 0x7fab3c100570 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 -- 172.21.15.3:0/2529910127 >> 172.21.15.3:0/2529910127 conn(0x7fab3c0fb6c0 msgr2=0x7fab3c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 -- 172.21.15.3:0/2529910127 shutdown_connections 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.163+0000 7fab41a4e640 1 -- 172.21.15.3:0/2529910127 wait complete. 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 Processor -- start 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 -- start start 2024-02-09T22:51:02.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab3c100150 0x7fab3c196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab3c101340 0x7fab3c196750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 0x7fab3c19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c19dd00 con 0x7fab3c101d00 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c19de70 con 0x7fab3c101340 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab41a4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c19e030 con 0x7fab3c100150 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 0x7fab3c19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab3b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 0x7fab3c19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40072/0 (socket says 172.21.15.3:40072) 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab3b7fe640 1 -- 172.21.15.3:0/268654054 learned_addr learned my addr 172.21.15.3:0/268654054 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.164+0000 7fab3affd640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab3c100150 0x7fab3c196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3b7fe640 1 -- 172.21.15.3:0/268654054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab3c100150 msgr2=0x7fab3c196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3b7fe640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab3c100150 0x7fab3c196210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3b7fe640 1 -- 172.21.15.3:0/268654054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab3c101340 msgr2=0x7fab3c196750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3a7fc640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab3c101340 0x7fab3c196750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3b7fe640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab3c101340 0x7fab3c196750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3b7fe640 1 -- 172.21.15.3:0/268654054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34009560 con 0x7fab3c101d00 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab3b7fe640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 0x7fab3c19d7c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00eca0 tx=0x7fab2c00b690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c013070 con 0x7fab3c101d00 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab2c005020 con 0x7fab3c101d00 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3c068560 con 0x7fab3c101d00 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c0099b0 con 0x7fab3c101d00 2024-02-09T22:51:02.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.165+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c068a20 con 0x7fab3c101d00 2024-02-09T22:51:02.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.166+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c069400 con 0x7fab3c101d00 2024-02-09T22:51:02.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.167+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fab2c005520 con 0x7fab3c101d00 2024-02-09T22:51:02.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.168+0000 7fab40a4c640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab280a20c0 0x7fab280a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.168+0000 7fab3affd640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab280a20c0 0x7fab280a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.168+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fab2c0cf6a0 con 0x7fab3c101d00 2024-02-09T22:51:02.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.168+0000 7fab3affd640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab280a20c0 0x7fab280a4580 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab3402c6a0 tx=0x7fab34005fd0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:02.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.172+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fab2c051d20 con 0x7fab3c101d00 2024-02-09T22:51:02.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.355+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab3c107040 con 0x7fab3c101d00 2024-02-09T22:51:02.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.356+0000 7fab40a4c640 1 -- 172.21.15.3:0/268654054 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab2c098680 con 0x7fab3c101d00 2024-02-09T22:51:02.359 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:51:02.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab280a20c0 msgr2=0x7fab280a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab280a20c0 0x7fab280a4580 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fab3402c6a0 tx=0x7fab34005fd0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 msgr2=0x7fab3c19d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 0x7fab3c19d7c0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fab2c00eca0 tx=0x7fab2c00b690 comp rx=0 tx=0).stop 2024-02-09T22:51:02.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 shutdown_connections 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab280a20c0 0x7fab280a4580 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab3c101340 0x7fab3c196750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab3c101d00 0x7fab3c19d7c0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 --2- 172.21.15.3:0/268654054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab3c100150 0x7fab3c196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 >> 172.21.15.3:0/268654054 conn(0x7fab3c0fb6c0 msgr2=0x7fab3c104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 shutdown_connections 2024-02-09T22:51:02.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.358+0000 7fab41a4e640 1 -- 172.21.15.3:0/268654054 wait complete. 2024-02-09T22:51:02.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:51:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 -- 172.21.15.3:0/2725461389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 msgr2=0x7fc030108590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2725461389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030108590 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc020009ca0 tx=0x7fc02002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 -- 172.21.15.3:0/2725461389 shutdown_connections 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2725461389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030101340 0x7fc0301017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2725461389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030108590 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2725461389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc030100150 0x7fc030100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 -- 172.21.15.3:0/2725461389 >> 172.21.15.3:0/2725461389 conn(0x7fc0300fb6c0 msgr2=0x7fc0300fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.476+0000 7fc0370c9640 1 -- 172.21.15.3:0/2725461389 shutdown_connections 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 -- 172.21.15.3:0/2725461389 wait complete. 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 Processor -- start 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 -- start start 2024-02-09T22:51:02.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030100150 0x7fc0301040e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc030101340 0x7fc030102720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030102c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0301033c0 con 0x7fc030101d00 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc030103530 con 0x7fc030100150 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc0370c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0301a13a0 con 0x7fc030101340 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc035666640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030102c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc035666640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030102c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40088/0 (socket says 172.21.15.3:40088) 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc035666640 1 -- 172.21.15.3:0/2494716361 learned_addr learned my addr 172.21.15.3:0/2494716361 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.477+0000 7fc034e65640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030100150 0x7fc0301040e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc035666640 1 -- 172.21.15.3:0/2494716361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc030101340 msgr2=0x7fc030102720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc02ffff640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc030101340 0x7fc030102720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc035666640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc030101340 0x7fc030102720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc035666640 1 -- 172.21.15.3:0/2494716361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030100150 msgr2=0x7fc0301040e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc035666640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030100150 0x7fc0301040e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc035666640 1 -- 172.21.15.3:0/2494716361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc020009900 con 0x7fc030101d00 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc034e65640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030100150 0x7fc0301040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc035666640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030102c60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc020032650 tx=0x7fc020034d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020032ed0 con 0x7fc030101d00 2024-02-09T22:51:02.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0301a15f0 con 0x7fc030101d00 2024-02-09T22:51:02.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc020003e50 con 0x7fc030101d00 2024-02-09T22:51:02.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc020040450 con 0x7fc030101d00 2024-02-09T22:51:02.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.478+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0301a1aa0 con 0x7fc030101d00 2024-02-09T22:51:02.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.479+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03010bb60 con 0x7fc030101d00 2024-02-09T22:51:02.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.480+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fc02000a040 con 0x7fc030101d00 2024-02-09T22:51:02.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.481+0000 7fc02dffb640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0180a20c0 0x7fc0180a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.481+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fc0200f0be0 con 0x7fc030101d00 2024-02-09T22:51:02.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.481+0000 7fc034e65640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0180a20c0 0x7fc0180a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.481+0000 7fc034e65640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0180a20c0 0x7fc0180a4580 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc028005d00 tx=0x7fc028005c90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:02.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.485+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fc0200b9bc0 con 0x7fc030101d00 2024-02-09T22:51:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:02 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1376681119' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:02 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1324402071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:02 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/268654054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.665+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0301a1cc0 con 0x7fc030101d00 2024-02-09T22:51:02.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.665+0000 7fc02dffb640 1 -- 172.21.15.3:0/2494716361 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc0200bd280 con 0x7fc030101d00 2024-02-09T22:51:02.668 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:51:02.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0180a20c0 msgr2=0x7fc0180a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0180a20c0 0x7fc0180a4580 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fc028005d00 tx=0x7fc028005c90 comp rx=0 tx=0).stop 2024-02-09T22:51:02.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 msgr2=0x7fc030102c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030102c60 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc020032650 tx=0x7fc020034d40 comp rx=0 tx=0).stop 2024-02-09T22:51:02.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 shutdown_connections 2024-02-09T22:51:02.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fc0180a20c0 0x7fc0180a4580 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc030101d00 0x7fc030102c60 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc030101340 0x7fc030102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 --2- 172.21.15.3:0/2494716361 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc030100150 0x7fc0301040e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 >> 172.21.15.3:0/2494716361 conn(0x7fc0300fb6c0 msgr2=0x7fc0300fd140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 shutdown_connections 2024-02-09T22:51:02.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.667+0000 7fc0370c9640 1 -- 172.21.15.3:0/2494716361 wait complete. 2024-02-09T22:51:02.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:51:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:02 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1376681119' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:02 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1324402071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:02.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:02 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/268654054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 -- 172.21.15.3:0/800640973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8104600 msgr2=0x7f3ee81069f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 --2- 172.21.15.3:0/800640973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8104600 0x7f3ee81069f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f3edc009900 tx=0x7f3edc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:02.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 -- 172.21.15.3:0/800640973 shutdown_connections 2024-02-09T22:51:02.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 --2- 172.21.15.3:0/800640973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee8106f30 0x7f3ee8109380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 --2- 172.21.15.3:0/800640973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8104600 0x7f3ee81069f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 --2- 172.21.15.3:0/800640973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8101cd0 0x7f3ee81040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 -- 172.21.15.3:0/800640973 >> 172.21.15.3:0/800640973 conn(0x7f3ee80fb6c0 msgr2=0x7f3ee80fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 -- 172.21.15.3:0/800640973 shutdown_connections 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.784+0000 7f3eee577640 1 -- 172.21.15.3:0/800640973 wait complete. 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 Processor -- start 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 -- start start 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee8101cd0 0x7f3ee8101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8104600 0x7f3ee80ffc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 0x7f3ee81001c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8100920 con 0x7f3ee8106f30 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee8100a90 con 0x7f3ee8104600 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eee577640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ee81a13d0 con 0x7f3ee8101cd0 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 0x7f3ee81001c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 0x7f3ee81001c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40100/0 (socket says 172.21.15.3:40100) 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 -- 172.21.15.3:0/1461072674 learned_addr learned my addr 172.21.15.3:0/1461072674 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3ee7fff640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee8101cd0 0x7f3ee8101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3ee77fe640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8104600 0x7f3ee80ffc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 -- 172.21.15.3:0/1461072674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee8101cd0 msgr2=0x7f3ee8101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee8101cd0 0x7f3ee8101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 -- 172.21.15.3:0/1461072674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8104600 msgr2=0x7f3ee80ffc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8104600 0x7f3ee80ffc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3eecb14640 1 -- 172.21.15.3:0/1461072674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc009560 con 0x7f3ee8106f30 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.785+0000 7f3ee77fe640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8104600 0x7f3ee80ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.786+0000 7f3eecb14640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 0x7f3ee81001c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000cb00 tx=0x7f3ee000cf00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.786+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee000adc0 con 0x7f3ee8106f30 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.786+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3ee0005020 con 0x7f3ee8106f30 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.786+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ee81a1680 con 0x7f3ee8106f30 2024-02-09T22:51:02.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.786+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ee000f6b0 con 0x7f3ee8106f30 2024-02-09T22:51:02.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.786+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ee81a1be0 con 0x7f3ee8106f30 2024-02-09T22:51:02.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.787+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ee80fd1b0 con 0x7f3ee8106f30 2024-02-09T22:51:02.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.788+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3ee0005190 con 0x7f3ee8106f30 2024-02-09T22:51:02.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.788+0000 7f3ee57fa640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ed00a20c0 0x7f3ed00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:02.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.788+0000 7f3ee7fff640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ed00a20c0 0x7f3ed00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:02.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.788+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3ee00cdf90 con 0x7f3ee8106f30 2024-02-09T22:51:02.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.789+0000 7f3ee7fff640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ed00a20c0 0x7f3ed00a4580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009990 tx=0x7f3ed8009110 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:02.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.793+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3ee0096f70 con 0x7f3ee8106f30 2024-02-09T22:51:02.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1376681119' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1324402071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:02.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/268654054' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:02.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.973+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3ee802cdb0 con 0x7f3ee8106f30 2024-02-09T22:51:02.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.974+0000 7f3ee57fa640 1 -- 172.21.15.3:0/1461072674 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3ee001d020 con 0x7f3ee8106f30 2024-02-09T22:51:02.977 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:51:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.975+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ed00a20c0 msgr2=0x7f3ed00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.975+0000 7f3eee577640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ed00a20c0 0x7f3ed00a4580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009990 tx=0x7f3ed8009110 comp rx=0 tx=0).stop 2024-02-09T22:51:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.975+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 msgr2=0x7f3ee81001c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.975+0000 7f3eee577640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 0x7f3ee81001c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f3ee000cb00 tx=0x7f3ee000cf00 comp rx=0 tx=0).stop 2024-02-09T22:51:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 shutdown_connections 2024-02-09T22:51:02.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3ed00a20c0 0x7f3ed00a4580 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3ee8104600 0x7f3ee80ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3ee8106f30 0x7f3ee81001c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 --2- 172.21.15.3:0/1461072674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3ee8101cd0 0x7f3ee8101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:02.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 >> 172.21.15.3:0/1461072674 conn(0x7f3ee80fb6c0 msgr2=0x7f3ee8102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:02.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 shutdown_connections 2024-02-09T22:51:02.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:02.976+0000 7f3eee577640 1 -- 172.21.15.3:0/1461072674 wait complete. 2024-02-09T22:51:02.990 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-02-09T22:51:02.990 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:51:03.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 -- 172.21.15.3:0/2176962850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44441004a0 msgr2=0x7f4444100920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 --2- 172.21.15.3:0/2176962850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44441004a0 0x7f4444100920 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f4438009930 tx=0x7f443802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 -- 172.21.15.3:0/2176962850 shutdown_connections 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 --2- 172.21.15.3:0/2176962850 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4444100e60 0x7f4444108700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 --2- 172.21.15.3:0/2176962850 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f44441004a0 0x7f4444100920 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 --2- 172.21.15.3:0/2176962850 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 0x7f4444101b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 -- 172.21.15.3:0/2176962850 >> 172.21.15.3:0/2176962850 conn(0x7f44440faeb0 msgr2=0x7f44440fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 -- 172.21.15.3:0/2176962850 shutdown_connections 2024-02-09T22:51:03.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 -- 172.21.15.3:0/2176962850 wait complete. 2024-02-09T22:51:03.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 Processor -- start 2024-02-09T22:51:03.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.090+0000 7f4449def640 1 -- start start 2024-02-09T22:51:03.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44441004a0 0x7f4444196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 0x7f4444196740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 0x7f444419d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444419dcf0 con 0x7f4444100e60 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444419de60 con 0x7f44441004a0 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444419e020 con 0x7f4444101770 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 0x7f4444196740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 0x7f4444196740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40120/0 (socket says 172.21.15.3:40120) 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 -- 172.21.15.3:0/364335959 learned_addr learned my addr 172.21.15.3:0/364335959 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4443fff640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 0x7f444419d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f44437fe640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44441004a0 0x7f4444196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 -- 172.21.15.3:0/364335959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 msgr2=0x7f444419d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 0x7f444419d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 -- 172.21.15.3:0/364335959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44441004a0 msgr2=0x7f4444196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44441004a0 0x7f4444196200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 -- 172.21.15.3:0/364335959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4438009560 con 0x7f4444100e60 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4443fff640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 0x7f444419d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f44437fe640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44441004a0 0x7f4444196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4442ffd640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 0x7f4444196740 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4438009230 tx=0x7f44380042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44380044e0 con 0x7f4444100e60 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44380054e0 con 0x7f4444100e60 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444419e2a0 con 0x7f4444100e60 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f443803e450 con 0x7f4444100e60 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.091+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444419e7b0 con 0x7f4444100e60 2024-02-09T22:51:03.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.092+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44441075e0 con 0x7f4444100e60 2024-02-09T22:51:03.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.093+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f4438003680 con 0x7f4444100e60 2024-02-09T22:51:03.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.094+0000 7f4440ff9640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f442c0a20c0 0x7f442c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.094+0000 7f44437fe640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f442c0a20c0 0x7f442c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.094+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f44380f0140 con 0x7f4444100e60 2024-02-09T22:51:03.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.094+0000 7f44437fe640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f442c0a20c0 0x7f442c0a4580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f44340058c0 tx=0x7f4434005850 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:03.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.098+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f44380b9120 con 0x7f4444100e60 2024-02-09T22:51:03.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.319+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f444402cde0 con 0x7f4444100e60 2024-02-09T22:51:03.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.320+0000 7f4440ff9640 1 -- 172.21.15.3:0/364335959 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f44380bc7e0 con 0x7f4444100e60 2024-02-09T22:51:03.323 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:03.324 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:51:00.398692+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:51:03.324 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2024-02-09T22:51:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f442c0a20c0 msgr2=0x7f442c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f442c0a20c0 0x7f442c0a4580 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f44340058c0 tx=0x7f4434005850 comp rx=0 tx=0).stop 2024-02-09T22:51:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 msgr2=0x7f4444196740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 0x7f4444196740 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f4438009230 tx=0x7f44380042f0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 shutdown_connections 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f442c0a20c0 0x7f442c0a4580 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4444101770 0x7f444419d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4444100e60 0x7f4444196740 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 --2- 172.21.15.3:0/364335959 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f44441004a0 0x7f4444196200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 >> 172.21.15.3:0/364335959 conn(0x7f44440faeb0 msgr2=0x7f44441047b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 shutdown_connections 2024-02-09T22:51:03.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.322+0000 7f4449def640 1 -- 172.21.15.3:0/364335959 wait complete. 2024-02-09T22:51:03.336 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-02-09T22:51:03.339 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-02-09T22:51:03.339 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 -- 172.21.15.3:0/1509893154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc0ff2e0 msgr2=0x7fe3fc0ff760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 --2- 172.21.15.3:0/1509893154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc0ff2e0 0x7fe3fc0ff760 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe3e4009900 tx=0x7fe3e402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 -- 172.21.15.3:0/1509893154 shutdown_connections 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 --2- 172.21.15.3:0/1509893154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 0x7fe3fc109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 --2- 172.21.15.3:0/1509893154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc0ff2e0 0x7fe3fc0ff760 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 --2- 172.21.15.3:0/1509893154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3fc102870 0x7fe3fc102c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 -- 172.21.15.3:0/1509893154 >> 172.21.15.3:0/1509893154 conn(0x7fe3fc0fae60 msgr2=0x7fe3fc0fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:03.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 -- 172.21.15.3:0/1509893154 shutdown_connections 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.439+0000 7fe4013d0640 1 -- 172.21.15.3:0/1509893154 wait complete. 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 Processor -- start 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 -- start start 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3fc0ff2e0 0x7fe3fc101f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 0x7fe3fc1005b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 0x7fe3fc100af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc101250 con 0x7fe3fc102870 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc1013c0 con 0x7fe3fc0ff2e0 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe4013d0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc1a1340 con 0x7fe3fc0ffca0 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 0x7fe3fc100af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 0x7fe3fc100af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40144/0 (socket says 172.21.15.3:40144) 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 -- 172.21.15.3:0/3429605012 learned_addr learned my addr 172.21.15.3:0/3429605012 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:03.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3faffd640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3fc0ff2e0 0x7fe3fc101f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3f3fff640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 0x7fe3fc1005b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 -- 172.21.15.3:0/3429605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 msgr2=0x7fe3fc1005b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 0x7fe3fc1005b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 -- 172.21.15.3:0/3429605012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3fc0ff2e0 msgr2=0x7fe3fc101f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3fc0ff2e0 0x7fe3fc101f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 -- 172.21.15.3:0/3429605012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e4009560 con 0x7fe3fc102870 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3f3fff640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 0x7fe3fc1005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.440+0000 7fe3fb7fe640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 0x7fe3fc100af0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400ead0 tx=0x7fe3f400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.441+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400cde0 con 0x7fe3fc102870 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.441+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3fc1a15f0 con 0x7fe3fc102870 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.441+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3f4005020 con 0x7fe3fc102870 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.441+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f40116b0 con 0x7fe3fc102870 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.441+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3fc1a1b00 con 0x7fe3fc102870 2024-02-09T22:51:03.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.442+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3fc108730 con 0x7fe3fc102870 2024-02-09T22:51:03.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.443+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fe3f4005520 con 0x7fe3fc102870 2024-02-09T22:51:03.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.443+0000 7fe3f8ff9640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe3e80a20c0 0x7fe3e80a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.443+0000 7fe3faffd640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe3e80a20c0 0x7fe3e80a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.443+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe3f40cde90 con 0x7fe3fc102870 2024-02-09T22:51:03.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.444+0000 7fe3faffd640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe3e80a20c0 0x7fe3e80a4580 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc069630 tx=0x7fe3ec00a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:03.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.447+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe3f4096e70 con 0x7fe3fc102870 2024-02-09T22:51:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.625+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3fc02cec0 con 0x7fe3e80a20c0 2024-02-09T22:51:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:03 smithi003 ceph-mon[32678]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-02-09T22:51:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2494716361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1461072674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:03 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/364335959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:03.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.636+0000 7fe3f8ff9640 1 -- 172.21.15.3:0/3429605012 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+344232 (secure 0 0 0) 0x7fe3fc02cec0 con 0x7fe3e80a20c0 2024-02-09T22:51:03.643 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:51:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe3e80a20c0 msgr2=0x7fe3e80a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe3e80a20c0 0x7fe3e80a4580 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc069630 tx=0x7fe3ec00a2d0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 msgr2=0x7fe3fc100af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 0x7fe3fc100af0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fe3f400ead0 tx=0x7fe3f400ef90 comp rx=0 tx=0).stop 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 shutdown_connections 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe3e80a20c0 0x7fe3e80a4580 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3fc102870 0x7fe3fc100af0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3fc0ffca0 0x7fe3fc1005b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 --2- 172.21.15.3:0/3429605012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3fc0ff2e0 0x7fe3fc101f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 >> 172.21.15.3:0/3429605012 conn(0x7fe3fc0fae60 msgr2=0x7fe3fc105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 shutdown_connections 2024-02-09T22:51:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.639+0000 7fe4013d0640 1 -- 172.21.15.3:0/3429605012 wait complete. 2024-02-09T22:51:03.653 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-02-09T22:51:02.566447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657815,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1249,"num_read_kb":752,"num_write":429,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":853,"ondisk_log_size":853,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358444,"kb_used_data":13452,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583380,"statfs":{"total":1151940427776,"available":1151573381120,"internally_reserved":0,"allocated":13774848,"data_stored":7519761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":72,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002762"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582618+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:50:36.582618+0000","last_peered":"2024-02-09T22:50:36.582618+0000","last_clean":"2024-02-09T22:50:36.582618+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:50:36.582618+0000","last_undegraded":"2024-02-09T22:50:36.582618+0000","last_fullsized":"2024-02-09T22:50:36.582618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575819+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:50:36.575819+0000","last_peered":"2024-02-09T22:50:36.575819+0000","last_clean":"2024-02-09T22:50:36.575819+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:50:36.575819+0000","last_undegraded":"2024-02-09T22:50:36.575819+0000","last_fullsized":"2024-02-09T22:50:36.575819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582696+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:50:36.582696+0000","last_peered":"2024-02-09T22:50:36.582696+0000","last_clean":"2024-02-09T22:50:36.582696+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:50:36.582696+0000","last_undegraded":"2024-02-09T22:50:36.582696+0000","last_fullsized":"2024-02-09T22:50:36.582696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.564811+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:50:36.564811+0000","last_peered":"2024-02-09T22:50:36.564811+0000","last_clean":"2024-02-09T22:50:36.564811+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:50:36.564811+0000","last_undegraded":"2024-02-09T22:50:36.564811+0000","last_fullsized":"2024-02-09T22:50:36.564811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:50.286530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571412+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:50:36.571412+0000","last_peered":"2024-02-09T22:50:36.571412+0000","last_clean":"2024-02-09T22:50:36.571412+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:50:36.571412+0000","last_undegraded":"2024-02-09T22:50:36.571412+0000","last_fullsized":"2024-02-09T22:50:36.571412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577030+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:50:36.577030+0000","last_peered":"2024-02-09T22:50:36.577030+0000","last_clean":"2024-02-09T22:50:36.577030+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:50:36.577030+0000","last_undegraded":"2024-02-09T22:50:36.577030+0000","last_fullsized":"2024-02-09T22:50:36.577030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578903+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:50:36.578903+0000","last_peered":"2024-02-09T22:50:36.578903+0000","last_clean":"2024-02-09T22:50:36.578903+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:50:36.578903+0000","last_undegraded":"2024-02-09T22:50:36.578903+0000","last_fullsized":"2024-02-09T22:50:36.578903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.576625+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:50:36.576625+0000","last_peered":"2024-02-09T22:50:36.576625+0000","last_clean":"2024-02-09T22:50:36.576625+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:50:36.576625+0000","last_undegraded":"2024-02-09T22:50:36.576625+0000","last_fullsized":"2024-02-09T22:50:36.576625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584409+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:50:36.584409+0000","last_peered":"2024-02-09T22:50:36.584409+0000","last_clean":"2024-02-09T22:50:36.584409+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:50:36.584409+0000","last_undegraded":"2024-02-09T22:50:36.584409+0000","last_fullsized":"2024-02-09T22:50:36.584409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579365+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:50:36.579365+0000","last_peered":"2024-02-09T22:50:36.579365+0000","last_clean":"2024-02-09T22:50:36.579365+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:50:36.579365+0000","last_undegraded":"2024-02-09T22:50:36.579365+0000","last_fullsized":"2024-02-09T22:50:36.579365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.583991+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:50:36.583991+0000","last_peered":"2024-02-09T22:50:36.583991+0000","last_clean":"2024-02-09T22:50:36.583991+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:50:36.583991+0000","last_undegraded":"2024-02-09T22:50:36.583991+0000","last_fullsized":"2024-02-09T22:50:36.583991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572772+0000","last_change":"2024-02-09T22:50:08.517050+0000","last_active":"2024-02-09T22:50:36.572772+0000","last_peered":"2024-02-09T22:50:36.572772+0000","last_clean":"2024-02-09T22:50:36.572772+0000","last_became_active":"2024-02-09T22:50:08.516688+0000","last_became_peered":"2024-02-09T22:50:08.516688+0000","last_unstale":"2024-02-09T22:50:36.572772+0000","last_undegraded":"2024-02-09T22:50:36.572772+0000","last_fullsized":"2024-02-09T22:50:36.572772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:44:51.672735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578717+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:50:36.578717+0000","last_peered":"2024-02-09T22:50:36.578717+0000","last_clean":"2024-02-09T22:50:36.578717+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:50:36.578717+0000","last_undegraded":"2024-02-09T22:50:36.578717+0000","last_fullsized":"2024-02-09T22:50:36.578717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575908+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:50:36.575908+0000","last_peered":"2024-02-09T22:50:36.575908+0000","last_clean":"2024-02-09T22:50:36.575908+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:50:36.575908+0000","last_undegraded":"2024-02-09T22:50:36.575908+0000","last_fullsized":"2024-02-09T22:50:36.575908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568961+0000","last_change":"2024-02-09T22:50:08.616823+0000","last_active":"2024-02-09T22:50:36.568961+0000","last_peered":"2024-02-09T22:50:36.568961+0000","last_clean":"2024-02-09T22:50:36.568961+0000","last_became_active":"2024-02-09T22:50:08.616482+0000","last_became_peered":"2024-02-09T22:50:08.616482+0000","last_unstale":"2024-02-09T22:50:36.568961+0000","last_undegraded":"2024-02-09T22:50:36.568961+0000","last_fullsized":"2024-02-09T22:50:36.568961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:57:44.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582432+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:50:36.582432+0000","last_peered":"2024-02-09T22:50:36.582432+0000","last_clean":"2024-02-09T22:50:36.582432+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:50:36.582432+0000","last_undegraded":"2024-02-09T22:50:36.582432+0000","last_fullsized":"2024-02-09T22:50:36.582432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582696+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:50:36.582696+0000","last_peered":"2024-02-09T22:50:36.582696+0000","last_clean":"2024-02-09T22:50:36.582696+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:50:36.582696+0000","last_undegraded":"2024-02-09T22:50:36.582696+0000","last_fullsized":"2024-02-09T22:50:36.582696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573224+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:50:36.573224+0000","last_peered":"2024-02-09T22:50:36.573224+0000","last_clean":"2024-02-09T22:50:36.573224+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:50:36.573224+0000","last_undegraded":"2024-02-09T22:50:36.573224+0000","last_fullsized":"2024-02-09T22:50:36.573224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.570014+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:50:36.570014+0000","last_peered":"2024-02-09T22:50:36.570014+0000","last_clean":"2024-02-09T22:50:36.570014+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:50:36.570014+0000","last_undegraded":"2024-02-09T22:50:36.570014+0000","last_fullsized":"2024-02-09T22:50:36.570014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:39.090835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'14","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:59.266178+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:50:59.266178+0000","last_peered":"2024-02-09T22:50:59.266178+0000","last_clean":"2024-02-09T22:50:59.266178+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:50:59.266178+0000","last_undegraded":"2024-02-09T22:50:59.266178+0000","last_fullsized":"2024-02-09T22:50:59.266178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.569807+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:50:36.569807+0000","last_peered":"2024-02-09T22:50:36.569807+0000","last_clean":"2024-02-09T22:50:36.569807+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:50:36.569807+0000","last_undegraded":"2024-02-09T22:50:36.569807+0000","last_fullsized":"2024-02-09T22:50:36.569807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267136+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:50:08.267136+0000","last_peered":"2024-02-09T22:50:08.267136+0000","last_clean":"2024-02-09T22:50:08.267136+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:50:08.267136+0000","last_undegraded":"2024-02-09T22:50:08.267136+0000","last_fullsized":"2024-02-09T22:50:08.267136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582634+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:50:36.582634+0000","last_peered":"2024-02-09T22:50:36.582634+0000","last_clean":"2024-02-09T22:50:36.582634+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:50:36.582634+0000","last_undegraded":"2024-02-09T22:50:36.582634+0000","last_fullsized":"2024-02-09T22:50:36.582634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267805+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:50:08.267805+0000","last_peered":"2024-02-09T22:50:08.267805+0000","last_clean":"2024-02-09T22:50:08.267805+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:50:08.267805+0000","last_undegraded":"2024-02-09T22:50:08.267805+0000","last_fullsized":"2024-02-09T22:50:08.267805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584728+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:50:36.584728+0000","last_peered":"2024-02-09T22:50:36.584728+0000","last_clean":"2024-02-09T22:50:36.584728+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:50:36.584728+0000","last_undegraded":"2024-02-09T22:50:36.584728+0000","last_fullsized":"2024-02-09T22:50:36.584728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267118+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:50:08.267118+0000","last_peered":"2024-02-09T22:50:08.267118+0000","last_clean":"2024-02-09T22:50:08.267118+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:50:08.267118+0000","last_undegraded":"2024-02-09T22:50:08.267118+0000","last_fullsized":"2024-02-09T22:50:08.267118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.570889+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:50:36.570889+0000","last_peered":"2024-02-09T22:50:36.570889+0000","last_clean":"2024-02-09T22:50:36.570889+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:50:36.570889+0000","last_undegraded":"2024-02-09T22:50:36.570889+0000","last_fullsized":"2024-02-09T22:50:36.570889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.275449+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:51:00.275449+0000","last_peered":"2024-02-09T22:51:00.275449+0000","last_clean":"2024-02-09T22:51:00.275449+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:51:00.275449+0000","last_undegraded":"2024-02-09T22:51:00.275449+0000","last_fullsized":"2024-02-09T22:51:00.275449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567770+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:50:36.567770+0000","last_peered":"2024-02-09T22:50:36.567770+0000","last_clean":"2024-02-09T22:50:36.567770+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:50:36.567770+0000","last_undegraded":"2024-02-09T22:50:36.567770+0000","last_fullsized":"2024-02-09T22:50:36.567770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515862+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:50:08.515862+0000","last_peered":"2024-02-09T22:50:08.515862+0000","last_clean":"2024-02-09T22:50:08.515862+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:50:08.515862+0000","last_undegraded":"2024-02-09T22:50:08.515862+0000","last_fullsized":"2024-02-09T22:50:08.515862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571325+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:50:36.571325+0000","last_peered":"2024-02-09T22:50:36.571325+0000","last_clean":"2024-02-09T22:50:36.571325+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:50:36.571325+0000","last_undegraded":"2024-02-09T22:50:36.571325+0000","last_fullsized":"2024-02-09T22:50:36.571325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615818+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:50:08.615818+0000","last_peered":"2024-02-09T22:50:08.615818+0000","last_clean":"2024-02-09T22:50:08.615818+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:50:08.615818+0000","last_undegraded":"2024-02-09T22:50:08.615818+0000","last_fullsized":"2024-02-09T22:50:08.615818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574163+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:50:36.574163+0000","last_peered":"2024-02-09T22:50:36.574163+0000","last_clean":"2024-02-09T22:50:36.574163+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:50:36.574163+0000","last_undegraded":"2024-02-09T22:50:36.574163+0000","last_fullsized":"2024-02-09T22:50:36.574163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615165+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:50:08.615165+0000","last_peered":"2024-02-09T22:50:08.615165+0000","last_clean":"2024-02-09T22:50:08.615165+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:50:08.615165+0000","last_undegraded":"2024-02-09T22:50:08.615165+0000","last_fullsized":"2024-02-09T22:50:08.615165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572865+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:50:36.572865+0000","last_peered":"2024-02-09T22:50:36.572865+0000","last_clean":"2024-02-09T22:50:36.572865+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:50:36.572865+0000","last_undegraded":"2024-02-09T22:50:36.572865+0000","last_fullsized":"2024-02-09T22:50:36.572865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":240,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.274534+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:51:00.274534+0000","last_peered":"2024-02-09T22:51:00.274534+0000","last_clean":"2024-02-09T22:51:00.274534+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:51:00.274534+0000","last_undegraded":"2024-02-09T22:51:00.274534+0000","last_fullsized":"2024-02-09T22:51:00.274534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579648+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:50:36.579648+0000","last_peered":"2024-02-09T22:50:36.579648+0000","last_clean":"2024-02-09T22:50:36.579648+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:50:36.579648+0000","last_undegraded":"2024-02-09T22:50:36.579648+0000","last_fullsized":"2024-02-09T22:50:36.579648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264958+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:50:08.264958+0000","last_peered":"2024-02-09T22:50:08.264958+0000","last_clean":"2024-02-09T22:50:08.264958+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:50:08.264958+0000","last_undegraded":"2024-02-09T22:50:08.264958+0000","last_fullsized":"2024-02-09T22:50:08.264958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575891+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:50:36.575891+0000","last_peered":"2024-02-09T22:50:36.575891+0000","last_clean":"2024-02-09T22:50:36.575891+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:50:36.575891+0000","last_undegraded":"2024-02-09T22:50:36.575891+0000","last_fullsized":"2024-02-09T22:50:36.575891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615718+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:50:08.615718+0000","last_peered":"2024-02-09T22:50:08.615718+0000","last_clean":"2024-02-09T22:50:08.615718+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:50:08.615718+0000","last_undegraded":"2024-02-09T22:50:08.615718+0000","last_fullsized":"2024-02-09T22:50:08.615718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.585093+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:50:36.585093+0000","last_peered":"2024-02-09T22:50:36.585093+0000","last_clean":"2024-02-09T22:50:36.585093+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:50:36.585093+0000","last_undegraded":"2024-02-09T22:50:36.585093+0000","last_fullsized":"2024-02-09T22:50:36.585093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":277,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.275099+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:51:00.275099+0000","last_peered":"2024-02-09T22:51:00.275099+0000","last_clean":"2024-02-09T22:51:00.275099+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:51:00.275099+0000","last_undegraded":"2024-02-09T22:51:00.275099+0000","last_fullsized":"2024-02-09T22:51:00.275099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59951,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":171,"num_write":36,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577233+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:50:36.577233+0000","last_peered":"2024-02-09T22:50:36.577233+0000","last_clean":"2024-02-09T22:50:36.577233+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:50:36.577233+0000","last_undegraded":"2024-02-09T22:50:36.577233+0000","last_fullsized":"2024-02-09T22:50:36.577233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267676+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:50:08.267676+0000","last_peered":"2024-02-09T22:50:08.267676+0000","last_clean":"2024-02-09T22:50:08.267676+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:50:08.267676+0000","last_undegraded":"2024-02-09T22:50:08.267676+0000","last_fullsized":"2024-02-09T22:50:08.267676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574237+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:50:36.574237+0000","last_peered":"2024-02-09T22:50:36.574237+0000","last_clean":"2024-02-09T22:50:36.574237+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:50:36.574237+0000","last_undegraded":"2024-02-09T22:50:36.574237+0000","last_fullsized":"2024-02-09T22:50:36.574237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615556+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:50:08.615556+0000","last_peered":"2024-02-09T22:50:08.615556+0000","last_clean":"2024-02-09T22:50:08.615556+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:50:08.615556+0000","last_undegraded":"2024-02-09T22:50:08.615556+0000","last_fullsized":"2024-02-09T22:50:08.615556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573769+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:50:36.573769+0000","last_peered":"2024-02-09T22:50:36.573769+0000","last_clean":"2024-02-09T22:50:36.573769+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:50:36.573769+0000","last_undegraded":"2024-02-09T22:50:36.573769+0000","last_fullsized":"2024-02-09T22:50:36.573769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":131,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:56.292256+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:50:56.292256+0000","last_peered":"2024-02-09T22:50:56.292256+0000","last_clean":"2024-02-09T22:50:56.292256+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:50:56.292256+0000","last_undegraded":"2024-02-09T22:50:56.292256+0000","last_fullsized":"2024-02-09T22:50:56.292256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579110+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:50:36.579110+0000","last_peered":"2024-02-09T22:50:36.579110+0000","last_clean":"2024-02-09T22:50:36.579110+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:50:36.579110+0000","last_undegraded":"2024-02-09T22:50:36.579110+0000","last_fullsized":"2024-02-09T22:50:36.579110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265893+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:50:08.265893+0000","last_peered":"2024-02-09T22:50:08.265893+0000","last_clean":"2024-02-09T22:50:08.265893+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:50:08.265893+0000","last_undegraded":"2024-02-09T22:50:08.265893+0000","last_fullsized":"2024-02-09T22:50:08.265893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.562328+0000","last_change":"2024-02-09T22:50:08.539326+0000","last_active":"2024-02-09T22:50:36.562328+0000","last_peered":"2024-02-09T22:50:36.562328+0000","last_clean":"2024-02-09T22:50:36.562328+0000","last_became_active":"2024-02-09T22:50:08.539091+0000","last_became_peered":"2024-02-09T22:50:08.539091+0000","last_unstale":"2024-02-09T22:50:36.562328+0000","last_undegraded":"2024-02-09T22:50:36.562328+0000","last_fullsized":"2024-02-09T22:50:36.562328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:31:57.282733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267149+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:50:08.267149+0000","last_peered":"2024-02-09T22:50:08.267149+0000","last_clean":"2024-02-09T22:50:08.267149+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:50:08.267149+0000","last_undegraded":"2024-02-09T22:50:08.267149+0000","last_fullsized":"2024-02-09T22:50:08.267149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573178+0000","last_change":"2024-02-09T22:50:08.616790+0000","last_active":"2024-02-09T22:50:36.573178+0000","last_peered":"2024-02-09T22:50:36.573178+0000","last_clean":"2024-02-09T22:50:36.573178+0000","last_became_active":"2024-02-09T22:50:08.616404+0000","last_became_peered":"2024-02-09T22:50:08.616404+0000","last_unstale":"2024-02-09T22:50:36.573178+0000","last_undegraded":"2024-02-09T22:50:36.573178+0000","last_fullsized":"2024-02-09T22:50:36.573178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:14:18.902026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263341+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:50:08.263341+0000","last_peered":"2024-02-09T22:50:08.263341+0000","last_clean":"2024-02-09T22:50:08.263341+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:50:08.263341+0000","last_undegraded":"2024-02-09T22:50:08.263341+0000","last_fullsized":"2024-02-09T22:50:08.263341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575910+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:50:36.575910+0000","last_peered":"2024-02-09T22:50:36.575910+0000","last_clean":"2024-02-09T22:50:36.575910+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:50:36.575910+0000","last_undegraded":"2024-02-09T22:50:36.575910+0000","last_fullsized":"2024-02-09T22:50:36.575910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'6","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.268171+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:51:00.268171+0000","last_peered":"2024-02-09T22:51:00.268171+0000","last_clean":"2024-02-09T22:51:00.268171+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:51:00.268171+0000","last_undegraded":"2024-02-09T22:51:00.268171+0000","last_fullsized":"2024-02-09T22:51:00.268171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577911+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:50:36.577911+0000","last_peered":"2024-02-09T22:50:36.577911+0000","last_clean":"2024-02-09T22:50:36.577911+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:50:36.577911+0000","last_undegraded":"2024-02-09T22:50:36.577911+0000","last_fullsized":"2024-02-09T22:50:36.577911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572932+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:50:36.572932+0000","last_peered":"2024-02-09T22:50:36.572932+0000","last_clean":"2024-02-09T22:50:36.572932+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:50:36.572932+0000","last_undegraded":"2024-02-09T22:50:36.572932+0000","last_fullsized":"2024-02-09T22:50:36.572932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":133,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:59.296638+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:50:59.296638+0000","last_peered":"2024-02-09T22:50:59.296638+0000","last_clean":"2024-02-09T22:50:59.296638+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:50:59.296638+0000","last_undegraded":"2024-02-09T22:50:59.296638+0000","last_fullsized":"2024-02-09T22:50:59.296638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575148+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:50:36.575148+0000","last_peered":"2024-02-09T22:50:36.575148+0000","last_clean":"2024-02-09T22:50:36.575148+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:50:36.575148+0000","last_undegraded":"2024-02-09T22:50:36.575148+0000","last_fullsized":"2024-02-09T22:50:36.575148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538316+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:50:08.538316+0000","last_peered":"2024-02-09T22:50:08.538316+0000","last_clean":"2024-02-09T22:50:08.538316+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:50:08.538316+0000","last_undegraded":"2024-02-09T22:50:08.538316+0000","last_fullsized":"2024-02-09T22:50:08.538316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264739+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:50:08.264739+0000","last_peered":"2024-02-09T22:50:08.264739+0000","last_clean":"2024-02-09T22:50:08.264739+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:50:08.264739+0000","last_undegraded":"2024-02-09T22:50:08.264739+0000","last_fullsized":"2024-02-09T22:50:08.264739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.569342+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:50:36.569342+0000","last_peered":"2024-02-09T22:50:36.569342+0000","last_clean":"2024-02-09T22:50:36.569342+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:50:36.569342+0000","last_undegraded":"2024-02-09T22:50:36.569342+0000","last_fullsized":"2024-02-09T22:50:36.569342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264784+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:50:08.264784+0000","last_peered":"2024-02-09T22:50:08.264784+0000","last_clean":"2024-02-09T22:50:08.264784+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:50:08.264784+0000","last_undegraded":"2024-02-09T22:50:08.264784+0000","last_fullsized":"2024-02-09T22:50:08.264784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.278066+0000","last_change":"2024-02-09T22:50:08.278066+0000","last_active":"2024-02-09T22:50:08.278066+0000","last_peered":"2024-02-09T22:50:08.278066+0000","last_clean":"2024-02-09T22:50:08.278066+0000","last_became_active":"2024-02-09T22:50:08.271486+0000","last_became_peered":"2024-02-09T22:50:08.271486+0000","last_unstale":"2024-02-09T22:50:08.278066+0000","last_undegraded":"2024-02-09T22:50:08.278066+0000","last_fullsized":"2024-02-09T22:50:08.278066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:54:12.295421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567960+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:50:36.567960+0000","last_peered":"2024-02-09T22:50:36.567960+0000","last_clean":"2024-02-09T22:50:36.567960+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:50:36.567960+0000","last_undegraded":"2024-02-09T22:50:36.567960+0000","last_fullsized":"2024-02-09T22:50:36.567960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'13","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263515+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:50:08.263515+0000","last_peered":"2024-02-09T22:50:08.263515+0000","last_clean":"2024-02-09T22:50:08.263515+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:50:08.263515+0000","last_undegraded":"2024-02-09T22:50:08.263515+0000","last_fullsized":"2024-02-09T22:50:08.263515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574800+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:50:36.574800+0000","last_peered":"2024-02-09T22:50:36.574800+0000","last_clean":"2024-02-09T22:50:36.574800+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:50:36.574800+0000","last_undegraded":"2024-02-09T22:50:36.574800+0000","last_fullsized":"2024-02-09T22:50:36.574800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538049+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:50:08.538049+0000","last_peered":"2024-02-09T22:50:08.538049+0000","last_clean":"2024-02-09T22:50:08.538049+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:50:08.538049+0000","last_undegraded":"2024-02-09T22:50:08.538049+0000","last_fullsized":"2024-02-09T22:50:08.538049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578272+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:50:36.578272+0000","last_peered":"2024-02-09T22:50:36.578272+0000","last_clean":"2024-02-09T22:50:36.578272+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:50:36.578272+0000","last_undegraded":"2024-02-09T22:50:36.578272+0000","last_fullsized":"2024-02-09T22:50:36.578272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537439+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:50:08.537439+0000","last_peered":"2024-02-09T22:50:08.537439+0000","last_clean":"2024-02-09T22:50:08.537439+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:50:08.537439+0000","last_undegraded":"2024-02-09T22:50:08.537439+0000","last_fullsized":"2024-02-09T22:50:08.537439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.538911+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:50:36.538911+0000","last_peered":"2024-02-09T22:50:36.538911+0000","last_clean":"2024-02-09T22:50:36.538911+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:50:36.538911+0000","last_undegraded":"2024-02-09T22:50:36.538911+0000","last_fullsized":"2024-02-09T22:50:36.538911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266189+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:50:08.266189+0000","last_peered":"2024-02-09T22:50:08.266189+0000","last_clean":"2024-02-09T22:50:08.266189+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:50:08.266189+0000","last_undegraded":"2024-02-09T22:50:08.266189+0000","last_fullsized":"2024-02-09T22:50:08.266189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568139+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:50:36.568139+0000","last_peered":"2024-02-09T22:50:36.568139+0000","last_clean":"2024-02-09T22:50:36.568139+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:50:36.568139+0000","last_undegraded":"2024-02-09T22:50:36.568139+0000","last_fullsized":"2024-02-09T22:50:36.568139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.273440+0000","last_change":"2024-02-09T22:50:08.273440+0000","last_active":"2024-02-09T22:50:08.273440+0000","last_peered":"2024-02-09T22:50:08.273440+0000","last_clean":"2024-02-09T22:50:08.273440+0000","last_became_active":"2024-02-09T22:50:08.268960+0000","last_became_peered":"2024-02-09T22:50:08.268960+0000","last_unstale":"2024-02-09T22:50:08.273440+0000","last_undegraded":"2024-02-09T22:50:08.273440+0000","last_fullsized":"2024-02-09T22:50:08.273440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:41.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.576486+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:50:36.576486+0000","last_peered":"2024-02-09T22:50:36.576486+0000","last_clean":"2024-02-09T22:50:36.576486+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:50:36.576486+0000","last_undegraded":"2024-02-09T22:50:36.576486+0000","last_fullsized":"2024-02-09T22:50:36.576486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.542393+0000","last_change":"2024-02-09T22:50:08.542393+0000","last_active":"2024-02-09T22:50:08.542393+0000","last_peered":"2024-02-09T22:50:08.542393+0000","last_clean":"2024-02-09T22:50:08.542393+0000","last_became_active":"2024-02-09T22:50:08.539084+0000","last_became_peered":"2024-02-09T22:50:08.539084+0000","last_unstale":"2024-02-09T22:50:08.542393+0000","last_undegraded":"2024-02-09T22:50:08.542393+0000","last_fullsized":"2024-02-09T22:50:08.542393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:32:59.713756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584235+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:50:36.584235+0000","last_peered":"2024-02-09T22:50:36.584235+0000","last_clean":"2024-02-09T22:50:36.584235+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:50:36.584235+0000","last_undegraded":"2024-02-09T22:50:36.584235+0000","last_fullsized":"2024-02-09T22:50:36.584235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":131,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:56.269509+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:50:56.269509+0000","last_peered":"2024-02-09T22:50:56.269509+0000","last_clean":"2024-02-09T22:50:56.269509+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:50:56.269509+0000","last_undegraded":"2024-02-09T22:50:56.269509+0000","last_fullsized":"2024-02-09T22:50:56.269509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584413+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:50:36.584413+0000","last_peered":"2024-02-09T22:50:36.584413+0000","last_clean":"2024-02-09T22:50:36.584413+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:50:36.584413+0000","last_undegraded":"2024-02-09T22:50:36.584413+0000","last_fullsized":"2024-02-09T22:50:36.584413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615601+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:50:08.615601+0000","last_peered":"2024-02-09T22:50:08.615601+0000","last_clean":"2024-02-09T22:50:08.615601+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:50:08.615601+0000","last_undegraded":"2024-02-09T22:50:08.615601+0000","last_fullsized":"2024-02-09T22:50:08.615601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577804+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:50:36.577804+0000","last_peered":"2024-02-09T22:50:36.577804+0000","last_clean":"2024-02-09T22:50:36.577804+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:50:36.577804+0000","last_undegraded":"2024-02-09T22:50:36.577804+0000","last_fullsized":"2024-02-09T22:50:36.577804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615940+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:50:08.615940+0000","last_peered":"2024-02-09T22:50:08.615940+0000","last_clean":"2024-02-09T22:50:08.615940+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:50:08.615940+0000","last_undegraded":"2024-02-09T22:50:08.615940+0000","last_fullsized":"2024-02-09T22:50:08.615940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578172+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:50:36.578172+0000","last_peered":"2024-02-09T22:50:36.578172+0000","last_clean":"2024-02-09T22:50:36.578172+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:50:36.578172+0000","last_undegraded":"2024-02-09T22:50:36.578172+0000","last_fullsized":"2024-02-09T22:50:36.578172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573750+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:50:36.573750+0000","last_peered":"2024-02-09T22:50:36.573750+0000","last_clean":"2024-02-09T22:50:36.573750+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:50:36.573750+0000","last_undegraded":"2024-02-09T22:50:36.573750+0000","last_fullsized":"2024-02-09T22:50:36.573750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567974+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:50:36.567974+0000","last_peered":"2024-02-09T22:50:36.567974+0000","last_clean":"2024-02-09T22:50:36.567974+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:50:36.567974+0000","last_undegraded":"2024-02-09T22:50:36.567974+0000","last_fullsized":"2024-02-09T22:50:36.567974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:24:49.099644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.566404+0000","last_change":"2024-02-09T22:50:08.272126+0000","last_active":"2024-02-09T22:50:36.566404+0000","last_peered":"2024-02-09T22:50:36.566404+0000","last_clean":"2024-02-09T22:50:36.566404+0000","last_became_active":"2024-02-09T22:50:08.271722+0000","last_became_peered":"2024-02-09T22:50:08.271722+0000","last_unstale":"2024-02-09T22:50:36.566404+0000","last_undegraded":"2024-02-09T22:50:36.566404+0000","last_fullsized":"2024-02-09T22:50:36.566404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:43:31.304504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575883+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:50:36.575883+0000","last_peered":"2024-02-09T22:50:36.575883+0000","last_clean":"2024-02-09T22:50:36.575883+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:50:36.575883+0000","last_undegraded":"2024-02-09T22:50:36.575883+0000","last_fullsized":"2024-02-09T22:50:36.575883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579765+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:50:36.579765+0000","last_peered":"2024-02-09T22:50:36.579765+0000","last_clean":"2024-02-09T22:50:36.579765+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:50:36.579765+0000","last_undegraded":"2024-02-09T22:50:36.579765+0000","last_fullsized":"2024-02-09T22:50:36.579765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568845+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:50:36.568845+0000","last_peered":"2024-02-09T22:50:36.568845+0000","last_clean":"2024-02-09T22:50:36.568845+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:50:36.568845+0000","last_undegraded":"2024-02-09T22:50:36.568845+0000","last_fullsized":"2024-02-09T22:50:36.568845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.563750+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:50:36.563750+0000","last_peered":"2024-02-09T22:50:36.563750+0000","last_clean":"2024-02-09T22:50:36.563750+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:50:36.563750+0000","last_undegraded":"2024-02-09T22:50:36.563750+0000","last_fullsized":"2024-02-09T22:50:36.563750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:57:01.921327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571402+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:50:36.571402+0000","last_peered":"2024-02-09T22:50:36.571402+0000","last_clean":"2024-02-09T22:50:36.571402+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:50:36.571402+0000","last_undegraded":"2024-02-09T22:50:36.571402+0000","last_fullsized":"2024-02-09T22:50:36.571402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574138+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:50:36.574138+0000","last_peered":"2024-02-09T22:50:36.574138+0000","last_clean":"2024-02-09T22:50:36.574138+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:50:36.574138+0000","last_undegraded":"2024-02-09T22:50:36.574138+0000","last_fullsized":"2024-02-09T22:50:36.574138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574591+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:50:36.574591+0000","last_peered":"2024-02-09T22:50:36.574591+0000","last_clean":"2024-02-09T22:50:36.574591+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:50:36.574591+0000","last_undegraded":"2024-02-09T22:50:36.574591+0000","last_fullsized":"2024-02-09T22:50:36.574591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.585107+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:50:36.585107+0000","last_peered":"2024-02-09T22:50:36.585107+0000","last_clean":"2024-02-09T22:50:36.585107+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:50:36.585107+0000","last_undegraded":"2024-02-09T22:50:36.585107+0000","last_fullsized":"2024-02-09T22:50:36.585107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575799+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:50:36.575799+0000","last_peered":"2024-02-09T22:50:36.575799+0000","last_clean":"2024-02-09T22:50:36.575799+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:50:36.575799+0000","last_undegraded":"2024-02-09T22:50:36.575799+0000","last_fullsized":"2024-02-09T22:50:36.575799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571747+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:50:36.571747+0000","last_peered":"2024-02-09T22:50:36.571747+0000","last_clean":"2024-02-09T22:50:36.571747+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:50:36.571747+0000","last_undegraded":"2024-02-09T22:50:36.571747+0000","last_fullsized":"2024-02-09T22:50:36.571747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67213,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1113,"num_read_kb":633,"num_write":246,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":463574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63500000000000001,"15min":0.626},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.752}]},{"osd":2,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.44600000000000001},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.51500000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.88},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.81399999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.73299999999999998}]},{"osd":6,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.66600000000000004},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78600000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.747,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.82799999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92800000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.59699999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.96699999999999997}]}]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32800,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712352,"statfs":{"total":95995035648,"available":95961448448,"internally_reserved":0,"allocated":1662976,"data_stored":1114442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.437}]},{"osd":1,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.496,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.66900000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85399999999999998,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.877,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.72499999999999998,"15min":0.69299999999999995},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.36199999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.502}]},{"osd":5,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.747,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.441,"5min":0.433,"15min":0.433},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57299999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79500000000000004,"15min":0.79100000000000004},"min":{"1min":0.59799999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79900000000000004,"15min":0.80400000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0589999999999999,"5min":1.069,"15min":1.069},"last":0.70999999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.627,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0409999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.002,"5min":1.155,"15min":1.155},"last":0.65400000000000003}]},{"osd":11,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.83199999999999996}]}]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":463392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74199999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66000000000000003,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.63400000000000001},"min":{"1min":0.41699999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.89400000000000002,"5min":1.022,"15min":1.022},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.33500000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61499999999999999,"15min":0.61599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80100000000000005,"5min":2.246,"15min":2.246},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78500000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65800000000000003,"15min":0.67800000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85699999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70499999999999996,"15min":0.71899999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89700000000000002,"5min":1.01,"15min":1.01},"last":0.441}]},{"osd":6,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":0.69099999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76800000000000002,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":2.036,"15min":2.036},"last":0.47499999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.76400000000000001,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90300000000000002,"5min":1.177,"15min":1.177},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.77500000000000002,"15min":0.80100000000000005},"min":{"1min":0.496,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91400000000000003,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.49399999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.76800000000000002,"15min":0.78700000000000003},"min":{"1min":0.49099999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94899999999999995,"5min":1.173,"15min":1.173},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.80000000000000004,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.94199999999999995,"5min":1.387,"15min":1.387},"last":0.53600000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":1.089,"15min":1.089},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.76500000000000001,"15min":0.77600000000000002},"min":{"1min":0.56200000000000006,"5min":0.504,"15min":0.504},"max":{"1min":0.97499999999999998,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.627}]}]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":463658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77400000000000002}]},{"osd":1,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.60599999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.628},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89300000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.371,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.46100000000000002},"min":{"1min":0.35799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.67700000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97299999999999998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.55300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.63,"15min":0.621},"min":{"1min":0.46600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69899999999999995,"15min":0.69999999999999996},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.70599999999999996,"15min":0.68999999999999995},"min":{"1min":0.58099999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69899999999999995,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93700000000000006,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.71099999999999997,"15min":0.67500000000000004},"min":{"1min":0.63700000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0529999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.90800000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.75600000000000001,"15min":0.70299999999999996},"min":{"1min":0.67400000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1990000000000001,"5min":1.371,"15min":1.371},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.77000000000000002,"15min":0.71799999999999997},"min":{"1min":0.69899999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1870000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.88400000000000001}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.871},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.84999999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1056737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48099999999999998,"15min":0.46700000000000003},"min":{"1min":0.308,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.626,"5min":0.997,"15min":0.997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.505},"min":{"1min":0.35999999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":1.01,"15min":1.01},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65800000000000003,"15min":0.64900000000000002},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.874,"5min":1.111,"15min":1.111},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.41299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66600000000000004}]},{"osd":4,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.373,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.56699999999999995,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89900000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.50600000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89200000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.435,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92000000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.81200000000000006}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96599999999999997,"5min":1.016,"15min":1.016},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.73199999999999998,"15min":0.70399999999999996},"min":{"1min":0.60499999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.73699999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76500000000000001,"15min":0.77200000000000002},"min":{"1min":0.55100000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.038,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95399999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.71699999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.63,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.754},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.69699999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.028,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.80000000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83599999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":463417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55400000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.42199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55300000000000005,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.875,"15min":0.875},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.443,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625}]},{"osd":7,"last update":"Fri Feb 9 22:50:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78000000000000003,"5min":0.999,"15min":0.999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.55400000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57099999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84799999999999998,"5min":0.876,"15min":0.876},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.81200000000000006,"5min":0.995,"15min":0.995},"last":0.66200000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65200000000000002,"15min":0.68000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70599999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72099999999999997,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.82199999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.59099999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.81100000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.748}]}]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":523665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57699999999999996,"15min":0.52700000000000002},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.621},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.76300000000000001,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.80600000000000005,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.77700000000000002,"15min":0.60299999999999998},"min":{"1min":0.49399999999999999,"5min":0.373,"15min":0.37},"max":{"1min":0.92400000000000004,"5min":18.18,"15min":18.18},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.61899999999999999,"15min":0.52500000000000002},"min":{"1min":0.52100000000000002,"5min":0.38500000000000001,"15min":0.372},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.94099999999999995,"5min":1.206,"15min":1.206},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.35899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.498,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.79700000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.71899999999999997,"15min":0.623},"min":{"1min":0.59299999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.72399999999999998,"15min":0.63600000000000001},"min":{"1min":0.56899999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66600000000000004,"15min":0.64500000000000002},"min":{"1min":0.48999999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.66100000000000003,"15min":0.629},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.70599999999999996,"15min":0.67200000000000004},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0669999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.86299999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.353,"15min":1.581},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1439999999999999,"5min":18.132000000000001,"15min":18.132000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69399999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.57199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.86899999999999999,"15min":0.85899999999999999},"min":{"1min":0.72899999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.101,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.98399999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78800000000000003,"15min":0.78200000000000003},"min":{"1min":0.60999999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.871,"15min":0.85399999999999998},"min":{"1min":0.67000000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.92200000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.96199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":464448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56899999999999995,"15min":0.495},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.57599999999999996,"15min":0.498},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56699999999999995,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":4,"last update":"Fri Feb 9 22:50:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.745},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Feb 9 22:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.622},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.34599999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76200000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84999999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64200000000000002}]},{"osd":9,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.72099999999999997},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.80200000000000005,"5min":1.081,"15min":1.081},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.749},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95899999999999996,"5min":1.202,"15min":1.202},"last":0.79000000000000004}]},{"osd":10,"last update":"Fri Feb 9 22:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67600000000000005,"15min":0.70399999999999996},"min":{"1min":0.51100000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79000000000000004,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72099999999999997,"15min":0.73899999999999999},"min":{"1min":0.60399999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84799999999999998,"5min":1.137,"15min":1.137},"last":0.79900000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.65400000000000003},"min":{"1min":0.59699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.89900000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.875}]}]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":524417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.45800000000000002},"min":{"1min":0.28299999999999997,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.71199999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.48299999999999998},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.73899999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56899999999999995,"15min":0.49199999999999999},"min":{"1min":0.438,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.60599999999999998,"15min":0.504},"min":{"1min":0.45700000000000002,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55900000000000005,"15min":0.49099999999999999},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.48999999999999999},"min":{"1min":0.39900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.437}]},{"osd":4,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.65800000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.499,"15min":0.435},"min":{"1min":0.32700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71299999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.874,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58899999999999997}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90700000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.51900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.48699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.624}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]},{"osd":11,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.872},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67700000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":462934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57899999999999996,"5min":0.62,"15min":0.62},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.53100000000000003},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78100000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.53500000000000003},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.753,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.746}]},{"osd":2,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.434},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.372,"15min":0.372},"max":{"1min":0.83599999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.73299999999999998,"15min":0.627},"min":{"1min":0.378,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.6920000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86299999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59399999999999997,"15min":0.60999999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76400000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.747,"5min":0.873,"15min":0.873},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.82799999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63800000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.745,"15min":0.70699999999999996},"min":{"1min":0.52700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.877,"5min":5.8789999999999996,"15min":5.8789999999999996},"last":0.69599999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:50:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.69699999999999995,"15min":0.75900000000000001},"min":{"1min":0.378,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.442,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88400000000000001,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.55900000000000005}]},{"osd":10,"last update":"Fri Feb 9 22:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79500000000000004,"15min":0.82999999999999996},"min":{"1min":0.58299999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92500000000000004,"5min":1.26,"15min":1.26},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.74399999999999999},"min":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.94199999999999995,"5min":0.999,"15min":0.999},"last":0.65300000000000002}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.624},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.72899999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":462970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.433,"15min":0.40300000000000002},"min":{"1min":0.34200000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.63100000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46000000000000002,"15min":0.40300000000000002},"min":{"1min":0.38400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.443}]},{"osd":2,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.61799999999999999,"15min":0.57299999999999995},"min":{"1min":0.45200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.01,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.874},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.497,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.871,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.626,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.57999999999999996,"15min":0.56699999999999995},"min":{"1min":0.47599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80400000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.63800000000000001,"15min":0.58599999999999997},"min":{"1min":0.44800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.60399999999999998},"min":{"1min":0.55800000000000005,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78200000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.60199999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.056,"15min":1.056},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93400000000000005,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Feb 9 22:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88600000000000001,"5min":1.097,"15min":1.097},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.496,"5min":0.434,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.55100000000000005}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93300000000000005,"5min":1.129,"15min":1.129},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76900000000000002,"15min":0.748},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.264,"15min":1.264},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.50700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.77400000000000002,"15min":0.78000000000000003},"min":{"1min":0.60399999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.98599999999999999,"5min":1.196,"15min":1.196},"last":0.82999999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.61899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98499999999999999,"5min":1.179,"15min":1.179},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.624,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.99299999999999999,"5min":1.238,"15min":1.238},"last":0.64800000000000002}]},{"osd":11,"last update":"Fri Feb 9 22:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.95299999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1593344,"data_stored":1056107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.498},"min":{"1min":0.438,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66200000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.48099999999999998},"min":{"1min":0.432,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55000000000000004}]},{"osd":1,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.56699999999999995,"15min":0.51600000000000001},"min":{"1min":0.45200000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.749},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.57199999999999995,"15min":0.52500000000000002},"min":{"1min":0.47699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.43099999999999999,"15min":0.40300000000000002},"min":{"1min":0.38200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.374,"5min":0.217,"15min":0.217},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54500000000000004,"15min":0.51300000000000001},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.496},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57199999999999995,"15min":0.53400000000000003},"min":{"1min":0.45200000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.61499999999999999,"15min":0.55700000000000005},"min":{"1min":0.58799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59799999999999998,"15min":0.55000000000000004},"min":{"1min":0.54200000000000004,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60899999999999999,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85299999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.625},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60599999999999998,"15min":0.58299999999999996},"min":{"1min":0.47499999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.81299999999999994,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.64000000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.67200000000000004,"15min":0.65300000000000002},"min":{"1min":0.65100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.67200000000000004,"15min":0.65900000000000003},"min":{"1min":0.63900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90700000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.70499999999999996}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.872,"5min":0.996,"15min":0.996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.63100000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90800000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.58599999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94699999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.75700000000000001,"15min":0.74099999999999999},"min":{"1min":0.73699999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0049999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.751,"15min":0.72999999999999998},"min":{"1min":0.69199999999999995,"5min":0.499,"15min":0.499},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:51:03.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-09T22:51:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:03 smithi089 ceph-mon[36665]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-02-09T22:51:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2494716361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1461072674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:03 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/364335959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 -- 172.21.15.3:0/88780960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781013c0 msgr2=0x7f6478101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 --2- 172.21.15.3:0/88780960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781013c0 0x7f6478101840 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f646c009900 tx=0x7f646c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 -- 172.21.15.3:0/88780960 shutdown_connections 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 --2- 172.21.15.3:0/88780960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6478101d80 0x7f6478108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 --2- 172.21.15.3:0/88780960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781013c0 0x7f6478101840 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 --2- 172.21.15.3:0/88780960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781001d0 0x7f64781005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 -- 172.21.15.3:0/88780960 >> 172.21.15.3:0/88780960 conn(0x7f64780fb760 msgr2=0x7f64780fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 -- 172.21.15.3:0/88780960 shutdown_connections 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 -- 172.21.15.3:0/88780960 wait complete. 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 Processor -- start 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.794+0000 7f647fb77640 1 -- start start 2024-02-09T22:51:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 0x7f647806a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781013c0 0x7f647819c050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6478101d80 0x7f647819e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647819ee50 con 0x7f64781001d0 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647819efc0 con 0x7f6478101d80 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f647819f180 con 0x7f64781013c0 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 0x7f647806a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 0x7f647806a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40166/0 (socket says 172.21.15.3:40166) 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 -- 172.21.15.3:0/182829151 learned_addr learned my addr 172.21.15.3:0/182829151 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d112640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781013c0 0x7f647819c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 -- 172.21.15.3:0/182829151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781013c0 msgr2=0x7f647819c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781013c0 0x7f647819c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 -- 172.21.15.3:0/182829151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6478101d80 msgr2=0x7f647819e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6478101d80 0x7f647819e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 -- 172.21.15.3:0/182829151 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f646c009560 con 0x7f64781001d0 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d112640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781013c0 0x7f647819c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647d913640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 0x7f647806a480 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f647000c970 tx=0x7f647000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470004220 con 0x7f64781001d0 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6470007e90 con 0x7f64781001d0 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647819f400 con 0x7f64781001d0 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6470004a00 con 0x7f64781001d0 2024-02-09T22:51:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.795+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647819f8c0 con 0x7f64781001d0 2024-02-09T22:51:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.796+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64781074f0 con 0x7f64781001d0 2024-02-09T22:51:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.797+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f64700079b0 con 0x7f64781001d0 2024-02-09T22:51:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.798+0000 7f6476ffd640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64600a20c0 0x7f64600a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.798+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f64700cde60 con 0x7f64781001d0 2024-02-09T22:51:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.798+0000 7f647d112640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64600a20c0 0x7f64600a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.798+0000 7f647d112640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64600a20c0 0x7f64600a4580 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f646c036040 tx=0x7f646c000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.802+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6470096de0 con 0x7f64781001d0 2024-02-09T22:51:03.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:03 smithi064 ceph-mon[36638]: pgmap v346: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-02-09T22:51:03.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2494716361' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:03.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1461072674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:03.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:03 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/364335959' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:03.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.979+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f647802cfa0 con 0x7f64600a20c0 2024-02-09T22:51:03.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.990+0000 7f6476ffd640 1 -- 172.21.15.3:0/182829151 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+344232 (secure 0 0 0) 0x7f647802cfa0 con 0x7f64600a20c0 2024-02-09T22:51:03.997 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:03.999 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64600a20c0 msgr2=0x7f64600a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64600a20c0 0x7f64600a4580 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f646c036040 tx=0x7f646c000bc0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 msgr2=0x7f647806a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 0x7f647806a480 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f647000c970 tx=0x7f647000ce30 comp rx=0 tx=0).stop 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 shutdown_connections 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f64600a20c0 0x7f64600a4580 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6478101d80 0x7f647819e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f64781013c0 0x7f647819c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 --2- 172.21.15.3:0/182829151 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f64781001d0 0x7f647806a480 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.993+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 >> 172.21.15.3:0/182829151 conn(0x7f64780fb760 msgr2=0x7f64781046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.994+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 shutdown_connections 2024-02-09T22:51:04.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:03.994+0000 7f647fb77640 1 -- 172.21.15.3:0/182829151 wait complete. 2024-02-09T22:51:04.008 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-02-09T22:51:02.566447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657815,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1249,"num_read_kb":752,"num_write":429,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":853,"ondisk_log_size":853,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358444,"kb_used_data":13452,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583380,"statfs":{"total":1151940427776,"available":1151573381120,"internally_reserved":0,"allocated":13774848,"data_stored":7519761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":72,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002762"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582618+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:50:36.582618+0000","last_peered":"2024-02-09T22:50:36.582618+0000","last_clean":"2024-02-09T22:50:36.582618+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:50:36.582618+0000","last_undegraded":"2024-02-09T22:50:36.582618+0000","last_fullsized":"2024-02-09T22:50:36.582618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575819+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:50:36.575819+0000","last_peered":"2024-02-09T22:50:36.575819+0000","last_clean":"2024-02-09T22:50:36.575819+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:50:36.575819+0000","last_undegraded":"2024-02-09T22:50:36.575819+0000","last_fullsized":"2024-02-09T22:50:36.575819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582696+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:50:36.582696+0000","last_peered":"2024-02-09T22:50:36.582696+0000","last_clean":"2024-02-09T22:50:36.582696+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:50:36.582696+0000","last_undegraded":"2024-02-09T22:50:36.582696+0000","last_fullsized":"2024-02-09T22:50:36.582696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.564811+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:50:36.564811+0000","last_peered":"2024-02-09T22:50:36.564811+0000","last_clean":"2024-02-09T22:50:36.564811+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:50:36.564811+0000","last_undegraded":"2024-02-09T22:50:36.564811+0000","last_fullsized":"2024-02-09T22:50:36.564811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:50.286530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571412+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:50:36.571412+0000","last_peered":"2024-02-09T22:50:36.571412+0000","last_clean":"2024-02-09T22:50:36.571412+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:50:36.571412+0000","last_undegraded":"2024-02-09T22:50:36.571412+0000","last_fullsized":"2024-02-09T22:50:36.571412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577030+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:50:36.577030+0000","last_peered":"2024-02-09T22:50:36.577030+0000","last_clean":"2024-02-09T22:50:36.577030+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:50:36.577030+0000","last_undegraded":"2024-02-09T22:50:36.577030+0000","last_fullsized":"2024-02-09T22:50:36.577030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578903+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:50:36.578903+0000","last_peered":"2024-02-09T22:50:36.578903+0000","last_clean":"2024-02-09T22:50:36.578903+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:50:36.578903+0000","last_undegraded":"2024-02-09T22:50:36.578903+0000","last_fullsized":"2024-02-09T22:50:36.578903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.576625+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:50:36.576625+0000","last_peered":"2024-02-09T22:50:36.576625+0000","last_clean":"2024-02-09T22:50:36.576625+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:50:36.576625+0000","last_undegraded":"2024-02-09T22:50:36.576625+0000","last_fullsized":"2024-02-09T22:50:36.576625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584409+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:50:36.584409+0000","last_peered":"2024-02-09T22:50:36.584409+0000","last_clean":"2024-02-09T22:50:36.584409+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:50:36.584409+0000","last_undegraded":"2024-02-09T22:50:36.584409+0000","last_fullsized":"2024-02-09T22:50:36.584409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579365+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:50:36.579365+0000","last_peered":"2024-02-09T22:50:36.579365+0000","last_clean":"2024-02-09T22:50:36.579365+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:50:36.579365+0000","last_undegraded":"2024-02-09T22:50:36.579365+0000","last_fullsized":"2024-02-09T22:50:36.579365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.583991+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:50:36.583991+0000","last_peered":"2024-02-09T22:50:36.583991+0000","last_clean":"2024-02-09T22:50:36.583991+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:50:36.583991+0000","last_undegraded":"2024-02-09T22:50:36.583991+0000","last_fullsized":"2024-02-09T22:50:36.583991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572772+0000","last_change":"2024-02-09T22:50:08.517050+0000","last_active":"2024-02-09T22:50:36.572772+0000","last_peered":"2024-02-09T22:50:36.572772+0000","last_clean":"2024-02-09T22:50:36.572772+0000","last_became_active":"2024-02-09T22:50:08.516688+0000","last_became_peered":"2024-02-09T22:50:08.516688+0000","last_unstale":"2024-02-09T22:50:36.572772+0000","last_undegraded":"2024-02-09T22:50:36.572772+0000","last_fullsized":"2024-02-09T22:50:36.572772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:44:51.672735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578717+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:50:36.578717+0000","last_peered":"2024-02-09T22:50:36.578717+0000","last_clean":"2024-02-09T22:50:36.578717+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:50:36.578717+0000","last_undegraded":"2024-02-09T22:50:36.578717+0000","last_fullsized":"2024-02-09T22:50:36.578717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575908+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:50:36.575908+0000","last_peered":"2024-02-09T22:50:36.575908+0000","last_clean":"2024-02-09T22:50:36.575908+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:50:36.575908+0000","last_undegraded":"2024-02-09T22:50:36.575908+0000","last_fullsized":"2024-02-09T22:50:36.575908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568961+0000","last_change":"2024-02-09T22:50:08.616823+0000","last_active":"2024-02-09T22:50:36.568961+0000","last_peered":"2024-02-09T22:50:36.568961+0000","last_clean":"2024-02-09T22:50:36.568961+0000","last_became_active":"2024-02-09T22:50:08.616482+0000","last_became_peered":"2024-02-09T22:50:08.616482+0000","last_unstale":"2024-02-09T22:50:36.568961+0000","last_undegraded":"2024-02-09T22:50:36.568961+0000","last_fullsized":"2024-02-09T22:50:36.568961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:57:44.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582432+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:50:36.582432+0000","last_peered":"2024-02-09T22:50:36.582432+0000","last_clean":"2024-02-09T22:50:36.582432+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:50:36.582432+0000","last_undegraded":"2024-02-09T22:50:36.582432+0000","last_fullsized":"2024-02-09T22:50:36.582432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582696+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:50:36.582696+0000","last_peered":"2024-02-09T22:50:36.582696+0000","last_clean":"2024-02-09T22:50:36.582696+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:50:36.582696+0000","last_undegraded":"2024-02-09T22:50:36.582696+0000","last_fullsized":"2024-02-09T22:50:36.582696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573224+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:50:36.573224+0000","last_peered":"2024-02-09T22:50:36.573224+0000","last_clean":"2024-02-09T22:50:36.573224+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:50:36.573224+0000","last_undegraded":"2024-02-09T22:50:36.573224+0000","last_fullsized":"2024-02-09T22:50:36.573224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.570014+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:50:36.570014+0000","last_peered":"2024-02-09T22:50:36.570014+0000","last_clean":"2024-02-09T22:50:36.570014+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:50:36.570014+0000","last_undegraded":"2024-02-09T22:50:36.570014+0000","last_fullsized":"2024-02-09T22:50:36.570014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:39.090835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'14","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:59.266178+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:50:59.266178+0000","last_peered":"2024-02-09T22:50:59.266178+0000","last_clean":"2024-02-09T22:50:59.266178+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:50:59.266178+0000","last_undegraded":"2024-02-09T22:50:59.266178+0000","last_fullsized":"2024-02-09T22:50:59.266178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.569807+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:50:36.569807+0000","last_peered":"2024-02-09T22:50:36.569807+0000","last_clean":"2024-02-09T22:50:36.569807+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:50:36.569807+0000","last_undegraded":"2024-02-09T22:50:36.569807+0000","last_fullsized":"2024-02-09T22:50:36.569807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267136+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:50:08.267136+0000","last_peered":"2024-02-09T22:50:08.267136+0000","last_clean":"2024-02-09T22:50:08.267136+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:50:08.267136+0000","last_undegraded":"2024-02-09T22:50:08.267136+0000","last_fullsized":"2024-02-09T22:50:08.267136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582634+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:50:36.582634+0000","last_peered":"2024-02-09T22:50:36.582634+0000","last_clean":"2024-02-09T22:50:36.582634+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:50:36.582634+0000","last_undegraded":"2024-02-09T22:50:36.582634+0000","last_fullsized":"2024-02-09T22:50:36.582634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267805+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:50:08.267805+0000","last_peered":"2024-02-09T22:50:08.267805+0000","last_clean":"2024-02-09T22:50:08.267805+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:50:08.267805+0000","last_undegraded":"2024-02-09T22:50:08.267805+0000","last_fullsized":"2024-02-09T22:50:08.267805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584728+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:50:36.584728+0000","last_peered":"2024-02-09T22:50:36.584728+0000","last_clean":"2024-02-09T22:50:36.584728+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:50:36.584728+0000","last_undegraded":"2024-02-09T22:50:36.584728+0000","last_fullsized":"2024-02-09T22:50:36.584728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267118+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:50:08.267118+0000","last_peered":"2024-02-09T22:50:08.267118+0000","last_clean":"2024-02-09T22:50:08.267118+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:50:08.267118+0000","last_undegraded":"2024-02-09T22:50:08.267118+0000","last_fullsized":"2024-02-09T22:50:08.267118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.570889+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:50:36.570889+0000","last_peered":"2024-02-09T22:50:36.570889+0000","last_clean":"2024-02-09T22:50:36.570889+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:50:36.570889+0000","last_undegraded":"2024-02-09T22:50:36.570889+0000","last_fullsized":"2024-02-09T22:50:36.570889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.275449+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:51:00.275449+0000","last_peered":"2024-02-09T22:51:00.275449+0000","last_clean":"2024-02-09T22:51:00.275449+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:51:00.275449+0000","last_undegraded":"2024-02-09T22:51:00.275449+0000","last_fullsized":"2024-02-09T22:51:00.275449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567770+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:50:36.567770+0000","last_peered":"2024-02-09T22:50:36.567770+0000","last_clean":"2024-02-09T22:50:36.567770+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:50:36.567770+0000","last_undegraded":"2024-02-09T22:50:36.567770+0000","last_fullsized":"2024-02-09T22:50:36.567770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515862+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:50:08.515862+0000","last_peered":"2024-02-09T22:50:08.515862+0000","last_clean":"2024-02-09T22:50:08.515862+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:50:08.515862+0000","last_undegraded":"2024-02-09T22:50:08.515862+0000","last_fullsized":"2024-02-09T22:50:08.515862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571325+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:50:36.571325+0000","last_peered":"2024-02-09T22:50:36.571325+0000","last_clean":"2024-02-09T22:50:36.571325+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:50:36.571325+0000","last_undegraded":"2024-02-09T22:50:36.571325+0000","last_fullsized":"2024-02-09T22:50:36.571325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615818+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:50:08.615818+0000","last_peered":"2024-02-09T22:50:08.615818+0000","last_clean":"2024-02-09T22:50:08.615818+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:50:08.615818+0000","last_undegraded":"2024-02-09T22:50:08.615818+0000","last_fullsized":"2024-02-09T22:50:08.615818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574163+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:50:36.574163+0000","last_peered":"2024-02-09T22:50:36.574163+0000","last_clean":"2024-02-09T22:50:36.574163+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:50:36.574163+0000","last_undegraded":"2024-02-09T22:50:36.574163+0000","last_fullsized":"2024-02-09T22:50:36.574163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615165+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:50:08.615165+0000","last_peered":"2024-02-09T22:50:08.615165+0000","last_clean":"2024-02-09T22:50:08.615165+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:50:08.615165+0000","last_undegraded":"2024-02-09T22:50:08.615165+0000","last_fullsized":"2024-02-09T22:50:08.615165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572865+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:50:36.572865+0000","last_peered":"2024-02-09T22:50:36.572865+0000","last_clean":"2024-02-09T22:50:36.572865+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:50:36.572865+0000","last_undegraded":"2024-02-09T22:50:36.572865+0000","last_fullsized":"2024-02-09T22:50:36.572865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":240,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.274534+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:51:00.274534+0000","last_peered":"2024-02-09T22:51:00.274534+0000","last_clean":"2024-02-09T22:51:00.274534+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:51:00.274534+0000","last_undegraded":"2024-02-09T22:51:00.274534+0000","last_fullsized":"2024-02-09T22:51:00.274534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579648+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:50:36.579648+0000","last_peered":"2024-02-09T22:50:36.579648+0000","last_clean":"2024-02-09T22:50:36.579648+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:50:36.579648+0000","last_undegraded":"2024-02-09T22:50:36.579648+0000","last_fullsized":"2024-02-09T22:50:36.579648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264958+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:50:08.264958+0000","last_peered":"2024-02-09T22:50:08.264958+0000","last_clean":"2024-02-09T22:50:08.264958+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:50:08.264958+0000","last_undegraded":"2024-02-09T22:50:08.264958+0000","last_fullsized":"2024-02-09T22:50:08.264958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575891+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:50:36.575891+0000","last_peered":"2024-02-09T22:50:36.575891+0000","last_clean":"2024-02-09T22:50:36.575891+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:50:36.575891+0000","last_undegraded":"2024-02-09T22:50:36.575891+0000","last_fullsized":"2024-02-09T22:50:36.575891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615718+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:50:08.615718+0000","last_peered":"2024-02-09T22:50:08.615718+0000","last_clean":"2024-02-09T22:50:08.615718+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:50:08.615718+0000","last_undegraded":"2024-02-09T22:50:08.615718+0000","last_fullsized":"2024-02-09T22:50:08.615718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.585093+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:50:36.585093+0000","last_peered":"2024-02-09T22:50:36.585093+0000","last_clean":"2024-02-09T22:50:36.585093+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:50:36.585093+0000","last_undegraded":"2024-02-09T22:50:36.585093+0000","last_fullsized":"2024-02-09T22:50:36.585093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":277,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.275099+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:51:00.275099+0000","last_peered":"2024-02-09T22:51:00.275099+0000","last_clean":"2024-02-09T22:51:00.275099+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:51:00.275099+0000","last_undegraded":"2024-02-09T22:51:00.275099+0000","last_fullsized":"2024-02-09T22:51:00.275099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59951,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":171,"num_write":36,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577233+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:50:36.577233+0000","last_peered":"2024-02-09T22:50:36.577233+0000","last_clean":"2024-02-09T22:50:36.577233+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:50:36.577233+0000","last_undegraded":"2024-02-09T22:50:36.577233+0000","last_fullsized":"2024-02-09T22:50:36.577233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267676+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:50:08.267676+0000","last_peered":"2024-02-09T22:50:08.267676+0000","last_clean":"2024-02-09T22:50:08.267676+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:50:08.267676+0000","last_undegraded":"2024-02-09T22:50:08.267676+0000","last_fullsized":"2024-02-09T22:50:08.267676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574237+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:50:36.574237+0000","last_peered":"2024-02-09T22:50:36.574237+0000","last_clean":"2024-02-09T22:50:36.574237+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:50:36.574237+0000","last_undegraded":"2024-02-09T22:50:36.574237+0000","last_fullsized":"2024-02-09T22:50:36.574237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615556+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:50:08.615556+0000","last_peered":"2024-02-09T22:50:08.615556+0000","last_clean":"2024-02-09T22:50:08.615556+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:50:08.615556+0000","last_undegraded":"2024-02-09T22:50:08.615556+0000","last_fullsized":"2024-02-09T22:50:08.615556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573769+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:50:36.573769+0000","last_peered":"2024-02-09T22:50:36.573769+0000","last_clean":"2024-02-09T22:50:36.573769+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:50:36.573769+0000","last_undegraded":"2024-02-09T22:50:36.573769+0000","last_fullsized":"2024-02-09T22:50:36.573769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":131,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:56.292256+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:50:56.292256+0000","last_peered":"2024-02-09T22:50:56.292256+0000","last_clean":"2024-02-09T22:50:56.292256+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:50:56.292256+0000","last_undegraded":"2024-02-09T22:50:56.292256+0000","last_fullsized":"2024-02-09T22:50:56.292256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579110+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:50:36.579110+0000","last_peered":"2024-02-09T22:50:36.579110+0000","last_clean":"2024-02-09T22:50:36.579110+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:50:36.579110+0000","last_undegraded":"2024-02-09T22:50:36.579110+0000","last_fullsized":"2024-02-09T22:50:36.579110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265893+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:50:08.265893+0000","last_peered":"2024-02-09T22:50:08.265893+0000","last_clean":"2024-02-09T22:50:08.265893+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:50:08.265893+0000","last_undegraded":"2024-02-09T22:50:08.265893+0000","last_fullsized":"2024-02-09T22:50:08.265893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.562328+0000","last_change":"2024-02-09T22:50:08.539326+0000","last_active":"2024-02-09T22:50:36.562328+0000","last_peered":"2024-02-09T22:50:36.562328+0000","last_clean":"2024-02-09T22:50:36.562328+0000","last_became_active":"2024-02-09T22:50:08.539091+0000","last_became_peered":"2024-02-09T22:50:08.539091+0000","last_unstale":"2024-02-09T22:50:36.562328+0000","last_undegraded":"2024-02-09T22:50:36.562328+0000","last_fullsized":"2024-02-09T22:50:36.562328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:31:57.282733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267149+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:50:08.267149+0000","last_peered":"2024-02-09T22:50:08.267149+0000","last_clean":"2024-02-09T22:50:08.267149+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:50:08.267149+0000","last_undegraded":"2024-02-09T22:50:08.267149+0000","last_fullsized":"2024-02-09T22:50:08.267149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573178+0000","last_change":"2024-02-09T22:50:08.616790+0000","last_active":"2024-02-09T22:50:36.573178+0000","last_peered":"2024-02-09T22:50:36.573178+0000","last_clean":"2024-02-09T22:50:36.573178+0000","last_became_active":"2024-02-09T22:50:08.616404+0000","last_became_peered":"2024-02-09T22:50:08.616404+0000","last_unstale":"2024-02-09T22:50:36.573178+0000","last_undegraded":"2024-02-09T22:50:36.573178+0000","last_fullsized":"2024-02-09T22:50:36.573178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:14:18.902026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263341+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:50:08.263341+0000","last_peered":"2024-02-09T22:50:08.263341+0000","last_clean":"2024-02-09T22:50:08.263341+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:50:08.263341+0000","last_undegraded":"2024-02-09T22:50:08.263341+0000","last_fullsized":"2024-02-09T22:50:08.263341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575910+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:50:36.575910+0000","last_peered":"2024-02-09T22:50:36.575910+0000","last_clean":"2024-02-09T22:50:36.575910+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:50:36.575910+0000","last_undegraded":"2024-02-09T22:50:36.575910+0000","last_fullsized":"2024-02-09T22:50:36.575910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'6","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.268171+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:51:00.268171+0000","last_peered":"2024-02-09T22:51:00.268171+0000","last_clean":"2024-02-09T22:51:00.268171+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:51:00.268171+0000","last_undegraded":"2024-02-09T22:51:00.268171+0000","last_fullsized":"2024-02-09T22:51:00.268171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577911+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:50:36.577911+0000","last_peered":"2024-02-09T22:50:36.577911+0000","last_clean":"2024-02-09T22:50:36.577911+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:50:36.577911+0000","last_undegraded":"2024-02-09T22:50:36.577911+0000","last_fullsized":"2024-02-09T22:50:36.577911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572932+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:50:36.572932+0000","last_peered":"2024-02-09T22:50:36.572932+0000","last_clean":"2024-02-09T22:50:36.572932+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:50:36.572932+0000","last_undegraded":"2024-02-09T22:50:36.572932+0000","last_fullsized":"2024-02-09T22:50:36.572932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":133,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:59.296638+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:50:59.296638+0000","last_peered":"2024-02-09T22:50:59.296638+0000","last_clean":"2024-02-09T22:50:59.296638+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:50:59.296638+0000","last_undegraded":"2024-02-09T22:50:59.296638+0000","last_fullsized":"2024-02-09T22:50:59.296638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575148+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:50:36.575148+0000","last_peered":"2024-02-09T22:50:36.575148+0000","last_clean":"2024-02-09T22:50:36.575148+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:50:36.575148+0000","last_undegraded":"2024-02-09T22:50:36.575148+0000","last_fullsized":"2024-02-09T22:50:36.575148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538316+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:50:08.538316+0000","last_peered":"2024-02-09T22:50:08.538316+0000","last_clean":"2024-02-09T22:50:08.538316+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:50:08.538316+0000","last_undegraded":"2024-02-09T22:50:08.538316+0000","last_fullsized":"2024-02-09T22:50:08.538316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264739+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:50:08.264739+0000","last_peered":"2024-02-09T22:50:08.264739+0000","last_clean":"2024-02-09T22:50:08.264739+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:50:08.264739+0000","last_undegraded":"2024-02-09T22:50:08.264739+0000","last_fullsized":"2024-02-09T22:50:08.264739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.569342+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:50:36.569342+0000","last_peered":"2024-02-09T22:50:36.569342+0000","last_clean":"2024-02-09T22:50:36.569342+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:50:36.569342+0000","last_undegraded":"2024-02-09T22:50:36.569342+0000","last_fullsized":"2024-02-09T22:50:36.569342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264784+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:50:08.264784+0000","last_peered":"2024-02-09T22:50:08.264784+0000","last_clean":"2024-02-09T22:50:08.264784+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:50:08.264784+0000","last_undegraded":"2024-02-09T22:50:08.264784+0000","last_fullsized":"2024-02-09T22:50:08.264784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.278066+0000","last_change":"2024-02-09T22:50:08.278066+0000","last_active":"2024-02-09T22:50:08.278066+0000","last_peered":"2024-02-09T22:50:08.278066+0000","last_clean":"2024-02-09T22:50:08.278066+0000","last_became_active":"2024-02-09T22:50:08.271486+0000","last_became_peered":"2024-02-09T22:50:08.271486+0000","last_unstale":"2024-02-09T22:50:08.278066+0000","last_undegraded":"2024-02-09T22:50:08.278066+0000","last_fullsized":"2024-02-09T22:50:08.278066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:54:12.295421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567960+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:50:36.567960+0000","last_peered":"2024-02-09T22:50:36.567960+0000","last_clean":"2024-02-09T22:50:36.567960+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:50:36.567960+0000","last_undegraded":"2024-02-09T22:50:36.567960+0000","last_fullsized":"2024-02-09T22:50:36.567960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'13","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263515+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:50:08.263515+0000","last_peered":"2024-02-09T22:50:08.263515+0000","last_clean":"2024-02-09T22:50:08.263515+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:50:08.263515+0000","last_undegraded":"2024-02-09T22:50:08.263515+0000","last_fullsized":"2024-02-09T22:50:08.263515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574800+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:50:36.574800+0000","last_peered":"2024-02-09T22:50:36.574800+0000","last_clean":"2024-02-09T22:50:36.574800+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:50:36.574800+0000","last_undegraded":"2024-02-09T22:50:36.574800+0000","last_fullsized":"2024-02-09T22:50:36.574800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538049+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:50:08.538049+0000","last_peered":"2024-02-09T22:50:08.538049+0000","last_clean":"2024-02-09T22:50:08.538049+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:50:08.538049+0000","last_undegraded":"2024-02-09T22:50:08.538049+0000","last_fullsized":"2024-02-09T22:50:08.538049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578272+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:50:36.578272+0000","last_peered":"2024-02-09T22:50:36.578272+0000","last_clean":"2024-02-09T22:50:36.578272+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:50:36.578272+0000","last_undegraded":"2024-02-09T22:50:36.578272+0000","last_fullsized":"2024-02-09T22:50:36.578272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537439+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:50:08.537439+0000","last_peered":"2024-02-09T22:50:08.537439+0000","last_clean":"2024-02-09T22:50:08.537439+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:50:08.537439+0000","last_undegraded":"2024-02-09T22:50:08.537439+0000","last_fullsized":"2024-02-09T22:50:08.537439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.538911+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:50:36.538911+0000","last_peered":"2024-02-09T22:50:36.538911+0000","last_clean":"2024-02-09T22:50:36.538911+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:50:36.538911+0000","last_undegraded":"2024-02-09T22:50:36.538911+0000","last_fullsized":"2024-02-09T22:50:36.538911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266189+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:50:08.266189+0000","last_peered":"2024-02-09T22:50:08.266189+0000","last_clean":"2024-02-09T22:50:08.266189+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:50:08.266189+0000","last_undegraded":"2024-02-09T22:50:08.266189+0000","last_fullsized":"2024-02-09T22:50:08.266189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568139+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:50:36.568139+0000","last_peered":"2024-02-09T22:50:36.568139+0000","last_clean":"2024-02-09T22:50:36.568139+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:50:36.568139+0000","last_undegraded":"2024-02-09T22:50:36.568139+0000","last_fullsized":"2024-02-09T22:50:36.568139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.273440+0000","last_change":"2024-02-09T22:50:08.273440+0000","last_active":"2024-02-09T22:50:08.273440+0000","last_peered":"2024-02-09T22:50:08.273440+0000","last_clean":"2024-02-09T22:50:08.273440+0000","last_became_active":"2024-02-09T22:50:08.268960+0000","last_became_peered":"2024-02-09T22:50:08.268960+0000","last_unstale":"2024-02-09T22:50:08.273440+0000","last_undegraded":"2024-02-09T22:50:08.273440+0000","last_fullsized":"2024-02-09T22:50:08.273440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:41.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.576486+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:50:36.576486+0000","last_peered":"2024-02-09T22:50:36.576486+0000","last_clean":"2024-02-09T22:50:36.576486+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:50:36.576486+0000","last_undegraded":"2024-02-09T22:50:36.576486+0000","last_fullsized":"2024-02-09T22:50:36.576486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.542393+0000","last_change":"2024-02-09T22:50:08.542393+0000","last_active":"2024-02-09T22:50:08.542393+0000","last_peered":"2024-02-09T22:50:08.542393+0000","last_clean":"2024-02-09T22:50:08.542393+0000","last_became_active":"2024-02-09T22:50:08.539084+0000","last_became_peered":"2024-02-09T22:50:08.539084+0000","last_unstale":"2024-02-09T22:50:08.542393+0000","last_undegraded":"2024-02-09T22:50:08.542393+0000","last_fullsized":"2024-02-09T22:50:08.542393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:32:59.713756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584235+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:50:36.584235+0000","last_peered":"2024-02-09T22:50:36.584235+0000","last_clean":"2024-02-09T22:50:36.584235+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:50:36.584235+0000","last_undegraded":"2024-02-09T22:50:36.584235+0000","last_fullsized":"2024-02-09T22:50:36.584235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":131,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:56.269509+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:50:56.269509+0000","last_peered":"2024-02-09T22:50:56.269509+0000","last_clean":"2024-02-09T22:50:56.269509+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:50:56.269509+0000","last_undegraded":"2024-02-09T22:50:56.269509+0000","last_fullsized":"2024-02-09T22:50:56.269509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584413+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:50:36.584413+0000","last_peered":"2024-02-09T22:50:36.584413+0000","last_clean":"2024-02-09T22:50:36.584413+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:50:36.584413+0000","last_undegraded":"2024-02-09T22:50:36.584413+0000","last_fullsized":"2024-02-09T22:50:36.584413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615601+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:50:08.615601+0000","last_peered":"2024-02-09T22:50:08.615601+0000","last_clean":"2024-02-09T22:50:08.615601+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:50:08.615601+0000","last_undegraded":"2024-02-09T22:50:08.615601+0000","last_fullsized":"2024-02-09T22:50:08.615601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577804+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:50:36.577804+0000","last_peered":"2024-02-09T22:50:36.577804+0000","last_clean":"2024-02-09T22:50:36.577804+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:50:36.577804+0000","last_undegraded":"2024-02-09T22:50:36.577804+0000","last_fullsized":"2024-02-09T22:50:36.577804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615940+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:50:08.615940+0000","last_peered":"2024-02-09T22:50:08.615940+0000","last_clean":"2024-02-09T22:50:08.615940+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:50:08.615940+0000","last_undegraded":"2024-02-09T22:50:08.615940+0000","last_fullsized":"2024-02-09T22:50:08.615940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578172+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:50:36.578172+0000","last_peered":"2024-02-09T22:50:36.578172+0000","last_clean":"2024-02-09T22:50:36.578172+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:50:36.578172+0000","last_undegraded":"2024-02-09T22:50:36.578172+0000","last_fullsized":"2024-02-09T22:50:36.578172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573750+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:50:36.573750+0000","last_peered":"2024-02-09T22:50:36.573750+0000","last_clean":"2024-02-09T22:50:36.573750+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:50:36.573750+0000","last_undegraded":"2024-02-09T22:50:36.573750+0000","last_fullsized":"2024-02-09T22:50:36.573750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567974+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:50:36.567974+0000","last_peered":"2024-02-09T22:50:36.567974+0000","last_clean":"2024-02-09T22:50:36.567974+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:50:36.567974+0000","last_undegraded":"2024-02-09T22:50:36.567974+0000","last_fullsized":"2024-02-09T22:50:36.567974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:24:49.099644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.566404+0000","last_change":"2024-02-09T22:50:08.272126+0000","last_active":"2024-02-09T22:50:36.566404+0000","last_peered":"2024-02-09T22:50:36.566404+0000","last_clean":"2024-02-09T22:50:36.566404+0000","last_became_active":"2024-02-09T22:50:08.271722+0000","last_became_peered":"2024-02-09T22:50:08.271722+0000","last_unstale":"2024-02-09T22:50:36.566404+0000","last_undegraded":"2024-02-09T22:50:36.566404+0000","last_fullsized":"2024-02-09T22:50:36.566404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:43:31.304504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575883+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:50:36.575883+0000","last_peered":"2024-02-09T22:50:36.575883+0000","last_clean":"2024-02-09T22:50:36.575883+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:50:36.575883+0000","last_undegraded":"2024-02-09T22:50:36.575883+0000","last_fullsized":"2024-02-09T22:50:36.575883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579765+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:50:36.579765+0000","last_peered":"2024-02-09T22:50:36.579765+0000","last_clean":"2024-02-09T22:50:36.579765+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:50:36.579765+0000","last_undegraded":"2024-02-09T22:50:36.579765+0000","last_fullsized":"2024-02-09T22:50:36.579765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568845+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:50:36.568845+0000","last_peered":"2024-02-09T22:50:36.568845+0000","last_clean":"2024-02-09T22:50:36.568845+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:50:36.568845+0000","last_undegraded":"2024-02-09T22:50:36.568845+0000","last_fullsized":"2024-02-09T22:50:36.568845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.563750+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:50:36.563750+0000","last_peered":"2024-02-09T22:50:36.563750+0000","last_clean":"2024-02-09T22:50:36.563750+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:50:36.563750+0000","last_undegraded":"2024-02-09T22:50:36.563750+0000","last_fullsized":"2024-02-09T22:50:36.563750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:57:01.921327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571402+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:50:36.571402+0000","last_peered":"2024-02-09T22:50:36.571402+0000","last_clean":"2024-02-09T22:50:36.571402+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:50:36.571402+0000","last_undegraded":"2024-02-09T22:50:36.571402+0000","last_fullsized":"2024-02-09T22:50:36.571402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574138+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:50:36.574138+0000","last_peered":"2024-02-09T22:50:36.574138+0000","last_clean":"2024-02-09T22:50:36.574138+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:50:36.574138+0000","last_undegraded":"2024-02-09T22:50:36.574138+0000","last_fullsized":"2024-02-09T22:50:36.574138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574591+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:50:36.574591+0000","last_peered":"2024-02-09T22:50:36.574591+0000","last_clean":"2024-02-09T22:50:36.574591+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:50:36.574591+0000","last_undegraded":"2024-02-09T22:50:36.574591+0000","last_fullsized":"2024-02-09T22:50:36.574591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.585107+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:50:36.585107+0000","last_peered":"2024-02-09T22:50:36.585107+0000","last_clean":"2024-02-09T22:50:36.585107+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:50:36.585107+0000","last_undegraded":"2024-02-09T22:50:36.585107+0000","last_fullsized":"2024-02-09T22:50:36.585107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575799+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:50:36.575799+0000","last_peered":"2024-02-09T22:50:36.575799+0000","last_clean":"2024-02-09T22:50:36.575799+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:50:36.575799+0000","last_undegraded":"2024-02-09T22:50:36.575799+0000","last_fullsized":"2024-02-09T22:50:36.575799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571747+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:50:36.571747+0000","last_peered":"2024-02-09T22:50:36.571747+0000","last_clean":"2024-02-09T22:50:36.571747+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:50:36.571747+0000","last_undegraded":"2024-02-09T22:50:36.571747+0000","last_fullsized":"2024-02-09T22:50:36.571747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67213,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1113,"num_read_kb":633,"num_write":246,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":463574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63500000000000001,"15min":0.626},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.752}]},{"osd":2,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.44600000000000001},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.51500000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.88},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.81399999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.73299999999999998}]},{"osd":6,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.66600000000000004},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78600000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.747,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.82799999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92800000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.59699999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.96699999999999997}]}]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32800,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712352,"statfs":{"total":95995035648,"available":95961448448,"internally_reserved":0,"allocated":1662976,"data_stored":1114442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.437}]},{"osd":1,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.496,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.66900000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85399999999999998,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.877,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.72499999999999998,"15min":0.69299999999999995},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.36199999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.502}]},{"osd":5,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.747,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.441,"5min":0.433,"15min":0.433},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57299999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79500000000000004,"15min":0.79100000000000004},"min":{"1min":0.59799999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79900000000000004,"15min":0.80400000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0589999999999999,"5min":1.069,"15min":1.069},"last":0.70999999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.627,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0409999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.002,"5min":1.155,"15min":1.155},"last":0.65400000000000003}]},{"osd":11,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.83199999999999996}]}]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":463392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74199999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66000000000000003,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.63400000000000001},"min":{"1min":0.41699999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.89400000000000002,"5min":1.022,"15min":1.022},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.33500000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61499999999999999,"15min":0.61599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80100000000000005,"5min":2.246,"15min":2.246},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78500000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65800000000000003,"15min":0.67800000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85699999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70499999999999996,"15min":0.71899999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89700000000000002,"5min":1.01,"15min":1.01},"last":0.441}]},{"osd":6,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":0.69099999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76800000000000002,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":2.036,"15min":2.036},"last":0.47499999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.76400000000000001,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90300000000000002,"5min":1.177,"15min":1.177},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.77500000000000002,"15min":0.80100000000000005},"min":{"1min":0.496,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91400000000000003,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.49399999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.76800000000000002,"15min":0.78700000000000003},"min":{"1min":0.49099999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94899999999999995,"5min":1.173,"15min":1.173},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.80000000000000004,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.94199999999999995,"5min":1.387,"15min":1.387},"last":0.53600000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":1.089,"15min":1.089},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.76500000000000001,"15min":0.77600000000000002},"min":{"1min":0.56200000000000006,"5min":0.504,"15min":0.504},"max":{"1min":0.97499999999999998,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.627}]}]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":463658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77400000000000002}]},{"osd":1,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.60599999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.628},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89300000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.371,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.46100000000000002},"min":{"1min":0.35799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.67700000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97299999999999998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.55300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.63,"15min":0.621},"min":{"1min":0.46600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69899999999999995,"15min":0.69999999999999996},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.70599999999999996,"15min":0.68999999999999995},"min":{"1min":0.58099999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69899999999999995,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93700000000000006,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.71099999999999997,"15min":0.67500000000000004},"min":{"1min":0.63700000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0529999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.90800000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.75600000000000001,"15min":0.70299999999999996},"min":{"1min":0.67400000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1990000000000001,"5min":1.371,"15min":1.371},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.77000000000000002,"15min":0.71799999999999997},"min":{"1min":0.69899999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1870000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.88400000000000001}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.871},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.84999999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1056737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48099999999999998,"15min":0.46700000000000003},"min":{"1min":0.308,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.626,"5min":0.997,"15min":0.997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.505},"min":{"1min":0.35999999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":1.01,"15min":1.01},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65800000000000003,"15min":0.64900000000000002},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.874,"5min":1.111,"15min":1.111},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.41299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66600000000000004}]},{"osd":4,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.373,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.56699999999999995,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89900000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.50600000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89200000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.435,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92000000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.81200000000000006}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96599999999999997,"5min":1.016,"15min":1.016},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.73199999999999998,"15min":0.70399999999999996},"min":{"1min":0.60499999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.73699999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76500000000000001,"15min":0.77200000000000002},"min":{"1min":0.55100000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.038,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95399999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.71699999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.63,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.754},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.69699999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.028,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.80000000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83599999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":463417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55400000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.42199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55300000000000005,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.875,"15min":0.875},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.443,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625}]},{"osd":7,"last update":"Fri Feb 9 22:50:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78000000000000003,"5min":0.999,"15min":0.999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.55400000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57099999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84799999999999998,"5min":0.876,"15min":0.876},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.81200000000000006,"5min":0.995,"15min":0.995},"last":0.66200000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65200000000000002,"15min":0.68000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70599999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72099999999999997,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.82199999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.59099999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.81100000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.748}]}]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":523665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57699999999999996,"15min":0.52700000000000002},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.621},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.76300000000000001,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.80600000000000005,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.77700000000000002,"15min":0.60299999999999998},"min":{"1min":0.49399999999999999,"5min":0.373,"15min":0.37},"max":{"1min":0.92400000000000004,"5min":18.18,"15min":18.18},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.61899999999999999,"15min":0.52500000000000002},"min":{"1min":0.52100000000000002,"5min":0.38500000000000001,"15min":0.372},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.94099999999999995,"5min":1.206,"15min":1.206},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.35899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.498,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.79700000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.71899999999999997,"15min":0.623},"min":{"1min":0.59299999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.72399999999999998,"15min":0.63600000000000001},"min":{"1min":0.56899999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66600000000000004,"15min":0.64500000000000002},"min":{"1min":0.48999999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.66100000000000003,"15min":0.629},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.70599999999999996,"15min":0.67200000000000004},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0669999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.86299999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.353,"15min":1.581},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1439999999999999,"5min":18.132000000000001,"15min":18.132000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69399999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.57199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.86899999999999999,"15min":0.85899999999999999},"min":{"1min":0.72899999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.101,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.98399999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78800000000000003,"15min":0.78200000000000003},"min":{"1min":0.60999999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.871,"15min":0.85399999999999998},"min":{"1min":0.67000000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.92200000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.96199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":464448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56899999999999995,"15min":0.495},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.57599999999999996,"15min":0.498},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56699999999999995,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":4,"last update":"Fri Feb 9 22:50:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.745},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Feb 9 22:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.622},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.34599999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76200000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84999999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64200000000000002}]},{"osd":9,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.72099999999999997},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.80200000000000005,"5min":1.081,"15min":1.081},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.749},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95899999999999996,"5min":1.202,"15min":1.202},"last":0.79000000000000004}]},{"osd":10,"last update":"Fri Feb 9 22:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67600000000000005,"15min":0.70399999999999996},"min":{"1min":0.51100000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79000000000000004,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72099999999999997,"15min":0.73899999999999999},"min":{"1min":0.60399999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84799999999999998,"5min":1.137,"15min":1.137},"last":0.79900000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.65400000000000003},"min":{"1min":0.59699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.89900000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.875}]}]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":524417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.45800000000000002},"min":{"1min":0.28299999999999997,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.71199999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.48299999999999998},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.73899999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56899999999999995,"15min":0.49199999999999999},"min":{"1min":0.438,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.60599999999999998,"15min":0.504},"min":{"1min":0.45700000000000002,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55900000000000005,"15min":0.49099999999999999},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.48999999999999999},"min":{"1min":0.39900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.437}]},{"osd":4,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.65800000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.499,"15min":0.435},"min":{"1min":0.32700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71299999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.874,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58899999999999997}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90700000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.51900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.48699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.624}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]},{"osd":11,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.872},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67700000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":462934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57899999999999996,"5min":0.62,"15min":0.62},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.53100000000000003},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78100000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.53500000000000003},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.753,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.746}]},{"osd":2,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.434},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.372,"15min":0.372},"max":{"1min":0.83599999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.73299999999999998,"15min":0.627},"min":{"1min":0.378,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.6920000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86299999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59399999999999997,"15min":0.60999999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76400000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.747,"5min":0.873,"15min":0.873},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.82799999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63800000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.745,"15min":0.70699999999999996},"min":{"1min":0.52700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.877,"5min":5.8789999999999996,"15min":5.8789999999999996},"last":0.69599999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:50:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.69699999999999995,"15min":0.75900000000000001},"min":{"1min":0.378,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.442,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88400000000000001,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.55900000000000005}]},{"osd":10,"last update":"Fri Feb 9 22:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79500000000000004,"15min":0.82999999999999996},"min":{"1min":0.58299999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92500000000000004,"5min":1.26,"15min":1.26},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.74399999999999999},"min":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.94199999999999995,"5min":0.999,"15min":0.999},"last":0.65300000000000002}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.624},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.72899999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":462970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.433,"15min":0.40300000000000002},"min":{"1min":0.34200000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.63100000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46000000000000002,"15min":0.40300000000000002},"min":{"1min":0.38400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.443}]},{"osd":2,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.61799999999999999,"15min":0.57299999999999995},"min":{"1min":0.45200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.01,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.874},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.497,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.871,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.626,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.57999999999999996,"15min":0.56699999999999995},"min":{"1min":0.47599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80400000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.63800000000000001,"15min":0.58599999999999997},"min":{"1min":0.44800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.60399999999999998},"min":{"1min":0.55800000000000005,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78200000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.60199999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.056,"15min":1.056},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93400000000000005,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Feb 9 22:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88600000000000001,"5min":1.097,"15min":1.097},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.496,"5min":0.434,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.55100000000000005}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93300000000000005,"5min":1.129,"15min":1.129},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76900000000000002,"15min":0.748},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.264,"15min":1.264},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.50700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.77400000000000002,"15min":0.78000000000000003},"min":{"1min":0.60399999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.98599999999999999,"5min":1.196,"15min":1.196},"last":0.82999999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.61899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98499999999999999,"5min":1.179,"15min":1.179},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.624,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.99299999999999999,"5min":1.238,"15min":1.238},"last":0.64800000000000002}]},{"osd":11,"last update":"Fri Feb 9 22:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.95299999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1593344,"data_stored":1056107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.498},"min":{"1min":0.438,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66200000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.48099999999999998},"min":{"1min":0.432,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55000000000000004}]},{"osd":1,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.56699999999999995,"15min":0.51600000000000001},"min":{"1min":0.45200000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.749},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.57199999999999995,"15min":0.52500000000000002},"min":{"1min":0.47699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.43099999999999999,"15min":0.40300000000000002},"min":{"1min":0.38200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.374,"5min":0.217,"15min":0.217},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54500000000000004,"15min":0.51300000000000001},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.496},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57199999999999995,"15min":0.53400000000000003},"min":{"1min":0.45200000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.61499999999999999,"15min":0.55700000000000005},"min":{"1min":0.58799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59799999999999998,"15min":0.55000000000000004},"min":{"1min":0.54200000000000004,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60899999999999999,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85299999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.625},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60599999999999998,"15min":0.58299999999999996},"min":{"1min":0.47499999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.81299999999999994,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.64000000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.67200000000000004,"15min":0.65300000000000002},"min":{"1min":0.65100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.67200000000000004,"15min":0.65900000000000003},"min":{"1min":0.63900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90700000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.70499999999999996}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.872,"5min":0.996,"15min":0.996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.63100000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90800000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.58599999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94699999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.75700000000000001,"15min":0.74099999999999999},"min":{"1min":0.73699999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0049999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.751,"15min":0.72999999999999998},"min":{"1min":0.69199999999999995,"5min":0.499,"15min":0.499},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:51:04.015 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-02-09T22:51:04.015 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-02-09T22:51:04.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 -- 172.21.15.3:0/3349992362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102450 msgr2=0x7fe4381028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 --2- 172.21.15.3:0/3349992362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102450 0x7fe4381028d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7fe42c009900 tx=0x7fe42c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:04.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 -- 172.21.15.3:0/3349992362 shutdown_connections 2024-02-09T22:51:04.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 --2- 172.21.15.3:0/3349992362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438102e10 0x7fe4381096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 --2- 172.21.15.3:0/3349992362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102450 0x7fe4381028d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 --2- 172.21.15.3:0/3349992362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe438101260 0x7fe438101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 -- 172.21.15.3:0/3349992362 >> 172.21.15.3:0/3349992362 conn(0x7fe4380fc810 msgr2=0x7fe4380fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 -- 172.21.15.3:0/3349992362 shutdown_connections 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.148+0000 7fe43f756640 1 -- 172.21.15.3:0/3349992362 wait complete. 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 Processor -- start 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 -- start start 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438101260 0x7fe438074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe438102450 0x7fe438073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 0x7fe438073b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4380742c0 con 0x7fe438102e10 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438074430 con 0x7fe438102450 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43f756640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4381a13d0 con 0x7fe438101260 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 0x7fe438073b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 0x7fe438073b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40196/0 (socket says 172.21.15.3:40196) 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 -- 172.21.15.3:0/1660207624 learned_addr learned my addr 172.21.15.3:0/1660207624 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:04.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43d4f2640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438101260 0x7fe438074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 -- 172.21.15.3:0/1660207624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438101260 msgr2=0x7fe438074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438101260 0x7fe438074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 -- 172.21.15.3:0/1660207624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe438102450 msgr2=0x7fe438073620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe438102450 0x7fe438073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43dcf3640 1 -- 172.21.15.3:0/1660207624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42c009560 con 0x7fe438102e10 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.149+0000 7fe43d4f2640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438101260 0x7fe438074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.150+0000 7fe43dcf3640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 0x7fe438073b60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe42800ead0 tx=0x7fe42800ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.150+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42800cde0 con 0x7fe438102e10 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.150+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe428005020 con 0x7fe438102e10 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.150+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4381a1680 con 0x7fe438102e10 2024-02-09T22:51:04.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.150+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4280116b0 con 0x7fe438102e10 2024-02-09T22:51:04.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.150+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4381a1c60 con 0x7fe438102e10 2024-02-09T22:51:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.151+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe438108580 con 0x7fe438102e10 2024-02-09T22:51:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.152+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fe428005520 con 0x7fe438102e10 2024-02-09T22:51:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.152+0000 7fe4367fc640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe4200a2140 0x7fe4200a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.152+0000 7fe43d4f2640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe4200a2140 0x7fe4200a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.152+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fe4280cdea0 con 0x7fe438102e10 2024-02-09T22:51:04.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.153+0000 7fe43d4f2640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe4200a2140 0x7fe4200a4600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe430006fd0 tx=0x7fe430008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:04.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.157+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe428096e80 con 0x7fe438102e10 2024-02-09T22:51:04.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.335+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe43802cf20 con 0x7fe4200a2140 2024-02-09T22:51:04.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.345+0000 7fe4367fc640 1 -- 172.21.15.3:0/1660207624 <== mgr.14148 v2:172.21.15.3:6800/284621866 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+344232 (secure 0 0 0) 0x7fe43802cf20 con 0x7fe4200a2140 2024-02-09T22:51:04.352 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:04.354 INFO:teuthology.orchestra.run.smithi003.stderr:dumped all 2024-02-09T22:51:04.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe4200a2140 msgr2=0x7fe4200a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe4200a2140 0x7fe4200a4600 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe430006fd0 tx=0x7fe430008040 comp rx=0 tx=0).stop 2024-02-09T22:51:04.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 msgr2=0x7fe438073b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 0x7fe438073b60 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fe42800ead0 tx=0x7fe42800ef90 comp rx=0 tx=0).stop 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 shutdown_connections 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fe4200a2140 0x7fe4200a4600 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe438102450 0x7fe438073620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe438102e10 0x7fe438073b60 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 --2- 172.21.15.3:0/1660207624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe438101260 0x7fe438074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 >> 172.21.15.3:0/1660207624 conn(0x7fe4380fc810 msgr2=0x7fe438105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 shutdown_connections 2024-02-09T22:51:04.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.349+0000 7fe43f756640 1 -- 172.21.15.3:0/1660207624 wait complete. 2024-02-09T22:51:04.363 INFO:teuthology.orchestra.run.smithi003.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-02-09T22:51:02.566447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":657815,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1249,"num_read_kb":752,"num_write":429,"num_write_kb":3079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1180848,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":853,"ondisk_log_size":853,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":358444,"kb_used_data":13452,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124583380,"statfs":{"total":1151940427776,"available":1151573381120,"internally_reserved":0,"allocated":13774848,"data_stored":7519761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":353154361},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":132,"num_read_kb":72,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002762"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582618+0000","last_change":"2024-02-09T22:49:07.264487+0000","last_active":"2024-02-09T22:50:36.582618+0000","last_peered":"2024-02-09T22:50:36.582618+0000","last_clean":"2024-02-09T22:50:36.582618+0000","last_became_active":"2024-02-09T22:49:07.264176+0000","last_became_peered":"2024-02-09T22:49:07.264176+0000","last_unstale":"2024-02-09T22:50:36.582618+0000","last_undegraded":"2024-02-09T22:50:36.582618+0000","last_fullsized":"2024-02-09T22:50:36.582618+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:18:02.292539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575819+0000","last_change":"2024-02-09T22:49:07.262645+0000","last_active":"2024-02-09T22:50:36.575819+0000","last_peered":"2024-02-09T22:50:36.575819+0000","last_clean":"2024-02-09T22:50:36.575819+0000","last_became_active":"2024-02-09T22:49:07.262302+0000","last_became_peered":"2024-02-09T22:49:07.262302+0000","last_unstale":"2024-02-09T22:50:36.575819+0000","last_undegraded":"2024-02-09T22:50:36.575819+0000","last_fullsized":"2024-02-09T22:50:36.575819+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:12:47.262031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582696+0000","last_change":"2024-02-09T22:49:07.263675+0000","last_active":"2024-02-09T22:50:36.582696+0000","last_peered":"2024-02-09T22:50:36.582696+0000","last_clean":"2024-02-09T22:50:36.582696+0000","last_became_active":"2024-02-09T22:49:07.263241+0000","last_became_peered":"2024-02-09T22:49:07.263241+0000","last_unstale":"2024-02-09T22:50:36.582696+0000","last_undegraded":"2024-02-09T22:50:36.582696+0000","last_fullsized":"2024-02-09T22:50:36.582696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:31.366165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.564811+0000","last_change":"2024-02-09T22:49:07.269526+0000","last_active":"2024-02-09T22:50:36.564811+0000","last_peered":"2024-02-09T22:50:36.564811+0000","last_clean":"2024-02-09T22:50:36.564811+0000","last_became_active":"2024-02-09T22:49:07.268682+0000","last_became_peered":"2024-02-09T22:49:07.268682+0000","last_unstale":"2024-02-09T22:50:36.564811+0000","last_undegraded":"2024-02-09T22:50:36.564811+0000","last_fullsized":"2024-02-09T22:50:36.564811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:38:50.286530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571412+0000","last_change":"2024-02-09T22:49:07.260437+0000","last_active":"2024-02-09T22:50:36.571412+0000","last_peered":"2024-02-09T22:50:36.571412+0000","last_clean":"2024-02-09T22:50:36.571412+0000","last_became_active":"2024-02-09T22:49:07.259999+0000","last_became_peered":"2024-02-09T22:49:07.259999+0000","last_unstale":"2024-02-09T22:50:36.571412+0000","last_undegraded":"2024-02-09T22:50:36.571412+0000","last_fullsized":"2024-02-09T22:50:36.571412+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:00:05.613146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577030+0000","last_change":"2024-02-09T22:49:07.255059+0000","last_active":"2024-02-09T22:50:36.577030+0000","last_peered":"2024-02-09T22:50:36.577030+0000","last_clean":"2024-02-09T22:50:36.577030+0000","last_became_active":"2024-02-09T22:49:07.253777+0000","last_became_peered":"2024-02-09T22:49:07.253777+0000","last_unstale":"2024-02-09T22:50:36.577030+0000","last_undegraded":"2024-02-09T22:50:36.577030+0000","last_fullsized":"2024-02-09T22:50:36.577030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:42:01.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578903+0000","last_change":"2024-02-09T22:49:07.264758+0000","last_active":"2024-02-09T22:50:36.578903+0000","last_peered":"2024-02-09T22:50:36.578903+0000","last_clean":"2024-02-09T22:50:36.578903+0000","last_became_active":"2024-02-09T22:49:07.264215+0000","last_became_peered":"2024-02-09T22:49:07.264215+0000","last_unstale":"2024-02-09T22:50:36.578903+0000","last_undegraded":"2024-02-09T22:50:36.578903+0000","last_fullsized":"2024-02-09T22:50:36.578903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:40:41.132982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.576625+0000","last_change":"2024-02-09T22:49:07.262510+0000","last_active":"2024-02-09T22:50:36.576625+0000","last_peered":"2024-02-09T22:50:36.576625+0000","last_clean":"2024-02-09T22:50:36.576625+0000","last_became_active":"2024-02-09T22:49:07.262059+0000","last_became_peered":"2024-02-09T22:49:07.262059+0000","last_unstale":"2024-02-09T22:50:36.576625+0000","last_undegraded":"2024-02-09T22:50:36.576625+0000","last_fullsized":"2024-02-09T22:50:36.576625+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:46:41.874244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584409+0000","last_change":"2024-02-09T22:49:07.266049+0000","last_active":"2024-02-09T22:50:36.584409+0000","last_peered":"2024-02-09T22:50:36.584409+0000","last_clean":"2024-02-09T22:50:36.584409+0000","last_became_active":"2024-02-09T22:49:07.265350+0000","last_became_peered":"2024-02-09T22:49:07.265350+0000","last_unstale":"2024-02-09T22:50:36.584409+0000","last_undegraded":"2024-02-09T22:50:36.584409+0000","last_fullsized":"2024-02-09T22:50:36.584409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:29:57.453934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579365+0000","last_change":"2024-02-09T22:49:07.262148+0000","last_active":"2024-02-09T22:50:36.579365+0000","last_peered":"2024-02-09T22:50:36.579365+0000","last_clean":"2024-02-09T22:50:36.579365+0000","last_became_active":"2024-02-09T22:49:07.261760+0000","last_became_peered":"2024-02-09T22:49:07.261760+0000","last_unstale":"2024-02-09T22:50:36.579365+0000","last_undegraded":"2024-02-09T22:50:36.579365+0000","last_fullsized":"2024-02-09T22:50:36.579365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:53.647019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.583991+0000","last_change":"2024-02-09T22:49:07.270882+0000","last_active":"2024-02-09T22:50:36.583991+0000","last_peered":"2024-02-09T22:50:36.583991+0000","last_clean":"2024-02-09T22:50:36.583991+0000","last_became_active":"2024-02-09T22:49:07.270538+0000","last_became_peered":"2024-02-09T22:49:07.270538+0000","last_unstale":"2024-02-09T22:50:36.583991+0000","last_undegraded":"2024-02-09T22:50:36.583991+0000","last_fullsized":"2024-02-09T22:50:36.583991+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:27:07.104221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572772+0000","last_change":"2024-02-09T22:50:08.517050+0000","last_active":"2024-02-09T22:50:36.572772+0000","last_peered":"2024-02-09T22:50:36.572772+0000","last_clean":"2024-02-09T22:50:36.572772+0000","last_became_active":"2024-02-09T22:50:08.516688+0000","last_became_peered":"2024-02-09T22:50:08.516688+0000","last_unstale":"2024-02-09T22:50:36.572772+0000","last_undegraded":"2024-02-09T22:50:36.572772+0000","last_fullsized":"2024-02-09T22:50:36.572772+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:44:51.672735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578717+0000","last_change":"2024-02-09T22:49:07.253453+0000","last_active":"2024-02-09T22:50:36.578717+0000","last_peered":"2024-02-09T22:50:36.578717+0000","last_clean":"2024-02-09T22:50:36.578717+0000","last_became_active":"2024-02-09T22:49:07.253088+0000","last_became_peered":"2024-02-09T22:49:07.253088+0000","last_unstale":"2024-02-09T22:50:36.578717+0000","last_undegraded":"2024-02-09T22:50:36.578717+0000","last_fullsized":"2024-02-09T22:50:36.578717+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:31:06.866064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575908+0000","last_change":"2024-02-09T22:49:07.264271+0000","last_active":"2024-02-09T22:50:36.575908+0000","last_peered":"2024-02-09T22:50:36.575908+0000","last_clean":"2024-02-09T22:50:36.575908+0000","last_became_active":"2024-02-09T22:49:07.263948+0000","last_became_peered":"2024-02-09T22:49:07.263948+0000","last_unstale":"2024-02-09T22:50:36.575908+0000","last_undegraded":"2024-02-09T22:50:36.575908+0000","last_fullsized":"2024-02-09T22:50:36.575908+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:30:50.987490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568961+0000","last_change":"2024-02-09T22:50:08.616823+0000","last_active":"2024-02-09T22:50:36.568961+0000","last_peered":"2024-02-09T22:50:36.568961+0000","last_clean":"2024-02-09T22:50:36.568961+0000","last_became_active":"2024-02-09T22:50:08.616482+0000","last_became_peered":"2024-02-09T22:50:08.616482+0000","last_unstale":"2024-02-09T22:50:36.568961+0000","last_undegraded":"2024-02-09T22:50:36.568961+0000","last_fullsized":"2024-02-09T22:50:36.568961+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:57:44.073265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582432+0000","last_change":"2024-02-09T22:49:07.269138+0000","last_active":"2024-02-09T22:50:36.582432+0000","last_peered":"2024-02-09T22:50:36.582432+0000","last_clean":"2024-02-09T22:50:36.582432+0000","last_became_active":"2024-02-09T22:49:07.268110+0000","last_became_peered":"2024-02-09T22:49:07.268110+0000","last_unstale":"2024-02-09T22:50:36.582432+0000","last_undegraded":"2024-02-09T22:50:36.582432+0000","last_fullsized":"2024-02-09T22:50:36.582432+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:50:27.942815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582696+0000","last_change":"2024-02-09T22:49:07.264383+0000","last_active":"2024-02-09T22:50:36.582696+0000","last_peered":"2024-02-09T22:50:36.582696+0000","last_clean":"2024-02-09T22:50:36.582696+0000","last_became_active":"2024-02-09T22:49:07.264032+0000","last_became_peered":"2024-02-09T22:49:07.264032+0000","last_unstale":"2024-02-09T22:50:36.582696+0000","last_undegraded":"2024-02-09T22:50:36.582696+0000","last_fullsized":"2024-02-09T22:50:36.582696+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:22:50.967486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573224+0000","last_change":"2024-02-09T22:49:07.258841+0000","last_active":"2024-02-09T22:50:36.573224+0000","last_peered":"2024-02-09T22:50:36.573224+0000","last_clean":"2024-02-09T22:50:36.573224+0000","last_became_active":"2024-02-09T22:49:07.258516+0000","last_became_peered":"2024-02-09T22:49:07.258516+0000","last_unstale":"2024-02-09T22:50:36.573224+0000","last_undegraded":"2024-02-09T22:50:36.573224+0000","last_fullsized":"2024-02-09T22:50:36.573224+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:34:11.655429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.570014+0000","last_change":"2024-02-09T22:49:07.260205+0000","last_active":"2024-02-09T22:50:36.570014+0000","last_peered":"2024-02-09T22:50:36.570014+0000","last_clean":"2024-02-09T22:50:36.570014+0000","last_became_active":"2024-02-09T22:49:07.259903+0000","last_became_peered":"2024-02-09T22:49:07.259903+0000","last_unstale":"2024-02-09T22:50:36.570014+0000","last_undegraded":"2024-02-09T22:50:36.570014+0000","last_fullsized":"2024-02-09T22:50:36.570014+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:39.090835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"88'14","reported_seq":141,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:59.266178+0000","last_change":"2024-02-09T22:49:06.258516+0000","last_active":"2024-02-09T22:50:59.266178+0000","last_peered":"2024-02-09T22:50:59.266178+0000","last_clean":"2024-02-09T22:50:59.266178+0000","last_became_active":"2024-02-09T22:49:06.255959+0000","last_became_peered":"2024-02-09T22:49:06.255959+0000","last_unstale":"2024-02-09T22:50:59.266178+0000","last_undegraded":"2024-02-09T22:50:59.266178+0000","last_fullsized":"2024-02-09T22:50:59.266178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:09:46.488373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":108,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.569807+0000","last_change":"2024-02-09T22:49:07.262894+0000","last_active":"2024-02-09T22:50:36.569807+0000","last_peered":"2024-02-09T22:50:36.569807+0000","last_clean":"2024-02-09T22:50:36.569807+0000","last_became_active":"2024-02-09T22:49:07.262565+0000","last_became_peered":"2024-02-09T22:49:07.262565+0000","last_unstale":"2024-02-09T22:50:36.569807+0000","last_undegraded":"2024-02-09T22:50:36.569807+0000","last_fullsized":"2024-02-09T22:50:36.569807+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:08:18.491313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267136+0000","last_change":"2024-02-09T22:49:06.254685+0000","last_active":"2024-02-09T22:50:08.267136+0000","last_peered":"2024-02-09T22:50:08.267136+0000","last_clean":"2024-02-09T22:50:08.267136+0000","last_became_active":"2024-02-09T22:49:06.254384+0000","last_became_peered":"2024-02-09T22:49:06.254384+0000","last_unstale":"2024-02-09T22:50:08.267136+0000","last_undegraded":"2024-02-09T22:50:08.267136+0000","last_fullsized":"2024-02-09T22:50:08.267136+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:36:03.673301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.582634+0000","last_change":"2024-02-09T22:49:07.267664+0000","last_active":"2024-02-09T22:50:36.582634+0000","last_peered":"2024-02-09T22:50:36.582634+0000","last_clean":"2024-02-09T22:50:36.582634+0000","last_became_active":"2024-02-09T22:49:07.263892+0000","last_became_peered":"2024-02-09T22:49:07.263892+0000","last_unstale":"2024-02-09T22:50:36.582634+0000","last_undegraded":"2024-02-09T22:50:36.582634+0000","last_fullsized":"2024-02-09T22:50:36.582634+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:17:45.048249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267805+0000","last_change":"2024-02-09T22:49:06.255684+0000","last_active":"2024-02-09T22:50:08.267805+0000","last_peered":"2024-02-09T22:50:08.267805+0000","last_clean":"2024-02-09T22:50:08.267805+0000","last_became_active":"2024-02-09T22:49:06.255363+0000","last_became_peered":"2024-02-09T22:49:06.255363+0000","last_unstale":"2024-02-09T22:50:08.267805+0000","last_undegraded":"2024-02-09T22:50:08.267805+0000","last_fullsized":"2024-02-09T22:50:08.267805+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:49.493015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584728+0000","last_change":"2024-02-09T22:49:07.265894+0000","last_active":"2024-02-09T22:50:36.584728+0000","last_peered":"2024-02-09T22:50:36.584728+0000","last_clean":"2024-02-09T22:50:36.584728+0000","last_became_active":"2024-02-09T22:49:07.265529+0000","last_became_peered":"2024-02-09T22:49:07.265529+0000","last_unstale":"2024-02-09T22:50:36.584728+0000","last_undegraded":"2024-02-09T22:50:36.584728+0000","last_fullsized":"2024-02-09T22:50:36.584728+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:33:49.739781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267118+0000","last_change":"2024-02-09T22:49:06.254054+0000","last_active":"2024-02-09T22:50:08.267118+0000","last_peered":"2024-02-09T22:50:08.267118+0000","last_clean":"2024-02-09T22:50:08.267118+0000","last_became_active":"2024-02-09T22:49:06.253744+0000","last_became_peered":"2024-02-09T22:49:06.253744+0000","last_unstale":"2024-02-09T22:50:08.267118+0000","last_undegraded":"2024-02-09T22:50:08.267118+0000","last_fullsized":"2024-02-09T22:50:08.267118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:32:54.537817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.570889+0000","last_change":"2024-02-09T22:49:07.266180+0000","last_active":"2024-02-09T22:50:36.570889+0000","last_peered":"2024-02-09T22:50:36.570889+0000","last_clean":"2024-02-09T22:50:36.570889+0000","last_became_active":"2024-02-09T22:49:07.265777+0000","last_became_peered":"2024-02-09T22:49:07.265777+0000","last_unstale":"2024-02-09T22:50:36.570889+0000","last_undegraded":"2024-02-09T22:50:36.570889+0000","last_fullsized":"2024-02-09T22:50:36.570889+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:21:29.117368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":138,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.275449+0000","last_change":"2024-02-09T22:49:06.253060+0000","last_active":"2024-02-09T22:51:00.275449+0000","last_peered":"2024-02-09T22:51:00.275449+0000","last_clean":"2024-02-09T22:51:00.275449+0000","last_became_active":"2024-02-09T22:49:06.252244+0000","last_became_peered":"2024-02-09T22:49:06.252244+0000","last_unstale":"2024-02-09T22:51:00.275449+0000","last_undegraded":"2024-02-09T22:51:00.275449+0000","last_fullsized":"2024-02-09T22:51:00.275449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:46:48.642250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567770+0000","last_change":"2024-02-09T22:49:07.262562+0000","last_active":"2024-02-09T22:50:36.567770+0000","last_peered":"2024-02-09T22:50:36.567770+0000","last_clean":"2024-02-09T22:50:36.567770+0000","last_became_active":"2024-02-09T22:49:07.262196+0000","last_became_peered":"2024-02-09T22:49:07.262196+0000","last_unstale":"2024-02-09T22:50:36.567770+0000","last_undegraded":"2024-02-09T22:50:36.567770+0000","last_fullsized":"2024-02-09T22:50:36.567770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:09:46.915442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.515862+0000","last_change":"2024-02-09T22:49:06.252852+0000","last_active":"2024-02-09T22:50:08.515862+0000","last_peered":"2024-02-09T22:50:08.515862+0000","last_clean":"2024-02-09T22:50:08.515862+0000","last_became_active":"2024-02-09T22:49:06.252542+0000","last_became_peered":"2024-02-09T22:49:06.252542+0000","last_unstale":"2024-02-09T22:50:08.515862+0000","last_undegraded":"2024-02-09T22:50:08.515862+0000","last_fullsized":"2024-02-09T22:50:08.515862+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:57:40.389056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571325+0000","last_change":"2024-02-09T22:49:07.259840+0000","last_active":"2024-02-09T22:50:36.571325+0000","last_peered":"2024-02-09T22:50:36.571325+0000","last_clean":"2024-02-09T22:50:36.571325+0000","last_became_active":"2024-02-09T22:49:07.258941+0000","last_became_peered":"2024-02-09T22:49:07.258941+0000","last_unstale":"2024-02-09T22:50:36.571325+0000","last_undegraded":"2024-02-09T22:50:36.571325+0000","last_fullsized":"2024-02-09T22:50:36.571325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:07:43.174773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615818+0000","last_change":"2024-02-09T22:49:06.246728+0000","last_active":"2024-02-09T22:50:08.615818+0000","last_peered":"2024-02-09T22:50:08.615818+0000","last_clean":"2024-02-09T22:50:08.615818+0000","last_became_active":"2024-02-09T22:49:06.246447+0000","last_became_peered":"2024-02-09T22:49:06.246447+0000","last_unstale":"2024-02-09T22:50:08.615818+0000","last_undegraded":"2024-02-09T22:50:08.615818+0000","last_fullsized":"2024-02-09T22:50:08.615818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:30:43.720811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574163+0000","last_change":"2024-02-09T22:49:07.258490+0000","last_active":"2024-02-09T22:50:36.574163+0000","last_peered":"2024-02-09T22:50:36.574163+0000","last_clean":"2024-02-09T22:50:36.574163+0000","last_became_active":"2024-02-09T22:49:07.258055+0000","last_became_peered":"2024-02-09T22:49:07.258055+0000","last_unstale":"2024-02-09T22:50:36.574163+0000","last_undegraded":"2024-02-09T22:50:36.574163+0000","last_fullsized":"2024-02-09T22:50:36.574163+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:55:26.999142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615165+0000","last_change":"2024-02-09T22:49:06.251782+0000","last_active":"2024-02-09T22:50:08.615165+0000","last_peered":"2024-02-09T22:50:08.615165+0000","last_clean":"2024-02-09T22:50:08.615165+0000","last_became_active":"2024-02-09T22:49:06.251490+0000","last_became_peered":"2024-02-09T22:49:06.251490+0000","last_unstale":"2024-02-09T22:50:08.615165+0000","last_undegraded":"2024-02-09T22:50:08.615165+0000","last_fullsized":"2024-02-09T22:50:08.615165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:58:31.312053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572865+0000","last_change":"2024-02-09T22:49:07.258146+0000","last_active":"2024-02-09T22:50:36.572865+0000","last_peered":"2024-02-09T22:50:36.572865+0000","last_clean":"2024-02-09T22:50:36.572865+0000","last_became_active":"2024-02-09T22:49:07.257837+0000","last_became_peered":"2024-02-09T22:49:07.257837+0000","last_unstale":"2024-02-09T22:50:36.572865+0000","last_undegraded":"2024-02-09T22:50:36.572865+0000","last_fullsized":"2024-02-09T22:50:36.572865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:51:35.891033+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'5","reported_seq":240,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.274534+0000","last_change":"2024-02-09T22:49:06.250763+0000","last_active":"2024-02-09T22:51:00.274534+0000","last_peered":"2024-02-09T22:51:00.274534+0000","last_clean":"2024-02-09T22:51:00.274534+0000","last_became_active":"2024-02-09T22:49:06.250451+0000","last_became_peered":"2024-02-09T22:49:06.250451+0000","last_unstale":"2024-02-09T22:51:00.274534+0000","last_undegraded":"2024-02-09T22:51:00.274534+0000","last_fullsized":"2024-02-09T22:51:00.274534+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:11:50.943098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":216,"num_read_kb":216,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579648+0000","last_change":"2024-02-09T22:49:07.266586+0000","last_active":"2024-02-09T22:50:36.579648+0000","last_peered":"2024-02-09T22:50:36.579648+0000","last_clean":"2024-02-09T22:50:36.579648+0000","last_became_active":"2024-02-09T22:49:07.264406+0000","last_became_peered":"2024-02-09T22:49:07.264406+0000","last_unstale":"2024-02-09T22:50:36.579648+0000","last_undegraded":"2024-02-09T22:50:36.579648+0000","last_fullsized":"2024-02-09T22:50:36.579648+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:36:01.664527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264958+0000","last_change":"2024-02-09T22:49:06.255227+0000","last_active":"2024-02-09T22:50:08.264958+0000","last_peered":"2024-02-09T22:50:08.264958+0000","last_clean":"2024-02-09T22:50:08.264958+0000","last_became_active":"2024-02-09T22:49:06.254913+0000","last_became_peered":"2024-02-09T22:49:06.254913+0000","last_unstale":"2024-02-09T22:50:08.264958+0000","last_undegraded":"2024-02-09T22:50:08.264958+0000","last_fullsized":"2024-02-09T22:50:08.264958+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:48:23.275709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575891+0000","last_change":"2024-02-09T22:49:07.271454+0000","last_active":"2024-02-09T22:50:36.575891+0000","last_peered":"2024-02-09T22:50:36.575891+0000","last_clean":"2024-02-09T22:50:36.575891+0000","last_became_active":"2024-02-09T22:49:07.270638+0000","last_became_peered":"2024-02-09T22:49:07.270638+0000","last_unstale":"2024-02-09T22:50:36.575891+0000","last_undegraded":"2024-02-09T22:50:36.575891+0000","last_fullsized":"2024-02-09T22:50:36.575891+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:39:51.886169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615718+0000","last_change":"2024-02-09T22:49:06.249147+0000","last_active":"2024-02-09T22:50:08.615718+0000","last_peered":"2024-02-09T22:50:08.615718+0000","last_clean":"2024-02-09T22:50:08.615718+0000","last_became_active":"2024-02-09T22:49:06.248821+0000","last_became_peered":"2024-02-09T22:49:06.248821+0000","last_unstale":"2024-02-09T22:50:08.615718+0000","last_undegraded":"2024-02-09T22:50:08.615718+0000","last_fullsized":"2024-02-09T22:50:08.615718+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:36:40.202068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.585093+0000","last_change":"2024-02-09T22:49:07.263216+0000","last_active":"2024-02-09T22:50:36.585093+0000","last_peered":"2024-02-09T22:50:36.585093+0000","last_clean":"2024-02-09T22:50:36.585093+0000","last_became_active":"2024-02-09T22:49:07.262628+0000","last_became_peered":"2024-02-09T22:49:07.262628+0000","last_unstale":"2024-02-09T22:50:36.585093+0000","last_undegraded":"2024-02-09T22:50:36.585093+0000","last_fullsized":"2024-02-09T22:50:36.585093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:30:05.472864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'34","reported_seq":277,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.275099+0000","last_change":"2024-02-09T22:49:06.254857+0000","last_active":"2024-02-09T22:51:00.275099+0000","last_peered":"2024-02-09T22:51:00.275099+0000","last_clean":"2024-02-09T22:51:00.275099+0000","last_became_active":"2024-02-09T22:49:06.254551+0000","last_became_peered":"2024-02-09T22:49:06.254551+0000","last_unstale":"2024-02-09T22:51:00.275099+0000","last_undegraded":"2024-02-09T22:51:00.275099+0000","last_fullsized":"2024-02-09T22:51:00.275099+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:05:30.167348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59951,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":224,"num_read_kb":171,"num_write":36,"num_write_kb":72,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577233+0000","last_change":"2024-02-09T22:49:07.255793+0000","last_active":"2024-02-09T22:50:36.577233+0000","last_peered":"2024-02-09T22:50:36.577233+0000","last_clean":"2024-02-09T22:50:36.577233+0000","last_became_active":"2024-02-09T22:49:07.255349+0000","last_became_peered":"2024-02-09T22:49:07.255349+0000","last_unstale":"2024-02-09T22:50:36.577233+0000","last_undegraded":"2024-02-09T22:50:36.577233+0000","last_fullsized":"2024-02-09T22:50:36.577233+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:16:14.982998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267676+0000","last_change":"2024-02-09T22:49:06.256649+0000","last_active":"2024-02-09T22:50:08.267676+0000","last_peered":"2024-02-09T22:50:08.267676+0000","last_clean":"2024-02-09T22:50:08.267676+0000","last_became_active":"2024-02-09T22:49:06.256185+0000","last_became_peered":"2024-02-09T22:49:06.256185+0000","last_unstale":"2024-02-09T22:50:08.267676+0000","last_undegraded":"2024-02-09T22:50:08.267676+0000","last_fullsized":"2024-02-09T22:50:08.267676+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T06:42:58.584746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574237+0000","last_change":"2024-02-09T22:49:07.258862+0000","last_active":"2024-02-09T22:50:36.574237+0000","last_peered":"2024-02-09T22:50:36.574237+0000","last_clean":"2024-02-09T22:50:36.574237+0000","last_became_active":"2024-02-09T22:49:07.258517+0000","last_became_peered":"2024-02-09T22:49:07.258517+0000","last_unstale":"2024-02-09T22:50:36.574237+0000","last_undegraded":"2024-02-09T22:50:36.574237+0000","last_fullsized":"2024-02-09T22:50:36.574237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:51:54.547817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615556+0000","last_change":"2024-02-09T22:49:06.249452+0000","last_active":"2024-02-09T22:50:08.615556+0000","last_peered":"2024-02-09T22:50:08.615556+0000","last_clean":"2024-02-09T22:50:08.615556+0000","last_became_active":"2024-02-09T22:49:06.249137+0000","last_became_peered":"2024-02-09T22:49:06.249137+0000","last_unstale":"2024-02-09T22:50:08.615556+0000","last_undegraded":"2024-02-09T22:50:08.615556+0000","last_fullsized":"2024-02-09T22:50:08.615556+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:05:41.306853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573769+0000","last_change":"2024-02-09T22:49:07.257380+0000","last_active":"2024-02-09T22:50:36.573769+0000","last_peered":"2024-02-09T22:50:36.573769+0000","last_clean":"2024-02-09T22:50:36.573769+0000","last_became_active":"2024-02-09T22:49:07.256680+0000","last_became_peered":"2024-02-09T22:49:07.256680+0000","last_unstale":"2024-02-09T22:50:36.573769+0000","last_undegraded":"2024-02-09T22:50:36.573769+0000","last_fullsized":"2024-02-09T22:50:36.573769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:01:58.162539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":131,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:56.292256+0000","last_change":"2024-02-09T22:49:06.250899+0000","last_active":"2024-02-09T22:50:56.292256+0000","last_peered":"2024-02-09T22:50:56.292256+0000","last_clean":"2024-02-09T22:50:56.292256+0000","last_became_active":"2024-02-09T22:49:06.250496+0000","last_became_peered":"2024-02-09T22:49:06.250496+0000","last_unstale":"2024-02-09T22:50:56.292256+0000","last_undegraded":"2024-02-09T22:50:56.292256+0000","last_fullsized":"2024-02-09T22:50:56.292256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T22:55:41.397801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":108,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579110+0000","last_change":"2024-02-09T22:49:07.259989+0000","last_active":"2024-02-09T22:50:36.579110+0000","last_peered":"2024-02-09T22:50:36.579110+0000","last_clean":"2024-02-09T22:50:36.579110+0000","last_became_active":"2024-02-09T22:49:07.259632+0000","last_became_peered":"2024-02-09T22:49:07.259632+0000","last_unstale":"2024-02-09T22:50:36.579110+0000","last_undegraded":"2024-02-09T22:50:36.579110+0000","last_fullsized":"2024-02-09T22:50:36.579110+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:38:06.496588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.265893+0000","last_change":"2024-02-09T22:49:06.252003+0000","last_active":"2024-02-09T22:50:08.265893+0000","last_peered":"2024-02-09T22:50:08.265893+0000","last_clean":"2024-02-09T22:50:08.265893+0000","last_became_active":"2024-02-09T22:49:06.251609+0000","last_became_peered":"2024-02-09T22:49:06.251609+0000","last_unstale":"2024-02-09T22:50:08.265893+0000","last_undegraded":"2024-02-09T22:50:08.265893+0000","last_fullsized":"2024-02-09T22:50:08.265893+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:03:59.163281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.562328+0000","last_change":"2024-02-09T22:50:08.539326+0000","last_active":"2024-02-09T22:50:36.562328+0000","last_peered":"2024-02-09T22:50:36.562328+0000","last_clean":"2024-02-09T22:50:36.562328+0000","last_became_active":"2024-02-09T22:50:08.539091+0000","last_became_peered":"2024-02-09T22:50:08.539091+0000","last_unstale":"2024-02-09T22:50:36.562328+0000","last_undegraded":"2024-02-09T22:50:36.562328+0000","last_fullsized":"2024-02-09T22:50:36.562328+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:31:57.282733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,4,1],"acting":[9,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.267149+0000","last_change":"2024-02-09T22:49:06.246775+0000","last_active":"2024-02-09T22:50:08.267149+0000","last_peered":"2024-02-09T22:50:08.267149+0000","last_clean":"2024-02-09T22:50:08.267149+0000","last_became_active":"2024-02-09T22:49:06.246448+0000","last_became_peered":"2024-02-09T22:49:06.246448+0000","last_unstale":"2024-02-09T22:50:08.267149+0000","last_undegraded":"2024-02-09T22:50:08.267149+0000","last_fullsized":"2024-02-09T22:50:08.267149+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:51:29.210304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573178+0000","last_change":"2024-02-09T22:50:08.616790+0000","last_active":"2024-02-09T22:50:36.573178+0000","last_peered":"2024-02-09T22:50:36.573178+0000","last_clean":"2024-02-09T22:50:36.573178+0000","last_became_active":"2024-02-09T22:50:08.616404+0000","last_became_peered":"2024-02-09T22:50:08.616404+0000","last_unstale":"2024-02-09T22:50:36.573178+0000","last_undegraded":"2024-02-09T22:50:36.573178+0000","last_fullsized":"2024-02-09T22:50:36.573178+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:14:18.902026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263341+0000","last_change":"2024-02-09T22:49:06.259416+0000","last_active":"2024-02-09T22:50:08.263341+0000","last_peered":"2024-02-09T22:50:08.263341+0000","last_clean":"2024-02-09T22:50:08.263341+0000","last_became_active":"2024-02-09T22:49:06.259105+0000","last_became_peered":"2024-02-09T22:49:06.259105+0000","last_unstale":"2024-02-09T22:50:08.263341+0000","last_undegraded":"2024-02-09T22:50:08.263341+0000","last_fullsized":"2024-02-09T22:50:08.263341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:32.693021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575910+0000","last_change":"2024-02-09T22:49:07.260905+0000","last_active":"2024-02-09T22:50:36.575910+0000","last_peered":"2024-02-09T22:50:36.575910+0000","last_clean":"2024-02-09T22:50:36.575910+0000","last_became_active":"2024-02-09T22:49:07.260560+0000","last_became_peered":"2024-02-09T22:49:07.260560+0000","last_unstale":"2024-02-09T22:50:36.575910+0000","last_undegraded":"2024-02-09T22:50:36.575910+0000","last_fullsized":"2024-02-09T22:50:36.575910+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:35:05.853090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"88'6","reported_seq":135,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:51:00.268171+0000","last_change":"2024-02-09T22:49:06.254641+0000","last_active":"2024-02-09T22:51:00.268171+0000","last_peered":"2024-02-09T22:51:00.268171+0000","last_clean":"2024-02-09T22:51:00.268171+0000","last_became_active":"2024-02-09T22:49:06.254266+0000","last_became_peered":"2024-02-09T22:49:06.254266+0000","last_unstale":"2024-02-09T22:51:00.268171+0000","last_undegraded":"2024-02-09T22:51:00.268171+0000","last_fullsized":"2024-02-09T22:51:00.268171+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:39:02.037179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":896,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577911+0000","last_change":"2024-02-09T22:49:07.258293+0000","last_active":"2024-02-09T22:50:36.577911+0000","last_peered":"2024-02-09T22:50:36.577911+0000","last_clean":"2024-02-09T22:50:36.577911+0000","last_became_active":"2024-02-09T22:49:07.257885+0000","last_became_peered":"2024-02-09T22:49:07.257885+0000","last_unstale":"2024-02-09T22:50:36.577911+0000","last_undegraded":"2024-02-09T22:50:36.577911+0000","last_fullsized":"2024-02-09T22:50:36.577911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:45:14.177858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.572932+0000","last_change":"2024-02-09T22:49:07.257226+0000","last_active":"2024-02-09T22:50:36.572932+0000","last_peered":"2024-02-09T22:50:36.572932+0000","last_clean":"2024-02-09T22:50:36.572932+0000","last_became_active":"2024-02-09T22:49:07.256654+0000","last_became_peered":"2024-02-09T22:49:07.256654+0000","last_unstale":"2024-02-09T22:50:36.572932+0000","last_undegraded":"2024-02-09T22:50:36.572932+0000","last_fullsized":"2024-02-09T22:50:36.572932+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:25:32.931614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":133,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:59.296638+0000","last_change":"2024-02-09T22:49:06.246043+0000","last_active":"2024-02-09T22:50:59.296638+0000","last_peered":"2024-02-09T22:50:59.296638+0000","last_clean":"2024-02-09T22:50:59.296638+0000","last_became_active":"2024-02-09T22:49:06.245725+0000","last_became_peered":"2024-02-09T22:49:06.245725+0000","last_unstale":"2024-02-09T22:50:59.296638+0000","last_undegraded":"2024-02-09T22:50:59.296638+0000","last_fullsized":"2024-02-09T22:50:59.296638+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:11:41.292465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":109,"num_read_kb":109,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575148+0000","last_change":"2024-02-09T22:49:07.260551+0000","last_active":"2024-02-09T22:50:36.575148+0000","last_peered":"2024-02-09T22:50:36.575148+0000","last_clean":"2024-02-09T22:50:36.575148+0000","last_became_active":"2024-02-09T22:49:07.258932+0000","last_became_peered":"2024-02-09T22:49:07.258932+0000","last_unstale":"2024-02-09T22:50:36.575148+0000","last_undegraded":"2024-02-09T22:50:36.575148+0000","last_fullsized":"2024-02-09T22:50:36.575148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:25.410162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538316+0000","last_change":"2024-02-09T22:49:06.257209+0000","last_active":"2024-02-09T22:50:08.538316+0000","last_peered":"2024-02-09T22:50:08.538316+0000","last_clean":"2024-02-09T22:50:08.538316+0000","last_became_active":"2024-02-09T22:49:06.256781+0000","last_became_peered":"2024-02-09T22:49:06.256781+0000","last_unstale":"2024-02-09T22:50:08.538316+0000","last_undegraded":"2024-02-09T22:50:08.538316+0000","last_fullsized":"2024-02-09T22:50:08.538316+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:19:24.190475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":354,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264739+0000","last_change":"2024-02-09T22:47:54.172239+0000","last_active":"2024-02-09T22:50:08.264739+0000","last_peered":"2024-02-09T22:50:08.264739+0000","last_clean":"2024-02-09T22:50:08.264739+0000","last_became_active":"2024-02-09T22:47:54.171705+0000","last_became_peered":"2024-02-09T22:47:54.171705+0000","last_unstale":"2024-02-09T22:50:08.264739+0000","last_undegraded":"2024-02-09T22:50:08.264739+0000","last_fullsized":"2024-02-09T22:50:08.264739+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:44:33.041288+0000","last_clean_scrub_stamp":"2024-02-09T22:44:33.041288+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:54:24.624652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.569342+0000","last_change":"2024-02-09T22:49:07.257729+0000","last_active":"2024-02-09T22:50:36.569342+0000","last_peered":"2024-02-09T22:50:36.569342+0000","last_clean":"2024-02-09T22:50:36.569342+0000","last_became_active":"2024-02-09T22:49:07.257387+0000","last_became_peered":"2024-02-09T22:49:07.257387+0000","last_unstale":"2024-02-09T22:50:36.569342+0000","last_undegraded":"2024-02-09T22:50:36.569342+0000","last_fullsized":"2024-02-09T22:50:36.569342+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:13:19.081060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.264784+0000","last_change":"2024-02-09T22:49:06.248981+0000","last_active":"2024-02-09T22:50:08.264784+0000","last_peered":"2024-02-09T22:50:08.264784+0000","last_clean":"2024-02-09T22:50:08.264784+0000","last_became_active":"2024-02-09T22:49:06.248687+0000","last_became_peered":"2024-02-09T22:49:06.248687+0000","last_unstale":"2024-02-09T22:50:08.264784+0000","last_undegraded":"2024-02-09T22:50:08.264784+0000","last_fullsized":"2024-02-09T22:50:08.264784+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:35:38.324728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.278066+0000","last_change":"2024-02-09T22:50:08.278066+0000","last_active":"2024-02-09T22:50:08.278066+0000","last_peered":"2024-02-09T22:50:08.278066+0000","last_clean":"2024-02-09T22:50:08.278066+0000","last_became_active":"2024-02-09T22:50:08.271486+0000","last_became_peered":"2024-02-09T22:50:08.271486+0000","last_unstale":"2024-02-09T22:50:08.278066+0000","last_undegraded":"2024-02-09T22:50:08.278066+0000","last_fullsized":"2024-02-09T22:50:08.278066+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:54:12.295421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567960+0000","last_change":"2024-02-09T22:49:07.250465+0000","last_active":"2024-02-09T22:50:36.567960+0000","last_peered":"2024-02-09T22:50:36.567960+0000","last_clean":"2024-02-09T22:50:36.567960+0000","last_became_active":"2024-02-09T22:49:07.250128+0000","last_became_peered":"2024-02-09T22:49:07.250128+0000","last_unstale":"2024-02-09T22:50:36.567960+0000","last_undegraded":"2024-02-09T22:50:36.567960+0000","last_fullsized":"2024-02-09T22:50:36.567960+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:38:22.287887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'13","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.263515+0000","last_change":"2024-02-09T22:49:06.255330+0000","last_active":"2024-02-09T22:50:08.263515+0000","last_peered":"2024-02-09T22:50:08.263515+0000","last_clean":"2024-02-09T22:50:08.263515+0000","last_became_active":"2024-02-09T22:49:06.255006+0000","last_became_peered":"2024-02-09T22:49:06.255006+0000","last_unstale":"2024-02-09T22:50:08.263515+0000","last_undegraded":"2024-02-09T22:50:08.263515+0000","last_fullsized":"2024-02-09T22:50:08.263515+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:28:46.224189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574800+0000","last_change":"2024-02-09T22:49:07.258723+0000","last_active":"2024-02-09T22:50:36.574800+0000","last_peered":"2024-02-09T22:50:36.574800+0000","last_clean":"2024-02-09T22:50:36.574800+0000","last_became_active":"2024-02-09T22:49:07.258346+0000","last_became_peered":"2024-02-09T22:49:07.258346+0000","last_unstale":"2024-02-09T22:50:36.574800+0000","last_undegraded":"2024-02-09T22:50:36.574800+0000","last_fullsized":"2024-02-09T22:50:36.574800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:31:49.939955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.538049+0000","last_change":"2024-02-09T22:49:06.246617+0000","last_active":"2024-02-09T22:50:08.538049+0000","last_peered":"2024-02-09T22:50:08.538049+0000","last_clean":"2024-02-09T22:50:08.538049+0000","last_became_active":"2024-02-09T22:49:06.246271+0000","last_became_peered":"2024-02-09T22:49:06.246271+0000","last_unstale":"2024-02-09T22:50:08.538049+0000","last_undegraded":"2024-02-09T22:50:08.538049+0000","last_fullsized":"2024-02-09T22:50:08.538049+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:24:18.938927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578272+0000","last_change":"2024-02-09T22:49:07.269950+0000","last_active":"2024-02-09T22:50:36.578272+0000","last_peered":"2024-02-09T22:50:36.578272+0000","last_clean":"2024-02-09T22:50:36.578272+0000","last_became_active":"2024-02-09T22:49:07.269619+0000","last_became_peered":"2024-02-09T22:49:07.269619+0000","last_unstale":"2024-02-09T22:50:36.578272+0000","last_undegraded":"2024-02-09T22:50:36.578272+0000","last_fullsized":"2024-02-09T22:50:36.578272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:42:20.695900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.537439+0000","last_change":"2024-02-09T22:49:06.252018+0000","last_active":"2024-02-09T22:50:08.537439+0000","last_peered":"2024-02-09T22:50:08.537439+0000","last_clean":"2024-02-09T22:50:08.537439+0000","last_became_active":"2024-02-09T22:49:06.251467+0000","last_became_peered":"2024-02-09T22:49:06.251467+0000","last_unstale":"2024-02-09T22:50:08.537439+0000","last_undegraded":"2024-02-09T22:50:08.537439+0000","last_fullsized":"2024-02-09T22:50:08.537439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T07:48:45.068833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.538911+0000","last_change":"2024-02-09T22:49:07.259429+0000","last_active":"2024-02-09T22:50:36.538911+0000","last_peered":"2024-02-09T22:50:36.538911+0000","last_clean":"2024-02-09T22:50:36.538911+0000","last_became_active":"2024-02-09T22:49:07.258742+0000","last_became_peered":"2024-02-09T22:49:07.258742+0000","last_unstale":"2024-02-09T22:50:36.538911+0000","last_undegraded":"2024-02-09T22:50:36.538911+0000","last_fullsized":"2024-02-09T22:50:36.538911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:21:18.210601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.266189+0000","last_change":"2024-02-09T22:49:06.253277+0000","last_active":"2024-02-09T22:50:08.266189+0000","last_peered":"2024-02-09T22:50:08.266189+0000","last_clean":"2024-02-09T22:50:08.266189+0000","last_became_active":"2024-02-09T22:49:06.252842+0000","last_became_peered":"2024-02-09T22:49:06.252842+0000","last_unstale":"2024-02-09T22:50:08.266189+0000","last_undegraded":"2024-02-09T22:50:08.266189+0000","last_fullsized":"2024-02-09T22:50:08.266189+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:10:47.050460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568139+0000","last_change":"2024-02-09T22:49:07.264858+0000","last_active":"2024-02-09T22:50:36.568139+0000","last_peered":"2024-02-09T22:50:36.568139+0000","last_clean":"2024-02-09T22:50:36.568139+0000","last_became_active":"2024-02-09T22:49:07.264537+0000","last_became_peered":"2024-02-09T22:49:07.264537+0000","last_unstale":"2024-02-09T22:50:36.568139+0000","last_undegraded":"2024-02-09T22:50:36.568139+0000","last_fullsized":"2024-02-09T22:50:36.568139+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:14:12.171961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.273440+0000","last_change":"2024-02-09T22:50:08.273440+0000","last_active":"2024-02-09T22:50:08.273440+0000","last_peered":"2024-02-09T22:50:08.273440+0000","last_clean":"2024-02-09T22:50:08.273440+0000","last_became_active":"2024-02-09T22:50:08.268960+0000","last_became_peered":"2024-02-09T22:50:08.268960+0000","last_unstale":"2024-02-09T22:50:08.273440+0000","last_undegraded":"2024-02-09T22:50:08.273440+0000","last_fullsized":"2024-02-09T22:50:08.273440+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:26:41.735330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.576486+0000","last_change":"2024-02-09T22:49:07.259880+0000","last_active":"2024-02-09T22:50:36.576486+0000","last_peered":"2024-02-09T22:50:36.576486+0000","last_clean":"2024-02-09T22:50:36.576486+0000","last_became_active":"2024-02-09T22:49:07.259297+0000","last_became_peered":"2024-02-09T22:49:07.259297+0000","last_unstale":"2024-02-09T22:50:36.576486+0000","last_undegraded":"2024-02-09T22:50:36.576486+0000","last_fullsized":"2024-02-09T22:50:36.576486+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:25:19.586920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":36,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.542393+0000","last_change":"2024-02-09T22:50:08.542393+0000","last_active":"2024-02-09T22:50:08.542393+0000","last_peered":"2024-02-09T22:50:08.542393+0000","last_clean":"2024-02-09T22:50:08.542393+0000","last_became_active":"2024-02-09T22:50:08.539084+0000","last_became_peered":"2024-02-09T22:50:08.539084+0000","last_unstale":"2024-02-09T22:50:08.542393+0000","last_undegraded":"2024-02-09T22:50:08.542393+0000","last_fullsized":"2024-02-09T22:50:08.542393+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:32:59.713756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584235+0000","last_change":"2024-02-09T22:49:07.272068+0000","last_active":"2024-02-09T22:50:36.584235+0000","last_peered":"2024-02-09T22:50:36.584235+0000","last_clean":"2024-02-09T22:50:36.584235+0000","last_became_active":"2024-02-09T22:49:07.271632+0000","last_became_peered":"2024-02-09T22:49:07.271632+0000","last_unstale":"2024-02-09T22:50:36.584235+0000","last_undegraded":"2024-02-09T22:50:36.584235+0000","last_fullsized":"2024-02-09T22:50:36.584235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:13:07.328084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'5","reported_seq":131,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:56.269509+0000","last_change":"2024-02-09T22:49:06.259200+0000","last_active":"2024-02-09T22:50:56.269509+0000","last_peered":"2024-02-09T22:50:56.269509+0000","last_clean":"2024-02-09T22:50:56.269509+0000","last_became_active":"2024-02-09T22:49:06.258804+0000","last_became_peered":"2024-02-09T22:49:06.258804+0000","last_unstale":"2024-02-09T22:50:56.269509+0000","last_undegraded":"2024-02-09T22:50:56.269509+0000","last_fullsized":"2024-02-09T22:50:56.269509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:18:29.615926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":107,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.584413+0000","last_change":"2024-02-09T22:49:07.270097+0000","last_active":"2024-02-09T22:50:36.584413+0000","last_peered":"2024-02-09T22:50:36.584413+0000","last_clean":"2024-02-09T22:50:36.584413+0000","last_became_active":"2024-02-09T22:49:07.269713+0000","last_became_peered":"2024-02-09T22:49:07.269713+0000","last_unstale":"2024-02-09T22:50:36.584413+0000","last_undegraded":"2024-02-09T22:50:36.584413+0000","last_fullsized":"2024-02-09T22:50:36.584413+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:40:28.504813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615601+0000","last_change":"2024-02-09T22:49:06.252248+0000","last_active":"2024-02-09T22:50:08.615601+0000","last_peered":"2024-02-09T22:50:08.615601+0000","last_clean":"2024-02-09T22:50:08.615601+0000","last_became_active":"2024-02-09T22:49:06.251925+0000","last_became_peered":"2024-02-09T22:49:06.251925+0000","last_unstale":"2024-02-09T22:50:08.615601+0000","last_undegraded":"2024-02-09T22:50:08.615601+0000","last_fullsized":"2024-02-09T22:50:08.615601+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:11:51.153254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.577804+0000","last_change":"2024-02-09T22:49:07.258526+0000","last_active":"2024-02-09T22:50:36.577804+0000","last_peered":"2024-02-09T22:50:36.577804+0000","last_clean":"2024-02-09T22:50:36.577804+0000","last_became_active":"2024-02-09T22:49:07.258165+0000","last_became_peered":"2024-02-09T22:49:07.258165+0000","last_unstale":"2024-02-09T22:50:36.577804+0000","last_undegraded":"2024-02-09T22:50:36.577804+0000","last_fullsized":"2024-02-09T22:50:36.577804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:24:05.120406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:08.615940+0000","last_change":"2024-02-09T22:49:06.256720+0000","last_active":"2024-02-09T22:50:08.615940+0000","last_peered":"2024-02-09T22:50:08.615940+0000","last_clean":"2024-02-09T22:50:08.615940+0000","last_became_active":"2024-02-09T22:49:06.256420+0000","last_became_peered":"2024-02-09T22:49:06.256420+0000","last_unstale":"2024-02-09T22:50:08.615940+0000","last_undegraded":"2024-02-09T22:50:08.615940+0000","last_fullsized":"2024-02-09T22:50:08.615940+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:05.218276+0000","last_clean_scrub_stamp":"2024-02-09T22:49:05.218276+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:59:23.526400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.578172+0000","last_change":"2024-02-09T22:49:07.259972+0000","last_active":"2024-02-09T22:50:36.578172+0000","last_peered":"2024-02-09T22:50:36.578172+0000","last_clean":"2024-02-09T22:50:36.578172+0000","last_became_active":"2024-02-09T22:49:07.259594+0000","last_became_peered":"2024-02-09T22:49:07.259594+0000","last_unstale":"2024-02-09T22:50:36.578172+0000","last_undegraded":"2024-02-09T22:50:36.578172+0000","last_fullsized":"2024-02-09T22:50:36.578172+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:00:18.371155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.573750+0000","last_change":"2024-02-09T22:49:07.257250+0000","last_active":"2024-02-09T22:50:36.573750+0000","last_peered":"2024-02-09T22:50:36.573750+0000","last_clean":"2024-02-09T22:50:36.573750+0000","last_became_active":"2024-02-09T22:49:07.256891+0000","last_became_peered":"2024-02-09T22:49:07.256891+0000","last_unstale":"2024-02-09T22:50:36.573750+0000","last_undegraded":"2024-02-09T22:50:36.573750+0000","last_fullsized":"2024-02-09T22:50:36.573750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T01:38:24.366970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.567974+0000","last_change":"2024-02-09T22:49:07.258778+0000","last_active":"2024-02-09T22:50:36.567974+0000","last_peered":"2024-02-09T22:50:36.567974+0000","last_clean":"2024-02-09T22:50:36.567974+0000","last_became_active":"2024-02-09T22:49:07.258461+0000","last_became_peered":"2024-02-09T22:49:07.258461+0000","last_unstale":"2024-02-09T22:50:36.567974+0000","last_undegraded":"2024-02-09T22:50:36.567974+0000","last_fullsized":"2024-02-09T22:50:36.567974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:24:49.099644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.566404+0000","last_change":"2024-02-09T22:50:08.272126+0000","last_active":"2024-02-09T22:50:36.566404+0000","last_peered":"2024-02-09T22:50:36.566404+0000","last_clean":"2024-02-09T22:50:36.566404+0000","last_became_active":"2024-02-09T22:50:08.271722+0000","last_became_peered":"2024-02-09T22:50:08.271722+0000","last_unstale":"2024-02-09T22:50:36.566404+0000","last_undegraded":"2024-02-09T22:50:36.566404+0000","last_fullsized":"2024-02-09T22:50:36.566404+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T00:43:31.304504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575883+0000","last_change":"2024-02-09T22:49:07.271305+0000","last_active":"2024-02-09T22:50:36.575883+0000","last_peered":"2024-02-09T22:50:36.575883+0000","last_clean":"2024-02-09T22:50:36.575883+0000","last_became_active":"2024-02-09T22:49:07.270444+0000","last_became_peered":"2024-02-09T22:49:07.270444+0000","last_unstale":"2024-02-09T22:50:36.575883+0000","last_undegraded":"2024-02-09T22:50:36.575883+0000","last_fullsized":"2024-02-09T22:50:36.575883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T09:18:52.190373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.579765+0000","last_change":"2024-02-09T22:49:07.261735+0000","last_active":"2024-02-09T22:50:36.579765+0000","last_peered":"2024-02-09T22:50:36.579765+0000","last_clean":"2024-02-09T22:50:36.579765+0000","last_became_active":"2024-02-09T22:49:07.261400+0000","last_became_peered":"2024-02-09T22:49:07.261400+0000","last_unstale":"2024-02-09T22:50:36.579765+0000","last_undegraded":"2024-02-09T22:50:36.579765+0000","last_fullsized":"2024-02-09T22:50:36.579765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:46:03.942575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.568845+0000","last_change":"2024-02-09T22:49:07.258461+0000","last_active":"2024-02-09T22:50:36.568845+0000","last_peered":"2024-02-09T22:50:36.568845+0000","last_clean":"2024-02-09T22:50:36.568845+0000","last_became_active":"2024-02-09T22:49:07.258022+0000","last_became_peered":"2024-02-09T22:49:07.258022+0000","last_unstale":"2024-02-09T22:50:36.568845+0000","last_undegraded":"2024-02-09T22:50:36.568845+0000","last_fullsized":"2024-02-09T22:50:36.568845+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-10T23:45:51.283761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.563750+0000","last_change":"2024-02-09T22:49:07.263423+0000","last_active":"2024-02-09T22:50:36.563750+0000","last_peered":"2024-02-09T22:50:36.563750+0000","last_clean":"2024-02-09T22:50:36.563750+0000","last_became_active":"2024-02-09T22:49:07.262722+0000","last_became_peered":"2024-02-09T22:49:07.262722+0000","last_unstale":"2024-02-09T22:50:36.563750+0000","last_undegraded":"2024-02-09T22:50:36.563750+0000","last_fullsized":"2024-02-09T22:50:36.563750+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:57:01.921327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571402+0000","last_change":"2024-02-09T22:49:07.269250+0000","last_active":"2024-02-09T22:50:36.571402+0000","last_peered":"2024-02-09T22:50:36.571402+0000","last_clean":"2024-02-09T22:50:36.571402+0000","last_became_active":"2024-02-09T22:49:07.268861+0000","last_became_peered":"2024-02-09T22:49:07.268861+0000","last_unstale":"2024-02-09T22:50:36.571402+0000","last_undegraded":"2024-02-09T22:50:36.571402+0000","last_fullsized":"2024-02-09T22:50:36.571402+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T05:31:30.583289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574138+0000","last_change":"2024-02-09T22:49:07.262011+0000","last_active":"2024-02-09T22:50:36.574138+0000","last_peered":"2024-02-09T22:50:36.574138+0000","last_clean":"2024-02-09T22:50:36.574138+0000","last_became_active":"2024-02-09T22:49:07.261468+0000","last_became_peered":"2024-02-09T22:49:07.261468+0000","last_unstale":"2024-02-09T22:50:36.574138+0000","last_undegraded":"2024-02-09T22:50:36.574138+0000","last_fullsized":"2024-02-09T22:50:36.574138+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T03:33:28.030875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.574591+0000","last_change":"2024-02-09T22:49:07.257678+0000","last_active":"2024-02-09T22:50:36.574591+0000","last_peered":"2024-02-09T22:50:36.574591+0000","last_clean":"2024-02-09T22:50:36.574591+0000","last_became_active":"2024-02-09T22:49:07.257099+0000","last_became_peered":"2024-02-09T22:49:07.257099+0000","last_unstale":"2024-02-09T22:50:36.574591+0000","last_undegraded":"2024-02-09T22:50:36.574591+0000","last_fullsized":"2024-02-09T22:50:36.574591+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T08:11:27.274029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.585107+0000","last_change":"2024-02-09T22:49:07.270229+0000","last_active":"2024-02-09T22:50:36.585107+0000","last_peered":"2024-02-09T22:50:36.585107+0000","last_clean":"2024-02-09T22:50:36.585107+0000","last_became_active":"2024-02-09T22:49:07.269302+0000","last_became_peered":"2024-02-09T22:49:07.269302+0000","last_unstale":"2024-02-09T22:50:36.585107+0000","last_undegraded":"2024-02-09T22:50:36.585107+0000","last_fullsized":"2024-02-09T22:50:36.585107+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T10:09:04.621711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.575799+0000","last_change":"2024-02-09T22:49:07.264865+0000","last_active":"2024-02-09T22:50:36.575799+0000","last_peered":"2024-02-09T22:50:36.575799+0000","last_clean":"2024-02-09T22:50:36.575799+0000","last_became_active":"2024-02-09T22:49:07.263185+0000","last_became_peered":"2024-02-09T22:49:07.263185+0000","last_unstale":"2024-02-09T22:50:36.575799+0000","last_undegraded":"2024-02-09T22:50:36.575799+0000","last_fullsized":"2024-02-09T22:50:36.575799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T02:42:19.023883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-02-09T22:50:36.571747+0000","last_change":"2024-02-09T22:49:07.264241+0000","last_active":"2024-02-09T22:50:36.571747+0000","last_peered":"2024-02-09T22:50:36.571747+0000","last_clean":"2024-02-09T22:50:36.571747+0000","last_became_active":"2024-02-09T22:49:07.263897+0000","last_became_peered":"2024-02-09T22:49:07.263897+0000","last_unstale":"2024-02-09T22:50:36.571747+0000","last_undegraded":"2024-02-09T22:50:36.571747+0000","last_fullsized":"2024-02-09T22:50:36.571747+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-02-09T22:49:06.226932+0000","last_clean_scrub_stamp":"2024-02-09T22:49:06.226932+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-02-11T04:05:34.280899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67213,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1113,"num_read_kb":633,"num_write":246,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":200715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351013,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":463574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.51200000000000001,"15min":0.504},"min":{"1min":0.38300000000000001,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.55100000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.63500000000000001,"15min":0.626},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.752}]},{"osd":2,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.45200000000000001,"15min":0.44600000000000001},"min":{"1min":0.35599999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.46999999999999997,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.64700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":4,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.64600000000000002,"15min":0.63200000000000001},"min":{"1min":0.51500000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.88},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.45000000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.81399999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.56399999999999995,"15min":0.55800000000000005},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.62,"5min":0.56999999999999995,"15min":0.56200000000000006},"min":{"1min":0.42499999999999999,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.73299999999999998}]},{"osd":6,"last update":"Fri Feb 9 22:49:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56000000000000005}]},{"osd":7,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.72199999999999998,"15min":0.70999999999999996},"min":{"1min":0.55400000000000005,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.67200000000000004,"15min":0.66600000000000004},"min":{"1min":0.47999999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.86499999999999999,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.84099999999999997}]},{"osd":8,"last update":"Fri Feb 9 22:50:09 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":1.002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.78600000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.73299999999999998,"15min":0.72899999999999998},"min":{"1min":0.53500000000000003,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":0.95999999999999996,"5min":1.04,"15min":1.04},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.747,"15min":0.73799999999999999},"min":{"1min":0.57199999999999995,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.82799999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:22 2024","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.52400000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.92800000000000005,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.75700000000000001,"15min":0.749},"min":{"1min":0.59699999999999998,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":0.999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.96699999999999997}]}]},{"osd":10,"up_from":73,"seq":313532612649,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32800,"kb_used_data":1624,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712352,"statfs":{"total":95995035648,"available":95961448448,"internally_reserved":0,"allocated":1662976,"data_stored":1114442,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.437}]},{"osd":1,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.496,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.69799999999999995,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.52000000000000002,"15min":0.52800000000000002},"min":{"1min":0.32800000000000001,"5min":0.314,"15min":0.314},"max":{"1min":0.66900000000000004,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.45400000000000001}]},{"osd":2,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64800000000000002,"15min":0.65600000000000003},"min":{"1min":0.47099999999999997,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.85399999999999998,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.66300000000000003,"15min":0.65700000000000003},"min":{"1min":0.45200000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.877,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.72499999999999998,"15min":0.69299999999999995},"min":{"1min":0.438,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":6.3280000000000003,"5min":6.3280000000000003,"15min":6.3280000000000003},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.63,"15min":0.64200000000000002},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.68500000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.59499999999999997,"15min":0.58799999999999997},"min":{"1min":0.36199999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.502}]},{"osd":5,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.67700000000000005,"15min":0.66700000000000004},"min":{"1min":0.505,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.747,"15min":0.753},"min":{"1min":0.48799999999999999,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":0.67100000000000004}]},{"osd":6,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.73299999999999998,"15min":0.745},"min":{"1min":0.441,"5min":0.433,"15min":0.433},"max":{"1min":1.0740000000000001,"5min":1.0740000000000001,"15min":1.0740000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.73199999999999998,"15min":0.72899999999999998},"min":{"1min":0.57299999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Feb 9 22:48:52 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.747,"5min":0.747,"15min":0.747},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":1.056,"5min":1.056,"15min":1.056},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.79500000000000004,"15min":0.79100000000000004},"min":{"1min":0.59799999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1399999999999999,"5min":1.1399999999999999,"15min":1.1399999999999999},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.79900000000000004,"15min":0.80400000000000005},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0589999999999999,"5min":1.069,"15min":1.069},"last":0.70999999999999996}]},{"osd":9,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.627,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.0409999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.80700000000000005,"15min":0.81999999999999995},"min":{"1min":0.56000000000000005,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.002,"5min":1.155,"15min":1.155},"last":0.65400000000000003}]},{"osd":11,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1240000000000001,"5min":1.1240000000000001,"15min":1.1240000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.84999999999999998,"5min":0.82899999999999996,"15min":0.82499999999999996},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.83199999999999996}]}]},{"osd":9,"up_from":67,"seq":287762808878,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":463392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.29899999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.74199999999999999,"5min":1.6040000000000001,"15min":1.6040000000000001},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.28899999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66000000000000003,"5min":1.5629999999999999,"15min":1.5629999999999999},"last":0.29999999999999999}]},{"osd":1,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.626,"15min":0.63400000000000001},"min":{"1min":0.41699999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.89400000000000002,"5min":1.022,"15min":1.022},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.62,"15min":0.63200000000000001},"min":{"1min":0.33500000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71199999999999997,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.58899999999999997}]},{"osd":2,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61499999999999999,"15min":0.61599999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.61499999999999999,"15min":0.625},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.42399999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.54300000000000004,"15min":0.54400000000000004},"min":{"1min":0.31900000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80100000000000005,"5min":2.246,"15min":2.246},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.35499999999999998,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.78500000000000003,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.33300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.65800000000000003,"15min":0.67800000000000005},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.85699999999999998,"5min":0.95299999999999996,"15min":0.95299999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.63,"5min":0.69199999999999995,"15min":0.71099999999999997},"min":{"1min":0.42699999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.83699999999999997,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.51400000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.68200000000000005,"15min":0.70399999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.83799999999999997,"5min":1.0960000000000001,"15min":1.0960000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.70499999999999996,"15min":0.71899999999999997},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.89700000000000002,"5min":1.01,"15min":1.01},"last":0.441}]},{"osd":6,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.65900000000000003,"15min":0.69099999999999995},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.76800000000000002,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.69699999999999995,"15min":0.73299999999999998},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.76800000000000002,"5min":2.036,"15min":2.036},"last":0.47499999999999998}]},{"osd":7,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.76400000000000001,"15min":0.79400000000000004},"min":{"1min":0.50800000000000001,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.90300000000000002,"5min":1.177,"15min":1.177},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.77500000000000002,"15min":0.80100000000000005},"min":{"1min":0.496,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.91400000000000003,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.49399999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.76800000000000002,"15min":0.78700000000000003},"min":{"1min":0.49099999999999999,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.94899999999999995,"5min":1.173,"15min":1.173},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.80000000000000004,"15min":0.82999999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.94199999999999995,"5min":1.387,"15min":1.387},"last":0.53600000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.746,"15min":0.751},"min":{"1min":0.48299999999999998,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":0.95999999999999996,"5min":1.089,"15min":1.089},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.70599999999999996,"5min":0.76500000000000001,"15min":0.77600000000000002},"min":{"1min":0.56200000000000006,"5min":0.504,"15min":0.504},"max":{"1min":0.97499999999999998,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.627}]}]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":463658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40999999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54400000000000004,"15min":0.54700000000000004},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.79400000000000004,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.77400000000000002}]},{"osd":1,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.71099999999999997,"5min":0.60599999999999998,"15min":0.59099999999999997},"min":{"1min":0.39700000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.628},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.61099999999999999,"15min":0.60499999999999998},"min":{"1min":0.438,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.89300000000000002,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.49099999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.44,"15min":0.42199999999999999},"min":{"1min":0.371,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.46800000000000003,"15min":0.46100000000000002},"min":{"1min":0.35799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.65200000000000002}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.54800000000000004,"15min":0.53800000000000003},"min":{"1min":0.373,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.56599999999999995,"15min":0.55800000000000005},"min":{"1min":0.43099999999999999,"5min":0.375,"15min":0.375},"max":{"1min":0.84399999999999997,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.67700000000000005}]},{"osd":4,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.625,"15min":0.61699999999999999},"min":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.97299999999999998,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.748,"5min":0.64100000000000001,"15min":0.627},"min":{"1min":0.55300000000000005,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.0529999999999999,"5min":1.0529999999999999,"15min":1.0529999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.53500000000000003,"15min":0.53900000000000003},"min":{"1min":0.32100000000000001,"5min":0.313,"15min":0.313},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.38800000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.63,"15min":0.621},"min":{"1min":0.46600000000000003,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.73999999999999999,"5min":0.67600000000000005,"15min":0.67000000000000004},"min":{"1min":0.47299999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.3260000000000001,"5min":1.3260000000000001,"15min":1.3260000000000001},"last":0.76400000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.69899999999999995,"15min":0.69999999999999996},"min":{"1min":0.56100000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.70599999999999996,"15min":0.68999999999999995},"min":{"1min":0.58099999999999996,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.82299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.78100000000000003,"5min":0.69899999999999995,"15min":0.66300000000000003},"min":{"1min":0.56699999999999995,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93700000000000006,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.81699999999999995,"5min":0.71099999999999997,"15min":0.67500000000000004},"min":{"1min":0.63700000000000001,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0529999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.90800000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.75600000000000001,"15min":0.70299999999999996},"min":{"1min":0.67400000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.1990000000000001,"5min":1.371,"15min":1.371},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.92700000000000005,"5min":0.77000000000000002,"15min":0.71799999999999997},"min":{"1min":0.69899999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1870000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.88400000000000001}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"min":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.871},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.84999999999999998}]}]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28560,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716592,"statfs":{"total":95995035648,"available":95965790208,"internally_reserved":0,"allocated":1581056,"data_stored":1056737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.57399999999999995,"15min":0.57099999999999995},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.91700000000000004,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.498},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.79000000000000004,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.47499999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.48099999999999998,"15min":0.46700000000000003},"min":{"1min":0.308,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.626,"5min":0.997,"15min":0.997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.52800000000000002,"15min":0.505},"min":{"1min":0.35999999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.59199999999999997,"15min":0.60199999999999998},"min":{"1min":0.38100000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.80400000000000005,"5min":1.01,"15min":1.01},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.65800000000000003,"15min":0.64900000000000002},"min":{"1min":0.46500000000000002,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.57799999999999996}]},{"osd":3,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.63600000000000001,"15min":0.628},"min":{"1min":0.38700000000000001,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.874,"5min":1.111,"15min":1.111},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.628,"15min":0.63500000000000001},"min":{"1min":0.41299999999999998,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.66600000000000004}]},{"osd":4,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.61199999999999999,"15min":0.622},"min":{"1min":0.373,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.69099999999999995,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.54900000000000004,"15min":0.53900000000000003},"min":{"1min":0.41399999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.77200000000000002,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.69799999999999995,"5min":0.66600000000000004,"15min":0.65800000000000003},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.85799999999999998,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.71899999999999997,"15min":0.70499999999999996},"min":{"1min":0.56699999999999995,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.89900000000000002,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.76600000000000001}]},{"osd":6,"last update":"Fri Feb 9 22:50:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.73399999999999999,"15min":0.72399999999999998},"min":{"1min":0.50600000000000001,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.89200000000000002,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.435,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.92000000000000004,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.81200000000000006}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.748,"15min":0.72799999999999998},"min":{"1min":0.53300000000000003,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96599999999999997,"5min":1.016,"15min":1.016},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.73199999999999998,"15min":0.70399999999999996},"min":{"1min":0.60499999999999998,"5min":0.498,"15min":0.498},"max":{"1min":0.95299999999999996,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.73699999999999999}]},{"osd":9,"last update":"Fri Feb 9 22:50:28 2024","interfaces":[{"interface":"back","average":{"1min":0.73699999999999999,"5min":0.76500000000000001,"15min":0.77200000000000002},"min":{"1min":0.55100000000000005,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.038,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.79300000000000004,"15min":0.79900000000000004},"min":{"1min":0.59799999999999998,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":0.95399999999999996,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.71699999999999997}]},{"osd":10,"last update":"Fri Feb 9 22:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.82299999999999995,"15min":0.82299999999999995},"min":{"1min":0.63,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":0.999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.754},{"interface":"front","average":{"1min":0.81399999999999995,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.69699999999999995,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.028,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.80000000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.83599999999999997}]}]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":463417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.378,"5min":0.36799999999999999,"15min":0.34399999999999997},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.54400000000000004,"5min":0.621,"15min":0.621},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.373,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.28799999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.47599999999999998,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.55400000000000005}]},{"osd":1,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.55500000000000005,"15min":0.53500000000000003},"min":{"1min":0.40899999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.56100000000000005,"15min":0.54400000000000004},"min":{"1min":0.42199999999999999,"5min":0.371,"15min":0.371},"max":{"1min":0.82799999999999996,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68899999999999995}]},{"osd":2,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.55300000000000005,"15min":0.52600000000000002},"min":{"1min":0.40699999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.76000000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.56599999999999995,"15min":0.54200000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.86799999999999999,"5min":0.875,"15min":0.875},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.42599999999999999,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.51700000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.441,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.628,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.45900000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.82199999999999995,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.624,"15min":0.61799999999999999},"min":{"1min":0.443,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.89700000000000002,"5min":0.91400000000000003,"15min":0.91400000000000003},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Feb 9 22:50:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60099999999999998,"15min":0.58399999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78600000000000003,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.629,"5min":0.66300000000000003,"15min":0.64100000000000001},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.84799999999999998,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.625}]},{"osd":7,"last update":"Fri Feb 9 22:50:48 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.64200000000000002,"15min":0.624},"min":{"1min":0.498,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.78000000000000003,"5min":0.999,"15min":0.999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.65500000000000003,"15min":0.63700000000000001},"min":{"1min":0.55400000000000005,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.81000000000000005,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57099999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.66100000000000003,"15min":0.67100000000000004},"min":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.84799999999999998,"5min":0.876,"15min":0.876},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.628,"5min":0.69199999999999995,"15min":0.69599999999999995},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.81200000000000006,"5min":0.995,"15min":0.995},"last":0.66200000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:31 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.65200000000000002,"15min":0.68700000000000006},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.80700000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.65200000000000002,"15min":0.68000000000000005},"min":{"1min":0.40899999999999997,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.78500000000000003,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.70599999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.68200000000000005,"5min":0.72099999999999997,"15min":0.73399999999999999},"min":{"1min":0.54800000000000004,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.82199999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.749,"15min":0.76700000000000002},"min":{"1min":0.59099999999999997,"5min":0.499,"15min":0.499},"max":{"1min":0.81100000000000005,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.748}]}]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":523665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.57699999999999996,"15min":0.52700000000000002},"min":{"1min":0.33200000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.97999999999999998,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.621},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.76300000000000001,"15min":0.59899999999999998},"min":{"1min":0.36699999999999999,"5min":0.33100000000000002,"15min":0.28999999999999998},"max":{"1min":0.80600000000000005,"5min":18.152000000000001,"15min":18.152000000000001},"last":0.52900000000000003}]},{"osd":2,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.77700000000000002,"15min":0.60299999999999998},"min":{"1min":0.49399999999999999,"5min":0.373,"15min":0.37},"max":{"1min":0.92400000000000004,"5min":18.18,"15min":18.18},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.61899999999999999,"15min":0.52500000000000002},"min":{"1min":0.52100000000000002,"5min":0.38500000000000001,"15min":0.372},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.49099999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.61599999999999999,"15min":0.57199999999999995},"min":{"1min":0.39900000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.94099999999999995,"5min":1.206,"15min":1.206},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.58699999999999997,"15min":0.54200000000000004},"min":{"1min":0.50700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78400000000000003,"5min":1.139,"15min":1.139},"last":0.59799999999999998}]},{"osd":4,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.50900000000000001,"15min":0.46100000000000002},"min":{"1min":0.35899999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.498,"15min":0.435},"min":{"1min":0.34000000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.79700000000000004}]},{"osd":5,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.71899999999999997,"15min":0.623},"min":{"1min":0.59299999999999997,"5min":0.44,"15min":0.44},"max":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"last":0.77400000000000002},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.72399999999999998,"15min":0.63600000000000001},"min":{"1min":0.56899999999999995,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.1839999999999999,"5min":1.1839999999999999,"15min":1.1839999999999999},"last":0.66400000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71399999999999997,"5min":0.66600000000000004,"15min":0.64500000000000002},"min":{"1min":0.48999999999999999,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.80300000000000005,"5min":0.72099999999999997,"15min":0.69699999999999995},"min":{"1min":0.53800000000000003,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.66100000000000003,"15min":0.629},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.70599999999999996,"15min":0.67200000000000004},"min":{"1min":0.44900000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.0669999999999999,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.86299999999999999}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.877,"5min":1.353,"15min":1.581},"min":{"1min":0.56699999999999995,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.1439999999999999,"5min":18.132000000000001,"15min":18.132000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.71799999999999997,"15min":0.69699999999999995},"min":{"1min":0.55900000000000005,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.69399999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.82499999999999996,"15min":0.81499999999999995},"min":{"1min":0.57199999999999995,"5min":0.54500000000000004,"15min":0.54500000000000004},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.86899999999999999,"15min":0.85899999999999999},"min":{"1min":0.72899999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.101,"5min":1.1819999999999999,"15min":1.1819999999999999},"last":0.98399999999999999}]},{"osd":10,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.78800000000000003,"15min":0.78200000000000003},"min":{"1min":0.60999999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.871,"15min":0.85399999999999998},"min":{"1min":0.67000000000000004,"5min":0.58499999999999996,"15min":0.58499999999999996},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.92200000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.748,"5min":0.748,"15min":0.748},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.96199999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":464448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Feb 9 22:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.56899999999999995,"15min":0.495},"min":{"1min":0.378,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.748,"5min":0.89900000000000002,"15min":0.89900000000000002},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.57599999999999996,"15min":0.498},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":6.6219999999999999,"5min":6.6219999999999999,"15min":6.6219999999999999},"last":0.45600000000000002}]},{"osd":2,"last update":"Fri Feb 9 22:50:41 2024","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.56699999999999995,"15min":0.52700000000000002},"min":{"1min":0.35499999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.395,"5min":1.395,"15min":1.395},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57499999999999996,"15min":0.51700000000000002},"min":{"1min":0.378,"5min":0.33600000000000002,"15min":0.28799999999999998},"max":{"1min":0.74299999999999999,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73399999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.377}]},{"osd":4,"last update":"Fri Feb 9 22:50:32 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.42599999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77200000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.745},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.61299999999999999,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.379,"5min":1.379,"15min":1.379},"last":0.55600000000000005}]},{"osd":5,"last update":"Fri Feb 9 22:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.61199999999999999,"15min":0.58599999999999997},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.622},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55900000000000005,"15min":0.57099999999999995},"min":{"1min":0.34599999999999997,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.76200000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Feb 9 22:50:13 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.88600000000000001,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56499999999999995,"15min":0.55900000000000005},"min":{"1min":0.35699999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84999999999999998,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.70499999999999996}]},{"osd":7,"last update":"Fri Feb 9 22:48:40 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.61299999999999999,"15min":0.60499999999999998},"min":{"1min":0.42099999999999999,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.5},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.68000000000000005,"15min":0.67600000000000005},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.069,"5min":1.069,"15min":1.069},"last":0.56399999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.70699999999999996,"15min":0.72499999999999998},"min":{"1min":0.42699999999999999,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.88600000000000001,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.63,"15min":0.64400000000000002},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.86099999999999999,"5min":1.0449999999999999,"15min":1.0449999999999999},"last":0.64200000000000002}]},{"osd":9,"last update":"Fri Feb 9 22:50:29 2024","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":0.72099999999999997},"min":{"1min":0.35399999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.80200000000000005,"5min":1.081,"15min":1.081},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.70099999999999996,"15min":0.749},"min":{"1min":0.42099999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.95899999999999996,"5min":1.202,"15min":1.202},"last":0.79000000000000004}]},{"osd":10,"last update":"Fri Feb 9 22:51:00 2024","interfaces":[{"interface":"back","average":{"1min":0.62,"5min":0.67600000000000005,"15min":0.70399999999999996},"min":{"1min":0.51100000000000001,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":0.79000000000000004,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.70399999999999996,"5min":0.72099999999999997,"15min":0.73899999999999999},"min":{"1min":0.60399999999999998,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.84799999999999998,"5min":1.137,"15min":1.137},"last":0.79900000000000004}]},{"osd":11,"last update":"Fri Feb 9 22:50:21 2024","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.71399999999999997,"15min":0.70999999999999996},"min":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.66400000000000003,"15min":0.65400000000000003},"min":{"1min":0.59699999999999998,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.89900000000000002,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.875}]}]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1052672,"data_stored":524417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.48599999999999999,"15min":0.45800000000000002},"min":{"1min":0.28299999999999997,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.71199999999999997,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.52500000000000002,"15min":0.48299999999999998},"min":{"1min":0.26500000000000001,"5min":0.224,"15min":0.224},"max":{"1min":0.73899999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.60799999999999998}]},{"osd":1,"last update":"Fri Feb 9 22:50:46 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56899999999999995,"15min":0.49199999999999999},"min":{"1min":0.438,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.82999999999999996,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.60599999999999998,"15min":0.504},"min":{"1min":0.45700000000000002,"5min":0.31900000000000001,"15min":0.30399999999999999},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.83699999999999997}]},{"osd":3,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.55900000000000005,"15min":0.49099999999999999},"min":{"1min":0.44700000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.92600000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.52600000000000002,"15min":0.48999999999999999},"min":{"1min":0.39900000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.80900000000000005,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.437}]},{"osd":4,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.65800000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.499,"15min":0.435},"min":{"1min":0.32700000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.45000000000000001},"min":{"1min":0.34999999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.71299999999999997,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48899999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.60499999999999998,"15min":0.58899999999999997},"min":{"1min":0.41599999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.874,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.40000000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.58899999999999997}]},{"osd":7,"last update":"Fri Feb 9 22:48:39 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.59999999999999998,"15min":0.60299999999999998},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64900000000000002,"15min":0.65000000000000002},"min":{"1min":0.48399999999999999,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.85899999999999999,"5min":0.874,"15min":0.874},"last":0.59999999999999998}]},{"osd":8,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.56699999999999995,"15min":0.54500000000000004},"min":{"1min":0.38300000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.90700000000000003,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.53200000000000003,"15min":0.51600000000000001},"min":{"1min":0.51900000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.77000000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.57299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:30 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.65200000000000002,"15min":0.65000000000000002},"min":{"1min":0.48699999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.64900000000000002,"15min":0.64100000000000001},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.624}]},{"osd":10,"last update":"Fri Feb 9 22:49:54 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.65900000000000003,"15min":0.64700000000000002},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.66400000000000003,"15min":0.65300000000000002},"min":{"1min":0.54400000000000004,"5min":0.44,"15min":0.44},"max":{"1min":0.88200000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"last":0.751}]},{"osd":11,"last update":"Fri Feb 9 22:50:07 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.872},{"interface":"front","average":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"min":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.67700000000000005}]}]},{"osd":3,"up_from":25,"seq":107374182476,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32140,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713012,"statfs":{"total":95995035648,"available":95962124288,"internally_reserved":0,"allocated":987136,"data_stored":462934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.371,"15min":0.36399999999999999},"min":{"1min":0.26900000000000002,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.57899999999999996,"5min":0.62,"15min":0.62},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.40200000000000002,"15min":0.39800000000000002},"min":{"1min":0.27900000000000003,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.64300000000000002,"5min":0.66700000000000004,"15min":0.66700000000000004},"last":0.32800000000000001}]},{"osd":1,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57499999999999996,"15min":0.53100000000000003},"min":{"1min":0.44900000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.78100000000000003,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.58299999999999996,"15min":0.53500000000000003},"min":{"1min":0.47299999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.753,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.746}]},{"osd":2,"last update":"Fri Feb 9 22:50:01 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.56599999999999995,"15min":0.55700000000000005},"min":{"1min":0.36399999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72299999999999998,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.434},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.58599999999999997,"15min":0.55400000000000005},"min":{"1min":0.39300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.52300000000000002}]},{"osd":4,"last update":"Fri Feb 9 22:50:33 2024","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.621,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.372,"15min":0.372},"max":{"1min":0.83599999999999997,"5min":1.1539999999999999,"15min":1.1539999999999999},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.73299999999999998,"15min":0.627},"min":{"1min":0.378,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":6.6920000000000002,"5min":6.6920000000000002,"15min":6.6920000000000002},"last":0.66100000000000003}]},{"osd":5,"last update":"Fri Feb 9 22:50:51 2024","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.60599999999999998,"15min":0.62},"min":{"1min":0.45000000000000001,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.86299999999999999,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.505},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.59399999999999997,"15min":0.60999999999999999},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.76400000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.49199999999999999}]},{"osd":6,"last update":"Fri Feb 9 22:50:15 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.5,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.747,"5min":0.873,"15min":0.873},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.56599999999999995,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.82799999999999996,"5min":0.97699999999999998,"15min":0.97699999999999998},"last":0.63800000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:43 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.46999999999999997,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.745,"15min":0.70699999999999996},"min":{"1min":0.52700000000000002,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.877,"5min":5.8789999999999996,"15min":5.8789999999999996},"last":0.69599999999999995}]},{"osd":8,"last update":"Fri Feb 9 22:49:00 2024","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.54500000000000004,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.745,"5min":1.335,"15min":1.4330000000000001},"min":{"1min":0.57699999999999996,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.94199999999999995,"5min":14.837,"15min":14.837},"last":0.97999999999999998}]},{"osd":9,"last update":"Fri Feb 9 22:50:27 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.69699999999999995,"15min":0.75900000000000001},"min":{"1min":0.378,"5min":0.374,"15min":0.374},"max":{"1min":0.77500000000000002,"5min":0.98599999999999999,"15min":0.98599999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.74199999999999999,"15min":0.79100000000000004},"min":{"1min":0.442,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.88400000000000001,"5min":1.0369999999999999,"15min":1.0369999999999999},"last":0.55900000000000005}]},{"osd":10,"last update":"Fri Feb 9 22:50:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.79500000000000004,"15min":0.82999999999999996},"min":{"1min":0.58299999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":0.92500000000000004,"5min":1.26,"15min":1.26},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.72799999999999998,"15min":0.74399999999999999},"min":{"1min":0.51400000000000001,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":0.94199999999999995,"5min":0.999,"15min":0.999},"last":0.65300000000000002}]},{"osd":11,"last update":"Fri Feb 9 22:50:06 2024","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"last":0.624},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.72899999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":462970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:48:28 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.78400000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.53300000000000003,"15min":0.52800000000000002},"min":{"1min":0.42899999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.625}]},{"osd":1,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.433,"15min":0.40300000000000002},"min":{"1min":0.34200000000000003,"5min":0.255,"15min":0.255},"max":{"1min":0.63100000000000001,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46000000000000002,"15min":0.40300000000000002},"min":{"1min":0.38400000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.71499999999999997,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.443}]},{"osd":2,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.61799999999999999,"15min":0.57299999999999995},"min":{"1min":0.45200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.01,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.874},{"interface":"front","average":{"1min":0.71199999999999997,"5min":0.63,"15min":0.57899999999999996},"min":{"1min":0.497,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.871,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.76200000000000001}]},{"osd":3,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.626,"15min":0.58199999999999996},"min":{"1min":0.42999999999999999,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.57999999999999996,"15min":0.56699999999999995},"min":{"1min":0.47599999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80400000000000005,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.73799999999999999}]},{"osd":5,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.63800000000000001,"15min":0.58599999999999997},"min":{"1min":0.44800000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.626},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.67700000000000005,"15min":0.60399999999999998},"min":{"1min":0.55800000000000005,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.78200000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:20 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.60199999999999998},"min":{"1min":0.39800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.0149999999999999,"5min":1.056,"15min":1.056},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65500000000000003,"15min":0.63300000000000001},"min":{"1min":0.42999999999999999,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.93400000000000005,"5min":1.2809999999999999,"15min":1.2809999999999999},"last":0.53100000000000003}]},{"osd":7,"last update":"Fri Feb 9 22:50:40 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.66300000000000003,"15min":0.65300000000000002},"min":{"1min":0.55200000000000005,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.88600000000000001,"5min":1.097,"15min":1.097},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.65200000000000002,"15min":0.63200000000000001},"min":{"1min":0.496,"5min":0.434,"15min":0.434},"max":{"1min":0.88600000000000001,"5min":1.1220000000000001,"15min":1.1220000000000001},"last":0.55100000000000005}]},{"osd":8,"last update":"Fri Feb 9 22:50:02 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.72999999999999998,"15min":0.72599999999999998},"min":{"1min":0.61799999999999999,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.93300000000000005,"5min":1.129,"15min":1.129},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.76900000000000002,"15min":0.748},"min":{"1min":0.56499999999999995,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.98599999999999999,"5min":1.264,"15min":1.264},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Feb 9 22:50:35 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.80300000000000005,"15min":0.81100000000000005},"min":{"1min":0.50700000000000001,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.0169999999999999,"5min":1.3049999999999999,"15min":1.3049999999999999},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.77400000000000002,"15min":0.78000000000000003},"min":{"1min":0.60399999999999998,"5min":0.439,"15min":0.439},"max":{"1min":0.98599999999999999,"5min":1.196,"15min":1.196},"last":0.82999999999999996}]},{"osd":10,"last update":"Fri Feb 9 22:50:53 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83799999999999997,"15min":0.84099999999999997},"min":{"1min":0.61899999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.98499999999999999,"5min":1.179,"15min":1.179},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.76900000000000002,"15min":0.77000000000000002},"min":{"1min":0.624,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.99299999999999999,"5min":1.238,"15min":1.238},"last":0.64800000000000002}]},{"osd":11,"last update":"Fri Feb 9 22:50:05 2024","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.876,"5min":0.876,"15min":0.876},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.95299999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32732,"kb_used_data":1556,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712420,"statfs":{"total":95995035648,"available":95961518080,"internally_reserved":0,"allocated":1593344,"data_stored":1056107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.53800000000000003,"15min":0.498},"min":{"1min":0.438,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.66200000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.51900000000000002,"15min":0.48099999999999998},"min":{"1min":0.432,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.65700000000000003,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.55000000000000004}]},{"osd":1,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.56699999999999995,"15min":0.51600000000000001},"min":{"1min":0.45200000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.749},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.57199999999999995,"15min":0.52500000000000002},"min":{"1min":0.47699999999999998,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.60999999999999999}]},{"osd":2,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.43099999999999999,"15min":0.40300000000000002},"min":{"1min":0.38200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.72699999999999998,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.441,"15min":0.40500000000000003},"min":{"1min":0.374,"5min":0.217,"15min":0.217},"max":{"1min":0.82999999999999996,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.42499999999999999}]},{"osd":3,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.54500000000000004,"15min":0.51300000000000001},"min":{"1min":0.38800000000000001,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.496},{"interface":"front","average":{"1min":0.61799999999999999,"5min":0.57199999999999995,"15min":0.53400000000000003},"min":{"1min":0.45200000000000001,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.80300000000000005,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.58899999999999997}]},{"osd":4,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.70799999999999996,"5min":0.61499999999999999,"15min":0.55700000000000005},"min":{"1min":0.58799999999999997,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.86299999999999999,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.59799999999999998,"15min":0.55000000000000004},"min":{"1min":0.54200000000000004,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Feb 9 22:50:19 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.60899999999999999,"15min":0.59199999999999997},"min":{"1min":0.45100000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.85299999999999998,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.625},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.60599999999999998,"15min":0.58299999999999996},"min":{"1min":0.47499999999999998,"5min":0.315,"15min":0.315},"max":{"1min":0.81299999999999994,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.64000000000000001}]},{"osd":7,"last update":"Fri Feb 9 22:50:42 2024","interfaces":[{"interface":"back","average":{"1min":0.76200000000000001,"5min":0.67200000000000004,"15min":0.65300000000000002},"min":{"1min":0.65100000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":0.88400000000000001,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.67200000000000004,"15min":0.65900000000000003},"min":{"1min":0.63900000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90700000000000003,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.70499999999999996}]},{"osd":8,"last update":"Fri Feb 9 22:50:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.623,"15min":0.61799999999999999},"min":{"1min":0.438,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.872,"5min":0.996,"15min":0.996},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.61599999999999999,"15min":0.60399999999999998},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.69299999999999995}]},{"osd":9,"last update":"Fri Feb 9 22:50:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.72099999999999997,"15min":0.71499999999999997},"min":{"1min":0.63100000000000001,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":0.90800000000000003,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.71999999999999997,"15min":0.71899999999999997},"min":{"1min":0.58599999999999997,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.94699999999999995,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.78100000000000003}]},{"osd":10,"last update":"Fri Feb 9 22:50:58 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.75700000000000001,"15min":0.74099999999999999},"min":{"1min":0.73699999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.0049999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.83599999999999997,"5min":0.751,"15min":0.72999999999999998},"min":{"1min":0.69199999999999995,"5min":0.499,"15min":0.499},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.80300000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":61395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":62147,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1030,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3982,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":61687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-02-09T22:51:04.371 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 -- 172.21.15.3:0/614551099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900101d40 msgr2=0x7f1900104130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 --2- 172.21.15.3:0/614551099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900101d40 0x7f1900104130 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f18fc009900 tx=0x7f18fc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 -- 172.21.15.3:0/614551099 shutdown_connections 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 --2- 172.21.15.3:0/614551099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900104670 0x7f1900106a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 --2- 172.21.15.3:0/614551099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900106fa0 0x7f1900109390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 --2- 172.21.15.3:0/614551099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900101d40 0x7f1900104130 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 -- 172.21.15.3:0/614551099 >> 172.21.15.3:0/614551099 conn(0x7f19000fb6d0 msgr2=0x7f19000fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 -- 172.21.15.3:0/614551099 shutdown_connections 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.504+0000 7f1908348640 1 -- 172.21.15.3:0/614551099 wait complete. 2024-02-09T22:51:04.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 Processor -- start 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 -- start start 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900101d40 0x7f19001016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 0x7f19000ffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900106fa0 0x7f1900100230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900100990 con 0x7f1900104670 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1900100b00 con 0x7f1900106fa0 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f1908348640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19001a13d0 con 0x7f1900101d40 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 0x7f19000ffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 0x7f19000ffcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40218/0 (socket says 172.21.15.3:40218) 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 -- 172.21.15.3:0/1404445982 learned_addr learned my addr 172.21.15.3:0/1404445982 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19068e5640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900106fa0 0x7f1900100230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19060e4640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900101d40 0x7f19001016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 -- 172.21.15.3:0/1404445982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900101d40 msgr2=0x7f19001016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900101d40 0x7f19001016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 -- 172.21.15.3:0/1404445982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900106fa0 msgr2=0x7f1900100230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900106fa0 0x7f1900100230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19058e3640 1 -- 172.21.15.3:0/1404445982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18fc009560 con 0x7f1900104670 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19068e5640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900106fa0 0x7f1900100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.505+0000 7f19060e4640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900101d40 0x7f19001016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.506+0000 7f19058e3640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 0x7f19000ffcf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f18f000b480 tx=0x7f18f000b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.506+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f0010870 con 0x7f1900104670 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.506+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18f000f3c0 con 0x7f1900104670 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.506+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19001a1680 con 0x7f1900104670 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.506+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18f00175c0 con 0x7f1900104670 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.506+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19001a1c60 con 0x7f1900104670 2024-02-09T22:51:04.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.507+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19000fd1c0 con 0x7f1900104670 2024-02-09T22:51:04.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.508+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f18f000f8c0 con 0x7f1900104670 2024-02-09T22:51:04.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.509+0000 7f18f77fe640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f18ec0a2140 0x7f18ec0a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.509+0000 7f19060e4640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f18ec0a2140 0x7f18ec0a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.509+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f18f00cd690 con 0x7f1900104670 2024-02-09T22:51:04.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.509+0000 7f19060e4640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f18ec0a2140 0x7f18ec0a4600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f18fc036040 tx=0x7f18fc000bc0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:04.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.513+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f18f0096700 con 0x7f1900104670 2024-02-09T22:51:04.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.696+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f190010ccf0 con 0x7f1900104670 2024-02-09T22:51:04.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.697+0000 7f18f77fe640 1 -- 172.21.15.3:0/1404445982 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7f18f0099dc0 con 0x7f1900104670 2024-02-09T22:51:04.701 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:04.701 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f18ec0a2140 msgr2=0x7f18ec0a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f18ec0a2140 0x7f18ec0a4600 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f18fc036040 tx=0x7f18fc000bc0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 msgr2=0x7f19000ffcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 0x7f19000ffcf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f18f000b480 tx=0x7f18f000b940 comp rx=0 tx=0).stop 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 shutdown_connections 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f18ec0a2140 0x7f18ec0a4600 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1900106fa0 0x7f1900100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1900104670 0x7f19000ffcf0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 --2- 172.21.15.3:0/1404445982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1900101d40 0x7f19001016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 >> 172.21.15.3:0/1404445982 conn(0x7f19000fb6d0 msgr2=0x7f1900102800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.699+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 shutdown_connections 2024-02-09T22:51:04.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.700+0000 7f1908348640 1 -- 172.21.15.3:0/1404445982 wait complete. 2024-02-09T22:51:04.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T22:51:04.714 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- 172.21.15.3:0/2230823865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 msgr2=0x7fb490100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- 172.21.15.3:0/2230823865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490100570 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fb478009900 tx=0x7fb47802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- 172.21.15.3:0/2230823865 shutdown_connections 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- 172.21.15.3:0/2230823865 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb490101340 0x7fb4901017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- 172.21.15.3:0/2230823865 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb490101d00 0x7fb490108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- 172.21.15.3:0/2230823865 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490100570 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- 172.21.15.3:0/2230823865 >> 172.21.15.3:0/2230823865 conn(0x7fb4900fb6e0 msgr2=0x7fb4900fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- 172.21.15.3:0/2230823865 shutdown_connections 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- 172.21.15.3:0/2230823865 wait complete. 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 Processor -- start 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- start start 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490195160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb490101340 0x7fb4901956a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb490101d00 0x7fb49019c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49019cc50 con 0x7fb490100150 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49019cdc0 con 0x7fb490101340 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.817+0000 7fb495577640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb49019cf80 con 0x7fb490101d00 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490195160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490195160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40234/0 (socket says 172.21.15.3:40234) 2024-02-09T22:51:04.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 -- 172.21.15.3:0/3688606157 learned_addr learned my addr 172.21.15.3:0/3688606157 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48f7fe640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb490101d00 0x7fb49019c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48e7fc640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb490101340 0x7fb4901956a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 -- 172.21.15.3:0/3688606157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb490101d00 msgr2=0x7fb49019c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb490101d00 0x7fb49019c710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 -- 172.21.15.3:0/3688606157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb490101340 msgr2=0x7fb4901956a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb490101340 0x7fb4901956a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 -- 172.21.15.3:0/3688606157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb478009560 con 0x7fb490100150 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb48effd640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490195160 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb478000c00 tx=0x7fb478004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb478004df0 con 0x7fb490100150 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49019d200 con 0x7fb490100150 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb478005430 con 0x7fb490100150 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb47803e450 con 0x7fb490100150 2024-02-09T22:51:04.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.818+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb49019d790 con 0x7fb490100150 2024-02-09T22:51:04.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.819+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb490107470 con 0x7fb490100150 2024-02-09T22:51:04.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.821+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fb478003680 con 0x7fb490100150 2024-02-09T22:51:04.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.821+0000 7fb487fff640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb47c0a42d0 0x7fb47c0a6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:04.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.821+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fb4780f0200 con 0x7fb490100150 2024-02-09T22:51:04.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.821+0000 7fb48e7fc640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb47c0a42d0 0x7fb47c0a6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:04.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.822+0000 7fb48e7fc640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb47c0a42d0 0x7fb47c0a6790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb490196670 tx=0x7fb480008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:04.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:04.826+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4780b91e0 con 0x7fb490100150 2024-02-09T22:51:05.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.005+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb49002cdb0 con 0x7fb490100150 2024-02-09T22:51:05.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.006+0000 7fb487fff640 1 -- 172.21.15.3:0/3688606157 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb4780bc8a0 con 0x7fb490100150 2024-02-09T22:51:05.009 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T22:51:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb47c0a42d0 msgr2=0x7fb47c0a6790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb47c0a42d0 0x7fb47c0a6790 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fb490196670 tx=0x7fb480008040 comp rx=0 tx=0).stop 2024-02-09T22:51:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 msgr2=0x7fb490195160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490195160 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fb478000c00 tx=0x7fb478004270 comp rx=0 tx=0).stop 2024-02-09T22:51:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 shutdown_connections 2024-02-09T22:51:05.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fb47c0a42d0 0x7fb47c0a6790 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb490101340 0x7fb4901956a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb490101d00 0x7fb49019c710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 --2- 172.21.15.3:0/3688606157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb490100150 0x7fb490195160 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 >> 172.21.15.3:0/3688606157 conn(0x7fb4900fb6e0 msgr2=0x7fb490104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 shutdown_connections 2024-02-09T22:51:05.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.008+0000 7fb495577640 1 -- 172.21.15.3:0/3688606157 wait complete. 2024-02-09T22:51:05.022 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 -- 172.21.15.3:0/3561102217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 msgr2=0x7f514c102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 --2- 172.21.15.3:0/3561102217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c102a20 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5134009900 tx=0x7f513402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 -- 172.21.15.3:0/3561102217 shutdown_connections 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 --2- 172.21.15.3:0/3561102217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c102f60 0x7f514c1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 --2- 172.21.15.3:0/3561102217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c102a20 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 --2- 172.21.15.3:0/3561102217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f514c0ff530 0x7f514c0ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 -- 172.21.15.3:0/3561102217 >> 172.21.15.3:0/3561102217 conn(0x7f514c0fae40 msgr2=0x7f514c0fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 -- 172.21.15.3:0/3561102217 shutdown_connections 2024-02-09T22:51:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 -- 172.21.15.3:0/3561102217 wait complete. 2024-02-09T22:51:05.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 Processor -- start 2024-02-09T22:51:05.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.128+0000 7f51508c0640 1 -- start start 2024-02-09T22:51:05.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f514c0ff530 0x7f514c1972b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c102f60 0x7f514c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c19eda0 con 0x7f514c1025a0 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c19ef10 con 0x7f514c0ff530 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c19f0d0 con 0x7f514c102f60 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c1977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40264/0 (socket says 172.21.15.3:40264) 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 -- 172.21.15.3:0/2053095250 learned_addr learned my addr 172.21.15.3:0/2053095250 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f514ad9d640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c102f60 0x7f514c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f514a59c640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f514c0ff530 0x7f514c1972b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 -- 172.21.15.3:0/2053095250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c102f60 msgr2=0x7f514c19e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c102f60 0x7f514c19e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 -- 172.21.15.3:0/2053095250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f514c0ff530 msgr2=0x7f514c1972b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f514c0ff530 0x7f514c1972b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 -- 172.21.15.3:0/2053095250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5134009560 con 0x7f514c1025a0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f5143fff640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c1977f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f513402c6a0 tx=0x7f51340042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51340044e0 con 0x7f514c1025a0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51340054e0 con 0x7f514c1025a0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c19f350 con 0x7f514c1025a0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.129+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f513403e450 con 0x7f514c1025a0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.130+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c19f8b0 con 0x7f514c1025a0 2024-02-09T22:51:05.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.130+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c1086d0 con 0x7f514c1025a0 2024-02-09T22:51:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.132+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f5134003710 con 0x7f514c1025a0 2024-02-09T22:51:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.132+0000 7f51437fe640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51380a2140 0x7f51380a4600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.132+0000 7f514a59c640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51380a2140 0x7f51380a4600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.133+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f51340f09c0 con 0x7f514c1025a0 2024-02-09T22:51:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.133+0000 7f514a59c640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51380a2140 0x7f51380a4600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f513c005fd0 tx=0x7f513c00b410 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.136+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f51340b99a0 con 0x7f514c1025a0 2024-02-09T22:51:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.319+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f514c10cf10 con 0x7f514c1025a0 2024-02-09T22:51:05.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.319+0000 7f51437fe640 1 -- 172.21.15.3:0/2053095250 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f51340bd060 con 0x7f514c1025a0 2024-02-09T22:51:05.322 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T22:51:05.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51380a2140 msgr2=0x7f51380a4600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51380a2140 0x7f51380a4600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f513c005fd0 tx=0x7f513c00b410 comp rx=0 tx=0).stop 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 msgr2=0x7f514c1977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c1977f0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f513402c6a0 tx=0x7f51340042f0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 shutdown_connections 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f51380a2140 0x7f51380a4600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f514c102f60 0x7f514c19e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f514c1025a0 0x7f514c1977f0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.321+0000 7f51508c0640 1 --2- 172.21.15.3:0/2053095250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f514c0ff530 0x7f514c1972b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.322+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 >> 172.21.15.3:0/2053095250 conn(0x7f514c0fae40 msgr2=0x7f514c1058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.322+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 shutdown_connections 2024-02-09T22:51:05.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.322+0000 7f51508c0640 1 -- 172.21.15.3:0/2053095250 wait complete. 2024-02-09T22:51:05.334 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:05 smithi003 ceph-mon[32678]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:05 smithi003 ceph-mon[32678]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:05 smithi003 ceph-mon[32678]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:05 smithi003 ceph-mon[32678]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-02-09T22:51:05.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:05 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1404445982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:05.335 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:05 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3688606157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:05.336 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T22:51:05.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:05 smithi064 ceph-mon[36638]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:05 smithi064 ceph-mon[36638]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:05 smithi064 ceph-mon[36638]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:05 smithi064 ceph-mon[36638]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-02-09T22:51:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:05 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1404445982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:05.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:05 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3688606157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:05.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.438+0000 7f3e33dd0640 1 -- 172.21.15.3:0/3847715312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 msgr2=0x7f3e2c1028d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.438+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/3847715312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c1028d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009900 tx=0x7f3e2002bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- 172.21.15.3:0/3847715312 shutdown_connections 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/3847715312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e2c102e10 0x7f3e2c1096a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/3847715312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c1028d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/3847715312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e2c101260 0x7f3e2c101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- 172.21.15.3:0/3847715312 >> 172.21.15.3:0/3847715312 conn(0x7f3e2c0fc7f0 msgr2=0x7f3e2c0fec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- 172.21.15.3:0/3847715312 shutdown_connections 2024-02-09T22:51:05.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- 172.21.15.3:0/3847715312 wait complete. 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 Processor -- start 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- start start 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e2c101260 0x7f3e2c1972c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c197800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e2c102e10 0x7f3e2c19e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c19edb0 con 0x7f3e2c102450 2024-02-09T22:51:05.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c19ef20 con 0x7f3e2c101260 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e33dd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2c19f0e0 con 0x7f3e2c102e10 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.439+0000 7f3e3136b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c197800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c197800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40288/0 (socket says 172.21.15.3:40288) 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 -- 172.21.15.3:0/1560929107 learned_addr learned my addr 172.21.15.3:0/1560929107 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e31b6c640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e2c101260 0x7f3e2c1972c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3236d640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e2c102e10 0x7f3e2c19e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 -- 172.21.15.3:0/1560929107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e2c102e10 msgr2=0x7f3e2c19e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e2c102e10 0x7f3e2c19e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 -- 172.21.15.3:0/1560929107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e2c101260 msgr2=0x7f3e2c1972c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e2c101260 0x7f3e2c1972c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 -- 172.21.15.3:0/1560929107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e20009560 con 0x7f3e2c102450 2024-02-09T22:51:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e3136b640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c197800 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3e20006010 tx=0x7f3e200040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:05.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e200044c0 con 0x7f3e2c102450 2024-02-09T22:51:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e2c19f360 con 0x7f3e2c102450 2024-02-09T22:51:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e200052d0 con 0x7f3e2c102450 2024-02-09T22:51:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e20003e10 con 0x7f3e2c102450 2024-02-09T22:51:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.440+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2c19f870 con 0x7f3e2c102450 2024-02-09T22:51:05.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.441+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e2c108520 con 0x7f3e2c102450 2024-02-09T22:51:05.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.442+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f3e20005440 con 0x7f3e2c102450 2024-02-09T22:51:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.443+0000 7f3e2affd640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e140a20c0 0x7f3e140a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.443+0000 7f3e31b6c640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e140a20c0 0x7f3e140a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.443+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3e200f0560 con 0x7f3e2c102450 2024-02-09T22:51:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.443+0000 7f3e31b6c640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e140a20c0 0x7f3e140a4580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c1022b0 tx=0x7f3e24005f90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:05.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.446+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e200b9540 con 0x7f3e2c102450 2024-02-09T22:51:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:05 smithi089 ceph-mon[36665]: from='client.15411 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:05 smithi089 ceph-mon[36665]: from='client.15417 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:05 smithi089 ceph-mon[36665]: from='client.15423 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-02-09T22:51:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:05 smithi089 ceph-mon[36665]: pgmap v347: 97 pgs: 97 active+clean; 642 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-02-09T22:51:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:05 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1404445982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:05.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:05 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3688606157' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T22:51:05.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.630+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3e2c10cec0 con 0x7f3e2c102450 2024-02-09T22:51:05.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.630+0000 7f3e2affd640 1 -- 172.21.15.3:0/1560929107 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3e20043020 con 0x7f3e2c102450 2024-02-09T22:51:05.633 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T22:51:05.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e140a20c0 msgr2=0x7f3e140a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e140a20c0 0x7f3e140a4580 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c1022b0 tx=0x7f3e24005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 msgr2=0x7f3e2c197800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c197800 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f3e20006010 tx=0x7f3e200040c0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 shutdown_connections 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f3e140a20c0 0x7f3e140a4580 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e2c102e10 0x7f3e2c19e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e2c102450 0x7f3e2c197800 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 --2- 172.21.15.3:0/1560929107 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e2c101260 0x7f3e2c1972c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.632+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 >> 172.21.15.3:0/1560929107 conn(0x7f3e2c0fc7f0 msgr2=0x7f3e2c1056f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.633+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 shutdown_connections 2024-02-09T22:51:05.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.633+0000 7f3e33dd0640 1 -- 172.21.15.3:0/1560929107 wait complete. 2024-02-09T22:51:05.646 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 -- 172.21.15.3:0/4165365756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10941013c0 msgr2=0x7f1094101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 --2- 172.21.15.3:0/4165365756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10941013c0 0x7f1094101840 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f1088009900 tx=0x7f108802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 -- 172.21.15.3:0/4165365756 shutdown_connections 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 --2- 172.21.15.3:0/4165365756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1094101d80 0x7f1094108610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 --2- 172.21.15.3:0/4165365756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f10941013c0 0x7f1094101840 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 --2- 172.21.15.3:0/4165365756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941001d0 0x7f10941005f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 -- 172.21.15.3:0/4165365756 >> 172.21.15.3:0/4165365756 conn(0x7f10940fb760 msgr2=0x7f10940fdb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 -- 172.21.15.3:0/4165365756 shutdown_connections 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 -- 172.21.15.3:0/4165365756 wait complete. 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 Processor -- start 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.746+0000 7f109a8f1640 1 -- start start 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10941001d0 0x7f10941973a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941013c0 0x7f10941978e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 0x7f109419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109419ee90 con 0x7f1094101d80 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109419f000 con 0x7f10941013c0 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109419f1c0 con 0x7f10941001d0 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 0x7f109419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 0x7f109419e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40304/0 (socket says 172.21.15.3:40304) 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 -- 172.21.15.3:0/214999349 learned_addr learned my addr 172.21.15.3:0/214999349 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:05.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1093fff640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10941001d0 0x7f10941973a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 -- 172.21.15.3:0/214999349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10941001d0 msgr2=0x7f10941973a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10941001d0 0x7f10941973a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 -- 172.21.15.3:0/214999349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941013c0 msgr2=0x7f10941978e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f10937fe640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941013c0 0x7f10941978e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941013c0 0x7f10941978e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 -- 172.21.15.3:0/214999349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1088009560 con 0x7f1094101d80 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f10937fe640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941013c0 0x7f10941978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f1098e8e640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 0x7f109419e950 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f108c00eaa0 tx=0x7f108c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c00cdb0 con 0x7f1094101d80 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f109419f4a0 con 0x7f1094101d80 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f108c005020 con 0x7f1094101d80 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108c0116b0 con 0x7f1094101d80 2024-02-09T22:51:05.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.747+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f109419fa30 con 0x7f1094101d80 2024-02-09T22:51:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.748+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1094107490 con 0x7f1094101d80 2024-02-09T22:51:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.750+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f108c005520 con 0x7f1094101d80 2024-02-09T22:51:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.750+0000 7f10917fa640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f107c0a20c0 0x7f107c0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.750+0000 7f1093fff640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f107c0a20c0 0x7f107c0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:05.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.750+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f108c0ce330 con 0x7f1094101d80 2024-02-09T22:51:05.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.751+0000 7f1093fff640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f107c0a20c0 0x7f107c0a4580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1094101220 tx=0x7f1084008040 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:05.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.754+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f108c097310 con 0x7f1094101d80 2024-02-09T22:51:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.972+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f109410ccf0 con 0x7f1094101d80 2024-02-09T22:51:05.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.973+0000 7f10917fa640 1 -- 172.21.15.3:0/214999349 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f108c01f020 con 0x7f1094101d80 2024-02-09T22:51:05.976 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:05.976 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:51:00.398692+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:51:05.976 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2024-02-09T22:51:05.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.974+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f107c0a20c0 msgr2=0x7f107c0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.974+0000 7f109a8f1640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f107c0a20c0 0x7f107c0a4580 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f1094101220 tx=0x7f1084008040 comp rx=0 tx=0).stop 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.974+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 msgr2=0x7f109419e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.974+0000 7f109a8f1640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 0x7f109419e950 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f108c00eaa0 tx=0x7f108c00ef60 comp rx=0 tx=0).stop 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 shutdown_connections 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f107c0a20c0 0x7f107c0a4580 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f10941013c0 0x7f10941978e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1094101d80 0x7f109419e950 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 --2- 172.21.15.3:0/214999349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f10941001d0 0x7f10941973a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 >> 172.21.15.3:0/214999349 conn(0x7f10940fb760 msgr2=0x7f1094104660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 shutdown_connections 2024-02-09T22:51:05.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:05.975+0000 7f109a8f1640 1 -- 172.21.15.3:0/214999349 wait complete. 2024-02-09T22:51:05.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 -- 172.21.15.3:0/2077444678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94101d10 msgr2=0x7fab941085a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 --2- 172.21.15.3:0/2077444678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94101d10 0x7fab941085a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fab8c009ca0 tx=0x7fab8c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 -- 172.21.15.3:0/2077444678 shutdown_connections 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 --2- 172.21.15.3:0/2077444678 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94101d10 0x7fab941085a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 --2- 172.21.15.3:0/2077444678 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 0x7fab941017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 --2- 172.21.15.3:0/2077444678 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94100160 0x7fab94100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 -- 172.21.15.3:0/2077444678 >> 172.21.15.3:0/2077444678 conn(0x7fab940fb6d0 msgr2=0x7fab940fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 -- 172.21.15.3:0/2077444678 shutdown_connections 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.088+0000 7fab9a78a640 1 -- 172.21.15.3:0/2077444678 wait complete. 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 Processor -- start 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 -- start start 2024-02-09T22:51:06.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 0x7fab94199380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 0x7fab941998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94101d10 0x7fab941a0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a0e70 con 0x7fab94100160 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a0fe0 con 0x7fab94101d10 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab9a78a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab941a11a0 con 0x7fab94101350 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab93fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 0x7fab94199380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab93fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 0x7fab94199380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40324/0 (socket says 172.21.15.3:40324) 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab93fff640 1 -- 172.21.15.3:0/3818376288 learned_addr learned my addr 172.21.15.3:0/3818376288 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.089+0000 7fab98d27640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94101d10 0x7fab941a0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab937fe640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 0x7fab941998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab93fff640 1 -- 172.21.15.3:0/3818376288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 msgr2=0x7fab941998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab93fff640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 0x7fab941998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab93fff640 1 -- 172.21.15.3:0/3818376288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94101d10 msgr2=0x7fab941a0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab93fff640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94101d10 0x7fab941a0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab93fff640 1 -- 172.21.15.3:0/3818376288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c009900 con 0x7fab94100160 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab98d27640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94101d10 0x7fab941a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab937fe640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 0x7fab941998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab93fff640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 0x7fab94199380 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00d770 tx=0x7fab7c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c009c30 con 0x7fab94100160 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab941a1480 con 0x7fab94100160 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab7c00de80 con 0x7fab94100160 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c010460 con 0x7fab94100160 2024-02-09T22:51:06.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.090+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab941a19e0 con 0x7fab94100160 2024-02-09T22:51:06.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.091+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9410bb70 con 0x7fab94100160 2024-02-09T22:51:06.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.092+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7fab7c003f60 con 0x7fab94100160 2024-02-09T22:51:06.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.093+0000 7fab917fa640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab800a20c0 0x7fab800a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.093+0000 7fab937fe640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab800a20c0 0x7fab800a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.093+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7fab7c0cdaf0 con 0x7fab94100160 2024-02-09T22:51:06.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.093+0000 7fab937fe640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab800a20c0 0x7fab800a4580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fab9419a890 tx=0x7fab84005d90 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.097+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fab7c096a70 con 0x7fab94100160 2024-02-09T22:51:06.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.284+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab9410bd70 con 0x7fab94100160 2024-02-09T22:51:06.289 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.286+0000 7fab917fa640 1 -- 172.21.15.3:0/3818376288 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22460 (secure 0 0 0) 0x7fab7c09a130 con 0x7fab94100160 2024-02-09T22:51:06.289 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:06.289 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":88,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T22:50:08.260275+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab800a20c0 msgr2=0x7fab800a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab800a20c0 0x7fab800a4580 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fab9419a890 tx=0x7fab84005d90 comp rx=0 tx=0).stop 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 msgr2=0x7fab94199380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 0x7fab94199380 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fab7c00d770 tx=0x7fab7c00dc30 comp rx=0 tx=0).stop 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 shutdown_connections 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7fab800a20c0 0x7fab800a4580 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab94101d10 0x7fab941a0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab94101350 0x7fab941998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 --2- 172.21.15.3:0/3818376288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab94100160 0x7fab94199380 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 >> 172.21.15.3:0/3818376288 conn(0x7fab940fb6d0 msgr2=0x7fab940fd060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 shutdown_connections 2024-02-09T22:51:06.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.288+0000 7fab9a78a640 1 -- 172.21.15.3:0/3818376288 wait complete. 2024-02-09T22:51:06.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:06 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2053095250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:06.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:06 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1560929107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:06.301 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:06 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/214999349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:06.302 DEBUG:tasks.fwd_scrub:Yielding 2024-02-09T22:51:06.303 INFO:teuthology.run_tasks:Running task exec... 2024-02-09T22:51:06.312 INFO:teuthology.task.exec:Executing custom commands... 2024-02-09T22:51:06.312 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T22:51:06.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-02-09T22:51:06.313 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-02-09T22:51:06.313 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:51:06.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:51:06.313 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:51:06.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 -- 172.21.15.3:0/4197129573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4100160 msgr2=0x7f5ac4100580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/4197129573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4100160 0x7f5ac4100580 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5abc009900 tx=0x7f5abc02bc50 comp rx=0 tx=0).stop 2024-02-09T22:51:06.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 -- 172.21.15.3:0/4197129573 shutdown_connections 2024-02-09T22:51:06.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/4197129573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ac4101d10 0x7f5ac41085a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/4197129573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4101350 0x7f5ac41017d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/4197129573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4100160 0x7f5ac4100580 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 -- 172.21.15.3:0/4197129573 >> 172.21.15.3:0/4197129573 conn(0x7f5ac40fb6d0 msgr2=0x7f5ac40fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 -- 172.21.15.3:0/4197129573 shutdown_connections 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.414+0000 7f5ac8b35640 1 -- 172.21.15.3:0/4197129573 wait complete. 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 Processor -- start 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 -- start start 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 0x7f5ac41972e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4101350 0x7f5ac4197820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ac4101d10 0x7f5ac419e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac419ed80 con 0x7f5ac4101350 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac419eef0 con 0x7f5ac4101d10 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac8b35640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac419f0b0 con 0x7f5ac4100160 2024-02-09T22:51:06.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 0x7f5ac41972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac1d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4101350 0x7f5ac4197820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 0x7f5ac41972e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53360/0 (socket says 172.21.15.3:53360) 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac259c640 1 -- 172.21.15.3:0/73339296 learned_addr learned my addr 172.21.15.3:0/73339296 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac2d9d640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ac4101d10 0x7f5ac419e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.415+0000 7f5ac259c640 1 -- 172.21.15.3:0/73339296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ac4101d10 msgr2=0x7f5ac419e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac259c640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ac4101d10 0x7f5ac419e840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac259c640 1 -- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4101350 msgr2=0x7f5ac4197820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac259c640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4101350 0x7f5ac4197820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac259c640 1 -- 172.21.15.3:0/73339296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc009560 con 0x7f5ac4100160 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac1d9b640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4101350 0x7f5ac4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac259c640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 0x7f5ac41972e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5abc036040 tx=0x7f5abc0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0044e0 con 0x7f5ac4100160 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac419f330 con 0x7f5ac4100160 2024-02-09T22:51:06.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5abc0054e0 con 0x7f5ac4100160 2024-02-09T22:51:06.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc03e450 con 0x7f5ac4100160 2024-02-09T22:51:06.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.416+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac419f890 con 0x7f5ac4100160 2024-02-09T22:51:06.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 -- 172.21.15.3:0/1086884476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 msgr2=0x7f33f41028a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 --2- 172.21.15.3:0/1086884476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f41028a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009900 tx=0x7f33e402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:06.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 -- 172.21.15.3:0/1086884476 shutdown_connections 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 --2- 172.21.15.3:0/1086884476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33f4102de0 0x7f33f4109670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 --2- 172.21.15.3:0/1086884476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f41028a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 --2- 172.21.15.3:0/1086884476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33f4101230 0x7f33f4101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 -- 172.21.15.3:0/1086884476 >> 172.21.15.3:0/1086884476 conn(0x7f33f40fc760 msgr2=0x7f33f40febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 -- 172.21.15.3:0/1086884476 shutdown_connections 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f33fc156640 1 -- 172.21.15.3:0/1086884476 wait complete. 2024-02-09T22:51:06.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.417+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac410bb30 con 0x7f5ac4100160 2024-02-09T22:51:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 Processor -- start 2024-02-09T22:51:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 -- start start 2024-02-09T22:51:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33f4101230 0x7f33f41972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f4197810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33f4102de0 0x7f33f419e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f419edc0 con 0x7f33f4102420 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f419ef30 con 0x7f33f4102de0 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fc156640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33f419f0f0 con 0x7f33f4101230 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f4197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f4197810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40362/0 (socket says 172.21.15.3:40362) 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 -- 172.21.15.3:0/1930014968 learned_addr learned my addr 172.21.15.3:0/1930014968 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33f4101230 0x7f33f41972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33f4102de0 0x7f33f419e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 -- 172.21.15.3:0/1930014968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33f4101230 msgr2=0x7f33f41972d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33f4101230 0x7f33f41972d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 -- 172.21.15.3:0/1930014968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33f4102de0 msgr2=0x7f33f419e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33f4102de0 0x7f33f419e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 -- 172.21.15.3:0/1930014968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e4009560 con 0x7f33f4102420 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.418+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f4197810 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006010 tx=0x7f33e40040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f33eaffd640 1 -- 172.21.15.3:0/1930014968 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e40044c0 con 0x7f33f4102420 2024-02-09T22:51:06.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33f419f370 con 0x7f33f4102420 2024-02-09T22:51:06.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f33eaffd640 1 -- 172.21.15.3:0/1930014968 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33e40052d0 con 0x7f33f4102420 2024-02-09T22:51:06.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f33eaffd640 1 -- 172.21.15.3:0/1930014968 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e4003e10 con 0x7f33f4102420 2024-02-09T22:51:06.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33f419f8d0 con 0x7f33f4102420 2024-02-09T22:51:06.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f5abc003680 con 0x7f5ac4100160 2024-02-09T22:51:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.419+0000 7f5ab37fe640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5aac0a20b0 0x7f5aac0a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.420+0000 7f5ac1d9b640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5aac0a20b0 0x7f5aac0a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.420+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f5abc0f02a0 con 0x7f5ac4100160 2024-02-09T22:51:06.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.420+0000 7f5ac1d9b640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5aac0a20b0 0x7f5aac0a4570 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ac41987a0 tx=0x7f5ab800a2d0 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.421+0000 7f33eaffd640 1 -- 172.21.15.3:0/1930014968 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f33e4005440 con 0x7f33f4102420 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.421+0000 7f33fee08740 1 Processor -- start 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.421+0000 7f33fee08740 1 -- start start 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fee08740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2e1b0 0x556ee5f2e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fee08740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f336a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fee08740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f33d10 0x556ee5f361a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fee08740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5e9d000 con 0x556ee5f2eba0 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fee08740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5e9a530 con 0x556ee5f33d10 2024-02-09T22:51:06.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fee08740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5e95d10 con 0x556ee5f2e1b0 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33eaffd640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33e00a20c0 0x7f33e00a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33f9ef2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f336a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33f9ef2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f336a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40364/0 (socket says 172.21.15.3:40364) 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33f9ef2640 1 -- 172.21.15.3:0/3868052474 learned_addr learned my addr 172.21.15.3:0/3868052474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2e1b0 0x556ee5f2e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33e00a20c0 0x7f33e00a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33eaffd640 1 -- 172.21.15.3:0/1930014968 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f33e40eff90 con 0x7f33f4102420 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33f96f1640 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f33d10 0x556ee5f361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 -- 172.21.15.3:0/3868052474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f33d10 msgr2=0x556ee5f361a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f33d10 0x556ee5f361a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 -- 172.21.15.3:0/3868052474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 msgr2=0x556ee5f336a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f336a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 -- 172.21.15.3:0/3868052474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ee5e3c210 con 0x556ee5f2e1b0 2024-02-09T22:51:06.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33f96f1640 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f33d10 0x556ee5f361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:06.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33e00a20c0 0x7f33e00a4580 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f33ec011cb0 tx=0x7f33ec016000 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.422+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2e1b0 0x556ee5f2e5d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f33ec014e30 tx=0x7f33ec0117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fce10fb1c41a74fb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33dffff640 1 -- 172.21.15.3:0/3868052474 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec024070 con 0x556ee5f2e1b0 2024-02-09T22:51:06.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f5abc0b9220 con 0x7f5ac4100160 2024-02-09T22:51:06.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:06 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2053095250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:06.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:06 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1560929107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:06.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:06 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/214999349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33dffff640 1 -- 172.21.15.3:0/3868052474 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33ec01a040 con 0x556ee5f2e1b0 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33dffff640 1 -- 172.21.15.3:0/3868052474 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec014360 con 0x556ee5f2e1b0 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 -- 172.21.15.3:0/3868052474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2e1b0 msgr2=0x556ee5f2e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2e1b0 0x556ee5f2e5d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f33ec014e30 tx=0x7f33ec0117b0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 -- 172.21.15.3:0/3868052474 shutdown_connections 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f33d10 0x556ee5f361a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f336a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 --2- 172.21.15.3:0/3868052474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2e1b0 0x556ee5f2e5d0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 -- 172.21.15.3:0/3868052474 >> 172.21.15.3:0/3868052474 conn(0x556ee5f29720 msgr2=0x556ee5f29b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.423+0000 7f33fee08740 1 -- 172.21.15.3:0/3868052474 shutdown_connections 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 -- 172.21.15.3:0/3868052474 wait complete. 2024-02-09T22:51:06.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 -- start start 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 Processor -- start 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 0x556ee5f38360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2eba0 0x556ee5f388a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5e3ae60 con 0x556ee5f2e1b0 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5b694b0 con 0x556ee5f2eba0 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5e1b670 con 0x556ee5f33d10 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fee08740 10 client.0 _ll_register_callbacks cb 0x556ee5e33190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33f9ef2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 0x556ee5f38360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33f9ef2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 0x556ee5f38360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40376/0 (socket says 172.21.15.3:40376) 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33f9ef2640 1 -- 172.21.15.3:0/1802217257 learned_addr learned my addr 172.21.15.3:0/1802217257 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.424+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2eba0 0x556ee5f388a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f9ef2640 1 -- 172.21.15.3:0/1802217257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 msgr2=0x556ee5f3f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f9ef2640 1 -- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2eba0 msgr2=0x556ee5f388a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2eba0 0x556ee5f388a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f9ef2640 1 -- 172.21.15.3:0/1802217257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec014a10 con 0x556ee5f2e1b0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2eba0 0x556ee5f388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 0x556ee5f38360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f33f0012700 tx=0x7f33f0012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ed6a3c5dfdb632c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33dd7fa640 1 -- 172.21.15.3:0/1802217257 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0017070 con 0x556ee5f2e1b0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556ee5b95860 con 0x556ee5f2e1b0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33dd7fa640 1 -- 172.21.15.3:0/1802217257 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f33f0004050 con 0x556ee5f2e1b0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33dd7fa640 1 -- 172.21.15.3:0/1802217257 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f0004210 con 0x556ee5f2e1b0 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33fee08740 10 client.15477 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:06.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556ee5e94390 con 0x556ee5f2e1b0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.425+0000 7f33dd7fa640 1 -- 172.21.15.3:0/1802217257 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556ee5b95860 con 0x556ee5f2e1b0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33dd7fa640 1 -- 172.21.15.3:0/1802217257 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f33f0025070 con 0x556ee5f2e1b0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33fee08740 10 client.15477 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33fee08740 10 client.15477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33fee08740 20 client.15477 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33fee08740 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f441f0 0x556ee5f46610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33fee08740 4 client.15477 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ee5be6410 con 0x556ee5f441f0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f441f0 0x556ee5f46610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33dd7fa640 10 client.15477 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.426+0000 7f33f96f1640 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f441f0 0x556ee5f46610 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6147273707a426ed server_cookie=1acdfd7ca8429a48 in_seq=0 out_seq=0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33dd7fa640 1 -- 172.21.15.3:0/1802217257 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556ee5be6410 con 0x556ee5f441f0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33dd7fa640 10 client.15477 handle_command_reply: tid=0 2024-02-09T22:51:06.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 client.15477 shutdown 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 msgr2=0x556ee5f38360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 0x556ee5f38360 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f33f0012700 tx=0x7f33f0012bc0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 shutdown_connections 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f441f0 0x556ee5f46610 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2eba0 0x556ee5f388a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33dd7fa640 0 client.15477 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 --2- 172.21.15.3:0/1802217257 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2e1b0 0x556ee5f38360 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 >> 172.21.15.3:0/1802217257 conn(0x556ee5f29720 msgr2=0x556ee5f28920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 shutdown_connections 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.427+0000 7f33fee08740 1 -- 172.21.15.3:0/1802217257 wait complete. 2024-02-09T22:51:06.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.428+0000 7f33fee08740 20 client.15477 trim_cache size 0 max 16384 2024-02-09T22:51:06.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.442+0000 7f33fee08740 1 Processor -- start 2024-02-09T22:51:06.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 -- start start 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f28090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2eba0 0x556ee5f3f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 0x556ee5f285d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5be6410 con 0x556ee5f33d10 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5b95860 con 0x556ee5f2e1b0 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fee08740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5f2a690 con 0x556ee5f2eba0 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 0x556ee5f285d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 0x556ee5f285d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40380/0 (socket says 172.21.15.3:40380) 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 -- 172.21.15.3:0/207446240 learned_addr learned my addr 172.21.15.3:0/207446240 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33f96f1640 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2eba0 0x556ee5f3f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f28090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 -- 172.21.15.3:0/207446240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2eba0 msgr2=0x556ee5f3f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2eba0 0x556ee5f3f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 -- 172.21.15.3:0/207446240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 msgr2=0x556ee5f28090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f28090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 -- 172.21.15.3:0/207446240 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ee5e30c70 con 0x556ee5f33d10 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.443+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 0x556ee5f285d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f33ec005fd0 tx=0x7f33ec00a4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbff4b249b7e87e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33df7fe640 1 -- 172.21.15.3:0/207446240 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec024040 con 0x556ee5f33d10 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33df7fe640 1 -- 172.21.15.3:0/207446240 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33ec014810 con 0x556ee5f33d10 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33df7fe640 1 -- 172.21.15.3:0/207446240 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec0040c0 con 0x556ee5f33d10 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 -- 172.21.15.3:0/207446240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 msgr2=0x556ee5f285d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 0x556ee5f285d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f33ec005fd0 tx=0x7f33ec00a4f0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 -- 172.21.15.3:0/207446240 shutdown_connections 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f33d10 0x556ee5f285d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f2eba0 0x556ee5f3f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 --2- 172.21.15.3:0/207446240 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f28090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 -- 172.21.15.3:0/207446240 >> 172.21.15.3:0/207446240 conn(0x556ee5f29720 msgr2=0x556ee5f2f5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 -- 172.21.15.3:0/207446240 shutdown_connections 2024-02-09T22:51:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.444+0000 7f33fee08740 1 -- 172.21.15.3:0/207446240 wait complete. 2024-02-09T22:51:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 -- start start 2024-02-09T22:51:06.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 Processor -- start 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f3dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f3e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3ea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5f3fc20 con 0x556ee5f2eba0 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5f9a720 con 0x556ee5f2e1b0 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ee5f9a860 con 0x556ee5f33d10 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fee08740 10 client.0 _ll_register_callbacks cb 0x7f33ec00ae70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33f96f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f3e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33f96f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f3e510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40388/0 (socket says 172.21.15.3:40388) 2024-02-09T22:51:06.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33f96f1640 1 -- 172.21.15.3:0/2728093976 learned_addr learned my addr 172.21.15.3:0/2728093976 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f3dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.445+0000 7f33fa6f3640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33f96f1640 1 -- 172.21.15.3:0/2728093976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 msgr2=0x556ee5f3ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33f96f1640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33f96f1640 1 -- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 msgr2=0x556ee5f3dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33f96f1640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f3dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33f96f1640 1 -- 172.21.15.3:0/2728093976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ee5e3c210 con 0x556ee5f2eba0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33f96f1640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f3e510 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009340 tx=0x7f33e40bd490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4748d758e141d8cc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33dd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33dd7fa640 1 -- 172.21.15.3:0/2728093976 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33e40c5470 con 0x556ee5f2eba0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33ec01a040 con 0x556ee5f2eba0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33dd7fa640 1 -- 172.21.15.3:0/2728093976 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f33e40035f0 con 0x556ee5f2eba0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33dd7fa640 1 -- 172.21.15.3:0/2728093976 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33e40db1e0 con 0x556ee5f2eba0 2024-02-09T22:51:06.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 10 client.15483 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x556ee5e56ac0 con 0x556ee5f2eba0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33dd7fa640 1 -- 172.21.15.3:0/2728093976 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f33e409f080 con 0x556ee5f2eba0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33dd7fa640 1 -- 172.21.15.3:0/2728093976 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f33e40d0d20 con 0x556ee5f2eba0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 10 client.15483 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 10 client.15483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 20 client.15483 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f9c0a0 0x556ee5f9e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 4 client.15483 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.446+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556ee5bd27b0 con 0x556ee5f9c0a0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.447+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f9c0a0 0x556ee5f9e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.447+0000 7f33dd7fa640 10 client.15483 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.447+0000 7f33f9ef2640 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f9c0a0 0x556ee5f9e4c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38a9f84d5b322c53 server_cookie=f7e3a62aa5d9d9d7 in_seq=0 out_seq=0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33dd7fa640 1 -- 172.21.15.3:0/2728093976 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556ee5bd27b0 con 0x556ee5f9c0a0 2024-02-09T22:51:06.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33dd7fa640 10 client.15483 handle_command_reply: tid=0 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 client.15483 shutdown 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 msgr2=0x556ee5f3e510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f3e510 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f33e4009340 tx=0x7f33e40bd490 comp rx=0 tx=0).stop 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 shutdown_connections 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556ee5f9c0a0 0x556ee5f9e4c0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556ee5f33d10 0x556ee5f3ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556ee5f2eba0 0x556ee5f3e510 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33dd7fa640 0 client.15483 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 --2- 172.21.15.3:0/2728093976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556ee5f2e1b0 0x556ee5f3dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 >> 172.21.15.3:0/2728093976 conn(0x556ee5f29720 msgr2=0x556ee5f2c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 shutdown_connections 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.448+0000 7f33fee08740 1 -- 172.21.15.3:0/2728093976 wait complete. 2024-02-09T22:51:06.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fee08740 20 client.15483 trim_cache size 0 max 16384 2024-02-09T22:51:06.457 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:51:06.457 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:51:06.457 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "d9d10ee7-c3f1-4494-b951-4f7c0b5107e8", 2024-02-09T22:51:06.457 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:51:06.457 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33e00a20c0 msgr2=0x7f33e00a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33e00a20c0 0x7f33e00a4580 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f33ec011cb0 tx=0x7f33ec016000 comp rx=0 tx=0).stop 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 msgr2=0x7f33f4197810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f4197810 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f33e4006010 tx=0x7f33e40040c0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 shutdown_connections 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33e00a20c0 0x7f33e00a4580 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33f4102de0 0x7f33f419e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f33f4102420 0x7f33f4197810 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 --2- 172.21.15.3:0/1930014968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f33f4101230 0x7f33f41972d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 >> 172.21.15.3:0/1930014968 conn(0x7f33f40fc760 msgr2=0x7f33f41057c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 shutdown_connections 2024-02-09T22:51:06.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.449+0000 7f33fc156640 1 -- 172.21.15.3:0/1930014968 wait complete. 2024-02-09T22:51:06.467 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:51:06.467 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:51:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:06 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2053095250' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T22:51:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:06 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1560929107' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T22:51:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:06 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/214999349' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:06.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 -- 172.21.15.3:0/2459547962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101340 msgr2=0x7f0f141017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 --2- 172.21.15.3:0/2459547962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101340 0x7f0f141017c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009900 tx=0x7f0f0802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 -- 172.21.15.3:0/2459547962 shutdown_connections 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 --2- 172.21.15.3:0/2459547962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f14101d00 0x7f0f14108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 --2- 172.21.15.3:0/2459547962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101340 0x7f0f141017c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 --2- 172.21.15.3:0/2459547962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f14100150 0x7f0f14100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 -- 172.21.15.3:0/2459547962 >> 172.21.15.3:0/2459547962 conn(0x7f0f140fb6c0 msgr2=0x7f0f140fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 -- 172.21.15.3:0/2459547962 shutdown_connections 2024-02-09T22:51:06.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.573+0000 7f0f19b47640 1 -- 172.21.15.3:0/2459547962 wait complete. 2024-02-09T22:51:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 Processor -- start 2024-02-09T22:51:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 -- start start 2024-02-09T22:51:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f14100150 0x7f0f14197260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f14101340 0x7f0f141977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 0x7f0f1419e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1419ed50 con 0x7f0f14101d00 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1419eec0 con 0x7f0f14101340 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.574+0000 7f0f19b47640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1419f080 con 0x7f0f14100150 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 0x7f0f1419e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 0x7f0f1419e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39870/0 (socket says 172.21.15.3:39870) 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 -- 172.21.15.3:0/3656462065 learned_addr learned my addr 172.21.15.3:0/3656462065 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f14101340 0x7f0f141977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f137fe640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f14100150 0x7f0f14197260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 -- 172.21.15.3:0/3656462065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f14100150 msgr2=0x7f0f14197260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f14100150 0x7f0f14197260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 -- 172.21.15.3:0/3656462065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f14101340 msgr2=0x7f0f141977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f14101340 0x7f0f141977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 -- 172.21.15.3:0/3656462065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08009560 con 0x7f0f14101d00 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f13fff640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 0x7f0f1419e810 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00d750 tx=0x7f0f0c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f10ff9640 1 -- 172.21.15.3:0/3656462065 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c009bd0 con 0x7f0f14101d00 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f10ff9640 1 -- 172.21.15.3:0/3656462065 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f0c00de60 con 0x7f0f14101d00 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1419f360 con 0x7f0f14101d00 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f10ff9640 1 -- 172.21.15.3:0/3656462065 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f0c004e70 con 0x7f0f14101d00 2024-02-09T22:51:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.575+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1419f7f0 con 0x7f0f14101d00 2024-02-09T22:51:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f10ff9640 1 -- 172.21.15.3:0/3656462065 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 18) v1 ==== 136767+0+0 (secure 0 0 0) 0x7f0f0c005020 con 0x7f0f14101d00 2024-02-09T22:51:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 Processor -- start 2024-02-09T22:51:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 -- start start 2024-02-09T22:51:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f4302f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f431520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4382f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f3e7d00 con 0x557d4f431a60 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f39c2f0 con 0x557d4f42fed0 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.578+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f399820 con 0x557d4f4310a0 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4382f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f10ff9640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0efc0a20c0 0x7f0efc0a4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4382f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39876/0 (socket says 172.21.15.3:39876) 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2184477054 learned_addr learned my addr 172.21.15.3:0/2184477054 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f13fff640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0efc0a20c0 0x7f0efc0a4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f137fe640 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f431520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f4302f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f10ff9640 1 -- 172.21.15.3:0/3656462065 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0f0c0cd5d0 con 0x7f0f14101d00 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2184477054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 msgr2=0x557d4f431520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f431520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2184477054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 msgr2=0x557d4f4302f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f4302f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2184477054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d4f356ff0 con 0x557d4f431a60 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4382f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0f0802c840 tx=0x7f0f0802c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4073141784c2b69 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0f13fff640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0efc0a20c0 0x7f0efc0a4580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0a2580 tx=0x7f0f0c065510 comp rx=0 tx=0).ready entity=mgr.14148 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0ef9ffb640 1 -- 172.21.15.3:0/2184477054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08004810 con 0x557d4f431a60 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0ef9ffb640 1 -- 172.21.15.3:0/2184477054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f08004510 con 0x557d4f431a60 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.579+0000 7f0ef9ffb640 1 -- 172.21.15.3:0/2184477054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08005640 con 0x557d4f431a60 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2184477054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 msgr2=0x557d4f4382f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4382f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0f0802c840 tx=0x7f0f0802c920 comp rx=0 tx=0).stop 2024-02-09T22:51:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2184477054 shutdown_connections 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4382f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f431520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2184477054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f4302f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2184477054 >> 172.21.15.3:0/2184477054 conn(0x557d4f426eb0 msgr2=0x557d4f4272b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2184477054 shutdown_connections 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2184477054 wait complete. 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.580+0000 7f0f1c7f1740 1 -- start start 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 Processor -- start 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f42fed0 0x557d4f43a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4415a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f395000 con 0x557d4f431a60 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f068430 con 0x557d4f4310a0 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f024e00 con 0x557d4f42fed0 2024-02-09T22:51:06.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f1c7f1740 10 client.0 _ll_register_callbacks cb 0x557d4f3897f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4415a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4415a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39888/0 (socket says 172.21.15.3:39888) 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 -- 172.21.15.3:0/1512078357 learned_addr learned my addr 172.21.15.3:0/1512078357 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f42fed0 0x557d4f43a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f137fe640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 -- 172.21.15.3:0/1512078357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f42fed0 msgr2=0x557d4f43a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f42fed0 0x557d4f43a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 -- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 msgr2=0x557d4f43a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.581+0000 7f0f12ffd640 1 -- 172.21.15.3:0/1512078357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f14068500 con 0x557d4f431a60 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4415a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0f08000c00 tx=0x7f0f0802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1a87592cedf9d0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0ef2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/1512078357 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08042470 con 0x557d4f431a60 2024-02-09T22:51:06.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/1512078357 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0f08000e40 con 0x557d4f431a60 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557d4f0947e0 con 0x557d4f431a60 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/1512078357 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f080425d0 con 0x557d4f431a60 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 10 client.15495 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557d4f393680 con 0x557d4f431a60 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/1512078357 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557d4f0947e0 con 0x557d4f431a60 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/1512078357 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0f08061070 con 0x557d4f431a60 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 10 client.15495 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 10 client.15495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 20 client.15495 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f445ec0 0x557d4f4482e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 4 client.15495 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.582+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557d4f0d1720 con 0x557d4f445ec0 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.583+0000 7f0f137fe640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f445ec0 0x557d4f4482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.583+0000 7f0ef2ffd640 10 client.15495 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.583+0000 7f0f137fe640 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f445ec0 0x557d4f4482e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6329579351c53773 server_cookie=b9a67f75dd60199 in_seq=0 out_seq=0 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/1512078357 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557d4f0d1720 con 0x557d4f445ec0 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0ef2ffd640 10 client.15495 handle_command_reply: tid=0 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 client.15495 shutdown 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 msgr2=0x557d4f4415a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4415a0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0f08000c00 tx=0x7f0f0802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 shutdown_connections 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f445ec0 0x557d4f4482e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f4415a0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0ef2ffd640 0 client.15495 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/1512078357 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f42fed0 0x557d4f43a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 >> 172.21.15.3:0/1512078357 conn(0x557d4f426eb0 msgr2=0x557d4f4261f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 shutdown_connections 2024-02-09T22:51:06.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/1512078357 wait complete. 2024-02-09T22:51:06.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.584+0000 7f0f1c7f1740 20 client.15495 trim_cache size 0 max 16384 2024-02-09T22:51:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 Processor -- start 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 -- start start 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f427290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f4414c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f434bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f319fd0 con 0x557d4f431a60 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f0947e0 con 0x557d4f42fed0 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f422980 con 0x557d4f4310a0 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f434bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f434bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39892/0 (socket says 172.21.15.3:39892) 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.599+0000 7f0f12ffd640 1 -- 172.21.15.3:0/3990680700 learned_addr learned my addr 172.21.15.3:0/3990680700 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f427290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f137fe640 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f4414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 -- 172.21.15.3:0/3990680700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 msgr2=0x557d4f4414c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f4414c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 -- 172.21.15.3:0/3990680700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 msgr2=0x557d4f427290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f427290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 -- 172.21.15.3:0/3990680700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557d4f32fb30 con 0x557d4f431a60 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f137fe640 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f4414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f434bd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0f08034b80 tx=0x7f0f08034b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32d902ed3f3b189c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0ef97fa640 1 -- 172.21.15.3:0/3990680700 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f080055d0 con 0x557d4f431a60 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0ef97fa640 1 -- 172.21.15.3:0/3990680700 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f0805e070 con 0x557d4f431a60 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.600+0000 7f0ef97fa640 1 -- 172.21.15.3:0/3990680700 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08042950 con 0x557d4f431a60 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/3990680700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 msgr2=0x557d4f434bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f434bd0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f0f08034b80 tx=0x7f0f08034b10 comp rx=0 tx=0).stop 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/3990680700 shutdown_connections 2024-02-09T22:51:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f431a60 0x557d4f434bd0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f4310a0 0x557d4f4414c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/3990680700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f42fed0 0x557d4f427290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/3990680700 >> 172.21.15.3:0/3990680700 conn(0x557d4f426eb0 msgr2=0x557d4f422080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/3990680700 shutdown_connections 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/3990680700 wait complete. 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- start start 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 Processor -- start 2024-02-09T22:51:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 0x557d4f441270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f431a60 0x557d4f43fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f440300 con 0x557d4f42fed0 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f440470 con 0x557d4f4310a0 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557d4f440630 con 0x557d4f431a60 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f1c7f1740 10 client.0 _ll_register_callbacks cb 0x7f0f08005d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 0x557d4f441270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 0x557d4f441270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39898/0 (socket says 172.21.15.3:39898) 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.601+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2268082376 learned_addr learned my addr 172.21.15.3:0/2268082376 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f137fe640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f431a60 0x557d4f43fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2268082376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f431a60 msgr2=0x557d4f43fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f431a60 0x557d4f43fdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 msgr2=0x557d4f43f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 -- 172.21.15.3:0/2268082376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557d4f393680 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f137fe640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f12ffd640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 0x557d4f441270 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009850 tx=0x7f0f08005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f548954b8b64980f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0ef2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/2268082376 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f08034940 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/2268082376 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f0f08000e40 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f08004510 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/2268082376 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f08034c00 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f1c7f1740 10 client.15501 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x557d4f355850 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.602+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/2268082376 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f08074020 con 0x557d4f42fed0 2024-02-09T22:51:06.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/2268082376 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0f08061040 con 0x557d4f42fed0 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 10 client.15501 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 10 client.15501 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 20 client.15501 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f49d6b0 0x557d4f49fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 4 client.15501 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x557d4f120890 con 0x557d4f49d6b0 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f137fe640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f49d6b0 0x557d4f49fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0ef2ffd640 10 client.15501 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f137fe640 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f49d6b0 0x557d4f49fad0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4baf7c6aad65ca84 server_cookie=6ec9359a7c99480b in_seq=0 out_seq=0 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0ef2ffd640 1 -- 172.21.15.3:0/2268082376 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x557d4f120890 con 0x557d4f49d6b0 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0ef2ffd640 10 client.15501 handle_command_reply: tid=0 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.603+0000 7f0f1c7f1740 1 client.15501 shutdown 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 msgr2=0x557d4f441270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 0x557d4f441270 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f0f08009850 tx=0x7f0f08005fd0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 shutdown_connections 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557d4f49d6b0 0x557d4f49fad0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557d4f4310a0 0x557d4f43f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557d4f431a60 0x557d4f43fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0ef2ffd640 0 client.15501 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 --2- 172.21.15.3:0/2268082376 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557d4f42fed0 0x557d4f441270 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 >> 172.21.15.3:0/2268082376 conn(0x557d4f426eb0 msgr2=0x557d4f437dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 shutdown_connections 2024-02-09T22:51:06.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 1 -- 172.21.15.3:0/2268082376 wait complete. 2024-02-09T22:51:06.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f1c7f1740 20 client.15501 trim_cache size 0 max 16384 2024-02-09T22:51:06.615 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:51:06.615 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: "d9d10ee7-c3f1-4494-b951-4f7c0b5107e8": { 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "d9d10ee7-c3f1-4494-b951-4f7c0b5107e8", 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:51:06.616 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0efc0a20c0 msgr2=0x7f0efc0a4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0efc0a20c0 0x7f0efc0a4580 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c0a2580 tx=0x7f0f0c065510 comp rx=0 tx=0).stop 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 msgr2=0x7f0f1419e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 0x7f0f1419e810 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c00d750 tx=0x7f0f0c00dc10 comp rx=0 tx=0).stop 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 shutdown_connections 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f0efc0a20c0 0x7f0efc0a4580 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f14101340 0x7f0f141977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f14101d00 0x7f0f1419e810 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 --2- 172.21.15.3:0/3656462065 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f14100150 0x7f0f14197260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:06.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.604+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 >> 172.21.15.3:0/3656462065 conn(0x7f0f140fb6c0 msgr2=0x7f0f14104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:06.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.605+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 shutdown_connections 2024-02-09T22:51:06.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.605+0000 7f0f19b47640 1 -- 172.21.15.3:0/3656462065 wait complete. 2024-02-09T22:51:06.618 INFO:tasks.cephfs.filesystem:scrub status for tag:d9d10ee7-c3f1-4494-b951-4f7c0b5107e8 - {'path': '/', 'tag': 'd9d10ee7-c3f1-4494-b951-4f7c0b5107e8', 'options': 'recursive,force'} 2024-02-09T22:51:06.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:06.671+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5ac419fcc0 con 0x7f5ac4100160 2024-02-09T22:51:07.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.142+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mgrmap(e 19) v1 ==== 136784+0+0 (secure 0 0 0) 0x7f5abc0b95f0 con 0x7f5ac4100160 2024-02-09T22:51:07.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.142+0000 7f5ab37fe640 1 -- 172.21.15.3:0/73339296 <== mon.1 v2:172.21.15.89:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5abc0e9ac0 con 0x7f5ac4100160 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5aac0a20b0 msgr2=0x7f5aac0a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5aac0a20b0 0x7f5aac0a4570 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5ac41987a0 tx=0x7f5ab800a2d0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 msgr2=0x7f5ac41972e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 0x7f5ac41972e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5abc036040 tx=0x7f5abc0042f0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 shutdown_connections 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f5aac0a20b0 0x7f5aac0a4570 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ac4101d10 0x7f5ac419e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ac4101350 0x7f5ac4197820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 --2- 172.21.15.3:0/73339296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ac4100160 0x7f5ac41972e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 >> 172.21.15.3:0/73339296 conn(0x7f5ac40fb6d0 msgr2=0x7f5ac4068a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:07.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 shutdown_connections 2024-02-09T22:51:07.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.145+0000 7f5ac8b35640 1 -- 172.21.15.3:0/73339296 wait complete. 2024-02-09T22:51:07.160 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-02-09T22:51:07.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 -- 172.21.15.3:0/3669540395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338109130 msgr2=0x7f333810b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 --2- 172.21.15.3:0/3669540395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338109130 0x7f333810b5d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3330009ca0 tx=0x7f333002c180 comp rx=0 tx=0).stop 2024-02-09T22:51:07.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 -- 172.21.15.3:0/3669540395 shutdown_connections 2024-02-09T22:51:07.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 --2- 172.21.15.3:0/3669540395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338109130 0x7f333810b5d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 --2- 172.21.15.3:0/3669540395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338073c30 0x7f3338074090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 --2- 172.21.15.3:0/3669540395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3338075210 0x7f3338073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 -- 172.21.15.3:0/3669540395 >> 172.21.15.3:0/3669540395 conn(0x7f33380fc810 msgr2=0x7f33380fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 -- 172.21.15.3:0/3669540395 shutdown_connections 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.269+0000 7f333f302640 1 -- 172.21.15.3:0/3669540395 wait complete. 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 Processor -- start 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 -- start start 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338073c30 0x7f33381994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 0x7f3338199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3338109130 0x7f33381a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33381a0fc0 con 0x7f3338075210 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33381a1130 con 0x7f3338109130 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333f302640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33381a12f0 con 0x7f3338073c30 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333c89d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 0x7f3338199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333c89d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 0x7f3338199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39932/0 (socket says 172.21.15.3:39932) 2024-02-09T22:51:07.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333c89d640 1 -- 172.21.15.3:0/2491093814 learned_addr learned my addr 172.21.15.3:0/2491093814 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333d09e640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338073c30 0x7f33381994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.270+0000 7f333d89f640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3338109130 0x7f33381a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333c89d640 1 -- 172.21.15.3:0/2491093814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338073c30 msgr2=0x7f33381994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333c89d640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338073c30 0x7f33381994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333c89d640 1 -- 172.21.15.3:0/2491093814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3338109130 msgr2=0x7f33381a0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333c89d640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3338109130 0x7f33381a0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333c89d640 1 -- 172.21.15.3:0/2491093814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3330009900 con 0x7f3338075210 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333d09e640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338073c30 0x7f33381994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333c89d640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 0x7f3338199a10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f332800e9f0 tx=0x7f332800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328009700 con 0x7f3338075210 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33381a15d0 con 0x7f3338075210 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f332800cd80 con 0x7f3338075210 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3328018920 con 0x7f3338075210 2024-02-09T22:51:07.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.271+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33381a1b30 con 0x7f3338075210 2024-02-09T22:51:07.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.272+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3338107ef0 con 0x7f3338075210 2024-02-09T22:51:07.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.273+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 19) v1 ==== 136784+0+0 (secure 0 0 0) 0x7f3328018a80 con 0x7f3338075210 2024-02-09T22:51:07.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.274+0000 7f332e7fc640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33240a2100 0x7f33240a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.274+0000 7f333d09e640 1 -- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33240a2100 msgr2=0x7f33240a45c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:07.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.274+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f3328014070 con 0x7f3338075210 2024-02-09T22:51:07.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.274+0000 7f333d09e640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33240a2100 0x7f33240a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:51:07.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.279+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3328052d20 con 0x7f3338075210 2024-02-09T22:51:07.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: ignoring --setuser ceph since I am not root 2024-02-09T22:51:07.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: ignoring --setgroup ceph since I am not root 2024-02-09T22:51:07.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mgr[38020]: -- 172.21.15.64:0/4059364344 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a8cae041a0 con 0x55a8cadda800 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3818376288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: pgmap v348: 97 pgs: 97 active+clean; 647 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/73339296' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-02-09T22:51:07.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-mon[36638]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-02-09T22:51:07.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.462+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f3338107960 con 0x7f3338075210 2024-02-09T22:51:07.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.465+0000 7f332e7fc640 1 -- 172.21.15.3:0/2491093814 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v27)=0 v27) v1 ==== 138+0+0 (secure 0 0 0) 0x7f332809ac40 con 0x7f3338075210 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.467+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33240a2100 msgr2=0x7f33240a45c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.467+0000 7f333f302640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33240a2100 0x7f33240a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.467+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 msgr2=0x7f3338199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.467+0000 7f333f302640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 0x7f3338199a10 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f332800e9f0 tx=0x7f332800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 shutdown_connections 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f33240a2100 0x7f33240a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3338109130 0x7f33381a0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3338075210 0x7f3338199a10 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 --2- 172.21.15.3:0/2491093814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3338073c30 0x7f33381994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 >> 172.21.15.3:0/2491093814 conn(0x7f33380fc810 msgr2=0x7f33381050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:07.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 shutdown_connections 2024-02-09T22:51:07.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.468+0000 7f333f302640 1 -- 172.21.15.3:0/2491093814 wait complete. 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: ignoring --setuser ceph since I am not root 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: ignoring --setgroup ceph since I am not root 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mgr[38529]: -- 172.21.15.89:0/910140499 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d9c0f081a0 con 0x55d9c0edf000 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3818376288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:07.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:51:07.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:07.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: pgmap v348: 97 pgs: 97 active+clean; 647 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-02-09T22:51:07.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:51:07.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/73339296' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-02-09T22:51:07.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-mon[36665]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-02-09T22:51:07.481 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: ignoring --setuser ceph since I am not root 2024-02-09T22:51:07.481 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: ignoring --setgroup ceph since I am not root 2024-02-09T22:51:07.481 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3818376288' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: pgmap v348: 97 pgs: 97 active+clean; 647 KiB data, 350 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 938 B/s wr, 12 op/s 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/73339296' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-02-09T22:51:07.482 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-mon[32678]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-02-09T22:51:07.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-02-09T22:51:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 -- 172.21.15.3:0/1124119757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570102470 msgr2=0x7f05701028f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 --2- 172.21.15.3:0/1124119757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570102470 0x7f05701028f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f0560009900 tx=0x7f056002bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 -- 172.21.15.3:0/1124119757 shutdown_connections 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 --2- 172.21.15.3:0/1124119757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0570102e30 0x7f05701096c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 --2- 172.21.15.3:0/1124119757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570102470 0x7f05701028f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 --2- 172.21.15.3:0/1124119757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570101280 0x7f05701016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 -- 172.21.15.3:0/1124119757 >> 172.21.15.3:0/1124119757 conn(0x7f05700fc7f0 msgr2=0x7f05700fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 -- 172.21.15.3:0/1124119757 shutdown_connections 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.585+0000 7f057816c640 1 -- 172.21.15.3:0/1124119757 wait complete. 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 Processor -- start 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 -- start start 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 0x7f0570197310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0570102470 0x7f0570197850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570102e30 0x7f057019e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057019ee00 con 0x7f0570101280 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057019ef70 con 0x7f0570102e30 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f057816c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057019f130 con 0x7f0570102470 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 0x7f0570197310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 0x7f0570197310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39948/0 (socket says 172.21.15.3:39948) 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 -- 172.21.15.3:0/849889960 learned_addr learned my addr 172.21.15.3:0/849889960 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0576709640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570102e30 0x7f057019e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575707640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0570102470 0x7f0570197850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 -- 172.21.15.3:0/849889960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0570102470 msgr2=0x7f0570197850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0570102470 0x7f0570197850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 -- 172.21.15.3:0/849889960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570102e30 msgr2=0x7f057019e8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570102e30 0x7f057019e8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0575f08640 1 -- 172.21.15.3:0/849889960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0560009560 con 0x7f0570101280 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.586+0000 7f0576709640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570102e30 0x7f057019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.587+0000 7f0575f08640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 0x7f0570197310 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f056c00e9c0 tx=0x7f056c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.587+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c009700 con 0x7f0570101280 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.587+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f057019f410 con 0x7f0570101280 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.587+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f056c00cd80 con 0x7f0570101280 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.587+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c010430 con 0x7f0570101280 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.587+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f057019f8f0 con 0x7f0570101280 2024-02-09T22:51:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.588+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05701085a0 con 0x7f0570101280 2024-02-09T22:51:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.589+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 19) v1 ==== 136784+0+0 (secure 0 0 0) 0x7f056c003680 con 0x7f0570101280 2024-02-09T22:51:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.589+0000 7f0566ffd640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 0x7f055c0a45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.589+0000 7f0575707640 1 -- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 msgr2=0x7f055c0a45c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.590+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f056c014070 con 0x7f0570101280 2024-02-09T22:51:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.590+0000 7f0575707640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 0x7f055c0a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:51:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.593+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f056c0971f0 con 0x7f0570101280 2024-02-09T22:51:07.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.791+0000 7f0575707640 1 -- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 msgr2=0x7f055c0a45c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:07.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.791+0000 7f0575707640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 0x7f055c0a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T22:51:07.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.812+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f057010cc90 con 0x7f0570101280 2024-02-09T22:51:07.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.817+0000 7f0566ffd640 1 -- 172.21.15.3:0/849889960 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v28)=0 v28) v1 ==== 133+0+0 (secure 0 0 0) 0x7f056c09a8b0 con 0x7f0570101280 2024-02-09T22:51:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 msgr2=0x7f055c0a45c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:51:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 0x7f055c0a45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 msgr2=0x7f0570197310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 0x7f0570197310 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f056c00e9c0 tx=0x7f056c00ee80 comp rx=0 tx=0).stop 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 shutdown_connections 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f055c0a2100 0x7f055c0a45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0570102e30 0x7f057019e8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0570102470 0x7f0570197850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 --2- 172.21.15.3:0/849889960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0570101280 0x7f0570197310 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 >> 172.21.15.3:0/849889960 conn(0x7f05700fc7f0 msgr2=0x7f0570105770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 shutdown_connections 2024-02-09T22:51:07.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.819+0000 7f057816c640 1 -- 172.21.15.3:0/849889960 wait complete. 2024-02-09T22:51:07.833 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- 172.21.15.3:0/1640996184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 msgr2=0x7f1b1c106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- 172.21.15.3:0/1640996184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c106aa0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f1b18009900 tx=0x7f1b1802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- 172.21.15.3:0/1640996184 shutdown_connections 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- 172.21.15.3:0/1640996184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b1c106fe0 0x7f1b1c1093d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- 172.21.15.3:0/1640996184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c106aa0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- 172.21.15.3:0/1640996184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b1c101d80 0x7f1b1c104170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- 172.21.15.3:0/1640996184 >> 172.21.15.3:0/1640996184 conn(0x7f1b1c0fb6d0 msgr2=0x7f1b1c0fdb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:07.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- 172.21.15.3:0/1640996184 shutdown_connections 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- 172.21.15.3:0/1640996184 wait complete. 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 Processor -- start 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- start start 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b1c101d80 0x7f1b1c197320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c197860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b1c106fe0 0x7f1b1c19e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c19ee10 con 0x7f1b1c1046b0 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c19ef80 con 0x7f1b1c101d80 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.936+0000 7f1b23ff4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b1c19f140 con 0x7f1b1c106fe0 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c197860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c197860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39968/0 (socket says 172.21.15.3:39968) 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 -- 172.21.15.3:0/2287766015 learned_addr learned my addr 172.21.15.3:0/2287766015 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:07.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b1c101d80 0x7f1b1c197320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b22591640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b1c106fe0 0x7f1b1c19e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b1c106fe0 msgr2=0x7f1b1c19e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b1c106fe0 0x7f1b1c19e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b1c101d80 msgr2=0x7f1b1c197320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b1c101d80 0x7f1b1c197320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 -- 172.21.15.3:0/2287766015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b18009560 con 0x7f1b1c1046b0 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b1c101d80 0x7f1b1c197320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:07.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b2158f640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c197860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1b18005ab0 tx=0x7f1b18004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b18004aa0 con 0x7f1b1c1046b0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b180052d0 con 0x7f1b1c1046b0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b1c19f3c0 con 0x7f1b1c1046b0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b18003e80 con 0x7f1b1c1046b0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.937+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b1c19f920 con 0x7f1b1c1046b0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.939+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b1c0fd1c0 con 0x7f1b1c1046b0 2024-02-09T22:51:07.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.939+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 19) v1 ==== 136784+0+0 (secure 0 0 0) 0x7f1b18005440 con 0x7f1b1c1046b0 2024-02-09T22:51:07.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.940+0000 7f1b12ffd640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:07.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.940+0000 7f1b21d90640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:07.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.940+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8211+0+0 (secure 0 0 0) 0x7f1b180efe70 con 0x7f1b1c1046b0 2024-02-09T22:51:07.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.940+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-02-09T22:51:07.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:07.944+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b180b8e50 con 0x7f1b1c1046b0 2024-02-09T22:51:07.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:07.704+0000 7f45ae4f1200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:51:07.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:07.797+0000 7f45ae4f1200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:51:08.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:07.762+0000 7f0eb3e0c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:51:08.132 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:07.863+0000 7f0eb3e0c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:51:08.136 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:07.729+0000 7fb85a573200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-02-09T22:51:08.136 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:07.827+0000 7fb85a573200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-02-09T22:51:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:08.141+0000 7f1b21d90640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:08.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:08.141+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-02-09T22:51:08.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:08.168+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 --> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b1c061670 con 0x7f1b080a2180 2024-02-09T22:51:08.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-mon[36638]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-02-09T22:51:08.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-mon[36638]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-02-09T22:51:08.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2491093814' entity='client.admin' 2024-02-09T22:51:08.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/849889960' entity='client.admin' 2024-02-09T22:51:08.442 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-mon[36665]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-02-09T22:51:08.442 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-mon[36665]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-02-09T22:51:08.442 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2491093814' entity='client.admin' 2024-02-09T22:51:08.442 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/849889960' entity='client.admin' 2024-02-09T22:51:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:08.542+0000 7f1b21d90640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:08.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:08.542+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-02-09T22:51:08.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-mon[32678]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-02-09T22:51:08.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-mon[32678]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-02-09T22:51:08.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2491093814' entity='client.admin' 2024-02-09T22:51:08.547 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/849889960' entity='client.admin' 2024-02-09T22:51:08.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:08.439+0000 7f45ae4f1200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:51:08.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:08.531+0000 7f45ae4f1200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:51:08.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:08.672+0000 7f45ae4f1200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:51:08.839 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:08.493+0000 7fb85a573200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:51:08.839 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:08.590+0000 7fb85a573200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:51:08.839 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:08.736+0000 7fb85a573200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:51:08.839 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:08.835+0000 7fb85a573200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:51:08.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:08.544+0000 7f0eb3e0c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-02-09T22:51:08.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:08.639+0000 7f0eb3e0c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-02-09T22:51:08.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:08.783+0000 7f0eb3e0c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-02-09T22:51:09.111 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:08.766+0000 7f45ae4f1200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:51:09.111 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:08.848+0000 7f45ae4f1200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:51:09.185 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:08.921+0000 7fb85a573200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:51:09.237 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:08.881+0000 7f0eb3e0c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-02-09T22:51:09.237 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:08.966+0000 7f0eb3e0c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-02-09T22:51:09.238 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:09.234+0000 7f0eb3e0c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:51:09.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:09.343+0000 7f1b21d90640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:09.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:09.343+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-02-09T22:51:09.397 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:09.108+0000 7f45ae4f1200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:51:09.397 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:09.217+0000 7f45ae4f1200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:51:09.398 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:09.302+0000 7f45ae4f1200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:51:09.499 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:09.193+0000 7fb85a573200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-02-09T22:51:09.499 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:09.308+0000 7fb85a573200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:51:09.499 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:09.397+0000 7fb85a573200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:51:09.535 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:09.346+0000 7f0eb3e0c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-02-09T22:51:09.535 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:09.435+0000 7f0eb3e0c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-02-09T22:51:09.535 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:09.532+0000 7f0eb3e0c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:51:09.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:09.394+0000 7f45ae4f1200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:51:09.728 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:09.568+0000 7f45ae4f1200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:51:09.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:09.710+0000 7f0eb3e0c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:51:09.935 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:09.495+0000 7fb85a573200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-02-09T22:51:09.935 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:09.676+0000 7fb85a573200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-02-09T22:51:10.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:10.944+0000 7f1b21d90640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:10.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:10.944+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-02-09T22:51:10.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:10.521+0000 7f45ae4f1200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:51:10.977 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:10.650+0000 7f45ae4f1200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:51:11.109 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:10.705+0000 7f0eb3e0c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:51:11.109 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:10.840+0000 7f0eb3e0c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:51:11.181 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:10.695+0000 7fb85a573200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-02-09T22:51:11.181 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:10.831+0000 7fb85a573200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-02-09T22:51:11.268 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:10.979+0000 7f45ae4f1200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:51:11.268 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:11.069+0000 7f45ae4f1200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:51:11.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:11.181+0000 7f0eb3e0c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:51:11.382 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:11.275+0000 7f0eb3e0c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:51:11.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:11.177+0000 7fb85a573200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-02-09T22:51:11.435 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:11.272+0000 7fb85a573200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-02-09T22:51:11.727 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:11.265+0000 7f45ae4f1200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:51:11.728 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:11.448+0000 7f45ae4f1200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:51:11.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:11.478+0000 7f0eb3e0c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:51:11.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:11.669+0000 7f0eb3e0c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:51:11.936 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:11.474+0000 7fb85a573200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:51:11.936 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:11.665+0000 7fb85a573200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-02-09T22:51:11.991 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:11.730+0000 7f45ae4f1200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:51:12.231 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:11.959+0000 7fb85a573200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:51:12.235 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:11.969+0000 7f0eb3e0c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-02-09T22:51:12.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:11.988+0000 7f45ae4f1200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:51:12.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:12.222+0000 7f45ae4f1200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:51:12.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:12.233+0000 7f0eb3e0c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:51:12.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:12.478+0000 7f0eb3e0c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:51:12.685 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:12.228+0000 7fb85a573200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-02-09T22:51:12.685 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:12.473+0000 7fb85a573200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-02-09T22:51:13.477 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:13.020+0000 7f45ae4f1200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:51:13.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:13.317+0000 7f0eb3e0c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:51:13.685 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:13.310+0000 7fb85a573200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-02-09T22:51:13.897 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:13.586+0000 7f45ae4f1200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:51:13.898 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:13.683+0000 7f45ae4f1200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:51:13.898 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z[38525]: 2024-02-09T22:51:13.894+0000 7f45ae4f1200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:51:13.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:13.968+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mgrmap(e 20) v1 ==== 136784+0+0 (secure 0 0 0) 0x7f1b180035f0 con 0x7f1b1c1046b0 2024-02-09T22:51:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:14.147+0000 7f1b21d90640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.3:6800/284621866 2024-02-09T22:51:14.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:14.147+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-02-09T22:51:14.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-mon[36638]: Standby manager daemon z restarted 2024-02-09T22:51:14.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-mon[36638]: Standby manager daemon z started 2024-02-09T22:51:14.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-09T22:51:14.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:51:14.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-09T22:51:14.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:51:14.186 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:13.904+0000 7fb85a573200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:51:14.186 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:14.010+0000 7fb85a573200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:51:14.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-mon[36665]: Standby manager daemon z restarted 2024-02-09T22:51:14.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-mon[36665]: Standby manager daemon z started 2024-02-09T22:51:14.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-09T22:51:14.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:51:14.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-09T22:51:14.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:13 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-mon[32678]: Standby manager daemon z restarted 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-mon[32678]: Standby manager daemon z started 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.89:0/2357387961' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:13.905+0000 7f0eb3e0c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.011+0000 7f0eb3e0c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-02-09T22:51:14.241 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.238+0000 7f0eb3e0c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:51:14.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:14.262+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 8 ==== mgrmap(e 21) v1 ==== 136453+0+0 (secure 0 0 0) 0x7f1b180ed9d0 con 0x7f1b1c1046b0 2024-02-09T22:51:14.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:14.263+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 msgr2=0x7f1b080a4640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-02-09T22:51:14.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:14.263+0000 7f1b12ffd640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/284621866,v1:172.21.15.3:6801/284621866] conn(0x7f1b080a2180 0x7f1b080a4640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:14.524 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y[38016]: 2024-02-09T22:51:14.236+0000 7fb85a573200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.499+0000 7f0e168bd700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.499+0000 7f0e168bd700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.499+0000 7f0e168bd700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.499+0000 7f0e168bd700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.499+0000 7f0e168bd700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.500+0000 7f0e130b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.500+0000 7f0e130b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.882 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.500+0000 7f0e130b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.883 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.500+0000 7f0e130b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:14.883 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:51:14.500+0000 7f0e130b6700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:51:15.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: Standby manager daemon y restarted 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: Standby manager daemon y started 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: Active manager daemon x restarted 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: Activating manager daemon x 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: osdmap e89: 12 total, 12 up, 12 in 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: mgrmap e21: x(active, starting, since 0.0127779s), standbys: z, y 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:51:15.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T22:51:15.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:51:15.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: Manager daemon x is now available 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-09T22:51:15.231 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:14 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-09T22:51:15.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.268+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mon.0 v2:172.21.15.3:3300/0 9 ==== mgrmap(e 22) v1 ==== 136580+0+0 (secure 0 0 0) 0x7f1b180b9220 con 0x7f1b1c1046b0 2024-02-09T22:51:15.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.272+0000 7f1b12ffd640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b08034bb0 0x7f1b08034f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.272+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 --> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1b1c061670 con 0x7f1b08034bb0 2024-02-09T22:51:15.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.275+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b08034bb0 0x7f1b08034f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.277+0000 7f1b21d90640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b08034bb0 0x7f1b08034f90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009fd0 tx=0x7f1b0c005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:15.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.345+0000 7f1b12ffd640 1 -- 172.21.15.3:0/2287766015 <== mgr.15519 v2:172.21.15.3:6800/4119047166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1b1c061670 con 0x7f1b08034bb0 2024-02-09T22:51:15.349 INFO:teuthology.orchestra.run.smithi003.stdout:Schedule set for path / 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b08034bb0 msgr2=0x7f1b08034f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b08034bb0 0x7f1b08034f90 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1b0c009fd0 tx=0x7f1b0c005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 msgr2=0x7f1b1c197860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 --2- 172.21.15.3:0/2287766015 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b1c1046b0 0x7f1b1c197860 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1b18005ab0 tx=0x7f1b18004270 comp rx=0 tx=0).stop 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b22591640 1 -- 172.21.15.3:0/2287766015 reap_dead start 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 shutdown_connections 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 >> 172.21.15.3:0/2287766015 conn(0x7f1b1c0fb6d0 msgr2=0x7f1b1c102840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 shutdown_connections 2024-02-09T22:51:15.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.348+0000 7f1b23ff4640 1 -- 172.21.15.3:0/2287766015 wait complete. 2024-02-09T22:51:15.360 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: Standby manager daemon y restarted 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: Standby manager daemon y started 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: Active manager daemon x restarted 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: Activating manager daemon x 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: osdmap e89: 12 total, 12 up, 12 in 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: mgrmap e21: x(active, starting, since 0.0127779s), standbys: z, y 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-09T22:51:15.361 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-09T22:51:15.362 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T22:51:15.363 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:51:15.364 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: Manager daemon x is now available 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-09T22:51:15.365 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:14 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: mgrmap e20: x(active, since 9m), standbys: y, z 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: Standby manager daemon y restarted 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: Standby manager daemon y started 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: Active manager daemon x restarted 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: Activating manager daemon x 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: osdmap e89: 12 total, 12 up, 12 in 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: mgrmap e21: x(active, starting, since 0.0127779s), standbys: z, y 2024-02-09T22:51:15.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-02-09T22:51:15.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.? 172.21.15.64:0/1692933799' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-02-09T22:51:15.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-02-09T22:51:15.439 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: Manager daemon x is now available 2024-02-09T22:51:15.440 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:15.440 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:15.440 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-02-09T22:51:15.440 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:14 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-02-09T22:51:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 -- 172.21.15.3:0/92641242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc780ff580 msgr2=0x7efc780ff9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 --2- 172.21.15.3:0/92641242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc780ff580 0x7efc780ff9a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7efc74009900 tx=0x7efc7402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 -- 172.21.15.3:0/92641242 shutdown_connections 2024-02-09T22:51:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 --2- 172.21.15.3:0/92641242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc781025f0 0x7efc78102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 --2- 172.21.15.3:0/92641242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc78102fb0 0x7efc78109840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 --2- 172.21.15.3:0/92641242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc780ff580 0x7efc780ff9a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 -- 172.21.15.3:0/92641242 >> 172.21.15.3:0/92641242 conn(0x7efc780fae90 msgr2=0x7efc780fd2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 -- 172.21.15.3:0/92641242 shutdown_connections 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 -- 172.21.15.3:0/92641242 wait complete. 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 Processor -- start 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.469+0000 7efc7f89d640 1 -- start start 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc780ff580 0x7efc78197380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 0x7efc781978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc78102fb0 0x7efc7819e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7819ee70 con 0x7efc781025f0 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7819efe0 con 0x7efc780ff580 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7819f1a0 con 0x7efc78102fb0 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 0x7efc781978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7d639640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc780ff580 0x7efc78197380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 0x7efc781978c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40108/0 (socket says 172.21.15.3:40108) 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 -- 172.21.15.3:0/3767443785 learned_addr learned my addr 172.21.15.3:0/3767443785 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7de3a640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc78102fb0 0x7efc7819e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 -- 172.21.15.3:0/3767443785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc78102fb0 msgr2=0x7efc7819e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc78102fb0 0x7efc7819e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 -- 172.21.15.3:0/3767443785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc780ff580 msgr2=0x7efc78197380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc780ff580 0x7efc78197380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 -- 172.21.15.3:0/3767443785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc74009560 con 0x7efc781025f0 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7d639640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc780ff580 0x7efc78197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7ce38640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 0x7efc781978c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e9f0 tx=0x7efc6800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68009700 con 0x7efc781025f0 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc6800cd80 con 0x7efc781025f0 2024-02-09T22:51:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7819f480 con 0x7efc781025f0 2024-02-09T22:51:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc68018920 con 0x7efc781025f0 2024-02-09T22:51:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.470+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7819f9e0 con 0x7efc781025f0 2024-02-09T22:51:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.471+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc781087f0 con 0x7efc781025f0 2024-02-09T22:51:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.472+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 22) v1 ==== 136580+0+0 (secure 0 0 0) 0x7efc68018a80 con 0x7efc781025f0 2024-02-09T22:51:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.473+0000 7efc6e7fc640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc640a1f20 0x7efc640a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.473+0000 7efc7d639640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc640a1f20 0x7efc640a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.473+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efc68014070 con 0x7efc781025f0 2024-02-09T22:51:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.474+0000 7efc7d639640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc640a1f20 0x7efc640a43e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc74005ab0 tx=0x7efc74005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:15.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.478+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efc68052d20 con 0x7efc781025f0 2024-02-09T22:51:15.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.684+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 --> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7efc7802cec0 con 0x7efc640a1f20 2024-02-09T22:51:15.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.736+0000 7efc6e7fc640 1 -- 172.21.15.3:0/3767443785 <== mgr.15519 v2:172.21.15.3:6800/4119047166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7efc7802cec0 con 0x7efc640a1f20 2024-02-09T22:51:15.740 INFO:teuthology.orchestra.run.smithi003.stdout:Retention added to path / 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.738+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc640a1f20 msgr2=0x7efc640a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.738+0000 7efc7f89d640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc640a1f20 0x7efc640a43e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efc74005ab0 tx=0x7efc74005fd0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.738+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 msgr2=0x7efc781978c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.738+0000 7efc7f89d640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 0x7efc781978c0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7efc6800e9f0 tx=0x7efc6800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 shutdown_connections 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc640a1f20 0x7efc640a43e0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc78102fb0 0x7efc7819e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc781025f0 0x7efc781978c0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 --2- 172.21.15.3:0/3767443785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc780ff580 0x7efc78197380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 >> 172.21.15.3:0/3767443785 conn(0x7efc780fae90 msgr2=0x7efc781059c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 shutdown_connections 2024-02-09T22:51:15.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.739+0000 7efc7f89d640 1 -- 172.21.15.3:0/3767443785 wait complete. 2024-02-09T22:51:15.753 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-02-09T22:51:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 -- 172.21.15.3:0/1015478144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 msgr2=0x7f0e80102c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 --2- 172.21.15.3:0/1015478144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e80102c90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0e70009900 tx=0x7f0e7002bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 -- 172.21.15.3:0/1015478144 shutdown_connections 2024-02-09T22:51:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 --2- 172.21.15.3:0/1015478144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e800ff2e0 0x7f0e800ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 --2- 172.21.15.3:0/1015478144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e800ffca0 0x7f0e80109850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 --2- 172.21.15.3:0/1015478144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e80102c90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 -- 172.21.15.3:0/1015478144 >> 172.21.15.3:0/1015478144 conn(0x7f0e800fae60 msgr2=0x7f0e800fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 -- 172.21.15.3:0/1015478144 shutdown_connections 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.855+0000 7f0e8542a640 1 -- 172.21.15.3:0/1015478144 wait complete. 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 Processor -- start 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 -- start start 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e800ff2e0 0x7f0e80197350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e800ffca0 0x7f0e80197890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e8019e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8019ee40 con 0x7f0e80102870 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8019efb0 con 0x7f0e800ff2e0 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e8542a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e8019f170 con 0x7f0e800ffca0 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e8019e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e767fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e800ffca0 0x7f0e80197890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e8019e900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40132/0 (socket says 172.21.15.3:40132) 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 -- 172.21.15.3:0/3437390834 learned_addr learned my addr 172.21.15.3:0/3437390834 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7effd640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e800ff2e0 0x7f0e80197350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 -- 172.21.15.3:0/3437390834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e800ffca0 msgr2=0x7f0e80197890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e800ffca0 0x7f0e80197890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 -- 172.21.15.3:0/3437390834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e800ff2e0 msgr2=0x7f0e80197350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e800ff2e0 0x7f0e80197350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.856+0000 7f0e7f7fe640 1 -- 172.21.15.3:0/3437390834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e70009560 con 0x7f0e80102870 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.857+0000 7f0e7f7fe640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e8019e900 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0e78009fd0 tx=0x7f0e7800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.857+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800bea0 con 0x7f0e80102870 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.857+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e78009940 con 0x7f0e80102870 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.857+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e8019f450 con 0x7f0e80102870 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.857+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e780106c0 con 0x7f0e80102870 2024-02-09T22:51:15.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.857+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e8019f960 con 0x7f0e80102870 2024-02-09T22:51:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.858+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e80108800 con 0x7f0e80102870 2024-02-09T22:51:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.859+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 22) v1 ==== 136580+0+0 (secure 0 0 0) 0x7f0e78004240 con 0x7f0e80102870 2024-02-09T22:51:15.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.860+0000 7f0e7cff9640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e6c0a1f20 0x7f0e6c0a43e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:15.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.860+0000 7f0e7effd640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e6c0a1f20 0x7f0e6c0a43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:15.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.860+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0e780cd5a0 con 0x7f0e80102870 2024-02-09T22:51:15.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.860+0000 7f0e7effd640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e6c0a1f20 0x7f0e6c0a43e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005ab0 tx=0x7f0e70005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:15.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:15.864+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f0e78096820 con 0x7f0e80102870 2024-02-09T22:51:16.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.101+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 --> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e8002ce50 con 0x7f0e6c0a1f20 2024-02-09T22:51:16.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.110+0000 7f0e7cff9640 1 -- 172.21.15.3:0/3437390834 <== mgr.15519 v2:172.21.15.3:6800/4119047166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f0e8002ce50 con 0x7f0e6c0a1f20 2024-02-09T22:51:16.113 INFO:teuthology.orchestra.run.smithi003.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-02-09T00:00:00", "created": "2024-02-09T22:51:15", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-02-09T22:51:16.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e6c0a1f20 msgr2=0x7f0e6c0a43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e6c0a1f20 0x7f0e6c0a43e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0e70005ab0 tx=0x7f0e70005fd0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 msgr2=0x7f0e8019e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e8019e900 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f0e78009fd0 tx=0x7f0e7800b260 comp rx=0 tx=0).stop 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 shutdown_connections 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e6c0a1f20 0x7f0e6c0a43e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e80102870 0x7f0e8019e900 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e800ffca0 0x7f0e80197890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 --2- 172.21.15.3:0/3437390834 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e800ff2e0 0x7f0e80197350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 >> 172.21.15.3:0/3437390834 conn(0x7f0e800fae60 msgr2=0x7f0e801059d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 shutdown_connections 2024-02-09T22:51:16.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.112+0000 7f0e8542a640 1 -- 172.21.15.3:0/3437390834 wait complete. 2024-02-09T22:51:16.127 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-02-09T22:51:16.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:15 smithi064 ceph-mon[36638]: [09/Feb/2024:22:51:14] ENGINE Bus STARTING 2024-02-09T22:51:16.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:15 smithi064 ceph-mon[36638]: [09/Feb/2024:22:51:14] ENGINE Serving on http://172.21.15.3:8765 2024-02-09T22:51:16.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:15 smithi064 ceph-mon[36638]: [09/Feb/2024:22:51:15] ENGINE Serving on https://172.21.15.3:7150 2024-02-09T22:51:16.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:15 smithi064 ceph-mon[36638]: [09/Feb/2024:22:51:15] ENGINE Bus STARTED 2024-02-09T22:51:16.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:15 smithi064 ceph-mon[36638]: mgrmap e22: x(active, since 1.02108s), standbys: z, y 2024-02-09T22:51:16.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/1504533478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc100190 msgr2=0x7fb4cc1005b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/1504533478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc100190 0x7fb4cc1005b0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8009900 tx=0x7fb4c802bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:16.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/1504533478 shutdown_connections 2024-02-09T22:51:16.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/1504533478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4cc101d40 0x7fb4cc1085d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/1504533478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 0x7fb4cc101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/1504533478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc100190 0x7fb4cc1005b0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/1504533478 >> 172.21.15.3:0/1504533478 conn(0x7fb4cc0fb700 msgr2=0x7fb4cc0fdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/1504533478 shutdown_connections 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.247+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/1504533478 wait complete. 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 Processor -- start 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 -- start start 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4cc100190 0x7fb4cc104120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 0x7fb4cc102760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 0x7fb4cc102ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc103400 con 0x7fb4cc101d40 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc103570 con 0x7fb4cc100190 2024-02-09T22:51:16.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d3fc2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4cc1a13f0 con 0x7fb4cc101380 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d255f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 0x7fb4cc102ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d155d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 0x7fb4cc102760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d255f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 0x7fb4cc102ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40156/0 (socket says 172.21.15.3:40156) 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d255f640 1 -- 172.21.15.3:0/2283471133 learned_addr learned my addr 172.21.15.3:0/2283471133 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.248+0000 7fb4d1d5e640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4cc100190 0x7fb4cc104120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d155d640 1 -- 172.21.15.3:0/2283471133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4cc100190 msgr2=0x7fb4cc104120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d155d640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4cc100190 0x7fb4cc104120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d155d640 1 -- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 msgr2=0x7fb4cc102ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d155d640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 0x7fb4cc102ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d155d640 1 -- 172.21.15.3:0/2283471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c8009560 con 0x7fb4cc101380 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d255f640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 0x7fb4cc102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d155d640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 0x7fb4cc102760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00e9f0 tx=0x7fb4bc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc00ccc0 con 0x7fb4cc101380 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4cc1a16a0 con 0x7fb4cc101380 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc00ce20 con 0x7fb4cc101380 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc004a00 con 0x7fb4cc101380 2024-02-09T22:51:16.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.249+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4cc1a1b60 con 0x7fb4cc101380 2024-02-09T22:51:16.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.250+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4cc10bba0 con 0x7fb4cc101380 2024-02-09T22:51:16.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.251+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 22) v1 ==== 136580+0+0 (secure 0 0 0) 0x7fb4bc0050b0 con 0x7fb4cc101380 2024-02-09T22:51:16.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.252+0000 7fb4c2ffd640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4b80a1ea0 0x7fb4b80a4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:16.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.252+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb4bc014070 con 0x7fb4cc101380 2024-02-09T22:51:16.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.252+0000 7fb4d1d5e640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4b80a1ea0 0x7fb4b80a4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:16.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.252+0000 7fb4d1d5e640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4b80a1ea0 0x7fb4b80a4360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8005ab0 tx=0x7fb4c8005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:16.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.258+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb4bc096ae0 con 0x7fb4cc101380 2024-02-09T22:51:16.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:15 smithi003 ceph-mon[32678]: [09/Feb/2024:22:51:14] ENGINE Bus STARTING 2024-02-09T22:51:16.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:15 smithi003 ceph-mon[32678]: [09/Feb/2024:22:51:14] ENGINE Serving on http://172.21.15.3:8765 2024-02-09T22:51:16.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:15 smithi003 ceph-mon[32678]: [09/Feb/2024:22:51:15] ENGINE Serving on https://172.21.15.3:7150 2024-02-09T22:51:16.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:15 smithi003 ceph-mon[32678]: [09/Feb/2024:22:51:15] ENGINE Bus STARTED 2024-02-09T22:51:16.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:15 smithi003 ceph-mon[32678]: mgrmap e22: x(active, since 1.02108s), standbys: z, y 2024-02-09T22:51:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:15 smithi089 ceph-mon[36665]: [09/Feb/2024:22:51:14] ENGINE Bus STARTING 2024-02-09T22:51:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:15 smithi089 ceph-mon[36665]: [09/Feb/2024:22:51:14] ENGINE Serving on http://172.21.15.3:8765 2024-02-09T22:51:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:15 smithi089 ceph-mon[36665]: [09/Feb/2024:22:51:15] ENGINE Serving on https://172.21.15.3:7150 2024-02-09T22:51:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:15 smithi089 ceph-mon[36665]: [09/Feb/2024:22:51:15] ENGINE Bus STARTED 2024-02-09T22:51:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:15 smithi089 ceph-mon[36665]: mgrmap e22: x(active, since 1.02108s), standbys: z, y 2024-02-09T22:51:16.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.486+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 --> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fb4cc0fd900 con 0x7fb4b80a1ea0 2024-02-09T22:51:16.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.493+0000 7fb4c2ffd640 1 -- 172.21.15.3:0/2283471133 <== mgr.15519 v2:172.21.15.3:6800/4119047166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fb4cc0fd900 con 0x7fb4b80a1ea0 2024-02-09T22:51:16.496 INFO:teuthology.orchestra.run.smithi003.stdout:/ 1m 6m3h 2024-02-09T22:51:16.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4b80a1ea0 msgr2=0x7fb4b80a4360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4b80a1ea0 0x7fb4b80a4360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb4c8005ab0 tx=0x7fb4c8005fd0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 msgr2=0x7fb4cc102760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 0x7fb4cc102760 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc00e9f0 tx=0x7fb4bc00eeb0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 shutdown_connections 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4b80a1ea0 0x7fb4b80a4360 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4cc101d40 0x7fb4cc102ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4cc101380 0x7fb4cc102760 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 --2- 172.21.15.3:0/2283471133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4cc100190 0x7fb4cc104120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 >> 172.21.15.3:0/2283471133 conn(0x7fb4cc0fb700 msgr2=0x7fb4cc0fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 shutdown_connections 2024-02-09T22:51:16.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:16.495+0000 7fb4d3fc2640 1 -- 172.21.15.3:0/2283471133 wait complete. 2024-02-09T22:51:16.510 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-02-09T22:51:16.538 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-02-09T22:51:16.550 INFO:teuthology.run_tasks:Running task background_exec... 2024-02-09T22:51:16.559 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T22:51:16.559 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-02-09T22:51:16.581 INFO:teuthology.run_tasks:Running task workunit... 2024-02-09T22:51:16.593 INFO:tasks.workunit:Pulling workunits from ref 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:51:16.593 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-02-09T22:51:16.593 INFO:tasks.workunit:timeout=3h 2024-02-09T22:51:16.594 INFO:tasks.workunit:cleanup=True 2024-02-09T22:51:16.594 DEBUG:teuthology.orchestra.run.smithi003:> stat -- /home/ubuntu/cephtest/mnt.0 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:Access: 2024-02-09 22:49:13.192840758 +0000 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:Modify: 2024-02-09 22:49:13.192840758 +0000 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout:Change: 2024-02-09 22:50:59.721783990 +0000 2024-02-09T22:51:16.609 INFO:teuthology.orchestra.run.smithi003.stdout: Birth: 2024-02-09 22:49:13.192840758 +0000 2024-02-09T22:51:16.610 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-02-09T22:51:16.610 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-02-09T22:51:16.685 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 2024-02-09T22:51:16.740 INFO:tasks.workunit.client.0.smithi003.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-02-09T22:51:17.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:17.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:16 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:16 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='client.15537 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:17.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='client.15576 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:17.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:16 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:51:18.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:17 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:51:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:51:18.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:17 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:51:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='client.25168 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: mgrmap e23: x(active, since 3s), standbys: z, y 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:51:18.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:17 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:51:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-02-09T22:51:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Unable to set osd_memory_target on smithi089 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-02-09T22:51:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi003 to 177.2M 2024-02-09T22:51:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Unable to set osd_memory_target on smithi003 to 185898496: error parsing value: Value '185898496' is below minimum 939524096 2024-02-09T22:51:19.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi064 to 174.1M 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Unable to set osd_memory_target on smithi064 to 182641356: error parsing value: Value '182641356' is below minimum 939524096 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:19.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:51:19.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:51:19.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:18 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Unable to set osd_memory_target on smithi089 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi003 to 177.2M 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Unable to set osd_memory_target on smithi003 to 185898496: error parsing value: Value '185898496' is below minimum 939524096 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi064 to 174.1M 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Unable to set osd_memory_target on smithi064 to 182641356: error parsing value: Value '182641356' is below minimum 939524096 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:51:19.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:18 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Unable to set osd_memory_target on smithi089 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi003 to 177.2M 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Unable to set osd_memory_target on smithi003 to 185898496: error parsing value: Value '185898496' is below minimum 939524096 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi064 to 174.1M 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Unable to set osd_memory_target on smithi064 to 182641356: error parsing value: Value '182641356' is below minimum 939524096 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Updating smithi003:/etc/ceph/ceph.conf 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Updating smithi064:/etc/ceph/ceph.conf 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Updating smithi089:/etc/ceph/ceph.conf 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.conf 2024-02-09T22:51:19.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:51:19.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:18 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:19 smithi003 ceph-mon[32678]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-02-09T22:51:20.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:19 smithi064 ceph-mon[36638]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-02-09T22:51:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: Updating smithi064:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: Updating smithi003:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: Updating smithi089:/etc/ceph/ceph.client.admin.keyring 2024-02-09T22:51:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail 2024-02-09T22:51:20.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: Updating smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: Updating smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: Updating smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/config/ceph.client.admin.keyring 2024-02-09T22:51:20.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:19 smithi089 ceph-mon[36665]: mgrmap e24: x(active, since 4s), standbys: z, y 2024-02-09T22:51:22.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:21 smithi003 ceph-mon[32678]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-09T22:51:22.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:21 smithi064 ceph-mon[36638]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-09T22:51:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:21 smithi089 ceph-mon[36665]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 1 op/s 2024-02-09T22:51:24.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:24 smithi003 ceph-mon[32678]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-02-09T22:51:24.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:24 smithi064 ceph-mon[36638]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-02-09T22:51:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:24 smithi089 ceph-mon[36665]: pgmap v7: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 68 KiB/s rd, 132 KiB/s wr, 80 op/s 2024-02-09T22:51:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:26 smithi003 ceph-mon[32678]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-02-09T22:51:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:26 smithi064 ceph-mon[36638]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-02-09T22:51:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:26 smithi089 ceph-mon[36665]: pgmap v8: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 103 KiB/s wr, 62 op/s 2024-02-09T22:51:28.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:28 smithi003 ceph-mon[32678]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-02-09T22:51:28.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:28 smithi064 ceph-mon[36638]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-02-09T22:51:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:28 smithi089 ceph-mon[36665]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 84 KiB/s wr, 55 op/s 2024-02-09T22:51:29.717 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 -- 172.21.15.3:0/290090666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe770106510 msgr2=0x7fe77010e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 --2- 172.21.15.3:0/290090666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe770106510 0x7fe77010e0e0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fe768009cd0 tx=0x7fe76802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 -- 172.21.15.3:0/290090666 shutdown_connections 2024-02-09T22:51:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 --2- 172.21.15.3:0/290090666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe770106510 0x7fe77010e0e0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 --2- 172.21.15.3:0/290090666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe7700696f0 0x7fe770105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 --2- 172.21.15.3:0/290090666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe770068d40 0x7fe770069120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 -- 172.21.15.3:0/290090666 >> 172.21.15.3:0/290090666 conn(0x7fe770077220 msgr2=0x7fe770077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 -- 172.21.15.3:0/290090666 shutdown_connections 2024-02-09T22:51:29.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.714+0000 7fe774b0d640 1 -- 172.21.15.3:0/290090666 wait complete. 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 Processor -- start 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 -- start start 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe770068d40 0x7fe77019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe770106510 0x7fe77019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 0x7fe770197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770198170 con 0x7fe7701976a0 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7701982e0 con 0x7fe770106510 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe774b0d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7701984a0 con 0x7fe770068d40 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.715+0000 7fe76ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 0x7fe770197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 0x7fe770197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53230/0 (socket says 172.21.15.3:53230) 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 -- 172.21.15.3:0/1320852424 learned_addr learned my addr 172.21.15.3:0/1320852424 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe770106510 0x7fe77019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe770068d40 0x7fe77019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 -- 172.21.15.3:0/1320852424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe770068d40 msgr2=0x7fe77019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe770068d40 0x7fe77019d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 -- 172.21.15.3:0/1320852424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe770106510 msgr2=0x7fe77019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe770106510 0x7fe77019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 -- 172.21.15.3:0/1320852424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009900 con 0x7fe7701976a0 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 0x7fe770197b20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe768004af0 tx=0x7fe768034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe75f7fe640 1 -- 172.21.15.3:0/1320852424 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680328c0 con 0x7fe7701976a0 2024-02-09T22:51:29.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7701a43e0 con 0x7fe7701976a0 2024-02-09T22:51:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe75f7fe640 1 -- 172.21.15.3:0/1320852424 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe768003e50 con 0x7fe7701976a0 2024-02-09T22:51:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe75f7fe640 1 -- 172.21.15.3:0/1320852424 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768004860 con 0x7fe7701976a0 2024-02-09T22:51:29.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.716+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7701a48b0 con 0x7fe7701976a0 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.719+0000 7fe75f7fe640 1 -- 172.21.15.3:0/1320852424 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7fe76800a040 con 0x7fe7701976a0 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.719+0000 7fe7777b9740 1 Processor -- start 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.719+0000 7fe7777b9740 1 -- start start 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe75f7fe640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7580a22b0 0x7fe7580a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d3086490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d3091b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7580a22b0 0x7fe7580a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe7777b9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d2ff4a00 con 0x5629d30869d0 2024-02-09T22:51:29.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe7777b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d2ff1f30 con 0x5629d30860b0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe7777b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d2fed710 con 0x5629d308f7a0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe75f7fe640 1 -- 172.21.15.3:0/1320852424 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe7680f0e80 con 0x7fe7701976a0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d3086490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53244/0 (socket says 172.21.15.3:53244) 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76dd9b640 1 -- 172.21.15.3:0/1079732124 learned_addr learned my addr 172.21.15.3:0/1079732124 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d3091b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76e59c640 1 -- 172.21.15.3:0/1079732124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 msgr2=0x5629d3091b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d3091b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76e59c640 1 -- 172.21.15.3:0/1079732124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 msgr2=0x5629d308f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76e59c640 1 -- 172.21.15.3:0/1079732124 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629d2fb23b0 con 0x5629d30860b0 2024-02-09T22:51:29.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.720+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7580a22b0 0x7fe7580a4770 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe764013460 tx=0x7fe764013860 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.721+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d3086490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe76000bbe0 tx=0x7fe76000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f164dd3467fbdb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.721+0000 7fe75cff9640 1 -- 172.21.15.3:0/1079732124 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760004020 con 0x5629d30860b0 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.721+0000 7fe75cff9640 1 -- 172.21.15.3:0/1079732124 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe760004180 con 0x5629d30860b0 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.721+0000 7fe75cff9640 1 -- 172.21.15.3:0/1079732124 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7600116a0 con 0x5629d30860b0 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- 172.21.15.3:0/1079732124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 msgr2=0x5629d3086490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d3086490 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe76000bbe0 tx=0x7fe76000f730 comp rx=0 tx=0).stop 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- 172.21.15.3:0/1079732124 shutdown_connections 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d3091b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1079732124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d3086490 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- 172.21.15.3:0/1079732124 >> 172.21.15.3:0/1079732124 conn(0x5629d30809d0 msgr2=0x5629d3080db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- 172.21.15.3:0/1079732124 shutdown_connections 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- 172.21.15.3:0/1079732124 wait complete. 2024-02-09T22:51:29.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- start start 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 Processor -- start 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d308c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30869d0 0x5629d308cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 0x5629d308d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.722+0000 7fe7777b9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d2f92b90 con 0x5629d308f7a0 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe7777b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d2cc1250 con 0x5629d30860b0 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe7777b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d2f73200 con 0x5629d30869d0 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe7777b9740 10 client.0 _ll_register_callbacks cb 0x5629d2fa95d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 0x5629d308d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d308c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 0x5629d308d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53258/0 (socket says 172.21.15.3:53258) 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 -- 172.21.15.3:0/2421693746 learned_addr learned my addr 172.21.15.3:0/2421693746 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:29.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30869d0 0x5629d308cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 -- 172.21.15.3:0/2421693746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30869d0 msgr2=0x5629d308cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30869d0 0x5629d308cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 -- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 msgr2=0x5629d308c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d308c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 -- 172.21.15.3:0/2421693746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76000b810 con 0x5629d308f7a0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76e59c640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d308c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 0x5629d308d270 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe7680ba210 tx=0x7fe7680bd6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=927bfa9d49176bd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe755ffb640 1 -- 172.21.15.3:0/2421693746 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7680bd840 con 0x5629d308f7a0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe755ffb640 1 -- 172.21.15.3:0/2421693746 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe7680c1990 con 0x5629d308f7a0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.723+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5629d3013ee0 con 0x5629d308f7a0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe755ffb640 1 -- 172.21.15.3:0/2421693746 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7680c5fe0 con 0x5629d308f7a0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 10 client.15606 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5629d2febd90 con 0x5629d308f7a0 2024-02-09T22:51:29.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe755ffb640 1 -- 172.21.15.3:0/2421693746 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7680da080 con 0x5629d308f7a0 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe755ffb640 1 -- 172.21.15.3:0/2421693746 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe7680bdaa0 con 0x5629d308f7a0 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 10 client.15606 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 20 client.15606 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d309f860 0x5629d30a1c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 4 client.15606 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629d2ced600 con 0x5629d309f860 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.724+0000 7fe76e59c640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d309f860 0x5629d30a1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe755ffb640 10 client.15606 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe76e59c640 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d309f860 0x5629d30a1c50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1efc2a73fe9fce9 server_cookie=eab5160b82a404b6 in_seq=0 out_seq=0 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe755ffb640 1 -- 172.21.15.3:0/2421693746 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5629d2ced600 con 0x5629d309f860 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe755ffb640 10 client.15606 handle_command_reply: tid=0 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 client.15606 shutdown 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 msgr2=0x5629d308d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 0x5629d308d270 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fe7680ba210 tx=0x7fe7680bd6d0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 shutdown_connections 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d309f860 0x5629d30a1c50 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d308f7a0 0x5629d308d270 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30869d0 0x5629d308cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe755ffb640 0 client.15606 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 --2- 172.21.15.3:0/2421693746 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d308c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.725+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 >> 172.21.15.3:0/2421693746 conn(0x5629d30809d0 msgr2=0x5629d3082720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.726+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 shutdown_connections 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.726+0000 7fe7777b9740 1 -- 172.21.15.3:0/2421693746 wait complete. 2024-02-09T22:51:29.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.726+0000 7fe7777b9740 20 client.15606 trim_cache size 0 max 16384 2024-02-09T22:51:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 Processor -- start 2024-02-09T22:51:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 -- start start 2024-02-09T22:51:29.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30860b0 0x5629d308ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d308f7a0 0x5629d308b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d307c310 con 0x5629d30869d0 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d309c4d0 con 0x5629d308f7a0 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe7777b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d307aec0 con 0x5629d30860b0 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53266/0 (socket says 172.21.15.3:53266) 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 -- 172.21.15.3:0/4111979582 learned_addr learned my addr 172.21.15.3:0/4111979582 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d308f7a0 0x5629d308b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76e59c640 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30860b0 0x5629d308ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 -- 172.21.15.3:0/4111979582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30860b0 msgr2=0x5629d308ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30860b0 0x5629d308ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 -- 172.21.15.3:0/4111979582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d308f7a0 msgr2=0x5629d308b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d308f7a0 0x5629d308b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.742+0000 7fe76dd9b640 1 -- 172.21.15.3:0/4111979582 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629d2f88820 con 0x5629d30869d0 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308d090 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe764013fd0 tx=0x7fe76401f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da11f8493b42d1ec server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe757fff640 1 -- 172.21.15.3:0/4111979582 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764025070 con 0x5629d30869d0 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe757fff640 1 -- 172.21.15.3:0/4111979582 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe764012110 con 0x5629d30869d0 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe757fff640 1 -- 172.21.15.3:0/4111979582 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764018040 con 0x5629d30869d0 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 -- 172.21.15.3:0/4111979582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 msgr2=0x5629d308d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308d090 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fe764013fd0 tx=0x7fe76401f000 comp rx=0 tx=0).stop 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 -- 172.21.15.3:0/4111979582 shutdown_connections 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d308f7a0 0x5629d308b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d308d090 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 --2- 172.21.15.3:0/4111979582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d30860b0 0x5629d308ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.743+0000 7fe7777b9740 1 -- 172.21.15.3:0/4111979582 >> 172.21.15.3:0/4111979582 conn(0x5629d30809d0 msgr2=0x5629d3087ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 -- 172.21.15.3:0/4111979582 shutdown_connections 2024-02-09T22:51:29.751 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 -- 172.21.15.3:0/4111979582 wait complete. 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 -- start start 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 Processor -- start 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d30947b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d3094cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d30eed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d30ef2b0 con 0x5629d30869d0 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d30ef420 con 0x5629d30860b0 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629d30ef5e0 con 0x5629d308f7a0 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe7777b9740 10 client.0 _ll_register_callbacks cb 0x7fe768009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d3094cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d3094cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53268/0 (socket says 172.21.15.3:53268) 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 -- 172.21.15.3:0/1347053785 learned_addr learned my addr 172.21.15.3:0/1347053785 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:29.752 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d30947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76ed9d640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d30eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 -- 172.21.15.3:0/1347053785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 msgr2=0x5629d30eed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d30eed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 -- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 msgr2=0x5629d30947b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.744+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d30947b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe76dd9b640 1 -- 172.21.15.3:0/1347053785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe764004330 con 0x5629d30869d0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe76dd9b640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d3094cf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe764004090 tx=0x7fe764005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0d1e85e4698ed81 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe755ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe755ffb640 1 -- 172.21.15.3:0/1347053785 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764018070 con 0x5629d30869d0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe755ffb640 1 -- 172.21.15.3:0/1347053785 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe764009c10 con 0x5629d30869d0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe760004180 con 0x5629d30869d0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe755ffb640 1 -- 172.21.15.3:0/1347053785 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe764025070 con 0x5629d30869d0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 10 client.15612 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5629d2fae6f0 con 0x5629d30869d0 2024-02-09T22:51:29.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe755ffb640 1 -- 172.21.15.3:0/1347053785 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe760004180 con 0x5629d30869d0 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe755ffb640 1 -- 172.21.15.3:0/1347053785 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe764020450 con 0x5629d30869d0 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 10 client.15612 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 10 client.15612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 20 client.15612 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d30f6eb0 0x5629d30f92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 4 client.15612 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.745+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5629d2d3e1b0 con 0x5629d30f6eb0 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.746+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d30f6eb0 0x5629d30f92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.746+0000 7fe76e59c640 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d30f6eb0 0x5629d30f92d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=916c42f86f92e60b server_cookie=b5d2bcc86587f327 in_seq=0 out_seq=0 2024-02-09T22:51:29.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.746+0000 7fe755ffb640 10 client.15612 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe755ffb640 1 -- 172.21.15.3:0/1347053785 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5629d2d3e1b0 con 0x5629d30f6eb0 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe755ffb640 10 client.15612 handle_command_reply: tid=0 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 client.15612 shutdown 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 msgr2=0x5629d3094cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d3094cf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fe764004090 tx=0x7fe764005f10 comp rx=0 tx=0).stop 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 shutdown_connections 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629d30f6eb0 0x5629d30f92d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5629d308f7a0 0x5629d30eed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5629d30869d0 0x5629d3094cf0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe755ffb640 0 client.15612 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 --2- 172.21.15.3:0/1347053785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5629d30860b0 0x5629d30947b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 >> 172.21.15.3:0/1347053785 conn(0x5629d30809d0 msgr2=0x5629d307f1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:29.762 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 shutdown_connections 2024-02-09T22:51:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 1 -- 172.21.15.3:0/1347053785 wait complete. 2024-02-09T22:51:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.758+0000 7fe7777b9740 20 client.15612 trim_cache size 0 max 16384 2024-02-09T22:51:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:51:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:51:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:51:29.763 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:51:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7580a22b0 msgr2=0x7fe7580a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7580a22b0 0x7fe7580a4770 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe764013460 tx=0x7fe764013860 comp rx=0 tx=0).stop 2024-02-09T22:51:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 msgr2=0x7fe770197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 0x7fe770197b20 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fe768004af0 tx=0x7fe768034cc0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 shutdown_connections 2024-02-09T22:51:29.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7580a22b0 0x7fe7580a4770 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe770106510 0x7fe77019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe7701976a0 0x7fe770197b20 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 --2- 172.21.15.3:0/1320852424 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe770068d40 0x7fe77019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 >> 172.21.15.3:0/1320852424 conn(0x7fe770077220 msgr2=0x7fe7701076e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 shutdown_connections 2024-02-09T22:51:29.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:29.759+0000 7fe774b0d640 1 -- 172.21.15.3:0/1320852424 wait complete. 2024-02-09T22:51:30.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:30 smithi003 ceph-mon[32678]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-02-09T22:51:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:30 smithi064 ceph-mon[36638]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-02-09T22:51:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:30 smithi089 ceph-mon[36665]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 50 op/s 2024-02-09T22:51:32.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:32 smithi064 ceph-mon[36638]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-02-09T22:51:32.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:32 smithi089 ceph-mon[36665]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-02-09T22:51:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:32 smithi003 ceph-mon[32678]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 77 KiB/s wr, 51 op/s 2024-02-09T22:51:34.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:34 smithi064 ceph-mon[36638]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 58 op/s 2024-02-09T22:51:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:34 smithi089 ceph-mon[36665]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 58 op/s 2024-02-09T22:51:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:34 smithi003 ceph-mon[32678]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 79 KiB/s wr, 58 op/s 2024-02-09T22:51:36.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:36 smithi064 ceph-mon[36638]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-02-09T22:51:36.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:36 smithi089 ceph-mon[36665]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-02-09T22:51:36.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:51:36.620 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:51:36.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:36 smithi003 ceph-mon[32678]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-02-09T22:51:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.784+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2072831761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0105410 msgr2=0x7f2cb010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.784+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2072831761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0105410 0x7f2cb010cfe0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009cd0 tx=0x7f2ca802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.784+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2072831761 shutdown_connections 2024-02-09T22:51:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.784+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2072831761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0105410 0x7f2cb010cfe0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.784+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2072831761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cb0100cf0 0x7f2cb0104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.784+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2072831761 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0100340 0x7f2cb0100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.785+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2072831761 >> 172.21.15.3:0/2072831761 conn(0x7f2cb00772e0 msgr2=0x7f2cb00776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:36.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.785+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2072831761 shutdown_connections 2024-02-09T22:51:36.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.785+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2072831761 wait complete. 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.786+0000 7f2cb63e9640 1 Processor -- start 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.786+0000 7f2cb63e9640 1 -- start start 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.786+0000 7f2cb63e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 0x7f2cb019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2cb63e9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0105410 0x7f2cb019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2cb63e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cb0197730 0x7f2cb0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 0x7f2cb019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 0x7f2cb019d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42776/0 (socket says 172.21.15.3:42776) 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 -- 172.21.15.3:0/2346922987 learned_addr learned my addr 172.21.15.3:0/2346922987 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0198200 con 0x7f2cb0100340 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0198370 con 0x7f2cb0105410 2024-02-09T22:51:36.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0198530 con 0x7f2cb0197730 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0105410 0x7f2cb019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2cb4986640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cb0197730 0x7f2cb0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 -- 172.21.15.3:0/2346922987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cb0197730 msgr2=0x7f2cb0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cb0197730 0x7f2cb0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 -- 172.21.15.3:0/2346922987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0105410 msgr2=0x7f2cb019dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0105410 0x7f2cb019dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caffff640 1 -- 172.21.15.3:0/2346922987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca8009900 con 0x7f2cb0100340 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.787+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0105410 0x7f2cb019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.788+0000 7f2caffff640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 0x7f2cb019d670 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000d950 tx=0x7f2ca000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.788+0000 7f2cad7fa640 1 -- 172.21.15.3:0/2346922987 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca0009c30 con 0x7f2cb0100340 2024-02-09T22:51:36.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.788+0000 7f2cad7fa640 1 -- 172.21.15.3:0/2346922987 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ca0009d90 con 0x7f2cb0100340 2024-02-09T22:51:36.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.788+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb01a44d0 con 0x7f2cb0100340 2024-02-09T22:51:36.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.788+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb01a4a00 con 0x7f2cb0100340 2024-02-09T22:51:36.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.789+0000 7f2cad7fa640 1 -- 172.21.15.3:0/2346922987 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000f7e0 con 0x7f2cb0100340 2024-02-09T22:51:36.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.791+0000 7f2cad7fa640 1 -- 172.21.15.3:0/2346922987 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f2ca000f940 con 0x7f2cb0100340 2024-02-09T22:51:36.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.791+0000 7f2cb8fc4740 1 Processor -- start 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 -- start start 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228630 0x55e1a5228a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 0x55e1a523a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a523cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a51e3d00 con 0x55e1a523aa00 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a51982f0 con 0x55e1a5228630 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a5195820 con 0x55e1a5228f50 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a523cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 0x55e1a523a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb4986640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228630 0x55e1a5228a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 0x55e1a523a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35380/0 (socket says 172.21.15.3:35380) 2024-02-09T22:51:36.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cb4986640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228630 0x55e1a5228a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53416/0 (socket says 172.21.15.3:53416) 2024-02-09T22:51:36.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caffff640 1 -- 172.21.15.3:0/2506871092 learned_addr learned my addr 172.21.15.3:0/2506871092 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caffff640 1 -- 172.21.15.3:0/2506871092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228630 msgr2=0x55e1a5228a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caffff640 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228630 0x55e1a5228a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2caffff640 1 -- 172.21.15.3:0/2506871092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 msgr2=0x55e1a523cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.792+0000 7f2cad7fa640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2c980a2340 0x7f2c980a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2caffff640 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a523cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2caffff640 1 -- 172.21.15.3:0/2506871092 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1a5152ff0 con 0x55e1a5228f50 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a523cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2cb4986640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2c980a2340 0x7f2c980a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2cad7fa640 1 -- 172.21.15.3:0/2346922987 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2ca00cdfa0 con 0x7f2cb0100340 2024-02-09T22:51:36.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2caffff640 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 0x55e1a523a4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00cb570 tx=0x7f2ca00659e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60e06b0a57a6ba81 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:36.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2c967fc640 1 -- 172.21.15.3:0/2506871092 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca009a3e0 con 0x55e1a5228f50 2024-02-09T22:51:36.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2c967fc640 1 -- 172.21.15.3:0/2506871092 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ca009a200 con 0x55e1a5228f50 2024-02-09T22:51:36.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.793+0000 7f2cb4986640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2c980a2340 0x7f2c980a4800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8005cc0 tx=0x7f2ca8005c50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:36.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.794+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/2506871092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 msgr2=0x55e1a523a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.794+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 0x55e1a523a4c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2ca00cb570 tx=0x7f2ca00659e0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.794+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/2506871092 shutdown_connections 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a523cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228f50 0x55e1a523a4c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/2506871092 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228630 0x55e1a5228a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/2506871092 >> 172.21.15.3:0/2506871092 conn(0x55e1a5223050 msgr2=0x55e1a5223430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/2506871092 shutdown_connections 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/2506871092 wait complete. 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.795+0000 7f2cb8fc4740 1 -- start start 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.796+0000 7f2cb8fc4740 1 Processor -- start 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.796+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5234070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228f50 0x55e1a52345b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a5239d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5234070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5234070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35384/0 (socket says 172.21.15.3:35384) 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2caffff640 1 -- 172.21.15.3:0/1799580462 learned_addr learned my addr 172.21.15.3:0/1799580462 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2cb4986640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a5239d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228f50 0x55e1a52345b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a5191000 con 0x55e1a523aa00 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a4e64430 con 0x55e1a5228f50 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.797+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a4e20e00 con 0x55e1a5228630 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2cb8fc4740 10 client.0 _ll_register_callbacks cb 0x55e1a51857f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2caffff640 1 -- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228f50 msgr2=0x55e1a52345b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2caffff640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228f50 0x55e1a52345b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2caffff640 1 -- 172.21.15.3:0/1799580462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 msgr2=0x55e1a5239d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2caffff640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a5239d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2caffff640 1 -- 172.21.15.3:0/1799580462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb01a4a00 con 0x55e1a5228630 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2cb4986640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a5239d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:36.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2caffff640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5234070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0000c00 tx=0x7f2ca0096880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bffd8e43bf1abdc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2c8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1799580462 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca009a3e0 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1799580462 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2ca009a8a0 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.798+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e1a4e907e0 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1799580462 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ca00d1a50 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 10 client.25204 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e1a518f680 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1799580462 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e1a4e907e0 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1799580462 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2ca0023070 con 0x55e1a5228630 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 10 client.25204 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 10 client.25204 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 20 client.25204 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a5252ea0 0x55e1a52552e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 4 client.25204 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:36.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.799+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e1a4ecd720 con 0x55e1a5252ea0 2024-02-09T22:51:36.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.800+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a5252ea0 0x55e1a52552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.800+0000 7f2c8f7fe640 10 client.25204 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.800+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a5252ea0 0x55e1a52552e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8d081af5ea5ba1e server_cookie=a60de5bf1c97ddbb in_seq=0 out_seq=0 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1799580462 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e1a4ecd720 con 0x55e1a5252ea0 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2c8f7fe640 10 client.25204 handle_command_reply: tid=0 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 client.25204 shutdown 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 msgr2=0x55e1a5234070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5234070 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0000c00 tx=0x7f2ca0096880 comp rx=0 tx=0).stop 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 shutdown_connections 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a5252ea0 0x55e1a52552e0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a5228f50 0x55e1a52345b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2c8f7fe640 0 client.25204 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a523aa00 0x55e1a5239d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1799580462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5234070 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 >> 172.21.15.3:0/1799580462 conn(0x55e1a5223050 msgr2=0x55e1a5224da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 shutdown_connections 2024-02-09T22:51:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1799580462 wait complete. 2024-02-09T22:51:36.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.801+0000 7f2cb8fc4740 20 client.25204 trim_cache size 0 max 16384 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.831+0000 7f2cb8fc4740 1 Processor -- start 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.831+0000 7f2cb8fc4740 1 -- start start 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5239d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a522be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a522c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a521e780 con 0x55e1a5228f50 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a5115fd0 con 0x55e1a523aa00 2024-02-09T22:51:36.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb8fc4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a52308d0 con 0x55e1a5228630 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5239d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a522be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5239d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35396/0 (socket says 172.21.15.3:35396) 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 -- 172.21.15.3:0/4108099287 learned_addr learned my addr 172.21.15.3:0/4108099287 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2cb4986640 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a522c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 -- 172.21.15.3:0/4108099287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 msgr2=0x55e1a522c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a522c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 -- 172.21.15.3:0/4108099287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 msgr2=0x55e1a522be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a522be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.832+0000 7f2caffff640 1 -- 172.21.15.3:0/4108099287 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e1a512bb30 con 0x55e1a5228630 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.833+0000 7f2caffff640 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5239d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000f430 tx=0x7f2ca00cb710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a05199a19936bed2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.833+0000 7f2c95ffb640 1 -- 172.21.15.3:0/4108099287 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00d1860 con 0x55e1a5228630 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.833+0000 7f2c95ffb640 1 -- 172.21.15.3:0/4108099287 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ca00cb540 con 0x55e1a5228630 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.833+0000 7f2c95ffb640 1 -- 172.21.15.3:0/4108099287 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00a2d40 con 0x55e1a5228630 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/4108099287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 msgr2=0x55e1a5239d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5239d80 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000f430 tx=0x7f2ca00cb710 comp rx=0 tx=0).stop 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/4108099287 shutdown_connections 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a522c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a522be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/4108099287 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a5239d80 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/4108099287 >> 172.21.15.3:0/4108099287 conn(0x55e1a5223050 msgr2=0x55e1a522e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/4108099287 shutdown_connections 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.834+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/4108099287 wait complete. 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.835+0000 7f2cb8fc4740 1 -- start start 2024-02-09T22:51:36.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.835+0000 7f2cb8fc4740 1 Processor -- start 2024-02-09T22:51:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.835+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a52a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.835+0000 7f2cb8fc4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a52a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a52a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a52a2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42810/0 (socket says 172.21.15.3:42810) 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a52a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caf7fe640 1 -- 172.21.15.3:0/1336271386 learned_addr learned my addr 172.21.15.3:0/1336271386 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a529d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a529d9a0 con 0x55e1a5228f50 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a529db10 con 0x55e1a523aa00 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e1a529dcd0 con 0x55e1a5228630 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb4986640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a529d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caffff640 1 -- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 msgr2=0x55e1a529d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caffff640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a529d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caffff640 1 -- 172.21.15.3:0/1336271386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 msgr2=0x55e1a52a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caffff640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a52a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caffff640 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1a518f680 con 0x55e1a5228630 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb4986640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a529d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a52a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:36.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.836+0000 7f2cb8fc4740 10 client.0 _ll_register_callbacks cb 0x7f2ca00cb490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e1a4f1c890 con 0x55e1a5228630 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2caffff640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a52a2aa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0098b50 tx=0x7f2ca00d1eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d39267ba9369d4eb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2c8f7fe640 10 client.25210 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1336271386 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca00d1860 con 0x55e1a5228630 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1336271386 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2ca0085280 con 0x55e1a5228630 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1336271386 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2ca00d1a50 con 0x55e1a5228630 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 10 client.25210 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55e1a5151850 con 0x55e1a5228630 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1336271386 <== mon.1 v2:172.21.15.89:3300/0 4 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2ca0023030 con 0x55e1a5228630 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 10 client.25210 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 10 client.25210 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 20 client.25210 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a52aa240 0x55e1a52ac700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:36.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 4 client.25210 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.837+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e1a4e3c700 con 0x55e1a52aa240 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.838+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a52aa240 0x55e1a52ac700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.838+0000 7f2c8f7fe640 10 client.25210 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.838+0000 7f2caf7fe640 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a52aa240 0x55e1a52ac700 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daff98c119109d90 server_cookie=74047ab554181837 in_seq=0 out_seq=0 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.838+0000 7f2c8f7fe640 1 -- 172.21.15.3:0/1336271386 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e1a4e3c700 con 0x55e1a52aa240 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.838+0000 7f2c8f7fe640 10 client.25210 handle_command_reply: tid=0 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.838+0000 7f2cb8fc4740 1 client.25210 shutdown 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 msgr2=0x55e1a52a2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a52a2aa0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0098b50 tx=0x7f2ca00d1eb0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 shutdown_connections 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e1a52aa240 0x55e1a52ac700 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e1a523aa00 0x55e1a529d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e1a5228f50 0x55e1a52a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 --2- 172.21.15.3:0/1336271386 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e1a5228630 0x55e1a52a2aa0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2c8f7fe640 0 client.25210 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 >> 172.21.15.3:0/1336271386 conn(0x55e1a5223050 msgr2=0x55e1a5221960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 shutdown_connections 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 1 -- 172.21.15.3:0/1336271386 wait complete. 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb8fc4740 20 client.25210 trim_cache size 0 max 16384 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:51:36.846 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:51:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2c980a2340 msgr2=0x7f2c980a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2c980a2340 0x7f2c980a4800 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8005cc0 tx=0x7f2ca8005c50 comp rx=0 tx=0).stop 2024-02-09T22:51:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 msgr2=0x7f2cb019d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.839+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 0x7f2cb019d670 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f2ca000d950 tx=0x7f2ca000de10 comp rx=0 tx=0).stop 2024-02-09T22:51:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 shutdown_connections 2024-02-09T22:51:36.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2c980a2340 0x7f2c980a4800 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2cb0105410 0x7f2cb019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2cb0197730 0x7f2cb0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 --2- 172.21.15.3:0/2346922987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2cb0100340 0x7f2cb019d670 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:36.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 >> 172.21.15.3:0/2346922987 conn(0x7f2cb00772e0 msgr2=0x7f2cb00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:36.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 shutdown_connections 2024-02-09T22:51:36.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:36.840+0000 7f2cb63e9640 1 -- 172.21.15.3:0/2346922987 wait complete. 2024-02-09T22:51:36.858 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:51:36.858 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:51:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 -- 172.21.15.3:0/1786107255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c100cf0 msgr2=0x7f616c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 --2- 172.21.15.3:0/1786107255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c100cf0 0x7f616c104d30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f615c009900 tx=0x7f615c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 -- 172.21.15.3:0/1786107255 shutdown_connections 2024-02-09T22:51:37.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 --2- 172.21.15.3:0/1786107255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 0x7f616c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 --2- 172.21.15.3:0/1786107255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c100cf0 0x7f616c104d30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 --2- 172.21.15.3:0/1786107255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 -- 172.21.15.3:0/1786107255 >> 172.21.15.3:0/1786107255 conn(0x7f616c077220 msgr2=0x7f616c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 -- 172.21.15.3:0/1786107255 shutdown_connections 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.055+0000 7f6173f47640 1 -- 172.21.15.3:0/1786107255 wait complete. 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 Processor -- start 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 -- start start 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c1967a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 0x7f616c196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c19b010 0x7f616c19b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19ba00 con 0x7f616c100340 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19bb70 con 0x7f616c105410 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6173f47640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19bce0 con 0x7f616c19b010 2024-02-09T22:51:37.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6171ce3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c1967a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6171ce3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c1967a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42830/0 (socket says 172.21.15.3:42830) 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f6171ce3640 1 -- 172.21.15.3:0/2337994389 learned_addr learned my addr 172.21.15.3:0/2337994389 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f61714e2640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 0x7f616c196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.056+0000 7f61724e4640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c19b010 0x7f616c19b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f61714e2640 1 -- 172.21.15.3:0/2337994389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c19b010 msgr2=0x7f616c19b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f61714e2640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c19b010 0x7f616c19b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f61714e2640 1 -- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 msgr2=0x7f616c1967a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f61714e2640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c1967a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f61714e2640 1 -- 172.21.15.3:0/2337994389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6168009490 con 0x7f616c105410 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f6171ce3640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c1967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f61714e2640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 0x7f616c196ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f615c036040 tx=0x7f615c02c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c02dea0 con 0x7f616c105410 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615c009560 con 0x7f616c105410 2024-02-09T22:51:37.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f615c040070 con 0x7f616c105410 2024-02-09T22:51:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0696a0 con 0x7f616c105410 2024-02-09T22:51:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.057+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f615c003c20 con 0x7f616c105410 2024-02-09T22:51:37.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.059+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c101810 con 0x7f616c105410 2024-02-09T22:51:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.060+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f615c004df0 con 0x7f616c105410 2024-02-09T22:51:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.061+0000 7f6162ffd640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2340 0x7f61580a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.061+0000 7f6171ce3640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2340 0x7f61580a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.061+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f615c042070 con 0x7f616c105410 2024-02-09T22:51:37.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.062+0000 7f6171ce3640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2340 0x7f61580a4800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6168011690 tx=0x7f6168005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.065+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f615c0baf90 con 0x7f616c105410 2024-02-09T22:51:37.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.331+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f616c06a130 con 0x7f616c105410 2024-02-09T22:51:37.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.332+0000 7f6162ffd640 1 -- 172.21.15.3:0/2337994389 <== mon.2 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f615c0be650 con 0x7f616c105410 2024-02-09T22:51:37.335 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:51:37.336 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:51:00.398692+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:51:37.336 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.334+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2340 msgr2=0x7f61580a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.334+0000 7f6173f47640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2340 0x7f61580a4800 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6168011690 tx=0x7f6168005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.334+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 msgr2=0x7f616c196ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.334+0000 7f6173f47640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 0x7f616c196ce0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f615c036040 tx=0x7f615c02c720 comp rx=0 tx=0).stop 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 shutdown_connections 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2340 0x7f61580a4800 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c105410 0x7f616c196ce0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c19b010 0x7f616c19b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 --2- 172.21.15.3:0/2337994389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c100340 0x7f616c1967a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 >> 172.21.15.3:0/2337994389 conn(0x7f616c077220 msgr2=0x7f616c0fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 shutdown_connections 2024-02-09T22:51:37.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.335+0000 7f6173f47640 1 -- 172.21.15.3:0/2337994389 wait complete. 2024-02-09T22:51:37.349 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:51:37.349 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 -- 172.21.15.3:0/3568070094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b3810b820 msgr2=0x7f5b3810dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/3568070094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b3810b820 0x7f5b3810dc50 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b3002c180 comp rx=0 tx=0).stop 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 -- 172.21.15.3:0/3568070094 shutdown_connections 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/3568070094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b38102d70 0x7f5b3810b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/3568070094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b3810b820 0x7f5b3810dc50 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/3568070094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b38102450 0x7f5b38102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 -- 172.21.15.3:0/3568070094 >> 172.21.15.3:0/3568070094 conn(0x7f5b380772e0 msgr2=0x7f5b380776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 -- 172.21.15.3:0/3568070094 shutdown_connections 2024-02-09T22:51:37.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.451+0000 7f5b3e82a640 1 -- 172.21.15.3:0/3568070094 wait complete. 2024-02-09T22:51:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.452+0000 7f5b3e82a640 1 Processor -- start 2024-02-09T22:51:37.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.452+0000 7f5b3e82a640 1 -- start start 2024-02-09T22:51:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.452+0000 7f5b3e82a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b38102450 0x7f5b381000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.452+0000 7f5b3e82a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 0x7f5b381005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b3e82a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b3810b820 0x7f5b38100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b3e82a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b381a5bc0 con 0x7f5b38102d70 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b3e82a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b381a5d30 con 0x7f5b3810b820 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b3e82a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b381a5ef0 con 0x7f5b38102450 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 0x7f5b381005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 0x7f5b381005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42836/0 (socket says 172.21.15.3:42836) 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b37fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b38102450 0x7f5b381000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 -- 172.21.15.3:0/890384568 learned_addr learned my addr 172.21.15.3:0/890384568 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b3810b820 0x7f5b38100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 -- 172.21.15.3:0/890384568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b38102450 msgr2=0x7f5b381000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b38102450 0x7f5b381000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 -- 172.21.15.3:0/890384568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b3810b820 msgr2=0x7f5b38100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b3810b820 0x7f5b38100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 -- 172.21.15.3:0/890384568 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30009900 con 0x7f5b38102d70 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b377fe640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 0x7f5b381005e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00d950 tx=0x7f5b2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.453+0000 7f5b357fa640 1 -- 172.21.15.3:0/890384568 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c009a30 con 0x7f5b38102d70 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.454+0000 7f5b357fa640 1 -- 172.21.15.3:0/890384568 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b2c009b90 con 0x7f5b38102d70 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.454+0000 7f5b357fa640 1 -- 172.21.15.3:0/890384568 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c004a00 con 0x7f5b38102d70 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.454+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b381a61d0 con 0x7f5b38102d70 2024-02-09T22:51:37.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.454+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b381a6690 con 0x7f5b38102d70 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.456+0000 7f5b357fa640 1 -- 172.21.15.3:0/890384568 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f5b2c009d00 con 0x7f5b38102d70 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.456+0000 7f5b41400740 1 Processor -- start 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.456+0000 7f5b41400740 1 -- start start 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b41400740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92dee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b357fa640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5b200a23f0 0x7f5b200a48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b41400740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b41400740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92e7f60 0x55fea92ea3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b41400740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea929a020 con 0x55fea92df3e0 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b41400740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea924d320 con 0x55fea92dea30 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b41400740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea924a850 con 0x55fea92e7f60 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92dee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92dee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53498/0 (socket says 172.21.15.3:53498) 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 -- 172.21.15.3:0/1638230885 learned_addr learned my addr 172.21.15.3:0/1638230885 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b357fa640 1 -- 172.21.15.3:0/890384568 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5b2c0cdac0 con 0x7f5b38102d70 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b37fff640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5b200a23f0 0x7f5b200a48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92e7f60 0x55fea92ea3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 -- 172.21.15.3:0/1638230885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92e7f60 msgr2=0x55fea92ea3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92e7f60 0x55fea92ea3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 -- 172.21.15.3:0/1638230885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 msgr2=0x55fea92e7990 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b377fe640 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.457+0000 7f5b3cdc7640 1 -- 172.21.15.3:0/1638230885 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fea9208210 con 0x55fea92dea30 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.458+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92dee10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b30009360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76ed48f74cef8047 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.458+0000 7f5b37fff640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5b200a23f0 0x7f5b200a48b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009bd0 tx=0x7f5b28005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.458+0000 7f5b1e7fc640 1 -- 172.21.15.3:0/1638230885 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b30034740 con 0x55fea92dea30 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.458+0000 7f5b1e7fc640 1 -- 172.21.15.3:0/1638230885 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b30003e50 con 0x55fea92dea30 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.458+0000 7f5b1e7fc640 1 -- 172.21.15.3:0/1638230885 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b30004030 con 0x55fea92dea30 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 -- 172.21.15.3:0/1638230885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 msgr2=0x55fea92dee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92dee10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5b30009ca0 tx=0x7f5b30009360 comp rx=0 tx=0).stop 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 -- 172.21.15.3:0/1638230885 shutdown_connections 2024-02-09T22:51:37.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92e7f60 0x55fea92ea3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 --2- 172.21.15.3:0/1638230885 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92dee10 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 -- 172.21.15.3:0/1638230885 >> 172.21.15.3:0/1638230885 conn(0x55fea92d9100 msgr2=0x55fea92d94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 -- 172.21.15.3:0/1638230885 shutdown_connections 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 -- 172.21.15.3:0/1638230885 wait complete. 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 -- start start 2024-02-09T22:51:37.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.459+0000 7f5b41400740 1 Processor -- start 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92e7f60 0x55fea92e5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea9246030 con 0x55fea92df3e0 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8f19430 con 0x55fea92e7f60 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea8ed5e00 con 0x55fea92dea30 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e5420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42866/0 (socket says 172.21.15.3:42866) 2024-02-09T22:51:37.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 -- 172.21.15.3:0/2435732233 learned_addr learned my addr 172.21.15.3:0/2435732233 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b37fff640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92e7f60 0x55fea92e5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b41400740 10 client.0 _ll_register_callbacks cb 0x55fea923b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 -- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 msgr2=0x55fea92e4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e4ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 -- 172.21.15.3:0/2435732233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92e7f60 msgr2=0x55fea92e5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92e7f60 0x55fea92e5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 -- 172.21.15.3:0/2435732233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b3810e8d0 con 0x55fea92df3e0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b377fe640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e5420 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c009fb0 tx=0x7f5b2c099f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879341940b220764 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.460+0000 7f5b177fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b177fe640 1 -- 172.21.15.3:0/2435732233 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c09e460 con 0x55fea92df3e0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fea8f457e0 con 0x55fea92df3e0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b177fe640 1 -- 172.21.15.3:0/2435732233 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5b2c0a2ab0 con 0x55fea92df3e0 2024-02-09T22:51:37.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b177fe640 1 -- 172.21.15.3:0/2435732233 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b2c005680 con 0x55fea92df3e0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b41400740 10 client.15645 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fea92446b0 con 0x55fea92df3e0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b177fe640 1 -- 172.21.15.3:0/2435732233 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b2c09e100 con 0x55fea92df3e0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b177fe640 1 -- 172.21.15.3:0/2435732233 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5b2c0cb3d0 con 0x55fea92df3e0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b41400740 10 client.15645 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b41400740 10 client.15645 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.461+0000 7f5b41400740 20 client.15645 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.462+0000 7f5b41400740 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea92f83e0 0x55fea92fa840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.462+0000 7f5b41400740 4 client.15645 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.462+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fea8f457e0 con 0x55fea92f83e0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.462+0000 7f5b37fff640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea92f83e0 0x55fea92fa840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.462+0000 7f5b177fe640 10 client.15645 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.462+0000 7f5b37fff640 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea92f83e0 0x55fea92fa840 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe720b42e8be6180 server_cookie=f99afeaa7722af0 in_seq=0 out_seq=0 2024-02-09T22:51:37.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b177fe640 1 -- 172.21.15.3:0/2435732233 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fea8f457e0 con 0x55fea92f83e0 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b177fe640 10 client.15645 handle_command_reply: tid=0 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 client.15645 shutdown 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 msgr2=0x55fea92e5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e5420 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c009fb0 tx=0x7f5b2c099f10 comp rx=0 tx=0).stop 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 shutdown_connections 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea92f83e0 0x55fea92fa840 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92e7f60 0x55fea92e5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92df3e0 0x55fea92e5420 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 --2- 172.21.15.3:0/2435732233 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e4ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 >> 172.21.15.3:0/2435732233 conn(0x55fea92d9100 msgr2=0x55fea92dae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b177fe640 0 client.15645 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 shutdown_connections 2024-02-09T22:51:37.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 1 -- 172.21.15.3:0/2435732233 wait complete. 2024-02-09T22:51:37.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.463+0000 7f5b41400740 20 client.15645 trim_cache size 0 max 16384 2024-02-09T22:51:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.475+0000 7f5b41400740 1 Processor -- start 2024-02-09T22:51:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.475+0000 7f5b41400740 1 -- start start 2024-02-09T22:51:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.475+0000 7f5b41400740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.475+0000 7f5b41400740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 0x55fea92e5780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.475+0000 7f5b41400740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92ff1e0 0x55fea92ff640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b41400740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea91cb130 con 0x55fea92e7f60 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b41400740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea92d4ae0 con 0x55fea92ff1e0 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b41400740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea92dde20 con 0x55fea92dea30 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 0x55fea92e5780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 0x55fea92e5780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42882/0 (socket says 172.21.15.3:42882) 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 -- 172.21.15.3:0/3022702400 learned_addr learned my addr 172.21.15.3:0/3022702400 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b37fff640 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 -- 172.21.15.3:0/3022702400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 msgr2=0x55fea92e7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 -- 172.21.15.3:0/3022702400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92ff1e0 msgr2=0x55fea92ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92ff1e0 0x55fea92ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 -- 172.21.15.3:0/3022702400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fea91e0c90 con 0x55fea92e7f60 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b377fe640 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 0x55fea92e5780 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c096770 tx=0x7f5b2c098820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca22ba50b3646c99 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b1dffb640 1 -- 172.21.15.3:0/3022702400 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c0a2ab0 con 0x55fea92e7f60 2024-02-09T22:51:37.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b1dffb640 1 -- 172.21.15.3:0/3022702400 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b2c09e460 con 0x55fea92e7f60 2024-02-09T22:51:37.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.476+0000 7f5b1dffb640 1 -- 172.21.15.3:0/3022702400 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c099800 con 0x55fea92e7f60 2024-02-09T22:51:37.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 -- 172.21.15.3:0/3022702400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 msgr2=0x55fea92e5780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 0x55fea92e5780 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c096770 tx=0x7f5b2c098820 comp rx=0 tx=0).stop 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 -- 172.21.15.3:0/3022702400 shutdown_connections 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92ff1e0 0x55fea92ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92e7f60 0x55fea92e5780 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 --2- 172.21.15.3:0/3022702400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92dea30 0x55fea92e7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 -- 172.21.15.3:0/3022702400 >> 172.21.15.3:0/3022702400 conn(0x55fea92d9100 msgr2=0x55fea92d94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 -- 172.21.15.3:0/3022702400 shutdown_connections 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 -- 172.21.15.3:0/3022702400 wait complete. 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.477+0000 7f5b41400740 1 -- start start 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 Processor -- start 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92ef2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 0x55fea92ed5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92edb10 0x55fea92edf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea934e970 con 0x55fea92ff1e0 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea934eab0 con 0x55fea92dea30 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fea934ec70 con 0x55fea92edb10 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b41400740 10 client.0 _ll_register_callbacks cb 0x7f5b30037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 0x55fea92ed5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b377fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 0x55fea92ed5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42886/0 (socket says 172.21.15.3:42886) 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b377fe640 1 -- 172.21.15.3:0/2830530064 learned_addr learned my addr 172.21.15.3:0/2830530064 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b37fff640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92ef2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.478+0000 7f5b3cdc7640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92edb10 0x55fea92edf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b377fe640 1 -- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92edb10 msgr2=0x55fea92edf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b377fe640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92edb10 0x55fea92edf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b377fe640 1 -- 172.21.15.3:0/2830530064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 msgr2=0x55fea92ef2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b377fe640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92ef2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b377fe640 1 -- 172.21.15.3:0/2830530064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fea92446b0 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b377fe640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 0x55fea92ed5d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c000c00 tx=0x7f5b2c096290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2211cdadf565e2e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b177fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b177fe640 1 -- 172.21.15.3:0/2830530064 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2c0b6060 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b177fe640 1 -- 172.21.15.3:0/2830530064 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5b2c005680 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b30003e50 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b177fe640 1 -- 172.21.15.3:0/2830530064 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b2c07a030 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b41400740 10 client.15651 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.479+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55fea9206a70 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b177fe640 1 -- 172.21.15.3:0/2830530064 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b30003e50 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b177fe640 1 -- 172.21.15.3:0/2830530064 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5b2c09f380 con 0x55fea92ff1e0 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b41400740 10 client.15651 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b41400740 10 client.15651 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b41400740 20 client.15651 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b41400740 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea934ff40 0x55fea9352330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b41400740 4 client.15651 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fea8f82720 con 0x55fea934ff40 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.480+0000 7f5b37fff640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea934ff40 0x55fea9352330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b177fe640 10 client.15651 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b37fff640 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea934ff40 0x55fea9352330 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9cef39b547b0fb7c server_cookie=283ea8ecb28e8671 in_seq=0 out_seq=0 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b177fe640 1 -- 172.21.15.3:0/2830530064 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fea8f82720 con 0x55fea934ff40 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b177fe640 10 client.15651 handle_command_reply: tid=0 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 client.15651 shutdown 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 msgr2=0x55fea92ed5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 0x55fea92ed5d0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c000c00 tx=0x7f5b2c096290 comp rx=0 tx=0).stop 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 shutdown_connections 2024-02-09T22:51:37.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fea934ff40 0x55fea9352330 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fea92edb10 0x55fea92edf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fea92ff1e0 0x55fea92ed5d0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 --2- 172.21.15.3:0/2830530064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fea92dea30 0x55fea92ef2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b177fe640 0 client.15651 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 >> 172.21.15.3:0/2830530064 conn(0x55fea92d9100 msgr2=0x55fea92e0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 shutdown_connections 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 1 -- 172.21.15.3:0/2830530064 wait complete. 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.481+0000 7f5b41400740 20 client.15651 trim_cache size 0 max 16384 2024-02-09T22:51:37.490 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:51:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5b200a23f0 msgr2=0x7f5b200a48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5b200a23f0 0x7f5b200a48b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f5b28009bd0 tx=0x7f5b28005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:37.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 msgr2=0x7f5b381005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 0x7f5b381005e0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f5b2c00d950 tx=0x7f5b2c00de10 comp rx=0 tx=0).stop 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 shutdown_connections 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5b200a23f0 0x7f5b200a48b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5b3810b820 0x7f5b38100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5b38102d70 0x7f5b381005e0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 --2- 172.21.15.3:0/890384568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5b38102450 0x7f5b381000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 >> 172.21.15.3:0/890384568 conn(0x7f5b380772e0 msgr2=0x7f5b380fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 shutdown_connections 2024-02-09T22:51:37.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.482+0000 7f5b3e82a640 1 -- 172.21.15.3:0/890384568 wait complete. 2024-02-09T22:51:37.496 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:51:37.496 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:51:37.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 -- 172.21.15.3:0/2652963800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102450 msgr2=0x7fcd30102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 --2- 172.21.15.3:0/2652963800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102450 0x7fcd30102830 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009900 tx=0x7fcd2002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 -- 172.21.15.3:0/2652963800 shutdown_connections 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 --2- 172.21.15.3:0/2652963800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd3010b820 0x7fcd3010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 --2- 172.21.15.3:0/2652963800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd30102d70 0x7fcd3010b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 --2- 172.21.15.3:0/2652963800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102450 0x7fcd30102830 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 -- 172.21.15.3:0/2652963800 >> 172.21.15.3:0/2652963800 conn(0x7fcd300772e0 msgr2=0x7fcd300776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 -- 172.21.15.3:0/2652963800 shutdown_connections 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.600+0000 7fcd365a1640 1 -- 172.21.15.3:0/2652963800 wait complete. 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 Processor -- start 2024-02-09T22:51:37.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 -- start start 2024-02-09T22:51:37.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd30102450 0x7fcd3019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 0x7fcd3019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd3010b820 0x7fcd30197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30197df0 con 0x7fcd30102d70 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30197f60 con 0x7fcd30102450 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd365a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30198120 con 0x7fcd3010b820 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 0x7fcd3019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 0x7fcd3019dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42916/0 (socket says 172.21.15.3:42916) 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/1200660584 learned_addr learned my addr 172.21.15.3:0/1200660584 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd30102450 0x7fcd3019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd3010b820 0x7fcd30197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/1200660584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd3010b820 msgr2=0x7fcd30197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd3010b820 0x7fcd30197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/1200660584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd30102450 msgr2=0x7fcd3019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.601+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd30102450 0x7fcd3019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/1200660584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd20009560 con 0x7fcd30102d70 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd3010b820 0x7fcd30197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 0x7fcd3019dae0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400d950 tx=0x7fcd2400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd2d7fa640 1 -- 172.21.15.3:0/1200660584 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24009a30 con 0x7fcd30102d70 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd30198400 con 0x7fcd30102d70 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd2d7fa640 1 -- 172.21.15.3:0/1200660584 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd24009b90 con 0x7fcd30102d70 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd301a4520 con 0x7fcd30102d70 2024-02-09T22:51:37.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.602+0000 7fcd2d7fa640 1 -- 172.21.15.3:0/1200660584 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd24004a00 con 0x7fcd30102d70 2024-02-09T22:51:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.604+0000 7fcd2d7fa640 1 -- 172.21.15.3:0/1200660584 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7fcd24009d00 con 0x7fcd30102d70 2024-02-09T22:51:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.604+0000 7fcd39179740 1 Processor -- start 2024-02-09T22:51:37.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 -- start start 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce3cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 0x56120ce4e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ce50ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120cdf8020 con 0x56120ce4eab0 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120cdab320 con 0x56120ce3c6e0 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd39179740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120cda8850 con 0x56120ce3d000 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ce50ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 0x56120ce4e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 0x56120ce4e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35504/0 (socket says 172.21.15.3:35504) 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 -- 172.21.15.3:0/647320749 learned_addr learned my addr 172.21.15.3:0/647320749 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce3cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 -- 172.21.15.3:0/647320749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 msgr2=0x56120ce3cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce3cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 -- 172.21.15.3:0/647320749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 msgr2=0x56120ce50ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ce50ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2ffff640 1 -- 172.21.15.3:0/647320749 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56120cd66210 con 0x56120ce3d000 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.605+0000 7fcd2d7fa640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd180a22c0 0x7fcd180a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd180a22c0 0x7fcd180a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 0x56120ce4e570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009900 tx=0x7fcd200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa01977d14fe93b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ce50ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd2d7fa640 1 -- 172.21.15.3:0/1200660584 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd240cd640 con 0x7fcd30102d70 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd167fc640 1 -- 172.21.15.3:0/647320749 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20004970 con 0x56120ce3d000 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd167fc640 1 -- 172.21.15.3:0/647320749 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd20004ad0 con 0x56120ce3d000 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.606+0000 7fcd167fc640 1 -- 172.21.15.3:0/647320749 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20005260 con 0x56120ce3d000 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 -- 172.21.15.3:0/647320749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 msgr2=0x56120ce4e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd180a22c0 0x7fcd180a4780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcd28005fd0 tx=0x7fcd280095b0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 0x56120ce4e570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fcd20009900 tx=0x7fcd200046c0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 -- 172.21.15.3:0/647320749 shutdown_connections 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ce50ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3d000 0x56120ce4e570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 --2- 172.21.15.3:0/647320749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce3cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 -- 172.21.15.3:0/647320749 >> 172.21.15.3:0/647320749 conn(0x56120ce37100 msgr2=0x56120ce374e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 -- 172.21.15.3:0/647320749 shutdown_connections 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 -- 172.21.15.3:0/647320749 wait complete. 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.607+0000 7fcd39179740 1 -- start start 2024-02-09T22:51:37.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 Processor -- start 2024-02-09T22:51:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce480a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce485e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce4eab0 0x56120ce4ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120cda4030 con 0x56120ce3d000 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ca77430 con 0x56120ce3c6e0 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ca33e00 con 0x56120ce4eab0 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd39179740 10 client.0 _ll_register_callbacks cb 0x56120cd99030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce485e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42938/0 (socket says 172.21.15.3:42938) 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/2912137960 learned_addr learned my addr 172.21.15.3:0/2912137960 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd2ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce480a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.608+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce4eab0 0x56120ce4ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2ffff640 1 -- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce4eab0 msgr2=0x56120ce4ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce4eab0 0x56120ce4ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2ffff640 1 -- 172.21.15.3:0/2912137960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 msgr2=0x56120ce485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2ffff640 1 -- 172.21.15.3:0/2912137960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3010e8d0 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce480a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd2002c210 tx=0x7fcd20000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36103b7bef226099 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/2912137960 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd20004970 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56120caa37e0 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/2912137960 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd2004a450 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/2912137960 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd200044f0 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd39179740 10 client.25202 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56120cda26b0 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.609+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/2912137960 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56120caa37e0 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/2912137960 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcd2003c050 con 0x56120ce3c6e0 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd39179740 10 client.25202 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd39179740 10 client.25202 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd39179740 20 client.25202 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd39179740 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120ce66ea0 0x56120ce692b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd39179740 4 client.25202 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56120cae0720 con 0x56120ce66ea0 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120ce66ea0 0x56120ce692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120ce66ea0 0x56120ce692b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c8d0d6a9ea3a699e server_cookie=7dbf522640593bdc in_seq=0 out_seq=0 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.610+0000 7fcd0f7fe640 10 client.25202 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/2912137960 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56120cae0720 con 0x56120ce66ea0 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd0f7fe640 10 client.25202 handle_command_reply: tid=0 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 client.25202 shutdown 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 msgr2=0x56120ce480a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce480a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd2002c210 tx=0x7fcd20000bc0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 shutdown_connections 2024-02-09T22:51:37.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120ce66ea0 0x56120ce692b0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce4eab0 0x56120ce4ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd0f7fe640 0 client.25202 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 --2- 172.21.15.3:0/2912137960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3c6e0 0x56120ce480a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 >> 172.21.15.3:0/2912137960 conn(0x56120ce37100 msgr2=0x56120ce38e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 shutdown_connections 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.611+0000 7fcd39179740 1 -- 172.21.15.3:0/2912137960 wait complete. 2024-02-09T22:51:37.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.612+0000 7fcd39179740 20 client.25202 trim_cache size 0 max 16384 2024-02-09T22:51:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.623+0000 7fcd39179740 1 Processor -- start 2024-02-09T22:51:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.623+0000 7fcd39179740 1 -- start start 2024-02-09T22:51:37.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.623+0000 7fcd39179740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ce43440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.623+0000 7fcd39179740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce4dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.623+0000 7fcd39179740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce4eab0 0x56120ce400f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd39179740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120cd29130 con 0x56120ce3d000 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd39179740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ce32ab0 con 0x56120ce4eab0 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd39179740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ce43ee0 con 0x56120ce3c6e0 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce4dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce4dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42946/0 (socket says 172.21.15.3:42946) 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/2821811647 learned_addr learned my addr 172.21.15.3:0/2821811647 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ce43440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce4eab0 0x56120ce400f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/2821811647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 msgr2=0x56120ce43440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ce43440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/2821811647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce4eab0 msgr2=0x56120ce400f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce4eab0 0x56120ce400f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 -- 172.21.15.3:0/2821811647 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56120cd3ec90 con 0x56120ce3d000 2024-02-09T22:51:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce4dd40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009610 tx=0x7fcd24005340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120b2eb57022b520 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd15ffb640 1 -- 172.21.15.3:0/2821811647 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd2409a5f0 con 0x56120ce3d000 2024-02-09T22:51:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd15ffb640 1 -- 172.21.15.3:0/2821811647 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd24099ef0 con 0x56120ce3d000 2024-02-09T22:51:37.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.624+0000 7fcd15ffb640 1 -- 172.21.15.3:0/2821811647 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd240cafb0 con 0x56120ce3d000 2024-02-09T22:51:37.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 -- 172.21.15.3:0/2821811647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 msgr2=0x56120ce4dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce4dd40 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fcd24009610 tx=0x7fcd24005340 comp rx=0 tx=0).stop 2024-02-09T22:51:37.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 -- 172.21.15.3:0/2821811647 shutdown_connections 2024-02-09T22:51:37.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce4eab0 0x56120ce400f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce3d000 0x56120ce4dd40 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 --2- 172.21.15.3:0/2821811647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ce43440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 -- 172.21.15.3:0/2821811647 >> 172.21.15.3:0/2821811647 conn(0x56120ce37100 msgr2=0x56120ce42d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 -- 172.21.15.3:0/2821811647 shutdown_connections 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.625+0000 7fcd39179740 1 -- 172.21.15.3:0/2821811647 wait complete. 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 -- start start 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 Processor -- start 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ceb6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3d000 0x56120ceb7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ceb12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ceb18b0 con 0x56120ce4eab0 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ceb1a20 con 0x56120ce3d000 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120ceb1be0 con 0x56120ce3c6e0 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd39179740 10 client.0 _ll_register_callbacks cb 0x7fcd20005e40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd34b3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ceb12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd34b3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ceb12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42962/0 (socket says 172.21.15.3:42962) 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd34b3e640 1 -- 172.21.15.3:0/3565445184 learned_addr learned my addr 172.21.15.3:0/3565445184 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd2f7fe640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3d000 0x56120ceb7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.626+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ceb6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd34b3e640 1 -- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 msgr2=0x56120ceb6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ceb6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd34b3e640 1 -- 172.21.15.3:0/3565445184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3d000 msgr2=0x56120ceb7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3d000 0x56120ceb7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd34b3e640 1 -- 172.21.15.3:0/3565445184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd301a4520 con 0x56120ce4eab0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd34b3e640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ceb12f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcd28012960 tx=0x7fcd28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fed59a1af8e738 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd0f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/3565445184 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd28021dd0 con 0x56120ce4eab0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd20004ad0 con 0x56120ce4eab0 2024-02-09T22:51:37.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/3565445184 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd28018070 con 0x56120ce4eab0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/3565445184 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd2802a440 con 0x56120ce4eab0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd39179740 10 client.15669 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.627+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56120cd64a70 con 0x56120ce4eab0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/3565445184 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd20004ad0 con 0x56120ce4eab0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/3565445184 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcd2801f820 con 0x56120ce4eab0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd39179740 10 client.15669 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd39179740 10 client.15669 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd39179740 20 client.15669 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd39179740 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120cebecc0 0x56120cec10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd39179740 4 client.15669 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56120cb2f890 con 0x56120cebecc0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120cebecc0 0x56120cec10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd0f7fe640 10 client.15669 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:51:37.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.628+0000 7fcd2ffff640 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120cebecc0 0x56120cec10e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7b8c2372eca7df37 server_cookie=1cb31fb575016a0e in_seq=0 out_seq=0 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd0f7fe640 1 -- 172.21.15.3:0/3565445184 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56120cb2f890 con 0x56120cebecc0 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd0f7fe640 10 client.15669 handle_command_reply: tid=0 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 client.15669 shutdown 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 msgr2=0x56120ceb12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ceb12f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fcd28012960 tx=0x7fcd28004a40 comp rx=0 tx=0).stop 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 shutdown_connections 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56120cebecc0 0x56120cec10e0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120ce3d000 0x56120ceb7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd0f7fe640 0 client.15669 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120ce4eab0 0x56120ceb12f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 --2- 172.21.15.3:0/3565445184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120ce3c6e0 0x56120ceb6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 >> 172.21.15.3:0/3565445184 conn(0x56120ce37100 msgr2=0x56120ce359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 shutdown_connections 2024-02-09T22:51:37.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 1 -- 172.21.15.3:0/3565445184 wait complete. 2024-02-09T22:51:37.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.629+0000 7fcd39179740 20 client.15669 trim_cache size 0 max 16384 2024-02-09T22:51:37.638 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd180a22c0 msgr2=0x7fcd180a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd180a22c0 0x7fcd180a4780 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fcd28005fd0 tx=0x7fcd280095b0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 msgr2=0x7fcd3019dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 0x7fcd3019dae0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fcd2400d950 tx=0x7fcd2400de10 comp rx=0 tx=0).stop 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 shutdown_connections 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd180a22c0 0x7fcd180a4780 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd3010b820 0x7fcd30197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd30102d70 0x7fcd3019dae0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 --2- 172.21.15.3:0/1200660584 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd30102450 0x7fcd3019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 >> 172.21.15.3:0/1200660584 conn(0x7fcd300772e0 msgr2=0x7fcd300fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 shutdown_connections 2024-02-09T22:51:37.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.630+0000 7fcd365a1640 1 -- 172.21.15.3:0/1200660584 wait complete. 2024-02-09T22:51:37.644 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:51:37.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 -- 172.21.15.3:0/916845789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f1410db80 msgr2=0x7f6f1410ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/916845789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f1410db80 0x7f6f1410ff70 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ca0 tx=0x7f6f0c02c180 comp rx=0 tx=0).stop 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 -- 172.21.15.3:0/916845789 shutdown_connections 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/916845789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f1410db80 0x7f6f1410ff70 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/916845789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f141008f0 0x7f6f1410d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/916845789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f140fffd0 0x7f6f141003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.747+0000 7f6f1be0f640 1 -- 172.21.15.3:0/916845789 >> 172.21.15.3:0/916845789 conn(0x7f6f140772e0 msgr2=0x7f6f140776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 -- 172.21.15.3:0/916845789 shutdown_connections 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 -- 172.21.15.3:0/916845789 wait complete. 2024-02-09T22:51:37.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 Processor -- start 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 -- start start 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 0x7f6f1410d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f141008f0 0x7f6f1410a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f1410db80 0x7f6f1410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1410b190 con 0x7f6f140fffd0 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f1410b300 con 0x7f6f141008f0 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.748+0000 7f6f1be0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f141aa400 con 0x7f6f1410db80 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 0x7f6f1410d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 0x7f6f1410d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42980/0 (socket says 172.21.15.3:42980) 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 -- 172.21.15.3:0/1049788594 learned_addr learned my addr 172.21.15.3:0/1049788594 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f1410db80 0x7f6f1410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f193aa640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f141008f0 0x7f6f1410a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 -- 172.21.15.3:0/1049788594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f1410db80 msgr2=0x7f6f1410aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f1410db80 0x7f6f1410aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 -- 172.21.15.3:0/1049788594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f141008f0 msgr2=0x7f6f1410a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f141008f0 0x7f6f1410a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 -- 172.21.15.3:0/1049788594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0c009900 con 0x7f6f140fffd0 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f193aa640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f141008f0 0x7f6f1410a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 0x7f6f1410d3d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000d9b0 tx=0x7f6f1000de70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.749+0000 7f6f0affd640 1 -- 172.21.15.3:0/1049788594 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f10009bd0 con 0x7f6f140fffd0 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.750+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f141aa600 con 0x7f6f140fffd0 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.750+0000 7f6f0affd640 1 -- 172.21.15.3:0/1049788594 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f10009d30 con 0x7f6f140fffd0 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.750+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f141aab60 con 0x7f6f140fffd0 2024-02-09T22:51:37.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.750+0000 7f6f0affd640 1 -- 172.21.15.3:0/1049788594 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f1000f7c0 con 0x7f6f140fffd0 2024-02-09T22:51:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.753+0000 7f6f0affd640 1 -- 172.21.15.3:0/1049788594 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f6f1000f920 con 0x7f6f140fffd0 2024-02-09T22:51:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.753+0000 7f6f1e9dc740 1 Processor -- start 2024-02-09T22:51:37.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.753+0000 7f6f1e9dc740 1 -- start start 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed18ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18f3e0 0x558aed197990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 0x558aed19a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed14a020 con 0x558aed18f3e0 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed0fd320 con 0x558aed197f60 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed0fa850 con 0x558aed18ea30 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f0affd640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6f000a22c0 0x7f6f000a4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f0affd640 1 -- 172.21.15.3:0/1049788594 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6f100cd780 con 0x7f6f140fffd0 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6f000a22c0 0x7f6f000a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f193aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 0x558aed19a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f193aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 0x558aed19a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53606/0 (socket says 172.21.15.3:53606) 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f193aa640 1 -- 172.21.15.3:0/2882080696 learned_addr learned my addr 172.21.15.3:0/2882080696 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f19bab640 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18f3e0 0x558aed197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f193aa640 1 -- 172.21.15.3:0/2882080696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 msgr2=0x558aed18ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f193aa640 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed18ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.754+0000 7f6f193aa640 1 -- 172.21.15.3:0/2882080696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18f3e0 msgr2=0x558aed197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6f193aa640 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18f3e0 0x558aed197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6f193aa640 1 -- 172.21.15.3:0/2882080696 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558aed0b8210 con 0x558aed197f60 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6f19bab640 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18f3e0 0x558aed197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6f193aa640 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 0x558aed19a3b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009c30 tx=0x7f6f040117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53ee5ea53d8e083 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6f000a22c0 0x7f6f000a4780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c02c790 tx=0x7f6f0c009800 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6efffff640 1 -- 172.21.15.3:0/2882080696 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f04012450 con 0x558aed197f60 2024-02-09T22:51:37.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6efffff640 1 -- 172.21.15.3:0/2882080696 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f04012a90 con 0x558aed197f60 2024-02-09T22:51:37.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.755+0000 7f6efffff640 1 -- 172.21.15.3:0/2882080696 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0400fbb0 con 0x558aed197f60 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2882080696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 msgr2=0x558aed19a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 0x558aed19a3b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f6f04009c30 tx=0x7f6f040117b0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2882080696 shutdown_connections 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed197f60 0x558aed19a3b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18f3e0 0x558aed197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2882080696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed18ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2882080696 >> 172.21.15.3:0/2882080696 conn(0x558aed189100 msgr2=0x558aed1894e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2882080696 shutdown_connections 2024-02-09T22:51:37.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2882080696 wait complete. 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.756+0000 7f6f1e9dc740 1 -- start start 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 Processor -- start 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed194f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed195460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1959a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed0f6030 con 0x558aed197f60 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aecdc9430 con 0x558aed18f3e0 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aecd85e00 con 0x558aed18ea30 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1e9dc740 10 client.0 _ll_register_callbacks cb 0x558aed0eb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1a3ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1a3ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1959a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42996/0 (socket says 172.21.15.3:42996) 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1200627580 learned_addr learned my addr 172.21.15.3:0/1200627580 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.757+0000 7f6f193aa640 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed195460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 msgr2=0x558aed194f20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed194f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1200627580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 msgr2=0x558aed195460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed195460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1200627580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f04009860 con 0x558aed197f60 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1959a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ca0 tx=0x7f6f0c0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8ab57104f2ac36 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6efd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6efd7fa640 1 -- 172.21.15.3:0/1200627580 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c04d670 con 0x558aed197f60 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6efd7fa640 1 -- 172.21.15.3:0/1200627580 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6f0c032e70 con 0x558aed197f60 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558aecdf57e0 con 0x558aed197f60 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6efd7fa640 1 -- 172.21.15.3:0/1200627580 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f0c0348a0 con 0x558aed197f60 2024-02-09T22:51:37.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1e9dc740 10 client.15681 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.758+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558aed0f46b0 con 0x558aed197f60 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6efd7fa640 1 -- 172.21.15.3:0/1200627580 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558aecdf57e0 con 0x558aed197f60 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6efd7fa640 1 -- 172.21.15.3:0/1200627580 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f0c049040 con 0x558aed197f60 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f1e9dc740 10 client.15681 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f1e9dc740 10 client.15681 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f1e9dc740 20 client.15681 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed1aaa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f1e9dc740 4 client.15681 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558aece32720 con 0x558aed1a8620 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.759+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed1aaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6efd7fa640 10 client.15681 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed1aaa40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b8536c8a78ab09cc server_cookie=31f14dceea92215d in_seq=0 out_seq=0 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6efd7fa640 1 -- 172.21.15.3:0/1200627580 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558aece32720 con 0x558aed1a8620 2024-02-09T22:51:37.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6efd7fa640 10 client.15681 handle_command_reply: tid=0 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6f1e9dc740 1 client.15681 shutdown 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 msgr2=0x558aed1959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.760+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1959a0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ca0 tx=0x7f6f0c0095c0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 shutdown_connections 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed1aaa40 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6efd7fa640 0 client.15681 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed195460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1959a0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1200627580 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed194f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 >> 172.21.15.3:0/1200627580 conn(0x558aed189100 msgr2=0x558aed18ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 shutdown_connections 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1200627580 wait complete. 2024-02-09T22:51:37.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.761+0000 7f6f1e9dc740 20 client.15681 trim_cache size 0 max 16384 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 Processor -- start 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 -- start start 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed197650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed1957c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1af3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed1848b0 con 0x558aed197f60 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed07b130 con 0x558aed18f3e0 2024-02-09T22:51:37.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed1a3fd0 con 0x558aed18ea30 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1a3ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1af3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1a3ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1af3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43004/0 (socket says 172.21.15.3:43004) 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1811764367 learned_addr learned my addr 172.21.15.3:0/1811764367 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.773+0000 7f6f193aa640 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed1957c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1811764367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 msgr2=0x558aed197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1811764367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 msgr2=0x558aed1957c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed1957c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f1a3ac640 1 -- 172.21.15.3:0/1811764367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558aed090c90 con 0x558aed197f60 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f19bab640 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1af3c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c005c00 tx=0x7f6f0c005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0f09e74e6cbf0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6eff7fe640 1 -- 172.21.15.3:0/1811764367 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c04b420 con 0x558aed197f60 2024-02-09T22:51:37.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6eff7fe640 1 -- 172.21.15.3:0/1811764367 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f0c0404d0 con 0x558aed197f60 2024-02-09T22:51:37.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.774+0000 7f6eff7fe640 1 -- 172.21.15.3:0/1811764367 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c04d6d0 con 0x558aed197f60 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1811764367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 msgr2=0x558aed1af3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1af3c0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c005c00 tx=0x7f6f0c005c30 comp rx=0 tx=0).stop 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1811764367 shutdown_connections 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed1957c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed197f60 0x558aed1af3c0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/1811764367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed18ea30 0x558aed197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1811764367 >> 172.21.15.3:0/1811764367 conn(0x558aed189100 msgr2=0x558aed18f7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1811764367 shutdown_connections 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/1811764367 wait complete. 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 -- start start 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.775+0000 7f6f1e9dc740 1 Processor -- start 2024-02-09T22:51:37.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 0x558aed19f000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed19d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed197f60 0x558aed19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed19f8b0 con 0x558aed18ea30 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 0x558aed19f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed19e010 con 0x558aed18f3e0 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558aed19e1d0 con 0x558aed197f60 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 0x558aed19f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43010/0 (socket says 172.21.15.3:43010) 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 -- 172.21.15.3:0/2006685164 learned_addr learned my addr 172.21.15.3:0/2006685164 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1e9dc740 10 client.0 _ll_register_callbacks cb 0x7f6f040095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f193aa640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f1a3ac640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed197f60 0x558aed19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 -- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed197f60 msgr2=0x558aed19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed197f60 0x558aed19dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 -- 172.21.15.3:0/2006685164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 msgr2=0x558aed19d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed19d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 -- 172.21.15.3:0/2006685164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f141aa600 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6f19bab640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 0x558aed19f000 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000d980 tx=0x7f6f10000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363d522a0ec5f913 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.776+0000 7f6efd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6efd7fa640 1 -- 172.21.15.3:0/2006685164 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f100cb2f0 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6efd7fa640 1 -- 172.21.15.3:0/2006685164 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6f100983f0 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f04012a90 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6efd7fa640 1 -- 172.21.15.3:0/2006685164 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f10094ba0 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 10 client.15687 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558aed0f46b0 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6efd7fa640 1 -- 172.21.15.3:0/2006685164 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f04012a90 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6efd7fa640 1 -- 172.21.15.3:0/2006685164 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6f10099480 con 0x558aed18ea30 2024-02-09T22:51:37.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 10 client.15687 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 10 client.15687 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 20 client.15687 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed201f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 4 client.15687 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.777+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558aece81890 con 0x558aed1a8620 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.778+0000 7f6f193aa640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed201f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.778+0000 7f6efd7fa640 10 client.15687 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.778+0000 7f6f193aa640 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed201f30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d6a4327c114f7880 server_cookie=3b3f5a190b6b964c in_seq=0 out_seq=0 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.778+0000 7f6efd7fa640 1 -- 172.21.15.3:0/2006685164 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558aece81890 con 0x558aed1a8620 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.778+0000 7f6efd7fa640 10 client.15687 handle_command_reply: tid=0 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.778+0000 7f6f1e9dc740 1 client.15687 shutdown 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 msgr2=0x558aed19f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 0x558aed19f000 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000d980 tx=0x7f6f10000bc0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 shutdown_connections 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558aed1a8620 0x558aed201f30 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558aed18f3e0 0x558aed19d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6efd7fa640 0 client.15687 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558aed197f60 0x558aed19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 --2- 172.21.15.3:0/2006685164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558aed18ea30 0x558aed19f000 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 >> 172.21.15.3:0/2006685164 conn(0x558aed189100 msgr2=0x558aed1af3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 shutdown_connections 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 1 -- 172.21.15.3:0/2006685164 wait complete. 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1e9dc740 20 client.15687 trim_cache size 0 max 16384 2024-02-09T22:51:37.787 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:51:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.779+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6f000a22c0 msgr2=0x7f6f000a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6f000a22c0 0x7f6f000a4780 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c02c790 tx=0x7f6f0c009800 comp rx=0 tx=0).stop 2024-02-09T22:51:37.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 msgr2=0x7f6f1410d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 0x7f6f1410d3d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6f1000d9b0 tx=0x7f6f1000de70 comp rx=0 tx=0).stop 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 shutdown_connections 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6f000a22c0 0x7f6f000a4780 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6f141008f0 0x7f6f1410a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6f1410db80 0x7f6f1410aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 --2- 172.21.15.3:0/1049788594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6f140fffd0 0x7f6f1410d3d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 >> 172.21.15.3:0/1049788594 conn(0x7f6f140772e0 msgr2=0x7f6f1410f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 shutdown_connections 2024-02-09T22:51:37.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.780+0000 7f6f1be0f640 1 -- 172.21.15.3:0/1049788594 wait complete. 2024-02-09T22:51:37.793 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:51:37.794 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:51:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 -- 172.21.15.3:0/1189064983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26241008f0 msgr2=0x7f262410d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 --2- 172.21.15.3:0/1189064983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26241008f0 0x7f262410d640 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f2618009900 tx=0x7f261802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 -- 172.21.15.3:0/1189064983 shutdown_connections 2024-02-09T22:51:37.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 --2- 172.21.15.3:0/1189064983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262410db80 0x7f262410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 --2- 172.21.15.3:0/1189064983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26241008f0 0x7f262410d640 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 --2- 172.21.15.3:0/1189064983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f26240fffd0 0x7f26241003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 -- 172.21.15.3:0/1189064983 >> 172.21.15.3:0/1189064983 conn(0x7f26240772e0 msgr2=0x7f26240776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 -- 172.21.15.3:0/1189064983 shutdown_connections 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 -- 172.21.15.3:0/1189064983 wait complete. 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.898+0000 7f262a040640 1 Processor -- start 2024-02-09T22:51:37.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 -- start start 2024-02-09T22:51:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 0x7f2624199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 0x7f262419a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262419e630 0x7f262419eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262419f020 con 0x7f26240fffd0 2024-02-09T22:51:37.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262419f190 con 0x7f262419e630 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f262a040640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262419f300 con 0x7f262410db80 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f26237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 0x7f2624199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f26237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 0x7f2624199d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43018/0 (socket says 172.21.15.3:43018) 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 0x7f262419a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 0x7f262419a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35590/0 (socket says 172.21.15.3:35590) 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f26237fe640 1 -- 172.21.15.3:0/2794540774 learned_addr learned my addr 172.21.15.3:0/2794540774 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2623fff640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262419e630 0x7f262419eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 -- 172.21.15.3:0/2794540774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262419e630 msgr2=0x7f262419eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262419e630 0x7f262419eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 -- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 msgr2=0x7f2624199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 0x7f2624199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.899+0000 7f2622ffd640 1 -- 172.21.15.3:0/2794540774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2618009560 con 0x7f262410db80 2024-02-09T22:51:37.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f2623fff640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262419e630 0x7f262419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f26237fe640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 0x7f2624199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 0x7f262419a2b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2618008000 tx=0x7f261802c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f2620ff9640 1 -- 172.21.15.3:0/2794540774 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26180042d0 con 0x7f262410db80 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26241a68e0 con 0x7f262410db80 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26241a6d60 con 0x7f262410db80 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.900+0000 7f2620ff9640 1 -- 172.21.15.3:0/2794540774 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2618004430 con 0x7f262410db80 2024-02-09T22:51:37.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.901+0000 7f2620ff9640 1 -- 172.21.15.3:0/2794540774 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2618003ab0 con 0x7f262410db80 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.903+0000 7f2620ff9640 1 -- 172.21.15.3:0/2794540774 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f2618003c10 con 0x7f262410db80 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.903+0000 7f262cc14740 1 Processor -- start 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.903+0000 7f262cc14740 1 -- start start 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f262cc14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x563688912e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f262cc14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5636889133e0 0x56368891b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f262cc14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x56368891e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f262cc14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636888ce020 con 0x563688912a30 2024-02-09T22:51:37.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f262cc14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563688881320 con 0x56368891bf60 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f262cc14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56368887e850 con 0x5636889133e0 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x563688912e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x563688912e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43030/0 (socket says 172.21.15.3:43030) 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 -- 172.21.15.3:0/174460379 learned_addr learned my addr 172.21.15.3:0/174460379 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2620ff9640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f260c0a21a0 0x7f260c0a4660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2622ffd640 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x56368891e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f26237fe640 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5636889133e0 0x56368891b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 -- 172.21.15.3:0/174460379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5636889133e0 msgr2=0x56368891b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5636889133e0 0x56368891b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 -- 172.21.15.3:0/174460379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 msgr2=0x56368891e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x56368891e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2623fff640 1 -- 172.21.15.3:0/174460379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56368883c210 con 0x563688912a30 2024-02-09T22:51:37.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.904+0000 7f2622ffd640 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x56368891e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.905+0000 7f2623fff640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f260c0a21a0 0x7f260c0a4660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.905+0000 7f2623fff640 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x563688912e10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f261c011790 tx=0x7f261c011c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0a3855c9c9a19 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.905+0000 7f2620ff9640 1 -- 172.21.15.3:0/2794540774 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f26180f0570 con 0x7f262410db80 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.905+0000 7f2609ffb640 1 -- 172.21.15.3:0/174460379 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c021070 con 0x563688912a30 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.905+0000 7f2609ffb640 1 -- 172.21.15.3:0/174460379 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f261c003d50 con 0x563688912a30 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.905+0000 7f2609ffb640 1 -- 172.21.15.3:0/174460379 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261c01c870 con 0x563688912a30 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 -- 172.21.15.3:0/174460379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 msgr2=0x563688912e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x563688912e10 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f261c011790 tx=0x7f261c011c50 comp rx=0 tx=0).stop 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f2623fff640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f260c0a21a0 0x7f260c0a4660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f261c020000 tx=0x7f261c021000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 -- 172.21.15.3:0/174460379 shutdown_connections 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x56368891e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5636889133e0 0x56368891b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 --2- 172.21.15.3:0/174460379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x563688912e10 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 -- 172.21.15.3:0/174460379 >> 172.21.15.3:0/174460379 conn(0x56368890d100 msgr2=0x56368890d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 -- 172.21.15.3:0/174460379 shutdown_connections 2024-02-09T22:51:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 -- 172.21.15.3:0/174460379 wait complete. 2024-02-09T22:51:37.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 -- start start 2024-02-09T22:51:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.906+0000 7f262cc14740 1 Processor -- start 2024-02-09T22:51:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x563688918e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5636889133e0 0x5636889193a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x5636889198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56368887a030 con 0x5636889133e0 2024-02-09T22:51:37.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56368854d430 con 0x563688912a30 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563688509e00 con 0x56368891bf60 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f262cc14740 10 client.0 _ll_register_callbacks cb 0x56368886f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f26237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x563688918e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2622ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5636889133e0 0x5636889193a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f26237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x563688918e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53666/0 (socket says 172.21.15.3:53666) 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f26237fe640 1 -- 172.21.15.3:0/2424417326 learned_addr learned my addr 172.21.15.3:0/2424417326 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2623fff640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x5636889198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2622ffd640 1 -- 172.21.15.3:0/2424417326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 msgr2=0x5636889198e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x5636889198e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2622ffd640 1 -- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 msgr2=0x563688918e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x563688918e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f2622ffd640 1 -- 172.21.15.3:0/2424417326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f261c011440 con 0x5636889133e0 2024-02-09T22:51:37.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.907+0000 7f26237fe640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x563688918e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5636889133e0 0x5636889193a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f26180edd10 tx=0x7f261802bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1c177cee23e525c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f2602ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f2602ffd640 1 -- 172.21.15.3:0/2424417326 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26180ee0e0 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636885b6720 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f2602ffd640 1 -- 172.21.15.3:0/2424417326 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f26180bd630 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f2602ffd640 1 -- 172.21.15.3:0/2424417326 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26180bc890 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f262cc14740 10 client.15699 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5636888786b0 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.908+0000 7f2602ffd640 1 -- 172.21.15.3:0/2424417326 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5636885b6720 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f2602ffd640 1 -- 172.21.15.3:0/2424417326 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f26180c0060 con 0x5636889133e0 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f262cc14740 10 client.15699 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f262cc14740 10 client.15699 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f262cc14740 20 client.15699 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f262cc14740 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56368892c140 0x56368892e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f262cc14740 4 client.15699 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.909+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563688605890 con 0x56368892c140 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.910+0000 7f26237fe640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56368892c140 0x56368892e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.910+0000 7f26237fe640 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56368892c140 0x56368892e560 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=43a60b9985f281c server_cookie=3ec1d09bd1c6e4c4 in_seq=0 out_seq=0 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.910+0000 7f2602ffd640 10 client.15699 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f2602ffd640 1 -- 172.21.15.3:0/2424417326 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563688605890 con 0x56368892c140 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f2602ffd640 10 client.15699 handle_command_reply: tid=0 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 client.15699 shutdown 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5636889133e0 msgr2=0x5636889193a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5636889133e0 0x5636889193a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f26180edd10 tx=0x7f261802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 shutdown_connections 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56368892c140 0x56368892e560 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x5636889198e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5636889133e0 0x5636889193a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 --2- 172.21.15.3:0/2424417326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x563688918e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 >> 172.21.15.3:0/2424417326 conn(0x56368890d100 msgr2=0x56368890ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f2602ffd640 0 client.15699 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:51:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 shutdown_connections 2024-02-09T22:51:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 1 -- 172.21.15.3:0/2424417326 wait complete. 2024-02-09T22:51:37.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.911+0000 7f262cc14740 20 client.15699 trim_cache size 0 max 16384 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.928+0000 7f262cc14740 1 Processor -- start 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 -- start start 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x56368891b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x5636889160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563688916600 0x563688916a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636887ff130 con 0x563688912a30 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636885b6720 con 0x56368891bf60 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f262cc14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56368890df00 con 0x563688916600 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x56368891b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x56368891b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43046/0 (socket says 172.21.15.3:43046) 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 -- 172.21.15.3:0/248485126 learned_addr learned my addr 172.21.15.3:0/248485126 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f2623fff640 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563688916600 0x563688916a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f2622ffd640 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x5636889160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 -- 172.21.15.3:0/248485126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563688916600 msgr2=0x563688916a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563688916600 0x563688916a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 -- 172.21.15.3:0/248485126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 msgr2=0x5636889160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x5636889160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.929+0000 7f26237fe640 1 -- 172.21.15.3:0/248485126 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563688814c90 con 0x563688912a30 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.930+0000 7f26237fe640 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x56368891b590 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2614005e00 tx=0x7f2614005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6142f46c606f973a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.930+0000 7f26097fa640 1 -- 172.21.15.3:0/248485126 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614017040 con 0x563688912a30 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.930+0000 7f26097fa640 1 -- 172.21.15.3:0/248485126 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2614018040 con 0x563688912a30 2024-02-09T22:51:37.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.930+0000 7f26097fa640 1 -- 172.21.15.3:0/248485126 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2614004a00 con 0x563688912a30 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 -- 172.21.15.3:0/248485126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 msgr2=0x56368891b590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x56368891b590 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f2614005e00 tx=0x7f2614005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 -- 172.21.15.3:0/248485126 shutdown_connections 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56368891bf60 0x5636889160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563688916600 0x563688916a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 --2- 172.21.15.3:0/248485126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688912a30 0x56368891b590 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 -- 172.21.15.3:0/248485126 >> 172.21.15.3:0/248485126 conn(0x56368890d100 msgr2=0x563688914170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 -- 172.21.15.3:0/248485126 shutdown_connections 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 -- 172.21.15.3:0/248485126 wait complete. 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.931+0000 7f262cc14740 1 -- start start 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 Processor -- start 2024-02-09T22:51:37.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x56368897ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688916600 0x56368897bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x563688922320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636889229d0 con 0x563688916600 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563688922b40 con 0x563688912a30 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f26237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x56368897ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f262cc14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563688922d00 con 0x56368891bf60 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f2622ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688916600 0x56368897bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f26237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x56368897ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53686/0 (socket says 172.21.15.3:53686) 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.932+0000 7f26237fe640 1 -- 172.21.15.3:0/2979986011 learned_addr learned my addr 172.21.15.3:0/2979986011 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f262cc14740 10 client.0 _ll_register_callbacks cb 0x7f261c009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f26237fe640 1 -- 172.21.15.3:0/2979986011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 msgr2=0x563688922320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f26237fe640 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x563688922320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f26237fe640 1 -- 172.21.15.3:0/2979986011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688916600 msgr2=0x56368897bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f26237fe640 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688916600 0x56368897bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f26237fe640 1 -- 172.21.15.3:0/2979986011 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2614012840 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688916600 0x56368897bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f26237fe640 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x56368897ba40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2614009640 tx=0x7f26140097a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6657ea2a923f2102 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f2602ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f2602ffd640 1 -- 172.21.15.3:0/2979986011 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f261400fc40 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f2602ffd640 1 -- 172.21.15.3:0/2979986011 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f261400fda0 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.933+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636885797e0 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f2602ffd640 1 -- 172.21.15.3:0/2979986011 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2614005230 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 10 client.25244 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x56368883aa70 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f2602ffd640 1 -- 172.21.15.3:0/2979986011 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5636885797e0 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f2602ffd640 1 -- 172.21.15.3:0/2979986011 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f2614021450 con 0x563688912a30 2024-02-09T22:51:37.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 10 client.25244 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 10 client.25244 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 20 client.25244 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5636889847e0 0x563688986c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 4 client.25244 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563688525700 con 0x5636889847e0 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.934+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5636889847e0 0x563688986c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f2602ffd640 10 client.25244 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f2622ffd640 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5636889847e0 0x563688986c20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b7a58257478d3d5c server_cookie=6e0881b528d6d66e in_seq=0 out_seq=0 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f2602ffd640 1 -- 172.21.15.3:0/2979986011 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563688525700 con 0x5636889847e0 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f2602ffd640 10 client.25244 handle_command_reply: tid=0 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 client.25244 shutdown 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 msgr2=0x56368897ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x56368897ba40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f2614009640 tx=0x7f26140097a0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 shutdown_connections 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5636889847e0 0x563688986c20 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56368891bf60 0x563688922320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f2602ffd640 0 client.25244 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563688916600 0x56368897bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 --2- 172.21.15.3:0/2979986011 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563688912a30 0x56368897ba40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 >> 172.21.15.3:0/2979986011 conn(0x56368890d100 msgr2=0x563688918e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 shutdown_connections 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.935+0000 7f262cc14740 1 -- 172.21.15.3:0/2979986011 wait complete. 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262cc14740 20 client.25244 trim_cache size 0 max 16384 2024-02-09T22:51:37.943 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:51:37.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f260c0a21a0 msgr2=0x7f260c0a4660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f260c0a21a0 0x7f260c0a4660 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f261c020000 tx=0x7f261c021000 comp rx=0 tx=0).stop 2024-02-09T22:51:37.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 msgr2=0x7f262419a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 0x7f262419a2b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2618008000 tx=0x7f261802c670 comp rx=0 tx=0).stop 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 shutdown_connections 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f260c0a21a0 0x7f260c0a4660 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f262419e630 0x7f262419eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f262410db80 0x7f262419a2b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 --2- 172.21.15.3:0/2794540774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f26240fffd0 0x7f2624199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 >> 172.21.15.3:0/2794540774 conn(0x7f26240772e0 msgr2=0x7f262410f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 shutdown_connections 2024-02-09T22:51:37.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:37.936+0000 7f262a040640 1 -- 172.21.15.3:0/2794540774 wait complete. 2024-02-09T22:51:37.953 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:51:37.954 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:51:38.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 -- 172.21.15.3:0/333012689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 msgr2=0x7f716010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 --2- 172.21.15.3:0/333012689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716010cfe0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f7158009cd0 tx=0x7f715802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 -- 172.21.15.3:0/333012689 shutdown_connections 2024-02-09T22:51:38.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 --2- 172.21.15.3:0/333012689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7160100cf0 0x7f7160104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 --2- 172.21.15.3:0/333012689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716010cfe0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 --2- 172.21.15.3:0/333012689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160100340 0x7f7160100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 -- 172.21.15.3:0/333012689 >> 172.21.15.3:0/333012689 conn(0x7f71600772e0 msgr2=0x7f71600776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 -- 172.21.15.3:0/333012689 shutdown_connections 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.058+0000 7f71668f4640 1 -- 172.21.15.3:0/333012689 wait complete. 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 Processor -- start 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 -- start start 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7160100340 0x7f716019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716019fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160199860 0x7f7160199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716019a250 con 0x7f7160105410 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716019a3c0 con 0x7f7160100340 2024-02-09T22:51:38.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f71668f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716019a580 con 0x7f7160199860 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f715f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716019fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f715f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716019fbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43072/0 (socket says 172.21.15.3:43072) 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f715f7fe640 1 -- 172.21.15.3:0/883204353 learned_addr learned my addr 172.21.15.3:0/883204353 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.059+0000 7f715ffff640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7160100340 0x7f716019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f7164e91640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160199860 0x7f7160199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715f7fe640 1 -- 172.21.15.3:0/883204353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160199860 msgr2=0x7f7160199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715f7fe640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160199860 0x7f7160199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715f7fe640 1 -- 172.21.15.3:0/883204353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7160100340 msgr2=0x7f716019f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715f7fe640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7160100340 0x7f716019f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715f7fe640 1 -- 172.21.15.3:0/883204353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7158009900 con 0x7f7160105410 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f7164e91640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160199860 0x7f7160199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715f7fe640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716019fbf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f715400e9c0 tx=0x7f715400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715d7fa640 1 -- 172.21.15.3:0/883204353 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154009700 con 0x7f7160105410 2024-02-09T22:51:38.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71601a6570 con 0x7f7160105410 2024-02-09T22:51:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71601a6ad0 con 0x7f7160105410 2024-02-09T22:51:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715d7fa640 1 -- 172.21.15.3:0/883204353 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f715400cd80 con 0x7f7160105410 2024-02-09T22:51:38.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.060+0000 7f715d7fa640 1 -- 172.21.15.3:0/883204353 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7154010430 con 0x7f7160105410 2024-02-09T22:51:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 Processor -- start 2024-02-09T22:51:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 -- start start 2024-02-09T22:51:38.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba05e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 0x55a42ba0e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba113b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b9c1020 con 0x55a42ba063e0 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b974320 con 0x55a42ba0ef60 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f71695a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b971850 con 0x55a42ba05a30 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f715ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 0x55a42ba0e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f715ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 0x55a42ba0e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43086/0 (socket says 172.21.15.3:43086) 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.063+0000 7f715ffff640 1 -- 172.21.15.3:0/1155259138 learned_addr learned my addr 172.21.15.3:0/1155259138 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f7164e91640 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba05e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715d7fa640 1 -- 172.21.15.3:0/883204353 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f7154005020 con 0x7f7160105410 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715ffff640 1 -- 172.21.15.3:0/1155259138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 msgr2=0x55a42ba05e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715ffff640 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba05e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715ffff640 1 -- 172.21.15.3:0/1155259138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 msgr2=0x55a42ba113b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715ffff640 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715ffff640 1 -- 172.21.15.3:0/1155259138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a42b92f210 con 0x55a42ba063e0 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715ffff640 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 0x55a42ba0e990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7150004540 tx=0x7f7150005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0ddd19325b7e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f71467fc640 1 -- 172.21.15.3:0/1155259138 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715001a070 con 0x55a42ba063e0 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f71467fc640 1 -- 172.21.15.3:0/1155259138 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f715000bde0 con 0x55a42ba063e0 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f71467fc640 1 -- 172.21.15.3:0/1155259138 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150010040 con 0x55a42ba063e0 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.064+0000 7f715d7fa640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71480a2340 0x7f71480a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f7164e91640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71480a2340 0x7f71480a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f715d7fa640 1 -- 172.21.15.3:0/883204353 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7154014070 con 0x7f7160105410 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f7164e91640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71480a2340 0x7f71480a4800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7158006010 tx=0x7f7158004a80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 -- 172.21.15.3:0/1155259138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 msgr2=0x55a42ba0e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 0x55a42ba0e990 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f7150004540 tx=0x7f7150005f90 comp rx=0 tx=0).stop 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 -- 172.21.15.3:0/1155259138 shutdown_connections 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba113b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba063e0 0x55a42ba0e990 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 --2- 172.21.15.3:0/1155259138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba05e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 -- 172.21.15.3:0/1155259138 >> 172.21.15.3:0/1155259138 conn(0x55a42ba00100 msgr2=0x55a42ba004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 -- 172.21.15.3:0/1155259138 shutdown_connections 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.065+0000 7f71695a2740 1 -- 172.21.15.3:0/1155259138 wait complete. 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.066+0000 7f71695a2740 1 -- start start 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.066+0000 7f71695a2740 1 Processor -- start 2024-02-09T22:51:38.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.066+0000 7f71695a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba0c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.066+0000 7f71695a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba0c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.066+0000 7f71695a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0cac0 0x55a42ba1b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.066+0000 7f7164e91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0cac0 0x55a42ba1b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba0c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f7164e91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0cac0 0x55a42ba1b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43102/0 (socket says 172.21.15.3:43102) 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f7164e91640 1 -- 172.21.15.3:0/3074425708 learned_addr learned my addr 172.21.15.3:0/3074425708 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b96d030 con 0x55a42ba0cac0 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b640430 con 0x55a42ba0ef60 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b5fce00 con 0x55a42ba05a30 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715f7fe640 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba0c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f71695a2740 10 client.0 _ll_register_callbacks cb 0x55a42b962030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 -- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 msgr2=0x55a42ba0c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba0c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 -- 172.21.15.3:0/3074425708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0cac0 msgr2=0x55a42ba1b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0cac0 0x55a42ba1b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71500041f0 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a42b66c7e0 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f715ffff640 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba0c040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55a42ba01170 tx=0x7f7150016720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a572545254350c99 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f713f7fe640 10 client.25288 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.067+0000 7f713f7fe640 1 -- 172.21.15.3:0/3074425708 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150023070 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f713f7fe640 1 -- 172.21.15.3:0/3074425708 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7150016b20 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f713f7fe640 1 -- 172.21.15.3:0/3074425708 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7150010070 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 10 client.25288 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({fsmap=25}) v3 -- 0x55a42b96b6b0 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f713f7fe640 1 -- 172.21.15.3:0/3074425708 <== mon.1 v2:172.21.15.89:3300/0 4 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7150026030 con 0x55a42ba05a30 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 10 client.25288 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 10 client.25288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:38.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 20 client.25288 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba21d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 4 client.25288 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.068+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a42b6a9720 con 0x55a42ba1f940 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.069+0000 7f715f7fe640 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba21d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.069+0000 7f713f7fe640 10 client.25288 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.069+0000 7f715f7fe640 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba21d30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a4bcab602a4694b server_cookie=ab717d0fa1cfd207 in_seq=0 out_seq=0 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.070+0000 7f713f7fe640 1 -- 172.21.15.3:0/3074425708 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a42b6a9720 con 0x55a42ba1f940 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.070+0000 7f713f7fe640 10 client.25288 handle_command_reply: tid=0 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.070+0000 7f71695a2740 1 client.25288 shutdown 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 msgr2=0x55a42ba0c040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba0c040 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x55a42ba01170 tx=0x7f7150016720 comp rx=0 tx=0).stop 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 shutdown_connections 2024-02-09T22:51:38.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba21d30 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0ef60 0x55a42ba0c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0cac0 0x55a42ba1b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f713f7fe640 0 client.25288 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 --2- 172.21.15.3:0/3074425708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba0c040 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 >> 172.21.15.3:0/3074425708 conn(0x55a42ba00100 msgr2=0x55a42ba01e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 shutdown_connections 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 1 -- 172.21.15.3:0/3074425708 wait complete. 2024-02-09T22:51:38.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.071+0000 7f71695a2740 20 client.25288 trim_cache size 0 max 16384 2024-02-09T22:51:38.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.094+0000 7f71695a2740 1 Processor -- start 2024-02-09T22:51:38.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.094+0000 7f71695a2740 1 -- start start 2024-02-09T22:51:38.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.094+0000 7f71695a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 0x55a42ba0b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.094+0000 7f71695a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.094+0000 7f71695a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba0ef60 0x55a42ba09820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f71695a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42b8f2130 con 0x55a42ba05a30 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f71695a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42ba1c5b0 con 0x55a42ba0cac0 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f71695a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42ba1f740 con 0x55a42ba0ef60 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 0x55a42ba0b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 0x55a42ba0b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43110/0 (socket says 172.21.15.3:43110) 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 -- 172.21.15.3:0/3512449718 learned_addr learned my addr 172.21.15.3:0/3512449718 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f7164e91640 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba0ef60 0x55a42ba09820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715f7fe640 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 -- 172.21.15.3:0/3512449718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba0ef60 msgr2=0x55a42ba09820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba0ef60 0x55a42ba09820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 -- 172.21.15.3:0/3512449718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 msgr2=0x55a42ba092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 -- 172.21.15.3:0/3512449718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a42b907c90 con 0x55a42ba05a30 2024-02-09T22:51:38.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f7164e91640 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba0ef60 0x55a42ba09820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:38.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f715ffff640 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 0x55a42ba0b040 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7150005d60 tx=0x7f715001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6360fdc55be6d820 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:38.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.095+0000 7f7145ffb640 1 -- 172.21.15.3:0/3512449718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7150023040 con 0x55a42ba05a30 2024-02-09T22:51:38.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f7145ffb640 1 -- 172.21.15.3:0/3512449718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a42b66c7e0 con 0x55a42ba05a30 2024-02-09T22:51:38.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f7145ffb640 1 -- 172.21.15.3:0/3512449718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715001a040 con 0x55a42ba05a30 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 -- 172.21.15.3:0/3512449718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 msgr2=0x55a42ba0b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 0x55a42ba0b040 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f7150005d60 tx=0x7f715001b000 comp rx=0 tx=0).stop 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 -- 172.21.15.3:0/3512449718 shutdown_connections 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba0ef60 0x55a42ba09820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 --2- 172.21.15.3:0/3512449718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba05a30 0x55a42ba0b040 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 -- 172.21.15.3:0/3512449718 >> 172.21.15.3:0/3512449718 conn(0x55a42ba00100 msgr2=0x55a42ba0a930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 -- 172.21.15.3:0/3512449718 shutdown_connections 2024-02-09T22:51:38.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.096+0000 7f71695a2740 1 -- 172.21.15.3:0/3512449718 wait complete. 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.097+0000 7f71695a2740 1 -- start start 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.097+0000 7f71695a2740 1 Processor -- start 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.097+0000 7f71695a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba162e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.097+0000 7f71695a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba14680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f71695a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 0x55a42ba14bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f71695a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42ba152a0 con 0x55a42ba0ef60 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f71695a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42ba15410 con 0x55a42ba0cac0 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f71695a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a42ba75df0 con 0x55a42ba05a30 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f71695a2740 10 client.0 _ll_register_callbacks cb 0x7f7150004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f7164e91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 0x55a42ba14bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba162e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba14680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f7164e91640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 0x55a42ba14bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43118/0 (socket says 172.21.15.3:43118) 2024-02-09T22:51:38.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f7164e91640 1 -- 172.21.15.3:0/3380152889 learned_addr learned my addr 172.21.15.3:0/3380152889 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 -- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 msgr2=0x55a42ba14680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba14680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 -- 172.21.15.3:0/3380152889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 msgr2=0x55a42ba14bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 0x55a42ba14bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 -- 172.21.15.3:0/3380152889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a42b96b6b0 con 0x55a42ba05a30 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f7164e91640 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 0x55a42ba14bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f715ffff640 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba162e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7150005e00 tx=0x7f7150003f60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5651b0c5411388bb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.098+0000 7f713f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f713f7fe640 1 -- 172.21.15.3:0/3380152889 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f715001a070 con 0x55a42ba05a30 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f715000bde0 con 0x55a42ba05a30 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f713f7fe640 1 -- 172.21.15.3:0/3380152889 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7150004eb0 con 0x55a42ba05a30 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f713f7fe640 1 -- 172.21.15.3:0/3380152889 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7150010040 con 0x55a42ba05a30 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f71695a2740 10 client.25294 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:38.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a42b92da70 con 0x55a42ba05a30 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f713f7fe640 1 -- 172.21.15.3:0/3380152889 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f715000bde0 con 0x55a42ba05a30 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f713f7fe640 1 -- 172.21.15.3:0/3380152889 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7150026070 con 0x55a42ba05a30 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f71695a2740 10 client.25294 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f71695a2740 10 client.25294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.099+0000 7f71695a2740 20 client.25294 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f71695a2740 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba790c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f71695a2740 4 client.25294 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a42b6f8890 con 0x55a42ba1f940 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f715f7fe640 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba790c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f715f7fe640 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba790c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3630d21ea5dab8f server_cookie=aa1cd610ede34f99 in_seq=0 out_seq=0 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f713f7fe640 10 client.25294 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f713f7fe640 1 -- 172.21.15.3:0/3380152889 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a42b6f8890 con 0x55a42ba1f940 2024-02-09T22:51:38.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.100+0000 7f713f7fe640 10 client.25294 handle_command_reply: tid=0 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 client.25294 shutdown 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 msgr2=0x55a42ba162e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba162e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7150005e00 tx=0x7f7150003f60 comp rx=0 tx=0).stop 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 shutdown_connections 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a42ba1f940 0x55a42ba790c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f713f7fe640 0 client.25294 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a42ba0cac0 0x55a42ba14680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a42ba0ef60 0x55a42ba14bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 --2- 172.21.15.3:0/3380152889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a42ba05a30 0x55a42ba162e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 >> 172.21.15.3:0/3380152889 conn(0x55a42ba00100 msgr2=0x55a42b9ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.101+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 shutdown_connections 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.102+0000 7f71695a2740 1 -- 172.21.15.3:0/3380152889 wait complete. 2024-02-09T22:51:38.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.102+0000 7f71695a2740 20 client.25294 trim_cache size 0 max 16384 2024-02-09T22:51:38.110 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.102+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71480a2340 msgr2=0x7f71480a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.102+0000 7f71668f4640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71480a2340 0x7f71480a4800 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7158006010 tx=0x7f7158004a80 comp rx=0 tx=0).stop 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.102+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 msgr2=0x7f716019fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716019fbf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f715400e9c0 tx=0x7f715400ee80 comp rx=0 tx=0).stop 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 shutdown_connections 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71480a2340 0x7f71480a4800 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7160199860 0x7f7160199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7160105410 0x7f716019fbf0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 --2- 172.21.15.3:0/883204353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7160100340 0x7f716019f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 >> 172.21.15.3:0/883204353 conn(0x7f71600772e0 msgr2=0x7f71600fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 shutdown_connections 2024-02-09T22:51:38.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:38.103+0000 7f71668f4640 1 -- 172.21.15.3:0/883204353 wait complete. 2024-02-09T22:51:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:38 smithi003 ceph-mon[32678]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-02-09T22:51:38.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:38 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2337994389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:38 smithi064 ceph-mon[36638]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-02-09T22:51:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:38 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2337994389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:38 smithi089 ceph-mon[36665]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-02-09T22:51:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:38 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2337994389' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:51:39.117 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:51:39.118 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:51:39.118 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:51:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 -- 172.21.15.3:0/1512930872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9440691c0 msgr2=0x7ff944105e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1512930872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9440691c0 0x7ff944105e90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff934009930 tx=0x7ff93402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 -- 172.21.15.3:0/1512930872 shutdown_connections 2024-02-09T22:51:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1512930872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944106570 0x7ff94410e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1512930872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9440691c0 0x7ff944105e90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1512930872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff944068810 0x7ff944068bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 -- 172.21.15.3:0/1512930872 >> 172.21.15.3:0/1512930872 conn(0x7ff944077230 msgr2=0x7ff944077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 -- 172.21.15.3:0/1512930872 shutdown_connections 2024-02-09T22:51:39.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.228+0000 7ff94bd77640 1 -- 172.21.15.3:0/1512930872 wait complete. 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 Processor -- start 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 -- start start 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 0x7ff94419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff944106570 0x7ff94419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff944197670 0x7ff944197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944198140 con 0x7ff944197670 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9441982b0 con 0x7ff944068810 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94bd77640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff944198470 con 0x7ff944106570 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff949b13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 0x7ff94419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff949b13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 0x7ff94419d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53766/0 (socket says 172.21.15.3:53766) 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff949b13640 1 -- 172.21.15.3:0/1939125170 learned_addr learned my addr 172.21.15.3:0/1939125170 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff94a314640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff944197670 0x7ff944197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.229+0000 7ff949312640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff944106570 0x7ff94419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff949b13640 1 -- 172.21.15.3:0/1939125170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff944106570 msgr2=0x7ff94419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff949b13640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff944106570 0x7ff94419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff949b13640 1 -- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff944197670 msgr2=0x7ff944197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff949b13640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff944197670 0x7ff944197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff949b13640 1 -- 172.21.15.3:0/1939125170 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff934009560 con 0x7ff944068810 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff94a314640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff944197670 0x7ff944197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff949b13640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 0x7ff94419d5b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff94000d950 tx=0x7ff94000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff93affd640 1 -- 172.21.15.3:0/1939125170 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940009a30 con 0x7ff944068810 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9441a4410 con 0x7ff944068810 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9441a4940 con 0x7ff944068810 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff93affd640 1 -- 172.21.15.3:0/1939125170 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff940009b90 con 0x7ff944068810 2024-02-09T22:51:39.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.230+0000 7ff93affd640 1 -- 172.21.15.3:0/1939125170 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff940004a00 con 0x7ff944068810 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.233+0000 7ff94ea26740 1 Processor -- start 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.233+0000 7ff93affd640 1 -- 172.21.15.3:0/1939125170 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7ff940009d00 con 0x7ff944068810 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 -- start start 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b60570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b69150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b6ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5ace000 con 0x558fa5b69690 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5acb530 con 0x558fa5b60190 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94ea26740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5ac6d10 con 0x558fa5b60ab0 2024-02-09T22:51:39.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94a314640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b60570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b6ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94a314640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b60570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53776/0 (socket says 172.21.15.3:53776) 2024-02-09T22:51:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff94a314640 1 -- 172.21.15.3:0/3662283976 learned_addr learned my addr 172.21.15.3:0/3662283976 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949b13640 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b69150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff93affd640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9300a2340 0x7ff9300a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949312640 1 -- 172.21.15.3:0/3662283976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 msgr2=0x558fa5b69150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949312640 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b69150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949312640 1 -- 172.21.15.3:0/3662283976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 msgr2=0x558fa5b60570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949312640 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b60570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.234+0000 7ff949312640 1 -- 172.21.15.3:0/3662283976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fa5a6d210 con 0x558fa5b69690 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.235+0000 7ff93affd640 1 -- 172.21.15.3:0/1939125170 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9400cd9b0 con 0x7ff944068810 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.235+0000 7ff94a314640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9300a2340 0x7ff9300a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.235+0000 7ff949312640 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b6ba80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff934036040 tx=0x7ff934005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=434c9e4da928b688 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.235+0000 7ff92ffff640 1 -- 172.21.15.3:0/3662283976 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934004800 con 0x558fa5b69690 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.235+0000 7ff92ffff640 1 -- 172.21.15.3:0/3662283976 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff934004960 con 0x558fa5b69690 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.235+0000 7ff92ffff640 1 -- 172.21.15.3:0/3662283976 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff934005050 con 0x558fa5b69690 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 -- 172.21.15.3:0/3662283976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 msgr2=0x558fa5b6ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b6ba80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff934036040 tx=0x7ff934005c00 comp rx=0 tx=0).stop 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94a314640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9300a2340 0x7ff9300a4800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00f600 tx=0x7ff93c011690 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 -- 172.21.15.3:0/3662283976 shutdown_connections 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b6ba80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b69150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 --2- 172.21.15.3:0/3662283976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b60570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 -- 172.21.15.3:0/3662283976 >> 172.21.15.3:0/3662283976 conn(0x558fa5b5a8c0 msgr2=0x558fa5b5aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 -- 172.21.15.3:0/3662283976 shutdown_connections 2024-02-09T22:51:39.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 -- 172.21.15.3:0/3662283976 wait complete. 2024-02-09T22:51:39.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.236+0000 7ff94ea26740 1 -- start start 2024-02-09T22:51:39.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 Processor -- start 2024-02-09T22:51:39.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 0x558fa5b66600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b66b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b69690 0x558fa5b67080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5a6be60 con 0x558fa5b60190 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa579a4b0 con 0x558fa5b69690 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5a4c670 con 0x558fa5b60ab0 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 0x558fa5b66600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 0x558fa5b66600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43162/0 (socket says 172.21.15.3:43162) 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94ea26740 10 client.0 _ll_register_callbacks cb 0x558fa5a64190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 -- 172.21.15.3:0/1214406696 learned_addr learned my addr 172.21.15.3:0/1214406696 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff94a314640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b69690 0x558fa5b67080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 -- 172.21.15.3:0/1214406696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 msgr2=0x558fa5b66b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b66b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 -- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b69690 msgr2=0x558fa5b67080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b69690 0x558fa5b67080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.237+0000 7ff949b13640 1 -- 172.21.15.3:0/1214406696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff94410e930 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff94a314640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b69690 0x558fa5b67080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff949b13640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 0x558fa5b66600 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff940000c00 tx=0x7ff940099e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de501951401f7a96 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff92d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff92d7fa640 1 -- 172.21.15.3:0/1214406696 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94009e3e0 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff92d7fa640 1 -- 172.21.15.3:0/1214406696 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9400a29a0 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558fa57c6860 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff92d7fa640 1 -- 172.21.15.3:0/1214406696 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff940005600 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff94ea26740 10 client.15735 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558fa5ac5390 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff92d7fa640 1 -- 172.21.15.3:0/1214406696 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558fa57c6860 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.238+0000 7ff92d7fa640 1 -- 172.21.15.3:0/1214406696 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff9400cdd20 con 0x558fa5b60190 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff94ea26740 10 client.15735 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff94ea26740 10 client.15735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:39.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff94ea26740 20 client.15735 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5b7da80 0x558fa5b7fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff94ea26740 4 client.15735 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fa5817410 con 0x558fa5b7da80 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff949312640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5b7da80 0x558fa5b7fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff949312640 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5b7da80 0x558fa5b7fee0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=73fe24879429a4e4 server_cookie=3bd44f4d6ff45592 in_seq=0 out_seq=0 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.239+0000 7ff92d7fa640 10 client.15735 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff92d7fa640 1 -- 172.21.15.3:0/1214406696 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558fa5817410 con 0x558fa5b7da80 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff92d7fa640 10 client.15735 handle_command_reply: tid=0 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 client.15735 shutdown 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 msgr2=0x558fa5b66600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 0x558fa5b66600 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7ff940000c00 tx=0x7ff940099e90 comp rx=0 tx=0).stop 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 shutdown_connections 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5b7da80 0x558fa5b7fee0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b69690 0x558fa5b67080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b66b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1214406696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b60190 0x558fa5b66600 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff92d7fa640 0 client.15735 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 >> 172.21.15.3:0/1214406696 conn(0x558fa5b5a8c0 msgr2=0x558fa5b5c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 shutdown_connections 2024-02-09T22:51:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.240+0000 7ff94ea26740 1 -- 172.21.15.3:0/1214406696 wait complete. 2024-02-09T22:51:39.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.241+0000 7ff94ea26740 20 client.15735 trim_cache size 0 max 16384 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.256+0000 7ff94ea26740 1 Processor -- start 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 -- start start 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b68d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b64ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b65410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5b7a520 con 0x558fa5b69690 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5b5b8c0 con 0x558fa5b60190 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94ea26740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5b5fbd0 con 0x558fa5b60ab0 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b65410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b65410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43176/0 (socket says 172.21.15.3:43176) 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 -- 172.21.15.3:0/1281588026 learned_addr learned my addr 172.21.15.3:0/1281588026 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff949312640 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b64ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff949b13640 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b68d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 -- 172.21.15.3:0/1281588026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 msgr2=0x558fa5b64ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b64ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 -- 172.21.15.3:0/1281588026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 msgr2=0x558fa5b68d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b68d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff94a314640 1 -- 172.21.15.3:0/1281588026 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fa5a61c70 con 0x558fa5b69690 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.257+0000 7ff949b13640 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b68d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.258+0000 7ff94a314640 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b65410 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff93c009770 tx=0x7ff93c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03ade2d10b801f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.258+0000 7ff92f7fe640 1 -- 172.21.15.3:0/1281588026 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c024070 con 0x558fa5b69690 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.258+0000 7ff92f7fe640 1 -- 172.21.15.3:0/1281588026 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff93c00ec10 con 0x558fa5b69690 2024-02-09T22:51:39.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.258+0000 7ff92f7fe640 1 -- 172.21.15.3:0/1281588026 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff93c029be0 con 0x558fa5b69690 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.258+0000 7ff94ea26740 1 -- 172.21.15.3:0/1281588026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 msgr2=0x558fa5b65410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b65410 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7ff93c009770 tx=0x7ff93c015000 comp rx=0 tx=0).stop 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 -- 172.21.15.3:0/1281588026 shutdown_connections 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b65410 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5b64ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 --2- 172.21.15.3:0/1281588026 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5b68d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 -- 172.21.15.3:0/1281588026 >> 172.21.15.3:0/1281588026 conn(0x558fa5b5a8c0 msgr2=0x558fa5b61930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 -- 172.21.15.3:0/1281588026 shutdown_connections 2024-02-09T22:51:39.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 -- 172.21.15.3:0/1281588026 wait complete. 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 -- start start 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.259+0000 7ff94ea26740 1 Processor -- start 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5bcd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5bcd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b73c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5b74330 con 0x558fa5b69690 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5b744a0 con 0x558fa5b60190 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fa5b74660 con 0x558fa5b60ab0 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94ea26740 10 client.0 _ll_register_callbacks cb 0x7ff934009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949b13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5bcd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949312640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5bcd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff94a314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b73c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949312640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5bcd890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35714/0 (socket says 172.21.15.3:35714) 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949312640 1 -- 172.21.15.3:0/38202377 learned_addr learned my addr 172.21.15.3:0/38202377 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949b13640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5bcd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53796/0 (socket says 172.21.15.3:53796) 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949b13640 1 -- 172.21.15.3:0/38202377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 msgr2=0x558fa5bcd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949b13640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5bcd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949b13640 1 -- 172.21.15.3:0/38202377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 msgr2=0x558fa5b73c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.260+0000 7ff949b13640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b73c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.261+0000 7ff949b13640 1 -- 172.21.15.3:0/38202377 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fa5ac5390 con 0x558fa5b60190 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.261+0000 7ff949312640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5bcd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:39.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.261+0000 7ff94a314640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b73c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:39.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.262+0000 7ff92d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:51:39.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.262+0000 7ff949b13640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5bcd350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff940096c60 tx=0x7ff94009a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fcb4fd63aa246f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.262+0000 7ff92d7fa640 1 -- 172.21.15.3:0/38202377 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9400a29a0 con 0x558fa5b60190 2024-02-09T22:51:39.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.262+0000 7ff92d7fa640 1 -- 172.21.15.3:0/38202377 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff940096900 con 0x558fa5b60190 2024-02-09T22:51:39.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.262+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff934004960 con 0x558fa5b60190 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff92d7fa640 1 -- 172.21.15.3:0/38202377 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff940096ac0 con 0x558fa5b60190 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 10 client.25274 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7ff93c011d50 con 0x558fa5b60190 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff92d7fa640 1 -- 172.21.15.3:0/38202377 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff934004960 con 0x558fa5b60190 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff92d7fa640 1 -- 172.21.15.3:0/38202377 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff9400cdd20 con 0x558fa5b60190 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 10 client.25274 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 10 client.25274 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 20 client.25274 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5bd5f90 0x558fa5bd8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 4 client.25274 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.263+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558fa58037b0 con 0x558fa5bd5f90 2024-02-09T22:51:39.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.264+0000 7ff949312640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5bd5f90 0x558fa5bd8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.264+0000 7ff949312640 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5bd5f90 0x558fa5bd8380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7eef65df32569b8e server_cookie=ea928d52ccb1d418 in_seq=0 out_seq=0 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.264+0000 7ff92d7fa640 10 client.25274 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.265+0000 7ff92d7fa640 1 -- 172.21.15.3:0/38202377 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558fa58037b0 con 0x558fa5bd5f90 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.265+0000 7ff92d7fa640 10 client.25274 handle_command_reply: tid=0 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.265+0000 7ff94ea26740 1 client.25274 shutdown 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 msgr2=0x558fa5bcd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5bcd350 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff940096c60 tx=0x7ff94009a150 comp rx=0 tx=0).stop 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 shutdown_connections 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558fa5bd5f90 0x558fa5bd8380 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558fa5b69690 0x558fa5b73c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558fa5b60ab0 0x558fa5bcd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 --2- 172.21.15.3:0/38202377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558fa5b60190 0x558fa5bcd350 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 >> 172.21.15.3:0/38202377 conn(0x558fa5b5a8c0 msgr2=0x558fa5b64e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff92d7fa640 0 client.25274 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 shutdown_connections 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 1 -- 172.21.15.3:0/38202377 wait complete. 2024-02-09T22:51:39.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.266+0000 7ff94ea26740 20 client.25274 trim_cache size 0 max 16384 2024-02-09T22:51:39.272 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:51:39.272 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:51:39.272 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "69425400-49d0-4701-8db7-8ea438cb50a0", 2024-02-09T22:51:39.272 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:51:39.272 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.267+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9300a2340 msgr2=0x7ff9300a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.267+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9300a2340 0x7ff9300a4800 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff93c00f600 tx=0x7ff93c011690 comp rx=0 tx=0).stop 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.267+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 msgr2=0x7ff94419d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.267+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 0x7ff94419d5b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff94000d950 tx=0x7ff94000de10 comp rx=0 tx=0).stop 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 shutdown_connections 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9300a2340 0x7ff9300a4800 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff944197670 0x7ff944197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff944106570 0x7ff94419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 --2- 172.21.15.3:0/1939125170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff944068810 0x7ff94419d5b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 >> 172.21.15.3:0/1939125170 conn(0x7ff944077230 msgr2=0x7ff9440fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 shutdown_connections 2024-02-09T22:51:39.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.268+0000 7ff94bd77640 1 -- 172.21.15.3:0/1939125170 wait complete. 2024-02-09T22:51:39.282 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:51:39.282 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:51:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 -- 172.21.15.3:0/2336798069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c105410 msgr2=0x7f212c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2336798069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c105410 0x7f212c10cfe0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f2124009cd0 tx=0x7f212402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 -- 172.21.15.3:0/2336798069 shutdown_connections 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2336798069 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c100cf0 0x7f212c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2336798069 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c105410 0x7f212c10cfe0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2336798069 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212c100340 0x7f212c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 -- 172.21.15.3:0/2336798069 >> 172.21.15.3:0/2336798069 conn(0x7f212c0772e0 msgr2=0x7f212c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 -- 172.21.15.3:0/2336798069 shutdown_connections 2024-02-09T22:51:39.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.389+0000 7f2130cc3640 1 -- 172.21.15.3:0/2336798069 wait complete. 2024-02-09T22:51:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f2130cc3640 1 Processor -- start 2024-02-09T22:51:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f2130cc3640 1 -- start start 2024-02-09T22:51:39.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f2130cc3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 0x7f212c195630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f2130cc3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c105410 0x7f212c195b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f2130cc3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212c199ef0 0x7f212c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f212a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 0x7f212c195630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f212a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 0x7f212c195630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43204/0 (socket says 172.21.15.3:43204) 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.390+0000 7f212a59c640 1 -- 172.21.15.3:0/2835249308 learned_addr learned my addr 172.21.15.3:0/2835249308 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c19a8e0 con 0x7f212c100340 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c19aa50 con 0x7f212c105410 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212c19abc0 con 0x7f212c199ef0 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f2129d9b640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c105410 0x7f212c195b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212ad9d640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212c199ef0 0x7f212c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212a59c640 1 -- 172.21.15.3:0/2835249308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212c199ef0 msgr2=0x7f212c19a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212a59c640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212c199ef0 0x7f212c19a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212a59c640 1 -- 172.21.15.3:0/2835249308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c105410 msgr2=0x7f212c195b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212a59c640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c105410 0x7f212c195b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212a59c640 1 -- 172.21.15.3:0/2835249308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2124009900 con 0x7f212c100340 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f2129d9b640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c105410 0x7f212c195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f212a59c640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 0x7f212c195630 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f211c00e9f0 tx=0x7f211c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.391+0000 7f211b7fe640 1 -- 172.21.15.3:0/2835249308 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c009720 con 0x7f212c100340 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.392+0000 7f211b7fe640 1 -- 172.21.15.3:0/2835249308 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f211c00cd80 con 0x7f212c100340 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.392+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212c1a21a0 con 0x7f212c100340 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.392+0000 7f211b7fe640 1 -- 172.21.15.3:0/2835249308 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c010690 con 0x7f212c100340 2024-02-09T22:51:39.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.392+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c1a2650 con 0x7f212c100340 2024-02-09T22:51:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f211b7fe640 1 -- 172.21.15.3:0/2835249308 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f211c0046c0 con 0x7f212c100340 2024-02-09T22:51:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 Processor -- start 2024-02-09T22:51:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 -- start start 2024-02-09T22:51:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff892bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89348e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff8937300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff88e6d00 con 0x55cff892c330 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff889b2f0 con 0x55cff892b980 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.395+0000 7f2133971740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8898820 con 0x55cff8934eb0 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89348e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89348e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43220/0 (socket says 172.21.15.3:43220) 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 -- 172.21.15.3:0/2397280136 learned_addr learned my addr 172.21.15.3:0/2397280136 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f2129d9b640 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff8937300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212ad9d640 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff892bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f211b7fe640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21140a2350 0x7f21140a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 -- 172.21.15.3:0/2397280136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 msgr2=0x55cff8937300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff8937300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 -- 172.21.15.3:0/2397280136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 msgr2=0x55cff892bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff892bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 -- 172.21.15.3:0/2397280136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cff8855ff0 con 0x55cff892c330 2024-02-09T22:51:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212ad9d640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21140a2350 0x7f21140a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f212a59c640 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89348e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f211c0cbd70 tx=0x7f211c0d64b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca479e7b1edb691 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.396+0000 7f2118ff9640 1 -- 172.21.15.3:0/2397280136 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c097120 con 0x55cff892c330 2024-02-09T22:51:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f2118ff9640 1 -- 172.21.15.3:0/2397280136 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f211c0969f0 con 0x55cff892c330 2024-02-09T22:51:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f2118ff9640 1 -- 172.21.15.3:0/2397280136 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c0a3620 con 0x55cff892c330 2024-02-09T22:51:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f211b7fe640 1 -- 172.21.15.3:0/2835249308 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f211c014070 con 0x7f212c100340 2024-02-09T22:51:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f2133971740 1 -- 172.21.15.3:0/2397280136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 msgr2=0x55cff89348e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f2133971740 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89348e0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f211c0cbd70 tx=0x7f211c0d64b0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f212ad9d640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21140a2350 0x7f21140a4810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2124005cc0 tx=0x7f21240095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.397+0000 7f2133971740 1 -- 172.21.15.3:0/2397280136 shutdown_connections 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff8937300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89348e0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 --2- 172.21.15.3:0/2397280136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff892bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- 172.21.15.3:0/2397280136 >> 172.21.15.3:0/2397280136 conn(0x55cff8926050 msgr2=0x55cff8926430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- 172.21.15.3:0/2397280136 shutdown_connections 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- 172.21.15.3:0/2397280136 wait complete. 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- start start 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 Processor -- start 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8931d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89322a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff89327e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8894000 con 0x55cff892c330 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8567430 con 0x55cff892b980 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8523e00 con 0x55cff8934eb0 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.398+0000 7f2133971740 10 client.0 _ll_register_callbacks cb 0x55cff88887f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89322a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89322a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43236/0 (socket says 172.21.15.3:43236) 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 -- 172.21.15.3:0/1635282189 learned_addr learned my addr 172.21.15.3:0/1635282189 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f212ad9d640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff89327e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f212a59c640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8931d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 -- 172.21.15.3:0/1635282189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 msgr2=0x55cff89327e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff89327e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 -- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 msgr2=0x55cff8931d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8931d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 -- 172.21.15.3:0/1635282189 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212c1a2650 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f212a59c640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8931d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2129d9b640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89322a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f21200099f0 tx=0x7f2120009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d06a29b6929381 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2111ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2111ffb640 1 -- 172.21.15.3:0/1635282189 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2120018070 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2111ffb640 1 -- 172.21.15.3:0/1635282189 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2120004460 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.399+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cff85937e0 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2111ffb640 1 -- 172.21.15.3:0/1635282189 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21200130d0 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 10 client.15753 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55cff8892680 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2111ffb640 1 -- 172.21.15.3:0/1635282189 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cff85937e0 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2111ffb640 1 -- 172.21.15.3:0/1635282189 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f21200118f0 con 0x55cff892c330 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 10 client.15753 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:39.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 10 client.15753 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 20 client.15753 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff8947870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 4 client.15753 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.400+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cff85d0720 con 0x55cff8945450 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.401+0000 7f212a59c640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff8947870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.401+0000 7f212a59c640 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff8947870 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cb542f50dab67c1 server_cookie=485773e1a995347c in_seq=0 out_seq=0 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.401+0000 7f2111ffb640 10 client.15753 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.401+0000 7f2111ffb640 1 -- 172.21.15.3:0/1635282189 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cff85d0720 con 0x55cff8945450 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.401+0000 7f2111ffb640 10 client.15753 handle_command_reply: tid=0 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 client.15753 shutdown 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 msgr2=0x55cff89322a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89322a0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f21200099f0 tx=0x7f2120009eb0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 shutdown_connections 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff8947870 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff8934eb0 0x55cff89327e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2111ffb640 0 client.15753 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff89322a0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 --2- 172.21.15.3:0/1635282189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8931d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 >> 172.21.15.3:0/1635282189 conn(0x55cff8926050 msgr2=0x55cff8927da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 shutdown_connections 2024-02-09T22:51:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 1 -- 172.21.15.3:0/1635282189 wait complete. 2024-02-09T22:51:39.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.402+0000 7f2133971740 20 client.15753 trim_cache size 0 max 16384 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.417+0000 7f2133971740 1 Processor -- start 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.417+0000 7f2133971740 1 -- start start 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.417+0000 7f2133971740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8934490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.417+0000 7f2133971740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892c330 0x55cff89211e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.417+0000 7f2133971740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 0x55cff8926430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f2133971740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8818fd0 con 0x55cff8934eb0 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 0x55cff8926430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 0x55cff8926430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43252/0 (socket says 172.21.15.3:43252) 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f2129d9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892c330 0x55cff89211e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 -- 172.21.15.3:0/3743259417 learned_addr learned my addr 172.21.15.3:0/3743259417 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff88fbab0 con 0x55cff892b980 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff8921720 con 0x55cff892c330 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212a59c640 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8934490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 -- 172.21.15.3:0/3743259417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892c330 msgr2=0x55cff89211e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892c330 0x55cff89211e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 -- 172.21.15.3:0/3743259417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 msgr2=0x55cff8934490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8934490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 -- 172.21.15.3:0/3743259417 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cff882eb30 con 0x55cff8934eb0 2024-02-09T22:51:39.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f2129d9b640 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892c330 0x55cff89211e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:39.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.418+0000 7f212ad9d640 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 0x55cff8926430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f212402c700 tx=0x7f2124043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c76a0a66e80027a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.419+0000 7f2113fff640 1 -- 172.21.15.3:0/3743259417 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2124034c90 con 0x55cff8934eb0 2024-02-09T22:51:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.419+0000 7f2113fff640 1 -- 172.21.15.3:0/3743259417 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2124034df0 con 0x55cff8934eb0 2024-02-09T22:51:39.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.419+0000 7f2113fff640 1 -- 172.21.15.3:0/3743259417 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2124004330 con 0x55cff8934eb0 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 msgr2=0x55cff8926430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 0x55cff8926430 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f212402c700 tx=0x7f2124043000 comp rx=0 tx=0).stop 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 shutdown_connections 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff8934eb0 0x55cff8926430 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892c330 0x55cff89211e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 --2- 172.21.15.3:0/3743259417 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff892b980 0x55cff8934490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 >> 172.21.15.3:0/3743259417 conn(0x55cff8926050 msgr2=0x55cff89315f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 shutdown_connections 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 -- 172.21.15.3:0/3743259417 wait complete. 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.420+0000 7f2133971740 1 -- start start 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 Processor -- start 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892b980 0x55cff893c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff893a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff8934eb0 0x55cff893aa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff893af40 con 0x55cff892c330 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2129d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff893a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2129d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff893a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43254/0 (socket says 172.21.15.3:43254) 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2129d9b640 1 -- 172.21.15.3:0/2857584764 learned_addr learned my addr 172.21.15.3:0/2857584764 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff893b080 con 0x55cff8934eb0 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cff893b240 con 0x55cff892b980 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.421+0000 7f212a59c640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892b980 0x55cff893c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212ad9d640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff8934eb0 0x55cff893aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2133971740 10 client.0 _ll_register_callbacks cb 0x7f211c0cbb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212a59c640 1 -- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff8934eb0 msgr2=0x55cff893aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212a59c640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff8934eb0 0x55cff893aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212a59c640 1 -- 172.21.15.3:0/2857584764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 msgr2=0x55cff893a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212a59c640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff893a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212a59c640 1 -- 172.21.15.3:0/2857584764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cff8892680 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2129d9b640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff893a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f212a59c640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892b980 0x55cff893c090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f211c09e000 tx=0x7f211c004b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1e2a96e283bab93 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2111ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2111ffb640 1 -- 172.21.15.3:0/2857584764 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211c0d1cc0 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f211c0969f0 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2111ffb640 1 -- 172.21.15.3:0/2857584764 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f211c09b580 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2111ffb640 1 -- 172.21.15.3:0/2857584764 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f211c09ac50 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2133971740 10 client.25327 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.422+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7f212c10d7f0 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2111ffb640 1 -- 172.21.15.3:0/2857584764 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f211c0969f0 con 0x55cff892b980 2024-02-09T22:51:39.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2111ffb640 1 -- 172.21.15.3:0/2857584764 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f211c01f060 con 0x55cff892b980 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2133971740 10 client.25327 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2133971740 10 client.25327 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2133971740 20 client.25327 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2133971740 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff899eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2133971740 4 client.25327 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cff861f890 con 0x55cff8945450 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.423+0000 7f2129d9b640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff899eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2111ffb640 10 client.25327 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2129d9b640 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff899eb20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba49332dce5c2192 server_cookie=3247fb78de7e52da in_seq=0 out_seq=0 2024-02-09T22:51:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2111ffb640 1 -- 172.21.15.3:0/2857584764 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55cff861f890 con 0x55cff8945450 2024-02-09T22:51:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2111ffb640 10 client.25327 handle_command_reply: tid=0 2024-02-09T22:51:39.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 client.25327 shutdown 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892b980 msgr2=0x55cff893c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892b980 0x55cff893c090 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f211c09e000 tx=0x7f211c004b30 comp rx=0 tx=0).stop 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 shutdown_connections 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cff8945450 0x55cff899eb20 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cff8934eb0 0x55cff893aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2111ffb640 0 client.25327 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cff892c330 0x55cff893a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 --2- 172.21.15.3:0/2857584764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cff892b980 0x55cff893c090 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 >> 172.21.15.3:0/2857584764 conn(0x55cff8926050 msgr2=0x55cff8930650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 shutdown_connections 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 1 -- 172.21.15.3:0/2857584764 wait complete. 2024-02-09T22:51:39.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.424+0000 7f2133971740 20 client.25327 trim_cache size 0 max 16384 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: "69425400-49d0-4701-8db7-8ea438cb50a0": { 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "69425400-49d0-4701-8db7-8ea438cb50a0", 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:51:39.433 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:51:39.434 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21140a2350 msgr2=0x7f21140a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21140a2350 0x7f21140a4810 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2124005cc0 tx=0x7f21240095c0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 msgr2=0x7f212c195630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 0x7f212c195630 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f211c00e9f0 tx=0x7f211c00edf0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 shutdown_connections 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21140a2350 0x7f21140a4810 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f212c105410 0x7f212c195b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f212c199ef0 0x7f212c19a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 --2- 172.21.15.3:0/2835249308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f212c100340 0x7f212c195630 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 >> 172.21.15.3:0/2835249308 conn(0x7f212c0772e0 msgr2=0x7f212c0fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 shutdown_connections 2024-02-09T22:51:39.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:51:39.425+0000 7f2130cc3640 1 -- 172.21.15.3:0/2835249308 wait complete. 2024-02-09T22:51:39.443 INFO:tasks.cephfs.filesystem:scrub status for tag:69425400-49d0-4701-8db7-8ea438cb50a0 - {'path': '/', 'tag': '69425400-49d0-4701-8db7-8ea438cb50a0', 'options': 'recursive,force'} 2024-02-09T22:51:40.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:40 smithi064 ceph-mon[36638]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-02-09T22:51:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:40 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:51:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:40 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:40 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:51:40.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:40 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:40 smithi089 ceph-mon[36665]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-02-09T22:51:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:40 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:51:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:40 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:40 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:51:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:40 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:40 smithi003 ceph-mon[32678]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-02-09T22:51:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:40 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:51:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:40 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:40 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:51:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:40 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:51:42.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:42 smithi064 ceph-mon[36638]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-02-09T22:51:42.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:51:42.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:42 smithi089 ceph-mon[36665]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-02-09T22:51:42.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:51:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:42 smithi003 ceph-mon[32678]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-02-09T22:51:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.893+0000 7f5670b25640 1 -- 172.21.15.3:0/1379795127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 msgr2=0x7f566c0fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.893+0000 7f5670b25640 1 --2- 172.21.15.3:0/1379795127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c0fff50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5654009900 tx=0x7f565402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 -- 172.21.15.3:0/1379795127 shutdown_connections 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 --2- 172.21.15.3:0/1379795127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 --2- 172.21.15.3:0/1379795127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c0fff50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 --2- 172.21.15.3:0/1379795127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f566c105860 0x7f566c105c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 -- 172.21.15.3:0/1379795127 >> 172.21.15.3:0/1379795127 conn(0x7f566c075450 msgr2=0x7f566c075860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:42.897 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 -- 172.21.15.3:0/1379795127 shutdown_connections 2024-02-09T22:51:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.894+0000 7f5670b25640 1 -- 172.21.15.3:0/1379795127 wait complete. 2024-02-09T22:51:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 Processor -- start 2024-02-09T22:51:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 -- start start 2024-02-09T22:51:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c10fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f566c105860 0x7f566c10e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.899 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c10ec10 con 0x7f566c100490 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c19ba70 con 0x7f566c105860 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5670b25640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566c19bc30 con 0x7f566c0ffad0 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f566a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c10fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43262/0 (socket says 172.21.15.3:43262) 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f5661d9b640 1 -- 172.21.15.3:0/2027049701 learned_addr learned my addr 172.21.15.3:0/2027049701 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.895+0000 7f566ad9d640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f566c105860 0x7f566c10e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5661d9b640 1 -- 172.21.15.3:0/2027049701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 msgr2=0x7f566c10fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5661d9b640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c10fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5661d9b640 1 -- 172.21.15.3:0/2027049701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f566c105860 msgr2=0x7f566c10e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5661d9b640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f566c105860 0x7f566c10e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5661d9b640 1 -- 172.21.15.3:0/2027049701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5654009560 con 0x7f566c100490 2024-02-09T22:51:42.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f566a59c640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c10fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5661d9b640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10de40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5654000c00 tx=0x7f565402c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5663fff640 1 -- 172.21.15.3:0/2027049701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654004410 con 0x7f566c100490 2024-02-09T22:51:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5663fff640 1 -- 172.21.15.3:0/2027049701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56540052d0 con 0x7f566c100490 2024-02-09T22:51:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.896+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f566c19beb0 con 0x7f566c100490 2024-02-09T22:51:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.897+0000 7f5663fff640 1 -- 172.21.15.3:0/2027049701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654003ca0 con 0x7f566c100490 2024-02-09T22:51:42.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.897+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c19c3c0 con 0x7f566c100490 2024-02-09T22:51:42.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.900+0000 7f5663fff640 1 -- 172.21.15.3:0/2027049701 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7f5654005440 con 0x7f566c100490 2024-02-09T22:51:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.900+0000 7f5663fff640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56580a22b0 0x7f56580a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.900+0000 7f56737d3740 1 Processor -- start 2024-02-09T22:51:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f5663fff640 1 -- 172.21.15.3:0/2027049701 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f56540f0500 con 0x7f566c100490 2024-02-09T22:51:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f566a59c640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56580a22b0 0x7f56580a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 -- start start 2024-02-09T22:51:42.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890060b0 0x565289006490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x56528900f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528900f7a0 0x565289011b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288f74a00 con 0x5652890069d0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288f71f30 con 0x5652890060b0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f56737d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288f6d710 con 0x56528900f7a0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f566a59c640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56580a22b0 0x7f56580a4770 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f565c005fd0 tx=0x7f565c009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.901+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x56528900f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x56528900f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43278/0 (socket says 172.21.15.3:43278) 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 -- 172.21.15.3:0/3708779618 learned_addr learned my addr 172.21.15.3:0/3708779618 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f566ad9d640 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528900f7a0 0x565289011b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f566a59c640 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890060b0 0x565289006490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 -- 172.21.15.3:0/3708779618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528900f7a0 msgr2=0x565289011b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528900f7a0 0x565289011b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 -- 172.21.15.3:0/3708779618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890060b0 msgr2=0x565289006490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890060b0 0x565289006490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 -- 172.21.15.3:0/3708779618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565288f323b0 con 0x5652890069d0 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f566ad9d640 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528900f7a0 0x565289011b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.902+0000 7f5661d9b640 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x56528900f260 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f56540bce50 tx=0x7f565402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19df5102c102500f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.903+0000 7f5660d99640 1 -- 172.21.15.3:0/3708779618 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56540bcec0 con 0x5652890069d0 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.903+0000 7f5660d99640 1 -- 172.21.15.3:0/3708779618 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56540c1010 con 0x5652890069d0 2024-02-09T22:51:42.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.903+0000 7f5660d99640 1 -- 172.21.15.3:0/3708779618 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56540ee0d0 con 0x5652890069d0 2024-02-09T22:51:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 -- 172.21.15.3:0/3708779618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 msgr2=0x56528900f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x56528900f260 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f56540bce50 tx=0x7f565402bc30 comp rx=0 tx=0).stop 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 -- 172.21.15.3:0/3708779618 shutdown_connections 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56528900f7a0 0x565289011b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x56528900f260 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 --2- 172.21.15.3:0/3708779618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890060b0 0x565289006490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 -- 172.21.15.3:0/3708779618 >> 172.21.15.3:0/3708779618 conn(0x5652890009d0 msgr2=0x565289000db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 -- 172.21.15.3:0/3708779618 shutdown_connections 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.904+0000 7f56737d3740 1 -- 172.21.15.3:0/3708779618 wait complete. 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.905+0000 7f56737d3740 1 -- start start 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.905+0000 7f56737d3740 1 Processor -- start 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x56528900c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 0x56528900cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 0x56528900d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f566a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x56528900c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f566a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x56528900c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35804/0 (socket says 172.21.15.3:35804) 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f566ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 0x56528900d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f566ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 0x56528900d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43282/0 (socket says 172.21.15.3:43282) 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 0x56528900cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 0x56528900cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53878/0 (socket says 172.21.15.3:53878) 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f566a59c640 1 -- 172.21.15.3:0/2548721578 learned_addr learned my addr 172.21.15.3:0/2548721578 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288f12b90 con 0x56528900f7a0 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288c41250 con 0x5652890069d0 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288ef3200 con 0x5652890060b0 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.906+0000 7f56737d3740 10 client.0 _ll_register_callbacks cb 0x565288f295d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566a59c640 1 -- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 msgr2=0x56528900cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566a59c640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 0x56528900cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566a59c640 1 -- 172.21.15.3:0/2548721578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 msgr2=0x56528900d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566a59c640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 0x56528900d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566a59c640 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f566c19c3c0 con 0x5652890060b0 2024-02-09T22:51:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f5661d9b640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 0x56528900cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566ad9d640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 0x56528900d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f566a59c640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x56528900c800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f565c00fbe0 tx=0x7f565c004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cde6cda09ff3f596 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:42.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f5651ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f5651ffb640 1 -- 172.21.15.3:0/2548721578 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c022070 con 0x5652890060b0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565288f93ee0 con 0x5652890060b0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f5651ffb640 1 -- 172.21.15.3:0/2548721578 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f565c0043a0 con 0x5652890060b0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f5651ffb640 1 -- 172.21.15.3:0/2548721578 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f565c004560 con 0x5652890060b0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f56737d3740 10 client.25339 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x565288f6bd90 con 0x5652890060b0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.907+0000 7f5651ffb640 1 -- 172.21.15.3:0/2548721578 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f565c034020 con 0x5652890060b0 2024-02-09T22:51:42.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f5651ffb640 1 -- 172.21.15.3:0/2548721578 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f565c026410 con 0x5652890060b0 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f56737d3740 10 client.25339 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f56737d3740 10 client.25339 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f56737d3740 20 client.25339 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f56737d3740 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56528901f870 0x565289021c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f56737d3740 4 client.25339 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565288c6d600 con 0x56528901f870 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.908+0000 7f5661d9b640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56528901f870 0x565289021c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f5661d9b640 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56528901f870 0x565289021c90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f338224007cbe1f server_cookie=262e45d4371c55be in_seq=0 out_seq=0 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f5651ffb640 10 client.25339 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f5651ffb640 1 -- 172.21.15.3:0/2548721578 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x565288c6d600 con 0x56528901f870 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f5651ffb640 10 client.25339 handle_command_reply: tid=0 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 client.25339 shutdown 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 msgr2=0x56528900c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x56528900c800 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f565c00fbe0 tx=0x7f565c004ab0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 shutdown_connections 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56528901f870 0x565289021c90 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652890069d0 0x56528900cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56528900f7a0 0x56528900d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f5651ffb640 0 client.25339 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 --2- 172.21.15.3:0/2548721578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x56528900c800 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.909+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 >> 172.21.15.3:0/2548721578 conn(0x5652890009d0 msgr2=0x565289002720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.910+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 shutdown_connections 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.910+0000 7f56737d3740 1 -- 172.21.15.3:0/2548721578 wait complete. 2024-02-09T22:51:42.916 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.910+0000 7f56737d3740 20 client.25339 trim_cache size 0 max 16384 2024-02-09T22:51:42.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.936+0000 7f56737d3740 1 Processor -- start 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.936+0000 7f56737d3740 1 -- start start 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.936+0000 7f56737d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 0x56528900ef30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.936+0000 7f56737d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890069d0 0x565289026610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.936+0000 7f56737d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x56528900b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f56737d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652890019d0 con 0x5652890060b0 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f56737d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56528901c2e0 con 0x56528900f7a0 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 0x56528900ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f56737d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565288ffaec0 con 0x5652890069d0 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 0x56528900ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43298/0 (socket says 172.21.15.3:43298) 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 -- 172.21.15.3:0/3332140856 learned_addr learned my addr 172.21.15.3:0/3332140856 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f5661d9b640 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890069d0 0x565289026610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566ad9d640 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x56528900b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 -- 172.21.15.3:0/3332140856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890069d0 msgr2=0x565289026610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890069d0 0x565289026610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 -- 172.21.15.3:0/3332140856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 msgr2=0x56528900b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x56528900b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 -- 172.21.15.3:0/3332140856 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565288f08820 con 0x5652890060b0 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f5661d9b640 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890069d0 0x565289026610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f566a59c640 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 0x56528900ef30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f565c016040 tx=0x7f565c00f760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c20ddcd9b0087ae9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.937+0000 7f5653fff640 1 -- 172.21.15.3:0/3332140856 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c022070 con 0x5652890060b0 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.938+0000 7f5653fff640 1 -- 172.21.15.3:0/3332140856 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f565c034070 con 0x5652890060b0 2024-02-09T22:51:42.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.938+0000 7f5653fff640 1 -- 172.21.15.3:0/3332140856 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565c0306e0 con 0x5652890060b0 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.938+0000 7f56737d3740 1 -- 172.21.15.3:0/3332140856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 msgr2=0x56528900ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.938+0000 7f56737d3740 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 0x56528900ef30 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f565c016040 tx=0x7f565c00f760 comp rx=0 tx=0).stop 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 -- 172.21.15.3:0/3332140856 shutdown_connections 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x56528900b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890069d0 0x565289026610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 --2- 172.21.15.3:0/3332140856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890060b0 0x56528900ef30 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 -- 172.21.15.3:0/3332140856 >> 172.21.15.3:0/3332140856 conn(0x5652890009d0 msgr2=0x565289007060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 -- 172.21.15.3:0/3332140856 shutdown_connections 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 -- 172.21.15.3:0/3332140856 wait complete. 2024-02-09T22:51:42.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 -- start start 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 Processor -- start 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x565289016040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x565289016580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.939+0000 7f56737d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x565289014470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f56737d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565289014b40 con 0x5652890069d0 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f56737d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565289014cb0 con 0x56528900f7a0 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f56737d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565289014e70 con 0x5652890060b0 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x565289016580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f56737d3740 10 client.0 _ll_register_callbacks cb 0x7f565c00f830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x565289016580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:43308/0 (socket says 172.21.15.3:43308) 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 -- 172.21.15.3:0/1783051070 learned_addr learned my addr 172.21.15.3:0/1783051070 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:42.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f566a59c640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x565289016040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f566ad9d640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x565289014470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 -- 172.21.15.3:0/1783051070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 msgr2=0x565289016040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x565289016040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 -- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 msgr2=0x565289014470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x565289014470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 -- 172.21.15.3:0/1783051070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565288f6bd90 con 0x5652890069d0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5661d9b640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x565289016580 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5654003780 tx=0x7f5654005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd27efa55d44abf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5651ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.940+0000 7f5651ffb640 1 -- 172.21.15.3:0/1783051070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56540c56e0 con 0x5652890069d0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.941+0000 7f5651ffb640 1 -- 172.21.15.3:0/1783051070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f56540bcec0 con 0x5652890069d0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.941+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f56540c1010 con 0x5652890069d0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.941+0000 7f5651ffb640 1 -- 172.21.15.3:0/1783051070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f56540b95f0 con 0x5652890069d0 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.941+0000 7f56737d3740 10 client.15777 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:42.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.941+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x565288f2e6f0 con 0x5652890069d0 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.941+0000 7f5651ffb640 1 -- 172.21.15.3:0/1783051070 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56540a6080 con 0x5652890069d0 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f5651ffb640 1 -- 172.21.15.3:0/1783051070 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f565404f020 con 0x5652890069d0 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f56737d3740 10 client.15777 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f56737d3740 10 client.15777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f56737d3740 20 client.15777 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f56737d3740 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565289076be0 0x565289079000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f56737d3740 4 client.15777 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.942+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f56540c1010 con 0x565289076be0 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.943+0000 7f566a59c640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565289076be0 0x565289079000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.943+0000 7f5651ffb640 10 client.15777 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:42.949 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.943+0000 7f566a59c640 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565289076be0 0x565289079000 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6beffaa600a31cf2 server_cookie=9c40cb9ed2efa4c8 in_seq=0 out_seq=0 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f5651ffb640 1 -- 172.21.15.3:0/1783051070 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f56540c1010 con 0x565289076be0 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f5651ffb640 10 client.15777 handle_command_reply: tid=0 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 client.15777 shutdown 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 msgr2=0x565289016580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x565289016580 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f5654003780 tx=0x7f5654005fd0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 shutdown_connections 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565289076be0 0x565289079000 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56528900f7a0 0x565289014470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652890069d0 0x565289016580 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 --2- 172.21.15.3:0/1783051070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652890060b0 0x565289016040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 >> 172.21.15.3:0/1783051070 conn(0x5652890009d0 msgr2=0x565289026610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f5651ffb640 0 client.15777 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:42.951 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 shutdown_connections 2024-02-09T22:51:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.947+0000 7f56737d3740 1 -- 172.21.15.3:0/1783051070 wait complete. 2024-02-09T22:51:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f56737d3740 20 client.15777 trim_cache size 0 max 16384 2024-02-09T22:51:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:51:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:51:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:51:42.952 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56580a22b0 msgr2=0x7f56580a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56580a22b0 0x7f56580a4770 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f565c005fd0 tx=0x7f565c009500 comp rx=0 tx=0).stop 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 msgr2=0x7f566c10de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10de40 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f5654000c00 tx=0x7f565402c6a0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 shutdown_connections 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56580a22b0 0x7f56580a4770 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f566c105860 0x7f566c10e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f566c100490 0x7f566c10de40 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 --2- 172.21.15.3:0/2027049701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f566c0ffad0 0x7f566c10fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.948+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 >> 172.21.15.3:0/2027049701 conn(0x7f566c075450 msgr2=0x7f566c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.949+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 shutdown_connections 2024-02-09T22:51:42.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:42.949+0000 7f5670b25640 1 -- 172.21.15.3:0/2027049701 wait complete. 2024-02-09T22:51:44.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:44 smithi064 ceph-mon[36638]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-02-09T22:51:44.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:44 smithi089 ceph-mon[36665]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-02-09T22:51:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:44 smithi003 ceph-mon[32678]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.5 KiB/s wr, 17 op/s 2024-02-09T22:51:46.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:46 smithi064 ceph-mon[36638]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-02-09T22:51:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:46 smithi089 ceph-mon[36665]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-02-09T22:51:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:46 smithi003 ceph-mon[32678]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 9 op/s 2024-02-09T22:51:48.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:48 smithi064 ceph-mon[36638]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-02-09T22:51:48.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:48 smithi089 ceph-mon[36665]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-02-09T22:51:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:48 smithi003 ceph-mon[32678]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 255 B/s wr, 14 op/s 2024-02-09T22:51:50.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:50 smithi064 ceph-mon[36638]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-02-09T22:51:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:50 smithi089 ceph-mon[36665]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-02-09T22:51:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:50 smithi003 ceph-mon[32678]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 9 op/s 2024-02-09T22:51:52.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:52 smithi064 ceph-mon[36638]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-02-09T22:51:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:52 smithi089 ceph-mon[36665]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-02-09T22:51:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:52 smithi003 ceph-mon[32678]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 10 op/s 2024-02-09T22:51:54.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:54 smithi064 ceph-mon[36638]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-02-09T22:51:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:54 smithi089 ceph-mon[36665]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-02-09T22:51:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:54 smithi003 ceph-mon[32678]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-02-09T22:51:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 -- 172.21.15.3:0/1893676992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd01024a0 msgr2=0x7fecd0102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 --2- 172.21.15.3:0/1893676992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd01024a0 0x7fecd0102880 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fecc0009900 tx=0x7fecc002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 -- 172.21.15.3:0/1893676992 shutdown_connections 2024-02-09T22:51:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 --2- 172.21.15.3:0/1893676992 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecd0102dc0 0x7fecd010b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 --2- 172.21.15.3:0/1893676992 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecd010b870 0x7fecd010dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 --2- 172.21.15.3:0/1893676992 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd01024a0 0x7fecd0102880 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 -- 172.21.15.3:0/1893676992 >> 172.21.15.3:0/1893676992 conn(0x7fecd0077250 msgr2=0x7fecd0077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:56.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 -- 172.21.15.3:0/1893676992 shutdown_connections 2024-02-09T22:51:56.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.050+0000 7fecd5257640 1 -- 172.21.15.3:0/1893676992 wait complete. 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 Processor -- start 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 -- start start 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecd01024a0 0x7fecd019f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 0x7fecd019fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecd010b870 0x7fecd01999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd019a020 con 0x7fecd0102dc0 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd019a190 con 0x7fecd010b870 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecd5257640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecd019a350 con 0x7fecd01024a0 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 0x7fecd019fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 0x7fecd019fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52362/0 (socket says 172.21.15.3:52362) 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.051+0000 7fecce59c640 1 -- 172.21.15.3:0/2972641583 learned_addr learned my addr 172.21.15.3:0/2972641583 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:56.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecced9d640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecd01024a0 0x7fecd019f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7feccf59e640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecd010b870 0x7fecd01999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecce59c640 1 -- 172.21.15.3:0/2972641583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecd01024a0 msgr2=0x7fecd019f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecce59c640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecd01024a0 0x7fecd019f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecce59c640 1 -- 172.21.15.3:0/2972641583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecd010b870 msgr2=0x7fecd01999d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecce59c640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecd010b870 0x7fecd01999d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecce59c640 1 -- 172.21.15.3:0/2972641583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc0009560 con 0x7fecd0102dc0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecce59c640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 0x7fecd019fd10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fecc400d950 tx=0x7fecc400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecb7fff640 1 -- 172.21.15.3:0/2972641583 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc4009c30 con 0x7fecd0102dc0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecb7fff640 1 -- 172.21.15.3:0/2972641583 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecc4009d90 con 0x7fecd0102dc0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecd019a630 con 0x7fecd0102dc0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecb7fff640 1 -- 172.21.15.3:0/2972641583 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc400f7e0 con 0x7fecd0102dc0 2024-02-09T22:51:56.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.052+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecd01a6870 con 0x7fecd0102dc0 2024-02-09T22:51:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.055+0000 7fecb7fff640 1 -- 172.21.15.3:0/2972641583 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 24) v1 ==== 136943+0+0 (secure 0 0 0) 0x7fecc4010ad0 con 0x7fecd0102dc0 2024-02-09T22:51:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 Processor -- start 2024-02-09T22:51:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 -- start start 2024-02-09T22:51:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x55648759a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487508a00 con 0x55648759a9d0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487505f30 con 0x55648759a0b0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecd7e20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487501710 con 0x5564875a37a0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52372/0 (socket says 172.21.15.3:52372) 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecced9d640 1 -- 172.21.15.3:0/2614939321 learned_addr learned my addr 172.21.15.3:0/2614939321 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7feccf59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x55648759a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecb7fff640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecb80a22b0 0x7fecb80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.056+0000 7fecce59c640 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecb80a22b0 0x7fecb80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7fecb7fff640 1 -- 172.21.15.3:0/2972641583 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fecc40cdf50 con 0x7fecd0102dc0 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 -- 172.21.15.3:0/2614939321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 msgr2=0x5564875a5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 -- 172.21.15.3:0/2614939321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 msgr2=0x5564875a3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 -- 172.21.15.3:0/2614939321 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564874c63b0 con 0x55648759a0b0 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7fecced9d640 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x55648759a490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecd006a490 tx=0x7fecc8012de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=138330cd45e59992 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7feccf59e640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecb80a22b0 0x7fecb80a4770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fecc801b000 tx=0x7fecc801f000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.057+0000 7fecb57fa640 1 -- 172.21.15.3:0/2614939321 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc8003b70 con 0x55648759a0b0 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecb57fa640 1 -- 172.21.15.3:0/2614939321 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecc801ed00 con 0x55648759a0b0 2024-02-09T22:51:56.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecb57fa640 1 -- 172.21.15.3:0/2614939321 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc80044b0 con 0x55648759a0b0 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 -- 172.21.15.3:0/2614939321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 msgr2=0x55648759a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x55648759a490 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fecd006a490 tx=0x7fecc8012de0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 -- 172.21.15.3:0/2614939321 shutdown_connections 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a5b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2614939321 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x55648759a490 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 -- 172.21.15.3:0/2614939321 >> 172.21.15.3:0/2614939321 conn(0x5564875949d0 msgr2=0x556487594db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 -- 172.21.15.3:0/2614939321 shutdown_connections 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.058+0000 7fecd7e20740 1 -- 172.21.15.3:0/2614939321 wait complete. 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 -- start start 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 Processor -- start 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x5564875a0780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564874a6b90 con 0x55648759a9d0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a0cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52380/0 (socket says 172.21.15.3:52380) 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecce59c640 1 -- 172.21.15.3:0/2272284699 learned_addr learned my addr 172.21.15.3:0/2272284699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564871d5250 con 0x55648759a0b0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487487200 con 0x5564875a37a0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecd7e20740 10 client.0 _ll_register_callbacks cb 0x5564874bd5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7fecced9d640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x5564875a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.059+0000 7feccf59e640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecce59c640 1 -- 172.21.15.3:0/2272284699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 msgr2=0x5564875a1200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecce59c640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a1200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecce59c640 1 -- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 msgr2=0x5564875a0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecce59c640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x5564875a0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecce59c640 1 -- 172.21.15.3:0/2272284699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc8012530 con 0x55648759a9d0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecce59c640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a0cc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fecc409a500 tx=0x7fecc406a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc8fe825658b634 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:56.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2272284699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc409a160 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2272284699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fecc4065d60 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556487527ee0 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2272284699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecc409af00 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecd7e20740 10 client.15789 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5564874ffd90 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.060+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2272284699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecc40950f0 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2272284699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fecc40cb8c0 con 0x55648759a9d0 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecd7e20740 10 client.15789 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecd7e20740 10 client.15789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecd7e20740 20 client.15789 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x5564875b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecd7e20740 4 client.15789 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:51:56.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556487527ee0 con 0x5564875b38e0 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.061+0000 7fecced9d640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x5564875b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecb2ff5640 10 client.15789 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecced9d640 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x5564875b5d10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcbd95bad26f6bf8 server_cookie=d8264688912bab29 in_seq=0 out_seq=0 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2272284699 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556487527ee0 con 0x5564875b38e0 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecb2ff5640 10 client.15789 handle_command_reply: tid=0 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecd7e20740 1 client.15789 shutdown 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 msgr2=0x5564875a0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.062+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a0cc0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fecc409a500 tx=0x7fecc406a660 comp rx=0 tx=0).stop 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 shutdown_connections 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x5564875b5d10 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564875a37a0 0x5564875a1200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecb2ff5640 0 client.15789 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a0cc0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2272284699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55648759a0b0 0x5564875a0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 >> 172.21.15.3:0/2272284699 conn(0x5564875949d0 msgr2=0x556487596720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 shutdown_connections 2024-02-09T22:51:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 1 -- 172.21.15.3:0/2272284699 wait complete. 2024-02-09T22:51:56.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.063+0000 7fecd7e20740 20 client.15789 trim_cache size 0 max 16384 2024-02-09T22:51:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 Processor -- start 2024-02-09T22:51:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 -- start start 2024-02-09T22:51:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a0b0 0x5564875a2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a1020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.080 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x55648759da00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487599990 con 0x55648759a9d0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55648759c380 con 0x5564875a37a0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.076+0000 7fecd7e20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564875baf40 con 0x55648759a0b0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a1020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52386/0 (socket says 172.21.15.3:52386) 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 -- 172.21.15.3:0/1278887613 learned_addr learned my addr 172.21.15.3:0/1278887613 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecced9d640 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a0b0 0x5564875a2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7feccf59e640 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x55648759da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 -- 172.21.15.3:0/1278887613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a0b0 msgr2=0x5564875a2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a0b0 0x5564875a2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 -- 172.21.15.3:0/1278887613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 msgr2=0x55648759da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x55648759da00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 -- 172.21.15.3:0/1278887613 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55648749c820 con 0x55648759a9d0 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7feccf59e640 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x55648759da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:51:56.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecce59c640 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a1020 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fecc40971d0 tx=0x7fecc40ce2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=107dfe9f3db74987 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecb4ff9640 1 -- 172.21.15.3:0/1278887613 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc40cb550 con 0x55648759a9d0 2024-02-09T22:51:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecb4ff9640 1 -- 172.21.15.3:0/1278887613 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecc4099ca0 con 0x55648759a9d0 2024-02-09T22:51:56.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.077+0000 7fecb4ff9640 1 -- 172.21.15.3:0/1278887613 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc409a800 con 0x55648759a9d0 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 -- 172.21.15.3:0/1278887613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 msgr2=0x5564875a1020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a1020 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fecc40971d0 tx=0x7fecc40ce2a0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 -- 172.21.15.3:0/1278887613 shutdown_connections 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x55648759da00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a9d0 0x5564875a1020 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 --2- 172.21.15.3:0/1278887613 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a0b0 0x5564875a2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 -- 172.21.15.3:0/1278887613 >> 172.21.15.3:0/1278887613 conn(0x5564875949d0 msgr2=0x55648759f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:56.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 -- 172.21.15.3:0/1278887613 shutdown_connections 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.078+0000 7fecd7e20740 1 -- 172.21.15.3:0/1278887613 wait complete. 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.079+0000 7fecd7e20740 1 -- start start 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.079+0000 7fecd7e20740 1 Processor -- start 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 0x5564875aa950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a9d0 0x5564875a8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 0x5564875aa950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 0x5564875aa950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52388/0 (socket says 172.21.15.3:52388) 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 -- 172.21.15.3:0/2922491866 learned_addr learned my addr 172.21.15.3:0/2922491866 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x5564875a93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564875a9ae0 con 0x55648759a0b0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487609e40 con 0x5564875a37a0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556487609f80 con 0x55648759a9d0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecce59c640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a9d0 0x5564875a8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecd7e20740 10 client.0 _ll_register_callbacks cb 0x7fecc40cb4a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7feccf59e640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x5564875a93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.087 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 -- 172.21.15.3:0/2922491866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a9d0 msgr2=0x5564875a8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a9d0 0x5564875a8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 -- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 msgr2=0x5564875a93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x5564875a93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.080+0000 7fecced9d640 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5564874ffd90 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecced9d640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 0x5564875aa950 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fecc0036040 tx=0x7fecc002c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a2750e98dd9f1a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecb2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2922491866 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecc0044040 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fecc801ed00 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2922491866 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fecc004b3b0 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2922491866 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecc0004510 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecd7e20740 10 client.15795 fetch_fsmap learned FSMap version 25 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5564874c26f0 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.081+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2922491866 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecc801ed00 con 0x55648759a0b0 2024-02-09T22:51:56.088 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2922491866 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fecc0053020 con 0x55648759a0b0 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecd7e20740 10 client.15795 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecd7e20740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecd7e20740 20 client.15795 populate_metadata read hostname 'smithi003' 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x55648760d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecd7e20740 4 client.15795 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556487201600 con 0x5564875b38e0 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.082+0000 7fecce59c640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x55648760d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.083+0000 7fecce59c640 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x55648760d610 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a9c31641e6e1818 server_cookie=52523f74e24091dc in_seq=0 out_seq=0 2024-02-09T22:51:56.089 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.083+0000 7fecb2ff5640 10 client.15795 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecb2ff5640 1 -- 172.21.15.3:0/2922491866 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556487201600 con 0x5564875b38e0 2024-02-09T22:51:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecb2ff5640 10 client.15795 handle_command_reply: tid=0 2024-02-09T22:51:56.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 client.15795 shutdown 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 msgr2=0x5564875aa950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 0x5564875aa950 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fecc0036040 tx=0x7fecc002c400 comp rx=0 tx=0).stop 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 shutdown_connections 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5564875b38e0 0x55648760d610 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecb2ff5640 0 client.15795 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564875a37a0 0x5564875a93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55648759a9d0 0x5564875a8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 --2- 172.21.15.3:0/2922491866 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55648759a0b0 0x5564875aa950 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 >> 172.21.15.3:0/2922491866 conn(0x5564875949d0 msgr2=0x5564875931a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 shutdown_connections 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 1 -- 172.21.15.3:0/2922491866 wait complete. 2024-02-09T22:51:56.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.090+0000 7fecd7e20740 20 client.15795 trim_cache size 0 max 16384 2024-02-09T22:51:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:51:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:51:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:51:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:51:56.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecb80a22b0 msgr2=0x7fecb80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecb80a22b0 0x7fecb80a4770 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fecc801b000 tx=0x7fecc801f000 comp rx=0 tx=0).stop 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 msgr2=0x7fecd019fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 0x7fecd019fd10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fecc400d950 tx=0x7fecc400de10 comp rx=0 tx=0).stop 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 shutdown_connections 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecb80a22b0 0x7fecb80a4770 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fecd010b870 0x7fecd01999d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fecd0102dc0 0x7fecd019fd10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 --2- 172.21.15.3:0/2972641583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fecd01024a0 0x7fecd019f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 >> 172.21.15.3:0/2972641583 conn(0x7fecd0077250 msgr2=0x7fecd0107790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 shutdown_connections 2024-02-09T22:51:56.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:51:56.091+0000 7fecd5257640 1 -- 172.21.15.3:0/2972641583 wait complete. 2024-02-09T22:51:56.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:56 smithi064 ceph-mon[36638]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-02-09T22:51:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:56 smithi089 ceph-mon[36665]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-02-09T22:51:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:56 smithi003 ceph-mon[32678]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-02-09T22:51:58.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:51:58 smithi064 ceph-mon[36638]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-02-09T22:51:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:51:58 smithi089 ceph-mon[36665]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-02-09T22:51:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:51:58 smithi003 ceph-mon[32678]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-02-09T22:52:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:00 smithi003 ceph-mon[32678]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-02-09T22:52:00.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:00 smithi064 ceph-mon[36638]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-02-09T22:52:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:00 smithi089 ceph-mon[36665]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-02-09T22:52:01.043 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:52:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:52:00.748+0000 7f0df4439700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:52:01.043 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:52:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:52:00.748+0000 7f0df4439700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:52:01.044 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:52:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:52:00.748+0000 7f0df4439700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:52:01.044 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:52:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:52:00.748+0000 7f0df4439700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:52:01.044 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 22:52:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x[32910]: 2024-02-09T22:52:00.748+0000 7f0df4439700 -1 client.0 error registering admin socket command: (17) File exists 2024-02-09T22:52:02.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:02 smithi003 ceph-mon[32678]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 9 op/s 2024-02-09T22:52:02.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:02 smithi064 ceph-mon[36638]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 9 op/s 2024-02-09T22:52:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:02 smithi089 ceph-mon[36665]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 9 op/s 2024-02-09T22:52:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:04 smithi089 ceph-mon[36665]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-02-09T22:52:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:04 smithi089 ceph-mon[36665]: mgrmap e25: x(active, since 48s), standbys: z, y 2024-02-09T22:52:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:04 smithi003 ceph-mon[32678]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-02-09T22:52:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:04 smithi003 ceph-mon[32678]: mgrmap e25: x(active, since 48s), standbys: z, y 2024-02-09T22:52:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:04 smithi064 ceph-mon[36638]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-02-09T22:52:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:04 smithi064 ceph-mon[36638]: mgrmap e25: x(active, since 48s), standbys: z, y 2024-02-09T22:52:05.119 INFO:tasks.workunit.client.0.smithi003.stderr:Updating files: 97% (11813/12104) Updating files: 98% (11862/12104) Updating files: 99% (11983/12104) Updating files: 100% (12104/12104) Updating files: 100% (12104/12104), done. 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr:Note: switching to '290212a1bfd33563ba76ee5b7f0b944fdb6c5d90'. 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr:changes and commit them, and you can discard any commits you make in this 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr:state without impacting any branches by switching back to a branch. 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.501 INFO:tasks.workunit.client.0.smithi003.stderr:If you want to create a new branch to retain commits you create, you may 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr:do so (now or later) by using -c with the switch command. Example: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: git switch -c 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr:Or undo this operation with: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: git switch - 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr: 2024-02-09T22:52:05.502 INFO:tasks.workunit.client.0.smithi003.stderr:HEAD is now at 290212a1bfd Merge pull request #53521 from xxhdx1985126/wip-62857 2024-02-09T22:52:05.505 DEBUG:teuthology.orchestra.run.smithi003:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-02-09T22:52:05.541 INFO:tasks.workunit.client.0.smithi003.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-02-09T22:52:05.543 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-09T22:52:05.543 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-02-09T22:52:06.248 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-02-09T22:52:06.303 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-02-09T22:52:06.343 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-02-09T22:52:06.344 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-09T22:52:06.344 INFO:tasks.workunit.client.0.smithi003.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-02-09T22:52:06.383 INFO:tasks.workunit.client.0.smithi003.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-02-09T22:52:06.386 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T22:52:06.386 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-02-09T22:52:06.442 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-02-09T22:52:06.443 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-02-09T22:52:06.444 DEBUG:teuthology.orchestra.run.smithi003:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-02-09T22:52:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:06 smithi089 ceph-mon[36665]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-02-09T22:52:06.502 INFO:tasks.workunit.client.0.smithi003.stderr:+ dbench 1 2024-02-09T22:52:06.510 INFO:tasks.workunit.client.0.smithi003.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-02-09T22:52:06.510 INFO:tasks.workunit.client.0.smithi003.stdout: 2024-02-09T22:52:06.511 INFO:tasks.workunit.client.0.smithi003.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-02-09T22:52:06.518 INFO:tasks.workunit.client.0.smithi003.stdout:0 of 1 processes prepared for launch 0 sec 2024-02-09T22:52:06.618 INFO:tasks.workunit.client.0.smithi003.stdout:1 of 1 processes prepared for launch 0 sec 2024-02-09T22:52:06.618 INFO:tasks.workunit.client.0.smithi003.stdout:releasing clients 2024-02-09T22:52:06.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:06 smithi003 ceph-mon[32678]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-02-09T22:52:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:06 smithi064 ceph-mon[36638]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 767 B/s wr, 10 op/s 2024-02-09T22:52:07.618 INFO:tasks.workunit.client.0.smithi003.stdout: 1 3650 38.63 MB/sec warmup 1 sec latency 6.166 ms 2024-02-09T22:52:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:08 smithi089 ceph-mon[36665]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-02-09T22:52:08.618 INFO:tasks.workunit.client.0.smithi003.stdout: 1 6752 26.70 MB/sec warmup 2 sec latency 6.090 ms 2024-02-09T22:52:08.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:08 smithi003 ceph-mon[32678]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-02-09T22:52:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:08 smithi064 ceph-mon[36638]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 23 KiB/s wr, 29 op/s 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.184+0000 7fe314c8f640 1 -- 172.21.15.3:0/1798012457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101e10 msgr2=0x7fe310105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.184+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1798012457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101e10 0x7fe310105e50 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fe304009930 tx=0x7fe30402bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 -- 172.21.15.3:0/1798012457 shutdown_connections 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1798012457 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 0x7fe31010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1798012457 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101e10 0x7fe310105e50 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1798012457 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe310101460 0x7fe310101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 -- 172.21.15.3:0/1798012457 >> 172.21.15.3:0/1798012457 conn(0x7fe3100fccc0 msgr2=0x7fe3100ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 -- 172.21.15.3:0/1798012457 shutdown_connections 2024-02-09T22:52:09.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 -- 172.21.15.3:0/1798012457 wait complete. 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 Processor -- start 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.185+0000 7fe314c8f640 1 -- start start 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 0x7fe31019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 0x7fe31019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe310197710 0x7fe310197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3101981e0 con 0x7fe310101460 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe310198350 con 0x7fe310197710 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe310198510 con 0x7fe310106530 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 0x7fe31019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 0x7fe31019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57144/0 (socket says 172.21.15.3:57144) 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 -- 172.21.15.3:0/1225160429 learned_addr learned my addr 172.21.15.3:0/1225160429 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 0x7fe31019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe310197710 0x7fe310197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 -- 172.21.15.3:0/1225160429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 msgr2=0x7fe31019db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 0x7fe31019db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 -- 172.21.15.3:0/1225160429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe310197710 msgr2=0x7fe310197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe310197710 0x7fe310197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 -- 172.21.15.3:0/1225160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe304009560 con 0x7fe310101460 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 0x7fe31019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe30e59c640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 0x7fe31019d600 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe30000e9f0 tx=0x7fe30000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe2ff7fe640 1 -- 172.21.15.3:0/1225160429 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300009700 con 0x7fe310101460 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe2ff7fe640 1 -- 172.21.15.3:0/1225160429 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe30000cd80 con 0x7fe310101460 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3101a4460 con 0x7fe310101460 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe2ff7fe640 1 -- 172.21.15.3:0/1225160429 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300018920 con 0x7fe310101460 2024-02-09T22:52:09.193 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.186+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101a4990 con 0x7fe310101460 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.188+0000 7fe2ff7fe640 1 -- 172.21.15.3:0/1225160429 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7fe300018a80 con 0x7fe310101460 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 Processor -- start 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 -- start start 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe2ff7fe640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe2f80a23c0 0x7fe2f80a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a7830f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78318260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a7831ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7827da00 con 0x555a7830f0b0 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7827af30 con 0x555a7830f9d0 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe31785e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a78276710 con 0x555a783187a0 2024-02-09T22:52:09.198 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a7830f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe2f80a23c0 0x7fe2f80a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe30dd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78318260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a7830f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57146/0 (socket says 172.21.15.3:57146) 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe30e59c640 1 -- 172.21.15.3:0/3861986494 learned_addr learned my addr 172.21.15.3:0/3861986494 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a7831ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.189+0000 7fe2ff7fe640 1 -- 172.21.15.3:0/1225160429 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe300014070 con 0x7fe310101460 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30e59c640 1 -- 172.21.15.3:0/3861986494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 msgr2=0x555a7831ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30e59c640 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a7831ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30e59c640 1 -- 172.21.15.3:0/3861986494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 msgr2=0x555a78318260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30e59c640 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78318260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30e59c640 1 -- 172.21.15.3:0/3861986494 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a7823b3b0 con 0x555a7830f0b0 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78318260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe2f80a23c0 0x7fe2f80a4880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe308005fd0 tx=0x7fe308009e90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe30e59c640 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a7830f490 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe3000cc390 tx=0x7fe300005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0c86ca1e989b37a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe2fcff9640 1 -- 172.21.15.3:0/3861986494 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300052ea0 con 0x555a7830f0b0 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe2fcff9640 1 -- 172.21.15.3:0/3861986494 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3000ad630 con 0x555a7830f0b0 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.190+0000 7fe2fcff9640 1 -- 172.21.15.3:0/3861986494 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe300099000 con 0x555a7830f0b0 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 -- 172.21.15.3:0/3861986494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 msgr2=0x555a7830f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a7830f490 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fe3000cc390 tx=0x7fe300005760 comp rx=0 tx=0).stop 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 -- 172.21.15.3:0/3861986494 shutdown_connections 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78318260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a7831ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 --2- 172.21.15.3:0/3861986494 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a7830f490 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 -- 172.21.15.3:0/3861986494 >> 172.21.15.3:0/3861986494 conn(0x555a783099d0 msgr2=0x555a78309db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 -- 172.21.15.3:0/3861986494 shutdown_connections 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 -- 172.21.15.3:0/3861986494 wait complete. 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 -- start start 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.191+0000 7fe31785e740 1 Processor -- start 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78315890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7821bb90 con 0x555a7830f0b0 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a77f4a250 con 0x555a7830f9d0 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a781fc200 con 0x555a783187a0 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78315890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe31785e740 10 client.0 _ll_register_callbacks cb 0x555a782325d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78315890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57158/0 (socket says 172.21.15.3:57158) 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 -- 172.21.15.3:0/191803510 learned_addr learned my addr 172.21.15.3:0/191803510 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.201 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 -- 172.21.15.3:0/191803510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 msgr2=0x555a78316310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 -- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 msgr2=0x555a78315dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 -- 172.21.15.3:0/191803510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101a4990 con 0x555a7830f0b0 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe30e59c640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78315890 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe30009ae20 tx=0x7fe3000a3580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bc1e7935fb0fb9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.202 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.192+0000 7fe2f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/191803510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30009ae90 con 0x555a7830f0b0 2024-02-09T22:52:09.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/191803510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe3000aee30 con 0x555a7830f0b0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555a7829cee0 con 0x555a7830f0b0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/191803510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3000af030 con 0x555a7830f0b0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 10 client.15810 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555a78274d90 con 0x555a7830f0b0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/191803510 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555a7829cee0 con 0x555a7830f0b0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/191803510 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe30009a2e0 con 0x555a7830f0b0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 10 client.15810 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 10 client.15810 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 20 client.15810 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78328700 0x555a7832aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 4 client.15810 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555a77f76600 con 0x555a78328700 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.195+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78328700 0x555a7832aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.196+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78328700 0x555a7832aaf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4463df96e070aab server_cookie=2ac9872b7e6d0443 in_seq=0 out_seq=0 2024-02-09T22:52:09.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.196+0000 7fe2f5ffb640 10 client.15810 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/191803510 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555a77f76600 con 0x555a78328700 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe2f5ffb640 10 client.15810 handle_command_reply: tid=0 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 client.15810 shutdown 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 msgr2=0x555a78315890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78315890 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7fe30009ae20 tx=0x7fe3000a3580 comp rx=0 tx=0).stop 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 shutdown_connections 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78328700 0x555a7832aaf0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe2f5ffb640 0 client.15810 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 --2- 172.21.15.3:0/191803510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78315890 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 >> 172.21.15.3:0/191803510 conn(0x555a783099d0 msgr2=0x555a7830b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 shutdown_connections 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.197+0000 7fe31785e740 1 -- 172.21.15.3:0/191803510 wait complete. 2024-02-09T22:52:09.206 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.198+0000 7fe31785e740 20 client.15810 trim_cache size 0 max 16384 2024-02-09T22:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.211+0000 7fe31785e740 1 Processor -- start 2024-02-09T22:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.211+0000 7fe31785e740 1 -- start start 2024-02-09T22:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe31785e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78317fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe31785e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe31785e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe31785e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a783050d0 con 0x555a7830f0b0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe31785e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7830a9d0 con 0x555a7830f9d0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe31785e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a783254b0 con 0x555a783187a0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78317fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78317fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57168/0 (socket says 172.21.15.3:57168) 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 -- 172.21.15.3:0/988868201 learned_addr learned my addr 172.21.15.3:0/988868201 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 -- 172.21.15.3:0/988868201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 msgr2=0x555a78316320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 -- 172.21.15.3:0/988868201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 msgr2=0x555a78315de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 -- 172.21.15.3:0/988868201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555a78211820 con 0x555a7830f0b0 2024-02-09T22:52:09.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe30e59c640 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78317fc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe300097860 tx=0x7fe3000993a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67593408856b1636 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe2f7fff640 1 -- 172.21.15.3:0/988868201 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe30009ae90 con 0x555a7830f0b0 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe2f7fff640 1 -- 172.21.15.3:0/988868201 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe300085e90 con 0x555a7830f0b0 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.212+0000 7fe2f7fff640 1 -- 172.21.15.3:0/988868201 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3000cbef0 con 0x555a7830f0b0 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 -- 172.21.15.3:0/988868201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 msgr2=0x555a78317fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78317fc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe300097860 tx=0x7fe3000993a0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 -- 172.21.15.3:0/988868201 shutdown_connections 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a7830f9d0 0x555a78315de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a783187a0 0x555a78316320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 --2- 172.21.15.3:0/988868201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f0b0 0x555a78317fc0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 -- 172.21.15.3:0/988868201 >> 172.21.15.3:0/988868201 conn(0x555a783099d0 msgr2=0x555a78314160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 -- 172.21.15.3:0/988868201 shutdown_connections 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 -- 172.21.15.3:0/988868201 wait complete. 2024-02-09T22:52:09.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 -- start start 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.213+0000 7fe31785e740 1 Processor -- start 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a7830f0b0 0x555a7831de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 0x555a7831e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a783187a0 0x555a7831e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7837eba0 con 0x555a7830f9d0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7837ece0 con 0x555a783187a0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555a7837ee20 con 0x555a7830f0b0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 10 client.0 _ll_register_callbacks cb 0x7fe3080093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 0x555a7831e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 0x555a7831e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57174/0 (socket says 172.21.15.3:57174) 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 -- 172.21.15.3:0/99972597 learned_addr learned my addr 172.21.15.3:0/99972597 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30ed9d640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a783187a0 0x555a7831e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30e59c640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a7830f0b0 0x555a7831de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 -- 172.21.15.3:0/99972597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a7830f0b0 msgr2=0x555a7831de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a7830f0b0 0x555a7831de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 -- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a783187a0 msgr2=0x555a7831e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a783187a0 0x555a7831e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 -- 172.21.15.3:0/99972597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe308019040 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe30dd9b640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 0x555a7831e350 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe304009900 tx=0x7fe30402ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6881f7dc3768ed09 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe2f5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/99972597 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe304049be0 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3000ad630 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.214+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/99972597 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe304046040 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/99972597 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe304049d40 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 10 client.15816 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x555a78274d90 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/99972597 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3000ad630 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/99972597 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe30403c040 con 0x555a7830f9d0 2024-02-09T22:52:09.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 10 client.15816 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 10 client.15816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 20 client.15816 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78380640 0x555a78382a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 4 client.15816 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555a77fc71b0 con 0x555a78380640 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.215+0000 7fe30e59c640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78380640 0x555a78382a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.216+0000 7fe2f5ffb640 10 client.15816 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.216+0000 7fe30e59c640 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78380640 0x555a78382a60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a2f74e1bae828 server_cookie=d247a25f70cf364a in_seq=0 out_seq=0 2024-02-09T22:52:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.297+0000 7fe2f5ffb640 1 -- 172.21.15.3:0/99972597 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555a77fc71b0 con 0x555a78380640 2024-02-09T22:52:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe2f5ffb640 10 client.15816 handle_command_reply: tid=0 2024-02-09T22:52:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 client.15816 shutdown 2024-02-09T22:52:09.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 msgr2=0x555a7831e350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 0x555a7831e350 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe304009900 tx=0x7fe30402ca80 comp rx=0 tx=0).stop 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 shutdown_connections 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555a78380640 0x555a78382a60 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555a783187a0 0x555a7831e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555a7830f9d0 0x555a7831e350 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe2f5ffb640 0 client.15816 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 --2- 172.21.15.3:0/99972597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555a7830f0b0 0x555a7831de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 >> 172.21.15.3:0/99972597 conn(0x555a783099d0 msgr2=0x555a7830ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 shutdown_connections 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 1 -- 172.21.15.3:0/99972597 wait complete. 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe31785e740 20 client.15816 trim_cache size 0 max 16384 2024-02-09T22:52:09.303 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:52:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:52:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:52:09.304 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe2f80a23c0 msgr2=0x7fe2f80a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe2f80a23c0 0x7fe2f80a4880 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe308005fd0 tx=0x7fe308009e90 comp rx=0 tx=0).stop 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 msgr2=0x7fe31019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 0x7fe31019d600 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fe30000e9f0 tx=0x7fe30000eeb0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 shutdown_connections 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.298+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe2f80a23c0 0x7fe2f80a4880 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.299+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe310197710 0x7fe310197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.299+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe310106530 0x7fe31019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.299+0000 7fe314c8f640 1 --2- 172.21.15.3:0/1225160429 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe310101460 0x7fe31019d600 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.299+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 >> 172.21.15.3:0/1225160429 conn(0x7fe3100fccc0 msgr2=0x7fe3100fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.299+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 shutdown_connections 2024-02-09T22:52:09.305 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:09.299+0000 7fe314c8f640 1 -- 172.21.15.3:0/1225160429 wait complete. 2024-02-09T22:52:09.445 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:52:09.445 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 -- 172.21.15.3:0/4161483558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 msgr2=0x7fb6241003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 --2- 172.21.15.3:0/4161483558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb6241003b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009860 tx=0x7fb61c02bc20 comp rx=0 tx=0).stop 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 -- 172.21.15.3:0/4161483558 shutdown_connections 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 --2- 172.21.15.3:0/4161483558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6241008f0 0x7fb62410d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 --2- 172.21.15.3:0/4161483558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb62410db80 0x7fb62410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 --2- 172.21.15.3:0/4161483558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb6241003b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 -- 172.21.15.3:0/4161483558 >> 172.21.15.3:0/4161483558 conn(0x7fb6240772e0 msgr2=0x7fb6240776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 -- 172.21.15.3:0/4161483558 shutdown_connections 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.548+0000 7fb62949d640 1 -- 172.21.15.3:0/4161483558 wait complete. 2024-02-09T22:52:09.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.549+0000 7fb62949d640 1 Processor -- start 2024-02-09T22:52:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.549+0000 7fb62949d640 1 -- start start 2024-02-09T22:52:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.549+0000 7fb62949d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb62419f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb62949d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6241008f0 0x7fb62419fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb62949d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb62410db80 0x7fb6241999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb62949d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62419a000 con 0x7fb6241008f0 2024-02-09T22:52:09.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb62949d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62419a170 con 0x7fb62410db80 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb62949d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62419a330 con 0x7fb6240fffd0 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6241008f0 0x7fb62419fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb62419f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb62419f800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:58704/0 (socket says 172.21.15.3:58704) 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6241008f0 0x7fb62419fd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57190/0 (socket says 172.21.15.3:57190) 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 -- 172.21.15.3:0/2830047181 learned_addr learned my addr 172.21.15.3:0/2830047181 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb6237fe640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb62410db80 0x7fb6241999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 -- 172.21.15.3:0/2830047181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb62410db80 msgr2=0x7fb6241999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb62410db80 0x7fb6241999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 -- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6241008f0 msgr2=0x7fb62419fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6241008f0 0x7fb62419fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 -- 172.21.15.3:0/2830047181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb614009900 con 0x7fb6240fffd0 2024-02-09T22:52:09.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb622ffd640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb62419f800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009980 tx=0x7fb61c034670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb60bfff640 1 -- 172.21.15.3:0/2830047181 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c02c7c0 con 0x7fb6240fffd0 2024-02-09T22:52:09.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb60bfff640 1 -- 172.21.15.3:0/2830047181 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb61c034a70 con 0x7fb6240fffd0 2024-02-09T22:52:09.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c009490 con 0x7fb6240fffd0 2024-02-09T22:52:09.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.550+0000 7fb60bfff640 1 -- 172.21.15.3:0/2830047181 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c03d650 con 0x7fb6240fffd0 2024-02-09T22:52:09.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.551+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6241a66c0 con 0x7fb6240fffd0 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.553+0000 7fb60bfff640 1 -- 172.21.15.3:0/2830047181 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7fb61c03b040 con 0x7fb6240fffd0 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.553+0000 7fb62c074740 1 Processor -- start 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.553+0000 7fb62c074740 1 -- start start 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb62c074740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a808d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb62c074740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a8118e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb62c074740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a811eb0 0x55823a814300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb62c074740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a7c3d00 con 0x55823a809330 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb62c074740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a7782f0 con 0x55823a808980 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb62c074740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a775820 con 0x55823a811eb0 2024-02-09T22:52:09.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a8118e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb60bfff640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2350 0x7fb60c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a808d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a811eb0 0x55823a814300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a8118e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57192/0 (socket says 172.21.15.3:57192) 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a808d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43332/0 (socket says 172.21.15.3:43332) 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb622ffd640 1 -- 172.21.15.3:0/2536188007 learned_addr learned my addr 172.21.15.3:0/2536188007 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 -- 172.21.15.3:0/2536188007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 msgr2=0x55823a808d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a808d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6237fe640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2350 0x7fb60c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 -- 172.21.15.3:0/2536188007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 msgr2=0x55823a8118e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a8118e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 -- 172.21.15.3:0/2536188007 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55823a732ff0 con 0x55823a811eb0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb60bfff640 1 -- 172.21.15.3:0/2830047181 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb61c0efb00 con 0x7fb6240fffd0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.554+0000 7fb6227fc640 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a811eb0 0x55823a814300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb62419b1a0 tx=0x7fb618011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8605219b5db121d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb6097fa640 1 -- 172.21.15.3:0/2536188007 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb618004310 con 0x55823a811eb0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb6237fe640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2350 0x7fb60c0a4810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb614009fd0 tx=0x7fb614004ab0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb6097fa640 1 -- 172.21.15.3:0/2536188007 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb618004470 con 0x55823a811eb0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb6097fa640 1 -- 172.21.15.3:0/2536188007 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6180035f0 con 0x55823a811eb0 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 -- 172.21.15.3:0/2536188007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a811eb0 msgr2=0x55823a814300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a811eb0 0x55823a814300 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb62419b1a0 tx=0x7fb618011440 comp rx=0 tx=0).stop 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 -- 172.21.15.3:0/2536188007 shutdown_connections 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a811eb0 0x55823a814300 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a8118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 --2- 172.21.15.3:0/2536188007 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a808d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 -- 172.21.15.3:0/2536188007 >> 172.21.15.3:0/2536188007 conn(0x55823a803050 msgr2=0x55823a803430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 -- 172.21.15.3:0/2536188007 shutdown_connections 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.555+0000 7fb62c074740 1 -- 172.21.15.3:0/2536188007 wait complete. 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 -- start start 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 Processor -- start 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a809330 0x55823a80f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a80f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a771000 con 0x55823a811eb0 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a444430 con 0x55823a809330 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a400e00 con 0x55823a808980 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb62c074740 10 client.0 _ll_register_callbacks cb 0x55823a7657f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb6237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a80f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:58720/0 (socket says 172.21.15.3:58720) 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb6237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a80f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57204/0 (socket says 172.21.15.3:57204) 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 -- 172.21.15.3:0/3431549336 learned_addr learned my addr 172.21.15.3:0/3431549336 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb6227fc640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a809330 0x55823a80f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 -- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a809330 msgr2=0x55823a80f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a809330 0x55823a80f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 -- 172.21.15.3:0/3431549336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 msgr2=0x55823a80f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a80f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb622ffd640 1 -- 172.21.15.3:0/3431549336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb618009830 con 0x55823a808980 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.556+0000 7fb6237fe640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a80f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb622ffd640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80eca0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb61c04c000 tx=0x7fb61c0dc0b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=792bbd59688b4ce8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb606ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb606ff5640 1 -- 172.21.15.3:0/3431549336 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0bc8f0 con 0x55823a808980 2024-02-09T22:52:09.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb606ff5640 1 -- 172.21.15.3:0/3431549336 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb61c003e30 con 0x55823a808980 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55823a4707e0 con 0x55823a808980 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb606ff5640 1 -- 172.21.15.3:0/3431549336 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb61c003990 con 0x55823a808980 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 10 client.25399 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55823a76f680 con 0x55823a808980 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb606ff5640 1 -- 172.21.15.3:0/3431549336 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55823a4707e0 con 0x55823a808980 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb606ff5640 1 -- 172.21.15.3:0/3431549336 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb61c0c00d0 con 0x55823a808980 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 10 client.25399 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 10 client.25399 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 20 client.25399 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a821ea0 0x55823a824290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 4 client.25399 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.557+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55823a4ad720 con 0x55823a821ea0 2024-02-09T22:52:09.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.558+0000 7fb6227fc640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a821ea0 0x55823a824290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.558+0000 7fb6227fc640 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a821ea0 0x55823a824290 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7cbd61d3f7cf8b7a server_cookie=448f2559917ee54b in_seq=0 out_seq=0 2024-02-09T22:52:09.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.558+0000 7fb606ff5640 10 client.25399 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.559+0000 7fb606ff5640 1 -- 172.21.15.3:0/3431549336 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55823a4ad720 con 0x55823a821ea0 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.559+0000 7fb606ff5640 10 client.25399 handle_command_reply: tid=0 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.559+0000 7fb62c074740 1 client.25399 shutdown 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.559+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 msgr2=0x55823a80eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.559+0000 7fb62c074740 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80eca0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb61c04c000 tx=0x7fb61c0dc0b0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 shutdown_connections 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a821ea0 0x55823a824290 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a809330 0x55823a80f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a80f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 --2- 172.21.15.3:0/3431549336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80eca0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb606ff5640 0 client.25399 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 >> 172.21.15.3:0/3431549336 conn(0x55823a803050 msgr2=0x55823a804da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 shutdown_connections 2024-02-09T22:52:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 1 -- 172.21.15.3:0/3431549336 wait complete. 2024-02-09T22:52:09.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.560+0000 7fb62c074740 20 client.25399 trim_cache size 0 max 16384 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 Processor -- start 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 -- start start 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a80f540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a811eb0 0x55823a802070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a6f5fd0 con 0x55823a809330 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a80a630 con 0x55823a811eb0 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.575+0000 7fb62c074740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a4707e0 con 0x55823a808980 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a80f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a80f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57214/0 (socket says 172.21.15.3:57214) 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:58730/0 (socket says 172.21.15.3:58730) 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 -- 172.21.15.3:0/692246738 learned_addr learned my addr 172.21.15.3:0/692246738 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6237fe640 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a811eb0 0x55823a802070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 -- 172.21.15.3:0/692246738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 msgr2=0x55823a80de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 -- 172.21.15.3:0/692246738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a811eb0 msgr2=0x55823a802070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a811eb0 0x55823a802070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 -- 172.21.15.3:0/692246738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55823a70bb30 con 0x55823a809330 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb622ffd640 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb6227fc640 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a80f540 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb6180058c0 tx=0x7fb618009b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d59c340a2dcbd623 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb608ff9640 1 -- 172.21.15.3:0/692246738 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb618004680 con 0x55823a809330 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb608ff9640 1 -- 172.21.15.3:0/692246738 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb618004110 con 0x55823a809330 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.576+0000 7fb608ff9640 1 -- 172.21.15.3:0/692246738 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb618003b50 con 0x55823a809330 2024-02-09T22:52:09.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 -- 172.21.15.3:0/692246738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 msgr2=0x55823a80f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a80f540 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fb6180058c0 tx=0x7fb618009b40 comp rx=0 tx=0).stop 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 -- 172.21.15.3:0/692246738 shutdown_connections 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a811eb0 0x55823a802070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a809330 0x55823a80f540 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 --2- 172.21.15.3:0/692246738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a808980 0x55823a80de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 -- 172.21.15.3:0/692246738 >> 172.21.15.3:0/692246738 conn(0x55823a803050 msgr2=0x55823a801960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 -- 172.21.15.3:0/692246738 shutdown_connections 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 -- 172.21.15.3:0/692246738 wait complete. 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 -- start start 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 Processor -- start 2024-02-09T22:52:09.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.577+0000 7fb62c074740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a817a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 0x55823a817fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a818500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a878440 con 0x55823a811eb0 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a8785b0 con 0x55823a808980 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55823a878770 con 0x55823a809330 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 10 client.0 _ll_register_callbacks cb 0x7fb618009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 0x55823a817fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a818500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a817a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 0x55823a817fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:58738/0 (socket says 172.21.15.3:58738) 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 -- 172.21.15.3:0/2514426153 learned_addr learned my addr 172.21.15.3:0/2514426153 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a818500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57216/0 (socket says 172.21.15.3:57216) 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb622ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a817a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43358/0 (socket says 172.21.15.3:43358) 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 -- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 msgr2=0x55823a817a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a817a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 -- 172.21.15.3:0/2514426153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 msgr2=0x55823a818500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a818500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 -- 172.21.15.3:0/2514426153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55823a76f680 con 0x55823a809330 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb6227fc640 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 0x55823a817fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb618005890 tx=0x7fb618003730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff807ae61a0b5259 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb606ff5640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb606ff5640 1 -- 172.21.15.3:0/2514426153 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb618037070 con 0x55823a809330 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb606ff5640 1 -- 172.21.15.3:0/2514426153 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb618003ae0 con 0x55823a809330 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.578+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb618004470 con 0x55823a809330 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb606ff5640 1 -- 172.21.15.3:0/2514426153 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb618003ca0 con 0x55823a809330 2024-02-09T22:52:09.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 10 client.25405 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fb61801d040 con 0x55823a809330 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb606ff5640 1 -- 172.21.15.3:0/2514426153 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb618040020 con 0x55823a809330 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb606ff5640 1 -- 172.21.15.3:0/2514426153 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb61803a030 con 0x55823a809330 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 10 client.25405 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 10 client.25405 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 20 client.25405 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a879c80 0x55823a87c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 4 client.25405 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55823a4fc890 con 0x55823a879c80 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb622ffd640 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a879c80 0x55823a87c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb622ffd640 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a879c80 0x55823a87c0a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bac731106bb26d0c server_cookie=e28376073f696d81 in_seq=0 out_seq=0 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.579+0000 7fb606ff5640 10 client.25405 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb606ff5640 1 -- 172.21.15.3:0/2514426153 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55823a4fc890 con 0x55823a879c80 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb606ff5640 10 client.25405 handle_command_reply: tid=0 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 client.25405 shutdown 2024-02-09T22:52:09.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 msgr2=0x55823a817fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 0x55823a817fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fb618005890 tx=0x7fb618003730 comp rx=0 tx=0).stop 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 shutdown_connections 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55823a879c80 0x55823a87c0a0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55823a811eb0 0x55823a818500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb606ff5640 0 client.25405 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55823a809330 0x55823a817fc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 --2- 172.21.15.3:0/2514426153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55823a808980 0x55823a817a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 >> 172.21.15.3:0/2514426153 conn(0x55823a803050 msgr2=0x55823a804050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 shutdown_connections 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 1 -- 172.21.15.3:0/2514426153 wait complete. 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62c074740 20 client.25405 trim_cache size 0 max 16384 2024-02-09T22:52:09.591 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:52:09.592 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:52:09.592 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:52:09.592 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2350 msgr2=0x7fb60c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62949d640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2350 0x7fb60c0a4810 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb614009fd0 tx=0x7fb614004ab0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 msgr2=0x7fb62419f800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.580+0000 7fb62949d640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb62419f800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009980 tx=0x7fb61c034670 comp rx=0 tx=0).stop 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 shutdown_connections 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2350 0x7fb60c0a4810 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb62410db80 0x7fb6241999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6241008f0 0x7fb62419fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 --2- 172.21.15.3:0/2830047181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6240fffd0 0x7fb62419f800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 >> 172.21.15.3:0/2830047181 conn(0x7fb6240772e0 msgr2=0x7fb62410f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 shutdown_connections 2024-02-09T22:52:09.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.581+0000 7fb62949d640 1 -- 172.21.15.3:0/2830047181 wait complete. 2024-02-09T22:52:09.596 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:52:09.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:52:09.618 INFO:tasks.workunit.client.0.smithi003.stdout: 1 9693 22.41 MB/sec warmup 3 sec latency 32.658 ms 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.696+0000 7f8a7119a640 1 -- 172.21.15.3:0/3271869462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c100d00 msgr2=0x7f8a6c104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.696+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3271869462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c100d00 0x7f8a6c104d40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009930 tx=0x7f8a5402bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 -- 172.21.15.3:0/3271869462 shutdown_connections 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3271869462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c105420 0x7f8a6c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3271869462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c100d00 0x7f8a6c104d40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3271869462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 0x7f8a6c100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 -- 172.21.15.3:0/3271869462 >> 172.21.15.3:0/3271869462 conn(0x7f8a6c077370 msgr2=0x7f8a6c077780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 -- 172.21.15.3:0/3271869462 shutdown_connections 2024-02-09T22:52:09.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 -- 172.21.15.3:0/3271869462 wait complete. 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.697+0000 7f8a7119a640 1 Processor -- start 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 -- start start 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 0x7f8a6c1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 0x7f8a6c198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c1991a0 0x7f8a6c1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1a3df0 con 0x7f8a6c105420 2024-02-09T22:52:09.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1a3f60 con 0x7f8a6c1991a0 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a7119a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a6c1a4120 con 0x7f8a6c100350 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 0x7f8a6c198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 0x7f8a6c198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57238/0 (socket says 172.21.15.3:57238) 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6b59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c1991a0 0x7f8a6c1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 -- 172.21.15.3:0/3753337686 learned_addr learned my addr 172.21.15.3:0/3753337686 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6ad9d640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 0x7f8a6c1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 -- 172.21.15.3:0/3753337686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 msgr2=0x7f8a6c1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 0x7f8a6c1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 -- 172.21.15.3:0/3753337686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c1991a0 msgr2=0x7f8a6c1a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c1991a0 0x7f8a6c1a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6a59c640 1 -- 172.21.15.3:0/3753337686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a64009490 con 0x7f8a6c105420 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6ad9d640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 0x7f8a6c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.698+0000 7f8a6b59e640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c1991a0 0x7f8a6c1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.699+0000 7f8a6a59c640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 0x7f8a6c198c10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8a54005ab0 tx=0x7f8a5402c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.699+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a540042d0 con 0x7f8a6c105420 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.699+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a54009560 con 0x7f8a6c105420 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.699+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a54004430 con 0x7f8a6c105420 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.699+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a54003be0 con 0x7f8a6c105420 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.699+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a6c1a46f0 con 0x7f8a6c105420 2024-02-09T22:52:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.700+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a6c069160 con 0x7f8a6c105420 2024-02-09T22:52:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.701+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f8a54003d40 con 0x7f8a6c105420 2024-02-09T22:52:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.701+0000 7f8a63fff640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8a580a2350 0x7f8a580a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.701+0000 7f8a6ad9d640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8a580a2350 0x7f8a580a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.701+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8a540f08c0 con 0x7f8a6c105420 2024-02-09T22:52:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.702+0000 7f8a6ad9d640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8a580a2350 0x7f8a580a4810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011630 tx=0x7f8a64005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:09.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.706+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f8a540b9440 con 0x7f8a6c105420 2024-02-09T22:52:09.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.930+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a6c108650 con 0x7f8a6c105420 2024-02-09T22:52:09.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.931+0000 7f8a63fff640 1 -- 172.21.15.3:0/3753337686 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f8a54043090 con 0x7f8a6c105420 2024-02-09T22:52:09.935 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:52:09.935 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:51:00.398692+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:52:09.935 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 25 2024-02-09T22:52:09.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8a580a2350 msgr2=0x7f8a580a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8a580a2350 0x7f8a580a4810 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8a64011630 tx=0x7f8a64005f90 comp rx=0 tx=0).stop 2024-02-09T22:52:09.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 msgr2=0x7f8a6c198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:09.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 0x7f8a6c198c10 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8a54005ab0 tx=0x7f8a5402c5f0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 shutdown_connections 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8a580a2350 0x7f8a580a4810 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8a6c1991a0 0x7f8a6c1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8a6c105420 0x7f8a6c198c10 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 --2- 172.21.15.3:0/3753337686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8a6c100350 0x7f8a6c1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 >> 172.21.15.3:0/3753337686 conn(0x7f8a6c077370 msgr2=0x7f8a6c0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 shutdown_connections 2024-02-09T22:52:09.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:09.933+0000 7f8a7119a640 1 -- 172.21.15.3:0/3753337686 wait complete. 2024-02-09T22:52:09.948 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:09.948 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.092+0000 7f69cc6f4640 1 -- 172.21.15.3:0/3369121483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4100cf0 msgr2=0x7f69c4104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.092+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/3369121483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4100cf0 0x7f69c4104d30 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009930 tx=0x7f69b802bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- 172.21.15.3:0/3369121483 shutdown_connections 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/3369121483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c4105410 0x7f69c410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/3369121483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4100cf0 0x7f69c4104d30 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/3369121483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4100340 0x7f69c4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- 172.21.15.3:0/3369121483 >> 172.21.15.3:0/3369121483 conn(0x7f69c40772e0 msgr2=0x7f69c40776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- 172.21.15.3:0/3369121483 shutdown_connections 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- 172.21.15.3:0/3369121483 wait complete. 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 Processor -- start 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- start start 2024-02-09T22:52:10.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c4100340 0x7f69c4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 0x7f69c4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4199140 0x7f69c41a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41a3d90 con 0x7f69c4105410 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41a3f00 con 0x7f69c4100340 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.093+0000 7f69cc6f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69c41a40c0 con 0x7f69c4199140 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 0x7f69c4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 0x7f69c4198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57244/0 (socket says 172.21.15.3:57244) 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1733657589 learned_addr learned my addr 172.21.15.3:0/1733657589 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69cac91640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4199140 0x7f69c41a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69ca490640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c4100340 0x7f69c4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1733657589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4199140 msgr2=0x7f69c41a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4199140 0x7f69c41a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1733657589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c4100340 msgr2=0x7f69c4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c4100340 0x7f69c4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1733657589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b8009560 con 0x7f69c4105410 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 0x7f69c4198bb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009230 tx=0x7f69b8004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c37fe640 1 -- 172.21.15.3:0/1733657589 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80043d0 con 0x7f69c4105410 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c37fe640 1 -- 172.21.15.3:0/1733657589 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b80052d0 con 0x7f69c4105410 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69c41a4340 con 0x7f69c4105410 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69c37fe640 1 -- 172.21.15.3:0/1733657589 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b8003d80 con 0x7f69c4105410 2024-02-09T22:52:10.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.094+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69c41a4850 con 0x7f69c4105410 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69c37fe640 1 -- 172.21.15.3:0/1733657589 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f69b8005440 con 0x7f69c4105410 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 Processor -- start 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 -- start start 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de64e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de653e0 0x55e39de6d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de6df60 0x55e39de703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39de20020 con 0x55e39de6df60 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39ddd3320 con 0x55e39de64a30 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cf39b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39ddd0850 con 0x55e39de653e0 2024-02-09T22:52:10.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de6df60 0x55e39de703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cac91640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de64e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cac91640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de64e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43400/0 (socket says 172.21.15.3:43400) 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69cac91640 1 -- 172.21.15.3:0/2254432118 learned_addr learned my addr 172.21.15.3:0/2254432118 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de6df60 0x55e39de703b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57258/0 (socket says 172.21.15.3:57258) 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69ca490640 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de653e0 0x55e39de6d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.097+0000 7f69c37fe640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69ac0a2350 0x7f69ac0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 -- 172.21.15.3:0/2254432118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de653e0 msgr2=0x55e39de6d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de653e0 0x55e39de6d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 -- 172.21.15.3:0/2254432118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de6df60 msgr2=0x55e39de703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de6df60 0x55e39de703b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 -- 172.21.15.3:0/2254432118 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39dd8e210 con 0x55e39de64a30 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69c37fe640 1 -- 172.21.15.3:0/1733657589 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f69b80f0390 con 0x7f69c4105410 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69ac0a2350 0x7f69ac0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69cac91640 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de64e10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009d90 tx=0x7f69b4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48260092cd487273 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69c0ff9640 1 -- 172.21.15.3:0/2254432118 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b401b420 con 0x55e39de64a30 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69c0ff9640 1 -- 172.21.15.3:0/2254432118 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b400f8a0 con 0x55e39de64a30 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.098+0000 7f69c0ff9640 1 -- 172.21.15.3:0/2254432118 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b400e750 con 0x55e39de64a30 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 -- 172.21.15.3:0/2254432118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 msgr2=0x55e39de64e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cac91640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69ac0a2350 0x7f69ac0a4810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69b4023000 tx=0x7f69b4024000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de64e10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f69b4009d90 tx=0x7f69b4011810 comp rx=0 tx=0).stop 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 -- 172.21.15.3:0/2254432118 shutdown_connections 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de6df60 0x55e39de703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de653e0 0x55e39de6d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 --2- 172.21.15.3:0/2254432118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de64e10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 -- 172.21.15.3:0/2254432118 >> 172.21.15.3:0/2254432118 conn(0x55e39de5f100 msgr2=0x55e39de5f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 -- 172.21.15.3:0/2254432118 shutdown_connections 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 -- 172.21.15.3:0/2254432118 wait complete. 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 -- start start 2024-02-09T22:52:10.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.099+0000 7f69cf39b740 1 Processor -- start 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de64a30 0x55e39de6af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de6df60 0x55e39de6b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39ddcc030 con 0x55e39de653e0 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39da9f430 con 0x55e39de6df60 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39da5be00 con 0x55e39de64a30 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 10 client.0 _ll_register_callbacks cb 0x55e39ddc1030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57274/0 (socket says 172.21.15.3:57274) 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1238158811 learned_addr learned my addr 172.21.15.3:0/1238158811 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69ca490640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de64a30 0x55e39de6af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cac91640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de6df60 0x55e39de6b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de64a30 msgr2=0x55e39de6af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de64a30 0x55e39de6af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1238158811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de6df60 msgr2=0x55e39de6b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de6df60 0x55e39de6b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1238158811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b4009910 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cac91640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de6df60 0x55e39de6b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b490 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009900 tx=0x7f69b8005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abf9807a62f1bf0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1238158811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80bc7b0 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1238158811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f69b80bd370 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e39dacb7e0 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1238158811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69b80bc5d0 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 10 client.15852 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.100+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e39ddca6b0 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1238158811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b80d9020 con 0x55e39de653e0 2024-02-09T22:52:10.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1238158811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f69b803c070 con 0x55e39de653e0 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69cf39b740 10 client.15852 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69cf39b740 10 client.15852 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69cf39b740 20 client.15852 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39de7e2d0 0x55e39de806c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69cf39b740 4 client.15852 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e39db08720 con 0x55e39de7e2d0 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69ca490640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39de7e2d0 0x55e39de806c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69a9ffb640 10 client.15852 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.101+0000 7f69ca490640 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39de7e2d0 0x55e39de806c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f9f911bab615620 server_cookie=e6cc01ea9bf74c29 in_seq=0 out_seq=0 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1238158811 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e39db08720 con 0x55e39de7e2d0 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69a9ffb640 10 client.15852 handle_command_reply: tid=0 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 client.15852 shutdown 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 msgr2=0x55e39de6b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b490 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009900 tx=0x7f69b8005c90 comp rx=0 tx=0).stop 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 shutdown_connections 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39de7e2d0 0x55e39de806c0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de6df60 0x55e39de6b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b490 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1238158811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de64a30 0x55e39de6af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 >> 172.21.15.3:0/1238158811 conn(0x55e39de5f100 msgr2=0x55e39de60e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69a9ffb640 0 client.15852 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 shutdown_connections 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 1 -- 172.21.15.3:0/1238158811 wait complete. 2024-02-09T22:52:10.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.102+0000 7f69cf39b740 20 client.15852 trim_cache size 0 max 16384 2024-02-09T22:52:10.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 Processor -- start 2024-02-09T22:52:10.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 -- start start 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de6d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de681b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39dd51130 con 0x55e39de653e0 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39de858b0 con 0x55e39de64a30 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69cf39b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39de64530 con 0x55e39de6df60 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57290/0 (socket says 172.21.15.3:57290) 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1676416614 learned_addr learned my addr 172.21.15.3:0/1676416614 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69ca490640 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de6d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1676416614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 msgr2=0x55e39de681b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de681b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1676416614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 msgr2=0x55e39de6d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de6d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.114+0000 7f69c9c8f640 1 -- 172.21.15.3:0/1676416614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e39dd66c90 con 0x55e39de653e0 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b7f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f69b8005ec0 tx=0x7f69b80bcb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7bdc864ba6b6c74 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69abfff640 1 -- 172.21.15.3:0/1676416614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80f3b50 con 0x55e39de653e0 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69abfff640 1 -- 172.21.15.3:0/1676416614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e39dacb7e0 con 0x55e39de653e0 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69abfff640 1 -- 172.21.15.3:0/1676416614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b80bcc70 con 0x55e39de653e0 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 -- 172.21.15.3:0/1676416614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 msgr2=0x55e39de6b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b7f0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f69b8005ec0 tx=0x7f69b80bcb00 comp rx=0 tx=0).stop 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 -- 172.21.15.3:0/1676416614 shutdown_connections 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de681b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de653e0 0x55e39de6b7f0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1676416614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de64a30 0x55e39de6d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 -- 172.21.15.3:0/1676416614 >> 172.21.15.3:0/1676416614 conn(0x55e39de5f100 msgr2=0x55e39de66210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 -- 172.21.15.3:0/1676416614 shutdown_connections 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.115+0000 7f69cf39b740 1 -- 172.21.15.3:0/1676416614 wait complete. 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 -- start start 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 Processor -- start 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 0x55e39de75460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de653e0 0x55e39de73800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de73f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39de74610 con 0x55e39de64a30 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39ded47b0 con 0x55e39de653e0 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e39ded48f0 con 0x55e39de6df60 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cf39b740 10 client.0 _ll_register_callbacks cb 0x7f69b4009770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 0x55e39de75460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 0x55e39de75460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57300/0 (socket says 172.21.15.3:57300) 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 -- 172.21.15.3:0/1276337874 learned_addr learned my addr 172.21.15.3:0/1276337874 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de653e0 0x55e39de73800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cac91640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de73f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 -- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 msgr2=0x55e39de73f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de73f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 -- 172.21.15.3:0/1276337874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de653e0 msgr2=0x55e39de73800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de653e0 0x55e39de73800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 -- 172.21.15.3:0/1276337874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e39ddca6b0 con 0x55e39de64a30 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69cac91640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de73f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69ca490640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 0x55e39de75460 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f69bc000f80 tx=0x7f69bc010cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd131ff1b0e38536 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.116+0000 7f69a9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1276337874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69bc015030 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1276337874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f69bc004d20 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69b400f8a0 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1276337874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69bc01c9e0 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 10 client.15858 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55e39dd8ca70 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1276337874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b400f8a0 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1276337874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f69bc005030 con 0x55e39de64a30 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 10 client.15858 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 10 client.15858 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 20 client.15858 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39ded60c0 0x55e39ded84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 4 client.15858 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.117+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e39db57890 con 0x55e39ded60c0 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39ded60c0 0x55e39ded84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69a9ffb640 10 client.15858 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69c9c8f640 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39ded60c0 0x55e39ded84e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b7ad0c5da55c76c5 server_cookie=62e191e002db786c in_seq=0 out_seq=0 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69a9ffb640 1 -- 172.21.15.3:0/1276337874 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e39db57890 con 0x55e39ded60c0 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69a9ffb640 10 client.15858 handle_command_reply: tid=0 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 client.15858 shutdown 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 msgr2=0x55e39de75460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 0x55e39de75460 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f69bc000f80 tx=0x7f69bc010cc0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 shutdown_connections 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e39ded60c0 0x55e39ded84e0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e39de653e0 0x55e39de73800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e39de6df60 0x55e39de73f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 --2- 172.21.15.3:0/1276337874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e39de64a30 0x55e39de75460 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69a9ffb640 0 client.15858 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 >> 172.21.15.3:0/1276337874 conn(0x55e39de5f100 msgr2=0x55e39de5d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 shutdown_connections 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 1 -- 172.21.15.3:0/1276337874 wait complete. 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.118+0000 7f69cf39b740 20 client.15858 trim_cache size 0 max 16384 2024-02-09T22:52:10.130 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69ac0a2350 msgr2=0x7f69ac0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69ac0a2350 0x7f69ac0a4810 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f69b4023000 tx=0x7f69b4024000 comp rx=0 tx=0).stop 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 msgr2=0x7f69c4198bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 0x7f69c4198bb0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f69b8009230 tx=0x7f69b8004c00 comp rx=0 tx=0).stop 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 shutdown_connections 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69ac0a2350 0x7f69ac0a4810 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69c4199140 0x7f69c41a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69c4105410 0x7f69c4198bb0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 --2- 172.21.15.3:0/1733657589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f69c4100340 0x7f69c4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 >> 172.21.15.3:0/1733657589 conn(0x7f69c40772e0 msgr2=0x7f69c41065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 shutdown_connections 2024-02-09T22:52:10.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.119+0000 7f69cc6f4640 1 -- 172.21.15.3:0/1733657589 wait complete. 2024-02-09T22:52:10.134 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:10.134 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 -- 172.21.15.3:0/3388388188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 msgr2=0x7f36a0100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 --2- 172.21.15.3:0/3388388188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a0100710 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f369c009930 tx=0x7f369c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 -- 172.21.15.3:0/3388388188 shutdown_connections 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 --2- 172.21.15.3:0/3388388188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0100ce0 0x7f36a0104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 --2- 172.21.15.3:0/3388388188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a0105400 0x7f36a010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 --2- 172.21.15.3:0/3388388188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a0100710 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 -- 172.21.15.3:0/3388388188 >> 172.21.15.3:0/3388388188 conn(0x7f36a0077250 msgr2=0x7f36a0077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 -- 172.21.15.3:0/3388388188 shutdown_connections 2024-02-09T22:52:10.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.236+0000 7f36a80c3640 1 -- 172.21.15.3:0/3388388188 wait complete. 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 Processor -- start 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 -- start start 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a0105400 0x7f36a019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0197660 0x7f36a0197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a0198130 con 0x7f36a0100330 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a01982a0 con 0x7f36a0197660 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a80c3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36a0198460 con 0x7f36a0105400 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57322/0 (socket says 172.21.15.3:57322) 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 -- 172.21.15.3:0/699269363 learned_addr learned my addr 172.21.15.3:0/699269363 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a6660640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0197660 0x7f36a0197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a565e640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a0105400 0x7f36a019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 -- 172.21.15.3:0/699269363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a0105400 msgr2=0x7f36a019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a0105400 0x7f36a019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 -- 172.21.15.3:0/699269363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0197660 msgr2=0x7f36a0197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0197660 0x7f36a0197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a5e5f640 1 -- 172.21.15.3:0/699269363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c009560 con 0x7f36a0100330 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.237+0000 7f36a6660640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0197660 0x7f36a0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.238+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a019d5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f369c009230 tx=0x7f369c02c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.238+0000 7f3696ffd640 1 -- 172.21.15.3:0/699269363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c004350 con 0x7f36a0100330 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.238+0000 7f3696ffd640 1 -- 172.21.15.3:0/699269363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f369c0044b0 con 0x7f36a0100330 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.238+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36a01a43a0 con 0x7f36a0100330 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.238+0000 7f3696ffd640 1 -- 172.21.15.3:0/699269363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c003d70 con 0x7f36a0100330 2024-02-09T22:52:10.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.238+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36a01a48d0 con 0x7f36a0100330 2024-02-09T22:52:10.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.240+0000 7f3696ffd640 1 -- 172.21.15.3:0/699269363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f369c004df0 con 0x7f36a0100330 2024-02-09T22:52:10.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 Processor -- start 2024-02-09T22:52:10.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 -- start start 2024-02-09T22:52:10.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f3598a30 0x5607f3598e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35993e0 0x5607f35a1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f3554020 con 0x5607f35993e0 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f3507320 con 0x5607f35a1f60 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36aad74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f3504850 con 0x5607f3598a30 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f3696ffd640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f368c0a23d0 0x7f368c0a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35993e0 0x5607f35a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35993e0 0x5607f35a1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57334/0 (socket says 172.21.15.3:57334) 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36a565e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36a5e5f640 1 -- 172.21.15.3:0/3274250418 learned_addr learned my addr 172.21.15.3:0/3274250418 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36a6660640 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f3598a30 0x5607f3598e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.241+0000 7f36a6660640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f368c0a23d0 0x7f368c0a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f3696ffd640 1 -- 172.21.15.3:0/699269363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f369c0f0250 con 0x7f36a0100330 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a565e640 1 -- 172.21.15.3:0/3274250418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f3598a30 msgr2=0x5607f3598e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a565e640 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f3598a30 0x5607f3598e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a565e640 1 -- 172.21.15.3:0/3274250418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35993e0 msgr2=0x5607f35a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a565e640 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35993e0 0x5607f35a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a565e640 1 -- 172.21.15.3:0/3274250418 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607f34c2210 con 0x5607f35a1f60 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a565e640 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36a00695d0 tx=0x7f3690011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fbc18030ca92e21 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f36a6660640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f368c0a23d0 0x7f368c0a4890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3698009fd0 tx=0x7f3698009600 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f368bfff640 1 -- 172.21.15.3:0/3274250418 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690004050 con 0x5607f35a1f60 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f368bfff640 1 -- 172.21.15.3:0/3274250418 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3690005500 con 0x5607f35a1f60 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.242+0000 7f368bfff640 1 -- 172.21.15.3:0/3274250418 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3690004d00 con 0x5607f35a1f60 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 -- 172.21.15.3:0/3274250418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 msgr2=0x5607f35a43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a43b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f36a00695d0 tx=0x7f3690011810 comp rx=0 tx=0).stop 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 -- 172.21.15.3:0/3274250418 shutdown_connections 2024-02-09T22:52:10.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a43b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35993e0 0x5607f35a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 --2- 172.21.15.3:0/3274250418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f3598a30 0x5607f3598e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 -- 172.21.15.3:0/3274250418 >> 172.21.15.3:0/3274250418 conn(0x5607f3593100 msgr2=0x5607f35934e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 -- 172.21.15.3:0/3274250418 shutdown_connections 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 -- 172.21.15.3:0/3274250418 wait complete. 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.243+0000 7f36aad74740 1 -- start start 2024-02-09T22:52:10.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 Processor -- start 2024-02-09T22:52:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f359ef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35993e0 0x5607f359f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35a1f60 0x5607f359f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f3500030 con 0x5607f3598a30 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f31d3430 con 0x5607f35993e0 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f318fe00 con 0x5607f35a1f60 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f359ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36aad74740 10 client.0 _ll_register_callbacks cb 0x5607f34f5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f359ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57348/0 (socket says 172.21.15.3:57348) 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36a5e5f640 1 -- 172.21.15.3:0/1087658148 learned_addr learned my addr 172.21.15.3:0/1087658148 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36a565e640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35993e0 0x5607f359f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.244+0000 7f36a6660640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35a1f60 0x5607f359f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a5e5f640 1 -- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35a1f60 msgr2=0x5607f359f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35a1f60 0x5607f359f9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a5e5f640 1 -- 172.21.15.3:0/1087658148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35993e0 msgr2=0x5607f359f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35993e0 0x5607f359f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a5e5f640 1 -- 172.21.15.3:0/1087658148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3690009850 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a6660640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35a1f60 0x5607f359f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a565e640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35993e0 0x5607f359f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f359ef40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f369c0b94e0 tx=0x7f369c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=864dfd20bb172f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36897fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36897fa640 1 -- 172.21.15.3:0/1087658148 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c0bcb10 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36897fa640 1 -- 172.21.15.3:0/1087658148 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f369c0bde40 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607f31ff7e0 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36897fa640 1 -- 172.21.15.3:0/1087658148 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f369c0c0c60 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36aad74740 10 client.15870 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5607f34fe6b0 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.245+0000 7f36897fa640 1 -- 172.21.15.3:0/1087658148 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5607f31ff7e0 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36897fa640 1 -- 172.21.15.3:0/1087658148 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f369c03c070 con 0x5607f3598a30 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36aad74740 10 client.15870 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36aad74740 10 client.15870 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36aad74740 20 client.15870 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36aad74740 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f35b2240 0x5607f35b4650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36aad74740 4 client.15870 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607f323c720 con 0x5607f35b2240 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36a565e640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f35b2240 0x5607f35b4650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36a565e640 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f35b2240 0x5607f35b4650 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ed73c2505fe92279 server_cookie=57c182f963f6e93c in_seq=0 out_seq=0 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.246+0000 7f36897fa640 10 client.15870 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36897fa640 1 -- 172.21.15.3:0/1087658148 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5607f323c720 con 0x5607f35b2240 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36897fa640 10 client.15870 handle_command_reply: tid=0 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 client.15870 shutdown 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 msgr2=0x5607f359ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f359ef40 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f369c0b94e0 tx=0x7f369c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 shutdown_connections 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f35b2240 0x5607f35b4650 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35993e0 0x5607f359f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35a1f60 0x5607f359f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36897fa640 0 client.15870 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 --2- 172.21.15.3:0/1087658148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f359ef40 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 >> 172.21.15.3:0/1087658148 conn(0x5607f3593100 msgr2=0x5607f3594e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 shutdown_connections 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 1 -- 172.21.15.3:0/1087658148 wait complete. 2024-02-09T22:52:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.247+0000 7f36aad74740 20 client.15870 trim_cache size 0 max 16384 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 Processor -- start 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 -- start start 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f3598a30 0x5607f35a1670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f359f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 0x5607f359f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f3485130 con 0x5607f35a1f60 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f359ff10 con 0x5607f3598a30 2024-02-09T22:52:10.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36aad74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f358e7f0 con 0x5607f35993e0 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36a6660640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 0x5607f359f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.260+0000 7f36a6660640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 0x5607f359f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57358/0 (socket says 172.21.15.3:57358) 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 -- 172.21.15.3:0/3495519209 learned_addr learned my addr 172.21.15.3:0/3495519209 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f3598a30 0x5607f35a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a565e640 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f359f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 -- 172.21.15.3:0/3495519209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 msgr2=0x5607f359f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f359f490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 -- 172.21.15.3:0/3495519209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f3598a30 msgr2=0x5607f35a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f3598a30 0x5607f35a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 -- 172.21.15.3:0/3495519209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607f349ac90 con 0x5607f35a1f60 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f36a6660640 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 0x5607f359f9d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3698012e30 tx=0x7f369800c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a5471bfb29f32b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f368b7fe640 1 -- 172.21.15.3:0/3495519209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369800ae50 con 0x5607f35a1f60 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f368b7fe640 1 -- 172.21.15.3:0/3495519209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3698005020 con 0x5607f35a1f60 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.261+0000 7f368b7fe640 1 -- 172.21.15.3:0/3495519209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36980296b0 con 0x5607f35a1f60 2024-02-09T22:52:10.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 -- 172.21.15.3:0/3495519209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 msgr2=0x5607f359f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 0x5607f359f9d0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f3698012e30 tx=0x7f369800c000 comp rx=0 tx=0).stop 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 -- 172.21.15.3:0/3495519209 shutdown_connections 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f35a1f60 0x5607f359f9d0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f359f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 --2- 172.21.15.3:0/3495519209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f3598a30 0x5607f35a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 -- 172.21.15.3:0/3495519209 >> 172.21.15.3:0/3495519209 conn(0x5607f3593100 msgr2=0x5607f35b8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 -- 172.21.15.3:0/3495519209 shutdown_connections 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 -- 172.21.15.3:0/3495519209 wait complete. 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 -- start start 2024-02-09T22:52:10.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.262+0000 7f36aad74740 1 Processor -- start 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f35a9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f35a75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f35a8200 con 0x5607f3598a30 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f35a8370 con 0x5607f35a1f60 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607f3608750 con 0x5607f35993e0 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 10 client.0 _ll_register_callbacks cb 0x7f36900097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f35a9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f35a9250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57370/0 (socket says 172.21.15.3:57370) 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 -- 172.21.15.3:0/4007495337 learned_addr learned my addr 172.21.15.3:0/4007495337 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a565e640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f35a75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a6660640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 -- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 msgr2=0x5607f35a75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f35a75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 -- 172.21.15.3:0/4007495337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 msgr2=0x5607f35a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 -- 172.21.15.3:0/4007495337 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369801a040 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a6660640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36a5e5f640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f35a9250 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f369c0ed8e0 tx=0x7f369c0b8e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69017290180c8194 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36897fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36897fa640 1 -- 172.21.15.3:0/4007495337 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369c0bc230 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36897fa640 1 -- 172.21.15.3:0/4007495337 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f369c0bde40 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.263+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3690005500 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36897fa640 1 -- 172.21.15.3:0/4007495337 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f369c0bc470 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 10 client.15876 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x5607f34c0a70 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36897fa640 1 -- 172.21.15.3:0/4007495337 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3690005500 con 0x5607f3598a30 2024-02-09T22:52:10.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36897fa640 1 -- 172.21.15.3:0/4007495337 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f369c03c040 con 0x5607f3598a30 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 10 client.15876 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 10 client.15876 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 20 client.15876 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f36098f0 0x5607f360bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 4 client.15876 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5607f328b890 con 0x5607f36098f0 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36a565e640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f36098f0 0x5607f360bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36a565e640 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f36098f0 0x5607f360bce0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7c92ae6a91a8f3f5 server_cookie=c08b218eda69439 in_seq=0 out_seq=0 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.264+0000 7f36897fa640 10 client.15876 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36897fa640 1 -- 172.21.15.3:0/4007495337 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5607f328b890 con 0x5607f36098f0 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36897fa640 10 client.15876 handle_command_reply: tid=0 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 client.15876 shutdown 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 msgr2=0x5607f35a9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f35a9250 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f369c0ed8e0 tx=0x7f369c0b8e30 comp rx=0 tx=0).stop 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 shutdown_connections 2024-02-09T22:52:10.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5607f36098f0 0x5607f360bce0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5607f35a1f60 0x5607f35a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5607f35993e0 0x5607f35a75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36897fa640 0 client.15876 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 --2- 172.21.15.3:0/4007495337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5607f3598a30 0x5607f35a9250 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 >> 172.21.15.3:0/4007495337 conn(0x5607f3593100 msgr2=0x5607f35919b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 shutdown_connections 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 1 -- 172.21.15.3:0/4007495337 wait complete. 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36aad74740 20 client.15876 trim_cache size 0 max 16384 2024-02-09T22:52:10.276 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f368c0a23d0 msgr2=0x7f368c0a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36a80c3640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f368c0a23d0 0x7f368c0a4890 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3698009fd0 tx=0x7f3698009600 comp rx=0 tx=0).stop 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 msgr2=0x7f36a019d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36a80c3640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a019d5a0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f369c009230 tx=0x7f369c02c6c0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.265+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 shutdown_connections 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f368c0a23d0 0x7f368c0a4890 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f36a0197660 0x7f36a0197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36a0105400 0x7f36a019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 --2- 172.21.15.3:0/699269363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f36a0100330 0x7f36a019d5a0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 >> 172.21.15.3:0/699269363 conn(0x7f36a0077250 msgr2=0x7f36a00fe350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 shutdown_connections 2024-02-09T22:52:10.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.266+0000 7f36a80c3640 1 -- 172.21.15.3:0/699269363 wait complete. 2024-02-09T22:52:10.280 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:10.281 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:52:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/613230427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 msgr2=0x7fe79c10dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/613230427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c10dc50 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe794009ca0 tx=0x7fe79402c180 comp rx=0 tx=0).stop 2024-02-09T22:52:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/613230427 shutdown_connections 2024-02-09T22:52:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/613230427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe79c102d70 0x7fe79c10b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/613230427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c10dc50 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/613230427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 0x7fe79c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/613230427 >> 172.21.15.3:0/613230427 conn(0x7fe79c077220 msgr2=0x7fe79c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/613230427 shutdown_connections 2024-02-09T22:52:10.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/613230427 wait complete. 2024-02-09T22:52:10.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 Processor -- start 2024-02-09T22:52:10.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.385+0000 7fe7a1f7a640 1 -- start start 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 0x7fe79c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe79c102d70 0x7fe79c19fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c199f60 con 0x7fe79c10b820 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c19a0d0 con 0x7fe79c102d70 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79c19a290 con 0x7fe79c102450 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c199910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57376/0 (socket says 172.21.15.3:57376) 2024-02-09T22:52:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 -- 172.21.15.3:0/3358451447 learned_addr learned my addr 172.21.15.3:0/3358451447 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79affd640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe79c102d70 0x7fe79c19fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 0x7fe79c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 -- 172.21.15.3:0/3358451447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 msgr2=0x7fe79c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 0x7fe79c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 -- 172.21.15.3:0/3358451447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe79c102d70 msgr2=0x7fe79c19fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe79c102d70 0x7fe79c19fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 -- 172.21.15.3:0/3358451447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe794009900 con 0x7fe79c10b820 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 0x7fe79c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe79bfff640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c199910 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe79402c790 tx=0x7fe7940346c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe798ff9640 1 -- 172.21.15.3:0/3358451447 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794003d80 con 0x7fe79c10b820 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe798ff9640 1 -- 172.21.15.3:0/3358451447 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7940348b0 con 0x7fe79c10b820 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe79c19a510 con 0x7fe79c10b820 2024-02-09T22:52:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.386+0000 7fe798ff9640 1 -- 172.21.15.3:0/3358451447 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe794004790 con 0x7fe79c10b820 2024-02-09T22:52:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.387+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79c1a6800 con 0x7fe79c10b820 2024-02-09T22:52:10.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.388+0000 7fe798ff9640 1 -- 172.21.15.3:0/3358451447 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7fe79400a040 con 0x7fe79c10b820 2024-02-09T22:52:10.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 Processor -- start 2024-02-09T22:52:10.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 -- start start 2024-02-09T22:52:10.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe798ff9640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7880a2350 0x7fe7880a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0909e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f0912990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0912f60 0x55f8f09153b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe798ff9640 1 -- 172.21.15.3:0/3358451447 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe7940f0a80 con 0x7fe79c10b820 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f08c5020 con 0x55f8f090a3e0 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0878320 con 0x55f8f0912f60 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0875850 con 0x55f8f0909a30 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe79affd640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7880a2350 0x7fe7880a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe79affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f0912990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe79affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f0912990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57384/0 (socket says 172.21.15.3:57384) 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe79affd640 1 -- 172.21.15.3:0/4171719987 learned_addr learned my addr 172.21.15.3:0/4171719987 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe79b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0909e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.389+0000 7fe79bfff640 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0912f60 0x55f8f09153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79b7fe640 1 -- 172.21.15.3:0/4171719987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0912f60 msgr2=0x55f8f09153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0912f60 0x55f8f09153b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79b7fe640 1 -- 172.21.15.3:0/4171719987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 msgr2=0x55f8f0912990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f0912990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79b7fe640 1 -- 172.21.15.3:0/4171719987 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8f0833210 con 0x55f8f0909a30 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79affd640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7880a2350 0x7fe7880a4810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe78c011930 tx=0x7fe78c009b60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0909e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe784009c00 tx=0x7fe7840117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=306ea56f9ea495b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe791ffb640 1 -- 172.21.15.3:0/4171719987 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784005020 con 0x55f8f0909a30 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe791ffb640 1 -- 172.21.15.3:0/4171719987 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe784005660 con 0x55f8f0909a30 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.390+0000 7fe791ffb640 1 -- 172.21.15.3:0/4171719987 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7840126b0 con 0x55f8f0909a30 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/4171719987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 msgr2=0x55f8f0909e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0909e10 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe784009c00 tx=0x7fe7840117b0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/4171719987 shutdown_connections 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0912f60 0x55f8f09153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f0912990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/4171719987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0909e10 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/4171719987 >> 172.21.15.3:0/4171719987 conn(0x55f8f0904100 msgr2=0x55f8f09044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/4171719987 shutdown_connections 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/4171719987 wait complete. 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 -- start start 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.391+0000 7fe7a4c2f740 1 Processor -- start 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f090fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0912f60 0x55f8f0910900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0871030 con 0x55f8f090a3e0 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0544430 con 0x55f8f0909a30 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0500e00 con 0x55f8f0912f60 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 10 client.0 _ll_register_callbacks cb 0x55f8f0866030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f090fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09103c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57392/0 (socket says 172.21.15.3:57392) 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0912f60 0x55f8f0910900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 -- 172.21.15.3:0/3735788182 learned_addr learned my addr 172.21.15.3:0/3735788182 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 -- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0912f60 msgr2=0x55f8f0910900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0912f60 0x55f8f0910900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 -- 172.21.15.3:0/3735788182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 msgr2=0x55f8f090fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f090fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 -- 172.21.15.3:0/3735788182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe784009860 con 0x55f8f090a3e0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe79affd640 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09103c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009d20 tx=0x7fe78c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796f1ce675ea12ef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe77affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe77affd640 1 -- 172.21.15.3:0/3735788182 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe78c016040 con 0x55f8f090a3e0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe77affd640 1 -- 172.21.15.3:0/3735788182 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe78c00fe10 con 0x55f8f090a3e0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.392+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f8f05707e0 con 0x55f8f090a3e0 2024-02-09T22:52:10.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe77affd640 1 -- 172.21.15.3:0/3735788182 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe78c010b40 con 0x55f8f090a3e0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 10 client.15888 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f8f086f6b0 con 0x55f8f090a3e0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe77affd640 1 -- 172.21.15.3:0/3735788182 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f8f05707e0 con 0x55f8f090a3e0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe77affd640 1 -- 172.21.15.3:0/3735788182 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe78c026030 con 0x55f8f090a3e0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 10 client.15888 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 10 client.15888 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 20 client.15888 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f09234e0 0x55f8f0925920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 4 client.15888 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f8f05ad720 con 0x55f8f09234e0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f09234e0 0x55f8f0925920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe77affd640 10 client.15888 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.393+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f09234e0 0x55f8f0925920 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4890d90a18f02bec server_cookie=1bf50c2ffa127434 in_seq=0 out_seq=0 2024-02-09T22:52:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe77affd640 1 -- 172.21.15.3:0/3735788182 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f8f05ad720 con 0x55f8f09234e0 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe77affd640 10 client.15888 handle_command_reply: tid=0 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 client.15888 shutdown 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 msgr2=0x55f8f09103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09103c0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe78c009d20 tx=0x7fe78c01d000 comp rx=0 tx=0).stop 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 shutdown_connections 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f09234e0 0x55f8f0925920 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0912f60 0x55f8f0910900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09103c0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe77affd640 0 client.15888 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3735788182 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f090fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 >> 172.21.15.3:0/3735788182 conn(0x55f8f0904100 msgr2=0x55f8f0905e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 shutdown_connections 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3735788182 wait complete. 2024-02-09T22:52:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.394+0000 7fe7a4c2f740 20 client.15888 trim_cache size 0 max 16384 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.406+0000 7fe7a4c2f740 1 Processor -- start 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.406+0000 7fe7a4c2f740 1 -- start start 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 0x55f8f09125b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0910720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f090a3e0 0x55f8f090d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f07f6130 con 0x55f8f0904100 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0904f30 con 0x55f8f090a3e0 2024-02-09T22:52:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0908ac0 con 0x55f8f0909a30 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 0x55f8f09125b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 0x55f8f09125b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57400/0 (socket says 172.21.15.3:57400) 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 -- 172.21.15.3:0/3534095514 learned_addr learned my addr 172.21.15.3:0/3534095514 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79affd640 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0910720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79bfff640 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f090a3e0 0x55f8f090d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 -- 172.21.15.3:0/3534095514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 msgr2=0x55f8f0910720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0910720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 -- 172.21.15.3:0/3534095514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f090a3e0 msgr2=0x55f8f090d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f090a3e0 0x55f8f090d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 -- 172.21.15.3:0/3534095514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f8f080bc90 con 0x55f8f0904100 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 0x55f8f09125b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe784005fd0 tx=0x7fe7840117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295bacb5de072e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7917fa640 1 -- 172.21.15.3:0/3534095514 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe784009cf0 con 0x55f8f0904100 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7917fa640 1 -- 172.21.15.3:0/3534095514 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7840052f0 con 0x55f8f0904100 2024-02-09T22:52:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.407+0000 7fe7917fa640 1 -- 172.21.15.3:0/3534095514 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7840294e0 con 0x55f8f0904100 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3534095514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 msgr2=0x55f8f09125b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 0x55f8f09125b0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fe784005fd0 tx=0x7fe7840117b0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3534095514 shutdown_connections 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f090a3e0 0x55f8f090d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0909a30 0x55f8f0910720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/3534095514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f0904100 0x55f8f09125b0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3534095514 >> 172.21.15.3:0/3534095514 conn(0x55f8f090ab30 msgr2=0x55f8f092a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3534095514 shutdown_connections 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/3534095514 wait complete. 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.408+0000 7fe7a4c2f740 1 -- start start 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 Processor -- start 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0904100 0x55f8f0918c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f0919160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09196c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0979bf0 con 0x55f8f090a3e0 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0979d60 con 0x55f8f0909a30 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f8f0979f20 con 0x55f8f0904100 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe7a4c2f740 10 client.0 _ll_register_callbacks cb 0x7fe7840095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09196c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57412/0 (socket says 172.21.15.3:57412) 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 -- 172.21.15.3:0/895173928 learned_addr learned my addr 172.21.15.3:0/895173928 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79affd640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f0919160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0904100 0x55f8f0918c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 -- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0904100 msgr2=0x55f8f0918c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0904100 0x55f8f0918c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 -- 172.21.15.3:0/895173928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 msgr2=0x55f8f0919160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f0919160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 -- 172.21.15.3:0/895173928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78401e040 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe79bfff640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09196c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe7940bd2f0 tx=0x7fe7940bcd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eff9959e15c1586 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.409+0000 7fe77affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe77affd640 1 -- 172.21.15.3:0/895173928 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7940bd7c0 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe77affd640 1 -- 172.21.15.3:0/895173928 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe7940bd360 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe784005660 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe77affd640 1 -- 172.21.15.3:0/895173928 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7940d8020 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 10 client.15894 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55f8f0831a70 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe77affd640 1 -- 172.21.15.3:0/895173928 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe784005660 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe77affd640 1 -- 172.21.15.3:0/895173928 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe7940ee360 con 0x55f8f090a3e0 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 10 client.15894 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 10 client.15894 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 20 client.15894 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f0912f60 0x55f8f097d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 4 client.15894 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f8f05fc890 con 0x55f8f0912f60 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.410+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f0912f60 0x55f8f097d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe79b7fe640 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f0912f60 0x55f8f097d510 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=17d9ef41ea6e4932 server_cookie=52b35256aaf96bee in_seq=0 out_seq=0 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe77affd640 10 client.15894 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe77affd640 1 -- 172.21.15.3:0/895173928 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f8f05fc890 con 0x55f8f0912f60 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe77affd640 10 client.15894 handle_command_reply: tid=0 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 client.15894 shutdown 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 msgr2=0x55f8f09196c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09196c0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7fe7940bd2f0 tx=0x7fe7940bcd80 comp rx=0 tx=0).stop 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 shutdown_connections 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f8f0912f60 0x55f8f097d510 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f8f0909a30 0x55f8f0919160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f8f090a3e0 0x55f8f09196c0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 --2- 172.21.15.3:0/895173928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f8f0904100 0x55f8f0918c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe77affd640 0 client.15894 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 >> 172.21.15.3:0/895173928 conn(0x55f8f090ab30 msgr2=0x55f8f09029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 shutdown_connections 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 1 -- 172.21.15.3:0/895173928 wait complete. 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.411+0000 7fe7a4c2f740 20 client.15894 trim_cache size 0 max 16384 2024-02-09T22:52:10.422 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7880a2350 msgr2=0x7fe7880a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7880a2350 0x7fe7880a4810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe78c011930 tx=0x7fe78c009b60 comp rx=0 tx=0).stop 2024-02-09T22:52:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 msgr2=0x7fe79c199910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c199910 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe79402c790 tx=0x7fe7940346c0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 shutdown_connections 2024-02-09T22:52:10.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe7880a2350 0x7fe7880a4810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe79c102d70 0x7fe79c19fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe79c10b820 0x7fe79c199910 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 --2- 172.21.15.3:0/3358451447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe79c102450 0x7fe79c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 >> 172.21.15.3:0/3358451447 conn(0x7fe79c077220 msgr2=0x7fe79c107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 shutdown_connections 2024-02-09T22:52:10.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.412+0000 7fe7a1f7a640 1 -- 172.21.15.3:0/3358451447 wait complete. 2024-02-09T22:52:10.426 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:10.426 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:52:10.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:10 smithi003 ceph-mon[32678]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-02-09T22:52:10.450 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3753337686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:52:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:10 smithi089 ceph-mon[36665]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-02-09T22:52:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3753337686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 -- 172.21.15.3:0/3482352892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102450 msgr2=0x7ff498102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/3482352892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102450 0x7ff498102830 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7ff494009ca0 tx=0x7ff49402c180 comp rx=0 tx=0).stop 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 -- 172.21.15.3:0/3482352892 shutdown_connections 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/3482352892 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff49810b820 0x7ff49810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/3482352892 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff498102d70 0x7ff49810b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/3482352892 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102450 0x7ff498102830 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 -- 172.21.15.3:0/3482352892 >> 172.21.15.3:0/3482352892 conn(0x7ff4980772e0 msgr2=0x7ff4980776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 -- 172.21.15.3:0/3482352892 shutdown_connections 2024-02-09T22:52:10.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 -- 172.21.15.3:0/3482352892 wait complete. 2024-02-09T22:52:10.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 Processor -- start 2024-02-09T22:52:10.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.530+0000 7ff4a07e2640 1 -- start start 2024-02-09T22:52:10.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff498102450 0x7ff49819d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 0x7ff49819dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff49810b820 0x7ff498197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff498197df0 con 0x7ff498102d70 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff498197f60 con 0x7ff498102450 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff498198120 con 0x7ff49810b820 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 0x7ff49819dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 0x7ff49819dae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57432/0 (socket says 172.21.15.3:57432) 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 -- 172.21.15.3:0/506263741 learned_addr learned my addr 172.21.15.3:0/506263741 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49e57e640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff498102450 0x7ff49819d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff49810b820 0x7ff498197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 -- 172.21.15.3:0/506263741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff49810b820 msgr2=0x7ff498197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff49810b820 0x7ff498197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 -- 172.21.15.3:0/506263741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff498102450 msgr2=0x7ff49819d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff498102450 0x7ff49819d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 -- 172.21.15.3:0/506263741 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494009900 con 0x7ff498102d70 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 0x7ff49819dae0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e9f0 tx=0x7ff48800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff48f7fe640 1 -- 172.21.15.3:0/506263741 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488009700 con 0x7ff498102d70 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff48f7fe640 1 -- 172.21.15.3:0/506263741 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff48800cd80 con 0x7ff498102d70 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff48f7fe640 1 -- 172.21.15.3:0/506263741 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff488018920 con 0x7ff498102d70 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.531+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff498198400 con 0x7ff498102d70 2024-02-09T22:52:10.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.532+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4981a4520 con 0x7ff498102d70 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.534+0000 7ff48f7fe640 1 -- 172.21.15.3:0/506263741 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7ff488018a80 con 0x7ff498102d70 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.534+0000 7ff4a33b2740 1 Processor -- start 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.534+0000 7ff4a33b2740 1 -- start start 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff48f7fe640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4840a2350 0x7ff4840a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 0x55dd5188de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd51896990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518993b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd51849020 con 0x55dd5188da30 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd517fc320 con 0x55dd5188e3e0 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd517f9850 con 0x55dd51896f60 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4840a2350 0x7ff4840a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 0x55dd5188de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff48f7fe640 1 -- 172.21.15.3:0/506263741 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff488014070 con 0x7ff498102d70 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 0x55dd5188de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57444/0 (socket says 172.21.15.3:57444) 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 -- 172.21.15.3:0/4168749881 learned_addr learned my addr 172.21.15.3:0/4168749881 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd51896990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 -- 172.21.15.3:0/4168749881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 msgr2=0x55dd518993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 -- 172.21.15.3:0/4168749881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 msgr2=0x55dd51896990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd51896990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 -- 172.21.15.3:0/4168749881 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd517b7210 con 0x55dd5188da30 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd51896990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49e57e640 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 0x55dd5188de10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff49402c790 tx=0x7ff494005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fb79e91813cb6c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.535+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4840a2350 0x7ff4840a4810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff490009fc0 tx=0x7ff490015850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff48cff9640 1 -- 172.21.15.3:0/4168749881 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494034640 con 0x55dd5188da30 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff48cff9640 1 -- 172.21.15.3:0/4168749881 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff494003e50 con 0x55dd5188da30 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff48cff9640 1 -- 172.21.15.3:0/4168749881 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff494032df0 con 0x55dd5188da30 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 -- 172.21.15.3:0/4168749881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 msgr2=0x55dd5188de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 0x55dd5188de10 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7ff49402c790 tx=0x7ff494005f70 comp rx=0 tx=0).stop 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 -- 172.21.15.3:0/4168749881 shutdown_connections 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd51896990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/4168749881 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188da30 0x55dd5188de10 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.536+0000 7ff4a33b2740 1 -- 172.21.15.3:0/4168749881 >> 172.21.15.3:0/4168749881 conn(0x55dd51888100 msgr2=0x55dd518884e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 -- 172.21.15.3:0/4168749881 shutdown_connections 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 -- 172.21.15.3:0/4168749881 wait complete. 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 -- start start 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 Processor -- start 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd5188e3e0 0x55dd51893f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 0x55dd51894460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd518949a0 0x55dd518a2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd517f5030 con 0x55dd51896f60 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd514c8430 con 0x55dd518949a0 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd51484e00 con 0x55dd5188e3e0 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff4a33b2740 10 client.0 _ll_register_callbacks cb 0x55dd517ea030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff49dd7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 0x55dd51894460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff49dd7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 0x55dd51894460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57446/0 (socket says 172.21.15.3:57446) 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff49dd7d640 1 -- 172.21.15.3:0/3638426084 learned_addr learned my addr 172.21.15.3:0/3638426084 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.537+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd518949a0 0x55dd518a2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49e57e640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd5188e3e0 0x55dd51893f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49dd7d640 1 -- 172.21.15.3:0/3638426084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd5188e3e0 msgr2=0x55dd51893f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd5188e3e0 0x55dd51893f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49dd7d640 1 -- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd518949a0 msgr2=0x55dd518a2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd518949a0 0x55dd518a2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49dd7d640 1 -- 172.21.15.3:0/3638426084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49810e8d0 con 0x55dd51896f60 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 0x55dd51894460 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff48809aea0 tx=0x7ff48809b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d745a760df978c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff481ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff481ffb640 1 -- 172.21.15.3:0/3638426084 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880a36b0 con 0x55dd51896f60 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff481ffb640 1 -- 172.21.15.3:0/3638426084 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff48808a7d0 con 0x55dd51896f60 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd514f47e0 con 0x55dd51896f60 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff481ffb640 1 -- 172.21.15.3:0/3638426084 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4880a3220 con 0x55dd51896f60 2024-02-09T22:52:10.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff4a33b2740 10 client.15906 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.538+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55dd517f36b0 con 0x55dd51896f60 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff481ffb640 1 -- 172.21.15.3:0/3638426084 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55dd514f47e0 con 0x55dd51896f60 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff481ffb640 1 -- 172.21.15.3:0/3638426084 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff48809a360 con 0x55dd51896f60 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff4a33b2740 10 client.15906 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff4a33b2740 10 client.15906 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff4a33b2740 20 client.15906 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd518a7990 0x55dd518a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff4a33b2740 4 client.15906 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd51531720 con 0x55dd518a7990 2024-02-09T22:52:10.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.539+0000 7ff49e57e640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd518a7990 0x55dd518a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.540+0000 7ff49e57e640 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd518a7990 0x55dd518a9db0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7d80df55398b1832 server_cookie=c1cacd944bd5800d in_seq=0 out_seq=0 2024-02-09T22:52:10.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.540+0000 7ff481ffb640 10 client.15906 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.540+0000 7ff481ffb640 1 -- 172.21.15.3:0/3638426084 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dd51531720 con 0x55dd518a7990 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.540+0000 7ff481ffb640 10 client.15906 handle_command_reply: tid=0 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.540+0000 7ff4a33b2740 1 client.15906 shutdown 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 msgr2=0x55dd51894460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 0x55dd51894460 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7ff48809aea0 tx=0x7ff48809b6d0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 shutdown_connections 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd518a7990 0x55dd518a9db0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd518949a0 0x55dd518a2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd51896f60 0x55dd51894460 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/3638426084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd5188e3e0 0x55dd51893f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff481ffb640 0 client.15906 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 >> 172.21.15.3:0/3638426084 conn(0x55dd51888100 msgr2=0x55dd51889e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 shutdown_connections 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 1 -- 172.21.15.3:0/3638426084 wait complete. 2024-02-09T22:52:10.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.541+0000 7ff4a33b2740 20 client.15906 trim_cache size 0 max 16384 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.552+0000 7ff4a33b2740 1 Processor -- start 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.552+0000 7ff4a33b2740 1 -- start start 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd518911a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 0x55dd518916e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd5177a130 con 0x55dd518949a0 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd518a3270 con 0x55dd5188e3e0 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd518ae940 con 0x55dd51896f60 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 0x55dd518916e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 0x55dd518916e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57458/0 (socket says 172.21.15.3:57458) 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 -- 172.21.15.3:0/390885548 learned_addr learned my addr 172.21.15.3:0/390885548 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49e57e640 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd518911a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 -- 172.21.15.3:0/390885548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 msgr2=0x55dd518937b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518937b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 -- 172.21.15.3:0/390885548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 msgr2=0x55dd518911a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd518911a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 -- 172.21.15.3:0/390885548 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd5178fc90 con 0x55dd518949a0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 0x55dd518916e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff48809ac10 tx=0x7ff488000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=708ac227897b257 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff483fff640 1 -- 172.21.15.3:0/390885548 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48809a870 con 0x55dd518949a0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff483fff640 1 -- 172.21.15.3:0/390885548 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4880cbc80 con 0x55dd518949a0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.553+0000 7ff483fff640 1 -- 172.21.15.3:0/390885548 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4880cc0a0 con 0x55dd518949a0 2024-02-09T22:52:10.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 -- 172.21.15.3:0/390885548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 msgr2=0x55dd518916e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 0x55dd518916e0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7ff48809ac10 tx=0x7ff488000bc0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 -- 172.21.15.3:0/390885548 shutdown_connections 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd51896f60 0x55dd518937b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd518949a0 0x55dd518916e0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/390885548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd5188e3e0 0x55dd518911a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 -- 172.21.15.3:0/390885548 >> 172.21.15.3:0/390885548 conn(0x55dd51888100 msgr2=0x55dd518927f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 -- 172.21.15.3:0/390885548 shutdown_connections 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 -- 172.21.15.3:0/390885548 wait complete. 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 -- start start 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.554+0000 7ff4a33b2740 1 Processor -- start 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 0x55dd518f85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd518949a0 0x55dd518f8b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd51896f60 0x55dd5189d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd5189dae0 con 0x55dd5188e3e0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd5189dc50 con 0x55dd51896f60 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd5189de10 con 0x55dd518949a0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 10 client.0 _ll_register_callbacks cb 0x7ff490005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 0x55dd518f85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 0x55dd518f85d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57466/0 (socket says 172.21.15.3:57466) 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 -- 172.21.15.3:0/1941948411 learned_addr learned my addr 172.21.15.3:0/1941948411 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd518949a0 0x55dd518f8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49ed7f640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd51896f60 0x55dd5189d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 -- 172.21.15.3:0/1941948411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd518949a0 msgr2=0x55dd518f8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd518949a0 0x55dd518f8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 -- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd51896f60 msgr2=0x55dd5189d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd51896f60 0x55dd5189d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 -- 172.21.15.3:0/1941948411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd517f36b0 con 0x55dd5188e3e0 2024-02-09T22:52:10.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff49e57e640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 0x55dd518f85d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff494000f80 tx=0x7ff494003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3aeb3fd369e39d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff481ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff481ffb640 1 -- 172.21.15.3:0/1941948411 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49404f040 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff481ffb640 1 -- 172.21.15.3:0/1941948411 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff494003c90 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.555+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff494003e50 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff481ffb640 1 -- 172.21.15.3:0/1941948411 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff494032600 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 10 client.15912 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55dd517b5a70 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff481ffb640 1 -- 172.21.15.3:0/1941948411 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff494003e50 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff481ffb640 1 -- 172.21.15.3:0/1941948411 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff494032cd0 con 0x55dd5188e3e0 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 10 client.15912 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 10 client.15912 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 20 client.15912 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd51900090 0x55dd519024b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 4 client.15912 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dd51580890 con 0x55dd51900090 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.556+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd51900090 0x55dd519024b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff49dd7d640 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd51900090 0x55dd519024b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3fb88f6bdcbe7de1 server_cookie=f1d84280928ec4da in_seq=0 out_seq=0 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff481ffb640 10 client.15912 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff481ffb640 1 -- 172.21.15.3:0/1941948411 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dd51580890 con 0x55dd51900090 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff481ffb640 10 client.15912 handle_command_reply: tid=0 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 client.15912 shutdown 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 msgr2=0x55dd518f85d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 0x55dd518f85d0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff494000f80 tx=0x7ff494003ba0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 shutdown_connections 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55dd51900090 0x55dd519024b0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dd51896f60 0x55dd5189d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff481ffb640 0 client.15912 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:10.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dd518949a0 0x55dd518f8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 --2- 172.21.15.3:0/1941948411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dd5188e3e0 0x55dd518f85d0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 >> 172.21.15.3:0/1941948411 conn(0x55dd51888100 msgr2=0x55dd518869b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 shutdown_connections 2024-02-09T22:52:10.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 1 -- 172.21.15.3:0/1941948411 wait complete. 2024-02-09T22:52:10.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.557+0000 7ff4a33b2740 20 client.15912 trim_cache size 0 max 16384 2024-02-09T22:52:10.568 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4840a2350 msgr2=0x7ff4840a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4840a2350 0x7ff4840a4810 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff490009fc0 tx=0x7ff490015850 comp rx=0 tx=0).stop 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 msgr2=0x7ff49819dae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 0x7ff49819dae0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7ff48800e9f0 tx=0x7ff48800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 shutdown_connections 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4840a2350 0x7ff4840a4810 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff49810b820 0x7ff498197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff498102d70 0x7ff49819dae0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 --2- 172.21.15.3:0/506263741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff498102450 0x7ff49819d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 >> 172.21.15.3:0/506263741 conn(0x7ff4980772e0 msgr2=0x7ff4980fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 shutdown_connections 2024-02-09T22:52:10.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.558+0000 7ff4a07e2640 1 -- 172.21.15.3:0/506263741 wait complete. 2024-02-09T22:52:10.572 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:10.572 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:52:10.618 INFO:tasks.workunit.client.0.smithi003.stdout: 1 13037 21.33 MB/sec warmup 4 sec latency 10.528 ms 2024-02-09T22:52:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:10 smithi064 ceph-mon[36638]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 361 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 25 op/s 2024-02-09T22:52:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3753337686' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 -- 172.21.15.3:0/3743411381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc1020d0 msgr2=0x7fa8bc1024b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/3743411381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc1024b0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8009900 tx=0x7fa8b802bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 -- 172.21.15.3:0/3743411381 shutdown_connections 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/3743411381 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc0ffbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/3743411381 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc100300 0x7fa8bc10fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/3743411381 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc1024b0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 -- 172.21.15.3:0/3743411381 >> 172.21.15.3:0/3743411381 conn(0x7fa8bc076630 msgr2=0x7fa8bc0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 -- 172.21.15.3:0/3743411381 shutdown_connections 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.733+0000 7fa8c39b5640 1 -- 172.21.15.3:0/3743411381 wait complete. 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 Processor -- start 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 -- start start 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8bc199880 0x7fa8bc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc19a270 con 0x7fa8bc0ff770 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc19a3e0 con 0x7fa8bc199880 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c39b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8bc19a5a0 con 0x7fa8bc1020d0 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c0f50640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc19f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57486/0 (socket says 172.21.15.3:57486) 2024-02-09T22:52:10.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c0f50640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59028/0 (socket says 172.21.15.3:59028) 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c1751640 1 -- 172.21.15.3:0/1297545511 learned_addr learned my addr 172.21.15.3:0/1297545511 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8bc199880 0x7fa8bc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c0f50640 1 -- 172.21.15.3:0/1297545511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8bc199880 msgr2=0x7fa8bc199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8bc199880 0x7fa8bc199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c0f50640 1 -- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 msgr2=0x7fa8bc19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.734+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8c0f50640 1 -- 172.21.15.3:0/1297545511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8009560 con 0x7fa8bc1020d0 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8c1751640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc19fc10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00e9f0 tx=0x7fa8ac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8b27fc640 1 -- 172.21.15.3:0/1297545511 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac00ccc0 con 0x7fa8bc1020d0 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8bc1a6590 con 0x7fa8bc1020d0 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8b27fc640 1 -- 172.21.15.3:0/1297545511 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8ac00ce20 con 0x7fa8bc1020d0 2024-02-09T22:52:10.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8b27fc640 1 -- 172.21.15.3:0/1297545511 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ac018920 con 0x7fa8bc1020d0 2024-02-09T22:52:10.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.735+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc1a6aa0 con 0x7fa8bc1020d0 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.737+0000 7fa8b27fc640 1 -- 172.21.15.3:0/1297545511 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7fa8ac018a80 con 0x7fa8bc1020d0 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 Processor -- start 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 -- start start 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8b27fc640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8a80a2350 0x7fa8a80a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a8be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a94990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a973b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546a47020 con 0x55a546a8ba30 2024-02-09T22:52:10.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8a80a2350 0x7fa8a80a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5469fa320 con 0x55a546a94f60 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c6588740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5469f7850 con 0x55a546a8c3e0 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a8be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8b27fc640 1 -- 172.21.15.3:0/1297545511 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa8ac014070 con 0x7fa8bc1020d0 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a8be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57502/0 (socket says 172.21.15.3:57502) 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c1751640 1 -- 172.21.15.3:0/1426727810 learned_addr learned my addr 172.21.15.3:0/1426727810 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a94990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.738+0000 7fa8c1751640 1 -- 172.21.15.3:0/1426727810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 msgr2=0x55a546a94990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c1751640 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a94990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c1751640 1 -- 172.21.15.3:0/1426727810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 msgr2=0x55a546a973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c1751640 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c1751640 1 -- 172.21.15.3:0/1426727810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5469b5210 con 0x55a546a8ba30 2024-02-09T22:52:10.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a94990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8a80a2350 0x7fa8a80a4810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc073690 tx=0x7fa8b4011b40 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8c1751640 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a8be10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8036040 tx=0x7fa8b802c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aba5eb8185827e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8a77fe640 1 -- 172.21.15.3:0/1426727810 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b802c7e0 con 0x55a546a8ba30 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8a77fe640 1 -- 172.21.15.3:0/1426727810 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8004510 con 0x55a546a8ba30 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.739+0000 7fa8a77fe640 1 -- 172.21.15.3:0/1426727810 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8005430 con 0x55a546a8ba30 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- 172.21.15.3:0/1426727810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 msgr2=0x55a546a8be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a8be10 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8036040 tx=0x7fa8b802c6e0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- 172.21.15.3:0/1426727810 shutdown_connections 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a94990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- 172.21.15.3:0/1426727810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a8be10 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- 172.21.15.3:0/1426727810 >> 172.21.15.3:0/1426727810 conn(0x55a546a86100 msgr2=0x55a546a864e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- 172.21.15.3:0/1426727810 shutdown_connections 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- 172.21.15.3:0/1426727810 wait complete. 2024-02-09T22:52:10.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- start start 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 Processor -- start 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a91ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a923e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a92920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5469f3030 con 0x55a546a8ba30 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5466c6430 con 0x55a546a94f60 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546682e00 con 0x55a546a8c3e0 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.740+0000 7fa8c6588740 10 client.0 _ll_register_callbacks cb 0x55a5469e8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a91ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c0f50640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a923e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1f52640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a92920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a91ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57518/0 (socket says 172.21.15.3:57518) 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 -- 172.21.15.3:0/628886697 learned_addr learned my addr 172.21.15.3:0/628886697 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 -- 172.21.15.3:0/628886697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 msgr2=0x55a546a923e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a923e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 -- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 msgr2=0x55a546a92920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a92920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 -- 172.21.15.3:0/628886697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8bc110af0 con 0x55a546a8ba30 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c1751640 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a91ea0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8005460 tx=0x7fa8b8005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9073bda77b83e8a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/628886697 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b803f850 con 0x55a546a8ba30 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/628886697 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa8b804d070 con 0x55a546a8ba30 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a5466f27e0 con 0x55a546a8ba30 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/628886697 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8b8004df0 con 0x55a546a8ba30 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.741+0000 7fa8c6588740 10 client.15924 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55a5469f16b0 con 0x55a546a8ba30 2024-02-09T22:52:10.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/628886697 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a5466f27e0 con 0x55a546a8ba30 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/628886697 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa8b803c050 con 0x55a546a8ba30 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 10 client.15924 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 10 client.15924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 20 client.15924 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546aa4fa0 0x55a546aa7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 4 client.15924 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a54672f720 con 0x55a546aa4fa0 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546aa4fa0 0x55a546aa7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8a4ff9640 10 client.15924 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.742+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546aa4fa0 0x55a546aa7390 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdd9d693f9c9266c server_cookie=92d698f1c24e6e1e in_seq=0 out_seq=0 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/628886697 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a54672f720 con 0x55a546aa4fa0 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8a4ff9640 10 client.15924 handle_command_reply: tid=0 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 client.15924 shutdown 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 msgr2=0x55a546a91ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a91ea0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8005460 tx=0x7fa8b8005490 comp rx=0 tx=0).stop 2024-02-09T22:52:10.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 shutdown_connections 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546aa4fa0 0x55a546aa7390 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a94f60 0x55a546a92920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8c3e0 0x55a546a923e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 --2- 172.21.15.3:0/628886697 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8ba30 0x55a546a91ea0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8a4ff9640 0 client.15924 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 >> 172.21.15.3:0/628886697 conn(0x55a546a86100 msgr2=0x55a546a87e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.743+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 shutdown_connections 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.744+0000 7fa8c6588740 1 -- 172.21.15.3:0/628886697 wait complete. 2024-02-09T22:52:10.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.744+0000 7fa8c6588740 20 client.15924 trim_cache size 0 max 16384 2024-02-09T22:52:10.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.756+0000 7fa8c6588740 1 Processor -- start 2024-02-09T22:52:10.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.756+0000 7fa8c6588740 1 -- start start 2024-02-09T22:52:10.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.756+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546a945d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.756+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546a923f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a92930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c6588740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546978130 con 0x55a546a8c3e0 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c6588740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546a92e70 con 0x55a546a86100 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c6588740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546a975d0 con 0x55a546a8ba30 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a92930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a92930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57534/0 (socket says 172.21.15.3:57534) 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 -- 172.21.15.3:0/3720259289 learned_addr learned my addr 172.21.15.3:0/3720259289 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1751640 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546a945d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546a923f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 -- 172.21.15.3:0/3720259289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 msgr2=0x55a546a923f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546a923f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 -- 172.21.15.3:0/3720259289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 msgr2=0x55a546a945d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546a945d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 -- 172.21.15.3:0/3720259289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a54698dc90 con 0x55a546a8c3e0 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1751640 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546a945d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8c1f52640 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a92930 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400fcb0 tx=0x7fa8b401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fa873ebba2c5e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8a6ffd640 1 -- 172.21.15.3:0/3720259289 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4004550 con 0x55a546a8c3e0 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8a6ffd640 1 -- 172.21.15.3:0/3720259289 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8b40046b0 con 0x55a546a8c3e0 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.757+0000 7fa8a6ffd640 1 -- 172.21.15.3:0/3720259289 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b4021680 con 0x55a546a8c3e0 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 -- 172.21.15.3:0/3720259289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 msgr2=0x55a546a92930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a92930 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fa8b400fcb0 tx=0x7fa8b401e000 comp rx=0 tx=0).stop 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 -- 172.21.15.3:0/3720259289 shutdown_connections 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a92930 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546a923f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3720259289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546a945d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 -- 172.21.15.3:0/3720259289 >> 172.21.15.3:0/3720259289 conn(0x55a546a8cb30 msgr2=0x55a546a90770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 -- 172.21.15.3:0/3720259289 shutdown_connections 2024-02-09T22:52:10.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 -- 172.21.15.3:0/3720259289 wait complete. 2024-02-09T22:52:10.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 -- start start 2024-02-09T22:52:10.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.758+0000 7fa8c6588740 1 Processor -- start 2024-02-09T22:52:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546af57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546af5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a9b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546a9b6d0 con 0x55a546a8c3e0 2024-02-09T22:52:10.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546a9b840 con 0x55a546a86100 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a546a9ba00 con 0x55a546a8ba30 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c6588740 10 client.0 _ll_register_callbacks cb 0x7fa8b40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1f52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a9b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546af57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546af57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43658/0 (socket says 172.21.15.3:43658) 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 -- 172.21.15.3:0/3011312578 learned_addr learned my addr 172.21.15.3:0/3011312578 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546af5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 -- 172.21.15.3:0/3011312578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 msgr2=0x55a546af5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546af5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 -- 172.21.15.3:0/3011312578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 msgr2=0x55a546a9b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a9b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.759+0000 7fa8c1751640 1 -- 172.21.15.3:0/3011312578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5469f16b0 con 0x55a546a86100 2024-02-09T22:52:10.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c1751640 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546af57c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa8b805afd0 tx=0x7fa8b8005b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab8e3d3b6075e2dd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8a4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/3011312578 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8b802cc10 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/3011312578 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa8b804d040 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8b8004510 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/3011312578 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8b805aba0 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 10 client.25460 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x7fa8b400fa70 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/3011312578 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8004510 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/3011312578 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa8b803c040 con 0x55a546a86100 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 10 client.25460 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 10 client.25460 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 20 client.25460 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546a94f60 0x55a546aff200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:10.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 4 client.25460 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a54677e890 con 0x55a546a94f60 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.760+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546a94f60 0x55a546aff200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.761+0000 7fa8c0f50640 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546a94f60 0x55a546aff200 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3088110028089f14 server_cookie=bd1ec1e4e81f0a60 in_seq=0 out_seq=0 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.761+0000 7fa8a4ff9640 10 client.25460 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8a4ff9640 1 -- 172.21.15.3:0/3011312578 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a54677e890 con 0x55a546a94f60 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8a4ff9640 10 client.25460 handle_command_reply: tid=0 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 client.25460 shutdown 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 msgr2=0x55a546af57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546af57c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa8b805afd0 tx=0x7fa8b8005b50 comp rx=0 tx=0).stop 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 shutdown_connections 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a546a94f60 0x55a546aff200 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a546a8c3e0 0x55a546a9b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8a4ff9640 0 client.25460 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:10.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a546a8ba30 0x55a546af5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 --2- 172.21.15.3:0/3011312578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a546a86100 0x55a546af57c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 >> 172.21.15.3:0/3011312578 conn(0x55a546a8cb30 msgr2=0x55a546a849b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.763+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 shutdown_connections 2024-02-09T22:52:10.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c6588740 1 -- 172.21.15.3:0/3011312578 wait complete. 2024-02-09T22:52:10.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c6588740 20 client.25460 trim_cache size 0 max 16384 2024-02-09T22:52:10.772 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8a80a2350 msgr2=0x7fa8a80a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8a80a2350 0x7fa8a80a4810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa8bc073690 tx=0x7fa8b4011b40 comp rx=0 tx=0).stop 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 msgr2=0x7fa8bc19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc19fc10 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa8ac00e9f0 tx=0x7fa8ac00eeb0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 shutdown_connections 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8a80a2350 0x7fa8a80a4810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8bc199880 0x7fa8bc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa8bc1020d0 0x7fa8bc19fc10 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 --2- 172.21.15.3:0/1297545511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa8bc0ff770 0x7fa8bc19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:10.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 >> 172.21.15.3:0/1297545511 conn(0x7fa8bc076630 msgr2=0x7fa8bc0fcd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:10.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 shutdown_connections 2024-02-09T22:52:10.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:10.764+0000 7fa8c39b5640 1 -- 172.21.15.3:0/1297545511 wait complete. 2024-02-09T22:52:11.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 15701 20.51 MB/sec warmup 5 sec latency 57.732 ms 2024-02-09T22:52:11.779 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:52:11.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:52:11.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 -- 172.21.15.3:0/4086044717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 msgr2=0x7f14201006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 --2- 172.21.15.3:0/4086044717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201006f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1408009900 tx=0x7f140802bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 -- 172.21.15.3:0/4086044717 shutdown_connections 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 --2- 172.21.15.3:0/4086044717 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1420100cc0 0x7f1420104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 --2- 172.21.15.3:0/4086044717 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14201053e0 0x7f142010cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 --2- 172.21.15.3:0/4086044717 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201006f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 -- 172.21.15.3:0/4086044717 >> 172.21.15.3:0/4086044717 conn(0x7f1420077230 msgr2=0x7f1420077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 -- 172.21.15.3:0/4086044717 shutdown_connections 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 -- 172.21.15.3:0/4086044717 wait complete. 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 Processor -- start 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.884+0000 7f14269e4640 1 -- start start 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f14269e4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f14269e4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14201053e0 0x7f1420198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f14269e4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 0x7f14201a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f14269e4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201a3dc0 con 0x7f1420199170 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f14269e4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201a3f30 con 0x7f14201053e0 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f14269e4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14201a40f0 con 0x7f1420100310 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 0x7f14201a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 0x7f14201a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57558/0 (socket says 172.21.15.3:57558) 2024-02-09T22:52:11.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f1424f81640 1 -- 172.21.15.3:0/1198335667 learned_addr learned my addr 172.21.15.3:0/1198335667 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141f7fe640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14201053e0 0x7f1420198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 -- 172.21.15.3:0/1198335667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14201053e0 msgr2=0x7f1420198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14201053e0 0x7f1420198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 -- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 msgr2=0x7f14201a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 0x7f14201a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 -- 172.21.15.3:0/1198335667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1408009560 con 0x7f1420100310 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f1424f81640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 0x7f14201a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141f7fe640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14201053e0 0x7f1420198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.885+0000 7f141ffff640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201986f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f140802c170 tx=0x7f140802c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.886+0000 7f141d7fa640 1 -- 172.21.15.3:0/1198335667 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140802dea0 con 0x7f1420100310 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.886+0000 7f141d7fa640 1 -- 172.21.15.3:0/1198335667 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14080042d0 con 0x7f1420100310 2024-02-09T22:52:11.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.886+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14201a4370 con 0x7f1420100310 2024-02-09T22:52:11.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.886+0000 7f141d7fa640 1 -- 172.21.15.3:0/1198335667 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080039f0 con 0x7f1420100310 2024-02-09T22:52:11.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.886+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14201a4850 con 0x7f1420100310 2024-02-09T22:52:11.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.888+0000 7f141d7fa640 1 -- 172.21.15.3:0/1198335667 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f1408005360 con 0x7f1420100310 2024-02-09T22:52:11.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.888+0000 7f141d7fa640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f140c0a2350 0x7f140c0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.888+0000 7f141f7fe640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f140c0a2350 0x7f140c0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.888+0000 7f141d7fa640 1 -- 172.21.15.3:0/1198335667 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14080f0760 con 0x7f1420100310 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.888+0000 7f14295b0740 1 Processor -- start 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.888+0000 7f14295b0740 1 -- start start 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14295b0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225ab570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14295b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14295b0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225b6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14295b0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522519000 con 0x55c5225b4690 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14295b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522516530 con 0x55c5225abab0 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14295b0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522511d10 con 0x55c5225ab190 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141f7fe640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f140c0a2350 0x7f140c0a4810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1420069000 tx=0x7f1410005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225b6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225ab570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225b6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57560/0 (socket says 172.21.15.3:57560) 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f1424f81640 1 -- 172.21.15.3:0/1171791664 learned_addr learned my addr 172.21.15.3:0/1171791664 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225ab570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59096/0 (socket says 172.21.15.3:59096) 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141f7fe640 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 -- 172.21.15.3:0/1171791664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 msgr2=0x55c5225b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 -- 172.21.15.3:0/1171791664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 msgr2=0x55c5225b6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225b6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 -- 172.21.15.3:0/1171791664 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5224b8210 con 0x55c5225ab190 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f1424f81640 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f141ffff640 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225ab570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14080ede70 tx=0x7f14080bc9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3222660c982a9bd6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14167fc640 1 -- 172.21.15.3:0/1171791664 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080d3020 con 0x55c5225ab190 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14167fc640 1 -- 172.21.15.3:0/1171791664 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14080bd040 con 0x55c5225ab190 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.889+0000 7f14167fc640 1 -- 172.21.15.3:0/1171791664 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080b7bb0 con 0x55c5225ab190 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 -- 172.21.15.3:0/1171791664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 msgr2=0x55c5225ab570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225ab570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14080ede70 tx=0x7f14080bc9a0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 -- 172.21.15.3:0/1171791664 shutdown_connections 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225b6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 --2- 172.21.15.3:0/1171791664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225ab570 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 -- 172.21.15.3:0/1171791664 >> 172.21.15.3:0/1171791664 conn(0x55c5225a58c0 msgr2=0x55c5225a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 -- 172.21.15.3:0/1171791664 shutdown_connections 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 -- 172.21.15.3:0/1171791664 wait complete. 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.890+0000 7f14295b0740 1 -- start start 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 Processor -- start 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225b16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225abab0 0x55c5225b1be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 0x55c5225b2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5224b6e60 con 0x55c5225abab0 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5221e54b0 con 0x55c5225ab190 2024-02-09T22:52:11.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522497670 con 0x55c5225b4690 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f14295b0740 10 client.0 _ll_register_callbacks cb 0x55c5224af190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225abab0 0x55c5225b1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225b16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f1424f81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 0x55c5225b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225b16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43694/0 (socket says 172.21.15.3:43694) 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225abab0 0x55c5225b1be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57576/0 (socket says 172.21.15.3:57576) 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f1424f81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 0x55c5225b2120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59112/0 (socket says 172.21.15.3:59112) 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 -- 172.21.15.3:0/142955402 learned_addr learned my addr 172.21.15.3:0/142955402 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 -- 172.21.15.3:0/142955402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 msgr2=0x55c5225b2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 0x55c5225b2120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 -- 172.21.15.3:0/142955402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225abab0 msgr2=0x55c5225b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225abab0 0x55c5225b1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f141ffff640 1 -- 172.21.15.3:0/142955402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14201a4850 con 0x55c5225ab190 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.891+0000 7f1424f81640 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 0x55c5225b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:11.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f141ffff640 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225b16a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14080b94a0 tx=0x7f14080b9300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f126f2bea38083e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f13ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f13ff7fe640 1 -- 172.21.15.3:0/142955402 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14080d3070 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f13ff7fe640 1 -- 172.21.15.3:0/142955402 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14080c57e0 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c522211860 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f13ff7fe640 1 -- 172.21.15.3:0/142955402 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14080b7db0 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 10 client.25472 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c522510390 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f13ff7fe640 1 -- 172.21.15.3:0/142955402 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c522211860 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f13ff7fe640 1 -- 172.21.15.3:0/142955402 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f140803c070 con 0x55c5225ab190 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 10 client.25472 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 10 client.25472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 20 client.25472 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:11.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5225c4070 0x55c5225c6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 4 client.25472 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.892+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c522262410 con 0x55c5225c4070 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.893+0000 7f141f7fe640 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5225c4070 0x55c5225c6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.893+0000 7f141f7fe640 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5225c4070 0x55c5225c6490 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16a8ce8bee7db15d server_cookie=5d05d4bcdf8e1f0 in_seq=0 out_seq=0 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.893+0000 7f13ff7fe640 10 client.25472 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.893+0000 7f13ff7fe640 1 -- 172.21.15.3:0/142955402 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c522262410 con 0x55c5225c4070 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.893+0000 7f13ff7fe640 10 client.25472 handle_command_reply: tid=0 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 client.25472 shutdown 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 msgr2=0x55c5225b16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225b16a0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f14080b94a0 tx=0x7f14080b9300 comp rx=0 tx=0).stop 2024-02-09T22:52:11.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 shutdown_connections 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5225c4070 0x55c5225c6490 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225b4690 0x55c5225b2120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225abab0 0x55c5225b1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 --2- 172.21.15.3:0/142955402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225b16a0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f13ff7fe640 0 client.25472 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 >> 172.21.15.3:0/142955402 conn(0x55c5225a58c0 msgr2=0x55c5225a7610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 shutdown_connections 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 1 -- 172.21.15.3:0/142955402 wait complete. 2024-02-09T22:52:11.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.894+0000 7f14295b0740 20 client.25472 trim_cache size 0 max 16384 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 Processor -- start 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 -- start start 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225ae920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5225a10d0 con 0x55c5225b4690 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5224b6df0 con 0x55c5225abab0 2024-02-09T22:52:11.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f14295b0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5225cb690 con 0x55c5225ab190 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225ae920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57584/0 (socket says 172.21.15.3:57584) 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 -- 172.21.15.3:0/2311808787 learned_addr learned my addr 172.21.15.3:0/2311808787 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f141ffff640 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f141f7fe640 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 -- 172.21.15.3:0/2311808787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 msgr2=0x55c5225b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 -- 172.21.15.3:0/2311808787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 msgr2=0x55c5225b1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.910+0000 7f1424f81640 1 -- 172.21.15.3:0/2311808787 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5224acc70 con 0x55c5225b4690 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f1424f81640 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225ae920 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f141800fcc0 tx=0x7f1418009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f10a5068f424c2bc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:11.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f1415ffb640 1 -- 172.21.15.3:0/2311808787 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418020070 con 0x55c5225b4690 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f1415ffb640 1 -- 172.21.15.3:0/2311808787 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1418018040 con 0x55c5225b4690 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f1415ffb640 1 -- 172.21.15.3:0/2311808787 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418017050 con 0x55c5225b4690 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 -- 172.21.15.3:0/2311808787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 msgr2=0x55c5225ae920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225ae920 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f141800fcc0 tx=0x7f1418009500 comp rx=0 tx=0).stop 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 -- 172.21.15.3:0/2311808787 shutdown_connections 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225abab0 0x55c5225b1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c5225ae920 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 --2- 172.21.15.3:0/2311808787 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225ab190 0x55c5225b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 -- 172.21.15.3:0/2311808787 >> 172.21.15.3:0/2311808787 conn(0x55c5225a58c0 msgr2=0x55c5225ac9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 -- 172.21.15.3:0/2311808787 shutdown_connections 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.911+0000 7f14295b0740 1 -- 172.21.15.3:0/2311808787 wait complete. 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 -- start start 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 Processor -- start 2024-02-09T22:52:11.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225ba660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225abab0 0x55c5225baba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c522614580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522614ac0 con 0x55c5225b4690 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522614c00 con 0x55c5225ab190 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c522614dc0 con 0x55c5225abab0 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f14295b0740 10 client.0 _ll_register_callbacks cb 0x7f1408005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c522614580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f1424f81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c522614580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57586/0 (socket says 172.21.15.3:57586) 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f141ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225ba660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f1424f81640 1 -- 172.21.15.3:0/2287513944 learned_addr learned my addr 172.21.15.3:0/2287513944 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.912+0000 7f141f7fe640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225abab0 0x55c5225baba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f1424f81640 1 -- 172.21.15.3:0/2287513944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225abab0 msgr2=0x55c5225baba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f1424f81640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225abab0 0x55c5225baba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f1424f81640 1 -- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 msgr2=0x55c5225ba660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:11.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f1424f81640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225ba660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f1424f81640 1 -- 172.21.15.3:0/2287513944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141800f970 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f141ffff640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225ba660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f1424f81640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c522614580 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1418012420 tx=0x7f1418004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb196603f4d135 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f13ff7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f13ff7fe640 1 -- 172.21.15.3:0/2287513944 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1418020040 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f13ff7fe640 1 -- 172.21.15.3:0/2287513944 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1418004fd0 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14080bd040 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f13ff7fe640 1 -- 172.21.15.3:0/2287513944 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1418009ec0 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 10 client.15948 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x55c5224d2ac0 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f13ff7fe640 1 -- 172.21.15.3:0/2287513944 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14080bd040 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f13ff7fe640 1 -- 172.21.15.3:0/2287513944 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1418010430 con 0x55c5225b4690 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 10 client.15948 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 10 client.15948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:11.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 20 client.15948 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c52261c680 0x55c52261eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 4 client.15948 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:52:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.913+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c52224e7b0 con 0x55c52261c680 2024-02-09T22:52:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.914+0000 7f141ffff640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c52261c680 0x55c52261eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.914+0000 7f13ff7fe640 10 client.15948 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:11.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.914+0000 7f141ffff640 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c52261c680 0x55c52261eaa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c331f92a7ef61e7f server_cookie=e206c31d4bcff75d in_seq=0 out_seq=0 2024-02-09T22:52:11.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f13ff7fe640 1 -- 172.21.15.3:0/2287513944 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c52224e7b0 con 0x55c52261c680 2024-02-09T22:52:11.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f13ff7fe640 10 client.15948 handle_command_reply: tid=0 2024-02-09T22:52:11.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 client.15948 shutdown 2024-02-09T22:52:11.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 msgr2=0x55c522614580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c522614580 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f1418012420 tx=0x7f1418004cf0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 shutdown_connections 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c52261c680 0x55c52261eaa0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5225b4690 0x55c522614580 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f13ff7fe640 0 client.15948 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5225abab0 0x55c5225baba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 --2- 172.21.15.3:0/2287513944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5225ab190 0x55c5225ba660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 >> 172.21.15.3:0/2287513944 conn(0x55c5225a58c0 msgr2=0x55c5225a4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 shutdown_connections 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 1 -- 172.21.15.3:0/2287513944 wait complete. 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.916+0000 7f14295b0740 20 client.15948 trim_cache size 0 max 16384 2024-02-09T22:52:11.925 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:52:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:52:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "f98afe9b-958d-42e4-bcb4-d296bbe228e9", 2024-02-09T22:52:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:52:11.926 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f140c0a2350 msgr2=0x7f140c0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f140c0a2350 0x7f140c0a4810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1420069000 tx=0x7f1410005f90 comp rx=0 tx=0).stop 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 msgr2=0x7f14201986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201986f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f140802c170 tx=0x7f140802c670 comp rx=0 tx=0).stop 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 shutdown_connections 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f140c0a2350 0x7f140c0a4810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14201053e0 0x7f1420198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1420199170 0x7f14201a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 --2- 172.21.15.3:0/1198335667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1420100310 0x7f14201986f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:11.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 >> 172.21.15.3:0/1198335667 conn(0x7f1420077230 msgr2=0x7f14200ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:11.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 shutdown_connections 2024-02-09T22:52:11.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:11.917+0000 7f14269e4640 1 -- 172.21.15.3:0/1198335667 wait complete. 2024-02-09T22:52:11.932 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:52:11.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:52:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 -- 172.21.15.3:0/3510096937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 msgr2=0x7fd644109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 --2- 172.21.15.3:0/3510096937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd644109870 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd634009930 tx=0x7fd63402bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 -- 172.21.15.3:0/3510096937 shutdown_connections 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 --2- 172.21.15.3:0/3510096937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd6441102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 --2- 172.21.15.3:0/3510096937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 0x7fd644103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 --2- 172.21.15.3:0/3510096937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd644109870 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 -- 172.21.15.3:0/3510096937 >> 172.21.15.3:0/3510096937 conn(0x7fd6440feea0 msgr2=0x7fd6441012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 -- 172.21.15.3:0/3510096937 shutdown_connections 2024-02-09T22:52:12.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.040+0000 7fd64882a640 1 -- 172.21.15.3:0/3510096937 wait complete. 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 Processor -- start 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 -- start start 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 0x7fd64419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd64419fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd6441998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd644199f40 con 0x7fd6441033f0 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64419a0b0 con 0x7fd644103db0 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.041+0000 7fd64882a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd64419a270 con 0x7fd644109490 2024-02-09T22:52:12.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd64419fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd64259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 0x7fd64419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd6441998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd64419fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43728/0 (socket says 172.21.15.3:43728) 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63bfff640 1 -- 172.21.15.3:0/3373995403 learned_addr learned my addr 172.21.15.3:0/3373995403 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 -- 172.21.15.3:0/3373995403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 msgr2=0x7fd64419fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd64419fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 -- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 msgr2=0x7fd64419f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 0x7fd64419f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 -- 172.21.15.3:0/3373995403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd634009560 con 0x7fd644109490 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63bfff640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd64419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:12.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd64259c640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 0x7fd64419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd6441998f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009fd0 tx=0x7fd63c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63b7fe640 1 -- 172.21.15.3:0/3373995403 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c00ce20 con 0x7fd644109490 2024-02-09T22:52:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63b7fe640 1 -- 172.21.15.3:0/3373995403 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd63c005020 con 0x7fd644109490 2024-02-09T22:52:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd64419a550 con 0x7fd644109490 2024-02-09T22:52:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.042+0000 7fd63b7fe640 1 -- 172.21.15.3:0/3373995403 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c010690 con 0x7fd644109490 2024-02-09T22:52:12.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.043+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6441a67e0 con 0x7fd644109490 2024-02-09T22:52:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.044+0000 7fd63b7fe640 1 -- 172.21.15.3:0/3373995403 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7fd63c005500 con 0x7fd644109490 2024-02-09T22:52:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.045+0000 7fd63b7fe640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6300a23d0 0x7fd6300a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.045+0000 7fd63b7fe640 1 -- 172.21.15.3:0/3373995403 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd63c014070 con 0x7fd644109490 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.045+0000 7fd64259c640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6300a23d0 0x7fd6300a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64259c640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6300a23d0 0x7fd6300a4890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd634005ab0 tx=0x7fd634005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 Processor -- start 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 -- start start 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f0d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f98e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 0x558f9c4fc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c4abd00 con 0x558f9c4f0980 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c4602f0 con 0x558f9c4f1330 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.046+0000 7fd64b402740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c45d820 con 0x558f9c4f9eb0 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd64259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 0x558f9c4fc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 0x558f9c4fc300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59146/0 (socket says 172.21.15.3:59146) 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd64259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57604/0 (socket says 172.21.15.3:57604) 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 -- 172.21.15.3:0/1099133608 learned_addr learned my addr 172.21.15.3:0/1099133608 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43734/0 (socket says 172.21.15.3:43734) 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 -- 172.21.15.3:0/1099133608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 msgr2=0x558f9c4f98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 -- 172.21.15.3:0/1099133608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 msgr2=0x558f9c4f0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 -- 172.21.15.3:0/1099133608 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f9c41aff0 con 0x558f9c4f9eb0 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd642d9d640 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 0x558f9c4fc300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd63c097560 tx=0x7fd63c09ad70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=364c9839e40a69c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd638ff9640 1 -- 172.21.15.3:0/1099133608 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c09f250 con 0x558f9c4f9eb0 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd638ff9640 1 -- 172.21.15.3:0/1099133608 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd63c0a38a0 con 0x558f9c4f9eb0 2024-02-09T22:52:12.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.047+0000 7fd638ff9640 1 -- 172.21.15.3:0/1099133608 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c0cc470 con 0x558f9c4f9eb0 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 -- 172.21.15.3:0/1099133608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 msgr2=0x558f9c4fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 0x558f9c4fc300 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd63c097560 tx=0x7fd63c09ad70 comp rx=0 tx=0).stop 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 -- 172.21.15.3:0/1099133608 shutdown_connections 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f9eb0 0x558f9c4fc300 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 --2- 172.21.15.3:0/1099133608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 -- 172.21.15.3:0/1099133608 >> 172.21.15.3:0/1099133608 conn(0x558f9c4eb050 msgr2=0x558f9c4eb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 -- 172.21.15.3:0/1099133608 shutdown_connections 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.048+0000 7fd64b402740 1 -- 172.21.15.3:0/1099133608 wait complete. 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 -- start start 2024-02-09T22:52:12.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 Processor -- start 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f6e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f1330 0x558f9c4f7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c459000 con 0x558f9c4f0980 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c12c430 con 0x558f9c4f9eb0 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c0e8e00 con 0x558f9c4f1330 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64b402740 10 client.0 _ll_register_callbacks cb 0x558f9c44d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd642d9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f6e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57620/0 (socket says 172.21.15.3:57620) 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64259c640 1 -- 172.21.15.3:0/3118524504 learned_addr learned my addr 172.21.15.3:0/3118524504 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd63bfff640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f1330 0x558f9c4f7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd642d9d640 1 -- 172.21.15.3:0/3118524504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f1330 msgr2=0x558f9c4f7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f1330 0x558f9c4f7340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd642d9d640 1 -- 172.21.15.3:0/3118524504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 msgr2=0x558f9c4f6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd642d9d640 1 -- 172.21.15.3:0/3118524504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6441a67e0 con 0x558f9c4f9eb0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd63bfff640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f1330 0x558f9c4f7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.049+0000 7fd64259c640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f7880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd63c005fd0 tx=0x7fd63c0cbf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4fcbd1025d1a40e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd629ffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd629ffb640 1 -- 172.21.15.3:0/3118524504 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63c09f2d0 con 0x558f9c4f9eb0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd629ffb640 1 -- 172.21.15.3:0/3118524504 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd63c0cc420 con 0x558f9c4f9eb0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f9c1587e0 con 0x558f9c4f9eb0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd629ffb640 1 -- 172.21.15.3:0/3118524504 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd63c09a550 con 0x558f9c4f9eb0 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 10 client.25490 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:12.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558f9c457680 con 0x558f9c4f9eb0 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd629ffb640 1 -- 172.21.15.3:0/3118524504 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd63c0a7080 con 0x558f9c4f9eb0 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd629ffb640 1 -- 172.21.15.3:0/3118524504 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd63c01f070 con 0x558f9c4f9eb0 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 10 client.25490 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 20 client.25490 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c50a110 0x558f9c50c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 4 client.25490 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.050+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f9c195720 con 0x558f9c50a110 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.051+0000 7fd64259c640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c50a110 0x558f9c50c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.051+0000 7fd629ffb640 10 client.25490 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.051+0000 7fd64259c640 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c50a110 0x558f9c50c500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef5ef8da4acf8655 server_cookie=58f2a34f5fb64551 in_seq=0 out_seq=0 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd629ffb640 1 -- 172.21.15.3:0/3118524504 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558f9c195720 con 0x558f9c50a110 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd629ffb640 10 client.25490 handle_command_reply: tid=0 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 client.25490 shutdown 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 msgr2=0x558f9c4f7880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f7880 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd63c005fd0 tx=0x7fd63c0cbf50 comp rx=0 tx=0).stop 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 shutdown_connections 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c50a110 0x558f9c50c500 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f7880 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f1330 0x558f9c4f7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 --2- 172.21.15.3:0/3118524504 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f0980 0x558f9c4f6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd629ffb640 0 client.25490 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 >> 172.21.15.3:0/3118524504 conn(0x558f9c4eb050 msgr2=0x558f9c4ecda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 shutdown_connections 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 1 -- 172.21.15.3:0/3118524504 wait complete. 2024-02-09T22:52:12.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.053+0000 7fd64b402740 20 client.25490 trim_cache size 0 max 16384 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 Processor -- start 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 -- start start 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4f9530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c3ddfd0 con 0x558f9c4f9eb0 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c4eab80 con 0x558f9c4f1330 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64b402740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c4f02d0 con 0x558f9c4f0980 2024-02-09T22:52:12.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4f9530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd642d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4f9530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59164/0 (socket says 172.21.15.3:59164) 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.068+0000 7fd64259c640 1 -- 172.21.15.3:0/2757253808 learned_addr learned my addr 172.21.15.3:0/2757253808 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd64259c640 1 -- 172.21.15.3:0/2757253808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 msgr2=0x558f9c4f20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd64259c640 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd64259c640 1 -- 172.21.15.3:0/2757253808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f9eb0 msgr2=0x558f9c4f2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd64259c640 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd64259c640 1 -- 172.21.15.3:0/2757253808 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f9c3f3b30 con 0x558f9c4f0980 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd64259c640 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4f9530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd634036040 tx=0x7fd63402bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bcac7056c770ee7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd62bfff640 1 -- 172.21.15.3:0/2757253808 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634004400 con 0x558f9c4f0980 2024-02-09T22:52:12.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd62bfff640 1 -- 172.21.15.3:0/2757253808 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd634003ad0 con 0x558f9c4f0980 2024-02-09T22:52:12.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.069+0000 7fd62bfff640 1 -- 172.21.15.3:0/2757253808 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634005390 con 0x558f9c4f0980 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 -- 172.21.15.3:0/2757253808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 msgr2=0x558f9c4f9530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4f9530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd634036040 tx=0x7fd63402bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 -- 172.21.15.3:0/2757253808 shutdown_connections 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f1330 0x558f9c4f20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f9eb0 0x558f9c4f2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 --2- 172.21.15.3:0/2757253808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4f9530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 -- 172.21.15.3:0/2757253808 >> 172.21.15.3:0/2757253808 conn(0x558f9c4eb050 msgr2=0x558f9c4eb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 -- 172.21.15.3:0/2757253808 shutdown_connections 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 -- 172.21.15.3:0/2757253808 wait complete. 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 -- start start 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.070+0000 7fd64b402740 1 Processor -- start 2024-02-09T22:52:12.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 0x558f9c4ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c500290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c5017b0 con 0x558f9c4f1330 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c560590 con 0x558f9c4f9eb0 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f9c5606d0 con 0x558f9c4f0980 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 10 client.0 _ll_register_callbacks cb 0x7fd63c097420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 0x558f9c4ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4ff810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59174/0 (socket says 172.21.15.3:59174) 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 0x558f9c4ffd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57644/0 (socket says 172.21.15.3:57644) 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64259c640 1 -- 172.21.15.3:0/3555510821 learned_addr learned my addr 172.21.15.3:0/3555510821 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd642d9d640 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c500290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 -- 172.21.15.3:0/3555510821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 msgr2=0x558f9c4ff810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4ff810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 -- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 msgr2=0x558f9c500290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c500290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 -- 172.21.15.3:0/3555510821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd644110bc0 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd63bfff640 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 0x558f9c4ffd50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x558f9c4f1730 tx=0x7fd62c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b354a8d82aafb3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd629ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd629ffb640 1 -- 172.21.15.3:0/3555510821 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62c004230 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd63c0a38a0 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.071+0000 7fd629ffb640 1 -- 172.21.15.3:0/3555510821 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd62c004390 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd629ffb640 1 -- 172.21.15.3:0/3555510821 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd62c005020 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 10 client.15966 fetch_fsmap learned FSMap version 25 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) v3 -- 0x558f9c419850 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd629ffb640 1 -- 172.21.15.3:0/3555510821 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd62c02a020 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd629ffb640 1 -- 172.21.15.3:0/3555510821 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 25) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd62c0035f0 con 0x558f9c4f1330 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 10 client.15966 fetch_fsmap finished waiting for FSMap version 25 2024-02-09T22:52:12.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 10 client.15966 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 20 client.15966 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c561f70 0x558f9c564390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 4 client.15966 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558f9c1e4890 con 0x558f9c561f70 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.072+0000 7fd64259c640 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c561f70 0x558f9c564390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.073+0000 7fd64259c640 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c561f70 0x558f9c564390 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85b26f8474a75e7c server_cookie=bddf4f8fa8467f94 in_seq=0 out_seq=0 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.073+0000 7fd629ffb640 10 client.15966 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.074+0000 7fd629ffb640 1 -- 172.21.15.3:0/3555510821 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558f9c1e4890 con 0x558f9c561f70 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.074+0000 7fd629ffb640 10 client.15966 handle_command_reply: tid=0 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.074+0000 7fd64b402740 1 client.15966 shutdown 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.074+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 msgr2=0x558f9c4ffd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.074+0000 7fd64b402740 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 0x558f9c4ffd50 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x558f9c4f1730 tx=0x7fd62c00f790 comp rx=0 tx=0).stop 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 shutdown_connections 2024-02-09T22:52:12.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f9c561f70 0x558f9c564390 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f9c4f9eb0 0x558f9c500290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd629ffb640 0 client.15966 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f9c4f1330 0x558f9c4ffd50 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 --2- 172.21.15.3:0/3555510821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f9c4f0980 0x558f9c4ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 >> 172.21.15.3:0/3555510821 conn(0x558f9c4eb050 msgr2=0x558f9c4f20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 shutdown_connections 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 1 -- 172.21.15.3:0/3555510821 wait complete. 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64b402740 20 client.15966 trim_cache size 0 max 16384 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:52:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "f98afe9b-958d-42e4-bcb4-d296bbe228e9": { 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "f98afe9b-958d-42e4-bcb4-d296bbe228e9", 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:52:12.084 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6300a23d0 msgr2=0x7fd6300a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6300a23d0 0x7fd6300a4890 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd634005ab0 tx=0x7fd634005fd0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 msgr2=0x7fd6441998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd6441998f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd63c009fd0 tx=0x7fd63c00ef10 comp rx=0 tx=0).stop 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 shutdown_connections 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6300a23d0 0x7fd6300a4890 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd644103db0 0x7fd64419fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd644109490 0x7fd6441998f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 --2- 172.21.15.3:0/3373995403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6441033f0 0x7fd64419f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 >> 172.21.15.3:0/3373995403 conn(0x7fd6440feea0 msgr2=0x7fd6440ff720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:12.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 shutdown_connections 2024-02-09T22:52:12.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:12.075+0000 7fd64882a640 1 -- 172.21.15.3:0/3373995403 wait complete. 2024-02-09T22:52:12.090 INFO:tasks.cephfs.filesystem:scrub status for tag:f98afe9b-958d-42e4-bcb4-d296bbe228e9 - {'path': '/', 'tag': 'f98afe9b-958d-42e4-bcb4-d296bbe228e9', 'options': 'recursive,force'} 2024-02-09T22:52:12.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:12 smithi003 ceph-mon[32678]: pgmap v31: 97 pgs: 97 active+clean; 1.9 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 78 KiB/s wr, 39 op/s 2024-02-09T22:52:12.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:12 smithi089 ceph-mon[36665]: pgmap v31: 97 pgs: 97 active+clean; 1.9 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 78 KiB/s wr, 39 op/s 2024-02-09T22:52:12.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 18363 19.43 MB/sec warmup 6 sec latency 101.852 ms 2024-02-09T22:52:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:12 smithi064 ceph-mon[36638]: pgmap v31: 97 pgs: 97 active+clean; 1.9 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 78 KiB/s wr, 39 op/s 2024-02-09T22:52:13.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:13 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:52:13.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:13 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:13.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:13 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:13.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:13 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:13.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 21736 19.39 MB/sec warmup 7 sec latency 6.624 ms 2024-02-09T22:52:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:13 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:52:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:13 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:13 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:13 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:13 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:52:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:13 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:13 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:13 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:14.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:14 smithi089 ceph-mon[36665]: pgmap v32: 97 pgs: 97 active+clean; 7.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 599 KiB/s wr, 159 op/s 2024-02-09T22:52:14.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 25123 19.32 MB/sec warmup 8 sec latency 27.444 ms 2024-02-09T22:52:14.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:14 smithi003 ceph-mon[32678]: pgmap v32: 97 pgs: 97 active+clean; 7.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 599 KiB/s wr, 159 op/s 2024-02-09T22:52:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:14 smithi064 ceph-mon[36638]: pgmap v32: 97 pgs: 97 active+clean; 7.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 198 KiB/s rd, 599 KiB/s wr, 159 op/s 2024-02-09T22:52:15.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 28249 18.82 MB/sec warmup 9 sec latency 19.090 ms 2024-02-09T22:52:15.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:15 smithi003 ceph-mon[32678]: pgmap v33: 97 pgs: 97 active+clean; 7.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 598 KiB/s wr, 153 op/s 2024-02-09T22:52:15.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:15 smithi064 ceph-mon[36638]: pgmap v33: 97 pgs: 97 active+clean; 7.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 598 KiB/s wr, 153 op/s 2024-02-09T22:52:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:15 smithi089 ceph-mon[36665]: pgmap v33: 97 pgs: 97 active+clean; 7.1 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 196 KiB/s rd, 598 KiB/s wr, 153 op/s 2024-02-09T22:52:16.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 31310 18.58 MB/sec warmup 10 sec latency 6.832 ms 2024-02-09T22:52:17.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 33022 17.73 MB/sec warmup 11 sec latency 84.478 ms 2024-02-09T22:52:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:17 smithi003 ceph-mon[32678]: pgmap v34: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.3 MiB/s wr, 269 op/s 2024-02-09T22:52:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:17 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:52:17.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:17 smithi064 ceph-mon[36638]: pgmap v34: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.3 MiB/s wr, 269 op/s 2024-02-09T22:52:17.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:17 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:52:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:17 smithi089 ceph-mon[36665]: pgmap v34: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 1.3 MiB/s wr, 269 op/s 2024-02-09T22:52:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:17 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:52:18.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35748 17.66 MB/sec warmup 12 sec latency 271.859 ms 2024-02-09T22:52:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:18 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:18 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:18 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:19.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35748 16.30 MB/sec warmup 13 sec latency 1271.900 ms 2024-02-09T22:52:19.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:19 smithi003 ceph-mon[32678]: pgmap v35: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.3 MiB/s wr, 250 op/s 2024-02-09T22:52:19.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:19 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:52:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:19 smithi064 ceph-mon[36638]: pgmap v35: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.3 MiB/s wr, 250 op/s 2024-02-09T22:52:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:19 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:52:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:19 smithi089 ceph-mon[36665]: pgmap v35: 97 pgs: 97 active+clean; 14 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.3 MiB/s wr, 250 op/s 2024-02-09T22:52:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:19 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:52:20.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35752 15.14 MB/sec warmup 14 sec latency 2106.951 ms 2024-02-09T22:52:20.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:20 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:20 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:20 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:52:20.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:20 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:52:20.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:20 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:20 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:20 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:20 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:52:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:20 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:52:20.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:20 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:20 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:20 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:20 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:52:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:20 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:52:20.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:20 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:52:21.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35752 14.13 MB/sec warmup 15 sec latency 1163.506 ms 2024-02-09T22:52:21.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:21 smithi003 ceph-mon[32678]: pgmap v36: 97 pgs: 97 active+clean; 17 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.6 MiB/s wr, 288 op/s 2024-02-09T22:52:21.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:21 smithi064 ceph-mon[36638]: pgmap v36: 97 pgs: 97 active+clean; 17 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.6 MiB/s wr, 288 op/s 2024-02-09T22:52:21.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:21 smithi089 ceph-mon[36665]: pgmap v36: 97 pgs: 97 active+clean; 17 MiB data, 436 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 1.6 MiB/s wr, 288 op/s 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 -- 172.21.15.3:0/1265244973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 msgr2=0x7f3b4010cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/1265244973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010cf00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009cd0 tx=0x7f3b3802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 -- 172.21.15.3:0/1265244973 shutdown_connections 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/1265244973 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010cf00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/1265244973 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b40100380 0x7f3b40100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/1265244973 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b401062c0 0x7f3b401066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 -- 172.21.15.3:0/1265244973 >> 172.21.15.3:0/1265244973 conn(0x7f3b40076ef0 msgr2=0x7f3b40077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 -- 172.21.15.3:0/1265244973 shutdown_connections 2024-02-09T22:52:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.397+0000 7f3b44d8a640 1 -- 172.21.15.3:0/1265244973 wait complete. 2024-02-09T22:52:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.398+0000 7f3b44d8a640 1 Processor -- start 2024-02-09T22:52:22.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.398+0000 7f3b44d8a640 1 -- start start 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.398+0000 7f3b44d8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 0x7f3b4010d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b44d8a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b44d8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b401062c0 0x7f3b4010e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b44d8a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401a5c10 con 0x7f3b40100d40 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b44d8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401a5d80 con 0x7f3b40100380 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b44d8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b401a5f40 con 0x7f3b401062c0 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b3e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 0x7f3b4010d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b3e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 0x7f3b4010d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:44484/0 (socket says 172.21.15.3:44484) 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49464/0 (socket says 172.21.15.3:49464) 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b3e59c640 1 -- 172.21.15.3:0/679770753 learned_addr learned my addr 172.21.15.3:0/679770753 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:22.405 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b401062c0 0x7f3b4010e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 -- 172.21.15.3:0/679770753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b401062c0 msgr2=0x7f3b4010e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b401062c0 0x7f3b4010e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 -- 172.21.15.3:0/679770753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 msgr2=0x7f3b4010d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 0x7f3b4010d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 -- 172.21.15.3:0/679770753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b38009900 con 0x7f3b40100d40 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 0x7f3b4010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.399+0000 7f3b37fff640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010de90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800d6d0 tx=0x7f3b2800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.400+0000 7f3b377fe640 1 -- 172.21.15.3:0/679770753 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28005220 con 0x7f3b40100d40 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.400+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b401a6220 con 0x7f3b40100d40 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.400+0000 7f3b377fe640 1 -- 172.21.15.3:0/679770753 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b28009d00 con 0x7f3b40100d40 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.400+0000 7f3b377fe640 1 -- 172.21.15.3:0/679770753 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b28004de0 con 0x7f3b40100d40 2024-02-09T22:52:22.406 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.400+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b401a6730 con 0x7f3b40100d40 2024-02-09T22:52:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 Processor -- start 2024-02-09T22:52:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b377fe640 1 -- 172.21.15.3:0/679770753 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f3b28009820 con 0x7f3b40100d40 2024-02-09T22:52:22.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 -- start start 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d188490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 0x55725d191260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1917a0 0x55725d193b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d0f6a00 con 0x55725d1889d0 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d0f3f30 con 0x55725d1880b0 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d0ef710 con 0x55725d1917a0 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b3e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 0x55725d191260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b3e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 0x55725d191260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49480/0 (socket says 172.21.15.3:49480) 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.403+0000 7f3b3e59c640 1 -- 172.21.15.3:0/1348658772 learned_addr learned my addr 172.21.15.3:0/1348658772 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b377fe640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b2c0a2340 0x7f3b2c0a4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b37fff640 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1917a0 0x55725d193b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d188490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3e59c640 1 -- 172.21.15.3:0/1348658772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1917a0 msgr2=0x55725d193b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1917a0 0x55725d193b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b2c0a2340 0x7f3b2c0a4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3e59c640 1 -- 172.21.15.3:0/1348658772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 msgr2=0x55725d188490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d188490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3e59c640 1 -- 172.21.15.3:0/1348658772 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55725d0b43b0 con 0x55725d1889d0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b377fe640 1 -- 172.21.15.3:0/679770753 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3b280cde50 con 0x7f3b40100d40 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 0x55725d191260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000f3f0 tx=0x7f3b3000f8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=801f2712fa0211cf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b2c0a2340 0x7f3b2c0a4800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b38005cc0 tx=0x7f3b38005c50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b34ff9640 1 -- 172.21.15.3:0/1348658772 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30010960 con 0x55725d1889d0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b34ff9640 1 -- 172.21.15.3:0/1348658772 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b30018440 con 0x55725d1889d0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.404+0000 7f3b34ff9640 1 -- 172.21.15.3:0/1348658772 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b300165a0 con 0x55725d1889d0 2024-02-09T22:52:22.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1348658772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 msgr2=0x55725d191260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 0x55725d191260 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f3b3000f3f0 tx=0x7f3b3000f8b0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1348658772 shutdown_connections 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1917a0 0x55725d193b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1889d0 0x55725d191260 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1348658772 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d188490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1348658772 >> 172.21.15.3:0/1348658772 conn(0x55725d1829d0 msgr2=0x55725d182db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1348658772 shutdown_connections 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1348658772 wait complete. 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.405+0000 7f3b47a3a740 1 -- start start 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 Processor -- start 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d18e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d18ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d094b90 con 0x55725d1917a0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725cdc3250 con 0x55725d1880b0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d075200 con 0x55725d1889d0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b47a3a740 10 client.0 _ll_register_callbacks cb 0x55725d0ab5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49496/0 (socket says 172.21.15.3:49496) 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3562633727 learned_addr learned my addr 172.21.15.3:0/3562633727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b37fff640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d18ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d18e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3562633727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 msgr2=0x55725d18ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d18ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 msgr2=0x55725d18e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d18e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.406+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3562633727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3000ac60 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18f240 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009cd0 tx=0x7f3b380095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69385e11c26fd9bb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3562633727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38032ac0 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3562633727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3b38003970 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55725d115ee0 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3562633727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b38003b30 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 10 client.15978 fetch_fsmap learned FSMap version 26 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55725d0edd90 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3562633727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55725d115ee0 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3562633727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f3b3804d530 con 0x55725d1917a0 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 10 client.15978 fetch_fsmap finished waiting for FSMap version 26 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 10 client.15978 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 20 client.15978 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:22.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1a1a90 0x55725d1a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 4 client.15978 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.407+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55725cdef600 con 0x55725d1a1a90 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.408+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1a1a90 0x55725d1a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.408+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1a1a90 0x55725d1a3eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce30f229a614a07c server_cookie=3dc597207774138e in_seq=0 out_seq=0 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.408+0000 7f3b25ffb640 10 client.15978 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3562633727 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55725cdef600 con 0x55725d1a1a90 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b25ffb640 10 client.15978 handle_command_reply: tid=0 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 client.15978 shutdown 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 msgr2=0x55725d18f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18f240 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f3b38009cd0 tx=0x7f3b380095c0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 shutdown_connections 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1a1a90 0x55725d1a3eb0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18f240 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d18ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3562633727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d18e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b25ffb640 0 client.15978 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 >> 172.21.15.3:0/3562633727 conn(0x55725d1829d0 msgr2=0x55725d184720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 shutdown_connections 2024-02-09T22:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3562633727 wait complete. 2024-02-09T22:52:22.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.410+0000 7f3b47a3a740 20 client.15978 trim_cache size 0 max 16384 2024-02-09T22:52:22.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.422+0000 7f3b47a3a740 1 Processor -- start 2024-02-09T22:52:22.431 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 -- start start 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1880b0 0x55725d18f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1889d0 0x55725d18ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d19e4c0 con 0x55725d1917a0 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d1873c0 con 0x55725d1889d0 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d18cd10 con 0x55725d1880b0 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49500/0 (socket says 172.21.15.3:49500) 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/1758031510 learned_addr learned my addr 172.21.15.3:0/1758031510 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b37fff640 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1889d0 0x55725d18ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1880b0 0x55725d18f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/1758031510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1880b0 msgr2=0x55725d18f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.432 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1880b0 0x55725d18f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/1758031510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1889d0 msgr2=0x55725d18ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1889d0 0x55725d18ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/1758031510 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55725d08a820 con 0x55725d1917a0 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1880b0 0x55725d18f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18bf60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3b3802c130 tx=0x7f3b38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa87eafff89630f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b27fff640 1 -- 172.21.15.3:0/1758031510 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3804dd00 con 0x55725d1917a0 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b27fff640 1 -- 172.21.15.3:0/1758031510 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b38032400 con 0x55725d1917a0 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.423+0000 7f3b27fff640 1 -- 172.21.15.3:0/1758031510 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b38060bd0 con 0x55725d1917a0 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1758031510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 msgr2=0x55725d18bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.433 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18bf60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f3b3802c130 tx=0x7f3b38004a40 comp rx=0 tx=0).stop 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1758031510 shutdown_connections 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1889d0 0x55725d18ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d18bf60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/1758031510 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1880b0 0x55725d18f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1758031510 >> 172.21.15.3:0/1758031510 conn(0x55725d1829d0 msgr2=0x55725d18d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1758031510 shutdown_connections 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 -- 172.21.15.3:0/1758031510 wait complete. 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 -- start start 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.424+0000 7f3b47a3a740 1 Processor -- start 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d197230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d197770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d1f7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.434 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d198040 con 0x55725d1917a0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d1f8500 con 0x55725d1880b0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55725d1f8640 con 0x55725d1889d0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 10 client.0 _ll_register_callbacks cb 0x7f3b38044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d1f7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d1f7fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49514/0 (socket says 172.21.15.3:49514) 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3104618070 learned_addr learned my addr 172.21.15.3:0/3104618070 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b37fff640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d197230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3104618070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 msgr2=0x55725d197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 msgr2=0x55725d197230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d197230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 -- 172.21.15.3:0/3104618070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55725d0edd90 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b3ed9d640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d1f7fc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006010 tx=0x7f3b3804d480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161b1d5943dba0b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b25ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3104618070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3804dd00 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3104618070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3b38003970 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.425+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b30018440 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3104618070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b38003b50 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 10 client.15984 fetch_fsmap learned FSMap version 26 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) v3 -- 0x55725d0b06f0 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3104618070 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b30018440 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3104618070 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 26) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f3b3806b020 con 0x55725d1917a0 2024-02-09T22:52:22.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 10 client.15984 fetch_fsmap finished waiting for FSMap version 26 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 10 client.15984 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 20 client.15984 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1f97f0 0x55725d1fbc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 4 client.15984 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55725ce401b0 con 0x55725d1f97f0 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.426+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1f97f0 0x55725d1fbc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.427+0000 7f3b25ffb640 10 client.15984 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:22.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.427+0000 7f3b3e59c640 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1f97f0 0x55725d1fbc10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74b58d0423a58f1d server_cookie=a5a6f2a2506dbfef in_seq=0 out_seq=0 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b25ffb640 1 -- 172.21.15.3:0/3104618070 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55725ce401b0 con 0x55725d1f97f0 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b25ffb640 10 client.15984 handle_command_reply: tid=0 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 client.15984 shutdown 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 msgr2=0x55725d1f7fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d1f7fc0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3b38006010 tx=0x7f3b3804d480 comp rx=0 tx=0).stop 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 shutdown_connections 2024-02-09T22:52:22.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55725d1f97f0 0x55725d1fbc10 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55725d1917a0 0x55725d1f7fc0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55725d1889d0 0x55725d197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b25ffb640 0 client.15984 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 --2- 172.21.15.3:0/3104618070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55725d1880b0 0x55725d197230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 >> 172.21.15.3:0/3104618070 conn(0x55725d1829d0 msgr2=0x55725d18fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 shutdown_connections 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 1 -- 172.21.15.3:0/3104618070 wait complete. 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.551+0000 7f3b47a3a740 20 client.15984 trim_cache size 0 max 16384 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:52:22.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:52:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:52:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b2c0a2340 msgr2=0x7f3b2c0a4800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.558 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b2c0a2340 0x7f3b2c0a4800 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f3b38005cc0 tx=0x7f3b38005c50 comp rx=0 tx=0).stop 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 msgr2=0x7f3b4010de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010de90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3b2800d6d0 tx=0x7f3b2800db90 comp rx=0 tx=0).stop 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 shutdown_connections 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b2c0a2340 0x7f3b2c0a4800 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b401062c0 0x7f3b4010e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b40100d40 0x7f3b4010de90 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 --2- 172.21.15.3:0/679770753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b40100380 0x7f3b4010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 >> 172.21.15.3:0/679770753 conn(0x7f3b40076ef0 msgr2=0x7f3b400fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 shutdown_connections 2024-02-09T22:52:22.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:22.552+0000 7f3b44d8a640 1 -- 172.21.15.3:0/679770753 wait complete. 2024-02-09T22:52:22.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35752 13.24 MB/sec warmup 16 sec latency 2163.603 ms 2024-02-09T22:52:23.619 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35752 12.46 MB/sec warmup 17 sec latency 3163.717 ms 2024-02-09T22:52:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:23 smithi003 ceph-mon[32678]: pgmap v37: 97 pgs: 97 active+clean; 18 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-02-09T22:52:23.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:23 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:23 smithi064 ceph-mon[36638]: pgmap v37: 97 pgs: 97 active+clean; 18 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-02-09T22:52:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:23 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:23 smithi089 ceph-mon[36665]: pgmap v37: 97 pgs: 97 active+clean; 18 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 881 KiB/s rd, 1.7 MiB/s wr, 293 op/s 2024-02-09T22:52:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:23 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:24.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35752 11.77 MB/sec warmup 18 sec latency 4163.864 ms 2024-02-09T22:52:25.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35886 11.17 MB/sec warmup 19 sec latency 5002.458 ms 2024-02-09T22:52:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:26 smithi064 ceph-mon[36638]: pgmap v38: 97 pgs: 97 active+clean; 18 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:52:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:26 smithi089 ceph-mon[36665]: pgmap v38: 97 pgs: 97 active+clean; 18 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:52:26.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35895 10.61 MB/sec warmup 20 sec latency 988.923 ms 2024-02-09T22:52:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:26 smithi003 ceph-mon[32678]: pgmap v38: 97 pgs: 97 active+clean; 18 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:52:27.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35895 10.10 MB/sec warmup 21 sec latency 1989.014 ms 2024-02-09T22:52:28.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:28 smithi064 ceph-mon[36638]: pgmap v39: 97 pgs: 97 active+clean; 10 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.2 MiB/s wr, 265 op/s 2024-02-09T22:52:28.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:28 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:28 smithi089 ceph-mon[36665]: pgmap v39: 97 pgs: 97 active+clean; 10 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.2 MiB/s wr, 265 op/s 2024-02-09T22:52:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:28 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:28.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35895 9.65 MB/sec warmup 22 sec latency 2989.126 ms 2024-02-09T22:52:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:28 smithi003 ceph-mon[32678]: pgmap v39: 97 pgs: 97 active+clean; 10 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.2 MiB/s wr, 265 op/s 2024-02-09T22:52:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:28 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:29.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 35895 9.23 MB/sec warmup 23 sec latency 3989.268 ms 2024-02-09T22:52:30.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:30 smithi003 ceph-mon[32678]: pgmap v40: 97 pgs: 97 active+clean; 10 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 505 KiB/s wr, 149 op/s 2024-02-09T22:52:30.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:30 smithi064 ceph-mon[36638]: pgmap v40: 97 pgs: 97 active+clean; 10 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 505 KiB/s wr, 149 op/s 2024-02-09T22:52:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:30 smithi089 ceph-mon[36665]: pgmap v40: 97 pgs: 97 active+clean; 10 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 306 KiB/s rd, 505 KiB/s wr, 149 op/s 2024-02-09T22:52:30.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36059 8.84 MB/sec warmup 24 sec latency 4823.482 ms 2024-02-09T22:52:31.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36229 8.50 MB/sec warmup 25 sec latency 891.670 ms 2024-02-09T22:52:32.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:32 smithi064 ceph-mon[36638]: pgmap v41: 97 pgs: 97 active+clean; 10 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 522 KiB/s wr, 156 op/s 2024-02-09T22:52:32.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:32 smithi089 ceph-mon[36665]: pgmap v41: 97 pgs: 97 active+clean; 10 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 522 KiB/s wr, 156 op/s 2024-02-09T22:52:32.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36229 8.17 MB/sec warmup 26 sec latency 1891.723 ms 2024-02-09T22:52:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:32 smithi003 ceph-mon[32678]: pgmap v41: 97 pgs: 97 active+clean; 10 MiB data, 424 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 522 KiB/s wr, 156 op/s 2024-02-09T22:52:33.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:33 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:33.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:33 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:33.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36230 7.87 MB/sec warmup 27 sec latency 2319.901 ms 2024-02-09T22:52:33.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:33 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:34.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:34 smithi064 ceph-mon[36638]: pgmap v42: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 249 KiB/s wr, 122 op/s 2024-02-09T22:52:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:34 smithi089 ceph-mon[36665]: pgmap v42: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 249 KiB/s wr, 122 op/s 2024-02-09T22:52:34.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36230 7.59 MB/sec warmup 28 sec latency 1571.994 ms 2024-02-09T22:52:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:34 smithi003 ceph-mon[32678]: pgmap v42: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 249 KiB/s wr, 122 op/s 2024-02-09T22:52:35.620 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36230 7.33 MB/sec warmup 29 sec latency 2572.048 ms 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 -- 172.21.15.3:0/3827954792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 msgr2=0x7faae010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 --2- 172.21.15.3:0/3827954792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae010e090 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7faad8009ca0 tx=0x7faad802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 -- 172.21.15.3:0/3827954792 shutdown_connections 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 --2- 172.21.15.3:0/3827954792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faae0069840 0x7faae0069cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 --2- 172.21.15.3:0/3827954792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae010e090 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 --2- 172.21.15.3:0/3827954792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae0106200 0x7faae01065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 -- 172.21.15.3:0/3827954792 >> 172.21.15.3:0/3827954792 conn(0x7faae0076ef0 msgr2=0x7faae0077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 -- 172.21.15.3:0/3827954792 shutdown_connections 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 -- 172.21.15.3:0/3827954792 wait complete. 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 Processor -- start 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.652+0000 7faae86c0640 1 -- start start 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae86c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faae0069840 0x7faae01977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae86c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae0197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae86c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae0106200 0x7faae019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae86c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae019c610 con 0x7faae006a200 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae86c0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae019c780 con 0x7faae0069840 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae86c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae019c940 con 0x7faae0106200 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae0197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae0197ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47416/0 (socket says 172.21.15.3:47416) 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 -- 172.21.15.3:0/4058567002 learned_addr learned my addr 172.21.15.3:0/4058567002 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae645c640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faae0069840 0x7faae01977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae6c5d640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae0106200 0x7faae019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 -- 172.21.15.3:0/4058567002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae0106200 msgr2=0x7faae019c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae0106200 0x7faae019c010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 -- 172.21.15.3:0/4058567002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faae0069840 msgr2=0x7faae01977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faae0069840 0x7faae01977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 -- 172.21.15.3:0/4058567002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8009900 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.653+0000 7faae5c5b640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae0197ce0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faad000d950 tx=0x7faad000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.654+0000 7faad77fe640 1 -- 172.21.15.3:0/4058567002 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0009c30 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.654+0000 7faad77fe640 1 -- 172.21.15.3:0/4058567002 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faad0009d90 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.654+0000 7faad77fe640 1 -- 172.21.15.3:0/4058567002 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad000f7e0 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.654+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae019cc20 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.654+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae0100560 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faad77fe640 1 -- 172.21.15.3:0/4058567002 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7faad0010ad0 con 0x7faae006a200 2024-02-09T22:52:35.659 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 Processor -- start 2024-02-09T22:52:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 -- start start 2024-02-09T22:52:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f120b0 0x55ae36f12490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f1b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f1b7a0 0x55ae36f1db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36e80a00 con 0x55ae36f129d0 2024-02-09T22:52:35.666 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36e7df30 con 0x55ae36f120b0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faaeb36c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36e79710 con 0x55ae36f1b7a0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faae645c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f1b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faad77fe640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faacc0a23c0 0x7faacc0a4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faae645c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f1b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47424/0 (socket says 172.21.15.3:47424) 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.657+0000 7faae645c640 1 -- 172.21.15.3:0/716809943 learned_addr learned my addr 172.21.15.3:0/716809943 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae5c5b640 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f1b7a0 0x55ae36f1db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae6c5d640 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f120b0 0x55ae36f12490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae6c5d640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faacc0a23c0 0x7faacc0a4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae645c640 1 -- 172.21.15.3:0/716809943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f1b7a0 msgr2=0x55ae36f1db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae645c640 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f1b7a0 0x55ae36f1db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae645c640 1 -- 172.21.15.3:0/716809943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f120b0 msgr2=0x55ae36f12490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae645c640 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f120b0 0x55ae36f12490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae645c640 1 -- 172.21.15.3:0/716809943 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae36e3e3b0 con 0x55ae36f129d0 2024-02-09T22:52:35.667 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faad77fe640 1 -- 172.21.15.3:0/4058567002 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7faad00cde90 con 0x7faae006a200 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae6c5d640 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f120b0 0x55ae36f12490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae645c640 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f1b260 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7faadc00bcb0 tx=0x7faadc0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6cf23b0374f20f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faae6c5d640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faacc0a23c0 0x7faacc0a4880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faad8004af0 tx=0x7faad8005c50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faad4ff9640 1 -- 172.21.15.3:0/716809943 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc004020 con 0x55ae36f129d0 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faad4ff9640 1 -- 172.21.15.3:0/716809943 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faadc015070 con 0x55ae36f129d0 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.658+0000 7faad4ff9640 1 -- 172.21.15.3:0/716809943 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc004ba0 con 0x55ae36f129d0 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 -- 172.21.15.3:0/716809943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 msgr2=0x55ae36f1b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f1b260 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7faadc00bcb0 tx=0x7faadc0107a0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 -- 172.21.15.3:0/716809943 shutdown_connections 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f1b7a0 0x55ae36f1db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f1b260 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 --2- 172.21.15.3:0/716809943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f120b0 0x55ae36f12490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 -- 172.21.15.3:0/716809943 >> 172.21.15.3:0/716809943 conn(0x55ae36f0c9d0 msgr2=0x55ae36f0cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:35.668 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 -- 172.21.15.3:0/716809943 shutdown_connections 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.659+0000 7faaeb36c740 1 -- 172.21.15.3:0/716809943 wait complete. 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 -- start start 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 Processor -- start 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f187b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f18cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f19230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36e1eb90 con 0x55ae36f129d0 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36b4d250 con 0x55ae36f1b7a0 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36dff200 con 0x55ae36f120b0 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faaeb36c740 10 client.0 _ll_register_callbacks cb 0x55ae36e355d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f18cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f18cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47426/0 (socket says 172.21.15.3:47426) 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 -- 172.21.15.3:0/1334574884 learned_addr learned my addr 172.21.15.3:0/1334574884 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae6c5d640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f19230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.669 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae645c640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f187b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 -- 172.21.15.3:0/1334574884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 msgr2=0x55ae36f187b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f187b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 -- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 msgr2=0x55ae36f19230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f19230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae5c5b640 1 -- 172.21.15.3:0/1334574884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faadc00b8e0 con 0x55ae36f129d0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae6c5d640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f19230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.660+0000 7faae645c640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f187b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faae5c5b640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f18cf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7faad0096b70 tx=0x7faad0005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a62a78b5c487c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faac9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faac9ffb640 1 -- 172.21.15.3:0/1334574884 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad009e7b0 con 0x55ae36f129d0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faac9ffb640 1 -- 172.21.15.3:0/1334574884 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7faad00a2e00 con 0x55ae36f129d0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ae36e9fee0 con 0x55ae36f129d0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faac9ffb640 1 -- 172.21.15.3:0/1334574884 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad00ba070 con 0x55ae36f129d0 2024-02-09T22:52:35.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 10 client.15996 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ae36e77d90 con 0x55ae36f129d0 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faac9ffb640 1 -- 172.21.15.3:0/1334574884 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ae36e9fee0 con 0x55ae36f129d0 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faac9ffb640 1 -- 172.21.15.3:0/1334574884 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7faad0099b60 con 0x55ae36f129d0 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 10 client.15996 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 10 client.15996 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 20 client.15996 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f2ba50 0x55ae36f2deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 4 client.15996 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.661+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ae36b79600 con 0x55ae36f2ba50 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.662+0000 7faae645c640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f2ba50 0x55ae36f2deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.662+0000 7faae645c640 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f2ba50 0x55ae36f2deb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4eaeef5843480d3 server_cookie=4760211152b691b9 in_seq=0 out_seq=0 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.662+0000 7faac9ffb640 10 client.15996 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faac9ffb640 1 -- 172.21.15.3:0/1334574884 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ae36b79600 con 0x55ae36f2ba50 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faac9ffb640 10 client.15996 handle_command_reply: tid=0 2024-02-09T22:52:35.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 client.15996 shutdown 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 msgr2=0x55ae36f18cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f18cf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7faad0096b70 tx=0x7faad0005bf0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 shutdown_connections 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f2ba50 0x55ae36f2deb0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f19230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f18cf0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faac9ffb640 0 client.15996 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1334574884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f187b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 >> 172.21.15.3:0/1334574884 conn(0x55ae36f0c9d0 msgr2=0x55ae36f0e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 shutdown_connections 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 1 -- 172.21.15.3:0/1334574884 wait complete. 2024-02-09T22:52:35.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.663+0000 7faaeb36c740 20 client.15996 trim_cache size 0 max 16384 2024-02-09T22:52:35.683 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 Processor -- start 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 -- start start 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f1aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f129d0 0x55ae36f19050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 0x55ae36f15a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f2a1d0 con 0x55ae36f1b7a0 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f0c1b0 con 0x55ae36f129d0 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.676+0000 7faaeb36c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f0c530 con 0x55ae36f120b0 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 0x55ae36f15a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 0x55ae36f15a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47442/0 (socket says 172.21.15.3:47442) 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 -- 172.21.15.3:0/3140939168 learned_addr learned my addr 172.21.15.3:0/3140939168 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 -- 172.21.15.3:0/3140939168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 msgr2=0x55ae36f1aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f1aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 -- 172.21.15.3:0/3140939168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f129d0 msgr2=0x55ae36f19050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f129d0 0x55ae36f19050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 -- 172.21.15.3:0/3140939168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ae36e14820 con 0x55ae36f1b7a0 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faae6c5d640 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 0x55ae36f15a30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7faad8009ca0 tx=0x7faad803f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c38ee156a4a8360 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faacbfff640 1 -- 172.21.15.3:0/3140939168 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8034d50 con 0x55ae36f1b7a0 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faacbfff640 1 -- 172.21.15.3:0/3140939168 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faad8003970 con 0x55ae36f1b7a0 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.677+0000 7faacbfff640 1 -- 172.21.15.3:0/3140939168 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8004440 con 0x55ae36f1b7a0 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- 172.21.15.3:0/3140939168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 msgr2=0x55ae36f15a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 0x55ae36f15a30 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7faad8009ca0 tx=0x7faad803f040 comp rx=0 tx=0).stop 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- 172.21.15.3:0/3140939168 shutdown_connections 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f129d0 0x55ae36f19050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f1b7a0 0x55ae36f15a30 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- 172.21.15.3:0/3140939168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f1aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- 172.21.15.3:0/3140939168 >> 172.21.15.3:0/3140939168 conn(0x55ae36f0c9d0 msgr2=0x55ae36f17080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- 172.21.15.3:0/3140939168 shutdown_connections 2024-02-09T22:52:35.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- 172.21.15.3:0/3140939168 wait complete. 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- start start 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 Processor -- start 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f214e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f21a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f81fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f22140 con 0x55ae36f129d0 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f82520 con 0x55ae36f1b7a0 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ae36f82690 con 0x55ae36f120b0 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.678+0000 7faaeb36c740 10 client.0 _ll_register_callbacks cb 0x7faad0010d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f21a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f21a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47452/0 (socket says 172.21.15.3:47452) 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 -- 172.21.15.3:0/1021313976 learned_addr learned my addr 172.21.15.3:0/1021313976 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae645c640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f214e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae6c5d640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f81fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 -- 172.21.15.3:0/1021313976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 msgr2=0x55ae36f214e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f214e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 -- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 msgr2=0x55ae36f81fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f81fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 -- 172.21.15.3:0/1021313976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faad8045070 con 0x55ae36f129d0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae645c640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f214e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faae5c5b640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f21a20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7faad009a410 tx=0x7faad00969b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f362b75fde62ed8a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faac9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faac9ffb640 1 -- 172.21.15.3:0/1021313976 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad0014020 con 0x55ae36f129d0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faac9ffb640 1 -- 172.21.15.3:0/1021313976 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7faad00b2070 con 0x55ae36f129d0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faadc015070 con 0x55ae36f129d0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faac9ffb640 1 -- 172.21.15.3:0/1021313976 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7faad00cb3d0 con 0x55ae36f129d0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faaeb36c740 10 client.16002 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.679+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ae36e3a6f0 con 0x55ae36f129d0 2024-02-09T22:52:35.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faac9ffb640 1 -- 172.21.15.3:0/1021313976 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faadc015070 con 0x55ae36f129d0 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faac9ffb640 1 -- 172.21.15.3:0/1021313976 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7faad007f070 con 0x55ae36f129d0 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faaeb36c740 10 client.16002 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faaeb36c740 10 client.16002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faaeb36c740 20 client.16002 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f83b70 0x55ae36f85f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faaeb36c740 4 client.16002 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ae36bca1b0 con 0x55ae36f83b70 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faae645c640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f83b70 0x55ae36f85f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faac9ffb640 10 client.16002 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:35.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.680+0000 7faae645c640 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f83b70 0x55ae36f85f90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a231dc98664e3f7f server_cookie=513851b84c1a62be in_seq=0 out_seq=0 2024-02-09T22:52:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faac9ffb640 1 -- 172.21.15.3:0/1021313976 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ae36bca1b0 con 0x55ae36f83b70 2024-02-09T22:52:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faac9ffb640 10 client.16002 handle_command_reply: tid=0 2024-02-09T22:52:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 client.16002 shutdown 2024-02-09T22:52:35.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 msgr2=0x55ae36f21a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f21a20 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7faad009a410 tx=0x7faad00969b0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 shutdown_connections 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ae36f83b70 0x55ae36f85f90 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ae36f1b7a0 0x55ae36f81fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ae36f129d0 0x55ae36f21a20 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faac9ffb640 0 client.16002 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 --2- 172.21.15.3:0/1021313976 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ae36f120b0 0x55ae36f214e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 >> 172.21.15.3:0/1021313976 conn(0x55ae36f0c9d0 msgr2=0x55ae36f0b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 shutdown_connections 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 1 -- 172.21.15.3:0/1021313976 wait complete. 2024-02-09T22:52:35.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.708+0000 7faaeb36c740 20 client.16002 trim_cache size 0 max 16384 2024-02-09T22:52:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:52:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:52:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:52:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:52:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faacc0a23c0 msgr2=0x7faacc0a4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faacc0a23c0 0x7faacc0a4880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7faad8004af0 tx=0x7faad8005c50 comp rx=0 tx=0).stop 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 msgr2=0x7faae0197ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae0197ce0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7faad000d950 tx=0x7faad000de10 comp rx=0 tx=0).stop 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 shutdown_connections 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faacc0a23c0 0x7faacc0a4880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faae0106200 0x7faae019c010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faae006a200 0x7faae0197ce0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 --2- 172.21.15.3:0/4058567002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faae0069840 0x7faae01977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 >> 172.21.15.3:0/4058567002 conn(0x7faae0076ef0 msgr2=0x7faae00fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 shutdown_connections 2024-02-09T22:52:35.714 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:35.709+0000 7faae86c0640 1 -- 172.21.15.3:0/4058567002 wait complete. 2024-02-09T22:52:36.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:36 smithi064 ceph-mon[36638]: pgmap v43: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 82 KiB/s wr, 101 op/s 2024-02-09T22:52:36.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:36 smithi089 ceph-mon[36665]: pgmap v43: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 82 KiB/s wr, 101 op/s 2024-02-09T22:52:36.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36230 7.08 MB/sec warmup 30 sec latency 3572.117 ms 2024-02-09T22:52:36.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:36 smithi003 ceph-mon[32678]: pgmap v43: 97 pgs: 97 active+clean; 10 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 82 KiB/s wr, 101 op/s 2024-02-09T22:52:37.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36230 6.85 MB/sec warmup 31 sec latency 4572.234 ms 2024-02-09T22:52:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:38 smithi064 ceph-mon[36638]: pgmap v44: 97 pgs: 97 active+clean; 26 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-02-09T22:52:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:38 smithi089 ceph-mon[36665]: pgmap v44: 97 pgs: 97 active+clean; 26 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-02-09T22:52:38.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36421 6.65 MB/sec warmup 32 sec latency 4993.509 ms 2024-02-09T22:52:38.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:38 smithi003 ceph-mon[32678]: pgmap v44: 97 pgs: 97 active+clean; 26 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.1 MiB/s wr, 204 op/s 2024-02-09T22:52:39.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36421 6.45 MB/sec warmup 33 sec latency 1460.995 ms 2024-02-09T22:52:40.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:40 smithi064 ceph-mon[36638]: pgmap v45: 97 pgs: 97 active+clean; 26 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 4.1 MiB/s wr, 113 op/s 2024-02-09T22:52:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:40 smithi089 ceph-mon[36665]: pgmap v45: 97 pgs: 97 active+clean; 26 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 4.1 MiB/s wr, 113 op/s 2024-02-09T22:52:40.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36421 6.26 MB/sec warmup 34 sec latency 2461.100 ms 2024-02-09T22:52:40.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:40 smithi003 ceph-mon[32678]: pgmap v45: 97 pgs: 97 active+clean; 26 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 4.1 MiB/s wr, 113 op/s 2024-02-09T22:52:41.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36421 6.08 MB/sec warmup 35 sec latency 3461.218 ms 2024-02-09T22:52:42.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:52:42.091 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:52:42.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 -- 172.21.15.3:0/741071126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c101440 msgr2=0x7f556c101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 --2- 172.21.15.3:0/741071126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c101440 0x7f556c101820 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f555c009930 tx=0x7f555c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 -- 172.21.15.3:0/741071126 shutdown_connections 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 --2- 172.21.15.3:0/741071126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f556c101df0 0x7f556c105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 --2- 172.21.15.3:0/741071126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f556c106510 0x7f556c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 --2- 172.21.15.3:0/741071126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c101440 0x7f556c101820 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 -- 172.21.15.3:0/741071126 >> 172.21.15.3:0/741071126 conn(0x7f556c0fccc0 msgr2=0x7f556c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 -- 172.21.15.3:0/741071126 shutdown_connections 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.207+0000 7f55724c0640 1 -- 172.21.15.3:0/741071126 wait complete. 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 Processor -- start 2024-02-09T22:52:42.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 -- start start 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f556c101440 0x7f556c19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f556c106510 0x7f556c19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 0x7f556c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c198210 con 0x7f556c197740 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c198380 con 0x7f556c101440 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f55724c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556c198540 con 0x7f556c106510 2024-02-09T22:52:42.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f5570a5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 0x7f556c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f5570a5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 0x7f556c197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40996/0 (socket says 172.21.15.3:40996) 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f5570a5d640 1 -- 172.21.15.3:0/2951958605 learned_addr learned my addr 172.21.15.3:0/2951958605 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f556c106510 0x7f556c19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.208+0000 7f556bfff640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f556c101440 0x7f556c19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f5570a5d640 1 -- 172.21.15.3:0/2951958605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f556c106510 msgr2=0x7f556c19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f5570a5d640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f556c106510 0x7f556c19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f5570a5d640 1 -- 172.21.15.3:0/2951958605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f556c101440 msgr2=0x7f556c19d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f5570a5d640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f556c101440 0x7f556c19d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f5570a5d640 1 -- 172.21.15.3:0/2951958605 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c009560 con 0x7f556c197740 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f5570a5d640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 0x7f556c197bc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f556400eb10 tx=0x7f556400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f55697fa640 1 -- 172.21.15.3:0/2951958605 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564004b20 con 0x7f556c197740 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f55697fa640 1 -- 172.21.15.3:0/2951958605 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5564004c80 con 0x7f556c197740 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c1a4490 con 0x7f556c197740 2024-02-09T22:52:42.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f55697fa640 1 -- 172.21.15.3:0/2951958605 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564010650 con 0x7f556c197740 2024-02-09T22:52:42.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.209+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c1a49c0 con 0x7f556c197740 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.211+0000 7f55697fa640 1 -- 172.21.15.3:0/2951958605 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f5564005020 con 0x7f556c197740 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.211+0000 7f557516e740 1 Processor -- start 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 -- start start 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f55697fa640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f55540a23d0 0x7f55540a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x56132618bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326194d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x561326195770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561326146d00 con 0x561326195310 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613260fb2f0 con 0x56132618b860 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f557516e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613260f8820 con 0x56132618c210 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f55540a23d0 0x7f55540a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x561326195770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x561326195770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40998/0 (socket says 172.21.15.3:40998) 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f55697fa640 1 -- 172.21.15.3:0/2951958605 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5564014070 con 0x7f556c197740 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 -- 172.21.15.3:0/78798466 learned_addr learned my addr 172.21.15.3:0/78798466 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f556bfff640 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x56132618bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f556b7fe640 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326194d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 -- 172.21.15.3:0/78798466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 msgr2=0x561326194d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326194d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 -- 172.21.15.3:0/78798466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 msgr2=0x56132618bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x56132618bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f5570a5d640 1 -- 172.21.15.3:0/78798466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5613260b5ff0 con 0x561326195310 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.212+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f55540a23d0 0x7f55540a4890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f556c198f40 tx=0x7f55600127c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f5570a5d640 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x561326195770 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f55640049d0 tx=0x7f556400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f09ed596d6ee83 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f55527fc640 1 -- 172.21.15.3:0/78798466 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5564052ea0 con 0x561326195310 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f55527fc640 1 -- 172.21.15.3:0/78798466 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5564078750 con 0x561326195310 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f55527fc640 1 -- 172.21.15.3:0/78798466 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556409afe0 con 0x561326195310 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 -- 172.21.15.3:0/78798466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 msgr2=0x561326195770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x561326195770 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f55640049d0 tx=0x7f556400cf70 comp rx=0 tx=0).stop 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 -- 172.21.15.3:0/78798466 shutdown_connections 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x561326195770 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326194d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 --2- 172.21.15.3:0/78798466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x56132618bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.213+0000 7f557516e740 1 -- 172.21.15.3:0/78798466 >> 172.21.15.3:0/78798466 conn(0x561326186050 msgr2=0x561326186430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 -- 172.21.15.3:0/78798466 shutdown_connections 2024-02-09T22:52:42.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 -- 172.21.15.3:0/78798466 wait complete. 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 -- start start 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 Processor -- start 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 0x56132619b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326198bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561326195310 0x561326199310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5613260f4000 con 0x56132618b860 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561325dc7430 con 0x561326195310 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561325d83e00 con 0x56132618c210 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f557516e740 10 client.0 _ll_register_callbacks cb 0x5613260e87f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f556bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 0x56132619b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f556bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 0x56132619b840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41010/0 (socket says 172.21.15.3:41010) 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.214+0000 7f556bfff640 1 -- 172.21.15.3:0/2399088553 learned_addr learned my addr 172.21.15.3:0/2399088553 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f5570a5d640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561326195310 0x561326199310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556bfff640 1 -- 172.21.15.3:0/2399088553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 msgr2=0x561326198bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326198bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556bfff640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326198bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556bfff640 1 -- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561326195310 msgr2=0x561326199310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556bfff640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561326195310 0x561326199310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556bfff640 1 -- 172.21.15.3:0/2399088553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c1a49c0 con 0x56132618b860 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f5570a5d640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561326195310 0x561326199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f556bfff640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 0x56132619b840 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f555c009900 tx=0x7f555c02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1466f370145504f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f554b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f554b7fe640 1 -- 172.21.15.3:0/2399088553 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c004040 con 0x56132618b860 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f554b7fe640 1 -- 172.21.15.3:0/2399088553 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f555c0041a0 con 0x56132618b860 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561325df37e0 con 0x56132618b860 2024-02-09T22:52:42.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f554b7fe640 1 -- 172.21.15.3:0/2399088553 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f555c004360 con 0x56132618b860 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f557516e740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5613260f2680 con 0x56132618b860 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.215+0000 7f554b7fe640 1 -- 172.21.15.3:0/2399088553 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561325df37e0 con 0x56132618b860 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f554b7fe640 1 -- 172.21.15.3:0/2399088553 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f555c03c040 con 0x56132618b860 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f557516e740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f557516e740 10 client.16014 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f557516e740 20 client.16014 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f557516e740 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5613261a9670 0x5613261abab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f557516e740 4 client.16014 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561325e30720 con 0x5613261a9670 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5613261a9670 0x5613261abab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f556b7fe640 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5613261a9670 0x5613261abab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47ac0aca4141cc81 server_cookie=9afdc99bab89859f in_seq=0 out_seq=0 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.216+0000 7f554b7fe640 10 client.16014 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:42.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f554b7fe640 1 -- 172.21.15.3:0/2399088553 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561325e30720 con 0x5613261a9670 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f554b7fe640 10 client.16014 handle_command_reply: tid=0 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 client.16014 shutdown 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 msgr2=0x56132619b840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 0x56132619b840 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f555c009900 tx=0x7f555c02c740 comp rx=0 tx=0).stop 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 shutdown_connections 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5613261a9670 0x5613261abab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561326195310 0x561326199310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x561326198bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f554b7fe640 0 client.16014 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 --2- 172.21.15.3:0/2399088553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618b860 0x56132619b840 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 >> 172.21.15.3:0/2399088553 conn(0x561326186050 msgr2=0x561326187da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 shutdown_connections 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 1 -- 172.21.15.3:0/2399088553 wait complete. 2024-02-09T22:52:42.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.217+0000 7f557516e740 20 client.16014 trim_cache size 0 max 16384 2024-02-09T22:52:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.232+0000 7f557516e740 1 Processor -- start 2024-02-09T22:52:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.232+0000 7f557516e740 1 -- start start 2024-02-09T22:52:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f557516e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x561326196720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f557516e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x5613261a0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f557516e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x5613261a0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f557516e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561326181780 con 0x561326195310 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f557516e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561326078fd0 con 0x56132618b860 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f557516e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561326185b80 con 0x56132618c210 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x5613261a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x5613261a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41018/0 (socket says 172.21.15.3:41018) 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 -- 172.21.15.3:0/1042617365 learned_addr learned my addr 172.21.15.3:0/1042617365 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f556bfff640 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x561326196720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 -- 172.21.15.3:0/1042617365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 msgr2=0x5613261a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x5613261a0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 -- 172.21.15.3:0/1042617365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 msgr2=0x561326196720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x561326196720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 -- 172.21.15.3:0/1042617365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56132608eb30 con 0x561326195310 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f556bfff640 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x561326196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5570a5d640 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x5613261a0890 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f55640788c0 tx=0x7f5564096bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda544efc41041a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5551ffb640 1 -- 172.21.15.3:0/1042617365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55640cbcc0 con 0x561326195310 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5551ffb640 1 -- 172.21.15.3:0/1042617365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f556409b650 con 0x561326195310 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.233+0000 7f5551ffb640 1 -- 172.21.15.3:0/1042617365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55640107d0 con 0x561326195310 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 -- 172.21.15.3:0/1042617365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 msgr2=0x5613261a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x5613261a0890 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f55640788c0 tx=0x7f5564096bc0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 -- 172.21.15.3:0/1042617365 shutdown_connections 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561326195310 0x5613261a0890 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56132618c210 0x5613261a0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 --2- 172.21.15.3:0/1042617365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x561326196720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 -- 172.21.15.3:0/1042617365 >> 172.21.15.3:0/1042617365 conn(0x561326186050 msgr2=0x5613261b0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 -- 172.21.15.3:0/1042617365 shutdown_connections 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 -- 172.21.15.3:0/1042617365 wait complete. 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.234+0000 7f557516e740 1 -- start start 2024-02-09T22:52:42.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 Processor -- start 2024-02-09T22:52:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x5613261a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 0x56132619e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561326195310 0x56132619ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56132619f3b0 con 0x56132618c210 2024-02-09T22:52:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56132619f520 con 0x56132618b860 2024-02-09T22:52:42.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56132619f6e0 con 0x561326195310 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f557516e740 10 client.0 _ll_register_callbacks cb 0x7f555c005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 0x56132619e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 0x56132619e930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41032/0 (socket says 172.21.15.3:41032) 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 -- 172.21.15.3:0/920168944 learned_addr learned my addr 172.21.15.3:0/920168944 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556bfff640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x5613261a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f5570a5d640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561326195310 0x56132619ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 -- 172.21.15.3:0/920168944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561326195310 msgr2=0x56132619ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561326195310 0x56132619ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 -- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 msgr2=0x5613261a0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x5613261a0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 -- 172.21.15.3:0/920168944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5613260b5ff0 con 0x56132618c210 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556bfff640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x5613261a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:42.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.235+0000 7f556b7fe640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 0x56132619e930 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f556001e570 tx=0x7f55600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2013a04d50b46175 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f554b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f554b7fe640 1 -- 172.21.15.3:0/920168944 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560024040 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f554b7fe640 1 -- 172.21.15.3:0/920168944 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f556001ecb0 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5564078750 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f554b7fe640 1 -- 172.21.15.3:0/920168944 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5560017070 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 10 client.16020 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5613260b4850 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f554b7fe640 1 -- 172.21.15.3:0/920168944 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5564078750 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f554b7fe640 1 -- 172.21.15.3:0/920168944 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f5560027050 con 0x56132618c210 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 10 client.16020 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 10 client.16020 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 20 client.16020 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561326201280 0x5613262036a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 4 client.16020 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:52:42.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561325e7f890 con 0x561326201280 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.236+0000 7f556bfff640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561326201280 0x5613262036a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f556bfff640 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561326201280 0x5613262036a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d168d5ba2daf5d0e server_cookie=ab8518d6b03480d in_seq=0 out_seq=0 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f554b7fe640 10 client.16020 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f554b7fe640 1 -- 172.21.15.3:0/920168944 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561325e7f890 con 0x561326201280 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f554b7fe640 10 client.16020 handle_command_reply: tid=0 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 client.16020 shutdown 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 msgr2=0x56132619e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 0x56132619e930 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f556001e570 tx=0x7f55600046c0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 shutdown_connections 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561326201280 0x5613262036a0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561326195310 0x56132619ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56132618c210 0x56132619e930 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f554b7fe640 0 client.16020 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:42.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 --2- 172.21.15.3:0/920168944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56132618b860 0x5613261a0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 >> 172.21.15.3:0/920168944 conn(0x561326186050 msgr2=0x561326198460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 shutdown_connections 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 1 -- 172.21.15.3:0/920168944 wait complete. 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.237+0000 7f557516e740 20 client.16020 trim_cache size 0 max 16384 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:52:42.247 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:52:42.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f55540a23d0 msgr2=0x7f55540a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f55540a23d0 0x7f55540a4890 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f556c198f40 tx=0x7f55600127c0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 msgr2=0x7f556c197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 0x7f556c197bc0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f556400eb10 tx=0x7f556400c2d0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 shutdown_connections 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f55540a23d0 0x7f55540a4890 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f556c197740 0x7f556c197bc0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f556c106510 0x7f556c19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 --2- 172.21.15.3:0/2951958605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f556c101440 0x7f556c19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 >> 172.21.15.3:0/2951958605 conn(0x7f556c0fccc0 msgr2=0x7f556c0fe0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 shutdown_connections 2024-02-09T22:52:42.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.238+0000 7f55724c0640 1 -- 172.21.15.3:0/2951958605 wait complete. 2024-02-09T22:52:42.252 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:52:42.252 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 -- 172.21.15.3:0/1527989050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c100ce0 msgr2=0x7f954c104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 --2- 172.21.15.3:0/1527989050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c100ce0 0x7f954c104d20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f9540009900 tx=0x7f954002bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 -- 172.21.15.3:0/1527989050 shutdown_connections 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 --2- 172.21.15.3:0/1527989050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c105400 0x7f954c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 --2- 172.21.15.3:0/1527989050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c100ce0 0x7f954c104d20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 --2- 172.21.15.3:0/1527989050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f954c100330 0x7f954c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 -- 172.21.15.3:0/1527989050 >> 172.21.15.3:0/1527989050 conn(0x7f954c0fbbb0 msgr2=0x7f954c0fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.352+0000 7f95543bc640 1 -- 172.21.15.3:0/1527989050 shutdown_connections 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 -- 172.21.15.3:0/1527989050 wait complete. 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 Processor -- start 2024-02-09T22:52:42.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 -- start start 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c100330 0x7f954c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 0x7f954c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f954c1976d0 0x7f954c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1981a0 con 0x7f954c105400 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c198310 con 0x7f954c1976d0 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f95543bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c1984d0 con 0x7f954c100330 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f9551957640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 0x7f954c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f9551957640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 0x7f954c19db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41054/0 (socket says 172.21.15.3:41054) 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f9551957640 1 -- 172.21.15.3:0/1985396075 learned_addr learned my addr 172.21.15.3:0/1985396075 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.353+0000 7f9552158640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c100330 0x7f954c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9552959640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f954c1976d0 0x7f954c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9551957640 1 -- 172.21.15.3:0/1985396075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c100330 msgr2=0x7f954c19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9551957640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c100330 0x7f954c19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9551957640 1 -- 172.21.15.3:0/1985396075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f954c1976d0 msgr2=0x7f954c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9551957640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f954c1976d0 0x7f954c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9551957640 1 -- 172.21.15.3:0/1985396075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540009560 con 0x7f954c105400 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9552158640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c100330 0x7f954c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f9551957640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 0x7f954c19db00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9540036040 tx=0x7f9540004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540004490 con 0x7f954c105400 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f954002cbc0 con 0x7f954c105400 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c1a43c0 con 0x7f954c105400 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540003e80 con 0x7f954c105400 2024-02-09T22:52:42.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.354+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c1a48f0 con 0x7f954c105400 2024-02-09T22:52:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.355+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c101800 con 0x7f954c105400 2024-02-09T22:52:42.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.356+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f95400035f0 con 0x7f954c105400 2024-02-09T22:52:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.357+0000 7f954b7fe640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95340a23d0 0x7f95340a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.357+0000 7f9552158640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95340a23d0 0x7f95340a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.357+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f95400f0ab0 con 0x7f954c105400 2024-02-09T22:52:42.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.357+0000 7f9552158640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95340a23d0 0x7f95340a4890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95440058c0 tx=0x7f9544005850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.361+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f95400b95d0 con 0x7f954c105400 2024-02-09T22:52:42.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:42 smithi003 ceph-mon[32678]: pgmap v46: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 4.2 MiB/s wr, 130 op/s 2024-02-09T22:52:42.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:42 smithi064 ceph-mon[36638]: pgmap v46: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 4.2 MiB/s wr, 130 op/s 2024-02-09T22:52:42.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:42 smithi089 ceph-mon[36665]: pgmap v46: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 4.2 MiB/s wr, 130 op/s 2024-02-09T22:52:42.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.581+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f954c108730 con 0x7f954c105400 2024-02-09T22:52:42.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.582+0000 7f954b7fe640 1 -- 172.21.15.3:0/1985396075 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+11368 (secure 0 0 0) 0x7f95400bcc90 con 0x7f954c105400 2024-02-09T22:52:42.585 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:52:42.585 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":29,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:52:31.662567+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:52:42.585 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 29 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95340a23d0 msgr2=0x7f95340a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95340a23d0 0x7f95340a4890 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f95440058c0 tx=0x7f9544005850 comp rx=0 tx=0).stop 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 msgr2=0x7f954c19db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 0x7f954c19db00 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f9540036040 tx=0x7f9540004270 comp rx=0 tx=0).stop 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 shutdown_connections 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95340a23d0 0x7f95340a4890 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f954c1976d0 0x7f954c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f954c105400 0x7f954c19db00 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 --2- 172.21.15.3:0/1985396075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f954c100330 0x7f954c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 >> 172.21.15.3:0/1985396075 conn(0x7f954c0fbbb0 msgr2=0x7f954c0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 shutdown_connections 2024-02-09T22:52:42.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.584+0000 7f95543bc640 1 -- 172.21.15.3:0/1985396075 wait complete. 2024-02-09T22:52:42.598 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:42.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:52:42.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36421 5.91 MB/sec warmup 36 sec latency 4461.264 ms 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- 172.21.15.3:0/533182730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18101440 msgr2=0x7f3c18101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/533182730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18101440 0x7f3c18101820 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c009930 tx=0x7f3c0c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- 172.21.15.3:0/533182730 shutdown_connections 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/533182730 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c18101df0 0x7f3c18105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/533182730 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c18106510 0x7f3c1810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/533182730 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18101440 0x7f3c18101820 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- 172.21.15.3:0/533182730 >> 172.21.15.3:0/533182730 conn(0x7f3c180fccc0 msgr2=0x7f3c180ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- 172.21.15.3:0/533182730 shutdown_connections 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- 172.21.15.3:0/533182730 wait complete. 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 Processor -- start 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- start start 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c18101440 0x7f3c1819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c18106510 0x7f3c1819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 0x7f3c18197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c18198260 con 0x7f3c18197790 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c181983d0 con 0x7f3c18106510 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.700+0000 7f3c1f00e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c18198590 con 0x7f3c18101440 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 0x7f3c18197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 0x7f3c18197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41086/0 (socket says 172.21.15.3:41086) 2024-02-09T22:52:42.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1958511316 learned_addr learned my addr 172.21.15.3:0/1958511316 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c18101440 0x7f3c1819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c18106510 0x7f3c1819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1958511316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c18101440 msgr2=0x7f3c1819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c18101440 0x7f3c1819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1958511316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c18106510 msgr2=0x7f3c1819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c18106510 0x7f3c1819dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1958511316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c009560 con 0x7f3c18197790 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 0x7f3c18197c10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000eaa0 tx=0x7f3c1000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c15ffb640 1 -- 172.21.15.3:0/1958511316 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1000cdb0 con 0x7f3c18197790 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c15ffb640 1 -- 172.21.15.3:0/1958511316 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c10005020 con 0x7f3c18197790 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c181a44e0 con 0x7f3c18197790 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c15ffb640 1 -- 172.21.15.3:0/1958511316 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c100116b0 con 0x7f3c18197790 2024-02-09T22:52:42.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.701+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c181a49c0 con 0x7f3c18197790 2024-02-09T22:52:42.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.703+0000 7f3c15ffb640 1 -- 172.21.15.3:0/1958511316 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f3c10005520 con 0x7f3c18197790 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 Processor -- start 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 -- start start 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c15ffb640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3c080a23d0 0x7f3c080a4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a5ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a65990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14a683b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14a18020 con 0x55fe14a65f60 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe149cb320 con 0x55fe14a5d3e0 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe149c8850 con 0x55fe14a5ca30 2024-02-09T22:52:42.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3c080a23d0 0x7f3c080a4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14a683b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14a683b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41092/0 (socket says 172.21.15.3:41092) 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1305674209 learned_addr learned my addr 172.21.15.3:0/1305674209 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c15ffb640 1 -- 172.21.15.3:0/1958511316 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3c100ce580 con 0x7f3c18197790 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a65990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a5ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1305674209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 msgr2=0x55fe14a5ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a5ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1305674209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 msgr2=0x55fe14a65990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a65990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1305674209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe14986210 con 0x55fe14a65f60 2024-02-09T22:52:42.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.704+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a5ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3c080a23d0 0x7f3c080a4890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c04005fd0 tx=0x7f3c040157f0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14a683b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c100a2f60 tx=0x7f3c100970b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e02bd35339d82d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c02ffd640 1 -- 172.21.15.3:0/1305674209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c1009b3f0 con 0x55fe14a65f60 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c02ffd640 1 -- 172.21.15.3:0/1305674209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c1009acf0 con 0x55fe14a65f60 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c02ffd640 1 -- 172.21.15.3:0/1305674209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c10099080 con 0x55fe14a65f60 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1305674209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 msgr2=0x55fe14a683b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14a683b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f3c100a2f60 tx=0x7f3c100970b0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1305674209 shutdown_connections 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a65990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14a683b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1305674209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a5ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1305674209 >> 172.21.15.3:0/1305674209 conn(0x55fe14a57100 msgr2=0x55fe14a574e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.705+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1305674209 shutdown_connections 2024-02-09T22:52:42.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1305674209 wait complete. 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 -- start start 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 Processor -- start 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a62fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a63a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe149c4030 con 0x55fe14a5d3e0 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14697430 con 0x55fe14a65f60 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14653e00 con 0x55fe14a5ca30 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c21bdd740 10 client.0 _ll_register_callbacks cb 0x55fe149b9030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41098/0 (socket says 172.21.15.3:41098) 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.706+0000 7f3c17fff640 1 -- 172.21.15.3:0/1483798561 learned_addr learned my addr 172.21.15.3:0/1483798561 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a63a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a62fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c17fff640 1 -- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 msgr2=0x55fe14a62fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a62fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c17fff640 1 -- 172.21.15.3:0/1483798561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 msgr2=0x55fe14a63a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a63a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c17fff640 1 -- 172.21.15.3:0/1483798561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c181a49c0 con 0x55fe14a5d3e0 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a63a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63520 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c1806a810 tx=0x7f3c04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0ed440b2e90969 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3bf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1483798561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04025070 con 0x55fe14a5d3e0 2024-02-09T22:52:42.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1483798561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3c04005020 con 0x55fe14a5d3e0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe146c37e0 con 0x55fe14a5d3e0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1483798561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c040051e0 con 0x55fe14a5d3e0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c21bdd740 10 client.16038 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fe149c26b0 con 0x55fe14a5d3e0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.707+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1483798561 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fe146c37e0 con 0x55fe14a5d3e0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1483798561 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3c04028030 con 0x55fe14a5d3e0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c21bdd740 10 client.16038 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c21bdd740 10 client.16038 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c21bdd740 20 client.16038 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14a764b0 0x55fe14a788d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c21bdd740 4 client.16038 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe14700720 con 0x55fe14a764b0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14a764b0 0x55fe14a788d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14a764b0 0x55fe14a788d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6763bc936c55af6f server_cookie=a2d9686953489d95 in_seq=0 out_seq=0 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.708+0000 7f3bf7fff640 10 client.16038 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:42.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1483798561 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fe14700720 con 0x55fe14a764b0 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3bf7fff640 10 client.16038 handle_command_reply: tid=0 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 client.16038 shutdown 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 msgr2=0x55fe14a63520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63520 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f3c1806a810 tx=0x7f3c04004730 comp rx=0 tx=0).stop 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 shutdown_connections 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14a764b0 0x55fe14a788d0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a63a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63520 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3bf7fff640 0 client.16038 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1483798561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a62fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 >> 172.21.15.3:0/1483798561 conn(0x55fe14a57100 msgr2=0x55fe14a58e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 shutdown_connections 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1483798561 wait complete. 2024-02-09T22:52:42.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.709+0000 7f3c21bdd740 20 client.16038 trim_cache size 0 max 16384 2024-02-09T22:52:42.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 Processor -- start 2024-02-09T22:52:42.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 -- start start 2024-02-09T22:52:42.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a65710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a60240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14949130 con 0x55fe14a5d3e0 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14a7dae0 con 0x55fe14a65f60 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14a5de30 con 0x55fe14a5ca30 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c17fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41102/0 (socket says 172.21.15.3:41102) 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.721+0000 7f3c17fff640 1 -- 172.21.15.3:0/383233827 learned_addr learned my addr 172.21.15.3:0/383233827 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c17fff640 1 -- 172.21.15.3:0/383233827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 msgr2=0x55fe14a65710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c17fff640 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a65710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c17fff640 1 -- 172.21.15.3:0/383233827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 msgr2=0x55fe14a60240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c17fff640 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a60240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c17fff640 1 -- 172.21.15.3:0/383233827 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe1495ec90 con 0x55fe14a5d3e0 2024-02-09T22:52:42.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c17fff640 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63880 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3c04015d60 tx=0x7f3c04009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea64b9c54c314482 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c027fc640 1 -- 172.21.15.3:0/383233827 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04025070 con 0x55fe14a5d3e0 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c027fc640 1 -- 172.21.15.3:0/383233827 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c04009810 con 0x55fe14a5d3e0 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c027fc640 1 -- 172.21.15.3:0/383233827 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0401fcc0 con 0x55fe14a5d3e0 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 -- 172.21.15.3:0/383233827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 msgr2=0x55fe14a63880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63880 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f3c04015d60 tx=0x7f3c04009540 comp rx=0 tx=0).stop 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 -- 172.21.15.3:0/383233827 shutdown_connections 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a65f60 0x55fe14a60240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a5d3e0 0x55fe14a63880 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/383233827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a65710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.722+0000 7f3c21bdd740 1 -- 172.21.15.3:0/383233827 >> 172.21.15.3:0/383233827 conn(0x55fe14a57100 msgr2=0x55fe14a5e2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 -- 172.21.15.3:0/383233827 shutdown_connections 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 -- 172.21.15.3:0/383233827 wait complete. 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 -- start start 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 Processor -- start 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a6bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a6c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14acc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14a6c9d0 con 0x55fe14a65f60 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14a6cb40 con 0x55fe14a5d3e0 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe14accf50 con 0x55fe14a5ca30 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c21bdd740 10 client.0 _ll_register_callbacks cb 0x7f3c1000cf10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c1d5ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14acc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c1d5ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14acc9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41112/0 (socket says 172.21.15.3:41112) 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.723+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1129410464 learned_addr learned my addr 172.21.15.3:0/1129410464 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 msgr2=0x55fe14a6bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a6c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a6bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a6bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1129410464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 msgr2=0x55fe14a6c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a6c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1d5ab640 1 -- 172.21.15.3:0/1129410464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe149c26b0 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c17fff640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a6c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c1d5ab640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14acc9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3c100a2f60 tx=0x7f3c100cbba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e8cd30424c2c21 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3bf7fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1129410464 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c100151b0 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1129410464 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3c10005190 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c1009acf0 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1129410464 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c10005390 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c21bdd740 10 client.16044 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55fe14984a70 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1129410464 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c1009acf0 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.724+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1129410464 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f3c10018030 con 0x55fe14a65f60 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 10 client.16044 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 10 client.16044 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:52:42.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 20 client.16044 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14ace470 0x55fe14ad0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 4 client.16044 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fe1474f890 con 0x55fe14ace470 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14ace470 0x55fe14ad0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3bf7fff640 10 client.16044 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c1cdaa640 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14ace470 0x55fe14ad0890 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=398cccbbaf79aaa server_cookie=1d6a7d727efa3a6c in_seq=0 out_seq=0 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3bf7fff640 1 -- 172.21.15.3:0/1129410464 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fe1474f890 con 0x55fe14ace470 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3bf7fff640 10 client.16044 handle_command_reply: tid=0 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.725+0000 7f3c21bdd740 1 client.16044 shutdown 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 msgr2=0x55fe14acc9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14acc9c0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f3c100a2f60 tx=0x7f3c100cbba0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 shutdown_connections 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55fe14ace470 0x55fe14ad0890 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe14a5d3e0 0x55fe14a6c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe14a65f60 0x55fe14acc9c0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3bf7fff640 0 client.16044 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 --2- 172.21.15.3:0/1129410464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe14a5ca30 0x55fe14a6bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 >> 172.21.15.3:0/1129410464 conn(0x55fe14a57100 msgr2=0x55fe14a559b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 shutdown_connections 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 1 -- 172.21.15.3:0/1129410464 wait complete. 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c21bdd740 20 client.16044 trim_cache size 0 max 16384 2024-02-09T22:52:42.735 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3c080a23d0 msgr2=0x7f3c080a4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3c080a23d0 0x7f3c080a4890 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f3c04005fd0 tx=0x7f3c040157f0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 msgr2=0x7f3c18197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 0x7f3c18197c10 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000eaa0 tx=0x7f3c1000ef60 comp rx=0 tx=0).stop 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 shutdown_connections 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3c080a23d0 0x7f3c080a4890 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3c18106510 0x7f3c1819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3c18197790 0x7f3c18197c10 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 --2- 172.21.15.3:0/1958511316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3c18101440 0x7f3c1819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 >> 172.21.15.3:0/1958511316 conn(0x7f3c180fccc0 msgr2=0x7f3c180fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 shutdown_connections 2024-02-09T22:52:42.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.726+0000 7f3c1f00e640 1 -- 172.21.15.3:0/1958511316 wait complete. 2024-02-09T22:52:42.740 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:42.740 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 -- 172.21.15.3:0/637859489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c081065d0 msgr2=0x7f9c0810e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 --2- 172.21.15.3:0/637859489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c081065d0 0x7f9c0810e1c0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f9c00009cd0 tx=0x7f9c0002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 -- 172.21.15.3:0/637859489 shutdown_connections 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 --2- 172.21.15.3:0/637859489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c081007c0 0x7f9c08105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 --2- 172.21.15.3:0/637859489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c081065d0 0x7f9c0810e1c0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 --2- 172.21.15.3:0/637859489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c080ffe10 0x7f9c081001f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 -- 172.21.15.3:0/637859489 >> 172.21.15.3:0/637859489 conn(0x7f9c08075410 msgr2=0x7f9c08075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 -- 172.21.15.3:0/637859489 shutdown_connections 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.883+0000 7f9c102c3640 1 -- 172.21.15.3:0/637859489 wait complete. 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 Processor -- start 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 -- start start 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 0x7f9c08100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c081065d0 0x7f9c081012c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c08101800 0x7f9c081a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c081a6080 con 0x7f9c080ffe10 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c081a61f0 con 0x7f9c081065d0 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c102c3640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c081a63b0 con 0x7f9c08101800 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 0x7f9c08100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 0x7f9c08100d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41132/0 (socket says 172.21.15.3:41132) 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 -- 172.21.15.3:0/2746418356 learned_addr learned my addr 172.21.15.3:0/2746418356 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c081065d0 0x7f9c081012c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 -- 172.21.15.3:0/2746418356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c08101800 msgr2=0x7f9c081a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c08101800 0x7f9c081a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 -- 172.21.15.3:0/2746418356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c081065d0 msgr2=0x7f9c081012c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c081065d0 0x7f9c081012c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.884+0000 7f9c0e05f640 1 -- 172.21.15.3:0/2746418356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c00009900 con 0x7f9c080ffe10 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.885+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 0x7f9c08100d80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400d770 tx=0x7f9c0400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.885+0000 7f9bff7fe640 1 -- 172.21.15.3:0/2746418356 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04009c30 con 0x7f9c080ffe10 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.885+0000 7f9bff7fe640 1 -- 172.21.15.3:0/2746418356 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c0400de80 con 0x7f9c080ffe10 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.885+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c081a6690 con 0x7f9c080ffe10 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.885+0000 7f9bff7fe640 1 -- 172.21.15.3:0/2746418356 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04004e80 con 0x7f9c080ffe10 2024-02-09T22:52:42.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.885+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c081a6b50 con 0x7f9c080ffe10 2024-02-09T22:52:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.887+0000 7f9bff7fe640 1 -- 172.21.15.3:0/2746418356 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f9c04005020 con 0x7f9c080ffe10 2024-02-09T22:52:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.887+0000 7f9c12f75740 1 Processor -- start 2024-02-09T22:52:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 -- start start 2024-02-09T22:52:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9bff7fe640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bf40a2380 0x7f9bf40a4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6878de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68796990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d687993b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d68749020 con 0x564d6878da30 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d686fc320 con 0x564d6878e3e0 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c12f75740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d686f9850 con 0x564d68796f60 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9bff7fe640 1 -- 172.21.15.3:0/2746418356 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9c040cde60 con 0x7f9c080ffe10 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c0e860640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bf40a2380 0x7f9bf40a4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c0e05f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6878de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c0d85e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68796990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c0e05f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6878de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41146/0 (socket says 172.21.15.3:41146) 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1403280573 learned_addr learned my addr 172.21.15.3:0/1403280573 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.888+0000 7f9c0e860640 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d687993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1403280573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 msgr2=0x564d687993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d687993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1403280573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 msgr2=0x564d68796990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68796990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1403280573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d686b7210 con 0x564d6878da30 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68796990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e860640 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d687993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e860640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bf40a2380 0x7f9bf40a4840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c00006010 tx=0x7f9c00032450 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6878de10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009610 tx=0x7f9c0409a1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ed154c3c6e496fc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9bfcbfe640 1 -- 172.21.15.3:0/1403280573 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040637d0 con 0x564d6878da30 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9bfcbfe640 1 -- 172.21.15.3:0/1403280573 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c0409acd0 con 0x564d6878da30 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.889+0000 7f9bfcbfe640 1 -- 172.21.15.3:0/1403280573 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040ba0b0 con 0x564d6878da30 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- 172.21.15.3:0/1403280573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 msgr2=0x564d6878de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6878de10 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009610 tx=0x7f9c0409a1d0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- 172.21.15.3:0/1403280573 shutdown_connections 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68796990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d687993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1403280573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6878de10 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- 172.21.15.3:0/1403280573 >> 172.21.15.3:0/1403280573 conn(0x564d68788100 msgr2=0x564d687884e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- 172.21.15.3:0/1403280573 shutdown_connections 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- 172.21.15.3:0/1403280573 wait complete. 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- start start 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 Processor -- start 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d68793f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 0x564d687944b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d68796f60 0x564d687949f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d686f5030 con 0x564d6878e3e0 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d683c8430 con 0x564d68796f60 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d68384e00 con 0x564d6878da30 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c12f75740 10 client.0 _ll_register_callbacks cb 0x564d686ea030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c0d85e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 0x564d687944b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c0d85e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 0x564d687944b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41154/0 (socket says 172.21.15.3:41154) 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.890+0000 7f9c0d85e640 1 -- 172.21.15.3:0/2105575937 learned_addr learned my addr 172.21.15.3:0/2105575937 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d68793f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0d85e640 1 -- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 msgr2=0x564d68793f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d68793f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0d85e640 1 -- 172.21.15.3:0/2105575937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d68796f60 msgr2=0x564d687949f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d68796f60 0x564d687949f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0d85e640 1 -- 172.21.15.3:0/2105575937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c081a6b50 con 0x564d6878e3e0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d68793f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 0x564d687944b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800bd90 tx=0x7f9bf8012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f559fddbfabda3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9bf1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/2105575937 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bf8004230 con 0x564d6878e3e0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/2105575937 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9bf8004390 con 0x564d6878e3e0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d683f47e0 con 0x564d6878e3e0 2024-02-09T22:52:42.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/2105575937 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9bf80035f0 con 0x564d6878e3e0 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c12f75740 10 client.16056 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564d686f36b0 con 0x564d6878e3e0 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.891+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/2105575937 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564d683f47e0 con 0x564d6878e3e0 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/2105575937 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9bf8010ad0 con 0x564d6878e3e0 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c12f75740 10 client.16056 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c12f75740 10 client.16056 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c12f75740 20 client.16056 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c12f75740 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687a7580 0x564d687a99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c12f75740 4 client.16056 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d68431720 con 0x564d687a7580 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687a7580 0x564d687a99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687a7580 0x564d687a99c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ce1eb458cb2d10a5 server_cookie=e645daaa4f19f17a in_seq=0 out_seq=0 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.892+0000 7f9bf1ffb640 10 client.16056 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:42.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/2105575937 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d68431720 con 0x564d687a7580 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9bf1ffb640 10 client.16056 handle_command_reply: tid=0 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 client.16056 shutdown 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 msgr2=0x564d687944b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 0x564d687944b0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f9bf800bd90 tx=0x7f9bf8012820 comp rx=0 tx=0).stop 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 shutdown_connections 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687a7580 0x564d687a99c0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d68796f60 0x564d687949f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9bf1ffb640 0 client.16056 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878e3e0 0x564d687944b0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 --2- 172.21.15.3:0/2105575937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d68793f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.893+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 >> 172.21.15.3:0/2105575937 conn(0x564d68788100 msgr2=0x564d68789e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.894+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 shutdown_connections 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.894+0000 7f9c12f75740 1 -- 172.21.15.3:0/2105575937 wait complete. 2024-02-09T22:52:42.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.894+0000 7f9c12f75740 20 client.16056 trim_cache size 0 max 16384 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 Processor -- start 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 -- start start 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d687966a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68794810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 0x564d687911d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d6867a130 con 0x564d68796f60 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d68788e90 con 0x564d6878e3e0 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c12f75740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d68787c30 con 0x564d6878da30 2024-02-09T22:52:42.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c0e860640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 0x564d687911d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c0e860640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 0x564d687911d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41164/0 (socket says 172.21.15.3:41164) 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.905+0000 7f9c0e860640 1 -- 172.21.15.3:0/3705457629 learned_addr learned my addr 172.21.15.3:0/3705457629 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d687966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68794810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e860640 1 -- 172.21.15.3:0/3705457629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 msgr2=0x564d687966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e860640 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d687966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e860640 1 -- 172.21.15.3:0/3705457629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 msgr2=0x564d68794810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e860640 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68794810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e860640 1 -- 172.21.15.3:0/3705457629 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d6868fc90 con 0x564d68796f60 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68794810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9c0e860640 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 0x564d687911d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9c00036000 tx=0x7f9c00046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3feef118c072411b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9bf3fff640 1 -- 172.21.15.3:0/3705457629 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00048040 con 0x564d68796f60 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9bf3fff640 1 -- 172.21.15.3:0/3705457629 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c00034d50 con 0x564d68796f60 2024-02-09T22:52:42.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.906+0000 7f9bf3fff640 1 -- 172.21.15.3:0/3705457629 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c000042a0 con 0x564d68796f60 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- 172.21.15.3:0/3705457629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 msgr2=0x564d687911d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 0x564d687911d0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f9c00036000 tx=0x7f9c00046000 comp rx=0 tx=0).stop 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- 172.21.15.3:0/3705457629 shutdown_connections 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d68794810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d68796f60 0x564d687911d0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- 172.21.15.3:0/3705457629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d6878da30 0x564d687966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- 172.21.15.3:0/3705457629 >> 172.21.15.3:0/3705457629 conn(0x564d68788100 msgr2=0x564d68792840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- 172.21.15.3:0/3705457629 shutdown_connections 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- 172.21.15.3:0/3705457629 wait complete. 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- start start 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 Processor -- start 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6879ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d6879cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d6879d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d687fdb40 con 0x564d6878da30 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d687fdcb0 con 0x564d6878e3e0 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d687fde70 con 0x564d68796f60 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c12f75740 10 client.0 _ll_register_callbacks cb 0x7f9c04096150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c0e05f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6879ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c0e05f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6879ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41174/0 (socket says 172.21.15.3:41174) 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1614639157 learned_addr learned my addr 172.21.15.3:0/1614639157 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c0e860640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d6879d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.907+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d6879cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 msgr2=0x564d6879d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d6879d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1614639157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 msgr2=0x564d6879cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d6879cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c0e05f640 1 -- 172.21.15.3:0/1614639157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0810e9d0 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c0e05f640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6879ca70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9c040cb340 tx=0x7f9c040ba040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ceab6a2f74e936e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9bf1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/1614639157 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040a2d70 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/1614639157 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9c0409a5d0 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c0409acd0 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/1614639157 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c0409e720 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 10 client.16062 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x564d686b5a70 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/1614639157 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c0409acd0 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/1614639157 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9c04099a20 con 0x564d6878da30 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 10 client.16062 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:42.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 10 client.16062 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 20 client.16062 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687ff1e0 0x564d68801600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 4 client.16062 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.908+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564d68480890 con 0x564d687ff1e0 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687ff1e0 0x564d68801600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9bf1ffb640 10 client.16062 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c0d85e640 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687ff1e0 0x564d68801600 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=91f1ac785b1fe594 server_cookie=31979b99ea737eaf in_seq=0 out_seq=0 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9bf1ffb640 1 -- 172.21.15.3:0/1614639157 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564d68480890 con 0x564d687ff1e0 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9bf1ffb640 10 client.16062 handle_command_reply: tid=0 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 client.16062 shutdown 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 msgr2=0x564d6879ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6879ca70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f9c040cb340 tx=0x7f9c040ba040 comp rx=0 tx=0).stop 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 shutdown_connections 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564d687ff1e0 0x564d68801600 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d6878e3e0 0x564d6879cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d68796f60 0x564d6879d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9bf1ffb640 0 client.16062 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 --2- 172.21.15.3:0/1614639157 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d6878da30 0x564d6879ca70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 >> 172.21.15.3:0/1614639157 conn(0x564d68788100 msgr2=0x564d687869b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 shutdown_connections 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 1 -- 172.21.15.3:0/1614639157 wait complete. 2024-02-09T22:52:42.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.909+0000 7f9c12f75740 20 client.16062 trim_cache size 0 max 16384 2024-02-09T22:52:42.920 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bf40a2380 msgr2=0x7f9bf40a4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bf40a2380 0x7f9bf40a4840 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9c00006010 tx=0x7f9c00032450 comp rx=0 tx=0).stop 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 msgr2=0x7f9c08100d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 0x7f9c08100d80 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f9c0400d770 tx=0x7f9c0400dc30 comp rx=0 tx=0).stop 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 shutdown_connections 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bf40a2380 0x7f9bf40a4840 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c081065d0 0x7f9c081012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c08101800 0x7f9c081a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 --2- 172.21.15.3:0/2746418356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c080ffe10 0x7f9c08100d80 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:42.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 >> 172.21.15.3:0/2746418356 conn(0x7f9c08075410 msgr2=0x7f9c080fde10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:42.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 shutdown_connections 2024-02-09T22:52:42.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:42.910+0000 7f9c102c3640 1 -- 172.21.15.3:0/2746418356 wait complete. 2024-02-09T22:52:42.923 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:42.923 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:52:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 -- 172.21.15.3:0/3921416373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9510102d70 msgr2=0x7f951010b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 --2- 172.21.15.3:0/3921416373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9510102d70 0x7f951010b250 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f9500009900 tx=0x7f950002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 -- 172.21.15.3:0/3921416373 shutdown_connections 2024-02-09T22:52:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 --2- 172.21.15.3:0/3921416373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f951010b820 0x7f951010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 --2- 172.21.15.3:0/3921416373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9510102d70 0x7f951010b250 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 --2- 172.21.15.3:0/3921416373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9510102450 0x7f9510102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 -- 172.21.15.3:0/3921416373 >> 172.21.15.3:0/3921416373 conn(0x7f95100772e0 msgr2=0x7f95100776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 -- 172.21.15.3:0/3921416373 shutdown_connections 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 -- 172.21.15.3:0/3921416373 wait complete. 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 Processor -- start 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.027+0000 7f9516979640 1 -- start start 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9516979640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9510102450 0x7f95101000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9516979640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9510102d70 0x7f95101005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9516979640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 0x7f9510100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9516979640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101a5c30 con 0x7f951010b820 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9516979640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101a5da0 con 0x7f9510102d70 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9516979640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95101a5f60 con 0x7f9510102450 2024-02-09T22:52:43.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 0x7f9510100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 0x7f9510100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41204/0 (socket says 172.21.15.3:41204) 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 -- 172.21.15.3:0/1987751495 learned_addr learned my addr 172.21.15.3:0/1987751495 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f950ffff640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9510102450 0x7f95101000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9510102d70 0x7f95101005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 -- 172.21.15.3:0/1987751495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9510102450 msgr2=0x7f95101000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9510102450 0x7f95101000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 -- 172.21.15.3:0/1987751495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9510102d70 msgr2=0x7f95101005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9510102d70 0x7f95101005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 -- 172.21.15.3:0/1987751495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9500009560 con 0x7f951010b820 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9510102d70 0x7f95101005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.028+0000 7f9514f16640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 0x7f9510100b20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f950800da40 tx=0x7f950800df00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.029+0000 7f950d7fa640 1 -- 172.21.15.3:0/1987751495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950800bdd0 con 0x7f951010b820 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.029+0000 7f950d7fa640 1 -- 172.21.15.3:0/1987751495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9508009890 con 0x7f951010b820 2024-02-09T22:52:43.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.029+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95101a6240 con 0x7f951010b820 2024-02-09T22:52:43.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.029+0000 7f950d7fa640 1 -- 172.21.15.3:0/1987751495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080106c0 con 0x7f951010b820 2024-02-09T22:52:43.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.029+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101a6700 con 0x7f951010b820 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.031+0000 7f950d7fa640 1 -- 172.21.15.3:0/1987751495 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f95080041b0 con 0x7f951010b820 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.031+0000 7f950d7fa640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f94f80a2350 0x7f94f80a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.031+0000 7f951961f740 1 Processor -- start 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.031+0000 7f951961f740 1 -- start start 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.031+0000 7f950ffff640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f94f80a2350 0x7f94f80a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.031+0000 7f950d7fa640 1 -- 172.21.15.3:0/1987751495 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f95080cdf90 con 0x7f951010b820 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f951961f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c4a30 0x55db112c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f951961f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112cd990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f951961f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112cdf60 0x55db112d03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f951961f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db11280020 con 0x55db112cdf60 2024-02-09T22:52:43.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f951961f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db11233320 con 0x55db112c53e0 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f951961f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db11230850 con 0x55db112c4a30 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f950ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c4a30 0x55db112c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f950ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c4a30 0x55db112c4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:44774/0 (socket says 172.21.15.3:44774) 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f950ffff640 1 -- 172.21.15.3:0/1106962545 learned_addr learned my addr 172.21.15.3:0/1106962545 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112cdf60 0x55db112d03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112cd990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f950ffff640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f94f80a2350 0x7f94f80a4810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9504009c20 tx=0x7f9504011810 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 -- 172.21.15.3:0/1106962545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c4a30 msgr2=0x55db112c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c4a30 0x55db112c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 -- 172.21.15.3:0/1106962545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 msgr2=0x55db112cd990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112cd990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 -- 172.21.15.3:0/1106962545 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db111ee210 con 0x55db112cdf60 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:43.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f9514f16640 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112cdf60 0x55db112d03b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f95080cb590 tx=0x7f950809a2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3973a2ff47d69a96 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f94f67fc640 1 -- 172.21.15.3:0/1106962545 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950809a760 con 0x55db112cdf60 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f94f67fc640 1 -- 172.21.15.3:0/1106962545 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950809e8b0 con 0x55db112cdf60 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.032+0000 7f94f67fc640 1 -- 172.21.15.3:0/1106962545 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080cb870 con 0x55db112cdf60 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 -- 172.21.15.3:0/1106962545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112cdf60 msgr2=0x55db112d03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112cdf60 0x55db112d03b0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f95080cb590 tx=0x7f950809a2a0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 -- 172.21.15.3:0/1106962545 shutdown_connections 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112cd990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112cdf60 0x55db112d03b0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 --2- 172.21.15.3:0/1106962545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c4a30 0x55db112c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 -- 172.21.15.3:0/1106962545 >> 172.21.15.3:0/1106962545 conn(0x55db112bf100 msgr2=0x55db112bf4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 -- 172.21.15.3:0/1106962545 shutdown_connections 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 -- 172.21.15.3:0/1106962545 wait complete. 2024-02-09T22:52:43.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.033+0000 7f951961f740 1 -- start start 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 Processor -- start 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c4a30 0x55db112caf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 0x55db112cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112cdf60 0x55db112cb9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db1122c030 con 0x55db112c53e0 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db10eff430 con 0x55db112c4a30 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db10ebbe00 con 0x55db112cdf60 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f951961f740 10 client.0 _ll_register_callbacks cb 0x55db11221030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 0x55db112cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 0x55db112cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41232/0 (socket says 172.21.15.3:41232) 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 -- 172.21.15.3:0/2064479806 learned_addr learned my addr 172.21.15.3:0/2064479806 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950ffff640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c4a30 0x55db112caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f9514f16640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112cdf60 0x55db112cb9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 -- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112cdf60 msgr2=0x55db112cb9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112cdf60 0x55db112cb9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 -- 172.21.15.3:0/2064479806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c4a30 msgr2=0x55db112caf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c4a30 0x55db112caf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.034+0000 7f950f7fe640 1 -- 172.21.15.3:0/2064479806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95101a6700 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f950f7fe640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 0x55db112cb490 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9500005940 tx=0x7f950002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c37417b6e7c8563 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f94ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2064479806 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500004450 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2064479806 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9500044070 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db10f2b7e0 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2064479806 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f950002c670 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55db1122a6b0 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2064479806 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55db10f2b7e0 con 0x55db112c53e0 2024-02-09T22:52:43.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2064479806 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f95000038c0 con 0x55db112c53e0 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 10 client.16074 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 20 client.16074 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112de3e0 0x55db112e0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 4 client.16074 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.035+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db10f68720 con 0x55db112de3e0 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.036+0000 7f950ffff640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112de3e0 0x55db112e0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.036+0000 7f94ef7fe640 10 client.16074 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.036+0000 7f950ffff640 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112de3e0 0x55db112e0800 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=321f2b997052dd95 server_cookie=96777e8da44f6349 in_seq=0 out_seq=0 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f94ef7fe640 1 -- 172.21.15.3:0/2064479806 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55db10f68720 con 0x55db112de3e0 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f94ef7fe640 10 client.16074 handle_command_reply: tid=0 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 client.16074 shutdown 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 msgr2=0x55db112cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 0x55db112cb490 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f9500005940 tx=0x7f950002cb30 comp rx=0 tx=0).stop 2024-02-09T22:52:43.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 shutdown_connections 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112de3e0 0x55db112e0800 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112cdf60 0x55db112cb9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c53e0 0x55db112cb490 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f94ef7fe640 0 client.16074 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 --2- 172.21.15.3:0/2064479806 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c4a30 0x55db112caf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 >> 172.21.15.3:0/2064479806 conn(0x55db112bf100 msgr2=0x55db112c0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 shutdown_connections 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 1 -- 172.21.15.3:0/2064479806 wait complete. 2024-02-09T22:52:43.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.037+0000 7f951961f740 20 client.16074 trim_cache size 0 max 16384 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.049+0000 7f951961f740 1 Processor -- start 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.049+0000 7f951961f740 1 -- start start 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.049+0000 7f951961f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112bf100 0x55db112cb7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f951961f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db112be0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f951961f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c53e0 0x55db112d4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f951961f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db111b1130 con 0x55db112c4a30 2024-02-09T22:52:43.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f951961f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db112daf80 con 0x55db112bf100 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f951961f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db112bfd00 con 0x55db112c53e0 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db112be0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db112be0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41240/0 (socket says 172.21.15.3:41240) 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 -- 172.21.15.3:0/1526761503 learned_addr learned my addr 172.21.15.3:0/1526761503 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f9514f16640 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c53e0 0x55db112d4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950ffff640 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112bf100 0x55db112cb7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 -- 172.21.15.3:0/1526761503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c53e0 msgr2=0x55db112d4ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c53e0 0x55db112d4ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 -- 172.21.15.3:0/1526761503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112bf100 msgr2=0x55db112cb7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112bf100 0x55db112cb7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 -- 172.21.15.3:0/1526761503 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db111c6c90 con 0x55db112c4a30 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f9514f16640 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c53e0 0x55db112d4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f950f7fe640 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db112be0c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f950002c210 tx=0x7f950002bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecfe060c7e89ea58 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f94f5ffb640 1 -- 172.21.15.3:0/1526761503 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500003df0 con 0x55db112c4a30 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f94f5ffb640 1 -- 172.21.15.3:0/1526761503 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9500042070 con 0x55db112c4a30 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.050+0000 7f94f5ffb640 1 -- 172.21.15.3:0/1526761503 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9500049620 con 0x55db112c4a30 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 -- 172.21.15.3:0/1526761503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 msgr2=0x55db112be0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db112be0c0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f950002c210 tx=0x7f950002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 -- 172.21.15.3:0/1526761503 shutdown_connections 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112c53e0 0x55db112d4ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db112be0c0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 --2- 172.21.15.3:0/1526761503 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112bf100 0x55db112cb7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 -- 172.21.15.3:0/1526761503 >> 172.21.15.3:0/1526761503 conn(0x55db112c5b30 msgr2=0x55db112bd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 -- 172.21.15.3:0/1526761503 shutdown_connections 2024-02-09T22:52:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 -- 172.21.15.3:0/1526761503 wait complete. 2024-02-09T22:52:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 -- start start 2024-02-09T22:52:43.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.051+0000 7f951961f740 1 Processor -- start 2024-02-09T22:52:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112bf100 0x55db1132dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db1132e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112d40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db112d4790 con 0x55db112c4a30 2024-02-09T22:52:43.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db112d4900 con 0x55db112c53e0 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db112d4ac0 con 0x55db112bf100 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f951961f740 10 client.0 _ll_register_callbacks cb 0x7f95080048b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db1132e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112bf100 0x55db1132dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db1132e1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41252/0 (socket says 172.21.15.3:41252) 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 -- 172.21.15.3:0/3984991480 learned_addr learned my addr 172.21.15.3:0/3984991480 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112bf100 0x55db1132dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:44806/0 (socket says 172.21.15.3:44806) 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f9514f16640 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112d40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 -- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112bf100 msgr2=0x55db1132dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112bf100 0x55db1132dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 -- 172.21.15.3:0/3984991480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 msgr2=0x55db112d40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112d40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 -- 172.21.15.3:0/3984991480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55db1122a6b0 con 0x55db112c4a30 2024-02-09T22:52:43.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f950f7fe640 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db1132e1c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9500005db0 tx=0x7f9500004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8bf279bb8af477 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.052+0000 7f94ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f94ef7fe640 1 -- 172.21.15.3:0/3984991480 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f950002cb10 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f94ef7fe640 1 -- 172.21.15.3:0/3984991480 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9500044070 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f950809e8b0 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f94ef7fe640 1 -- 172.21.15.3:0/3984991480 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f950002cc70 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 10 client.16080 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55db111eca70 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f94ef7fe640 1 -- 172.21.15.3:0/3984991480 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f950809e8b0 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f94ef7fe640 1 -- 172.21.15.3:0/3984991480 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9500004df0 con 0x55db112c4a30 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 10 client.16080 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 10 client.16080 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 20 client.16080 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112cdf60 0x55db11338770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 4 client.16080 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55db10fb7890 con 0x55db112cdf60 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.053+0000 7f950ffff640 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112cdf60 0x55db11338770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f94ef7fe640 10 client.16080 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f950ffff640 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112cdf60 0x55db11338770 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5db0d2c190f798c8 server_cookie=dfac3dd64b72196d in_seq=0 out_seq=0 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f94ef7fe640 1 -- 172.21.15.3:0/3984991480 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55db10fb7890 con 0x55db112cdf60 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f94ef7fe640 10 client.16080 handle_command_reply: tid=0 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 client.16080 shutdown 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 msgr2=0x55db1132e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db1132e1c0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f9500005db0 tx=0x7f9500004af0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 shutdown_connections 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55db112cdf60 0x55db11338770 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db112c53e0 0x55db112d40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db112c4a30 0x55db1132e1c0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 --2- 172.21.15.3:0/3984991480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db112bf100 0x55db1132dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f94ef7fe640 0 client.16080 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:52:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 >> 172.21.15.3:0/3984991480 conn(0x55db112c5b30 msgr2=0x55db112be240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 shutdown_connections 2024-02-09T22:52:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 1 -- 172.21.15.3:0/3984991480 wait complete. 2024-02-09T22:52:43.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.054+0000 7f951961f740 20 client.16080 trim_cache size 0 max 16384 2024-02-09T22:52:43.064 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f94f80a2350 msgr2=0x7f94f80a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f94f80a2350 0x7f94f80a4810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9504009c20 tx=0x7f9504011810 comp rx=0 tx=0).stop 2024-02-09T22:52:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 msgr2=0x7f9510100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 0x7f9510100b20 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f950800da40 tx=0x7f950800df00 comp rx=0 tx=0).stop 2024-02-09T22:52:43.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 shutdown_connections 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f94f80a2350 0x7f94f80a4810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9510102d70 0x7f95101005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f951010b820 0x7f9510100b20 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 --2- 172.21.15.3:0/1987751495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9510102450 0x7f95101000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 >> 172.21.15.3:0/1987751495 conn(0x7f95100772e0 msgr2=0x7f95100fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 shutdown_connections 2024-02-09T22:52:43.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.055+0000 7f9516979640 1 -- 172.21.15.3:0/1987751495 wait complete. 2024-02-09T22:52:43.069 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:43.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:52:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 -- 172.21.15.3:0/2807932928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc100340 msgr2=0x7f9ffc100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 --2- 172.21.15.3:0/2807932928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc100340 0x7f9ffc100720 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9fec009930 tx=0x7f9fec02bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 -- 172.21.15.3:0/2807932928 shutdown_connections 2024-02-09T22:52:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 --2- 172.21.15.3:0/2807932928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ffc100cf0 0x7f9ffc104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 --2- 172.21.15.3:0/2807932928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ffc105410 0x7f9ffc10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 --2- 172.21.15.3:0/2807932928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc100340 0x7f9ffc100720 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 -- 172.21.15.3:0/2807932928 >> 172.21.15.3:0/2807932928 conn(0x7f9ffc0772e0 msgr2=0x7f9ffc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 -- 172.21.15.3:0/2807932928 shutdown_connections 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.171+0000 7fa001481640 1 -- 172.21.15.3:0/2807932928 wait complete. 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 Processor -- start 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 -- start start 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ffc100340 0x7f9ffc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ffc105410 0x7f9ffc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 0x7f9ffc1a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc1a3e10 con 0x7f9ffc1991c0 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc1a3f80 con 0x7f9ffc105410 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7fa001481640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ffc1a4140 con 0x7f9ffc100340 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7f9ffb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 0x7f9ffc1a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7f9ffaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ffc100340 0x7f9ffc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7f9ffb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 0x7f9ffc1a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41264/0 (socket says 172.21.15.3:41264) 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7f9ffb7fe640 1 -- 172.21.15.3:0/1018435071 learned_addr learned my addr 172.21.15.3:0/1018435071 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.172+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ffc105410 0x7f9ffc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.173+0000 7f9ffb7fe640 1 -- 172.21.15.3:0/1018435071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ffc100340 msgr2=0x7f9ffc1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.173+0000 7f9ffb7fe640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ffc100340 0x7f9ffc1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.173+0000 7f9ffb7fe640 1 -- 172.21.15.3:0/1018435071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ffc105410 msgr2=0x7f9ffc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.173+0000 7f9ffb7fe640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ffc105410 0x7f9ffc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.173+0000 7f9ffb7fe640 1 -- 172.21.15.3:0/1018435071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec009560 con 0x7f9ffc1991c0 2024-02-09T22:52:43.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.173+0000 7f9ffb7fe640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 0x7f9ffc1a3770 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400cab0 tx=0x7f9ff400cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.175+0000 7f9ff8ff9640 1 -- 172.21.15.3:0/1018435071 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff400ae20 con 0x7f9ffc1991c0 2024-02-09T22:52:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.175+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ffc1a4420 con 0x7f9ffc1991c0 2024-02-09T22:52:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.175+0000 7f9ff8ff9640 1 -- 172.21.15.3:0/1018435071 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ff400f930 con 0x7f9ffc1991c0 2024-02-09T22:52:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.175+0000 7f9ff8ff9640 1 -- 172.21.15.3:0/1018435071 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ff4018a70 con 0x7f9ffc1991c0 2024-02-09T22:52:43.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.175+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ffc1a48e0 con 0x7f9ffc1991c0 2024-02-09T22:52:43.179 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:43 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1985396075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:52:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 Processor -- start 2024-02-09T22:52:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ff8ff9640 1 -- 172.21.15.3:0/1018435071 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f9ff400faa0 con 0x7f9ffc1991c0 2024-02-09T22:52:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 -- start start 2024-02-09T22:52:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c730be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c7314990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c73173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c72c7020 con 0x5574c7314f60 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c727a320 con 0x5574c730c3e0 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7fa00404f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c7277850 con 0x5574c730ba30 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ff3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c73173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ff3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c73173b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41276/0 (socket says 172.21.15.3:41276) 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ff3fff640 1 -- 172.21.15.3:0/127474790 learned_addr learned my addr 172.21.15.3:0/127474790 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c730be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c7314990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.178+0000 7f9ff8ff9640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9fe80a2350 0x7f9fe80a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff3fff640 1 -- 172.21.15.3:0/127474790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 msgr2=0x5574c730be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c730be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff3fff640 1 -- 172.21.15.3:0/127474790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 msgr2=0x5574c7314990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c7314990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff3fff640 1 -- 172.21.15.3:0/127474790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574c7235210 con 0x5574c7314f60 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ffb7fe640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9fe80a2350 0x7f9fe80a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff8ff9640 1 -- 172.21.15.3:0/1018435071 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ff40cead0 con 0x7f9ffc1991c0 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c73173b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4012770 tx=0x7f9fe4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443057bc72e01e88 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ffb7fe640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9fe80a2350 0x7f9fe80a4810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40cc130 tx=0x7f9ff4018400 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff17fa640 1 -- 172.21.15.3:0/127474790 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4012cf0 con 0x5574c7314f60 2024-02-09T22:52:43.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff17fa640 1 -- 172.21.15.3:0/127474790 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fe4004510 con 0x5574c7314f60 2024-02-09T22:52:43.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.179+0000 7f9ff17fa640 1 -- 172.21.15.3:0/127474790 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4003850 con 0x5574c7314f60 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 -- 172.21.15.3:0/127474790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 msgr2=0x5574c73173b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c73173b0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f9fe4012770 tx=0x7f9fe4004730 comp rx=0 tx=0).stop 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 -- 172.21.15.3:0/127474790 shutdown_connections 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c7314990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c73173b0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 --2- 172.21.15.3:0/127474790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c730be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 -- 172.21.15.3:0/127474790 >> 172.21.15.3:0/127474790 conn(0x5574c7306100 msgr2=0x5574c73064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 -- 172.21.15.3:0/127474790 shutdown_connections 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 -- 172.21.15.3:0/127474790 wait complete. 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.180+0000 7fa00404f740 1 -- start start 2024-02-09T22:52:43.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 Processor -- start 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c7311f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c7312480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c7314f60 0x5574c73129c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c7273030 con 0x5574c730c3e0 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6f46430 con 0x5574c7314f60 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c6f02e00 con 0x5574c730ba30 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 10 client.0 _ll_register_callbacks cb 0x5574c7268030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c7312480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c7312480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41292/0 (socket says 172.21.15.3:41292) 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 -- 172.21.15.3:0/506939858 learned_addr learned my addr 172.21.15.3:0/506939858 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c7311f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c7314f60 0x5574c73129c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 -- 172.21.15.3:0/506939858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 msgr2=0x5574c7311f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c7311f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 -- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c7314f60 msgr2=0x5574c73129c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c7314f60 0x5574c73129c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 -- 172.21.15.3:0/506939858 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe401a040 con 0x5574c730c3e0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c7312480 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9fec036040 tx=0x7f9fec02cb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b193e9040516b50 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9fe27fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9fe27fc640 1 -- 172.21.15.3:0/506939858 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec004250 con 0x5574c730c3e0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7f9fe27fc640 1 -- 172.21.15.3:0/506939858 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9fec0043b0 con 0x5574c730c3e0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.181+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5574c6f727e0 con 0x5574c730c3e0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7f9fe27fc640 1 -- 172.21.15.3:0/506939858 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fec004df0 con 0x5574c730c3e0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 10 client.16092 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5574c72716b0 con 0x5574c730c3e0 2024-02-09T22:52:43.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7f9fe27fc640 1 -- 172.21.15.3:0/506939858 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5574c6f727e0 con 0x5574c730c3e0 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7f9fe27fc640 1 -- 172.21.15.3:0/506939858 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9fec03c040 con 0x5574c730c3e0 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 10 client.16092 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 10 client.16092 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 20 client.16092 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c7325520 0x5574c7327940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 4 client.16092 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5574c6faf720 con 0x5574c7325520 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.182+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c7325520 0x5574c7327940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.183+0000 7f9fe27fc640 10 client.16092 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.183+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c7325520 0x5574c7327940 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e4a1f974797c75d5 server_cookie=cb7c80f908001431 in_seq=0 out_seq=0 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.183+0000 7f9fe27fc640 1 -- 172.21.15.3:0/506939858 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5574c6faf720 con 0x5574c7325520 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.183+0000 7f9fe27fc640 10 client.16092 handle_command_reply: tid=0 2024-02-09T22:52:43.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.183+0000 7fa00404f740 1 client.16092 shutdown 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 msgr2=0x5574c7312480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c7312480 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f9fec036040 tx=0x7f9fec02cb50 comp rx=0 tx=0).stop 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 shutdown_connections 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c7325520 0x5574c7327940 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c7314f60 0x5574c73129c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c7312480 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 --2- 172.21.15.3:0/506939858 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c7311f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 >> 172.21.15.3:0/506939858 conn(0x5574c7306100 msgr2=0x5574c7307e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7f9fe27fc640 0 client.16092 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 shutdown_connections 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 1 -- 172.21.15.3:0/506939858 wait complete. 2024-02-09T22:52:43.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.184+0000 7fa00404f740 20 client.16092 trim_cache size 0 max 16384 2024-02-09T22:52:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.195+0000 7fa00404f740 1 Processor -- start 2024-02-09T22:52:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.195+0000 7fa00404f740 1 -- start start 2024-02-09T22:52:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7fa00404f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730ba30 0x5574c7314670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7fa00404f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c73127e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7fa00404f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c7314f60 0x5574c730d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7fa00404f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c71f8130 con 0x5574c730c3e0 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7fa00404f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c7301ab0 con 0x5574c730ba30 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7fa00404f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c7322040 con 0x5574c7314f60 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c73127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c73127e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41294/0 (socket says 172.21.15.3:41294) 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 -- 172.21.15.3:0/4256289124 learned_addr learned my addr 172.21.15.3:0/4256289124 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 -- 172.21.15.3:0/4256289124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c7314f60 msgr2=0x5574c730d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c7314f60 0x5574c730d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730ba30 0x5574c7314670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 -- 172.21.15.3:0/4256289124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730ba30 msgr2=0x5574c7314670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730ba30 0x5574c7314670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 -- 172.21.15.3:0/4256289124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574c720dc90 con 0x5574c730c3e0 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730ba30 0x5574c7314670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c73127e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9fec02c350 tx=0x7f9fec02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c4d492d7ad67ff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ff0ff9640 1 -- 172.21.15.3:0/4256289124 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec004450 con 0x5574c730c3e0 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ff0ff9640 1 -- 172.21.15.3:0/4256289124 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fec0035f0 con 0x5574c730c3e0 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.196+0000 7f9ff0ff9640 1 -- 172.21.15.3:0/4256289124 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec005510 con 0x5574c730c3e0 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 -- 172.21.15.3:0/4256289124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 msgr2=0x5574c73127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c73127e0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f9fec02c350 tx=0x7f9fec02bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 -- 172.21.15.3:0/4256289124 shutdown_connections 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c7314f60 0x5574c730d250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c730c3e0 0x5574c73127e0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 --2- 172.21.15.3:0/4256289124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730ba30 0x5574c7314670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 -- 172.21.15.3:0/4256289124 >> 172.21.15.3:0/4256289124 conn(0x5574c7306100 msgr2=0x5574c73064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 -- 172.21.15.3:0/4256289124 shutdown_connections 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 -- 172.21.15.3:0/4256289124 wait complete. 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.197+0000 7fa00404f740 1 -- start start 2024-02-09T22:52:43.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 Processor -- start 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c731c450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c731a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c731af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c731b650 con 0x5574c7314f60 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c737ba30 con 0x5574c730c3e0 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574c737bb70 con 0x5574c730ba30 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7fa00404f740 10 client.0 _ll_register_callbacks cb 0x7f9fe400baa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c731af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c731af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41310/0 (socket says 172.21.15.3:41310) 2024-02-09T22:52:43.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 -- 172.21.15.3:0/3203124549 learned_addr learned my addr 172.21.15.3:0/3203124549 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c731a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c731c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 -- 172.21.15.3:0/3203124549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 msgr2=0x5574c731c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c731c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 -- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 msgr2=0x5574c731a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c731a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 -- 172.21.15.3:0/3203124549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574c72716b0 con 0x5574c7314f60 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9ff3fff640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c731af80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9fe401e000 tx=0x7f9fe4014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce76ee518a57fc54 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.198+0000 7f9fe27fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7f9fe27fc640 1 -- 172.21.15.3:0/3203124549 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fe4023040 con 0x5574c7314f60 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7f9fe27fc640 1 -- 172.21.15.3:0/3203124549 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9fe4000e40 con 0x5574c7314f60 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9fe4004510 con 0x5574c7314f60 2024-02-09T22:52:43.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7f9fe27fc640 1 -- 172.21.15.3:0/3203124549 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9fe4010740 con 0x5574c7314f60 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 10 client.16098 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5574c7233a70 con 0x5574c7314f60 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7f9fe27fc640 1 -- 172.21.15.3:0/3203124549 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9fe4004510 con 0x5574c7314f60 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7f9fe27fc640 1 -- 172.21.15.3:0/3203124549 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f9fe401f460 con 0x5574c7314f60 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 10 client.16098 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 20 client.16098 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c737d210 0x5574c737f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 4 client.16098 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5574c6ffe890 con 0x5574c737d210 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.199+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c737d210 0x5574c737f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7f9fe27fc640 10 client.16098 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7f9ffaffd640 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c737d210 0x5574c737f630 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a67a26a7a1199adf server_cookie=72875ed2e0a91fb7 in_seq=0 out_seq=0 2024-02-09T22:52:43.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7f9fe27fc640 1 -- 172.21.15.3:0/3203124549 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5574c6ffe890 con 0x5574c737d210 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7f9fe27fc640 10 client.16098 handle_command_reply: tid=0 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 client.16098 shutdown 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 msgr2=0x5574c731af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c731af80 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7f9fe401e000 tx=0x7f9fe4014040 comp rx=0 tx=0).stop 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 shutdown_connections 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5574c737d210 0x5574c737f630 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5574c730c3e0 0x5574c731a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5574c7314f60 0x5574c731af80 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 --2- 172.21.15.3:0/3203124549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5574c730ba30 0x5574c731c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7f9fe27fc640 0 client.16098 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 >> 172.21.15.3:0/3203124549 conn(0x5574c7306100 msgr2=0x5574c730d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 shutdown_connections 2024-02-09T22:52:43.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 1 -- 172.21.15.3:0/3203124549 wait complete. 2024-02-09T22:52:43.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.200+0000 7fa00404f740 20 client.16098 trim_cache size 0 max 16384 2024-02-09T22:52:43.210 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9fe80a2350 msgr2=0x7f9fe80a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9fe80a2350 0x7f9fe80a4810 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40cc130 tx=0x7f9ff4018400 comp rx=0 tx=0).stop 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 msgr2=0x7f9ffc1a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 0x7f9ffc1a3770 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9ff400cab0 tx=0x7f9ff400cf70 comp rx=0 tx=0).stop 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 shutdown_connections 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9fe80a2350 0x7f9fe80a4810 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ffc105410 0x7f9ffc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ffc1991c0 0x7f9ffc1a3770 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 --2- 172.21.15.3:0/1018435071 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ffc100340 0x7f9ffc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 >> 172.21.15.3:0/1018435071 conn(0x7f9ffc0772e0 msgr2=0x7f9ffc0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 shutdown_connections 2024-02-09T22:52:43.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.201+0000 7fa001481640 1 -- 172.21.15.3:0/1018435071 wait complete. 2024-02-09T22:52:43.215 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:52:43.215 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:52:43.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/850743939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 msgr2=0x7fcdac10dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/850743939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac10dc50 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fcda4009ca0 tx=0x7fcda402c180 comp rx=0 tx=0).stop 2024-02-09T22:52:43.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/850743939 shutdown_connections 2024-02-09T22:52:43.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/850743939 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcdac102d70 0x7fcdac10b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/850743939 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac10dc50 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/850743939 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 0x7fcdac102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/850743939 >> 172.21.15.3:0/850743939 conn(0x7fcdac0772e0 msgr2=0x7fcdac0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/850743939 shutdown_connections 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.319+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/850743939 wait complete. 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 Processor -- start 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 -- start start 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 0x7fcdac19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcdac102d70 0x7fcdac19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac197ec0 con 0x7fcdac10b820 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac198030 con 0x7fcdac102d70 2024-02-09T22:52:43.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdb1d4b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1981f0 con 0x7fcdac102450 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac197730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41316/0 (socket says 172.21.15.3:41316) 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 -- 172.21.15.3:0/3387684396 learned_addr learned my addr 172.21.15.3:0/3387684396 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcdac102d70 0x7fcdac19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 0x7fcdac19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 -- 172.21.15.3:0/3387684396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 msgr2=0x7fcdac19d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 0x7fcdac19d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 -- 172.21.15.3:0/3387684396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcdac102d70 msgr2=0x7fcdac19db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcdac102d70 0x7fcdac19db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdabfff640 1 -- 172.21.15.3:0/3387684396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda4009900 con 0x7fcdac10b820 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.320+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 0x7fcdac19d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.321+0000 7fcdabfff640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac197730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcda402c790 tx=0x7fcda4034760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.321+0000 7fcda8ff9640 1 -- 172.21.15.3:0/3387684396 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4003d80 con 0x7fcdac10b820 2024-02-09T22:52:43.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.321+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac198470 con 0x7fcdac10b820 2024-02-09T22:52:43.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.321+0000 7fcda8ff9640 1 -- 172.21.15.3:0/3387684396 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcda4032c70 con 0x7fcdac10b820 2024-02-09T22:52:43.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.321+0000 7fcda8ff9640 1 -- 172.21.15.3:0/3387684396 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40047f0 con 0x7fcdac10b820 2024-02-09T22:52:43.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.321+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac1a45a0 con 0x7fcdac10b820 2024-02-09T22:52:43.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.323+0000 7fcda8ff9640 1 -- 172.21.15.3:0/3387684396 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7fcda400a040 con 0x7fcdac10b820 2024-02-09T22:52:43.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.323+0000 7fcdb49f4740 1 Processor -- start 2024-02-09T22:52:43.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.323+0000 7fcdb49f4740 1 -- start start 2024-02-09T22:52:43.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcda8ff9640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd940a2350 0x7fcd940a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac62999e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac6299a3e0 0x55ac629a2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62955020 con 0x55ac629a2f60 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62908320 con 0x55ac6299a3e0 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62905850 con 0x55ac62999a30 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd940a2350 0x7fcd940a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac62999e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41332/0 (socket says 172.21.15.3:41332) 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcda8ff9640 1 -- 172.21.15.3:0/3387684396 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcda40f0ef0 con 0x7fcdac10b820 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 -- 172.21.15.3:0/2658096981 learned_addr learned my addr 172.21.15.3:0/2658096981 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac6299a3e0 0x55ac629a2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 -- 172.21.15.3:0/2658096981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 msgr2=0x55ac62999e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac62999e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 -- 172.21.15.3:0/2658096981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac6299a3e0 msgr2=0x55ac629a2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac6299a3e0 0x55ac629a2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 -- 172.21.15.3:0/2658096981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac628c3210 con 0x55ac629a2f60 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd940a2350 0x7fcd940a4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcdac198f30 tx=0x7fcd9c0128d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcdabfff640 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a53b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcda40bd700 tx=0x7fcda40bd540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b45aec0ca930d7b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.324+0000 7fcd91ffb640 1 -- 172.21.15.3:0/2658096981 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40a1450 con 0x55ac629a2f60 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcd91ffb640 1 -- 172.21.15.3:0/2658096981 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcda404ac80 con 0x55ac629a2f60 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcd91ffb640 1 -- 172.21.15.3:0/2658096981 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda4032de0 con 0x55ac629a2f60 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 -- 172.21.15.3:0/2658096981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 msgr2=0x55ac629a53b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a53b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fcda40bd700 tx=0x7fcda40bd540 comp rx=0 tx=0).stop 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 -- 172.21.15.3:0/2658096981 shutdown_connections 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac6299a3e0 0x55ac629a2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a53b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/2658096981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac62999e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 -- 172.21.15.3:0/2658096981 >> 172.21.15.3:0/2658096981 conn(0x55ac62994100 msgr2=0x55ac629944e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 -- 172.21.15.3:0/2658096981 shutdown_connections 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.325+0000 7fcdb49f4740 1 -- 172.21.15.3:0/2658096981 wait complete. 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 -- start start 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 Processor -- start 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac6299fec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac6299a3e0 0x55ac629a0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62901030 con 0x55ac629a2f60 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac625d4430 con 0x55ac62999a30 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62590e00 con 0x55ac6299a3e0 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdb49f4740 10 client.0 _ll_register_callbacks cb 0x55ac628f6030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41344/0 (socket says 172.21.15.3:41344) 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 -- 172.21.15.3:0/3145876423 learned_addr learned my addr 172.21.15.3:0/3145876423 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac6299a3e0 0x55ac629a0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac6299fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 -- 172.21.15.3:0/3145876423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac6299a3e0 msgr2=0x55ac629a0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac6299a3e0 0x55ac629a0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 -- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 msgr2=0x55ac6299fec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac6299fec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.326+0000 7fcdabfff640 1 -- 172.21.15.3:0/3145876423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac1a45a0 con 0x55ac629a2f60 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdabfff640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a0940 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcda40bd470 tx=0x7fcda402c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7d92b2885dfa11a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcd8affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcd8affd640 1 -- 172.21.15.3:0/3145876423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcda40bd770 con 0x55ac629a2f60 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcd8affd640 1 -- 172.21.15.3:0/3145876423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcda40c18c0 con 0x55ac629a2f60 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ac626007e0 con 0x55ac629a2f60 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcd8affd640 1 -- 172.21.15.3:0/3145876423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcda40bde70 con 0x55ac629a2f60 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 10 client.16110 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:43.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ac628ff6b0 con 0x55ac629a2f60 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcd8affd640 1 -- 172.21.15.3:0/3145876423 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ac626007e0 con 0x55ac629a2f60 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcd8affd640 1 -- 172.21.15.3:0/3145876423 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcda40c5a80 con 0x55ac629a2f60 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 20 client.16110 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac629b2f60 0x55ac629b5380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 4 client.16110 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.327+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ac6263d720 con 0x55ac629b2f60 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.328+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac629b2f60 0x55ac629b5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.328+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac629b2f60 0x55ac629b5380 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=446ea998e8c75c5f server_cookie=c1a428807a30cdc3 in_seq=0 out_seq=0 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.328+0000 7fcd8affd640 10 client.16110 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcd8affd640 1 -- 172.21.15.3:0/3145876423 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ac6263d720 con 0x55ac629b2f60 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcd8affd640 10 client.16110 handle_command_reply: tid=0 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 client.16110 shutdown 2024-02-09T22:52:43.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 msgr2=0x55ac629a0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a0940 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcda40bd470 tx=0x7fcda402c180 comp rx=0 tx=0).stop 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 shutdown_connections 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac629b2f60 0x55ac629b5380 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac629a2f60 0x55ac629a0940 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac6299a3e0 0x55ac629a0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcd8affd640 0 client.16110 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/3145876423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac6299fec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 >> 172.21.15.3:0/3145876423 conn(0x55ac62994100 msgr2=0x55ac62995e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 shutdown_connections 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 1 -- 172.21.15.3:0/3145876423 wait complete. 2024-02-09T22:52:43.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.329+0000 7fcdb49f4740 20 client.16110 trim_cache size 0 max 16384 2024-02-09T22:52:43.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 Processor -- start 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 -- start start 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac629a25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac629a2f60 0x55ac6299eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62886130 con 0x55ac6299a3e0 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac6298fa40 con 0x55ac62999a30 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac62993f80 con 0x55ac629a2f60 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a0760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41346/0 (socket says 172.21.15.3:41346) 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1306097684 learned_addr learned my addr 172.21.15.3:0/1306097684 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac629a25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1306097684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac629a2f60 msgr2=0x55ac6299eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdabfff640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac629a2f60 0x55ac6299eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac629a2f60 0x55ac6299eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1306097684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 msgr2=0x55ac629a25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac629a25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1306097684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ac6289bc90 con 0x55ac6299a3e0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac629a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdabfff640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac629a2f60 0x55ac6299eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a0760 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c005d00 tx=0x7fcd9c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cda4e33d51f58bb5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.341+0000 7fcd917fa640 1 -- 172.21.15.3:0/1306097684 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c025070 con 0x55ac6299a3e0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcd917fa640 1 -- 172.21.15.3:0/1306097684 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd9c017070 con 0x55ac6299a3e0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcd917fa640 1 -- 172.21.15.3:0/1306097684 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c02a920 con 0x55ac6299a3e0 2024-02-09T22:52:43.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1306097684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 msgr2=0x55ac629a0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a0760 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c005d00 tx=0x7fcd9c01d000 comp rx=0 tx=0).stop 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1306097684 shutdown_connections 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac629a2f60 0x55ac6299eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a0760 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1306097684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac62999a30 0x55ac629a25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1306097684 >> 172.21.15.3:0/1306097684 conn(0x55ac62994100 msgr2=0x55ac6299e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1306097684 shutdown_connections 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1306097684 wait complete. 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.342+0000 7fcdb49f4740 1 -- start start 2024-02-09T22:52:43.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 Processor -- start 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac629a9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a9fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac629a2f60 0x55ac629a7f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac629a85b0 con 0x55ac6299a3e0 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac629a8720 con 0x55ac629a2f60 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ac629a88e0 con 0x55ac62999a30 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdb49f4740 10 client.0 _ll_register_callbacks cb 0x7fcda4004950 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a9fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a9fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41352/0 (socket says 172.21.15.3:41352) 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1725234168 learned_addr learned my addr 172.21.15.3:0/1725234168 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:43.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac629a9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdabfff640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac629a2f60 0x55ac629a7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1725234168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 msgr2=0x55ac629a9a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac629a9a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac629a2f60 msgr2=0x55ac629a7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac629a2f60 0x55ac629a7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 -- 172.21.15.3:0/1725234168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c00e4d0 con 0x55ac6299a3e0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcdaaffd640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a9fa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0207b0 tx=0x7fcd9c0207e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98ee41dea0968fe0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcd8affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcd8affd640 1 -- 172.21.15.3:0/1725234168 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c025040 con 0x55ac6299a3e0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.343+0000 7fcd8affd640 1 -- 172.21.15.3:0/1725234168 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd9c02a5a0 con 0x55ac6299a3e0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcda404ac80 con 0x55ac6299a3e0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcd8affd640 1 -- 172.21.15.3:0/1725234168 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd9c02a760 con 0x55ac6299a3e0 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 10 client.16116 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:43.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55ac628c1a70 con 0x55ac6299a3e0 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcd8affd640 1 -- 172.21.15.3:0/1725234168 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcda404ac80 con 0x55ac6299a3e0 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcd8affd640 1 -- 172.21.15.3:0/1725234168 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fcd9c035a90 con 0x55ac6299a3e0 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 10 client.16116 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 10 client.16116 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 20 client.16116 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac62a0e920 0x55ac62a10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 4 client.16116 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ac6268c890 con 0x55ac62a0e920 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.344+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac62a0e920 0x55ac62a10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcd8affd640 10 client.16116 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdab7fe640 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac62a0e920 0x55ac62a10d40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=91af75fb661a2e2e server_cookie=491f87bb45933ea8 in_seq=0 out_seq=0 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcd8affd640 1 -- 172.21.15.3:0/1725234168 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ac6268c890 con 0x55ac62a0e920 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcd8affd640 10 client.16116 handle_command_reply: tid=0 2024-02-09T22:52:43.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 client.16116 shutdown 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 msgr2=0x55ac629a9fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a9fa0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0207b0 tx=0x7fcd9c0207e0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 shutdown_connections 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ac62a0e920 0x55ac62a10d40 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ac629a2f60 0x55ac629a7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ac6299a3e0 0x55ac629a9fa0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcd8affd640 0 client.16116 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 --2- 172.21.15.3:0/1725234168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ac62999a30 0x55ac629a9a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 >> 172.21.15.3:0/1725234168 conn(0x55ac62994100 msgr2=0x55ac62993240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 shutdown_connections 2024-02-09T22:52:43.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 1 -- 172.21.15.3:0/1725234168 wait complete. 2024-02-09T22:52:43.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.345+0000 7fcdb49f4740 20 client.16116 trim_cache size 0 max 16384 2024-02-09T22:52:43.360 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd940a2350 msgr2=0x7fcd940a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd940a2350 0x7fcd940a4810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcdac198f30 tx=0x7fcd9c0128d0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 msgr2=0x7fcdac197730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac197730 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fcda402c790 tx=0x7fcda4034760 comp rx=0 tx=0).stop 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 shutdown_connections 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd940a2350 0x7fcd940a4810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcdac102d70 0x7fcdac19db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcdac10b820 0x7fcdac197730 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 --2- 172.21.15.3:0/3387684396 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcdac102450 0x7fcdac19d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:43.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 >> 172.21.15.3:0/3387684396 conn(0x7fcdac0772e0 msgr2=0x7fcdac0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:43.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 shutdown_connections 2024-02-09T22:52:43.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:43.346+0000 7fcdb1d4b640 1 -- 172.21.15.3:0/3387684396 wait complete. 2024-02-09T22:52:43.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:43 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1985396075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:52:43.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:43 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1985396075' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:52:43.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36488 5.76 MB/sec warmup 37 sec latency 4884.853 ms 2024-02-09T22:52:44.364 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:52:44.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:52:44.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:52:44.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:44 smithi064 ceph-mon[36638]: pgmap v47: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-02-09T22:52:44.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:44 smithi089 ceph-mon[36665]: pgmap v47: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-02-09T22:52:44.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 -- 172.21.15.3:0/2201214702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14106510 msgr2=0x7f8f1410e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 --2- 172.21.15.3:0/2201214702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14106510 0x7f8f1410e0e0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c009cd0 tx=0x7f8f0c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 -- 172.21.15.3:0/2201214702 shutdown_connections 2024-02-09T22:52:44.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 --2- 172.21.15.3:0/2201214702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14106510 0x7f8f1410e0e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 --2- 172.21.15.3:0/2201214702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f14101df0 0x7f8f14105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 --2- 172.21.15.3:0/2201214702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f14101440 0x7f8f14101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 -- 172.21.15.3:0/2201214702 >> 172.21.15.3:0/2201214702 conn(0x7f8f140fccc0 msgr2=0x7f8f140ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 -- 172.21.15.3:0/2201214702 shutdown_connections 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.481+0000 7f8f194f4640 1 -- 172.21.15.3:0/2201214702 wait complete. 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 Processor -- start 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 -- start start 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f14101440 0x7f8f1419d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f14106510 0x7f8f1419dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 0x7f8f14197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14198110 con 0x7f8f14197640 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14198280 con 0x7f8f14106510 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f194f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f14198440 con 0x7f8f14101440 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 0x7f8f14197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 0x7f8f14197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41364/0 (socket says 172.21.15.3:41364) 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 -- 172.21.15.3:0/3304951350 learned_addr learned my addr 172.21.15.3:0/3304951350 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f127fc640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f14106510 0x7f8f1419dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f14101440 0x7f8f1419d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 -- 172.21.15.3:0/3304951350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f14101440 msgr2=0x7f8f1419d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f14101440 0x7f8f1419d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 -- 172.21.15.3:0/3304951350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f14106510 msgr2=0x7f8f1419dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f14106510 0x7f8f1419dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.482+0000 7f8f137fe640 1 -- 172.21.15.3:0/3304951350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f0c009900 con 0x7f8f14197640 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.483+0000 7f8f137fe640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 0x7f8f14197ac0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c004af0 tx=0x7f8f0c034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.483+0000 7f8efbfff640 1 -- 172.21.15.3:0/3304951350 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c0328c0 con 0x7f8f14197640 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.483+0000 7f8efbfff640 1 -- 172.21.15.3:0/3304951350 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0c003e50 con 0x7f8f14197640 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.483+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f141a4380 con 0x7f8f14197640 2024-02-09T22:52:44.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.483+0000 7f8efbfff640 1 -- 172.21.15.3:0/3304951350 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0c004860 con 0x7f8f14197640 2024-02-09T22:52:44.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.483+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f141a4800 con 0x7f8f14197640 2024-02-09T22:52:44.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8efbfff640 1 -- 172.21.15.3:0/3304951350 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7f8f0c00a040 con 0x7f8f14197640 2024-02-09T22:52:44.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 Processor -- start 2024-02-09T22:52:44.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 -- start start 2024-02-09T22:52:44.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6b62a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 0x55893c6c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c6c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c624000 con 0x55893c6b67e0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c621530 con 0x55893c6c3aa0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c61cd10 con 0x55893c6b5ec0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 0x55893c6c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 0x55893c6c3560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41374/0 (socket says 172.21.15.3:41374) 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f12ffd640 1 -- 172.21.15.3:0/3272075690 learned_addr learned my addr 172.21.15.3:0/3272075690 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f127fc640 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6b62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8efbfff640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8efc0a2350 0x7f8efc0a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.486+0000 7f8f127fc640 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c6c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f137fe640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8efc0a2350 0x7f8efc0a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f12ffd640 1 -- 172.21.15.3:0/3272075690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 msgr2=0x55893c6b62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6b62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f12ffd640 1 -- 172.21.15.3:0/3272075690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 msgr2=0x55893c6c5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c6c5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f12ffd640 1 -- 172.21.15.3:0/3272075690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893c5c3210 con 0x55893c6b67e0 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8efbfff640 1 -- 172.21.15.3:0/3304951350 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f0c0f0ed0 con 0x7f8f14197640 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 0x55893c6c3560 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400bcb0 tx=0x7f8f0400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6a286bff5f1fc07 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8f137fe640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8efc0a2350 0x7f8efc0a4810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0ee640 tx=0x7f8f0c02c0f0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8ef97fa640 1 -- 172.21.15.3:0/3272075690 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f0401a460 con 0x55893c6b67e0 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8ef97fa640 1 -- 172.21.15.3:0/3272075690 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0401aaa0 con 0x55893c6b67e0 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.487+0000 7f8ef97fa640 1 -- 172.21.15.3:0/3272075690 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f04011ae0 con 0x55893c6b67e0 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/3272075690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 msgr2=0x55893c6c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 0x55893c6c3560 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f8f0400bcb0 tx=0x7f8f0400f730 comp rx=0 tx=0).stop 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/3272075690 shutdown_connections 2024-02-09T22:52:44.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c6c5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b67e0 0x55893c6c3560 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/3272075690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6b62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/3272075690 >> 172.21.15.3:0/3272075690 conn(0x55893c6b08c0 msgr2=0x55893c6b0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/3272075690 shutdown_connections 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/3272075690 wait complete. 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.488+0000 7f8f1c1a2740 1 -- start start 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 Processor -- start 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6c1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6c3aa0 0x55893c6c1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c5c1e60 con 0x55893c6b5ec0 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c2f04b0 con 0x55893c6b67e0 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c5a2670 con 0x55893c6c3aa0 2024-02-09T22:52:44.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f1c1a2740 10 client.0 _ll_register_callbacks cb 0x55893c5ba190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41382/0 (socket says 172.21.15.3:41382) 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 -- 172.21.15.3:0/1979382492 learned_addr learned my addr 172.21.15.3:0/1979382492 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6c1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6c3aa0 0x55893c6c1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 -- 172.21.15.3:0/1979382492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6c3aa0 msgr2=0x55893c6c1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6c3aa0 0x55893c6c1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 -- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 msgr2=0x55893c6c1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6c1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.489+0000 7f8f127fc640 1 -- 172.21.15.3:0/1979382492 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f0400b8e0 con 0x55893c6b5ec0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c0ad0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8f080044d0 tx=0x7f8f080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a9ea81ec1e58774 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8ef6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:44.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1979382492 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08016040 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1979382492 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f08005020 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55893c31c860 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1979382492 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f080051e0 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 10 client.16128 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55893c61b390 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1979382492 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55893c31c860 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1979382492 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f8f080116b0 con 0x55893c6b5ec0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 10 client.16128 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 20 client.16128 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c6d3bf0 0x55893c6d6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 4 client.16128 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.490+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55893c36d410 con 0x55893c6d3bf0 2024-02-09T22:52:44.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.491+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c6d3bf0 0x55893c6d6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.491+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c6d3bf0 0x55893c6d6010 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d651865cab55da36 server_cookie=4a489ed3224534c9 in_seq=0 out_seq=0 2024-02-09T22:52:44.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.491+0000 7f8ef6ff5640 10 client.16128 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1979382492 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55893c36d410 con 0x55893c6d3bf0 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8ef6ff5640 10 client.16128 handle_command_reply: tid=0 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 client.16128 shutdown 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 msgr2=0x55893c6c0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c0ad0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f8f080044d0 tx=0x7f8f080046c0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 shutdown_connections 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c6d3bf0 0x55893c6d6010 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6c1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6c3aa0 0x55893c6c1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8ef6ff5640 0 client.16128 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1979382492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c0ad0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 >> 172.21.15.3:0/1979382492 conn(0x55893c6b08c0 msgr2=0x55893c6b2610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.492+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 shutdown_connections 2024-02-09T22:52:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.493+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1979382492 wait complete. 2024-02-09T22:52:44.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.493+0000 7f8f1c1a2740 20 client.16128 trim_cache size 0 max 16384 2024-02-09T22:52:44.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 Processor -- start 2024-02-09T22:52:44.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 -- start start 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6c3200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6bdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 0x55893c6be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c31c860 con 0x55893c6c3aa0 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c6b6ce0 con 0x55893c6b67e0 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c6b2c10 con 0x55893c6b5ec0 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f127fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 0x55893c6be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f127fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 0x55893c6be270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41386/0 (socket says 172.21.15.3:41386) 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f127fc640 1 -- 172.21.15.3:0/2273786953 learned_addr learned my addr 172.21.15.3:0/2273786953 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f127fc640 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6c3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.508+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6bdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f127fc640 1 -- 172.21.15.3:0/2273786953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 msgr2=0x55893c6c3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f127fc640 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6c3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f127fc640 1 -- 172.21.15.3:0/2273786953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 msgr2=0x55893c6bdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f127fc640 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6bdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f127fc640 1 -- 172.21.15.3:0/2273786953 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893c5b7c70 con 0x55893c6c3aa0 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6bdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8f127fc640 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 0x55893c6be270 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8f08001eb0 tx=0x7f8f080092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3638ba2433a736f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8ef8ff9640 1 -- 172.21.15.3:0/2273786953 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f080116b0 con 0x55893c6c3aa0 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8ef8ff9640 1 -- 172.21.15.3:0/2273786953 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f08011810 con 0x55893c6c3aa0 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.509+0000 7f8ef8ff9640 1 -- 172.21.15.3:0/2273786953 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08029790 con 0x55893c6c3aa0 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/2273786953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 msgr2=0x55893c6be270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 0x55893c6be270 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f8f08001eb0 tx=0x7f8f080092d0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/2273786953 shutdown_connections 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6b67e0 0x55893c6bdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6c3aa0 0x55893c6be270 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/2273786953 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b5ec0 0x55893c6c3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/2273786953 >> 172.21.15.3:0/2273786953 conn(0x55893c6b08c0 msgr2=0x55893c6bf3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/2273786953 shutdown_connections 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/2273786953 wait complete. 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- start start 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 Processor -- start 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b67e0 0x55893c6c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c72a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c6ca5a0 con 0x55893c6b5ec0 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c72a640 con 0x55893c6c3aa0 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893c72a780 con 0x55893c6b67e0 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f1c1a2740 10 client.0 _ll_register_callbacks cb 0x7f8f0400b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f127fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f127fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c9790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41390/0 (socket says 172.21.15.3:41390) 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f127fc640 1 -- 172.21.15.3:0/1483472996 learned_addr learned my addr 172.21.15.3:0/1483472996 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.510+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b67e0 0x55893c6c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c72a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 -- 172.21.15.3:0/1483472996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b67e0 msgr2=0x55893c6c9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b67e0 0x55893c6c9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 -- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 msgr2=0x55893c72a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c72a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 -- 172.21.15.3:0/1483472996 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55893c61b390 con 0x55893c6b5ec0 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f127fc640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c9790 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8f0802b040 tx=0x7f8f08029430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d95ac5944760ec8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8ef6ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:44.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1483472996 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f08016030 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1483472996 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8f08000e40 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8f0401aaa0 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1483472996 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f08020890 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f1c1a2740 10 client.16134 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55893c5ddac0 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.511+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1483472996 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f0401aaa0 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1483472996 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f8f08029ce0 con 0x55893c6b5ec0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f1c1a2740 10 client.16134 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f1c1a2740 10 client.16134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f1c1a2740 20 client.16134 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c72bc90 0x55893c72e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f1c1a2740 4 client.16134 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55893c3597b0 con 0x55893c72bc90 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c72bc90 0x55893c72e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8ef6ff5640 10 client.16134 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.512+0000 7f8f12ffd640 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c72bc90 0x55893c72e0b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6959c016613ebe50 server_cookie=24a9cdce7d66d219 in_seq=0 out_seq=0 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8ef6ff5640 1 -- 172.21.15.3:0/1483472996 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55893c3597b0 con 0x55893c72bc90 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8ef6ff5640 10 client.16134 handle_command_reply: tid=0 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 client.16134 shutdown 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 msgr2=0x55893c6c9790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c9790 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f8f0802b040 tx=0x7f8f08029430 comp rx=0 tx=0).stop 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 shutdown_connections 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893c72bc90 0x55893c72e0b0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55893c6c3aa0 0x55893c72a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55893c6b67e0 0x55893c6c9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 --2- 172.21.15.3:0/1483472996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55893c6b5ec0 0x55893c6c9790 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 >> 172.21.15.3:0/1483472996 conn(0x55893c6b08c0 msgr2=0x55893c6af090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8ef6ff5640 0 client.16134 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 shutdown_connections 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 1 -- 172.21.15.3:0/1483472996 wait complete. 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.513+0000 7f8f1c1a2740 20 client.16134 trim_cache size 0 max 16384 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "5ec574ae-5cc4-4e6b-a2ae-f0fc615937e6", 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:52:44.524 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:52:44.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8efc0a2350 msgr2=0x7f8efc0a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8efc0a2350 0x7f8efc0a4810 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c0ee640 tx=0x7f8f0c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 msgr2=0x7f8f14197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 0x7f8f14197ac0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f8f0c004af0 tx=0x7f8f0c034cc0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 shutdown_connections 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8efc0a2350 0x7f8efc0a4810 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f14106510 0x7f8f1419dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f14197640 0x7f8f14197ac0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 --2- 172.21.15.3:0/3304951350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f14101440 0x7f8f1419d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 >> 172.21.15.3:0/3304951350 conn(0x7f8f140fccc0 msgr2=0x7f8f140fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 shutdown_connections 2024-02-09T22:52:44.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.514+0000 7f8f194f4640 1 -- 172.21.15.3:0/3304951350 wait complete. 2024-02-09T22:52:44.527 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:44 smithi003 ceph-mon[32678]: pgmap v47: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 172 KiB/s rd, 4.1 MiB/s wr, 124 op/s 2024-02-09T22:52:44.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:52:44.529 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:52:44.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 36663 5.64 MB/sec warmup 38 sec latency 1459.984 ms 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 -- 172.21.15.3:0/1459101083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4101430 msgr2=0x7efff4104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 --2- 172.21.15.3:0/1459101083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4101430 0x7efff4104d40 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7effe4009930 tx=0x7effe402bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 -- 172.21.15.3:0/1459101083 shutdown_connections 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 --2- 172.21.15.3:0/1459101083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efff4105420 0x7efff410cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 --2- 172.21.15.3:0/1459101083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4101430 0x7efff4104d40 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 --2- 172.21.15.3:0/1459101083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff4100a80 0x7efff4100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 -- 172.21.15.3:0/1459101083 >> 172.21.15.3:0/1459101083 conn(0x7efff4076630 msgr2=0x7efff40fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 -- 172.21.15.3:0/1459101083 shutdown_connections 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.634+0000 7efffc68f640 1 -- 172.21.15.3:0/1459101083 wait complete. 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 Processor -- start 2024-02-09T22:52:44.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 -- start start 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efff4100a80 0x7efff41986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 0x7efff4198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff41991b0 0x7efff41a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff41a3e00 con 0x7efff4105420 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff41a3f70 con 0x7efff4100a80 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffc68f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff41a4130 con 0x7efff41991b0 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efff9c2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 0x7efff4198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efff9c2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 0x7efff4198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41412/0 (socket says 172.21.15.3:41412) 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efff9c2a640 1 -- 172.21.15.3:0/766356188 learned_addr learned my addr 172.21.15.3:0/766356188 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffa42b640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efff4100a80 0x7efff41986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.635+0000 7efffac2c640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff41991b0 0x7efff41a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efff9c2a640 1 -- 172.21.15.3:0/766356188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff41991b0 msgr2=0x7efff41a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efff9c2a640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff41991b0 0x7efff41a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efff9c2a640 1 -- 172.21.15.3:0/766356188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efff4100a80 msgr2=0x7efff41986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efff9c2a640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efff4100a80 0x7efff41986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efff9c2a640 1 -- 172.21.15.3:0/766356188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff0009840 con 0x7efff4105420 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efff9c2a640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 0x7efff4198c20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7effe4000c00 tx=0x7effe402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7effeb7fe640 1 -- 172.21.15.3:0/766356188 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe40042d0 con 0x7efff4105420 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7effeb7fe640 1 -- 172.21.15.3:0/766356188 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effe4004430 con 0x7efff4105420 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe4009560 con 0x7efff4105420 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7effeb7fe640 1 -- 172.21.15.3:0/766356188 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe4003c90 con 0x7efff4105420 2024-02-09T22:52:44.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.636+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efff41a4730 con 0x7efff4105420 2024-02-09T22:52:44.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.638+0000 7effeb7fe640 1 -- 172.21.15.3:0/766356188 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 25) v1 ==== 137000+0+0 (secure 0 0 0) 0x7effe4004df0 con 0x7efff4105420 2024-02-09T22:52:44.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effeb7fe640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effe00a2350 0x7effe00a4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 Processor -- start 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effeb7fe640 1 -- 172.21.15.3:0/766356188 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7effe40f01b0 con 0x7efff4105420 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7efffa42b640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effe00a2350 0x7effe00a4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 -- start start 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f796d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f7a2300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f751d00 con 0x55f38f79feb0 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f7062f0 con 0x55f38f797330 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.639+0000 7effff25e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f703820 con 0x55f38f796980 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f7a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efff9c2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f7a2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41422/0 (socket says 172.21.15.3:41422) 2024-02-09T22:52:44.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 -- 172.21.15.3:0/2029525606 learned_addr learned my addr 172.21.15.3:0/2029525606 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffa42b640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effe00a2350 0x7effe00a4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efff0005fd0 tx=0x7efff0009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffa42b640 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f796d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 -- 172.21.15.3:0/2029525606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 msgr2=0x55f38f796d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f796d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 -- 172.21.15.3:0/2029525606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 msgr2=0x55f38f79f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 -- 172.21.15.3:0/2029525606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f38f6c0ff0 con 0x55f38f79feb0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efff9c2a640 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7efffac2c640 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f7a2300 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7effec009c20 tx=0x7effec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f70e2a3bada532 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7effe8ff9640 1 -- 172.21.15.3:0/2029525606 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec019630 con 0x55f38f79feb0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7effe8ff9640 1 -- 172.21.15.3:0/2029525606 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effec009e90 con 0x55f38f79feb0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.640+0000 7effe8ff9640 1 -- 172.21.15.3:0/2029525606 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec00eaa0 con 0x55f38f79feb0 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 -- 172.21.15.3:0/2029525606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 msgr2=0x55f38f7a2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f7a2300 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7effec009c20 tx=0x7effec011810 comp rx=0 tx=0).stop 2024-02-09T22:52:44.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 -- 172.21.15.3:0/2029525606 shutdown_connections 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f7a2300 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 --2- 172.21.15.3:0/2029525606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f796d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 -- 172.21.15.3:0/2029525606 >> 172.21.15.3:0/2029525606 conn(0x55f38f791050 msgr2=0x55f38f791430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 -- 172.21.15.3:0/2029525606 shutdown_connections 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 -- 172.21.15.3:0/2029525606 wait complete. 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.641+0000 7effff25e740 1 -- start start 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 Processor -- start 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f79cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f79d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f6ff000 con 0x55f38f79feb0 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f3d2430 con 0x55f38f797330 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f38ee00 con 0x55f38f796980 2024-02-09T22:52:44.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7effff25e740 10 client.0 _ll_register_callbacks cb 0x55f38f6f37f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7efffa42b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f79cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7efffa42b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f79cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:45026/0 (socket says 172.21.15.3:45026) 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7efff9c2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7efffa42b640 1 -- 172.21.15.3:0/1913554247 learned_addr learned my addr 172.21.15.3:0/1913554247 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.642+0000 7efffac2c640 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f79d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7efffa42b640 1 -- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 msgr2=0x55f38f79d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7efffa42b640 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7efffa42b640 1 -- 172.21.15.3:0/1913554247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 msgr2=0x55f38f79d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7efffa42b640 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f79d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7efffa42b640 1 -- 172.21.15.3:0/1913554247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effec0098d0 con 0x55f38f796980 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7efffa42b640 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f79cdb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efff0004180 tx=0x7efff0004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=67243f5a5ce7100e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effddffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effddffb640 1 -- 172.21.15.3:0/1913554247 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff000fcc0 con 0x55f38f796980 2024-02-09T22:52:44.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effddffb640 1 -- 172.21.15.3:0/1913554247 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efff000fe20 con 0x55f38f796980 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f38f3fe7e0 con 0x55f38f796980 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effddffb640 1 -- 172.21.15.3:0/1913554247 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7efff0028440 con 0x55f38f796980 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effff25e740 10 client.25669 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55f38f6fd680 con 0x55f38f796980 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.643+0000 7effddffb640 1 -- 172.21.15.3:0/1913554247 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f38f3fe7e0 con 0x55f38f796980 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effddffb640 1 -- 172.21.15.3:0/1913554247 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7efff001e730 con 0x55f38f796980 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effff25e740 10 client.25669 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effff25e740 10 client.25669 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effff25e740 20 client.25669 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effff25e740 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f7b04e0 0x55f38f7b2900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effff25e740 4 client.25669 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f38f43b720 con 0x55f38f7b04e0 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7efff9c2a640 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f7b04e0 0x55f38f7b2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7effddffb640 10 client.25669 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.644+0000 7efff9c2a640 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f7b04e0 0x55f38f7b2900 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa228d38dc43b92f server_cookie=353f55844d123fb8 in_seq=0 out_seq=0 2024-02-09T22:52:44.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effddffb640 1 -- 172.21.15.3:0/1913554247 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f38f43b720 con 0x55f38f7b04e0 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effddffb640 10 client.25669 handle_command_reply: tid=0 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 client.25669 shutdown 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 msgr2=0x55f38f79cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f79cdb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7efff0004180 tx=0x7efff0004ab0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 shutdown_connections 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f7b04e0 0x55f38f7b2900 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f797330 0x55f38f79d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f79d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 --2- 172.21.15.3:0/1913554247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f796980 0x55f38f79cdb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effddffb640 0 client.25669 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 >> 172.21.15.3:0/1913554247 conn(0x55f38f791050 msgr2=0x55f38f792da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 shutdown_connections 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 1 -- 172.21.15.3:0/1913554247 wait complete. 2024-02-09T22:52:44.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.645+0000 7effff25e740 20 client.25669 trim_cache size 0 max 16384 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 Processor -- start 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 -- start start 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f790070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 0x55f38f7a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f79feb0 0x55f38f7a7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f683fd0 con 0x55f38f797330 2024-02-09T22:52:44.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f7acea0 con 0x55f38f796980 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7effff25e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f7a22a0 con 0x55f38f79feb0 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 0x55f38f7a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efffa42b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f790070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 0x55f38f7a6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41438/0 (socket says 172.21.15.3:41438) 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 -- 172.21.15.3:0/2164726583 learned_addr learned my addr 172.21.15.3:0/2164726583 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efffac2c640 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f79feb0 0x55f38f7a7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 -- 172.21.15.3:0/2164726583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f79feb0 msgr2=0x55f38f7a7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f79feb0 0x55f38f7a7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 -- 172.21.15.3:0/2164726583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 msgr2=0x55f38f790070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f790070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efff9c2a640 1 -- 172.21.15.3:0/2164726583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f38f699b30 con 0x55f38f797330 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.661+0000 7efffa42b640 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f790070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.662+0000 7efff9c2a640 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 0x55f38f7a6ee0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7effe402c170 tx=0x7effe40a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4fed7e09a58489 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.662+0000 7effdffff640 1 -- 172.21.15.3:0/2164726583 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe40bd190 con 0x55f38f797330 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.662+0000 7effdffff640 1 -- 172.21.15.3:0/2164726583 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effe40bca90 con 0x55f38f797330 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.662+0000 7effdffff640 1 -- 172.21.15.3:0/2164726583 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effe40bae20 con 0x55f38f797330 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.662+0000 7effff25e740 1 -- 172.21.15.3:0/2164726583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 msgr2=0x55f38f7a6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.662+0000 7effff25e740 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 0x55f38f7a6ee0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7effe402c170 tx=0x7effe40a7040 comp rx=0 tx=0).stop 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- 172.21.15.3:0/2164726583 shutdown_connections 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f79feb0 0x55f38f7a7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f797330 0x55f38f7a6ee0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 --2- 172.21.15.3:0/2164726583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f790070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- 172.21.15.3:0/2164726583 >> 172.21.15.3:0/2164726583 conn(0x55f38f791050 msgr2=0x55f38f78f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- 172.21.15.3:0/2164726583 shutdown_connections 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- 172.21.15.3:0/2164726583 wait complete. 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- start start 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 Processor -- start 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f7a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f797330 0x55f38f7a6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f806a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f7a6b10 con 0x55f38f79feb0 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f806f70 con 0x55f38f796980 2024-02-09T22:52:44.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f38f8070b0 con 0x55f38f797330 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7effff25e740 10 client.0 _ll_register_callbacks cb 0x7effec009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7efffac2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f806a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7efffa42b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f7a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7efffac2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f806a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41452/0 (socket says 172.21.15.3:41452) 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7efffac2c640 1 -- 172.21.15.3:0/1973222914 learned_addr learned my addr 172.21.15.3:0/1973222914 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.663+0000 7efff9c2a640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f797330 0x55f38f7a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffac2c640 1 -- 172.21.15.3:0/1973222914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f797330 msgr2=0x55f38f7a6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffac2c640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f797330 0x55f38f7a6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffac2c640 1 -- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 msgr2=0x55f38f7a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffac2c640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f7a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffac2c640 1 -- 172.21.15.3:0/1973222914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f38f6fd680 con 0x55f38f79feb0 2024-02-09T22:52:44.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffa42b640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f7a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efff9c2a640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f797330 0x55f38f7a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7efffac2c640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f806a30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7effec009c20 tx=0x7effec0053b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff5a40deff80c43 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effddffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effddffb640 1 -- 172.21.15.3:0/1973222914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effec004220 con 0x55f38f79feb0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effddffb640 1 -- 172.21.15.3:0/1973222914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7effec02a070 con 0x55f38f79feb0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effec009e90 con 0x55f38f79feb0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effddffb640 1 -- 172.21.15.3:0/1973222914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7effec004380 con 0x55f38f79feb0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effff25e740 10 client.16152 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x7effe4009560 con 0x55f38f79feb0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effddffb640 1 -- 172.21.15.3:0/1973222914 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effec009e90 con 0x55f38f79feb0 2024-02-09T22:52:44.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.664+0000 7effddffb640 1 -- 172.21.15.3:0/1973222914 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7effec004af0 con 0x55f38f79feb0 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 10 client.16152 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 10 client.16152 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 20 client.16152 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f808850 0x55f38f80ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 4 client.16152 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f38f48a890 con 0x55f38f808850 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7efffa42b640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f808850 0x55f38f80ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7efffa42b640 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f808850 0x55f38f80ac70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5125a05bb5f086c8 server_cookie=4acf98bedb2d378c in_seq=0 out_seq=0 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effddffb640 10 client.16152 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effddffb640 1 -- 172.21.15.3:0/1973222914 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f38f48a890 con 0x55f38f808850 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effddffb640 10 client.16152 handle_command_reply: tid=0 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 client.16152 shutdown 2024-02-09T22:52:44.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 msgr2=0x55f38f806a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f806a30 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7effec009c20 tx=0x7effec0053b0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 shutdown_connections 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f38f808850 0x55f38f80ac70 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f38f79feb0 0x55f38f806a30 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effddffb640 0 client.16152 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f38f797330 0x55f38f7a6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 --2- 172.21.15.3:0/1973222914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f38f796980 0x55f38f7a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 >> 172.21.15.3:0/1973222914 conn(0x55f38f791050 msgr2=0x55f38f7901f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.665+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 shutdown_connections 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7effff25e740 1 -- 172.21.15.3:0/1973222914 wait complete. 2024-02-09T22:52:44.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7effff25e740 20 client.16152 trim_cache size 0 max 16384 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: "5ec574ae-5cc4-4e6b-a2ae-f0fc615937e6": { 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "5ec574ae-5cc4-4e6b-a2ae-f0fc615937e6", 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:52:44.679 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effe00a2350 msgr2=0x7effe00a4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effe00a2350 0x7effe00a4810 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efff0005fd0 tx=0x7efff0009500 comp rx=0 tx=0).stop 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 msgr2=0x7efff4198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 0x7efff4198c20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7effe4000c00 tx=0x7effe402c5d0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 shutdown_connections 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effe00a2350 0x7effe00a4810 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efff41991b0 0x7efff41a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efff4105420 0x7efff4198c20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 --2- 172.21.15.3:0/766356188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efff4100a80 0x7efff41986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 >> 172.21.15.3:0/766356188 conn(0x7efff4076630 msgr2=0x7efff4103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 shutdown_connections 2024-02-09T22:52:44.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:52:44.666+0000 7efffc68f640 1 -- 172.21.15.3:0/766356188 wait complete. 2024-02-09T22:52:44.682 INFO:tasks.cephfs.filesystem:scrub status for tag:5ec574ae-5cc4-4e6b-a2ae-f0fc615937e6 - {'path': '/', 'tag': '5ec574ae-5cc4-4e6b-a2ae-f0fc615937e6', 'options': 'recursive,force'} 2024-02-09T22:52:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:52:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:45 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.621 INFO:tasks.workunit.client.0.smithi003.stdout: 1 38869 5.81 MB/sec warmup 39 sec latency 297.639 ms 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:45 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:52:45.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:45.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:45 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:46 smithi089 ceph-mon[36665]: pgmap v48: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-02-09T22:52:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:46 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:52:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:46 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:46 smithi089 ceph-mon[36665]: mgrmap e26: x(active, since 90s), standbys: z, y 2024-02-09T22:52:46.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 38869 5.67 MB/sec warmup 40 sec latency 1297.777 ms 2024-02-09T22:52:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:46 smithi003 ceph-mon[32678]: pgmap v48: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-02-09T22:52:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:46 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:52:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:46 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:46 smithi003 ceph-mon[32678]: mgrmap e26: x(active, since 90s), standbys: z, y 2024-02-09T22:52:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:46 smithi064 ceph-mon[36638]: pgmap v48: 97 pgs: 97 active+clean; 26 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 4.1 MiB/s wr, 120 op/s 2024-02-09T22:52:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:46 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:52:46.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:46 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:52:46.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:46 smithi064 ceph-mon[36638]: mgrmap e26: x(active, since 90s), standbys: z, y 2024-02-09T22:52:47.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 38869 5.53 MB/sec warmup 41 sec latency 2297.842 ms 2024-02-09T22:52:48.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:48 smithi089 ceph-mon[36665]: pgmap v49: 97 pgs: 97 active+clean; 27 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 154 op/s 2024-02-09T22:52:48.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 38869 5.40 MB/sec warmup 42 sec latency 3297.931 ms 2024-02-09T22:52:48.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:48 smithi003 ceph-mon[32678]: pgmap v49: 97 pgs: 97 active+clean; 27 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 154 op/s 2024-02-09T22:52:48.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:48 smithi064 ceph-mon[36638]: pgmap v49: 97 pgs: 97 active+clean; 27 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 201 KiB/s rd, 4.2 MiB/s wr, 154 op/s 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.807+0000 7f7942125640 1 -- 172.21.15.3:0/2302093111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c0696f0 msgr2=0x7f793c105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.807+0000 7f7942125640 1 --2- 172.21.15.3:0/2302093111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c0696f0 0x7f793c105e30 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f7930009930 tx=0x7f793002bc30 comp rx=0 tx=0).stop 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 -- 172.21.15.3:0/2302093111 shutdown_connections 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 --2- 172.21.15.3:0/2302093111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c106510 0x7f793c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 --2- 172.21.15.3:0/2302093111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c0696f0 0x7f793c105e30 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 --2- 172.21.15.3:0/2302093111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f793c068d40 0x7f793c069120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 -- 172.21.15.3:0/2302093111 >> 172.21.15.3:0/2302093111 conn(0x7f793c077220 msgr2=0x7f793c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 -- 172.21.15.3:0/2302093111 shutdown_connections 2024-02-09T22:52:48.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 -- 172.21.15.3:0/2302093111 wait complete. 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 Processor -- start 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.808+0000 7f7942125640 1 -- start start 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f7942125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 0x7f793c1000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f7942125640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f793c106510 0x7f793c100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f7942125640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c100b70 0x7f793c1a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f7942125640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793c1a6070 con 0x7f793c068d40 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f7942125640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793c1a61e0 con 0x7f793c100b70 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f7942125640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f793c1a63a0 con 0x7f793c106510 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 0x7f793c1000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 0x7f793c1000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54382/0 (socket says 172.21.15.3:54382) 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 -- 172.21.15.3:0/1274113699 learned_addr learned my addr 172.21.15.3:0/1274113699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793affd640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f793c106510 0x7f793c100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793bfff640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c100b70 0x7f793c1a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 -- 172.21.15.3:0/1274113699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f793c106510 msgr2=0x7f793c100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f793c106510 0x7f793c100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 -- 172.21.15.3:0/1274113699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c100b70 msgr2=0x7f793c1a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c100b70 0x7f793c1a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 -- 172.21.15.3:0/1274113699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930009560 con 0x7f793c068d40 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793bfff640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c100b70 0x7f793c1a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.809+0000 7f793b7fe640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 0x7f793c1000f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f792c00e9f0 tx=0x7f792c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.810+0000 7f7938ff9640 1 -- 172.21.15.3:0/1274113699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c009720 con 0x7f793c068d40 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.810+0000 7f7938ff9640 1 -- 172.21.15.3:0/1274113699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f792c00cd80 con 0x7f793c068d40 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.810+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f793c1a6680 con 0x7f793c068d40 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.810+0000 7f7938ff9640 1 -- 172.21.15.3:0/1274113699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c010690 con 0x7f793c068d40 2024-02-09T22:52:48.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.810+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793c1a6b10 con 0x7f793c068d40 2024-02-09T22:52:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.812+0000 7f7938ff9640 1 -- 172.21.15.3:0/1274113699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f792c0046c0 con 0x7f793c068d40 2024-02-09T22:52:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.812+0000 7f7944dd9740 1 Processor -- start 2024-02-09T22:52:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.812+0000 7f7938ff9640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79240a2250 0x7f79240a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.822 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.812+0000 7f7944dd9740 1 -- start start 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793bfff640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79240a2250 0x7f79240a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8839490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7938ff9640 1 -- 172.21.15.3:0/1274113699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f792c014070 con 0x7f793c068d40 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd8842260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8844b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7944dd9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd87a7a00 con 0x563fd88427a0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7944dd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd87a4f30 con 0x563fd88399d0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f7944dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd87a0710 con 0x563fd88390b0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8844b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8839490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8844b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54390/0 (socket says 172.21.15.3:54390) 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793bfff640 1 -- 172.21.15.3:0/284733653 learned_addr learned my addr 172.21.15.3:0/284733653 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793affd640 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd8842260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.823 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 -- 172.21.15.3:0/284733653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 msgr2=0x563fd8842260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd8842260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 -- 172.21.15.3:0/284733653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 msgr2=0x563fd8844b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8844b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793bfff640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79240a2250 0x7f79240a4710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7934015730 tx=0x7f7934015b30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 -- 172.21.15.3:0/284733653 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563fd87653b0 con 0x563fd88390b0 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.813+0000 7f793b7fe640 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8839490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f792c0cbbf0 tx=0x7f792c09a8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d11bcc2dabab59d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7921ffb640 1 -- 172.21.15.3:0/284733653 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c09efe0 con 0x563fd88390b0 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7921ffb640 1 -- 172.21.15.3:0/284733653 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f792c0a3630 con 0x563fd88390b0 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7921ffb640 1 -- 172.21.15.3:0/284733653 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f792c09a2e0 con 0x563fd88390b0 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 -- 172.21.15.3:0/284733653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 msgr2=0x563fd8839490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8839490 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f792c0cbbf0 tx=0x7f792c09a8b0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 -- 172.21.15.3:0/284733653 shutdown_connections 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd8842260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.824 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8844b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 --2- 172.21.15.3:0/284733653 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8839490 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 -- 172.21.15.3:0/284733653 >> 172.21.15.3:0/284733653 conn(0x563fd88339d0 msgr2=0x563fd8833db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 -- 172.21.15.3:0/284733653 shutdown_connections 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.814+0000 7f7944dd9740 1 -- 172.21.15.3:0/284733653 wait complete. 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 -- start start 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 Processor -- start 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88390b0 0x563fd883f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd883fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88427a0 0x563fd8840250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd8745b90 con 0x563fd88399d0 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd8474250 con 0x563fd88390b0 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd8726200 con 0x563fd88427a0 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f7944dd9740 10 client.0 _ll_register_callbacks cb 0x563fd875c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:48.825 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f793affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd883fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f793affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd883fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54406/0 (socket says 172.21.15.3:54406) 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f793affd640 1 -- 172.21.15.3:0/3824907025 learned_addr learned my addr 172.21.15.3:0/3824907025 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.815+0000 7f793bfff640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88427a0 0x563fd8840250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793b7fe640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88390b0 0x563fd883f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793affd640 1 -- 172.21.15.3:0/3824907025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88427a0 msgr2=0x563fd8840250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793affd640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88427a0 0x563fd8840250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793affd640 1 -- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88390b0 msgr2=0x563fd883f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793affd640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88390b0 0x563fd883f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793affd640 1 -- 172.21.15.3:0/3824907025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793c1a6b10 con 0x563fd88399d0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793bfff640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88427a0 0x563fd8840250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f793affd640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd883fd10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7930009900 tx=0x7f793002ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a2734191f00858a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f791affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f791affd640 1 -- 172.21.15.3:0/3824907025 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f793002c640 con 0x563fd88399d0 2024-02-09T22:52:48.826 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f791affd640 1 -- 172.21.15.3:0/3824907025 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7930004050 con 0x563fd88399d0 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563fd87c6ee0 con 0x563fd88399d0 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f791affd640 1 -- 172.21.15.3:0/3824907025 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7930004210 con 0x563fd88399d0 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f7944dd9740 10 client.16164 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.816+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563fd879ed90 con 0x563fd88399d0 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f791affd640 1 -- 172.21.15.3:0/3824907025 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563fd87c6ee0 con 0x563fd88399d0 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f791affd640 1 -- 172.21.15.3:0/3824907025 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f793003c040 con 0x563fd88399d0 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f7944dd9740 10 client.16164 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f7944dd9740 10 client.16164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f7944dd9740 20 client.16164 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f7944dd9740 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd8852a10 0x563fd8854e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f7944dd9740 4 client.16164 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:52:48.827 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563fd84a0600 con 0x563fd8852a10 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.817+0000 7f793b7fe640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd8852a10 0x563fd8854e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f793b7fe640 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd8852a10 0x563fd8854e30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b5b9daeaa2e3a6e server_cookie=2f1833faff992b29 in_seq=0 out_seq=0 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f791affd640 10 client.16164 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f791affd640 1 -- 172.21.15.3:0/3824907025 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563fd84a0600 con 0x563fd8852a10 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f791affd640 10 client.16164 handle_command_reply: tid=0 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 client.16164 shutdown 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 msgr2=0x563fd883fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd883fd10 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f7930009900 tx=0x7f793002ca10 comp rx=0 tx=0).stop 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 shutdown_connections 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd8852a10 0x563fd8854e30 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88427a0 0x563fd8840250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd883fd10 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.828 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f791affd640 0 client.16164 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 --2- 172.21.15.3:0/3824907025 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88390b0 0x563fd883f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.818+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 >> 172.21.15.3:0/3824907025 conn(0x563fd88339d0 msgr2=0x563fd8835720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.819+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 shutdown_connections 2024-02-09T22:52:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.819+0000 7f7944dd9740 1 -- 172.21.15.3:0/3824907025 wait complete. 2024-02-09T22:52:48.829 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.819+0000 7f7944dd9740 20 client.16164 trim_cache size 0 max 16384 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 Processor -- start 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 -- start start 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8841f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd883a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8840070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd882f310 con 0x563fd88427a0 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd884f450 con 0x563fd88399d0 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f7944dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd88349d0 con 0x563fd88390b0 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f793bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8840070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f793bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8840070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54414/0 (socket says 172.21.15.3:54414) 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f793bfff640 1 -- 172.21.15.3:0/4083409784 learned_addr learned my addr 172.21.15.3:0/4083409784 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:48.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.832+0000 7f793b7fe640 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8841f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f793bfff640 1 -- 172.21.15.3:0/4083409784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 msgr2=0x563fd8841f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f793bfff640 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8841f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f793bfff640 1 -- 172.21.15.3:0/4083409784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 msgr2=0x563fd883a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f793bfff640 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd883a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f793bfff640 1 -- 172.21.15.3:0/4083409784 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563fd873b820 con 0x563fd88427a0 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f793bfff640 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8840070 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x563fd8834b60 tx=0x7f7934018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3d6f3fcdb3af03 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f79217fa640 1 -- 172.21.15.3:0/4083409784 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934015310 con 0x563fd88427a0 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f79217fa640 1 -- 172.21.15.3:0/4083409784 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7934025070 con 0x563fd88427a0 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.833+0000 7f79217fa640 1 -- 172.21.15.3:0/4083409784 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7934021d00 con 0x563fd88427a0 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- 172.21.15.3:0/4083409784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 msgr2=0x563fd8840070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8840070 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x563fd8834b60 tx=0x7f7934018000 comp rx=0 tx=0).stop 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- 172.21.15.3:0/4083409784 shutdown_connections 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88399d0 0x563fd883a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88427a0 0x563fd8840070 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- 172.21.15.3:0/4083409784 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8841f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- 172.21.15.3:0/4083409784 >> 172.21.15.3:0/4083409784 conn(0x563fd88339d0 msgr2=0x563fd883e0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- 172.21.15.3:0/4083409784 shutdown_connections 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- 172.21.15.3:0/4083409784 wait complete. 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- start start 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 Processor -- start 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8849950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd8847d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88427a0 0x563fd8848430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd8848b30 con 0x563fd88399d0 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd88a8f40 con 0x563fd88427a0 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563fd88a9080 con 0x563fd88390b0 2024-02-09T22:52:48.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.834+0000 7f7944dd9740 10 client.0 _ll_register_callbacks cb 0x7f7930009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd8847d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd8847d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54422/0 (socket says 172.21.15.3:54422) 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 -- 172.21.15.3:0/1529939696 learned_addr learned my addr 172.21.15.3:0/1529939696 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793b7fe640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8849950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 -- 172.21.15.3:0/1529939696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 msgr2=0x563fd8849950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8849950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793bfff640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88427a0 0x563fd8848430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 -- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88427a0 msgr2=0x563fd8848430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88427a0 0x563fd8848430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 -- 172.21.15.3:0/1529939696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f793401b070 con 0x563fd88399d0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f793affd640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd8847d40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7930003880 tx=0x7f7930003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1851e6f570dcbe10 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f791affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:52:48.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f791affd640 1 -- 172.21.15.3:0/1529939696 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7930055070 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f791affd640 1 -- 172.21.15.3:0/1529939696 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7930000e40 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f792c0a3630 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f791affd640 1 -- 172.21.15.3:0/1529939696 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f793002c9f0 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f7944dd9740 10 client.16170 fetch_fsmap learned FSMap version 29 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.835+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x563fd87616f0 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f791affd640 1 -- 172.21.15.3:0/1529939696 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f792c0a3630 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f791affd640 1 -- 172.21.15.3:0/1529939696 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 29) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f7930058030 con 0x563fd88399d0 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f7944dd9740 10 client.16170 fetch_fsmap finished waiting for FSMap version 29 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f7944dd9740 10 client.16170 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f7944dd9740 20 client.16170 populate_metadata read hostname 'smithi003' 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f7944dd9740 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd88aa530 0x563fd88ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f7944dd9740 4 client.16170 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563fd84f11b0 con 0x563fd88aa530 2024-02-09T22:52:48.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f793b7fe640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd88aa530 0x563fd88ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:52:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f791affd640 10 client.16170 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:48.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.836+0000 7f793b7fe640 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd88aa530 0x563fd88ac950 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=58a7f4af92c3eac4 server_cookie=3b2b1bca29af6807 in_seq=0 out_seq=0 2024-02-09T22:52:48.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f791affd640 1 -- 172.21.15.3:0/1529939696 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563fd84f11b0 con 0x563fd88aa530 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f791affd640 10 client.16170 handle_command_reply: tid=0 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 client.16170 shutdown 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 msgr2=0x563fd8847d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd8847d40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f7930003880 tx=0x7f7930003d40 comp rx=0 tx=0).stop 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 shutdown_connections 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563fd88aa530 0x563fd88ac950 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563fd88427a0 0x563fd8848430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563fd88399d0 0x563fd8847d40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f791affd640 0 client.16170 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 --2- 172.21.15.3:0/1529939696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563fd88390b0 0x563fd8849950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 >> 172.21.15.3:0/1529939696 conn(0x563fd88339d0 msgr2=0x563fd8834df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:48.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 shutdown_connections 2024-02-09T22:52:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 1 -- 172.21.15.3:0/1529939696 wait complete. 2024-02-09T22:52:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.858+0000 7f7944dd9740 20 client.16170 trim_cache size 0 max 16384 2024-02-09T22:52:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:52:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:52:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:52:48.865 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79240a2250 msgr2=0x7f79240a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79240a2250 0x7f79240a4710 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7934015730 tx=0x7f7934015b30 comp rx=0 tx=0).stop 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 msgr2=0x7f793c1000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 0x7f793c1000f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f792c00e9f0 tx=0x7f792c00edf0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 shutdown_connections 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79240a2250 0x7f79240a4710 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f793c100b70 0x7f793c1a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f793c106510 0x7f793c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 --2- 172.21.15.3:0/1274113699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f793c068d40 0x7f793c1000f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:52:48.866 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 >> 172.21.15.3:0/1274113699 conn(0x7f793c077220 msgr2=0x7f793c0fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:52:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 shutdown_connections 2024-02-09T22:52:48.867 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:52:48.859+0000 7f7942125640 1 -- 172.21.15.3:0/1274113699 wait complete. 2024-02-09T22:52:49.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 38869 5.27 MB/sec warmup 43 sec latency 4298.073 ms 2024-02-09T22:52:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:50 smithi089 ceph-mon[36665]: pgmap v50: 97 pgs: 97 active+clean; 27 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 166 KiB/s wr, 51 op/s 2024-02-09T22:52:50.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 39497 5.26 MB/sec warmup 44 sec latency 4765.915 ms 2024-02-09T22:52:50.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:50 smithi003 ceph-mon[32678]: pgmap v50: 97 pgs: 97 active+clean; 27 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 166 KiB/s wr, 51 op/s 2024-02-09T22:52:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:50 smithi064 ceph-mon[36638]: pgmap v50: 97 pgs: 97 active+clean; 27 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 166 KiB/s wr, 51 op/s 2024-02-09T22:52:51.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 39846 5.15 MB/sec warmup 45 sec latency 754.256 ms 2024-02-09T22:52:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:52 smithi089 ceph-mon[36665]: pgmap v51: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 199 KiB/s wr, 73 op/s 2024-02-09T22:52:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:52:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:52:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:52:52.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:52:52.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 39846 5.04 MB/sec warmup 46 sec latency 1754.343 ms 2024-02-09T22:52:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:52 smithi003 ceph-mon[32678]: pgmap v51: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 199 KiB/s wr, 73 op/s 2024-02-09T22:52:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:52:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:52:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:52:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:52:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:52 smithi064 ceph-mon[36638]: pgmap v51: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 257 KiB/s rd, 199 KiB/s wr, 73 op/s 2024-02-09T22:52:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:52:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:52:52.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:52:52.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:52:53.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:53 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:52:53.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 40166 4.96 MB/sec warmup 47 sec latency 2170.881 ms 2024-02-09T22:52:53.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:53 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:52:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:53 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:52:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:54 smithi089 ceph-mon[36665]: pgmap v52: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 130 KiB/s wr, 56 op/s 2024-02-09T22:52:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:54 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:54.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 40166 4.86 MB/sec warmup 48 sec latency 1418.079 ms 2024-02-09T22:52:54.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:54 smithi003 ceph-mon[32678]: pgmap v52: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 130 KiB/s wr, 56 op/s 2024-02-09T22:52:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:54 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:54 smithi064 ceph-mon[36638]: pgmap v52: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 130 KiB/s wr, 56 op/s 2024-02-09T22:52:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:54 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:52:55.622 INFO:tasks.workunit.client.0.smithi003.stdout: 1 40166 4.76 MB/sec warmup 49 sec latency 2418.218 ms 2024-02-09T22:52:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:56 smithi089 ceph-mon[36665]: pgmap v53: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 130 KiB/s wr, 56 op/s 2024-02-09T22:52:56.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:56 smithi064 ceph-mon[36638]: pgmap v53: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 130 KiB/s wr, 56 op/s 2024-02-09T22:52:56.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 40166 4.67 MB/sec warmup 50 sec latency 3418.309 ms 2024-02-09T22:52:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:56 smithi003 ceph-mon[32678]: pgmap v53: 97 pgs: 97 active+clean; 23 MiB data, 487 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 130 KiB/s wr, 56 op/s 2024-02-09T22:52:57.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 40166 4.57 MB/sec warmup 51 sec latency 4418.424 ms 2024-02-09T22:52:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:52:58 smithi089 ceph-mon[36665]: pgmap v54: 97 pgs: 97 active+clean; 24 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 226 KiB/s wr, 79 op/s 2024-02-09T22:52:58.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 42694 4.79 MB/sec warmup 52 sec latency 4573.037 ms 2024-02-09T22:52:58.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:52:58 smithi003 ceph-mon[32678]: pgmap v54: 97 pgs: 97 active+clean; 24 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 226 KiB/s wr, 79 op/s 2024-02-09T22:52:58.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:52:58 smithi064 ceph-mon[36638]: pgmap v54: 97 pgs: 97 active+clean; 24 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 324 KiB/s rd, 226 KiB/s wr, 79 op/s 2024-02-09T22:52:59.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 43264 4.73 MB/sec warmup 53 sec latency 461.688 ms 2024-02-09T22:53:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:00 smithi089 ceph-mon[36665]: pgmap v55: 97 pgs: 97 active+clean; 24 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 129 KiB/s wr, 44 op/s 2024-02-09T22:53:00.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 43264 4.64 MB/sec warmup 54 sec latency 1461.769 ms 2024-02-09T22:53:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:00 smithi003 ceph-mon[32678]: pgmap v55: 97 pgs: 97 active+clean; 24 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 129 KiB/s wr, 44 op/s 2024-02-09T22:53:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:00 smithi064 ceph-mon[36638]: pgmap v55: 97 pgs: 97 active+clean; 24 MiB data, 495 MiB used, 1.0 TiB / 1.0 TiB avail; 286 KiB/s rd, 129 KiB/s wr, 44 op/s 2024-02-09T22:53:01.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 43388 4.56 MB/sec warmup 55 sec latency 1696.565 ms 2024-02-09T22:53:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.956+0000 7f2d59fd0640 1 -- 172.21.15.3:0/956085308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d54107420 msgr2=0x7f2d54107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.956+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/956085308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d54107420 0x7f2d54107800 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009930 tx=0x7f2d4402bcd0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 -- 172.21.15.3:0/956085308 shutdown_connections 2024-02-09T22:53:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/956085308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d541004a0 0x7f2d5410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/956085308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d540ffae0 0x7f2d540fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.962 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/956085308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d54107420 0x7f2d54107800 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 -- 172.21.15.3:0/956085308 >> 172.21.15.3:0/956085308 conn(0x7f2d540753d0 msgr2=0x7f2d540757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 -- 172.21.15.3:0/956085308 shutdown_connections 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 -- 172.21.15.3:0/956085308 wait complete. 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 Processor -- start 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.957+0000 7f2d59fd0640 1 -- start start 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d59fd0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 0x7f2d54100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d59fd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d541004a0 0x7f2d54101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d59fd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d54107420 0x7f2d54101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d59fd0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541a5c80 con 0x7f2d540ffae0 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d59fd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541a5df0 con 0x7f2d54107420 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d59fd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d541a5fb0 con 0x7f2d541004a0 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 0x7f2d54100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.963 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 0x7f2d54100cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47536/0 (socket says 172.21.15.3:47536) 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 -- 172.21.15.3:0/3620801177 learned_addr learned my addr 172.21.15.3:0/3620801177 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d541004a0 0x7f2d54101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d53fff640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d54107420 0x7f2d54101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 -- 172.21.15.3:0/3620801177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d541004a0 msgr2=0x7f2d54101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d541004a0 0x7f2d54101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 -- 172.21.15.3:0/3620801177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d54107420 msgr2=0x7f2d54101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d54107420 0x7f2d54101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 -- 172.21.15.3:0/3620801177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44009560 con 0x7f2d540ffae0 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d53fff640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d54107420 0x7f2d54101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.958+0000 7f2d537fe640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 0x7f2d54100cc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000c00 tx=0x7f2d440045e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.959+0000 7f2d50ff9640 1 -- 172.21.15.3:0/3620801177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4402dc50 con 0x7f2d540ffae0 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.959+0000 7f2d50ff9640 1 -- 172.21.15.3:0/3620801177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d4402ddb0 con 0x7f2d540ffae0 2024-02-09T22:53:01.964 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.959+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d541a6230 con 0x7f2d540ffae0 2024-02-09T22:53:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.959+0000 7f2d50ff9640 1 -- 172.21.15.3:0/3620801177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d44003c20 con 0x7f2d540ffae0 2024-02-09T22:53:01.965 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.959+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d541a6790 con 0x7f2d540ffae0 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.961+0000 7f2d50ff9640 1 -- 172.21.15.3:0/3620801177 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f2d44004df0 con 0x7f2d540ffae0 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.961+0000 7f2d5cb9b740 1 Processor -- start 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.961+0000 7f2d5cb9b740 1 -- start start 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa0b0 0x557584daa490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 0x557584db5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584d18a00 con 0x557584db37a0 2024-02-09T22:53:01.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584d15f30 con 0x557584daa9d0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584d11710 con 0x557584daa0b0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d50ff9640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2d3c0a22a0 0x7f2d3c0a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 0x557584db5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 0x557584db5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47538/0 (socket says 172.21.15.3:47538) 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 -- 172.21.15.3:0/1166915206 learned_addr learned my addr 172.21.15.3:0/1166915206 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d537fe640 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d53fff640 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa0b0 0x557584daa490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d53fff640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2d3c0a22a0 0x7f2d3c0a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d50ff9640 1 -- 172.21.15.3:0/3620801177 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2d440f0660 con 0x7f2d540ffae0 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 -- 172.21.15.3:0/1166915206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa0b0 msgr2=0x557584daa490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa0b0 0x557584daa490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 -- 172.21.15.3:0/1166915206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 msgr2=0x557584db3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db3260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 -- 172.21.15.3:0/1166915206 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557584cd63b0 con 0x557584db37a0 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d537fe640 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.962+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 0x557584db5b90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800b740 tx=0x7f2d4800bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b17fa747a513e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d53fff640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2d3c0a22a0 0x7f2d3c0a4760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009fd0 tx=0x7f2d4c009280 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d39ffb640 1 -- 172.21.15.3:0/1166915206 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48003dd0 con 0x557584db37a0 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d39ffb640 1 -- 172.21.15.3:0/1166915206 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d4800be50 con 0x557584db37a0 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d39ffb640 1 -- 172.21.15.3:0/1166915206 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d48011440 con 0x557584db37a0 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1166915206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 msgr2=0x557584db5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 0x557584db5b90 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f2d4800b740 tx=0x7f2d4800bc00 comp rx=0 tx=0).stop 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1166915206 shutdown_connections 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584db37a0 0x557584db5b90 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1166915206 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa0b0 0x557584daa490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1166915206 >> 172.21.15.3:0/1166915206 conn(0x557584da49d0 msgr2=0x557584da4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1166915206 shutdown_connections 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.963+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1166915206 wait complete. 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 -- start start 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 Processor -- start 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584db0d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584cb6b90 con 0x557584daa0b0 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575849e5250 con 0x557584db37a0 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584c97200 con 0x557584daa9d0 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d5cb9b740 10 client.0 _ll_register_callbacks cb 0x557584ccd5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47540/0 (socket says 172.21.15.3:47540) 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.964+0000 7f2d537fe640 1 -- 172.21.15.3:0/1343120160 learned_addr learned my addr 172.21.15.3:0/1343120160 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584db0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d53fff640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d537fe640 1 -- 172.21.15.3:0/1343120160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 msgr2=0x557584db0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d537fe640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584db0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d537fe640 1 -- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 msgr2=0x557584db1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d537fe640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d537fe640 1 -- 172.21.15.3:0/1343120160 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d48004310 con 0x557584daa0b0 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584db0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d537fe640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db07c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2d440bce90 tx=0x7f2d440b94f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df12a0df03ba200 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:01.976 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d32ffd640 1 -- 172.21.15.3:0/1343120160 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d4408c1c0 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d32ffd640 1 -- 172.21.15.3:0/1343120160 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2d440bd890 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557584d37ee0 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d32ffd640 1 -- 172.21.15.3:0/1343120160 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d440bd190 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d5cb9b740 10 client.16182 fetch_fsmap learned FSMap version 30 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557584d0fd90 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.965+0000 7f2d32ffd640 1 -- 172.21.15.3:0/1343120160 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557584d37ee0 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d32ffd640 1 -- 172.21.15.3:0/1343120160 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 30) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2d4403c070 con 0x557584daa0b0 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d5cb9b740 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d5cb9b740 10 client.16182 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d5cb9b740 20 client.16182 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584dc3820 0x557584dc5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d5cb9b740 4 client.16182 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557584a11600 con 0x557584dc3820 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584dc3820 0x557584dc5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584dc3820 0x557584dc5c30 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96542c2686eedc5f server_cookie=5c3bf7ecd93556b7 in_seq=0 out_seq=0 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.966+0000 7f2d32ffd640 10 client.16182 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d32ffd640 1 -- 172.21.15.3:0/1343120160 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557584a11600 con 0x557584dc3820 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d32ffd640 10 client.16182 handle_command_reply: tid=0 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 client.16182 shutdown 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 msgr2=0x557584db07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db07c0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f2d440bce90 tx=0x7f2d440b94f0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 shutdown_connections 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584dc3820 0x557584dc5c30 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584db0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d32ffd640 0 client.16182 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/1343120160 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db07c0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.967+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 >> 172.21.15.3:0/1343120160 conn(0x557584da49d0 msgr2=0x557584da6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.968+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 shutdown_connections 2024-02-09T22:53:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.968+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/1343120160 wait complete. 2024-02-09T22:53:01.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.968+0000 7f2d5cb9b740 20 client.16182 trim_cache size 0 max 16384 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 Processor -- start 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 -- start start 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db1060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584db37a0 0x557584daf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584da4260 con 0x557584daa0b0 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584d9eec0 con 0x557584daa9d0 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.981+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584dc3620 con 0x557584db37a0 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47542/0 (socket says 172.21.15.3:47542) 2024-02-09T22:53:01.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 -- 172.21.15.3:0/2049144638 learned_addr learned my addr 172.21.15.3:0/2049144638 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d53fff640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584db37a0 0x557584daf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 -- 172.21.15.3:0/2049144638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584db37a0 msgr2=0x557584daf090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584db37a0 0x557584daf090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 -- 172.21.15.3:0/2049144638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 msgr2=0x557584db1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 -- 172.21.15.3:0/2049144638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557584cac820 con 0x557584daa0b0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d53fff640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584db37a0 0x557584daf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d537fe640 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db2e80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009900 tx=0x7f2d440036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79db4a307090dc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d397fa640 1 -- 172.21.15.3:0/2049144638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d440bccd0 con 0x557584daa0b0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d397fa640 1 -- 172.21.15.3:0/2049144638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d440bd890 con 0x557584daa0b0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.982+0000 7f2d397fa640 1 -- 172.21.15.3:0/2049144638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d440c5930 con 0x557584daa0b0 2024-02-09T22:53:01.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/2049144638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 msgr2=0x557584db2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db2e80 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f2d44009900 tx=0x7f2d440036f0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/2049144638 shutdown_connections 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584daa9d0 0x557584db1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584db37a0 0x557584daf090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/2049144638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584db2e80 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/2049144638 >> 172.21.15.3:0/2049144638 conn(0x557584da49d0 msgr2=0x557584dabb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/2049144638 shutdown_connections 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/2049144638 wait complete. 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 -- start start 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.983+0000 7f2d5cb9b740 1 Processor -- start 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584dba110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584dba650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db85d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584db8ca0 con 0x557584daa0b0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584db8e10 con 0x557584db37a0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557584db8fd0 con 0x557584daa9d0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 10 client.0 _ll_register_callbacks cb 0x7f2d4402df20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584dba110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584dba110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47556/0 (socket says 172.21.15.3:47556) 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 -- 172.21.15.3:0/829955689 learned_addr learned my addr 172.21.15.3:0/829955689 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584dba650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 -- 172.21.15.3:0/829955689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 msgr2=0x557584dba650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584dba650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 -- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 msgr2=0x557584db85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 -- 172.21.15.3:0/829955689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557584d0fd90 con 0x557584daa0b0 2024-02-09T22:53:01.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584dba650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d537fe640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584dba110 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2d440bda20 tx=0x7f2d440b94f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a323eaea89bc33d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d32ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d32ffd640 1 -- 172.21.15.3:0/829955689 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d440bccd0 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d32ffd640 1 -- 172.21.15.3:0/829955689 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2d440c5930 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d4800be50 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d32ffd640 1 -- 172.21.15.3:0/829955689 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d440c12e0 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 10 client.16188 fetch_fsmap learned FSMap version 30 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.984+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) v3 -- 0x557584cd26f0 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d32ffd640 1 -- 172.21.15.3:0/829955689 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d4800be50 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d32ffd640 1 -- 172.21.15.3:0/829955689 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 30) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f2d4403c070 con 0x557584daa0b0 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d5cb9b740 10 client.16188 fetch_fsmap finished waiting for FSMap version 30 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d5cb9b740 10 client.16188 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d5cb9b740 20 client.16188 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:01.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584e1a960 0x557584e1cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d5cb9b740 4 client.16188 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:53:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557584a621b0 con 0x557584e1a960 2024-02-09T22:53:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584e1a960 0x557584e1cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d32ffd640 10 client.16188 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:01.994 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:01.985+0000 7f2d52ffd640 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584e1a960 0x557584e1cd50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae718e8388c33ca3 server_cookie=daf411cd2facbefc in_seq=0 out_seq=0 2024-02-09T22:53:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.010+0000 7f2d32ffd640 1 -- 172.21.15.3:0/829955689 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557584a621b0 con 0x557584e1a960 2024-02-09T22:53:02.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.010+0000 7f2d32ffd640 10 client.16188 handle_command_reply: tid=0 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.010+0000 7f2d5cb9b740 1 client.16188 shutdown 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 msgr2=0x557584dba110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584dba110 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f2d440bda20 tx=0x7f2d440b94f0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 shutdown_connections 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557584e1a960 0x557584e1cd50 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557584db37a0 0x557584db85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557584daa9d0 0x557584dba650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d32ffd640 0 client.16188 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 --2- 172.21.15.3:0/829955689 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557584daa0b0 0x557584dba110 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 >> 172.21.15.3:0/829955689 conn(0x557584da49d0 msgr2=0x557584daf030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 shutdown_connections 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 1 -- 172.21.15.3:0/829955689 wait complete. 2024-02-09T22:53:02.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d5cb9b740 20 client.16188 trim_cache size 0 max 16384 2024-02-09T22:53:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:53:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:53:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:53:02.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2d3c0a22a0 msgr2=0x7f2d3c0a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2d3c0a22a0 0x7f2d3c0a4760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c009fd0 tx=0x7f2d4c009280 comp rx=0 tx=0).stop 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 msgr2=0x7f2d54100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.011+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 0x7f2d54100cc0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2d44000c00 tx=0x7f2d440045e0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 shutdown_connections 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2d3c0a22a0 0x7f2d3c0a4760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2d54107420 0x7f2d54101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2d541004a0 0x7f2d54101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 --2- 172.21.15.3:0/3620801177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2d540ffae0 0x7f2d54100cc0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:02.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 >> 172.21.15.3:0/3620801177 conn(0x7f2d540753d0 msgr2=0x7f2d540fe570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 shutdown_connections 2024-02-09T22:53:02.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:02.012+0000 7f2d59fd0640 1 -- 172.21.15.3:0/3620801177 wait complete. 2024-02-09T22:53:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:02 smithi089 ceph-mon[36665]: pgmap v56: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 260 KiB/s wr, 83 op/s 2024-02-09T22:53:02.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 43388 4.48 MB/sec warmup 56 sec latency 1684.665 ms 2024-02-09T22:53:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:02 smithi003 ceph-mon[32678]: pgmap v56: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 260 KiB/s wr, 83 op/s 2024-02-09T22:53:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:02 smithi064 ceph-mon[36638]: pgmap v56: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 260 KiB/s wr, 83 op/s 2024-02-09T22:53:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:03 smithi089 ceph-mon[36665]: pgmap v57: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 227 KiB/s wr, 62 op/s 2024-02-09T22:53:03.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 44049 4.48 MB/sec warmup 57 sec latency 2103.587 ms 2024-02-09T22:53:03.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:03 smithi003 ceph-mon[32678]: pgmap v57: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 227 KiB/s wr, 62 op/s 2024-02-09T22:53:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:03 smithi064 ceph-mon[36638]: pgmap v57: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 227 KiB/s wr, 62 op/s 2024-02-09T22:53:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:04 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:53:04.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46520 4.65 MB/sec warmup 58 sec latency 13.586 ms 2024-02-09T22:53:04.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:04 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:53:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:04 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:53:05.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:05 smithi089 ceph-mon[36665]: pgmap v58: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 227 KiB/s wr, 62 op/s 2024-02-09T22:53:05.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46814 4.57 MB/sec warmup 59 sec latency 715.819 ms 2024-02-09T22:53:05.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:05 smithi003 ceph-mon[32678]: pgmap v58: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 227 KiB/s wr, 62 op/s 2024-02-09T22:53:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:05 smithi064 ceph-mon[36638]: pgmap v58: 97 pgs: 97 active+clean; 26 MiB data, 503 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 227 KiB/s wr, 62 op/s 2024-02-09T22:53:06.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46814 4.49 MB/sec warmup 60 sec latency 1715.855 ms 2024-02-09T22:53:07.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46814 4.42 MB/sec warmup 61 sec latency 2715.978 ms 2024-02-09T22:53:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:07 smithi003 ceph-mon[32678]: pgmap v59: 97 pgs: 97 active+clean; 28 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 564 KiB/s wr, 149 op/s 2024-02-09T22:53:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:07 smithi064 ceph-mon[36638]: pgmap v59: 97 pgs: 97 active+clean; 28 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 564 KiB/s wr, 149 op/s 2024-02-09T22:53:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:07 smithi089 ceph-mon[36665]: pgmap v59: 97 pgs: 97 active+clean; 28 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 564 KiB/s wr, 149 op/s 2024-02-09T22:53:08.623 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46938 4.35 MB/sec warmup 62 sec latency 3133.312 ms 2024-02-09T22:53:09.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46938 4.29 MB/sec warmup 63 sec latency 1501.148 ms 2024-02-09T22:53:10.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:10 smithi064 ceph-mon[36638]: pgmap v60: 97 pgs: 97 active+clean; 28 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 467 KiB/s wr, 126 op/s 2024-02-09T22:53:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:10 smithi089 ceph-mon[36665]: pgmap v60: 97 pgs: 97 active+clean; 28 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 467 KiB/s wr, 126 op/s 2024-02-09T22:53:10.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46938 4.22 MB/sec warmup 64 sec latency 2501.237 ms 2024-02-09T22:53:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:10 smithi003 ceph-mon[32678]: pgmap v60: 97 pgs: 97 active+clean; 28 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 668 KiB/s rd, 467 KiB/s wr, 126 op/s 2024-02-09T22:53:11.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46938 4.15 MB/sec warmup 65 sec latency 3501.363 ms 2024-02-09T22:53:12.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:12 smithi064 ceph-mon[36638]: pgmap v61: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 520 KiB/s wr, 139 op/s 2024-02-09T22:53:12.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:12 smithi089 ceph-mon[36665]: pgmap v61: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 520 KiB/s wr, 139 op/s 2024-02-09T22:53:12.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 46938 4.09 MB/sec warmup 66 sec latency 4501.457 ms 2024-02-09T22:53:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:12 smithi003 ceph-mon[32678]: pgmap v61: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 520 KiB/s wr, 139 op/s 2024-02-09T22:53:13.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 48210 4.11 MB/sec warmup 67 sec latency 4918.083 ms 2024-02-09T22:53:14.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:14 smithi064 ceph-mon[36638]: pgmap v62: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 389 KiB/s wr, 100 op/s 2024-02-09T22:53:14.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:14 smithi089 ceph-mon[36665]: pgmap v62: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 389 KiB/s wr, 100 op/s 2024-02-09T22:53:14.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 50159 4.25 MB/sec warmup 68 sec latency 9.764 ms 2024-02-09T22:53:14.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:14 smithi003 ceph-mon[32678]: pgmap v62: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 389 KiB/s wr, 100 op/s 2024-02-09T22:53:14.684 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:53:14.684 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 -- 172.21.15.3:0/454446119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101de0 msgr2=0x7f3984105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 --2- 172.21.15.3:0/454446119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101de0 0x7f3984105e20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3974009930 tx=0x7f397402bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 -- 172.21.15.3:0/454446119 shutdown_connections 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 --2- 172.21.15.3:0/454446119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984106500 0x7f398410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 --2- 172.21.15.3:0/454446119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101de0 0x7f3984105e20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 --2- 172.21.15.3:0/454446119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3984101430 0x7f3984101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 -- 172.21.15.3:0/454446119 >> 172.21.15.3:0/454446119 conn(0x7f39840fccd0 msgr2=0x7f39840ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 -- 172.21.15.3:0/454446119 shutdown_connections 2024-02-09T22:53:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.787+0000 7f398c6d4640 1 -- 172.21.15.3:0/454446119 wait complete. 2024-02-09T22:53:14.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 Processor -- start 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 -- start start 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 0x7f398419d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3984106500 0x7f398419dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984197760 0x7f3984197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3984198230 con 0x7f3984101430 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39841983a0 con 0x7f3984106500 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398c6d4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3984198560 con 0x7f3984197760 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 0x7f398419d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 0x7f398419d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47428/0 (socket says 172.21.15.3:47428) 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 -- 172.21.15.3:0/3460077605 learned_addr learned my addr 172.21.15.3:0/3460077605 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f3989c6f640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3984106500 0x7f398419dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398ac71640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984197760 0x7f3984197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 -- 172.21.15.3:0/3460077605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984197760 msgr2=0x7f3984197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984197760 0x7f3984197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 -- 172.21.15.3:0/3460077605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3984106500 msgr2=0x7f398419dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3984106500 0x7f398419dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 -- 172.21.15.3:0/3460077605 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3974009560 con 0x7f3984101430 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.788+0000 7f398a470640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 0x7f398419d6a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f398000d950 tx=0x7f398000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.789+0000 7f397b7fe640 1 -- 172.21.15.3:0/3460077605 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980009a30 con 0x7f3984101430 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.789+0000 7f397b7fe640 1 -- 172.21.15.3:0/3460077605 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3980009b90 con 0x7f3984101430 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.789+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39841a4500 con 0x7f3984101430 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.789+0000 7f397b7fe640 1 -- 172.21.15.3:0/3460077605 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980004a00 con 0x7f3984101430 2024-02-09T22:53:14.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.789+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39841a49e0 con 0x7f3984101430 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f397b7fe640 1 -- 172.21.15.3:0/3460077605 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3980009d00 con 0x7f3984101430 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 Processor -- start 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 -- start start 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f397b7fe640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39700a2230 0x7f39700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2cdd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea288d00 con 0x5611ea2cd980 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea23d2f0 con 0x5611ea2d6eb0 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f398f389740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea23a820 con 0x5611ea2ce330 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.791+0000 7f397b7fe640 1 -- 172.21.15.3:0/3460077605 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39800cda70 con 0x7f3984101430 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398ac71640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39700a2230 0x7f39700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2cdd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2cdd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47440/0 (socket says 172.21.15.3:47440) 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 -- 172.21.15.3:0/2183236626 learned_addr learned my addr 172.21.15.3:0/2183236626 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f3989c6f640 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398ac71640 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 -- 172.21.15.3:0/2183236626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 msgr2=0x5611ea2d68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 -- 172.21.15.3:0/2183236626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 msgr2=0x5611ea2d9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 -- 172.21.15.3:0/2183236626 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611ea1f7ff0 con 0x5611ea2cd980 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398ac71640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39700a2230 0x7f39700a46f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f397c0070b0 tx=0x7f397c00c900 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f398a470640 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2cdd60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f39800968f0 tx=0x7f3980099eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9eedeab7b0cb7d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f3978ff9640 1 -- 172.21.15.3:0/2183236626 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39800ba0d0 con 0x5611ea2cd980 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f3978ff9640 1 -- 172.21.15.3:0/2183236626 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f398009e5e0 con 0x5611ea2cd980 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.792+0000 7f3978ff9640 1 -- 172.21.15.3:0/2183236626 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980096b00 con 0x5611ea2cd980 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 -- 172.21.15.3:0/2183236626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 msgr2=0x5611ea2cdd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2cdd60 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f39800968f0 tx=0x7f3980099eb0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 -- 172.21.15.3:0/2183236626 shutdown_connections 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 --2- 172.21.15.3:0/2183236626 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2cdd60 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 -- 172.21.15.3:0/2183236626 >> 172.21.15.3:0/2183236626 conn(0x5611ea2c8050 msgr2=0x5611ea2c8430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 -- 172.21.15.3:0/2183236626 shutdown_connections 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 -- 172.21.15.3:0/2183236626 wait complete. 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 -- start start 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.793+0000 7f398f389740 1 Processor -- start 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2d3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea236000 con 0x5611ea2cd980 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e9f09430 con 0x5611ea2d6eb0 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e9ec5e00 con 0x5611ea2ce330 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2d3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2d3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47456/0 (socket says 172.21.15.3:47456) 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 -- 172.21.15.3:0/4091103918 learned_addr learned my addr 172.21.15.3:0/4091103918 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 10 client.0 _ll_register_callbacks cb 0x5611ea22a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f3989c6f640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398ac71640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 -- 172.21.15.3:0/4091103918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 msgr2=0x5611ea2d4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 -- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 msgr2=0x5611ea2d4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 -- 172.21.15.3:0/4091103918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39841a49e0 con 0x5611ea2cd980 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398a470640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2d3dd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3980009610 tx=0x7f39800cb680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c3b049b1c6ba888 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f396dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f396dffb640 1 -- 172.21.15.3:0/4091103918 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980005680 con 0x5611ea2cd980 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f396dffb640 1 -- 172.21.15.3:0/4091103918 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39800a2c30 con 0x5611ea2cd980 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5611e9f357e0 con 0x5611ea2cd980 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.794+0000 7f396dffb640 1 -- 172.21.15.3:0/4091103918 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3980051220 con 0x5611ea2cd980 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 10 client.16200 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5611ea234680 con 0x5611ea2cd980 2024-02-09T22:53:14.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f396dffb640 1 -- 172.21.15.3:0/4091103918 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5611e9f357e0 con 0x5611ea2cd980 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f396dffb640 1 -- 172.21.15.3:0/4091103918 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f39800585c0 con 0x5611ea2cd980 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 10 client.16200 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 10 client.16200 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 20 client.16200 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea2e6fc0 0x5611ea2e93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 4 client.16200 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5611e9f72720 con 0x5611ea2e6fc0 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f3989c6f640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea2e6fc0 0x5611ea2e93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f3989c6f640 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea2e6fc0 0x5611ea2e93e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a04d6f99f687f05 server_cookie=67c74766f9260b78 in_seq=0 out_seq=0 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.795+0000 7f396dffb640 10 client.16200 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f396dffb640 1 -- 172.21.15.3:0/4091103918 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5611e9f72720 con 0x5611ea2e6fc0 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f396dffb640 10 client.16200 handle_command_reply: tid=0 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 client.16200 shutdown 2024-02-09T22:53:14.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 msgr2=0x5611ea2d3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2d3dd0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f3980009610 tx=0x7f39800cb680 comp rx=0 tx=0).stop 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 shutdown_connections 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea2e6fc0 0x5611ea2e93e0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2ce330 0x5611ea2d4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f396dffb640 0 client.16200 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 --2- 172.21.15.3:0/4091103918 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea2d3dd0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 >> 172.21.15.3:0/4091103918 conn(0x5611ea2c8050 msgr2=0x5611ea2c9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 shutdown_connections 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 1 -- 172.21.15.3:0/4091103918 wait complete. 2024-02-09T22:53:14.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.796+0000 7f398f389740 20 client.16200 trim_cache size 0 max 16384 2024-02-09T22:53:14.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 Processor -- start 2024-02-09T22:53:14.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 -- start start 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2cd980 0x5611ea2de260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 0x5611ea2d1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea1bafd0 con 0x5611ea2ce330 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611e9f357e0 con 0x5611ea2cd980 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f398f389740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea2de7a0 con 0x5611ea2d6eb0 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.811+0000 7f3989c6f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 0x5611ea2d1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 0x5611ea2d1050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47472/0 (socket says 172.21.15.3:47472) 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 -- 172.21.15.3:0/1679534618 learned_addr learned my addr 172.21.15.3:0/1679534618 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f398a470640 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2cd980 0x5611ea2de260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f398ac71640 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 -- 172.21.15.3:0/1679534618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 msgr2=0x5611ea2d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 -- 172.21.15.3:0/1679534618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2cd980 msgr2=0x5611ea2de260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2cd980 0x5611ea2de260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 -- 172.21.15.3:0/1679534618 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5611ea1d0b30 con 0x5611ea2ce330 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f398ac71640 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f3989c6f640 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 0x5611ea2d1050 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3974005ab0 tx=0x7f3974005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6beb93c3beb777f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f396ffff640 1 -- 172.21.15.3:0/1679534618 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974046040 con 0x5611ea2ce330 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f396ffff640 1 -- 172.21.15.3:0/1679534618 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3974049be0 con 0x5611ea2ce330 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.812+0000 7f396ffff640 1 -- 172.21.15.3:0/1679534618 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397402cab0 con 0x5611ea2ce330 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- 172.21.15.3:0/1679534618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 msgr2=0x5611ea2d1050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 0x5611ea2d1050 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f3974005ab0 tx=0x7f3974005fd0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- 172.21.15.3:0/1679534618 shutdown_connections 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea2d1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2ce330 0x5611ea2d1050 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- 172.21.15.3:0/1679534618 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2cd980 0x5611ea2de260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- 172.21.15.3:0/1679534618 >> 172.21.15.3:0/1679534618 conn(0x5611ea2c8050 msgr2=0x5611ea2d46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- 172.21.15.3:0/1679534618 shutdown_connections 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- 172.21.15.3:0/1679534618 wait complete. 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- start start 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 Processor -- start 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea33b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2ce330 0x5611ea33b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea3357f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea335db0 con 0x5611ea2cd980 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea335f20 con 0x5611ea2ce330 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5611ea3360e0 con 0x5611ea2d6eb0 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398f389740 10 client.0 _ll_register_callbacks cb 0x7f3980096050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea33b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398a470640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea33b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47478/0 (socket says 172.21.15.3:47478) 2024-02-09T22:53:14.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f398a470640 1 -- 172.21.15.3:0/3565663408 learned_addr learned my addr 172.21.15.3:0/3565663408 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.813+0000 7f3989c6f640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2ce330 0x5611ea33b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398ac71640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea3357f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398a470640 1 -- 172.21.15.3:0/3565663408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 msgr2=0x5611ea3357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398a470640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea3357f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398a470640 1 -- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2ce330 msgr2=0x5611ea33b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398a470640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2ce330 0x5611ea33b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398a470640 1 -- 172.21.15.3:0/3565663408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f398410e8e0 con 0x5611ea2cd980 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398a470640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea33b3e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3980098740 tx=0x7f39800989a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da14b96e24e495a1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f396dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f396dffb640 1 -- 172.21.15.3:0/3565663408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3980058370 con 0x5611ea2cd980 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f396dffb640 1 -- 172.21.15.3:0/3565663408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39800a2c30 con 0x5611ea2cd980 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f398009e5e0 con 0x5611ea2cd980 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f396dffb640 1 -- 172.21.15.3:0/3565663408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3980005680 con 0x5611ea2cd980 2024-02-09T22:53:14.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 10 client.16206 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5611ea1f6850 con 0x5611ea2cd980 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f396dffb640 1 -- 172.21.15.3:0/3565663408 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f398009e5e0 con 0x5611ea2cd980 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f396dffb640 1 -- 172.21.15.3:0/3565663408 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f39800b6060 con 0x5611ea2cd980 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 10 client.16206 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 10 client.16206 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 20 client.16206 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea342d60 0x5611ea345150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 4 client.16206 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.814+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5611e9fc1890 con 0x5611ea342d60 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f3989c6f640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea342d60 0x5611ea345150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f3989c6f640 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea342d60 0x5611ea345150 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62275e8104bf0137 server_cookie=c6bb4b02c989528f in_seq=0 out_seq=0 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f396dffb640 10 client.16206 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f396dffb640 1 -- 172.21.15.3:0/3565663408 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5611e9fc1890 con 0x5611ea342d60 2024-02-09T22:53:14.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f396dffb640 10 client.16206 handle_command_reply: tid=0 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 client.16206 shutdown 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 msgr2=0x5611ea33b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea33b3e0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f3980098740 tx=0x7f39800989a0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 shutdown_connections 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5611ea342d60 0x5611ea345150 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5611ea2ce330 0x5611ea33b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5611ea2d6eb0 0x5611ea3357f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 --2- 172.21.15.3:0/3565663408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5611ea2cd980 0x5611ea33b3e0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f396dffb640 0 client.16206 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 >> 172.21.15.3:0/3565663408 conn(0x5611ea2c8050 msgr2=0x5611ea2c6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 shutdown_connections 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 1 -- 172.21.15.3:0/3565663408 wait complete. 2024-02-09T22:53:14.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.815+0000 7f398f389740 20 client.16206 trim_cache size 0 max 16384 2024-02-09T22:53:14.826 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:53:14.826 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:53:14.826 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:53:14.826 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39700a2230 msgr2=0x7f39700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39700a2230 0x7f39700a46f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f397c0070b0 tx=0x7f397c00c900 comp rx=0 tx=0).stop 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 msgr2=0x7f398419d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 0x7f398419d6a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f398000d950 tx=0x7f398000de10 comp rx=0 tx=0).stop 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 shutdown_connections 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39700a2230 0x7f39700a46f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3984106500 0x7f398419dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3984197760 0x7f3984197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 --2- 172.21.15.3:0/3460077605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3984101430 0x7f398419d6a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 >> 172.21.15.3:0/3460077605 conn(0x7f39840fccd0 msgr2=0x7f39840ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 shutdown_connections 2024-02-09T22:53:14.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.816+0000 7f398c6d4640 1 -- 172.21.15.3:0/3460077605 wait complete. 2024-02-09T22:53:14.829 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:53:14.829 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:53:14.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 -- 172.21.15.3:0/179703568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10100330 msgr2=0x7fba10100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 --2- 172.21.15.3:0/179703568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10100330 0x7fba10100710 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fba00009930 tx=0x7fba0002bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 -- 172.21.15.3:0/179703568 shutdown_connections 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 --2- 172.21.15.3:0/179703568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100ce0 0x7fba10104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 --2- 172.21.15.3:0/179703568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba10105400 0x7fba1010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 --2- 172.21.15.3:0/179703568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10100330 0x7fba10100710 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 -- 172.21.15.3:0/179703568 >> 172.21.15.3:0/179703568 conn(0x7fba100fbbb0 msgr2=0x7fba100fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 -- 172.21.15.3:0/179703568 shutdown_connections 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.929+0000 7fba1544d640 1 -- 172.21.15.3:0/179703568 wait complete. 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 Processor -- start 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 -- start start 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100330 0x7fba1019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba10105400 0x7fba1019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 0x7fba10197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10198210 con 0x7fba10197760 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10198380 con 0x7fba10100330 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba1544d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba10198540 con 0x7fba10105400 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 0x7fba10197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100330 0x7fba1019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 0x7fba10197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47492/0 (socket says 172.21.15.3:47492) 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0f7fe640 1 -- 172.21.15.3:0/1331891818 learned_addr learned my addr 172.21.15.3:0/1331891818 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0e7fc640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba10105400 0x7fba1019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0effd640 1 -- 172.21.15.3:0/1331891818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba10105400 msgr2=0x7fba1019db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0effd640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba10105400 0x7fba1019db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0effd640 1 -- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 msgr2=0x7fba10197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0effd640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 0x7fba10197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0effd640 1 -- 172.21.15.3:0/1331891818 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00009560 con 0x7fba10100330 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.930+0000 7fba0f7fe640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 0x7fba10197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:14.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.931+0000 7fba0effd640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100330 0x7fba1019d610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba00009230 tx=0x7fba00005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.931+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00004350 con 0x7fba10100330 2024-02-09T22:53:14.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.931+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba101a4480 con 0x7fba10100330 2024-02-09T22:53:14.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.931+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba000044b0 con 0x7fba10100330 2024-02-09T22:53:14.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.931+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba00003c90 con 0x7fba10100330 2024-02-09T22:53:14.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.931+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba101a49b0 con 0x7fba10100330 2024-02-09T22:53:14.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.932+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba10101800 con 0x7fba10100330 2024-02-09T22:53:14.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.933+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fba00070a10 con 0x7fba10100330 2024-02-09T22:53:14.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.933+0000 7fb9f7fff640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb9f80a2230 0x7fb9f80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:14.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.933+0000 7fba0e7fc640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb9f80a2230 0x7fb9f80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:14.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.933+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba000f00d0 con 0x7fba10100330 2024-02-09T22:53:14.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.934+0000 7fba0e7fc640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb9f80a2230 0x7fb9f80a46f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba100fc430 tx=0x7fba04005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:14.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:14.937+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fba000b8ea0 con 0x7fba10100330 2024-02-09T22:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 -- 172.21.15.3:0/3610946931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 msgr2=0x7effe0069120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 --2- 172.21.15.3:0/3610946931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe0069120 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7effd0009930 tx=0x7effd002bc20 comp rx=0 tx=0).stop 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 -- 172.21.15.3:0/3610946931 shutdown_connections 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 --2- 172.21.15.3:0/3610946931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe00696f0 0x7effe0105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 --2- 172.21.15.3:0/3610946931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 0x7effe010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 --2- 172.21.15.3:0/3610946931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe0069120 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 -- 172.21.15.3:0/3610946931 >> 172.21.15.3:0/3610946931 conn(0x7effe0077220 msgr2=0x7effe0077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 -- 172.21.15.3:0/3610946931 shutdown_connections 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.100+0000 7effe6577640 1 -- 172.21.15.3:0/3610946931 wait complete. 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 Processor -- start 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 -- start start 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe019f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 0x7effe019fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0199940 0x7effe0199df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe019a330 con 0x7effe0068d40 2024-02-09T22:53:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe019a4a0 con 0x7effe0199940 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe6577640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe019a660 con 0x7effe0106510 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effdffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe019f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effdffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe019f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47518/0 (socket says 172.21.15.3:47518) 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effdf7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 0x7effe019fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effdffff640 1 -- 172.21.15.3:0/1393602868 learned_addr learned my addr 172.21.15.3:0/1393602868 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.101+0000 7effe4b14640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0199940 0x7effe0199df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdffff640 1 -- 172.21.15.3:0/1393602868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 msgr2=0x7effe019fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdffff640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 0x7effe019fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdffff640 1 -- 172.21.15.3:0/1393602868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0199940 msgr2=0x7effe0199df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdffff640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0199940 0x7effe0199df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdffff640 1 -- 172.21.15.3:0/1393602868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd0009560 con 0x7effe0068d40 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effe4b14640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0199940 0x7effe0199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdf7fe640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 0x7effe019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdffff640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe019f790 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7effd0009230 tx=0x7effd002c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdd7fa640 1 -- 172.21.15.3:0/1393602868 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0004350 con 0x7effe0068d40 2024-02-09T22:53:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe01a6650 con 0x7effe0068d40 2024-02-09T22:53:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdd7fa640 1 -- 172.21.15.3:0/1393602868 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd00044b0 con 0x7effe0068d40 2024-02-09T22:53:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effdd7fa640 1 -- 172.21.15.3:0/1393602868 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0003d00 con 0x7effe0068d40 2024-02-09T22:53:15.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.102+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe01a6b50 con 0x7effe0068d40 2024-02-09T22:53:15.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.104+0000 7effdd7fa640 1 -- 172.21.15.3:0/1393602868 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7effd0004df0 con 0x7effe0068d40 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 Processor -- start 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 -- start start 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e08241490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824a260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e081afa00 con 0x555e0824a7a0 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e081acf30 con 0x555e082410b0 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe9226740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e081a8710 con 0x555e082419d0 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe4b14640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e08241490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe4b14640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e08241490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:58488/0 (socket says 172.21.15.3:58488) 2024-02-09T22:53:15.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47532/0 (socket says 172.21.15.3:47532) 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe4b14640 1 -- 172.21.15.3:0/508707341 learned_addr learned my addr 172.21.15.3:0/508707341 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdd7fa640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effc80a2220 0x7effc80a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdffff640 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824a260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe4b14640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effc80a2220 0x7effc80a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdd7fa640 1 -- 172.21.15.3:0/1393602868 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7effd00efee0 con 0x7effe0068d40 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 -- 172.21.15.3:0/508707341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 msgr2=0x555e0824a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824a260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 -- 172.21.15.3:0/508707341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 msgr2=0x555e08241490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e08241490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effdf7fe640 1 -- 172.21.15.3:0/508707341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e0816d3b0 con 0x555e0824a7a0 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.105+0000 7effe4b14640 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e08241490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effdf7fe640 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824cb90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effe019b130 tx=0x7effd40118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daec1f8f1c6aff08 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effe4b14640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effc80a2220 0x7effc80a46e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7effd8005c40 tx=0x7effd8004ab0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effc67fc640 1 -- 172.21.15.3:0/508707341 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4004030 con 0x555e0824a7a0 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effc67fc640 1 -- 172.21.15.3:0/508707341 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd4005500 con 0x555e0824a7a0 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effc67fc640 1 -- 172.21.15.3:0/508707341 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4004ce0 con 0x555e0824a7a0 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effe9226740 1 -- 172.21.15.3:0/508707341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 msgr2=0x555e0824cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.106+0000 7effe9226740 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824cb90 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7effe019b130 tx=0x7effd40118d0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- 172.21.15.3:0/508707341 shutdown_connections 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824cb90 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824a260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 --2- 172.21.15.3:0/508707341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e08241490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- 172.21.15.3:0/508707341 >> 172.21.15.3:0/508707341 conn(0x555e0823b9d0 msgr2=0x555e0823bdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- 172.21.15.3:0/508707341 shutdown_connections 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- 172.21.15.3:0/508707341 wait complete. 2024-02-09T22:53:15.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- start start 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 Processor -- start 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 0x555e08247750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e08247c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e0824a7a0 0x555e082481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0814db90 con 0x555e082410b0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e07e7c250 con 0x555e0824a7a0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0812e200 con 0x555e082419d0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.107+0000 7effe9226740 10 client.0 _ll_register_callbacks cb 0x555e081645d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 0x555e08247750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 0x555e08247750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47546/0 (socket says 172.21.15.3:47546) 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 -- 172.21.15.3:0/3743558816 learned_addr learned my addr 172.21.15.3:0/3743558816 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdf7fe640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e08247c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effe4b14640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e0824a7a0 0x555e082481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 -- 172.21.15.3:0/3743558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 msgr2=0x555e08247c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e08247c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 -- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e0824a7a0 msgr2=0x555e082481d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e0824a7a0 0x555e082481d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 -- 172.21.15.3:0/3743558816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd4009c00 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effdffff640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 0x555e08247750 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7effd0009430 tx=0x7effd002c730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77406a0c6ee4cfae server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effbf7fe640 1 -- 172.21.15.3:0/3743558816 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00035f0 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effbf7fe640 1 -- 172.21.15.3:0/3743558816 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7effd00bd110 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e081ceee0 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effbf7fe640 1 -- 172.21.15.3:0/3743558816 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd00bca10 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effe9226740 10 client.16224 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.108+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555e081a6d90 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effbf7fe640 1 -- 172.21.15.3:0/3743558816 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555e081ceee0 con 0x555e082410b0 2024-02-09T22:53:15.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effbf7fe640 1 -- 172.21.15.3:0/3743558816 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7effd003c070 con 0x555e082410b0 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effe9226740 10 client.16224 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effe9226740 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effe9226740 20 client.16224 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effe9226740 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e0825a7b0 0x555e0825cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effe9226740 4 client.16224 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e07ea8600 con 0x555e0825a7b0 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effdf7fe640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e0825a7b0 0x555e0825cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effdf7fe640 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e0825a7b0 0x555e0825cbd0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6c781280cc629f3 server_cookie=53b1c7c1aa9d1ee1 in_seq=0 out_seq=0 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.109+0000 7effbf7fe640 10 client.16224 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effbf7fe640 1 -- 172.21.15.3:0/3743558816 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e07ea8600 con 0x555e0825a7b0 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effbf7fe640 10 client.16224 handle_command_reply: tid=0 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 client.16224 shutdown 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 msgr2=0x555e08247750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 0x555e08247750 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7effd0009430 tx=0x7effd002c730 comp rx=0 tx=0).stop 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 shutdown_connections 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e0825a7b0 0x555e0825cbd0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e0824a7a0 0x555e082481d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e08247c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 --2- 172.21.15.3:0/3743558816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e082410b0 0x555e08247750 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 >> 172.21.15.3:0/3743558816 conn(0x555e0823b9d0 msgr2=0x555e0823d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effbf7fe640 0 client.16224 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 shutdown_connections 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 1 -- 172.21.15.3:0/3743558816 wait complete. 2024-02-09T22:53:15.120 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.110+0000 7effe9226740 20 client.16224 trim_cache size 0 max 16384 2024-02-09T22:53:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.123+0000 7effe9226740 1 Processor -- start 2024-02-09T22:53:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.123+0000 7effe9226740 1 -- start start 2024-02-09T22:53:15.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe9226740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082410b0 0x555e08249e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe9226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082419d0 0x555e08248010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe9226740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e08246080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe9226740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0823cb90 con 0x555e0824a7a0 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe9226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0823b500 con 0x555e082419d0 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe9226740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e0824cb90 con 0x555e082410b0 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe4b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e08246080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082410b0 0x555e08249e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe4b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e08246080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47548/0 (socket says 172.21.15.3:47548) 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effe4b14640 1 -- 172.21.15.3:0/2637438946 learned_addr learned my addr 172.21.15.3:0/2637438946 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdf7fe640 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082419d0 0x555e08248010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 -- 172.21.15.3:0/2637438946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082419d0 msgr2=0x555e08248010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082419d0 0x555e08248010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 -- 172.21.15.3:0/2637438946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 msgr2=0x555e08246080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e08246080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 -- 172.21.15.3:0/2637438946 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e08143820 con 0x555e082410b0 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effdffff640 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082410b0 0x555e08249e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7effd00b93e0 tx=0x7effd00057e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1a1bbe99c1936d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effc5ffb640 1 -- 172.21.15.3:0/2637438946 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00bc370 con 0x555e082410b0 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effc5ffb640 1 -- 172.21.15.3:0/2637438946 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd00b8cb0 con 0x555e082410b0 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.124+0000 7effc5ffb640 1 -- 172.21.15.3:0/2637438946 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd00c0b60 con 0x555e082410b0 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 -- 172.21.15.3:0/2637438946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082410b0 msgr2=0x555e08249e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082410b0 0x555e08249e80 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7effd00b93e0 tx=0x7effd00057e0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 -- 172.21.15.3:0/2637438946 shutdown_connections 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082419d0 0x555e08248010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e08246080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 --2- 172.21.15.3:0/2637438946 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082410b0 0x555e08249e80 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 -- 172.21.15.3:0/2637438946 >> 172.21.15.3:0/2637438946 conn(0x555e0823b9d0 msgr2=0x555e08242ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 -- 172.21.15.3:0/2637438946 shutdown_connections 2024-02-09T22:53:15.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 -- 172.21.15.3:0/2637438946 wait complete. 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.125+0000 7effe9226740 1 -- start start 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 Processor -- start 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e0824f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e082afdd0 con 0x555e0824a7a0 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e082aff40 con 0x555e082410b0 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e082b0100 con 0x555e082419d0 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe9226740 10 client.0 _ll_register_callbacks cb 0x7effd00a5f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effdffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e0824f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47564/0 (socket says 172.21.15.3:47564) 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effdffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e0824f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:58506/0 (socket says 172.21.15.3:58506) 2024-02-09T22:53:15.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 -- 172.21.15.3:0/2733363547 learned_addr learned my addr 172.21.15.3:0/2733363547 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effdf7fe640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 -- 172.21.15.3:0/2733363547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 msgr2=0x555e0824f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 -- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 msgr2=0x555e0824f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e0824f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 -- 172.21.15.3:0/2733363547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e081a6d90 con 0x555e0824a7a0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effdffff640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e0824f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effe4b14640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824fef0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7effd8012040 tx=0x7effd8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d8cee09688d7446 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.126+0000 7effbf7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effbf7fe640 1 -- 172.21.15.3:0/2733363547 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8025040 con 0x555e0824a7a0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effbf7fe640 1 -- 172.21.15.3:0/2733363547 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7effd8004d20 con 0x555e0824a7a0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effd4005500 con 0x555e0824a7a0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effbf7fe640 1 -- 172.21.15.3:0/2733363547 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd801f600 con 0x555e0824a7a0 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 10 client.16230 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555e081696f0 con 0x555e0824a7a0 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effbf7fe640 1 -- 172.21.15.3:0/2733363547 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd4005500 con 0x555e0824a7a0 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effbf7fe640 1 -- 172.21.15.3:0/2733363547 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7effd8028030 con 0x555e0824a7a0 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 10 client.16230 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 10 client.16230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 20 client.16230 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e082b1a80 0x555e082b3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 4 client.16230 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555e07ef91b0 con 0x555e082b1a80 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.127+0000 7effdffff640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e082b1a80 0x555e082b3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.128+0000 7effbf7fe640 10 client.16230 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.128+0000 7effdffff640 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e082b1a80 0x555e082b3ea0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a00be4bdd97fc8a2 server_cookie=7fe79db77eaa48fc in_seq=0 out_seq=0 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effbf7fe640 1 -- 172.21.15.3:0/2733363547 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555e07ef91b0 con 0x555e082b1a80 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effbf7fe640 10 client.16230 handle_command_reply: tid=0 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 client.16230 shutdown 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 msgr2=0x555e0824fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824fef0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7effd8012040 tx=0x7effd8004a40 comp rx=0 tx=0).stop 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 shutdown_connections 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555e082b1a80 0x555e082b3ea0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e0824a7a0 0x555e0824fef0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e082419d0 0x555e0824f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effbf7fe640 0 client.16230 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 --2- 172.21.15.3:0/2733363547 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e082410b0 0x555e0824f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 >> 172.21.15.3:0/2733363547 conn(0x555e0823b9d0 msgr2=0x555e08246000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 shutdown_connections 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.154+0000 7effe9226740 1 -- 172.21.15.3:0/2733363547 wait complete. 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe9226740 20 client.16230 trim_cache size 0 max 16384 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:53:15.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:53:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effc80a2220 msgr2=0x7effc80a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effc80a2220 0x7effc80a46e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7effd8005c40 tx=0x7effd8004ab0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 msgr2=0x7effe019f790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe019f790 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7effd0009230 tx=0x7effd002c640 comp rx=0 tx=0).stop 2024-02-09T22:53:15.160 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 shutdown_connections 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effc80a2220 0x7effc80a46e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0199940 0x7effe0199df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0106510 0x7effe019fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 --2- 172.21.15.3:0/1393602868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0068d40 0x7effe019f790 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 >> 172.21.15.3:0/1393602868 conn(0x7effe0077220 msgr2=0x7effe010c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 shutdown_connections 2024-02-09T22:53:15.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:15.155+0000 7effe6577640 1 -- 172.21.15.3:0/1393602868 wait complete. 2024-02-09T22:53:15.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.167+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba10068500 con 0x7fba10100330 2024-02-09T22:53:15.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.169+0000 7fb9f7fff640 1 -- 172.21.15.3:0/1331891818 <== mon.2 v2:172.21.15.64:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11371 (secure 0 0 0) 0x7fba000bc560 con 0x7fba10100330 2024-02-09T22:53:15.172 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:53:15.172 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:53:02.760086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:53:15.172 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 31 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb9f80a2230 msgr2=0x7fb9f80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb9f80a2230 0x7fb9f80a46f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fba100fc430 tx=0x7fba04005f90 comp rx=0 tx=0).stop 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100330 msgr2=0x7fba1019d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100330 0x7fba1019d610 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fba00009230 tx=0x7fba00005e70 comp rx=0 tx=0).stop 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 shutdown_connections 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb9f80a2230 0x7fb9f80a46f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba10197760 0x7fba10197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba10105400 0x7fba1019db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 --2- 172.21.15.3:0/1331891818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba10100330 0x7fba1019d610 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 >> 172.21.15.3:0/1331891818 conn(0x7fba100fbbb0 msgr2=0x7fba10069cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 shutdown_connections 2024-02-09T22:53:15.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.171+0000 7fba1544d640 1 -- 172.21.15.3:0/1331891818 wait complete. 2024-02-09T22:53:15.185 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:15.185 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 -- 172.21.15.3:0/3572957401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 msgr2=0x7f619c102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 --2- 172.21.15.3:0/3572957401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c102830 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f6198009900 tx=0x7f619802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 -- 172.21.15.3:0/3572957401 shutdown_connections 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 --2- 172.21.15.3:0/3572957401 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 0x7f619c10b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 --2- 172.21.15.3:0/3572957401 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 0x7f619c10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 --2- 172.21.15.3:0/3572957401 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c102830 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 -- 172.21.15.3:0/3572957401 >> 172.21.15.3:0/3572957401 conn(0x7f619c077280 msgr2=0x7f619c077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 -- 172.21.15.3:0/3572957401 shutdown_connections 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.292+0000 7f61a3db6640 1 -- 172.21.15.3:0/3572957401 wait complete. 2024-02-09T22:53:15.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 Processor -- start 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 -- start start 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 0x7f619c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 0x7f619c197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c197e00 con 0x7f619c102450 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c197f70 con 0x7f619c102d70 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a3db6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c198130 con 0x7f619c10b820 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47580/0 (socket says 172.21.15.3:47580) 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 -- 172.21.15.3:0/2043434263 learned_addr learned my addr 172.21.15.3:0/2043434263 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1351640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 0x7f619c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a2353640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 0x7f619c197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.312 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 -- 172.21.15.3:0/2043434263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 msgr2=0x7f619c197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 0x7f619c197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 -- 172.21.15.3:0/2043434263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 msgr2=0x7f619c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 0x7f619c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1b52640 1 -- 172.21.15.3:0/2043434263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6198009560 con 0x7f619c102450 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.293+0000 7f61a1351640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 0x7f619c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f61a2353640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 0x7f619c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f61a1b52640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c19d5b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6198005940 tx=0x7f619802c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f6192ffd640 1 -- 172.21.15.3:0/2043434263 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61980043d0 con 0x7f619c102450 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f6192ffd640 1 -- 172.21.15.3:0/2043434263 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61980052d0 con 0x7f619c102450 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c1983b0 con 0x7f619c102450 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f6192ffd640 1 -- 172.21.15.3:0/2043434263 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198003d30 con 0x7f619c102450 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.294+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c1a4530 con 0x7f619c102450 2024-02-09T22:53:15.313 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f6192ffd640 1 -- 172.21.15.3:0/2043434263 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6198005440 con 0x7f619c102450 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 Processor -- start 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 -- start start 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4d2f60 0x5600cb4d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb485020 con 0x5600cb4ca3e0 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb438320 con 0x5600cb4d2f60 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a6985740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb435850 con 0x5600cb4c9a30 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f61a1b52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.297+0000 7f6192ffd640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61880a2230 0x7f61880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47592/0 (socket says 172.21.15.3:47592) 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 -- 172.21.15.3:0/3041208382 learned_addr learned my addr 172.21.15.3:0/3041208382 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a2353640 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a2353640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61880a2230 0x7f61880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1351640 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4d2f60 0x5600cb4d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f6192ffd640 1 -- 172.21.15.3:0/2043434263 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f61980f0820 con 0x7f619c102450 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 -- 172.21.15.3:0/3041208382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 msgr2=0x5600cb4c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 -- 172.21.15.3:0/3041208382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4d2f60 msgr2=0x5600cb4d53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4d2f60 0x5600cb4d53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 -- 172.21.15.3:0/3041208382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600cb3f3210 con 0x5600cb4ca3e0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a2353640 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d2990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6198006010 tx=0x7f619802bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9eb434881ab779 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f61a2353640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61880a2230 0x7f61880a46f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6194005c40 tx=0x7f6194009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f6187fff640 1 -- 172.21.15.3:0/3041208382 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61980bcdd0 con 0x5600cb4ca3e0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f6187fff640 1 -- 172.21.15.3:0/3041208382 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61980035f0 con 0x5600cb4ca3e0 2024-02-09T22:53:15.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.298+0000 7f6187fff640 1 -- 172.21.15.3:0/3041208382 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61980c5c80 con 0x5600cb4ca3e0 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 -- 172.21.15.3:0/3041208382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 msgr2=0x5600cb4d2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d2990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f6198006010 tx=0x7f619802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 -- 172.21.15.3:0/3041208382 shutdown_connections 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4d2f60 0x5600cb4d53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d2990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 --2- 172.21.15.3:0/3041208382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 -- 172.21.15.3:0/3041208382 >> 172.21.15.3:0/3041208382 conn(0x5600cb4c4100 msgr2=0x5600cb4c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 -- 172.21.15.3:0/3041208382 shutdown_connections 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 -- 172.21.15.3:0/3041208382 wait complete. 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 -- start start 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.299+0000 7f61a6985740 1 Processor -- start 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d0370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4d2f60 0x5600cb4d08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb431030 con 0x5600cb4ca3e0 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb104430 con 0x5600cb4c9a30 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb0c0e00 con 0x5600cb4d2f60 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a6985740 10 client.0 _ll_register_callbacks cb 0x5600cb426030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1351640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1351640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47594/0 (socket says 172.21.15.3:47594) 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1351640 1 -- 172.21.15.3:0/3934480064 learned_addr learned my addr 172.21.15.3:0/3934480064 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a2353640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4d2f60 0x5600cb4d08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1b52640 1 -- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4d2f60 msgr2=0x5600cb4d08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4d2f60 0x5600cb4d08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1b52640 1 -- 172.21.15.3:0/3934480064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 msgr2=0x5600cb4d0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1b52640 1 -- 172.21.15.3:0/3934480064 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c1a4530 con 0x5600cb4c9a30 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.300+0000 7f61a1351640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a1b52640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4cfe30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x5600cb4d5aa0 tx=0x7f61980bce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0f54edc2e4728e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61857fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61857fa640 1 -- 172.21.15.3:0/3934480064 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61980c15c0 con 0x5600cb4c9a30 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5600cb1307e0 con 0x5600cb4c9a30 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61857fa640 1 -- 172.21.15.3:0/3934480064 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f61980f3e40 con 0x5600cb4c9a30 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61857fa640 1 -- 172.21.15.3:0/3934480064 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61980c13a0 con 0x5600cb4c9a30 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 10 client.25715 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5600cb42f6b0 con 0x5600cb4c9a30 2024-02-09T22:53:15.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61857fa640 1 -- 172.21.15.3:0/3934480064 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6198086080 con 0x5600cb4c9a30 2024-02-09T22:53:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61857fa640 1 -- 172.21.15.3:0/3934480064 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f619803c070 con 0x5600cb4c9a30 2024-02-09T22:53:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 10 client.25715 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 10 client.25715 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 20 client.25715 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb4e2ce0 0x5600cb4e50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 4 client.25715 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.301+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600cb16d720 con 0x5600cb4e2ce0 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.302+0000 7f61a1351640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb4e2ce0 0x5600cb4e50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.302+0000 7f61857fa640 10 client.25715 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.302+0000 7f61a1351640 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb4e2ce0 0x5600cb4e50d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5f01ecb472b031ba server_cookie=45f5b3aa73f77d25 in_seq=0 out_seq=0 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.302+0000 7f61857fa640 1 -- 172.21.15.3:0/3934480064 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5600cb16d720 con 0x5600cb4e2ce0 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.302+0000 7f61857fa640 10 client.25715 handle_command_reply: tid=0 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.302+0000 7f61a6985740 1 client.25715 shutdown 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 msgr2=0x5600cb4cfe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4cfe30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x5600cb4d5aa0 tx=0x7f61980bce50 comp rx=0 tx=0).stop 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 shutdown_connections 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb4e2ce0 0x5600cb4e50d0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4d2f60 0x5600cb4d08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61857fa640 0 client.25715 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 --2- 172.21.15.3:0/3934480064 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4cfe30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 >> 172.21.15.3:0/3934480064 conn(0x5600cb4c4100 msgr2=0x5600cb4c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 shutdown_connections 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 1 -- 172.21.15.3:0/3934480064 wait complete. 2024-02-09T22:53:15.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.303+0000 7f61a6985740 20 client.25715 trim_cache size 0 max 16384 2024-02-09T22:53:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.314+0000 7f61a6985740 1 Processor -- start 2024-02-09T22:53:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.314+0000 7f61a6985740 1 -- start start 2024-02-09T22:53:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a6985740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4d2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a6985740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a6985740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb4cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a6985740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb3b6130 con 0x5600cb4d2f60 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a6985740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb4d0fa0 con 0x5600cb4ca3e0 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a6985740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb16d720 con 0x5600cb4c9a30 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb4cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb4cd090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47608/0 (socket says 172.21.15.3:47608) 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 -- 172.21.15.3:0/2875415750 learned_addr learned my addr 172.21.15.3:0/2875415750 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a1b52640 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4d2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a1351640 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 -- 172.21.15.3:0/2875415750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 msgr2=0x5600cb4d2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4d2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 -- 172.21.15.3:0/2875415750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4ca3e0 msgr2=0x5600cb4d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 -- 172.21.15.3:0/2875415750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600cb3cbc90 con 0x5600cb4d2f60 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a1351640 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61a2353640 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb4cd090 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6194014110 tx=0x7f6194005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6c918985df434e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61877fe640 1 -- 172.21.15.3:0/2875415750 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194003e00 con 0x5600cb4d2f60 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61877fe640 1 -- 172.21.15.3:0/2875415750 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6194014140 con 0x5600cb4d2f60 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.315+0000 7f61877fe640 1 -- 172.21.15.3:0/2875415750 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194012750 con 0x5600cb4d2f60 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 -- 172.21.15.3:0/2875415750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 msgr2=0x5600cb4cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb4cd090 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f6194014110 tx=0x7f6194005bd0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 -- 172.21.15.3:0/2875415750 shutdown_connections 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb4cd090 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 --2- 172.21.15.3:0/2875415750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4c9a30 0x5600cb4d2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 -- 172.21.15.3:0/2875415750 >> 172.21.15.3:0/2875415750 conn(0x5600cb4c4100 msgr2=0x5600cb4e9a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 -- 172.21.15.3:0/2875415750 shutdown_connections 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 -- 172.21.15.3:0/2875415750 wait complete. 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 -- start start 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.316+0000 7f61a6985740 1 Processor -- start 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4d88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb5391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb4d9560 con 0x5600cb4d2f60 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb4d96d0 con 0x5600cb4c9a30 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600cb539710 con 0x5600cb4ca3e0 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a6985740 10 client.0 _ll_register_callbacks cb 0x7f61980bd240 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a2353640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb5391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a2353640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb5391d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47618/0 (socket says 172.21.15.3:47618) 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a2353640 1 -- 172.21.15.3:0/4153632512 learned_addr learned my addr 172.21.15.3:0/4153632512 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1b52640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4d88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 -- 172.21.15.3:0/4153632512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 msgr2=0x5600cb4d88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4d88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 -- 172.21.15.3:0/4153632512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 msgr2=0x5600cb5391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb5391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 -- 172.21.15.3:0/4153632512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6194019040 con 0x5600cb4ca3e0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a2353640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb5391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61a1351640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d8de0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f61980bd120 tx=0x7f618c01d810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3fb1375618b3436 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61857fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.317+0000 7f61857fa640 1 -- 172.21.15.3:0/4153632512 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c017040 con 0x5600cb4ca3e0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61980035f0 con 0x5600cb4ca3e0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61857fa640 1 -- 172.21.15.3:0/4153632512 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f618c012cf0 con 0x5600cb4ca3e0 2024-02-09T22:53:15.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61857fa640 1 -- 172.21.15.3:0/4153632512 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f618c005020 con 0x5600cb4ca3e0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 10 client.25756 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5600cb3f1a70 con 0x5600cb4ca3e0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61857fa640 1 -- 172.21.15.3:0/4153632512 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61980035f0 con 0x5600cb4ca3e0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61857fa640 1 -- 172.21.15.3:0/4153632512 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f618c011440 con 0x5600cb4ca3e0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 10 client.25756 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 10 client.25756 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 20 client.25756 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb53afa0 0x5600cb53d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 4 client.25756 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5600cb1bc890 con 0x5600cb53afa0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.318+0000 7f61a1b52640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb53afa0 0x5600cb53d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61857fa640 10 client.25756 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a1b52640 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb53afa0 0x5600cb53d3c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=54d8d8c8fd9ee2bb server_cookie=37295db89ceb3cb6 in_seq=0 out_seq=0 2024-02-09T22:53:15.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61857fa640 1 -- 172.21.15.3:0/4153632512 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5600cb1bc890 con 0x5600cb53afa0 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61857fa640 10 client.25756 handle_command_reply: tid=0 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 client.25756 shutdown 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4ca3e0 msgr2=0x5600cb4d8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d8de0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f61980bd120 tx=0x7f618c01d810 comp rx=0 tx=0).stop 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 shutdown_connections 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5600cb53afa0 0x5600cb53d3c0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5600cb4d2f60 0x5600cb5391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5600cb4ca3e0 0x5600cb4d8de0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 --2- 172.21.15.3:0/4153632512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5600cb4c9a30 0x5600cb4d88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61857fa640 0 client.25756 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 >> 172.21.15.3:0/4153632512 conn(0x5600cb4c4100 msgr2=0x5600cb4c29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 shutdown_connections 2024-02-09T22:53:15.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 1 -- 172.21.15.3:0/4153632512 wait complete. 2024-02-09T22:53:15.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a6985740 20 client.25756 trim_cache size 0 max 16384 2024-02-09T22:53:15.336 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61880a2230 msgr2=0x7f61880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a3db6640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61880a2230 0x7f61880a46f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6194005c40 tx=0x7f6194009500 comp rx=0 tx=0).stop 2024-02-09T22:53:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 msgr2=0x7f619c19d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.319+0000 7f61a3db6640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c19d5b0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f6198005940 tx=0x7f619802c780 comp rx=0 tx=0).stop 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 shutdown_connections 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61880a2230 0x7f61880a46f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f619c102d70 0x7f619c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f619c10b820 0x7f619c197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 --2- 172.21.15.3:0/2043434263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f619c102450 0x7f619c19d5b0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 >> 172.21.15.3:0/2043434263 conn(0x7f619c077280 msgr2=0x7f619c0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 shutdown_connections 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.320+0000 7f61a3db6640 1 -- 172.21.15.3:0/2043434263 wait complete. 2024-02-09T22:53:15.338 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:15.338 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 -- 172.21.15.3:0/1663324616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102d70 msgr2=0x7efe2810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 --2- 172.21.15.3:0/1663324616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102d70 0x7efe2810b250 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efe18009900 tx=0x7efe1802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 -- 172.21.15.3:0/1663324616 shutdown_connections 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 --2- 172.21.15.3:0/1663324616 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe2810b820 0x7efe2810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 --2- 172.21.15.3:0/1663324616 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102d70 0x7efe2810b250 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 --2- 172.21.15.3:0/1663324616 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102450 0x7efe28102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 -- 172.21.15.3:0/1663324616 >> 172.21.15.3:0/1663324616 conn(0x7efe280772e0 msgr2=0x7efe280776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 -- 172.21.15.3:0/1663324616 shutdown_connections 2024-02-09T22:53:15.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 -- 172.21.15.3:0/1663324616 wait complete. 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 Processor -- start 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.441+0000 7efe3060e640 1 -- start start 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 0x7efe2819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102d70 0x7efe2819d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe2810b820 0x7efe281975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28197c10 con 0x7efe28102450 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28197d80 con 0x7efe2810b820 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe28197f40 con 0x7efe28102d70 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 0x7efe2819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 0x7efe2819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47624/0 (socket says 172.21.15.3:47624) 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 -- 172.21.15.3:0/2358737342 learned_addr learned my addr 172.21.15.3:0/2358737342 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2dba9640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102d70 0x7efe2819d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe2810b820 0x7efe281975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 -- 172.21.15.3:0/2358737342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102d70 msgr2=0x7efe2819d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102d70 0x7efe2819d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 -- 172.21.15.3:0/2358737342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe2810b820 msgr2=0x7efe281975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe2810b820 0x7efe281975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 -- 172.21.15.3:0/2358737342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18009560 con 0x7efe28102450 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2dba9640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102d70 0x7efe2819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 0x7efe2819d410 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efe2000c920 tx=0x7efe2000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe1f7fe640 1 -- 172.21.15.3:0/2358737342 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20007c70 con 0x7efe28102450 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe1f7fe640 1 -- 172.21.15.3:0/2358737342 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe20007dd0 con 0x7efe28102450 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe1f7fe640 1 -- 172.21.15.3:0/2358737342 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20018860 con 0x7efe28102450 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.442+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe28198220 con 0x7efe28102450 2024-02-09T22:53:15.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.443+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281a44b0 con 0x7efe28102450 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe1f7fe640 1 -- 172.21.15.3:0/2358737342 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efe2000fac0 con 0x7efe28102450 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 Processor -- start 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 -- start start 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc4be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc54990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc54f60 0x562b4cc573b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cc07020 con 0x562b4cc4c3e0 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cbba320 con 0x562b4cc54f60 2024-02-09T22:53:15.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe331e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cbb7850 con 0x562b4cc4ba30 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe2e3aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc54990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe2e3aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc54990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47626/0 (socket says 172.21.15.3:47626) 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe2e3aa640 1 -- 172.21.15.3:0/1953810597 learned_addr learned my addr 172.21.15.3:0/1953810597 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe2ebab640 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc4be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe1f7fe640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe140a2230 0x7efe140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.446+0000 7efe2dba9640 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc54f60 0x562b4cc573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2e3aa640 1 -- 172.21.15.3:0/1953810597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 msgr2=0x562b4cc4be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc4be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2e3aa640 1 -- 172.21.15.3:0/1953810597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc54f60 msgr2=0x562b4cc573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc54f60 0x562b4cc573b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2e3aa640 1 -- 172.21.15.3:0/1953810597 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b4cb75210 con 0x562b4cc4c3e0 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2ebab640 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc4be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe140a2230 0x7efe140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe1f7fe640 1 -- 172.21.15.3:0/2358737342 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe200ce0a0 con 0x7efe28102450 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc54990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efe200cb830 tx=0x7efe2000fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a029f38ac3c2f703 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe140a2230 0x7efe140a46f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe24005fd0 tx=0x7efe24014040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe1cbfe640 1 -- 172.21.15.3:0/1953810597 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe200d1770 con 0x562b4cc4c3e0 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe1cbfe640 1 -- 172.21.15.3:0/1953810597 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe2009be90 con 0x562b4cc4c3e0 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.447+0000 7efe1cbfe640 1 -- 172.21.15.3:0/1953810597 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2009ab60 con 0x562b4cc4c3e0 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 -- 172.21.15.3:0/1953810597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 msgr2=0x562b4cc54990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc54990 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7efe200cb830 tx=0x7efe2000fec0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 -- 172.21.15.3:0/1953810597 shutdown_connections 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc54f60 0x562b4cc573b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc54990 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 --2- 172.21.15.3:0/1953810597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc4be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 -- 172.21.15.3:0/1953810597 >> 172.21.15.3:0/1953810597 conn(0x562b4cc46100 msgr2=0x562b4cc464e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 -- 172.21.15.3:0/1953810597 shutdown_connections 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 -- 172.21.15.3:0/1953810597 wait complete. 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 -- start start 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 Processor -- start 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc51f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.448+0000 7efe331e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4c3e0 0x562b4cc52480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cbb3030 con 0x562b4cc54f60 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4c886430 con 0x562b4cc4c3e0 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4c842e00 con 0x562b4cc4ba30 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 10 client.0 _ll_register_callbacks cb 0x562b4cba8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc529c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47634/0 (socket says 172.21.15.3:47634) 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 -- 172.21.15.3:0/2454669541 learned_addr learned my addr 172.21.15.3:0/2454669541 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2e3aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc51f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2dba9640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4c3e0 0x562b4cc52480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 -- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 msgr2=0x562b4cc51f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc51f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 -- 172.21.15.3:0/2454669541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4c3e0 msgr2=0x562b4cc52480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4c3e0 0x562b4cc52480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 -- 172.21.15.3:0/2454669541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe281a44b0 con 0x562b4cc54f60 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc51f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc529c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efe240108e0 tx=0x7efe24009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbb7dafe98393c94 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe11ffb640 1 -- 172.21.15.3:0/2454669541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2401c890 con 0x562b4cc54f60 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562b4c8b27e0 con 0x562b4cc54f60 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe11ffb640 1 -- 172.21.15.3:0/2454669541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe2401c9f0 con 0x562b4cc54f60 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe11ffb640 1 -- 172.21.15.3:0/2454669541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe2401cbb0 con 0x562b4cc54f60 2024-02-09T22:53:15.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 10 client.16260 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.449+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562b4cbb16b0 con 0x562b4cc54f60 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe11ffb640 1 -- 172.21.15.3:0/2454669541 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562b4c8b27e0 con 0x562b4cc54f60 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe11ffb640 1 -- 172.21.15.3:0/2454669541 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7efe24027440 con 0x562b4cc54f60 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe331e8740 10 client.16260 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe331e8740 10 client.16260 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe331e8740 20 client.16260 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe331e8740 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4cc65580 0x562b4cc679a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe331e8740 4 client.16260 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b4c8ef720 con 0x562b4cc65580 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4cc65580 0x562b4cc679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4cc65580 0x562b4cc679a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8896614fef4668c6 server_cookie=6754e8ccd1ada2d9 in_seq=0 out_seq=0 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.450+0000 7efe11ffb640 10 client.16260 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe11ffb640 1 -- 172.21.15.3:0/2454669541 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b4c8ef720 con 0x562b4cc65580 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe11ffb640 10 client.16260 handle_command_reply: tid=0 2024-02-09T22:53:15.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 client.16260 shutdown 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 msgr2=0x562b4cc529c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc529c0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efe240108e0 tx=0x7efe24009500 comp rx=0 tx=0).stop 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 shutdown_connections 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4cc65580 0x562b4cc679a0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4c3e0 0x562b4cc52480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc529c0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe11ffb640 0 client.16260 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 --2- 172.21.15.3:0/2454669541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4ba30 0x562b4cc51f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 >> 172.21.15.3:0/2454669541 conn(0x562b4cc46100 msgr2=0x562b4cc47e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 shutdown_connections 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 1 -- 172.21.15.3:0/2454669541 wait complete. 2024-02-09T22:53:15.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.451+0000 7efe331e8740 20 client.16260 trim_cache size 0 max 16384 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 Processor -- start 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 -- start start 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc54670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4c3e0 0x562b4cc4c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc527e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cb38130 con 0x562b4cc54f60 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cc62070 con 0x562b4cc4ba30 2024-02-09T22:53:15.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe331e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4cc63d80 con 0x562b4cc4c3e0 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe2ebab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc527e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe2ebab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc527e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47648/0 (socket says 172.21.15.3:47648) 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe2ebab640 1 -- 172.21.15.3:0/2406045135 learned_addr learned my addr 172.21.15.3:0/2406045135 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc54670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.463+0000 7efe2dba9640 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4c3e0 0x562b4cc4c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe2ebab640 1 -- 172.21.15.3:0/2406045135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4c3e0 msgr2=0x562b4cc4c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4c3e0 0x562b4cc4c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe2ebab640 1 -- 172.21.15.3:0/2406045135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 msgr2=0x562b4cc54670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc54670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe2ebab640 1 -- 172.21.15.3:0/2406045135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b4cb4dc90 con 0x562b4cc54f60 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe2ebab640 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc527e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efe24004010 tx=0x7efe24009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa587f2c26ac5b4b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe13fff640 1 -- 172.21.15.3:0/2406045135 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe24004490 con 0x562b4cc54f60 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe13fff640 1 -- 172.21.15.3:0/2406045135 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe24027440 con 0x562b4cc54f60 2024-02-09T22:53:15.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe13fff640 1 -- 172.21.15.3:0/2406045135 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2401ca10 con 0x562b4cc54f60 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 -- 172.21.15.3:0/2406045135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 msgr2=0x562b4cc527e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc527e0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7efe24004010 tx=0x7efe24009ef0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 -- 172.21.15.3:0/2406045135 shutdown_connections 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc54f60 0x562b4cc527e0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc4c3e0 0x562b4cc4c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 --2- 172.21.15.3:0/2406045135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc54670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 -- 172.21.15.3:0/2406045135 >> 172.21.15.3:0/2406045135 conn(0x562b4cc46100 msgr2=0x562b4cc41210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 -- 172.21.15.3:0/2406045135 shutdown_connections 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.464+0000 7efe331e8740 1 -- 172.21.15.3:0/2406045135 wait complete. 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 -- start start 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 Processor -- start 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc5a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc5aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc54f60 0x562b4cc5b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4ccbba50 con 0x562b4cc4c3e0 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4ccbbb90 con 0x562b4cc4ba30 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b4ccbbcd0 con 0x562b4cc54f60 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe331e8740 10 client.0 _ll_register_callbacks cb 0x7efe200d1b10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc5aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc5aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47652/0 (socket says 172.21.15.3:47652) 2024-02-09T22:53:15.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2e3aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc5a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 -- 172.21.15.3:0/4013631285 learned_addr learned my addr 172.21.15.3:0/4013631285 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 -- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc54f60 msgr2=0x562b4cc5b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2ebab640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc54f60 0x562b4cc5b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc54f60 0x562b4cc5b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 -- 172.21.15.3:0/4013631285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 msgr2=0x562b4cc5a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc5a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2dba9640 1 -- 172.21.15.3:0/4013631285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b4cbb16b0 con 0x562b4cc4c3e0 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc5a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.465+0000 7efe2ebab640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc54f60 0x562b4cc5b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe2dba9640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc5aee0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efe18004c50 tx=0x7efe180042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63cfe31ebb51f0f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe11ffb640 1 -- 172.21.15.3:0/4013631285 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18044040 con 0x562b4cc4c3e0 2024-02-09T22:53:15.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe11ffb640 1 -- 172.21.15.3:0/4013631285 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe18004df0 con 0x562b4cc4c3e0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe2009be90 con 0x562b4cc4c3e0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe11ffb640 1 -- 172.21.15.3:0/4013631285 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe18004fb0 con 0x562b4cc4c3e0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 10 client.16266 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x562b4cb73a70 con 0x562b4cc4c3e0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe11ffb640 1 -- 172.21.15.3:0/4013631285 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe2009be90 con 0x562b4cc4c3e0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe11ffb640 1 -- 172.21.15.3:0/4013631285 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7efe18047030 con 0x562b4cc4c3e0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 10 client.16266 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 10 client.16266 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 20 client.16266 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4ccbd530 0x562b4ccbf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 4 client.16266 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.466+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562b4c93e890 con 0x562b4ccbd530 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4ccbd530 0x562b4ccbf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe2e3aa640 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4ccbd530 0x562b4ccbf950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=70220b8055bbcfe9 server_cookie=bd7160fb47698ab6 in_seq=0 out_seq=0 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe11ffb640 10 client.16266 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe11ffb640 1 -- 172.21.15.3:0/4013631285 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562b4c93e890 con 0x562b4ccbd530 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe11ffb640 10 client.16266 handle_command_reply: tid=0 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 client.16266 shutdown 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 msgr2=0x562b4cc5aee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc5aee0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7efe18004c50 tx=0x7efe180042e0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 shutdown_connections 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562b4ccbd530 0x562b4ccbf950 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562b4cc54f60 0x562b4cc5b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562b4cc4c3e0 0x562b4cc5aee0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe11ffb640 0 client.16266 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 --2- 172.21.15.3:0/4013631285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562b4cc4ba30 0x562b4cc5a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 >> 172.21.15.3:0/4013631285 conn(0x562b4cc46100 msgr2=0x562b4cc4d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 shutdown_connections 2024-02-09T22:53:15.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 1 -- 172.21.15.3:0/4013631285 wait complete. 2024-02-09T22:53:15.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.467+0000 7efe331e8740 20 client.16266 trim_cache size 0 max 16384 2024-02-09T22:53:15.480 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:15.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe140a2230 msgr2=0x7efe140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe140a2230 0x7efe140a46f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe24005fd0 tx=0x7efe24014040 comp rx=0 tx=0).stop 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 msgr2=0x7efe2819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 0x7efe2819d410 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efe2000c920 tx=0x7efe2000cde0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 shutdown_connections 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe140a2230 0x7efe140a46f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe2810b820 0x7efe281975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe28102d70 0x7efe2819d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 --2- 172.21.15.3:0/2358737342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe28102450 0x7efe2819d410 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 >> 172.21.15.3:0/2358737342 conn(0x7efe280772e0 msgr2=0x7efe280fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 shutdown_connections 2024-02-09T22:53:15.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.468+0000 7efe3060e640 1 -- 172.21.15.3:0/2358737342 wait complete. 2024-02-09T22:53:15.483 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:15.483 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:53:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 -- 172.21.15.3:0/2236709917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c100cf0 msgr2=0x7f379c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 --2- 172.21.15.3:0/2236709917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c100cf0 0x7f379c104d30 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f3790009930 tx=0x7f379002bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:15.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 -- 172.21.15.3:0/2236709917 shutdown_connections 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 --2- 172.21.15.3:0/2236709917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f379c105410 0x7f379c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 --2- 172.21.15.3:0/2236709917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c100cf0 0x7f379c104d30 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 --2- 172.21.15.3:0/2236709917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f379c100340 0x7f379c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 -- 172.21.15.3:0/2236709917 >> 172.21.15.3:0/2236709917 conn(0x7f379c0772e0 msgr2=0x7f379c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 -- 172.21.15.3:0/2236709917 shutdown_connections 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.585+0000 7f37a32eb640 1 -- 172.21.15.3:0/2236709917 wait complete. 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 Processor -- start 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 -- start start 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f379c100340 0x7f379c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 0x7f379c19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f379c1976d0 0x7f379c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c1981a0 con 0x7f379c105410 2024-02-09T22:53:15.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c198310 con 0x7f379c100340 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a32eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379c1984d0 con 0x7f379c1976d0 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a0886640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 0x7f379c19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a0886640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 0x7f379c19db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47674/0 (socket says 172.21.15.3:47674) 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a0886640 1 -- 172.21.15.3:0/4252877452 learned_addr learned my addr 172.21.15.3:0/4252877452 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a1087640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f379c100340 0x7f379c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.586+0000 7f37a1888640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f379c1976d0 0x7f379c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a0886640 1 -- 172.21.15.3:0/4252877452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f379c1976d0 msgr2=0x7f379c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a0886640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f379c1976d0 0x7f379c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a0886640 1 -- 172.21.15.3:0/4252877452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f379c100340 msgr2=0x7f379c19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a0886640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f379c100340 0x7f379c19d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a0886640 1 -- 172.21.15.3:0/4252877452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3790009560 con 0x7f379c105410 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a0886640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 0x7f379c19db50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3790009230 tx=0x7f379002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f379a7fc640 1 -- 172.21.15.3:0/4252877452 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790004350 con 0x7f379c105410 2024-02-09T22:53:15.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379c1a4410 con 0x7f379c105410 2024-02-09T22:53:15.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f379a7fc640 1 -- 172.21.15.3:0/4252877452 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37900044b0 con 0x7f379c105410 2024-02-09T22:53:15.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f379a7fc640 1 -- 172.21.15.3:0/4252877452 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3790003da0 con 0x7f379c105410 2024-02-09T22:53:15.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.587+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f379c1a48f0 con 0x7f379c105410 2024-02-09T22:53:15.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.589+0000 7f379a7fc640 1 -- 172.21.15.3:0/4252877452 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3790004df0 con 0x7f379c105410 2024-02-09T22:53:15.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 Processor -- start 2024-02-09T22:53:15.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 -- start start 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 0x557c8cab2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cabbdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cabc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8ca6e020 con 0x557c8cabc3c0 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f379a7fc640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f37840a2230 0x7f37840a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8ca21320 con 0x557c8cab2910 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a5f98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8ca1e850 con 0x557c8cab32c0 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a0886640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cabc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 0x557c8cab2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 0x557c8cab2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:58668/0 (socket says 172.21.15.3:58668) 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 -- 172.21.15.3:0/1692744241 learned_addr learned my addr 172.21.15.3:0/1692744241 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1087640 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cabbdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f37840a2230 0x7f37840a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f379a7fc640 1 -- 172.21.15.3:0/4252877452 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f37900f0080 con 0x7f379c105410 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 -- 172.21.15.3:0/1692744241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 msgr2=0x557c8cabbdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cabbdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 -- 172.21.15.3:0/1692744241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 msgr2=0x557c8cabc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.590+0000 7f37a1888640 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cabc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.591+0000 7f37a1888640 1 -- 172.21.15.3:0/1692744241 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c8c9dc210 con 0x557c8cab2910 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.591+0000 7f37a1888640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f37840a2230 0x7f37840a46f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f378c00fc90 tx=0x7f378c018000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.591+0000 7f37a1888640 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 0x557c8cab2cf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f378c005f40 tx=0x7f378c00f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e1dd700f9a81705 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.591+0000 7f37837fe640 1 -- 172.21.15.3:0/1692744241 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c00ec50 con 0x557c8cab2910 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.591+0000 7f37837fe640 1 -- 172.21.15.3:0/1692744241 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f378c009dc0 con 0x557c8cab2910 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.591+0000 7f37837fe640 1 -- 172.21.15.3:0/1692744241 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c004270 con 0x557c8cab2910 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 -- 172.21.15.3:0/1692744241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 msgr2=0x557c8cab2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 0x557c8cab2cf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f378c005f40 tx=0x7f378c00f790 comp rx=0 tx=0).stop 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 -- 172.21.15.3:0/1692744241 shutdown_connections 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cabc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cabbdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1692744241 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab2910 0x557c8cab2cf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 -- 172.21.15.3:0/1692744241 >> 172.21.15.3:0/1692744241 conn(0x557c8caad100 msgr2=0x557c8caad4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 -- 172.21.15.3:0/1692744241 shutdown_connections 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 -- 172.21.15.3:0/1692744241 wait complete. 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 -- start start 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.592+0000 7f37a5f98740 1 Processor -- start 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab2910 0x557c8cabfed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cac0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8ca1a030 con 0x557c8cabc3c0 2024-02-09T22:53:15.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8c6ed430 con 0x557c8cab32c0 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8c6a9e00 con 0x557c8cab2910 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 10 client.0 _ll_register_callbacks cb 0x557c8ca0f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cac0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1087640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab2910 0x557c8cabfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cac0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47690/0 (socket says 172.21.15.3:47690) 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 -- 172.21.15.3:0/602395246 learned_addr learned my addr 172.21.15.3:0/602395246 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a0886640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 -- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab2910 msgr2=0x557c8cabfed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab2910 0x557c8cabfed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 -- 172.21.15.3:0/602395246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 msgr2=0x557c8cac0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 -- 172.21.15.3:0/602395246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378c019040 con 0x557c8cabc3c0 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1087640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab2910 0x557c8cabfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a1888640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cac0950 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f378c003c70 tx=0x7f378c00ffd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3387f927f9ce35b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f3780ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f3780ff9640 1 -- 172.21.15.3:0/602395246 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f378c02b070 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f3780ff9640 1 -- 172.21.15.3:0/602395246 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f378c011890 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557c8c7197e0 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.593+0000 7f3780ff9640 1 -- 172.21.15.3:0/602395246 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f378c00ec20 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 10 client.16278 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557c8ca186b0 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f3780ff9640 1 -- 172.21.15.3:0/602395246 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557c8c7197e0 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f3780ff9640 1 -- 172.21.15.3:0/602395246 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f378c030540 con 0x557c8cabc3c0 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 10 client.16278 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 10 client.16278 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:53:15.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 20 client.16278 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cad0950 0x557c8cad2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 4 client.16278 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c8c756720 con 0x557c8cad0950 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.594+0000 7f37a1087640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cad0950 0x557c8cad2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a1087640 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cad0950 0x557c8cad2d40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=12cfa5463198ed2e server_cookie=be6659e46998ba0b in_seq=0 out_seq=0 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f3780ff9640 10 client.16278 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f3780ff9640 1 -- 172.21.15.3:0/602395246 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557c8c756720 con 0x557c8cad0950 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f3780ff9640 10 client.16278 handle_command_reply: tid=0 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 client.16278 shutdown 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 msgr2=0x557c8cac0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cac0950 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f378c003c70 tx=0x7f378c00ffd0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 shutdown_connections 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cad0950 0x557c8cad2d40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cabc3c0 0x557c8cac0950 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f3780ff9640 0 client.16278 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 --2- 172.21.15.3:0/602395246 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab2910 0x557c8cabfed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.595+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 >> 172.21.15.3:0/602395246 conn(0x557c8caad100 msgr2=0x557c8caaee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.596+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 shutdown_connections 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.596+0000 7f37a5f98740 1 -- 172.21.15.3:0/602395246 wait complete. 2024-02-09T22:53:15.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.596+0000 7f37a5f98740 20 client.16278 trim_cache size 0 max 16384 2024-02-09T22:53:15.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 Processor -- start 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 -- start start 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cabc3c0 0x557c8cabd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8c99f130 con 0x557c8cab2910 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8cad7f50 con 0x557c8cab32c0 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.607+0000 7f37a5f98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8cab2540 con 0x557c8cabc3c0 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47698/0 (socket says 172.21.15.3:47698) 2024-02-09T22:53:15.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 -- 172.21.15.3:0/1475438951 learned_addr learned my addr 172.21.15.3:0/1475438951 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a0886640 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 -- 172.21.15.3:0/1475438951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cabc3c0 msgr2=0x557c8cabd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cabc3c0 0x557c8cabd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 -- 172.21.15.3:0/1475438951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 msgr2=0x557c8cac0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 -- 172.21.15.3:0/1475438951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c8c9b4c90 con 0x557c8cab2910 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f37a1087640 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac2600 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x557c8cad3650 tx=0x7f3794005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebc96196354de3d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f3782ffd640 1 -- 172.21.15.3:0/1475438951 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794025030 con 0x557c8cab2910 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f3782ffd640 1 -- 172.21.15.3:0/1475438951 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3794018040 con 0x557c8cab2910 2024-02-09T22:53:15.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.608+0000 7f3782ffd640 1 -- 172.21.15.3:0/1475438951 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794017040 con 0x557c8cab2910 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- 172.21.15.3:0/1475438951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 msgr2=0x557c8cac2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac2600 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x557c8cad3650 tx=0x7f3794005f90 comp rx=0 tx=0).stop 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- 172.21.15.3:0/1475438951 shutdown_connections 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cab32c0 0x557c8cac0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cabc3c0 0x557c8cabd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- 172.21.15.3:0/1475438951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac2600 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- 172.21.15.3:0/1475438951 >> 172.21.15.3:0/1475438951 conn(0x557c8caad100 msgr2=0x557c8caba220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- 172.21.15.3:0/1475438951 shutdown_connections 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- 172.21.15.3:0/1475438951 wait complete. 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- start start 2024-02-09T22:53:15.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 Processor -- start 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cac6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cabc3c0 0x557c8cb20e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8cb213b0 con 0x557c8cab2910 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8cb214f0 con 0x557c8cabc3c0 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.609+0000 7f37a5f98740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c8cb216b0 con 0x557c8cab32c0 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a5f98740 10 client.0 _ll_register_callbacks cb 0x7f378c011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47706/0 (socket says 172.21.15.3:47706) 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 -- 172.21.15.3:0/73313589 learned_addr learned my addr 172.21.15.3:0/73313589 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a0886640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cac6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1888640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cabc3c0 0x557c8cb20e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 -- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 msgr2=0x557c8cac6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cac6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 -- 172.21.15.3:0/73313589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cabc3c0 msgr2=0x557c8cb20e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cabc3c0 0x557c8cb20e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 -- 172.21.15.3:0/73313589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3794012690 con 0x557c8cab2910 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1888640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cabc3c0 0x557c8cb20e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a1087640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac6670 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3794009fb0 tx=0x7f3794012110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee99b99fa2700d7c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f3780ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f3780ff9640 1 -- 172.21.15.3:0/73313589 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3794004040 con 0x557c8cab2910 2024-02-09T22:53:15.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f3780ff9640 1 -- 172.21.15.3:0/73313589 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3794026070 con 0x557c8cab2910 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f378c009dc0 con 0x557c8cab2910 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f3780ff9640 1 -- 172.21.15.3:0/73313589 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37940041a0 con 0x557c8cab2910 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a5f98740 10 client.16284 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x557c8c9daa70 con 0x557c8cab2910 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.610+0000 7f3780ff9640 1 -- 172.21.15.3:0/73313589 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f378c009dc0 con 0x557c8cab2910 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f3780ff9640 1 -- 172.21.15.3:0/73313589 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f3794004b30 con 0x557c8cab2910 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 10 client.16284 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 10 client.16284 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 20 client.16284 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cb28f50 0x557c8cb2b370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 4 client.16284 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:15.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557c8c7a5890 con 0x557c8cb28f50 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a0886640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cb28f50 0x557c8cb2b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a0886640 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cb28f50 0x557c8cb2b370 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d4a278c81ac8ac69 server_cookie=b72faf75da94e334 in_seq=0 out_seq=0 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f3780ff9640 10 client.16284 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f3780ff9640 1 -- 172.21.15.3:0/73313589 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557c8c7a5890 con 0x557c8cb28f50 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f3780ff9640 10 client.16284 handle_command_reply: tid=0 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.611+0000 7f37a5f98740 1 client.16284 shutdown 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 msgr2=0x557c8cac6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac6670 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f3794009fb0 tx=0x7f3794012110 comp rx=0 tx=0).stop 2024-02-09T22:53:15.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 shutdown_connections 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x557c8cb28f50 0x557c8cb2b370 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557c8cabc3c0 0x557c8cb20e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557c8cab32c0 0x557c8cac6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 --2- 172.21.15.3:0/73313589 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557c8cab2910 0x557c8cac6670 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f3780ff9640 0 client.16284 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 >> 172.21.15.3:0/73313589 conn(0x557c8caad100 msgr2=0x557c8cabad60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 shutdown_connections 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 1 -- 172.21.15.3:0/73313589 wait complete. 2024-02-09T22:53:15.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a5f98740 20 client.16284 trim_cache size 0 max 16384 2024-02-09T22:53:15.625 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f37840a2230 msgr2=0x7f37840a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f37840a2230 0x7f37840a46f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f378c00fc90 tx=0x7f378c018000 comp rx=0 tx=0).stop 2024-02-09T22:53:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 msgr2=0x7f379c19db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 0x7f379c19db50 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f3790009230 tx=0x7f379002c680 comp rx=0 tx=0).stop 2024-02-09T22:53:15.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 shutdown_connections 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f37840a2230 0x7f37840a46f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f379c1976d0 0x7f379c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f379c105410 0x7f379c19db50 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 --2- 172.21.15.3:0/4252877452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f379c100340 0x7f379c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 >> 172.21.15.3:0/4252877452 conn(0x7f379c0772e0 msgr2=0x7f379c0fe360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 shutdown_connections 2024-02-09T22:53:15.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.612+0000 7f37a32eb640 1 -- 172.21.15.3:0/4252877452 wait complete. 2024-02-09T22:53:15.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 50355 4.19 MB/sec warmup 69 sec latency 882.215 ms 2024-02-09T22:53:15.628 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:15.628 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:53:15.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 -- 172.21.15.3:0/556435947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98102440 msgr2=0x7fda98102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/556435947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98102440 0x7fda98102820 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7fda94009930 tx=0x7fda9402bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 -- 172.21.15.3:0/556435947 shutdown_connections 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/556435947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda98102df0 0x7fda98107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/556435947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda98107930 0x7fda9810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/556435947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98102440 0x7fda98102820 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 -- 172.21.15.3:0/556435947 >> 172.21.15.3:0/556435947 conn(0x7fda98077310 msgr2=0x7fda98077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 -- 172.21.15.3:0/556435947 shutdown_connections 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.730+0000 7fda9fa1e640 1 -- 172.21.15.3:0/556435947 wait complete. 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 Processor -- start 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 -- start start 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda98102440 0x7fda9819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 0x7fda9819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda98197840 0x7fda98197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981982f0 con 0x7fda98107930 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda98198460 con 0x7fda98197840 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9fa1e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda981985d0 con 0x7fda98102440 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 0x7fda9819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 0x7fda9819dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47728/0 (socket says 172.21.15.3:47728) 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 -- 172.21.15.3:0/3703542095 learned_addr learned my addr 172.21.15.3:0/3703542095 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda98197840 0x7fda98197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda98102440 0x7fda9819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 -- 172.21.15.3:0/3703542095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda98102440 msgr2=0x7fda9819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda98102440 0x7fda9819d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 -- 172.21.15.3:0/3703542095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda98197840 msgr2=0x7fda98197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda98197840 0x7fda98197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.731+0000 7fda9cfb9640 1 -- 172.21.15.3:0/3703542095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94009560 con 0x7fda98107930 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.732+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 0x7fda9819dbc0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e9f0 tx=0x7fda8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.732+0000 7fda8e7fc640 1 -- 172.21.15.3:0/3703542095 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880096a0 con 0x7fda98107930 2024-02-09T22:53:15.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.732+0000 7fda8e7fc640 1 -- 172.21.15.3:0/3703542095 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8800ccc0 con 0x7fda98107930 2024-02-09T22:53:15.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.732+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda981a4530 con 0x7fda98107930 2024-02-09T22:53:15.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.732+0000 7fda8e7fc640 1 -- 172.21.15.3:0/3703542095 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88018860 con 0x7fda98107930 2024-02-09T22:53:15.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.732+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda981a4a10 con 0x7fda98107930 2024-02-09T22:53:15.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.734+0000 7fda8e7fc640 1 -- 172.21.15.3:0/3703542095 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fda88010aa0 con 0x7fda98107930 2024-02-09T22:53:15.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.734+0000 7fdaa25f0740 1 Processor -- start 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.734+0000 7fdaa25f0740 1 -- start start 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.734+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 0x563aff866e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff86ff60 0x563aff8723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff822020 con 0x563aff86ff60 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff7d5320 con 0x563aff8673e0 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff7d2850 con 0x563aff866a30 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff86ff60 0x563aff8723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 0x563aff866e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff86ff60 0x563aff8723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47736/0 (socket says 172.21.15.3:47736) 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 0x563aff866e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46950/0 (socket says 172.21.15.3:46950) 2024-02-09T22:53:15.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9cfb9640 1 -- 172.21.15.3:0/3941911823 learned_addr learned my addr 172.21.15.3:0/3941911823 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda8e7fc640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda840a2230 0x7fda840a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 -- 172.21.15.3:0/3941911823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 msgr2=0x563aff86f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 -- 172.21.15.3:0/3941911823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff86ff60 msgr2=0x563aff8723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff86ff60 0x563aff8723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 -- 172.21.15.3:0/3941911823 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563aff790210 con 0x563aff866a30 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda8e7fc640 1 -- 172.21.15.3:0/3703542095 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda88014070 con 0x7fda98107930 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda840a2230 0x7fda840a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.735+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 0x563aff866e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fda900103f0 tx=0x7fda900108b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cc24092376a2b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda840a2230 0x7fda840a46f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fda90021000 tx=0x7fda9001e000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fda837fe640 1 -- 172.21.15.3:0/3941911823 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda900136c0 con 0x563aff866a30 2024-02-09T22:53:15.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fda837fe640 1 -- 172.21.15.3:0/3941911823 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda90013d00 con 0x563aff866a30 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fda837fe640 1 -- 172.21.15.3:0/3941911823 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9001ebc0 con 0x563aff866a30 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 -- 172.21.15.3:0/3941911823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 msgr2=0x563aff866e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 0x563aff866e10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fda900103f0 tx=0x7fda900108b0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 -- 172.21.15.3:0/3941911823 shutdown_connections 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff86ff60 0x563aff8723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/3941911823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff866a30 0x563aff866e10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 -- 172.21.15.3:0/3941911823 >> 172.21.15.3:0/3941911823 conn(0x563aff861100 msgr2=0x563aff8614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 -- 172.21.15.3:0/3941911823 shutdown_connections 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.736+0000 7fdaa25f0740 1 -- 172.21.15.3:0/3941911823 wait complete. 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 -- start start 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 Processor -- start 2024-02-09T22:53:15.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 0x563aff86cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff86d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff7ce030 con 0x563aff866a30 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff4a1430 con 0x563aff8673e0 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff45de00 con 0x563aff86ff60 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fdaa25f0740 10 client.0 _ll_register_callbacks cb 0x563aff7c3030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fda9d7ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 0x563aff86cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fda9d7ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 0x563aff86cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47744/0 (socket says 172.21.15.3:47744) 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fda9d7ba640 1 -- 172.21.15.3:0/1703144669 learned_addr learned my addr 172.21.15.3:0/1703144669 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.737+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff86d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda9d7ba640 1 -- 172.21.15.3:0/1703144669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 msgr2=0x563aff86d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff86d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda9d7ba640 1 -- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 msgr2=0x563aff86d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda9d7ba640 1 -- 172.21.15.3:0/1703144669 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda90009b50 con 0x563aff866a30 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 0x563aff86cf40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fda94006010 tx=0x7fda9402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae76bbd930a7de5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda80ff9640 1 -- 172.21.15.3:0/1703144669 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94005d50 con 0x563aff866a30 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda80ff9640 1 -- 172.21.15.3:0/1703144669 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda9402cc00 con 0x563aff866a30 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563aff4cd7e0 con 0x563aff866a30 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda80ff9640 1 -- 172.21.15.3:0/1703144669 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda9402c5d0 con 0x563aff866a30 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fdaa25f0740 10 client.16296 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563aff7cc6b0 con 0x563aff866a30 2024-02-09T22:53:15.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda80ff9640 1 -- 172.21.15.3:0/1703144669 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563aff4cd7e0 con 0x563aff866a30 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.738+0000 7fda80ff9640 1 -- 172.21.15.3:0/1703144669 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fda9403c040 con 0x563aff866a30 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fdaa25f0740 10 client.16296 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fdaa25f0740 10 client.16296 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fdaa25f0740 20 client.16296 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff880520 0x563aff882910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fdaa25f0740 4 client.16296 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563aff50a720 con 0x563aff880520 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff880520 0x563aff882910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fda80ff9640 10 client.16296 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.739+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff880520 0x563aff882910 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5268c1c6ed7233a7 server_cookie=f99f34441fc7fb7d in_seq=0 out_seq=0 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fda80ff9640 1 -- 172.21.15.3:0/1703144669 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563aff50a720 con 0x563aff880520 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fda80ff9640 10 client.16296 handle_command_reply: tid=0 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 client.16296 shutdown 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 msgr2=0x563aff86cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 0x563aff86cf40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fda94006010 tx=0x7fda9402ca10 comp rx=0 tx=0).stop 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 shutdown_connections 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff880520 0x563aff882910 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff8673e0 0x563aff86d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff86d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1703144669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff866a30 0x563aff86cf40 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fda80ff9640 0 client.16296 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 >> 172.21.15.3:0/1703144669 conn(0x563aff861100 msgr2=0x563aff862e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 shutdown_connections 2024-02-09T22:53:15.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1703144669 wait complete. 2024-02-09T22:53:15.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.740+0000 7fdaa25f0740 20 client.16296 trim_cache size 0 max 16384 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 Processor -- start 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 -- start start 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff86f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff86d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff868250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff85c8b0 con 0x563aff8673e0 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff753130 con 0x563aff866a30 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff865cf0 con 0x563aff86ff60 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff86d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff86d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47756/0 (socket says 172.21.15.3:47756) 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1908513794 learned_addr learned my addr 172.21.15.3:0/1908513794 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff868250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff86f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1908513794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 msgr2=0x563aff868250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff868250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1908513794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 msgr2=0x563aff86f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff86f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1908513794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563aff768c90 con 0x563aff8673e0 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff86f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.752+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff86d7e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fda880cbca0 tx=0x7fda88097260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af9de32b0ee56cc2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fda82ffd640 1 -- 172.21.15.3:0/1908513794 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8809ac60 con 0x563aff8673e0 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fda82ffd640 1 -- 172.21.15.3:0/1908513794 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8809a9d0 con 0x563aff8673e0 2024-02-09T22:53:15.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fda82ffd640 1 -- 172.21.15.3:0/1908513794 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda880a36c0 con 0x563aff8673e0 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1908513794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 msgr2=0x563aff86d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff86d7e0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fda880cbca0 tx=0x7fda88097260 comp rx=0 tx=0).stop 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1908513794 shutdown_connections 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff868250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff86d7e0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1908513794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff86f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1908513794 >> 172.21.15.3:0/1908513794 conn(0x563aff861100 msgr2=0x563aff8614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1908513794 shutdown_connections 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.753+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1908513794 wait complete. 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 -- start start 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 Processor -- start 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff875940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff875e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff8763c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff8778e0 con 0x563aff8673e0 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff8d6a20 con 0x563aff866a30 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563aff8d6b90 con 0x563aff86ff60 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fdaa25f0740 10 client.0 _ll_register_callbacks cb 0x7fda900099f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff875e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff875e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47766/0 (socket says 172.21.15.3:47766) 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1541516664 learned_addr learned my addr 172.21.15.3:0/1541516664 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff875940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1541516664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 msgr2=0x563aff8763c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9dfbb640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff8763c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff8763c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 msgr2=0x563aff875940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.754+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff875940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda9cfb9640 1 -- 172.21.15.3:0/1541516664 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda981a4a10 con 0x563aff8673e0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff875940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda9cfb9640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff875e80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fda88097860 tx=0x7fda8809a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845d73f8155dc101 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda80ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda80ff9640 1 -- 172.21.15.3:0/1541516664 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8809ae90 con 0x563aff8673e0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda80ff9640 1 -- 172.21.15.3:0/1541516664 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda8809efe0 con 0x563aff8673e0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda90013d00 con 0x563aff8673e0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda80ff9640 1 -- 172.21.15.3:0/1541516664 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda8809b590 con 0x563aff8673e0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 10 client.16302 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x563aff78ea70 con 0x563aff8673e0 2024-02-09T22:53:15.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda80ff9640 1 -- 172.21.15.3:0/1541516664 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda90013d00 con 0x563aff8673e0 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fda80ff9640 1 -- 172.21.15.3:0/1541516664 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fda880b3bc0 con 0x563aff8673e0 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 10 client.16302 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 10 client.16302 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 20 client.16302 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff8d7f80 0x563aff8da3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 4 client.16302 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.755+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563aff559890 con 0x563aff8d7f80 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff8d7f80 0x563aff8da3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fda9d7ba640 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff8d7f80 0x563aff8da3a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cf8c4e284a90b697 server_cookie=2c98a4a3805ff2ce in_seq=0 out_seq=0 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fda80ff9640 10 client.16302 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fda80ff9640 1 -- 172.21.15.3:0/1541516664 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563aff559890 con 0x563aff8d7f80 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fda80ff9640 10 client.16302 handle_command_reply: tid=0 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 client.16302 shutdown 2024-02-09T22:53:15.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 msgr2=0x563aff875e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff875e80 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fda88097860 tx=0x7fda8809a8c0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 shutdown_connections 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563aff8d7f80 0x563aff8da3a0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563aff86ff60 0x563aff8763c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fda80ff9640 0 client.16302 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563aff8673e0 0x563aff875e80 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 --2- 172.21.15.3:0/1541516664 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563aff866a30 0x563aff875940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 >> 172.21.15.3:0/1541516664 conn(0x563aff861100 msgr2=0x563aff868250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 shutdown_connections 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 1 -- 172.21.15.3:0/1541516664 wait complete. 2024-02-09T22:53:15.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.756+0000 7fdaa25f0740 20 client.16302 trim_cache size 0 max 16384 2024-02-09T22:53:15.768 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda840a2230 msgr2=0x7fda840a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda840a2230 0x7fda840a46f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fda90021000 tx=0x7fda9001e000 comp rx=0 tx=0).stop 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 msgr2=0x7fda9819dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 0x7fda9819dbc0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fda8800e9f0 tx=0x7fda8800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 shutdown_connections 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda840a2230 0x7fda840a46f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda98197840 0x7fda98197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda98107930 0x7fda9819dbc0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 --2- 172.21.15.3:0/3703542095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda98102440 0x7fda9819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 >> 172.21.15.3:0/3703542095 conn(0x7fda98077310 msgr2=0x7fda980ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 shutdown_connections 2024-02-09T22:53:15.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.757+0000 7fda9fa1e640 1 -- 172.21.15.3:0/3703542095 wait complete. 2024-02-09T22:53:15.771 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:15.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:53:15.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 -- 172.21.15.3:0/656250067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc8100860 msgr2=0x7f1fc8100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 --2- 172.21.15.3:0/656250067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc8100860 0x7f1fc8100cc0 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8009900 tx=0x7f1fb802bc80 comp rx=0 tx=0).stop 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 -- 172.21.15.3:0/656250067 shutdown_connections 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 --2- 172.21.15.3:0/656250067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fc8101200 0x7f1fc810cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 --2- 172.21.15.3:0/656250067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc8100860 0x7f1fc8100cc0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 --2- 172.21.15.3:0/656250067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc81050c0 0x7f1fc81054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 -- 172.21.15.3:0/656250067 >> 172.21.15.3:0/656250067 conn(0x7f1fc80776a0 msgr2=0x7f1fc80fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 -- 172.21.15.3:0/656250067 shutdown_connections 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.872+0000 7f1fd0464640 1 -- 172.21.15.3:0/656250067 wait complete. 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 Processor -- start 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 -- start start 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fc8100860 0x7f1fc819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc8101200 0x7f1fc819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 0x7f1fc8197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8197ea0 con 0x7f1fc81050c0 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8198010 con 0x7f1fc8101200 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fd0464640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc81981d0 con 0x7f1fc8100860 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fcea01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 0x7f1fc8197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fcea01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 0x7f1fc8197710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47774/0 (socket says 172.21.15.3:47774) 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fcea01640 1 -- 172.21.15.3:0/2205057444 learned_addr learned my addr 172.21.15.3:0/2205057444 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fce200640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fc8100860 0x7f1fc819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.873+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc8101200 0x7f1fc819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fcea01640 1 -- 172.21.15.3:0/2205057444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fc8100860 msgr2=0x7f1fc819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fcea01640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fc8100860 0x7f1fc819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fcea01640 1 -- 172.21.15.3:0/2205057444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc8101200 msgr2=0x7f1fc819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fcea01640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc8101200 0x7f1fc819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fcea01640 1 -- 172.21.15.3:0/2205057444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fb8009560 con 0x7f1fc81050c0 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fcea01640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 0x7f1fc8197710 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000eaa0 tx=0x7f1fc000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fbf7fe640 1 -- 172.21.15.3:0/2205057444 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc000cdb0 con 0x7f1fc81050c0 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fbf7fe640 1 -- 172.21.15.3:0/2205057444 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fc0005020 con 0x7f1fc81050c0 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc81984b0 con 0x7f1fc81050c0 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fbf7fe640 1 -- 172.21.15.3:0/2205057444 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00116b0 con 0x7f1fc81050c0 2024-02-09T22:53:15.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.874+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc81a45d0 con 0x7f1fc81050c0 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.876+0000 7f1fbf7fe640 1 -- 172.21.15.3:0/2205057444 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1fc0005520 con 0x7f1fc81050c0 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 Processor -- start 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 -- start start 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d10e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d19990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fbf7fe640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1fb40a2230 0x7f1fb40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d1c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457ccc020 con 0x55f457d19f60 2024-02-09T22:53:15.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457c7f320 con 0x55f457d113e0 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fd303a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457c7c850 con 0x55f457d10a30 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fcea01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d1c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1fb40a2230 0x7f1fb40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fcea01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d1c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47782/0 (socket says 172.21.15.3:47782) 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fcea01640 1 -- 172.21.15.3:0/214804526 learned_addr learned my addr 172.21.15.3:0/214804526 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fce200640 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d19990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.877+0000 7f1fbf7fe640 1 -- 172.21.15.3:0/2205057444 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1fc00ce280 con 0x7f1fc81050c0 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d10e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcea01640 1 -- 172.21.15.3:0/214804526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 msgr2=0x55f457d10e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcea01640 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d10e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcea01640 1 -- 172.21.15.3:0/214804526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 msgr2=0x55f457d19990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcea01640 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d19990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcea01640 1 -- 172.21.15.3:0/214804526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f457c3a210 con 0x55f457d19f60 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fce200640 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d19990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcea01640 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d1c3b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1fc009ac80 tx=0x7f1fc000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=253b964a4121f8e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1fb40a2230 0x7f1fb40a46f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8006010 tx=0x7f1fb8005b50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fbcbfe640 1 -- 172.21.15.3:0/214804526 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc009acf0 con 0x55f457d19f60 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fbcbfe640 1 -- 172.21.15.3:0/214804526 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fc009b5d0 con 0x55f457d19f60 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.878+0000 7f1fbcbfe640 1 -- 172.21.15.3:0/214804526 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00a3470 con 0x55f457d19f60 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 -- 172.21.15.3:0/214804526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 msgr2=0x55f457d1c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d1c3b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f1fc009ac80 tx=0x7f1fc000cf90 comp rx=0 tx=0).stop 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 -- 172.21.15.3:0/214804526 shutdown_connections 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d19990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d1c3b0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 --2- 172.21.15.3:0/214804526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d10e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 -- 172.21.15.3:0/214804526 >> 172.21.15.3:0/214804526 conn(0x55f457d0b100 msgr2=0x55f457d0b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 -- 172.21.15.3:0/214804526 shutdown_connections 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 -- 172.21.15.3:0/214804526 wait complete. 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 -- start start 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.879+0000 7f1fd303a740 1 Processor -- start 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 0x55f457d16f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d174d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d19f60 0x55f457d17a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457c78030 con 0x55f457d10a30 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f45794b430 con 0x55f457d113e0 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457907e00 con 0x55f457d19f60 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 0x55f457d16f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 10 client.0 _ll_register_callbacks cb 0x55f457c6d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 0x55f457d16f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47790/0 (socket says 172.21.15.3:47790) 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 -- 172.21.15.3:0/1215412931 learned_addr learned my addr 172.21.15.3:0/1215412931 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d174d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fcea01640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d19f60 0x55f457d17a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 -- 172.21.15.3:0/1215412931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d19f60 msgr2=0x55f457d17a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d19f60 0x55f457d17a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 -- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 msgr2=0x55f457d174d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d174d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 -- 172.21.15.3:0/1215412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc81a45d0 con 0x55f457d10a30 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fce200640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 0x55f457d16f90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400bad0 tx=0x7f1fc400bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93175d0bf9606be4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/1215412931 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4016070 con 0x55f457d10a30 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/1215412931 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1fc4003e00 con 0x55f457d10a30 2024-02-09T22:53:15.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f4579777e0 con 0x55f457d10a30 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/1215412931 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fc4011440 con 0x55f457d10a30 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 10 client.16314 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.880+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f457c766b0 con 0x55f457d10a30 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/1215412931 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f4579777e0 con 0x55f457d10a30 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/1215412931 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f1fc401b7d0 con 0x55f457d10a30 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fd303a740 10 client.16314 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fd303a740 10 client.16314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fd303a740 20 client.16314 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d2a740 0x55f457d2cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fd303a740 4 client.16314 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f4579b4720 con 0x55f457d2a740 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d2a740 0x55f457d2cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d2a740 0x55f457d2cb60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f992ee09544b7d62 server_cookie=e9e8eacacd7d401a in_seq=0 out_seq=0 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.881+0000 7f1fb1ffb640 10 client.16314 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/1215412931 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f4579b4720 con 0x55f457d2a740 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fb1ffb640 10 client.16314 handle_command_reply: tid=0 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 client.16314 shutdown 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 msgr2=0x55f457d16f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 0x55f457d16f90 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400bad0 tx=0x7f1fc400bf90 comp rx=0 tx=0).stop 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 shutdown_connections 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d2a740 0x55f457d2cb60 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d174d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d19f60 0x55f457d17a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fb1ffb640 0 client.16314 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1215412931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d10a30 0x55f457d16f90 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 >> 172.21.15.3:0/1215412931 conn(0x55f457d0b100 msgr2=0x55f457d0ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 shutdown_connections 2024-02-09T22:53:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 1 -- 172.21.15.3:0/1215412931 wait complete. 2024-02-09T22:53:15.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.882+0000 7f1fd303a740 20 client.16314 trim_cache size 0 max 16384 2024-02-09T22:53:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 Processor -- start 2024-02-09T22:53:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 -- start start 2024-02-09T22:53:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d196c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 0x55f457d17830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d19f60 0x55f457d14210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457bfd130 con 0x55f457d113e0 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457d31db0 con 0x55f457d19f60 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fd303a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457d11e30 con 0x55f457d10a30 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fcd9ff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 0x55f457d17830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fcd9ff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 0x55f457d17830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47792/0 (socket says 172.21.15.3:47792) 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.894+0000 7f1fcd9ff640 1 -- 172.21.15.3:0/1285808016 learned_addr learned my addr 172.21.15.3:0/1285808016 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcd9ff640 1 -- 172.21.15.3:0/1285808016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 msgr2=0x55f457d196c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d196c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcd9ff640 1 -- 172.21.15.3:0/1285808016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d19f60 msgr2=0x55f457d14210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcea01640 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d19f60 0x55f457d14210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d19f60 0x55f457d14210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcd9ff640 1 -- 172.21.15.3:0/1285808016 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f457c12c90 con 0x55f457d113e0 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcea01640 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d19f60 0x55f457d14210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:15.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 0x55f457d17830 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1fb802c1c0 tx=0x7f1fb8005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488243409e12c31e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fb3fff640 1 -- 172.21.15.3:0/1285808016 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb804a040 con 0x55f457d113e0 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fb3fff640 1 -- 172.21.15.3:0/1285808016 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fb8004490 con 0x55f457d113e0 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fb3fff640 1 -- 172.21.15.3:0/1285808016 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8003b50 con 0x55f457d113e0 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 -- 172.21.15.3:0/1285808016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 msgr2=0x55f457d17830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 0x55f457d17830 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f1fb802c1c0 tx=0x7f1fb8005e40 comp rx=0 tx=0).stop 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 -- 172.21.15.3:0/1285808016 shutdown_connections 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d19f60 0x55f457d14210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d113e0 0x55f457d17830 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 --2- 172.21.15.3:0/1285808016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d196c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 -- 172.21.15.3:0/1285808016 >> 172.21.15.3:0/1285808016 conn(0x55f457d0b100 msgr2=0x55f457d122f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 -- 172.21.15.3:0/1285808016 shutdown_connections 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.895+0000 7f1fd303a740 1 -- 172.21.15.3:0/1285808016 wait complete. 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 -- start start 2024-02-09T22:53:15.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 Processor -- start 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d7b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d7b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d201b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457d207f0 con 0x55f457d19f60 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457d20960 con 0x55f457d113e0 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f457d20b20 con 0x55f457d10a30 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fd303a740 10 client.0 _ll_register_callbacks cb 0x7f1fc0096870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d201b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d201b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47794/0 (socket says 172.21.15.3:47794) 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 -- 172.21.15.3:0/3505041061 learned_addr learned my addr 172.21.15.3:0/3505041061 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fce200640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d7b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcd9ff640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d7b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 -- 172.21.15.3:0/3505041061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 msgr2=0x55f457d7b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d7b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 -- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 msgr2=0x55f457d7b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d7b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.896+0000 7f1fcea01640 1 -- 172.21.15.3:0/3505041061 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc810d8e0 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fcea01640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d201b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0098fa0 tx=0x7f1fc009aac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23abe079438d75dd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/3505041061 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00053a0 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/3505041061 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1fc009acf0 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1fc009b5d0 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/3505041061 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fc00cbe20 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 10 client.16320 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55f457c38a70 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/3505041061 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fc007d020 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/3505041061 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f1fc004e050 con 0x55f457d19f60 2024-02-09T22:53:15.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 10 client.16320 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 10 client.16320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 20 client.16320 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d82b40 0x55f457d84f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 4 client.16320 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f457a03890 con 0x55f457d82b40 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.897+0000 7f1fce200640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d82b40 0x55f457d84f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fb1ffb640 10 client.16320 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fce200640 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d82b40 0x55f457d84f60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9011e3f4ccd4a325 server_cookie=8fa12145f89ccf1b in_seq=0 out_seq=0 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fb1ffb640 1 -- 172.21.15.3:0/3505041061 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f457a03890 con 0x55f457d82b40 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fb1ffb640 10 client.16320 handle_command_reply: tid=0 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 client.16320 shutdown 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 msgr2=0x55f457d201b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d201b0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0098fa0 tx=0x7f1fc009aac0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 shutdown_connections 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f457d82b40 0x55f457d84f60 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f457d113e0 0x55f457d7b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f457d19f60 0x55f457d201b0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 --2- 172.21.15.3:0/3505041061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f457d10a30 0x55f457d7b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fb1ffb640 0 client.16320 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 >> 172.21.15.3:0/3505041061 conn(0x55f457d0b100 msgr2=0x55f457d099b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 shutdown_connections 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 1 -- 172.21.15.3:0/3505041061 wait complete. 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.898+0000 7f1fd303a740 20 client.16320 trim_cache size 0 max 16384 2024-02-09T22:53:15.908 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1fb40a2230 msgr2=0x7f1fb40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1fb40a2230 0x7f1fb40a46f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8006010 tx=0x7f1fb8005b50 comp rx=0 tx=0).stop 2024-02-09T22:53:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 msgr2=0x7f1fc8197710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:15.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 0x7f1fc8197710 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f1fc000eaa0 tx=0x7f1fc000ef60 comp rx=0 tx=0).stop 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 shutdown_connections 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1fb40a2230 0x7f1fb40a46f0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1fc8101200 0x7f1fc819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1fc81050c0 0x7f1fc8197710 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 --2- 172.21.15.3:0/2205057444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1fc8100860 0x7f1fc819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 >> 172.21.15.3:0/2205057444 conn(0x7f1fc80776a0 msgr2=0x7f1fc80ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 shutdown_connections 2024-02-09T22:53:15.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:15.899+0000 7f1fd0464640 1 -- 172.21.15.3:0/2205057444 wait complete. 2024-02-09T22:53:16.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:16 smithi064 ceph-mon[36638]: pgmap v63: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 389 KiB/s wr, 100 op/s 2024-02-09T22:53:16.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:16 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1331891818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:53:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:16 smithi089 ceph-mon[36665]: pgmap v63: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 389 KiB/s wr, 100 op/s 2024-02-09T22:53:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:16 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1331891818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:53:16.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 50355 4.13 MB/sec warmup 70 sec latency 1882.331 ms 2024-02-09T22:53:16.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:16 smithi003 ceph-mon[32678]: pgmap v63: 97 pgs: 97 active+clean; 28 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 389 KiB/s wr, 100 op/s 2024-02-09T22:53:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:16 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1331891818' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:53:16.913 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:53:16.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:53:16.914 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:53:17.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/894991798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 msgr2=0x7f9ee010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/894991798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee010e0d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009cd0 tx=0x7f9ed802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/894991798 shutdown_connections 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/894991798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee010e0d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/894991798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee006a550 0x7f9ee0105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/894991798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 0x7f9ee0069f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/894991798 >> 172.21.15.3:0/894991798 conn(0x7f9ee0076ef0 msgr2=0x7f9ee0077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/894991798 shutdown_connections 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.031+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/894991798 wait complete. 2024-02-09T22:53:17.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 Processor -- start 2024-02-09T22:53:17.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 -- start start 2024-02-09T22:53:17.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 0x7f9ee0100450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee0100990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee0100ed0 0x7f9ee01a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01a5f90 con 0x7f9ee01064e0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01a6100 con 0x7f9ee0069ba0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9ee5b9a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee01a62c0 con 0x7f9ee0100ed0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee0100990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee0100990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56864/0 (socket says 172.21.15.3:56864) 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.032+0000 7f9edeffd640 1 -- 172.21.15.3:0/239821288 learned_addr learned my addr 172.21.15.3:0/239821288 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edffff640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee0100ed0 0x7f9ee01a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 0x7f9ee0100450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edeffd640 1 -- 172.21.15.3:0/239821288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee0100ed0 msgr2=0x7f9ee01a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edeffd640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee0100ed0 0x7f9ee01a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edeffd640 1 -- 172.21.15.3:0/239821288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 msgr2=0x7f9ee0100450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edeffd640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 0x7f9ee0100450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edeffd640 1 -- 172.21.15.3:0/239821288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed8009900 con 0x7f9ee01064e0 2024-02-09T22:53:17.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 0x7f9ee0100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:17.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edeffd640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee0100990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e9f0 tx=0x7f9ed400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edcff9640 1 -- 172.21.15.3:0/239821288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40096c0 con 0x7f9ee01064e0 2024-02-09T22:53:17.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee01a65a0 con 0x7f9ee01064e0 2024-02-09T22:53:17.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edcff9640 1 -- 172.21.15.3:0/239821288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed4005500 con 0x7f9ee01064e0 2024-02-09T22:53:17.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9edcff9640 1 -- 172.21.15.3:0/239821288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4010430 con 0x7f9ee01064e0 2024-02-09T22:53:17.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.033+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee01a6a60 con 0x7f9ee01064e0 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.035+0000 7f9edcff9640 1 -- 172.21.15.3:0/239821288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9ed4005020 con 0x7f9ee01064e0 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 Processor -- start 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edcff9640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9ec80a2230 0x7f9ec80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 -- start start 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edffff640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9ec80a2230 0x7f9ec80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415468570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edcff9640 1 -- 172.21.15.3:0/239821288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ed4014070 con 0x7f9ee01064e0 2024-02-09T22:53:17.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x559415471150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415471690 0x559415473a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594153d6000 con 0x559415468ab0 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594153d3530 con 0x559415468190 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9ee8775740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594153ced10 con 0x559415471690 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x559415471150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x559415471150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56876/0 (socket says 172.21.15.3:56876) 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 -- 172.21.15.3:0/2381195671 learned_addr learned my addr 172.21.15.3:0/2381195671 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415468570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edffff640 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415471690 0x559415473a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 -- 172.21.15.3:0/2381195671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415471690 msgr2=0x559415473a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415471690 0x559415473a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 -- 172.21.15.3:0/2381195671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 msgr2=0x559415468570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415468570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.036+0000 7f9edeffd640 1 -- 172.21.15.3:0/2381195671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559415375210 con 0x559415468ab0 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9edffff640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9ec80a2230 0x7f9ec80a46f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8005cc0 tx=0x7f9ed8032ef0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9edeffd640 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x559415471150 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40977f0 tx=0x7f9ed4005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97975d6ec6933620 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ec5ffb640 1 -- 172.21.15.3:0/2381195671 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed409ef70 con 0x559415468ab0 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ec5ffb640 1 -- 172.21.15.3:0/2381195671 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed40a35c0 con 0x559415468ab0 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ec5ffb640 1 -- 172.21.15.3:0/2381195671 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4097490 con 0x559415468ab0 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ee8775740 1 -- 172.21.15.3:0/2381195671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 msgr2=0x559415471150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ee8775740 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x559415471150 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f9ed40977f0 tx=0x7f9ed4005730 comp rx=0 tx=0).stop 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ee8775740 1 -- 172.21.15.3:0/2381195671 shutdown_connections 2024-02-09T22:53:17.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ee8775740 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415471690 0x559415473a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ee8775740 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x559415471150 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.037+0000 7f9ee8775740 1 --2- 172.21.15.3:0/2381195671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415468570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- 172.21.15.3:0/2381195671 >> 172.21.15.3:0/2381195671 conn(0x5594154628c0 msgr2=0x559415462ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- 172.21.15.3:0/2381195671 shutdown_connections 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- 172.21.15.3:0/2381195671 wait complete. 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- start start 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 Processor -- start 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x55941546ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x55941546f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559415373e60 con 0x559415468ab0 2024-02-09T22:53:17.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594150a24b0 con 0x559415471690 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.038+0000 7f9ee8775740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559415354670 con 0x559415468190 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ee8775740 10 client.0 _ll_register_callbacks cb 0x55941536c190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x55941546ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x55941546ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56878/0 (socket says 172.21.15.3:56878) 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 -- 172.21.15.3:0/3568562495 learned_addr learned my addr 172.21.15.3:0/3568562495 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edffff640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x55941546f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 -- 172.21.15.3:0/3568562495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 msgr2=0x55941546e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 -- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 msgr2=0x55941546f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x55941546f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 -- 172.21.15.3:0/3568562495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee01a6a60 con 0x559415468ab0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edffff640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x55941546f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9edeffd640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x55941546ec50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c6a0 tx=0x7f9ed409f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb4bd171f021dc7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ebeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ebeffd640 1 -- 172.21.15.3:0/3568562495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4098ec0 con 0x559415468ab0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ebeffd640 1 -- 172.21.15.3:0/3568562495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ed4099120 con 0x559415468ab0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594150ce860 con 0x559415468ab0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ebeffd640 1 -- 172.21.15.3:0/3568562495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ed40662c0 con 0x559415468ab0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ee8775740 10 client.16332 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.039+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5594153cd390 con 0x559415468ab0 2024-02-09T22:53:17.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ebeffd640 1 -- 172.21.15.3:0/3568562495 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5594150ce860 con 0x559415468ab0 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ebeffd640 1 -- 172.21.15.3:0/3568562495 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9ed401f060 con 0x559415468ab0 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ee8775740 10 client.16332 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ee8775740 10 client.16332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ee8775740 20 client.16332 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ee8775740 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559415481420 0x559415483840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ee8775740 4 client.16332 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55941511f410 con 0x559415481420 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559415481420 0x559415483840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559415481420 0x559415483840 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecc572367a88ac42 server_cookie=50355518f9f7bd19 in_seq=0 out_seq=0 2024-02-09T22:53:17.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.040+0000 7f9ebeffd640 10 client.16332 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ebeffd640 1 -- 172.21.15.3:0/3568562495 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55941511f410 con 0x559415481420 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ebeffd640 10 client.16332 handle_command_reply: tid=0 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 client.16332 shutdown 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 msgr2=0x55941546ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x55941546ec50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c6a0 tx=0x7f9ed409f040 comp rx=0 tx=0).stop 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 shutdown_connections 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559415481420 0x559415483840 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x55941546f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x55941546ec50 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ebeffd640 0 client.16332 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 --2- 172.21.15.3:0/3568562495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 >> 172.21.15.3:0/3568562495 conn(0x5594154628c0 msgr2=0x559415464610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 shutdown_connections 2024-02-09T22:53:17.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 1 -- 172.21.15.3:0/3568562495 wait complete. 2024-02-09T22:53:17.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.041+0000 7f9ee8775740 20 client.16332 trim_cache size 0 max 16384 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.056+0000 7f9ee8775740 1 Processor -- start 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.056+0000 7f9ee8775740 1 -- start start 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x5594154617a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x559415478540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ee8775740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559415467df0 con 0x559415468ab0 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ee8775740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559415467560 con 0x559415471690 2024-02-09T22:53:17.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ee8775740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594154638c0 con 0x559415468190 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x5594154617a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x5594154617a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56884/0 (socket says 172.21.15.3:56884) 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 -- 172.21.15.3:0/636462634 learned_addr learned my addr 172.21.15.3:0/636462634 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 -- 172.21.15.3:0/636462634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 msgr2=0x55941546efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 -- 172.21.15.3:0/636462634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 msgr2=0x559415478540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x559415478540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 -- 172.21.15.3:0/636462634 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559415369c70 con 0x559415468ab0 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9edeffd640 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x5594154617a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e9c0 tx=0x7f9ed409aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=665c0b47306248c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ec57fa640 1 -- 172.21.15.3:0/636462634 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed409a780 con 0x559415468ab0 2024-02-09T22:53:17.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ec57fa640 1 -- 172.21.15.3:0/636462634 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed4098ec0 con 0x559415468ab0 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.057+0000 7f9ec57fa640 1 -- 172.21.15.3:0/636462634 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed40663b0 con 0x559415468ab0 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 -- 172.21.15.3:0/636462634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 msgr2=0x5594154617a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x5594154617a0 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e9c0 tx=0x7f9ed409aa90 comp rx=0 tx=0).stop 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 -- 172.21.15.3:0/636462634 shutdown_connections 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415471690 0x559415478540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415468ab0 0x5594154617a0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 --2- 172.21.15.3:0/636462634 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468190 0x55941546efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 -- 172.21.15.3:0/636462634 >> 172.21.15.3:0/636462634 conn(0x5594154628c0 msgr2=0x559415461090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 -- 172.21.15.3:0/636462634 shutdown_connections 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 -- 172.21.15.3:0/636462634 wait complete. 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 -- start start 2024-02-09T22:53:17.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.058+0000 7f9ee8775740 1 Processor -- start 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415477830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468ab0 0x559415477d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 0x5594154d1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594154d1e70 con 0x559415471690 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594154d1fe0 con 0x559415468190 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594154d21a0 con 0x559415468ab0 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 10 client.0 _ll_register_callbacks cb 0x7f9ed40970c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 0x5594154d1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 0x5594154d1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56898/0 (socket says 172.21.15.3:56898) 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 -- 172.21.15.3:0/1778592377 learned_addr learned my addr 172.21.15.3:0/1778592377 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415477830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edeffd640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468ab0 0x559415477d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 -- 172.21.15.3:0/1778592377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468ab0 msgr2=0x559415477d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468ab0 0x559415477d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 -- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 msgr2=0x559415477830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415477830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 -- 172.21.15.3:0/1778592377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee01a6a60 con 0x559415471690 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415477830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9edffff640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 0x5594154d1930 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009cd0 tx=0x7f9ed80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39f5b519fcb36644 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ebeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:17.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ebeffd640 1 -- 172.21.15.3:0/1778592377 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed8032ac0 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ebeffd640 1 -- 172.21.15.3:0/1778592377 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9ed80324c0 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.059+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ed40a35c0 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ebeffd640 1 -- 172.21.15.3:0/1778592377 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ed8034e80 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 10 client.16338 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55941538fac0 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ebeffd640 1 -- 172.21.15.3:0/1778592377 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ed40a35c0 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ebeffd640 1 -- 172.21.15.3:0/1778592377 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9ed8041440 con 0x559415471690 2024-02-09T22:53:17.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 10 client.16338 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 10 client.16338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 20 client.16338 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5594154d9bd0 0x5594154dbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 4 client.16338 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55941510b7b0 con 0x5594154d9bd0 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.060+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5594154d9bd0 0x5594154dbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.061+0000 7f9edf7fe640 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5594154d9bd0 0x5594154dbff0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac8dc22beb5962e8 server_cookie=7bd201514162dd50 in_seq=0 out_seq=0 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.061+0000 7f9ebeffd640 10 client.16338 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ebeffd640 1 -- 172.21.15.3:0/1778592377 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55941510b7b0 con 0x5594154d9bd0 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ebeffd640 10 client.16338 handle_command_reply: tid=0 2024-02-09T22:53:17.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 client.16338 shutdown 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 msgr2=0x5594154d1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 0x5594154d1930 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8009cd0 tx=0x7f9ed80095c0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 shutdown_connections 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5594154d9bd0 0x5594154dbff0 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559415471690 0x5594154d1930 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559415468ab0 0x559415477d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 --2- 172.21.15.3:0/1778592377 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559415468190 0x559415477830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ebeffd640 0 client.16338 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 >> 172.21.15.3:0/1778592377 conn(0x5594154628c0 msgr2=0x559415461920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 shutdown_connections 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 1 -- 172.21.15.3:0/1778592377 wait complete. 2024-02-09T22:53:17.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.062+0000 7f9ee8775740 20 client.16338 trim_cache size 0 max 16384 2024-02-09T22:53:17.074 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:53:17.074 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:53:17.074 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "033b3854-deba-482e-880f-59ed884fef31", 2024-02-09T22:53:17.074 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:53:17.074 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9ec80a2230 msgr2=0x7f9ec80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9ec80a2230 0x7f9ec80a46f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9ed8005cc0 tx=0x7f9ed8032ef0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 msgr2=0x7f9ee0100990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee0100990 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400e9f0 tx=0x7f9ed400eeb0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 shutdown_connections 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9ec80a2230 0x7f9ec80a46f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9ee0100ed0 0x7f9ee01a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9ee01064e0 0x7f9ee0100990 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 --2- 172.21.15.3:0/239821288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9ee0069ba0 0x7f9ee0100450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 >> 172.21.15.3:0/239821288 conn(0x7f9ee0076ef0 msgr2=0x7f9ee00fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 shutdown_connections 2024-02-09T22:53:17.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.063+0000 7f9ee5b9a640 1 -- 172.21.15.3:0/239821288 wait complete. 2024-02-09T22:53:17.078 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:53:17.078 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:53:17.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 -- 172.21.15.3:0/505597943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c1020a0 msgr2=0x7f8b2c102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 --2- 172.21.15.3:0/505597943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c1020a0 0x7f8b2c102480 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c009900 tx=0x7f8b1c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:17.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 -- 172.21.15.3:0/505597943 shutdown_connections 2024-02-09T22:53:17.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 --2- 172.21.15.3:0/505597943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b2c101210 0x7f8b2c10d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 --2- 172.21.15.3:0/505597943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b2c100870 0x7f8b2c100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 --2- 172.21.15.3:0/505597943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c1020a0 0x7f8b2c102480 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.185+0000 7f8b3157a640 1 -- 172.21.15.3:0/505597943 >> 172.21.15.3:0/505597943 conn(0x7f8b2c076620 msgr2=0x7f8b2c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 -- 172.21.15.3:0/505597943 shutdown_connections 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 -- 172.21.15.3:0/505597943 wait complete. 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 Processor -- start 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 -- start start 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 0x7f8b2c0fba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b2c101210 0x7f8b2c0fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b2c1020a0 0x7f8b2c0fc4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c1a5db0 con 0x7f8b2c100870 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c1a5f20 con 0x7f8b2c1020a0 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b3157a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2c1a60e0 con 0x7f8b2c101210 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b2affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 0x7f8b2c0fba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b2affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 0x7f8b2c0fba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56924/0 (socket says 172.21.15.3:56924) 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b2affd640 1 -- 172.21.15.3:0/1437845989 learned_addr learned my addr 172.21.15.3:0/1437845989 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b2c1020a0 0x7f8b2c0fc4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.186+0000 7f8b23fff640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b2c101210 0x7f8b2c0fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b2affd640 1 -- 172.21.15.3:0/1437845989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b2c101210 msgr2=0x7f8b2c0fbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b2affd640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b2c101210 0x7f8b2c0fbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b2affd640 1 -- 172.21.15.3:0/1437845989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b2c1020a0 msgr2=0x7f8b2c0fc4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b2affd640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b2c1020a0 0x7f8b2c0fc4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b2affd640 1 -- 172.21.15.3:0/1437845989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1c009560 con 0x7f8b2c100870 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b2affd640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 0x7f8b2c0fba70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c005940 tx=0x7f8b1c02c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b28ff9640 1 -- 172.21.15.3:0/1437845989 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c0042d0 con 0x7f8b2c100870 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b28ff9640 1 -- 172.21.15.3:0/1437845989 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b1c004430 con 0x7f8b2c100870 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2c1a6360 con 0x7f8b2c100870 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b28ff9640 1 -- 172.21.15.3:0/1437845989 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c003c50 con 0x7f8b2c100870 2024-02-09T22:53:17.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.187+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2c1a6870 con 0x7f8b2c100870 2024-02-09T22:53:17.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.189+0000 7f8b28ff9640 1 -- 172.21.15.3:0/1437845989 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8b1c0054f0 con 0x7f8b2c100870 2024-02-09T22:53:17.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 Processor -- start 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b28ff9640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8b180a2230 0x7f8b180a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 -- start start 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8b180a2230 0x7f8b180a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b28ff9640 1 -- 172.21.15.3:0/1437845989 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8b1c0f0070 con 0x7f8b2c100870 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b77ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7838e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b786300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b735d00 con 0x55ea3b783eb0 2024-02-09T22:53:17.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b6ea2f0 con 0x55ea3b77a980 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b3422f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b6e7820 con 0x55ea3b77b330 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b2b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b786300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7838e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b2b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b786300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56930/0 (socket says 172.21.15.3:56930) 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/2191583036 learned_addr learned my addr 172.21.15.3:0/2191583036 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.190+0000 7f8b2affd640 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b77ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8b180a2230 0x7f8b180a46f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b24005fd0 tx=0x7f8b24011810 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b23fff640 1 -- 172.21.15.3:0/2191583036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 msgr2=0x55ea3b77ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b23fff640 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b77ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b23fff640 1 -- 172.21.15.3:0/2191583036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 msgr2=0x55ea3b786300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b23fff640 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b786300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b23fff640 1 -- 172.21.15.3:0/2191583036 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea3b6a4ff0 con 0x55ea3b77b330 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b786300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:17.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b23fff640 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7838e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009bd0 tx=0x7f8b14011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1728e1ec9da69b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b217fa640 1 -- 172.21.15.3:0/2191583036 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14004230 con 0x55ea3b77b330 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b217fa640 1 -- 172.21.15.3:0/2191583036 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b14009e40 con 0x55ea3b77b330 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.191+0000 7f8b217fa640 1 -- 172.21.15.3:0/2191583036 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14004d70 con 0x55ea3b77b330 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- 172.21.15.3:0/2191583036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 msgr2=0x55ea3b7838e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7838e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009bd0 tx=0x7f8b14011810 comp rx=0 tx=0).stop 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- 172.21.15.3:0/2191583036 shutdown_connections 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b786300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7838e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2191583036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b77ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- 172.21.15.3:0/2191583036 >> 172.21.15.3:0/2191583036 conn(0x55ea3b775050 msgr2=0x55ea3b775430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- 172.21.15.3:0/2191583036 shutdown_connections 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- 172.21.15.3:0/2191583036 wait complete. 2024-02-09T22:53:17.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- start start 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 Processor -- start 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b780db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7812f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b781830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b6e3000 con 0x55ea3b783eb0 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b3b6430 con 0x55ea3b77a980 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.192+0000 7f8b3422f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b372e00 con 0x55ea3b77b330 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b3422f740 10 client.0 _ll_register_callbacks cb 0x55ea3b6d77f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b781830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b781830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56942/0 (socket says 172.21.15.3:56942) 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/1074422945 learned_addr learned my addr 172.21.15.3:0/1074422945 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b23fff640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7812f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2affd640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b780db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/1074422945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 msgr2=0x55ea3b7812f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7812f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 msgr2=0x55ea3b780db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b780db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/1074422945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b14009830 con 0x55ea3b783eb0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b23fff640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7812f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b781830 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8b24004240 tx=0x7f8b24016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b80923bb5ff010aa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b127fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b127fc640 1 -- 172.21.15.3:0/1074422945 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2401e9c0 con 0x55ea3b783eb0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b127fc640 1 -- 172.21.15.3:0/1074422945 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8b2401eb20 con 0x55ea3b783eb0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ea3b3e27e0 con 0x55ea3b783eb0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b127fc640 1 -- 172.21.15.3:0/1074422945 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b2401ece0 con 0x55ea3b783eb0 2024-02-09T22:53:17.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b3422f740 10 client.16350 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ea3b6e1680 con 0x55ea3b783eb0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.193+0000 7f8b127fc640 1 -- 172.21.15.3:0/1074422945 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ea3b3e27e0 con 0x55ea3b783eb0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b127fc640 1 -- 172.21.15.3:0/1074422945 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f8b240275d0 con 0x55ea3b783eb0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b3422f740 10 client.16350 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b3422f740 10 client.16350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b3422f740 20 client.16350 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b3422f740 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7943e0 0x55ea3b796800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b3422f740 4 client.16350 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ea3b41f720 con 0x55ea3b7943e0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b2affd640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7943e0 0x55ea3b796800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b2affd640 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7943e0 0x55ea3b796800 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a675386699c17910 server_cookie=5e82bea2b959eba9 in_seq=0 out_seq=0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.194+0000 7f8b127fc640 10 client.16350 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b127fc640 1 -- 172.21.15.3:0/1074422945 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ea3b41f720 con 0x55ea3b7943e0 2024-02-09T22:53:17.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b127fc640 10 client.16350 handle_command_reply: tid=0 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 client.16350 shutdown 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 msgr2=0x55ea3b781830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b781830 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f8b24004240 tx=0x7f8b24016000 comp rx=0 tx=0).stop 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 shutdown_connections 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7943e0 0x55ea3b796800 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b781830 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77b330 0x55ea3b7812f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b127fc640 0 client.16350 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 --2- 172.21.15.3:0/1074422945 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77a980 0x55ea3b780db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 >> 172.21.15.3:0/1074422945 conn(0x55ea3b775050 msgr2=0x55ea3b776da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 shutdown_connections 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 1 -- 172.21.15.3:0/1074422945 wait complete. 2024-02-09T22:53:17.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.195+0000 7f8b3422f740 20 client.16350 trim_cache size 0 max 16384 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.210+0000 7f8b3422f740 1 Processor -- start 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.210+0000 7f8b3422f740 1 -- start start 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b7834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 0x55ea3b774070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b783eb0 0x55ea3b78aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b3422f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b667fd0 con 0x55ea3b77b330 2024-02-09T22:53:17.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b3422f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b3e27e0 con 0x55ea3b783eb0 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b3422f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b781d80 con 0x55ea3b77a980 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 0x55ea3b774070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 0x55ea3b774070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56958/0 (socket says 172.21.15.3:56958) 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 -- 172.21.15.3:0/2478974727 learned_addr learned my addr 172.21.15.3:0/2478974727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b783eb0 0x55ea3b78aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b2affd640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b7834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 -- 172.21.15.3:0/2478974727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 msgr2=0x55ea3b7834e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b7834e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 -- 172.21.15.3:0/2478974727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b783eb0 msgr2=0x55ea3b78aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b783eb0 0x55ea3b78aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 -- 172.21.15.3:0/2478974727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ea3b67db30 con 0x55ea3b77b330 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b2affd640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b7834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:17.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b783eb0 0x55ea3b78aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b23fff640 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 0x55ea3b774070 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009bd0 tx=0x7f8b14009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4bed5e946a2c687 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.211+0000 7f8b20ff9640 1 -- 172.21.15.3:0/2478974727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14005470 con 0x55ea3b77b330 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b20ff9640 1 -- 172.21.15.3:0/2478974727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b140055d0 con 0x55ea3b77b330 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b20ff9640 1 -- 172.21.15.3:0/2478974727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b14022a00 con 0x55ea3b77b330 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 -- 172.21.15.3:0/2478974727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 msgr2=0x55ea3b774070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 0x55ea3b774070 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009bd0 tx=0x7f8b14009b60 comp rx=0 tx=0).stop 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 -- 172.21.15.3:0/2478974727 shutdown_connections 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b783eb0 0x55ea3b78aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b77b330 0x55ea3b774070 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 --2- 172.21.15.3:0/2478974727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b7834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 -- 172.21.15.3:0/2478974727 >> 172.21.15.3:0/2478974727 conn(0x55ea3b775050 msgr2=0x55ea3b773960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 -- 172.21.15.3:0/2478974727 shutdown_connections 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.212+0000 7f8b3422f740 1 -- 172.21.15.3:0/2478974727 wait complete. 2024-02-09T22:53:17.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 -- start start 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 Processor -- start 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b78a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77b330 0x55ea3b78a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b7e4930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b7e4e70 con 0x55ea3b783eb0 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b7e4fb0 con 0x55ea3b77b330 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ea3b7e5170 con 0x55ea3b77a980 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b3422f740 10 client.0 _ll_register_callbacks cb 0x7f8b14009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b7e4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b7e4930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56974/0 (socket says 172.21.15.3:56974) 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/4287496186 learned_addr learned my addr 172.21.15.3:0/4287496186 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b23fff640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77b330 0x55ea3b78a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2affd640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b78a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/4287496186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 msgr2=0x55ea3b78a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b78a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77b330 msgr2=0x55ea3b78a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77b330 0x55ea3b78a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 -- 172.21.15.3:0/4287496186 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ea3b6a4ff0 con 0x55ea3b783eb0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2affd640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b78a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b2b7fe640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b7e4930 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8b24011d10 tx=0x7f8b240048d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=405ddb68a2692e54 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.213+0000 7f8b127fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b127fc640 1 -- 172.21.15.3:0/4287496186 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b24027d50 con 0x55ea3b783eb0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b127fc640 1 -- 172.21.15.3:0/4287496186 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8b24027410 con 0x55ea3b783eb0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b14009e40 con 0x55ea3b783eb0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b127fc640 1 -- 172.21.15.3:0/4287496186 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b240275d0 con 0x55ea3b783eb0 2024-02-09T22:53:17.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 10 client.16356 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55ea3b6a3850 con 0x55ea3b783eb0 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b127fc640 1 -- 172.21.15.3:0/4287496186 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b14009e40 con 0x55ea3b783eb0 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b127fc640 1 -- 172.21.15.3:0/4287496186 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f8b24005020 con 0x55ea3b783eb0 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 10 client.16356 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 10 client.16356 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 20 client.16356 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7eca70 0x55ea3b7eee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 4 client.16356 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ea3b46e890 con 0x55ea3b7eca70 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.214+0000 7f8b2affd640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7eca70 0x55ea3b7eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b2affd640 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7eca70 0x55ea3b7eee90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd2ca6eec8580172 server_cookie=64472362e3ad5113 in_seq=0 out_seq=0 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b127fc640 10 client.16356 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b127fc640 1 -- 172.21.15.3:0/4287496186 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ea3b46e890 con 0x55ea3b7eca70 2024-02-09T22:53:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b127fc640 10 client.16356 handle_command_reply: tid=0 2024-02-09T22:53:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 client.16356 shutdown 2024-02-09T22:53:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 msgr2=0x55ea3b7e4930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b7e4930 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8b24011d10 tx=0x7f8b240048d0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 shutdown_connections 2024-02-09T22:53:17.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ea3b7eca70 0x55ea3b7eee90 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ea3b77b330 0x55ea3b78a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b127fc640 0 client.16356 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ea3b783eb0 0x55ea3b7e4930 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 --2- 172.21.15.3:0/4287496186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ea3b77a980 0x55ea3b78a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 >> 172.21.15.3:0/4287496186 conn(0x55ea3b775050 msgr2=0x55ea3b77f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 shutdown_connections 2024-02-09T22:53:17.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 1 -- 172.21.15.3:0/4287496186 wait complete. 2024-02-09T22:53:17.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.215+0000 7f8b3422f740 20 client.16356 trim_cache size 0 max 16384 2024-02-09T22:53:17.232 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:53:17.233 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:53:17.233 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:53:17.233 INFO:teuthology.orchestra.run.smithi003.stdout: "033b3854-deba-482e-880f-59ed884fef31": { 2024-02-09T22:53:17.233 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:53:17.233 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "033b3854-deba-482e-880f-59ed884fef31", 2024-02-09T22:53:17.234 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:53:17.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:53:17.234 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:53:17.234 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:53:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8b180a2230 msgr2=0x7f8b180a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8b180a2230 0x7f8b180a46f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8b24005fd0 tx=0x7f8b24011810 comp rx=0 tx=0).stop 2024-02-09T22:53:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 msgr2=0x7f8b2c0fba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:17.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 0x7f8b2c0fba70 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c005940 tx=0x7f8b1c02c670 comp rx=0 tx=0).stop 2024-02-09T22:53:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 shutdown_connections 2024-02-09T22:53:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8b180a2230 0x7f8b180a46f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8b2c1020a0 0x7f8b2c0fc4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8b2c101210 0x7f8b2c0fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 --2- 172.21.15.3:0/1437845989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8b2c100870 0x7f8b2c0fba70 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:17.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 >> 172.21.15.3:0/1437845989 conn(0x7f8b2c076620 msgr2=0x7f8b2c0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:17.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 shutdown_connections 2024-02-09T22:53:17.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:17.216+0000 7f8b3157a640 1 -- 172.21.15.3:0/1437845989 wait complete. 2024-02-09T22:53:17.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:17 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:17.238 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:17 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:17 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:17.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:17 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:17 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:17.239 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:17 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.240 INFO:tasks.cephfs.filesystem:scrub status for tag:033b3854-deba-482e-880f-59ed884fef31 - {'path': '/', 'tag': '033b3854-deba-482e-880f-59ed884fef31', 'options': 'recursive,force'} 2024-02-09T22:53:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:17 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:17 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:17 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:17 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:17 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:17.437 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:17 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:17 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:17.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:17 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:17 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:17.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:17 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:17 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:17.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:17 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:17.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 50479 4.07 MB/sec warmup 71 sec latency 2359.486 ms 2024-02-09T22:53:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: pgmap v64: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 593 KiB/s wr, 153 op/s 2024-02-09T22:53:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:53:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:18 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: pgmap v64: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 593 KiB/s wr, 153 op/s 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:18 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 52139 4.11 MB/sec warmup 72 sec latency 854.502 ms 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: pgmap v64: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 979 KiB/s rd, 593 KiB/s wr, 153 op/s 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:18.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:18 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:19.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53839 4.22 MB/sec warmup 73 sec latency 17.692 ms 2024-02-09T22:53:20.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:20 smithi064 ceph-mon[36638]: pgmap v65: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 256 KiB/s wr, 66 op/s 2024-02-09T22:53:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:20 smithi089 ceph-mon[36665]: pgmap v65: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 256 KiB/s wr, 66 op/s 2024-02-09T22:53:20.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53903 4.16 MB/sec warmup 74 sec latency 975.344 ms 2024-02-09T22:53:20.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:20 smithi003 ceph-mon[32678]: pgmap v65: 97 pgs: 97 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 728 KiB/s rd, 256 KiB/s wr, 66 op/s 2024-02-09T22:53:21.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:21 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:53:21.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:21 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:21 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:21 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:21 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:21 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:53:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:21 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:21 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:21 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:21 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.624 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53903 4.11 MB/sec warmup 75 sec latency 1975.433 ms 2024-02-09T22:53:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:21 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:53:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:21 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:21 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:21 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:21 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:22.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:22 smithi064 ceph-mon[36638]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 780 KiB/s wr, 129 op/s 2024-02-09T22:53:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:22 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:53:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:22 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:53:22.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:22 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:22 smithi089 ceph-mon[36665]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 780 KiB/s wr, 129 op/s 2024-02-09T22:53:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:22 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:53:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:22 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:53:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:22 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:22.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53903 4.05 MB/sec warmup 76 sec latency 2975.527 ms 2024-02-09T22:53:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:22 smithi003 ceph-mon[32678]: pgmap v66: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 780 KiB/s wr, 129 op/s 2024-02-09T22:53:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:22 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:53:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:22 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:53:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:22 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:53:23.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53959 4.00 MB/sec warmup 77 sec latency 3393.200 ms 2024-02-09T22:53:24.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:24 smithi064 ceph-mon[36638]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 728 KiB/s wr, 116 op/s 2024-02-09T22:53:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:24 smithi089 ceph-mon[36665]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 728 KiB/s wr, 116 op/s 2024-02-09T22:53:24.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53959 3.95 MB/sec warmup 78 sec latency 1545.634 ms 2024-02-09T22:53:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:24 smithi003 ceph-mon[32678]: pgmap v67: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 728 KiB/s wr, 116 op/s 2024-02-09T22:53:25.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53959 3.90 MB/sec warmup 79 sec latency 2545.746 ms 2024-02-09T22:53:26.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:26 smithi064 ceph-mon[36638]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 728 KiB/s wr, 116 op/s 2024-02-09T22:53:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:26 smithi089 ceph-mon[36665]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 728 KiB/s wr, 116 op/s 2024-02-09T22:53:26.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53959 3.85 MB/sec warmup 80 sec latency 3545.858 ms 2024-02-09T22:53:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:26 smithi003 ceph-mon[32678]: pgmap v68: 97 pgs: 97 active+clean; 33 MiB data, 562 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 728 KiB/s wr, 116 op/s 2024-02-09T22:53:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:27.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:27.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 53959 3.80 MB/sec warmup 81 sec latency 4545.962 ms 2024-02-09T22:53:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:27.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 -- 172.21.15.3:0/2832375630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c0ff6c0 msgr2=0x7f889c0ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2832375630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c0ff6c0 0x7f889c0ffb40 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f8884009930 tx=0x7f888402bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 -- 172.21.15.3:0/2832375630 shutdown_connections 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2832375630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 0x7f889c10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2832375630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c0ff6c0 0x7f889c0ffb40 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2832375630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f889c106220 0x7f889c106600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 -- 172.21.15.3:0/2832375630 >> 172.21.15.3:0/2832375630 conn(0x7f889c076650 msgr2=0x7f889c0fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 -- 172.21.15.3:0/2832375630 shutdown_connections 2024-02-09T22:53:28.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.255+0000 7f88a0c55640 1 -- 172.21.15.3:0/2832375630 wait complete. 2024-02-09T22:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 Processor -- start 2024-02-09T22:53:28.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 -- start start 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f889c0ff6c0 0x7f889c1008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 0x7f889c100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 0x7f889c101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1a5d10 con 0x7f889c106220 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1a5e80 con 0x7f889c100080 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f88a0c55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f889c1a6040 con 0x7f889c0ff6c0 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 0x7f889c101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 0x7f889c101320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45810/0 (socket says 172.21.15.3:45810) 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.256+0000 7f889ad9d640 1 -- 172.21.15.3:0/2564681667 learned_addr learned my addr 172.21.15.3:0/2564681667 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889a59c640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f889c0ff6c0 0x7f889c1008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 0x7f889c100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889ad9d640 1 -- 172.21.15.3:0/2564681667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f889c0ff6c0 msgr2=0x7f889c1008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889ad9d640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f889c0ff6c0 0x7f889c1008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889ad9d640 1 -- 172.21.15.3:0/2564681667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 msgr2=0x7f889c100de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889ad9d640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 0x7f889c100de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889ad9d640 1 -- 172.21.15.3:0/2564681667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8884009560 con 0x7f889c106220 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 0x7f889c100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f889ad9d640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 0x7f889c101320 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f889400eaa0 tx=0x7f889400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f8893fff640 1 -- 172.21.15.3:0/2564681667 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f889400cdb0 con 0x7f889c106220 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f8893fff640 1 -- 172.21.15.3:0/2564681667 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8894005020 con 0x7f889c106220 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f889c1a6320 con 0x7f889c106220 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f8893fff640 1 -- 172.21.15.3:0/2564681667 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88940116b0 con 0x7f889c106220 2024-02-09T22:53:28.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.257+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c1a67b0 con 0x7f889c106220 2024-02-09T22:53:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.260+0000 7f8893fff640 1 -- 172.21.15.3:0/2564681667 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8894005520 con 0x7f889c106220 2024-02-09T22:53:28.271 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.260+0000 7f88a382d740 1 Processor -- start 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.260+0000 7f88a382d740 1 -- start start 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.260+0000 7f8893fff640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f88880a2220 0x7f88880a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.260+0000 7f88a382d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa8e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f88a382d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa97260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f88a382d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa977a0 0x55bf3aa99b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f88a382d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3a9fca00 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f88a382d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3a9f9f30 con 0x55bf3aa977a0 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f88a382d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3a9f5710 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f88880a2220 0x7f88880a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa8e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8893fff640 1 -- 172.21.15.3:0/2564681667 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f88940ce2c0 con 0x7f889c106220 2024-02-09T22:53:28.272 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa8e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45820/0 (socket says 172.21.15.3:45820) 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 -- 172.21.15.3:0/2367538663 learned_addr learned my addr 172.21.15.3:0/2367538663 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa97260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889ad9d640 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa977a0 0x55bf3aa99b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 -- 172.21.15.3:0/2367538663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 msgr2=0x55bf3aa97260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa97260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 -- 172.21.15.3:0/2367538663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa977a0 msgr2=0x55bf3aa99b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa977a0 0x55bf3aa99b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 -- 172.21.15.3:0/2367538663 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf3a9ba3b0 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f88880a2220 0x7f88880a46e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8884006010 tx=0x7f888402c650 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f889a59c640 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa8e490 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f888c009ca0 tx=0x7f888c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa22723fee91b48 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8890d99640 1 -- 172.21.15.3:0/2367538663 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c019620 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8890d99640 1 -- 172.21.15.3:0/2367538663 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f888c009e50 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.261+0000 7f8890d99640 1 -- 172.21.15.3:0/2367538663 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c018a70 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 -- 172.21.15.3:0/2367538663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 msgr2=0x55bf3aa8e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa8e490 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f888c009ca0 tx=0x7f888c011810 comp rx=0 tx=0).stop 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 -- 172.21.15.3:0/2367538663 shutdown_connections 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa977a0 0x55bf3aa99b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa97260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 --2- 172.21.15.3:0/2367538663 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa8e490 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 -- 172.21.15.3:0/2367538663 >> 172.21.15.3:0/2367538663 conn(0x55bf3aa889d0 msgr2=0x55bf3aa88db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 -- 172.21.15.3:0/2367538663 shutdown_connections 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.262+0000 7f88a382d740 1 -- 172.21.15.3:0/2367538663 wait complete. 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 -- start start 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 Processor -- start 2024-02-09T22:53:28.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa94730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa951b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3a99ab90 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3a6c9250 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3a97b200 con 0x55bf3aa977a0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f88a382d740 10 client.0 _ll_register_callbacks cb 0x55bf3a9b15d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa94730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa94730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45834/0 (socket says 172.21.15.3:45834) 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 -- 172.21.15.3:0/2798915180 learned_addr learned my addr 172.21.15.3:0/2798915180 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889ad9d640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 -- 172.21.15.3:0/2798915180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 msgr2=0x55bf3aa951b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa951b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 -- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e9d0 msgr2=0x55bf3aa94c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.263+0000 7f889a59c640 1 -- 172.21.15.3:0/2798915180 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f888c0098d0 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f889a59c640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa94730 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f888c011d10 tx=0x7f888c0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a31ab24bcef6e0d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8881ffb640 1 -- 172.21.15.3:0/2798915180 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f888c004220 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8881ffb640 1 -- 172.21.15.3:0/2798915180 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f888c004380 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bf3aa1bee0 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8881ffb640 1 -- 172.21.15.3:0/2798915180 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f888c018a00 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f88a382d740 10 client.16368 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bf3a9f3d90 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8881ffb640 1 -- 172.21.15.3:0/2798915180 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bf3aa1bee0 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f8881ffb640 1 -- 172.21.15.3:0/2798915180 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f888c028030 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f88a382d740 10 client.16368 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f88a382d740 10 client.16368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.264+0000 7f88a382d740 20 client.16368 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.265+0000 7f88a382d740 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaa7940 0x55bf3aaa9d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.265+0000 7f88a382d740 4 client.16368 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.265+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bf3a6f5600 con 0x55bf3aaa7940 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.265+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaa7940 0x55bf3aaa9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.265+0000 7f8891d9b640 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaa7940 0x55bf3aaa9d80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2123fc437e7e1b4e server_cookie=962eea126b598671 in_seq=0 out_seq=0 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.265+0000 7f8881ffb640 10 client.16368 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f8881ffb640 1 -- 172.21.15.3:0/2798915180 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bf3a6f5600 con 0x55bf3aaa7940 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f8881ffb640 10 client.16368 handle_command_reply: tid=0 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 client.16368 shutdown 2024-02-09T22:53:28.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 msgr2=0x55bf3aa94730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa94730 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f888c011d10 tx=0x7f888c0096c0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 shutdown_connections 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaa7940 0x55bf3aaa9d80 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f8881ffb640 0 client.16368 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 --2- 172.21.15.3:0/2798915180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa94730 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 >> 172.21.15.3:0/2798915180 conn(0x55bf3aa889d0 msgr2=0x55bf3aa8a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 shutdown_connections 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 1 -- 172.21.15.3:0/2798915180 wait complete. 2024-02-09T22:53:28.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.266+0000 7f88a382d740 20 client.16368 trim_cache size 0 max 16384 2024-02-09T22:53:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 Processor -- start 2024-02-09T22:53:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 -- start start 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa96e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa93000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3aa88550 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3aa90310 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f88a382d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3aa8f5a0 con 0x55bf3aa977a0 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f8891d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.280+0000 7f8891d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45846/0 (socket says 172.21.15.3:45846) 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 -- 172.21.15.3:0/3917903958 learned_addr learned my addr 172.21.15.3:0/3917903958 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f889ad9d640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa93000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f889a59c640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa96e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 -- 172.21.15.3:0/3917903958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 msgr2=0x55bf3aa93000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa93000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 -- 172.21.15.3:0/3917903958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 msgr2=0x55bf3aa96e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa96e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 -- 172.21.15.3:0/3917903958 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bf3a990820 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f889ad9d640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa93000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f889a59c640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa96e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8891d9b640 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94fd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8884004a10 tx=0x7f8884045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7cc10028d30b1f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8883fff640 1 -- 172.21.15.3:0/3917903958 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884053ea0 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8883fff640 1 -- 172.21.15.3:0/3917903958 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8884053830 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.281+0000 7f8883fff640 1 -- 172.21.15.3:0/3917903958 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8884003de0 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- 172.21.15.3:0/3917903958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 msgr2=0x55bf3aa94fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94fd0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f8884004a10 tx=0x7f8884045000 comp rx=0 tx=0).stop 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- 172.21.15.3:0/3917903958 shutdown_connections 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa977a0 0x55bf3aa93000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa94fd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- 172.21.15.3:0/3917903958 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa96e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- 172.21.15.3:0/3917903958 >> 172.21.15.3:0/3917903958 conn(0x55bf3aa889d0 msgr2=0x55bf3aaae720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- 172.21.15.3:0/3917903958 shutdown_connections 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- 172.21.15.3:0/3917903958 wait complete. 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- start start 2024-02-09T22:53:28.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 Processor -- start 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa9e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa9cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 0x55bf3aa9d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3aa9d850 con 0x55bf3aa977a0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3aa9d9c0 con 0x55bf3aa8e0b0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bf3aafdeb0 con 0x55bf3aa8e9d0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f88a382d740 10 client.0 _ll_register_callbacks cb 0x7f888c009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.282+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 0x55bf3aa9d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 0x55bf3aa9d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45848/0 (socket says 172.21.15.3:45848) 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 -- 172.21.15.3:0/903001428 learned_addr learned my addr 172.21.15.3:0/903001428 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889a59c640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa9e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f8891d9b640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa9cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 -- 172.21.15.3:0/903001428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 msgr2=0x55bf3aa9cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.293 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa9cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 -- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 msgr2=0x55bf3aa9e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa9e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 -- 172.21.15.3:0/903001428 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889c10ea00 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889a59c640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa9e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f889ad9d640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 0x55bf3aa9d150 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f88940cbaa0 tx=0x7f889409b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e63b7bbafd86335a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f8881ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f8881ffb640 1 -- 172.21.15.3:0/903001428 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8894005390 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f8881ffb640 1 -- 172.21.15.3:0/903001428 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f88940cd040 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f888c009e50 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f8881ffb640 1 -- 172.21.15.3:0/903001428 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f889404f3f0 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f88a382d740 10 client.16374 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55bf3a9b66f0 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.283+0000 7f8881ffb640 1 -- 172.21.15.3:0/903001428 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f888c009e50 con 0x55bf3aa977a0 2024-02-09T22:53:28.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f8881ffb640 1 -- 172.21.15.3:0/903001428 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f8894018030 con 0x55bf3aa977a0 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f88a382d740 10 client.16374 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f88a382d740 10 client.16374 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f88a382d740 20 client.16374 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f88a382d740 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaff7d0 0x55bf3ab01bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f88a382d740 4 client.16374 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bf3a7461b0 con 0x55bf3aaff7d0 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f889a59c640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaff7d0 0x55bf3ab01bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f8881ffb640 10 client.16374 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:28.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.284+0000 7f889a59c640 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaff7d0 0x55bf3ab01bf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf2269a25c0381b8 server_cookie=9a3fe612c936a442 in_seq=0 out_seq=0 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f8881ffb640 1 -- 172.21.15.3:0/903001428 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bf3a7461b0 con 0x55bf3aaff7d0 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f8881ffb640 10 client.16374 handle_command_reply: tid=0 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 client.16374 shutdown 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 msgr2=0x55bf3aa9d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 0x55bf3aa9d150 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f88940cbaa0 tx=0x7f889409b4a0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 shutdown_connections 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bf3aaff7d0 0x55bf3ab01bf0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bf3aa977a0 0x55bf3aa9d150 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bf3aa8e9d0 0x55bf3aa9cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f8881ffb640 0 client.16374 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 --2- 172.21.15.3:0/903001428 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bf3aa8e0b0 0x55bf3aa9e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 >> 172.21.15.3:0/903001428 conn(0x55bf3aa889d0 msgr2=0x55bf3aa92f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:28.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 shutdown_connections 2024-02-09T22:53:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 1 -- 172.21.15.3:0/903001428 wait complete. 2024-02-09T22:53:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.306+0000 7f88a382d740 20 client.16374 trim_cache size 0 max 16384 2024-02-09T22:53:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:53:28.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:53:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:53:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:53:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f88880a2220 msgr2=0x7f88880a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f88880a2220 0x7f88880a46e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8884006010 tx=0x7f888402c650 comp rx=0 tx=0).stop 2024-02-09T22:53:28.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 msgr2=0x7f889c101320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 0x7f889c101320 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f889400eaa0 tx=0x7f889400ef60 comp rx=0 tx=0).stop 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 shutdown_connections 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f88880a2220 0x7f88880a46e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f889c100080 0x7f889c100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f889c106220 0x7f889c101320 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 --2- 172.21.15.3:0/2564681667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f889c0ff6c0 0x7f889c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 >> 172.21.15.3:0/2564681667 conn(0x7f889c076650 msgr2=0x7f889c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:28.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 shutdown_connections 2024-02-09T22:53:28.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:28.307+0000 7f88a0c55640 1 -- 172.21.15.3:0/2564681667 wait complete. 2024-02-09T22:53:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:28 smithi089 ceph-mon[36665]: pgmap v69: 97 pgs: 97 active+clean; 33 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 801 KiB/s wr, 130 op/s 2024-02-09T22:53:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:28 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:28.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 54027 3.76 MB/sec warmup 82 sec latency 4961.387 ms 2024-02-09T22:53:28.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:28 smithi003 ceph-mon[32678]: pgmap v69: 97 pgs: 97 active+clean; 33 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 801 KiB/s wr, 130 op/s 2024-02-09T22:53:28.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:28 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:28 smithi064 ceph-mon[36638]: pgmap v69: 97 pgs: 97 active+clean; 33 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 801 KiB/s wr, 130 op/s 2024-02-09T22:53:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:28 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:29.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 54027 3.71 MB/sec warmup 83 sec latency 1534.768 ms 2024-02-09T22:53:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:30 smithi089 ceph-mon[36665]: pgmap v70: 97 pgs: 97 active+clean; 33 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 597 KiB/s wr, 77 op/s 2024-02-09T22:53:30.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 54027 3.67 MB/sec warmup 84 sec latency 2534.869 ms 2024-02-09T22:53:30.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:30 smithi003 ceph-mon[32678]: pgmap v70: 97 pgs: 97 active+clean; 33 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 597 KiB/s wr, 77 op/s 2024-02-09T22:53:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:30 smithi064 ceph-mon[36638]: pgmap v70: 97 pgs: 97 active+clean; 33 MiB data, 564 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 597 KiB/s wr, 77 op/s 2024-02-09T22:53:31.625 INFO:tasks.workunit.client.0.smithi003.stdout: 1 54027 3.63 MB/sec warmup 85 sec latency 3534.982 ms 2024-02-09T22:53:32.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:32 smithi089 ceph-mon[36665]: pgmap v71: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 866 KiB/s wr, 101 op/s 2024-02-09T22:53:32.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 54027 3.58 MB/sec warmup 86 sec latency 4535.139 ms 2024-02-09T22:53:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:32 smithi003 ceph-mon[32678]: pgmap v71: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 866 KiB/s wr, 101 op/s 2024-02-09T22:53:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:32 smithi064 ceph-mon[36638]: pgmap v71: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 866 KiB/s wr, 101 op/s 2024-02-09T22:53:33.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 55634 3.62 MB/sec warmup 87 sec latency 4952.769 ms 2024-02-09T22:53:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:34 smithi089 ceph-mon[36665]: pgmap v72: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 342 KiB/s wr, 37 op/s 2024-02-09T22:53:34.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 57341 3.72 MB/sec warmup 88 sec latency 17.650 ms 2024-02-09T22:53:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:34 smithi003 ceph-mon[32678]: pgmap v72: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 342 KiB/s wr, 37 op/s 2024-02-09T22:53:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:34 smithi064 ceph-mon[36638]: pgmap v72: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 342 KiB/s wr, 37 op/s 2024-02-09T22:53:35.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 60116 3.84 MB/sec warmup 89 sec latency 14.341 ms 2024-02-09T22:53:36.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:36 smithi089 ceph-mon[36665]: pgmap v73: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 342 KiB/s wr, 37 op/s 2024-02-09T22:53:36.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 60995 3.85 MB/sec warmup 90 sec latency 278.076 ms 2024-02-09T22:53:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:36 smithi003 ceph-mon[32678]: pgmap v73: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 342 KiB/s wr, 37 op/s 2024-02-09T22:53:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:36 smithi064 ceph-mon[36638]: pgmap v73: 97 pgs: 97 active+clean; 35 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 342 KiB/s wr, 37 op/s 2024-02-09T22:53:37.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 60995 3.80 MB/sec warmup 91 sec latency 1278.206 ms 2024-02-09T22:53:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:38 smithi089 ceph-mon[36665]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 872 KiB/s wr, 108 op/s 2024-02-09T22:53:38.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 62579 3.82 MB/sec warmup 92 sec latency 1694.232 ms 2024-02-09T22:53:38.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:38 smithi003 ceph-mon[32678]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 872 KiB/s wr, 108 op/s 2024-02-09T22:53:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:38 smithi064 ceph-mon[36638]: pgmap v74: 97 pgs: 97 active+clean; 35 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 872 KiB/s wr, 108 op/s 2024-02-09T22:53:39.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 64328 3.93 MB/sec warmup 93 sec latency 11.994 ms 2024-02-09T22:53:40.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:40 smithi089 ceph-mon[36665]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 799 KiB/s wr, 94 op/s 2024-02-09T22:53:40.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 64592 3.89 MB/sec warmup 94 sec latency 797.003 ms 2024-02-09T22:53:40.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:40 smithi003 ceph-mon[32678]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 799 KiB/s wr, 94 op/s 2024-02-09T22:53:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:40 smithi064 ceph-mon[36638]: pgmap v75: 97 pgs: 97 active+clean; 35 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 896 KiB/s rd, 799 KiB/s wr, 94 op/s 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.404+0000 7fc988b55640 1 -- 172.21.15.3:0/1673568047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9841054e0 msgr2=0x7fc98410d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.404+0000 7fc988b55640 1 --2- 172.21.15.3:0/1673568047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9841054e0 0x7fc98410d020 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fc97c009cd0 tx=0x7fc97c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 -- 172.21.15.3:0/1673568047 shutdown_connections 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 --2- 172.21.15.3:0/1673568047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9841054e0 0x7fc98410d020 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 --2- 172.21.15.3:0/1673568047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984101810 0x7fc984104db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 --2- 172.21.15.3:0/1673568047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc984100e60 0x7fc984101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 -- 172.21.15.3:0/1673568047 >> 172.21.15.3:0/1673568047 conn(0x7fc9840753d0 msgr2=0x7fc9840757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 -- 172.21.15.3:0/1673568047 shutdown_connections 2024-02-09T22:53:41.409 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 -- 172.21.15.3:0/1673568047 wait complete. 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 Processor -- start 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.405+0000 7fc988b55640 1 -- start start 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc988b55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984100e60 0x7fc9840fd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc988b55640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9841054e0 0x7fc9840fde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc988b55640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 0x7fc9841a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc988b55640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9841a5f50 con 0x7fc9840fe360 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc988b55640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9841a60c0 con 0x7fc9841054e0 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc988b55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9841a6280 con 0x7fc984100e60 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 0x7fc9841a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 0x7fc9841a5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44972/0 (socket says 172.21.15.3:44972) 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 -- 172.21.15.3:0/170746983 learned_addr learned my addr 172.21.15.3:0/170746983 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc98259c640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984100e60 0x7fc9840fd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc97bfff640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9841054e0 0x7fc9840fde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 -- 172.21.15.3:0/170746983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984100e60 msgr2=0x7fc9840fd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984100e60 0x7fc9840fd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 -- 172.21.15.3:0/170746983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9841054e0 msgr2=0x7fc9840fde20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9841054e0 0x7fc9840fde20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc982d9d640 1 -- 172.21.15.3:0/170746983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc97c009900 con 0x7fc9840fe360 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.406+0000 7fc98259c640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984100e60 0x7fc9840fd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.407+0000 7fc982d9d640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 0x7fc9841a5980 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc97c004af0 tx=0x7fc97c034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.407+0000 7fc97b7fe640 1 -- 172.21.15.3:0/170746983 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0328c0 con 0x7fc9840fe360 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.407+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9841a6500 con 0x7fc9840fe360 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.407+0000 7fc97b7fe640 1 -- 172.21.15.3:0/170746983 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c003e50 con 0x7fc9840fe360 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.407+0000 7fc97b7fe640 1 -- 172.21.15.3:0/170746983 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0047f0 con 0x7fc9840fe360 2024-02-09T22:53:41.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.407+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9841a69b0 con 0x7fc9840fe360 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.409+0000 7fc97b7fe640 1 -- 172.21.15.3:0/170746983 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc97c00a040 con 0x7fc9840fe360 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 Processor -- start 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 -- start start 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc97b7fe640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc9700a2250 0x7fc9700a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc97b7fe640 1 -- 172.21.15.3:0/170746983 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc97c0f09b0 con 0x7fc9840fe360 2024-02-09T22:53:41.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd19d0 0x55c93bcda260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc97bfff640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc9700a2250 0x7fc9700a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcda7a0 0x55c93bcdcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bc3fa00 con 0x55c93bcd10b0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bc3cf30 con 0x55c93bcd19d0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bc38710 con 0x55c93bcda7a0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd1490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44986/0 (socket says 172.21.15.3:44986) 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc98259c640 1 -- 172.21.15.3:0/814748470 learned_addr learned my addr 172.21.15.3:0/814748470 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc97bfff640 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd19d0 0x55c93bcda260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.410+0000 7fc982d9d640 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcda7a0 0x55c93bcdcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc98259c640 1 -- 172.21.15.3:0/814748470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcda7a0 msgr2=0x55c93bcdcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc98259c640 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcda7a0 0x55c93bcdcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc98259c640 1 -- 172.21.15.3:0/814748470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd19d0 msgr2=0x55c93bcda260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc98259c640 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd19d0 0x55c93bcda260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc98259c640 1 -- 172.21.15.3:0/814748470 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c93bbfd3b0 con 0x55c93bcd10b0 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc97bfff640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc9700a2250 0x7fc9700a4710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9840ff370 tx=0x7fc96c0137d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc98259c640 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd1490 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc97400f4c0 tx=0x7fc97400f980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a33fce0cda9628 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc978ff9640 1 -- 172.21.15.3:0/814748470 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400ab90 con 0x55c93bcd10b0 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc978ff9640 1 -- 172.21.15.3:0/814748470 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc974005700 con 0x55c93bcd10b0 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.411+0000 7fc978ff9640 1 -- 172.21.15.3:0/814748470 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc974010b80 con 0x55c93bcd10b0 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 -- 172.21.15.3:0/814748470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 msgr2=0x55c93bcd1490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd1490 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc97400f4c0 tx=0x7fc97400f980 comp rx=0 tx=0).stop 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 -- 172.21.15.3:0/814748470 shutdown_connections 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd19d0 0x55c93bcda260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcda7a0 0x55c93bcdcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/814748470 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd1490 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 -- 172.21.15.3:0/814748470 >> 172.21.15.3:0/814748470 conn(0x55c93bccb9d0 msgr2=0x55c93bccbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 -- 172.21.15.3:0/814748470 shutdown_connections 2024-02-09T22:53:41.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 -- 172.21.15.3:0/814748470 wait complete. 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 -- start start 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.412+0000 7fc98b7fd740 1 Processor -- start 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcd81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bbddb90 con 0x55c93bcd10b0 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93b90c250 con 0x55c93bcda7a0 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bbbe200 con 0x55c93bcd19d0 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98b7fd740 10 client.0 _ll_register_callbacks cb 0x55c93bbf45d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd7730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44990/0 (socket says 172.21.15.3:44990) 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 -- 172.21.15.3:0/3182908272 learned_addr learned my addr 172.21.15.3:0/3182908272 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc97bfff640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc982d9d640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcd81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 -- 172.21.15.3:0/3182908272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 msgr2=0x55c93bcd7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 -- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 msgr2=0x55c93bcd81b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcd81b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 -- 172.21.15.3:0/3182908272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc97400acf0 con 0x55c93bcd10b0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc98259c640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd7730 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc974000c00 tx=0x7fc974005e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e678d13271fd0ea4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.413+0000 7fc969ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc969ffb640 1 -- 172.21.15.3:0/3182908272 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9740054f0 con 0x55c93bcd10b0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc969ffb640 1 -- 172.21.15.3:0/3182908272 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc9740046c0 con 0x55c93bcd10b0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c93bc5eee0 con 0x55c93bcd10b0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc969ffb640 1 -- 172.21.15.3:0/3182908272 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc974004880 con 0x55c93bcd10b0 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 10 client.16386 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:41.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c93bc36d90 con 0x55c93bcd10b0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc969ffb640 1 -- 172.21.15.3:0/3182908272 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c93bc5eee0 con 0x55c93bcd10b0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc969ffb640 1 -- 172.21.15.3:0/3182908272 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc974020040 con 0x55c93bcd10b0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 10 client.16386 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 10 client.16386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 20 client.16386 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bcea9a0 0x55c93bcecdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 4 client.16386 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.414+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c93b938600 con 0x55c93bcea9a0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.415+0000 7fc97bfff640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bcea9a0 0x55c93bcecdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.415+0000 7fc97bfff640 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bcea9a0 0x55c93bcecdc0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed96b485cbf48694 server_cookie=d63eb4a004930c7 in_seq=0 out_seq=0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.415+0000 7fc969ffb640 10 client.16386 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc969ffb640 1 -- 172.21.15.3:0/3182908272 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c93b938600 con 0x55c93bcea9a0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc969ffb640 10 client.16386 handle_command_reply: tid=0 2024-02-09T22:53:41.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 client.16386 shutdown 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 msgr2=0x55c93bcd7730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd7730 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc974000c00 tx=0x7fc974005e60 comp rx=0 tx=0).stop 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 shutdown_connections 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bcea9a0 0x55c93bcecdc0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcd81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc969ffb640 0 client.16386 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/3182908272 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bcd7730 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 >> 172.21.15.3:0/3182908272 conn(0x55c93bccb9d0 msgr2=0x55c93bccd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 shutdown_connections 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 1 -- 172.21.15.3:0/3182908272 wait complete. 2024-02-09T22:53:41.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.416+0000 7fc98b7fd740 20 client.16386 trim_cache size 0 max 16384 2024-02-09T22:53:41.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 Processor -- start 2024-02-09T22:53:41.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 -- start start 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd10b0 0x55c93bcd9e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 0x55c93bcd4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bccb580 con 0x55c93bcda7a0 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bcdcbc0 con 0x55c93bcd10b0 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.429+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bcd5cb0 con 0x55c93bcd19d0 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 0x55c93bcd4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 0x55c93bcd4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45006/0 (socket says 172.21.15.3:45006) 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 -- 172.21.15.3:0/2249954322 learned_addr learned my addr 172.21.15.3:0/2249954322 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc98259c640 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd10b0 0x55c93bcd9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc97bfff640 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 -- 172.21.15.3:0/2249954322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 msgr2=0x55c93bcd7fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 -- 172.21.15.3:0/2249954322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd10b0 msgr2=0x55c93bcd9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd10b0 0x55c93bcd9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 -- 172.21.15.3:0/2249954322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c93bbd3820 con 0x55c93bcda7a0 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc982d9d640 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 0x55c93bcd4990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0bd180 tx=0x7fc97c004290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2192d65dd85155c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc96bfff640 1 -- 172.21.15.3:0/2249954322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0bcde0 con 0x55c93bcda7a0 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc96bfff640 1 -- 172.21.15.3:0/2249954322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc97c0dd0b0 con 0x55c93bcda7a0 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.430+0000 7fc96bfff640 1 -- 172.21.15.3:0/2249954322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97c0bb860 con 0x55c93bcda7a0 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- 172.21.15.3:0/2249954322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 msgr2=0x55c93bcd4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 0x55c93bcd4990 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0bd180 tx=0x7fc97c004290 comp rx=0 tx=0).stop 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- 172.21.15.3:0/2249954322 shutdown_connections 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcda7a0 0x55c93bcd4990 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcd7fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/2249954322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcd10b0 0x55c93bcd9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- 172.21.15.3:0/2249954322 >> 172.21.15.3:0/2249954322 conn(0x55c93bccb9d0 msgr2=0x55c93bcd6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- 172.21.15.3:0/2249954322 shutdown_connections 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- 172.21.15.3:0/2249954322 wait complete. 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- start start 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 Processor -- start 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bce1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcdfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcdfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bce0600 con 0x55c93bcd10b0 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bce0770 con 0x55c93bcda7a0 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c93bce0930 con 0x55c93bcd19d0 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.431+0000 7fc98b7fd740 10 client.0 _ll_register_callbacks cb 0x7fc96c0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:41.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bce1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bce1710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45008/0 (socket says 172.21.15.3:45008) 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 -- 172.21.15.3:0/235536624 learned_addr learned my addr 172.21.15.3:0/235536624 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc982d9d640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcdfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc97bfff640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcdfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 -- 172.21.15.3:0/235536624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 msgr2=0x55c93bcdfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcdfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 -- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 msgr2=0x55c93bcdfff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcdfff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 -- 172.21.15.3:0/235536624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9841a69b0 con 0x55c93bcd10b0 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98259c640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bce1710 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc974005f10 tx=0x7fc974010c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a53e9192c8ae189 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc969ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:41.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc969ffb640 1 -- 172.21.15.3:0/235536624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97400ab90 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc969ffb640 1 -- 172.21.15.3:0/235536624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc974000e40 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc974005700 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc969ffb640 1 -- 172.21.15.3:0/235536624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc974010760 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98b7fd740 10 client.16392 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.432+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55c93bbf96f0 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc969ffb640 1 -- 172.21.15.3:0/235536624 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc974005700 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc969ffb640 1 -- 172.21.15.3:0/235536624 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fc974020070 con 0x55c93bcd10b0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc98b7fd740 10 client.16392 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc98b7fd740 10 client.16392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc98b7fd740 20 client.16392 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bd422d0 0x55c93bd446f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc98b7fd740 4 client.16392 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c93b9891b0 con 0x55c93bd422d0 2024-02-09T22:53:41.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc97bfff640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bd422d0 0x55c93bd446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc969ffb640 10 client.16392 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:41.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.433+0000 7fc97bfff640 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bd422d0 0x55c93bd446f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=86f3ed5ed4fc4ec9 server_cookie=6828278aa7d9009e in_seq=0 out_seq=0 2024-02-09T22:53:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc969ffb640 1 -- 172.21.15.3:0/235536624 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c93b9891b0 con 0x55c93bd422d0 2024-02-09T22:53:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc969ffb640 10 client.16392 handle_command_reply: tid=0 2024-02-09T22:53:41.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 client.16392 shutdown 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 msgr2=0x55c93bce1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bce1710 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fc974005f10 tx=0x7fc974010c30 comp rx=0 tx=0).stop 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 shutdown_connections 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c93bd422d0 0x55c93bd446f0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c93bcda7a0 0x55c93bcdfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc969ffb640 0 client.16392 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c93bcd19d0 0x55c93bcdfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 --2- 172.21.15.3:0/235536624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c93bcd10b0 0x55c93bce1710 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 >> 172.21.15.3:0/235536624 conn(0x55c93bccb9d0 msgr2=0x55c93bcca1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 shutdown_connections 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 1 -- 172.21.15.3:0/235536624 wait complete. 2024-02-09T22:53:41.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.464+0000 7fc98b7fd740 20 client.16392 trim_cache size 0 max 16384 2024-02-09T22:53:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:53:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:53:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:53:41.470 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc9700a2250 msgr2=0x7fc9700a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc9700a2250 0x7fc9700a4710 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc9840ff370 tx=0x7fc96c0137d0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 msgr2=0x7fc9841a5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 0x7fc9841a5980 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fc97c004af0 tx=0x7fc97c034cc0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 shutdown_connections 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc9700a2250 0x7fc9700a4710 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc9841054e0 0x7fc9840fde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc9840fe360 0x7fc9841a5980 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 --2- 172.21.15.3:0/170746983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc984100e60 0x7fc9840fd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 >> 172.21.15.3:0/170746983 conn(0x7fc9840753d0 msgr2=0x7fc9840ffc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 shutdown_connections 2024-02-09T22:53:41.471 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:41.465+0000 7fc988b55640 1 -- 172.21.15.3:0/170746983 wait complete. 2024-02-09T22:53:41.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 64592 3.85 MB/sec warmup 95 sec latency 1797.104 ms 2024-02-09T22:53:42.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:42 smithi089 ceph-mon[36665]: pgmap v76: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T22:53:42.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 64592 3.81 MB/sec warmup 96 sec latency 2797.208 ms 2024-02-09T22:53:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:42 smithi003 ceph-mon[32678]: pgmap v76: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T22:53:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:42 smithi064 ceph-mon[36638]: pgmap v76: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T22:53:43.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 66217 3.84 MB/sec warmup 97 sec latency 3211.079 ms 2024-02-09T22:53:44.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:44 smithi089 ceph-mon[36665]: pgmap v77: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 910 KiB/s wr, 139 op/s 2024-02-09T22:53:44.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 67978 3.92 MB/sec warmup 98 sec latency 8.827 ms 2024-02-09T22:53:44.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:44 smithi003 ceph-mon[32678]: pgmap v77: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 910 KiB/s wr, 139 op/s 2024-02-09T22:53:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:44 smithi064 ceph-mon[36638]: pgmap v77: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 910 KiB/s wr, 139 op/s 2024-02-09T22:53:45.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68087 3.88 MB/sec warmup 99 sec latency 933.837 ms 2024-02-09T22:53:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:46 smithi089 ceph-mon[36665]: pgmap v78: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 910 KiB/s wr, 139 op/s 2024-02-09T22:53:46.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68087 3.84 MB/sec warmup 100 sec latency 1933.873 ms 2024-02-09T22:53:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:46 smithi003 ceph-mon[32678]: pgmap v78: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 910 KiB/s wr, 139 op/s 2024-02-09T22:53:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:46 smithi064 ceph-mon[36638]: pgmap v78: 97 pgs: 97 active+clean; 38 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 910 KiB/s wr, 139 op/s 2024-02-09T22:53:47.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:53:47.242 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:53:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.358+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1198732082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 msgr2=0x7ffbbc10d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.358+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1198732082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc10d0c0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4009930 tx=0x7ffbb402bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:47.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.358+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1198732082 shutdown_connections 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.358+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1198732082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc10d0c0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.358+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1198732082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 0x7ffbbc102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.358+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1198732082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 0x7ffbbc100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.359+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1198732082 >> 172.21.15.3:0/1198732082 conn(0x7ffbbc077310 msgr2=0x7ffbbc077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.359+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1198732082 shutdown_connections 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.359+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1198732082 wait complete. 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.359+0000 7ffbc17ac640 1 Processor -- start 2024-02-09T22:53:47.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.359+0000 7ffbc17ac640 1 -- start start 2024-02-09T22:53:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbc17ac640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 0x7ffbbc0fd070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbc17ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 0x7ffbbc0fd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbc17ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc0fdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbc17ac640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1a5c30 con 0x7ffbbc1026c0 2024-02-09T22:53:47.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbc17ac640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1a5da0 con 0x7ffbbc100440 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbc17ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbbc1a5f60 con 0x7ffbbc101d20 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc0fdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc0fdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35234/0 (socket says 172.21.15.3:35234) 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 -- 172.21.15.3:0/1050290812 learned_addr learned my addr 172.21.15.3:0/1050290812 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 0x7ffbbc0fd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 0x7ffbbc0fd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 -- 172.21.15.3:0/1050290812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 msgr2=0x7ffbbc0fd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 0x7ffbbc0fd5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 -- 172.21.15.3:0/1050290812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 msgr2=0x7ffbbc0fd070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 0x7ffbbc0fd070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 -- 172.21.15.3:0/1050290812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb4009560 con 0x7ffbbc1026c0 2024-02-09T22:53:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 0x7ffbbc0fd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 0x7ffbbc0fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffbbb7fe640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc0fdaf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4005940 tx=0x7ffbb402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.360+0000 7ffba3fff640 1 -- 172.21.15.3:0/1050290812 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4004350 con 0x7ffbbc1026c0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.361+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbbc1a61e0 con 0x7ffbbc1026c0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.361+0000 7ffba3fff640 1 -- 172.21.15.3:0/1050290812 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbb40044b0 con 0x7ffbbc1026c0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.361+0000 7ffba3fff640 1 -- 172.21.15.3:0/1050290812 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb4003cc0 con 0x7ffbbc1026c0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.361+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbbc1a6740 con 0x7ffbbc1026c0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffba3fff640 1 -- 172.21.15.3:0/1050290812 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ffbb4005580 con 0x7ffbbc1026c0 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 Processor -- start 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 -- start start 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 0x55fe1a83fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a8488e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a848eb0 0x55fe1a84b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a7fad00 con 0x55fe1a83f980 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a7af2f0 con 0x55fe1a840330 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.363+0000 7ffbc437d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a7ac820 con 0x55fe1a848eb0 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 0x55fe1a83fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffba3fff640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ffba40a2230 0x7ffba40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 0x55fe1a83fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35244/0 (socket says 172.21.15.3:35244) 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2194774474 learned_addr learned my addr 172.21.15.3:0/2194774474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a8488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbbb7fe640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ffba40a2230 0x7ffba40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a848eb0 0x55fe1a84b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffba3fff640 1 -- 172.21.15.3:0/1050290812 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ffbb40f01a0 con 0x7ffbbc1026c0 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2194774474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a848eb0 msgr2=0x55fe1a84b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a848eb0 0x55fe1a84b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2194774474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 msgr2=0x55fe1a8488e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a8488e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2194774474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe1a769ff0 con 0x55fe1a83f980 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 0x55fe1a83fd60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0013540 tx=0x7ffbb000b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c8d6120f9bd186 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffbbb7fe640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ffba40a2230 0x7ffba40a46f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40edae0 tx=0x7ffbb40b23a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffba17fa640 1 -- 172.21.15.3:0/2194774474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0004020 con 0x55fe1a83f980 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffba17fa640 1 -- 172.21.15.3:0/2194774474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbb0004180 con 0x55fe1a83f980 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.364+0000 7ffba17fa640 1 -- 172.21.15.3:0/2194774474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00116c0 con 0x55fe1a83f980 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 -- 172.21.15.3:0/2194774474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 msgr2=0x55fe1a83fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 0x55fe1a83fd60 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0013540 tx=0x7ffbb000b6b0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 -- 172.21.15.3:0/2194774474 shutdown_connections 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a8488e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a848eb0 0x55fe1a84b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2194774474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a83f980 0x55fe1a83fd60 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 -- 172.21.15.3:0/2194774474 >> 172.21.15.3:0/2194774474 conn(0x55fe1a83a050 msgr2=0x55fe1a83a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 -- 172.21.15.3:0/2194774474 shutdown_connections 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 -- 172.21.15.3:0/2194774474 wait complete. 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.365+0000 7ffbc437d740 1 -- start start 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 Processor -- start 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a845dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a846310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a846850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a7a8000 con 0x55fe1a848eb0 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a47b430 con 0x55fe1a83f980 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a437e00 con 0x55fe1a840330 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbc437d740 10 client.0 _ll_register_callbacks cb 0x55fe1a79c7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a846850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a846850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35260/0 (socket says 172.21.15.3:35260) 2024-02-09T22:53:47.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2278467125 learned_addr learned my addr 172.21.15.3:0/2278467125 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a846310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a845dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2278467125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 msgr2=0x55fe1a846310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a846310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 msgr2=0x55fe1a845dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a845dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbba7fc640 1 -- 172.21.15.3:0/2278467125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb000be00 con 0x55fe1a848eb0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.366+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a846310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a846850 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0003d00 tx=0x7ffbb0003d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9967ad2d6598c08d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffb9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffb9eff5640 1 -- 172.21.15.3:0/2278467125 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb0003670 con 0x55fe1a848eb0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe1a4a77e0 con 0x55fe1a848eb0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffb9eff5640 1 -- 172.21.15.3:0/2278467125 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ffbb00037d0 con 0x55fe1a848eb0 2024-02-09T22:53:47.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffb9eff5640 1 -- 172.21.15.3:0/2278467125 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbb0003990 con 0x55fe1a848eb0 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 10 client.16404 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fe1a7a6680 con 0x55fe1a848eb0 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffb9eff5640 1 -- 172.21.15.3:0/2278467125 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fe1a4a77e0 con 0x55fe1a848eb0 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffb9eff5640 1 -- 172.21.15.3:0/2278467125 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7ffbb001e460 con 0x55fe1a848eb0 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 10 client.16404 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 10 client.16404 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 20 client.16404 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a859310 0x55fe1a85b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 4 client.16404 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.367+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe1a4e4720 con 0x55fe1a859310 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.368+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a859310 0x55fe1a85b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.368+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a859310 0x55fe1a85b730 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f3c432c5dcc98a6 server_cookie=791ad63abd63e286 in_seq=0 out_seq=0 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.368+0000 7ffb9eff5640 10 client.16404 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffb9eff5640 1 -- 172.21.15.3:0/2278467125 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fe1a4e4720 con 0x55fe1a859310 2024-02-09T22:53:47.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffb9eff5640 10 client.16404 handle_command_reply: tid=0 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 client.16404 shutdown 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 msgr2=0x55fe1a846850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a846850 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ffbb0003d00 tx=0x7ffbb0003d60 comp rx=0 tx=0).stop 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 shutdown_connections 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a859310 0x55fe1a85b730 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a846850 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a846310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffb9eff5640 0 client.16404 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 --2- 172.21.15.3:0/2278467125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a845dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 >> 172.21.15.3:0/2278467125 conn(0x55fe1a83a050 msgr2=0x55fe1a83bda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 shutdown_connections 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 1 -- 172.21.15.3:0/2278467125 wait complete. 2024-02-09T22:53:47.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.369+0000 7ffbc437d740 20 client.16404 trim_cache size 0 max 16384 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 Processor -- start 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 -- start start 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a83f980 0x55fe1a848500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a860050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a843030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a72cfd0 con 0x55fe1a848eb0 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a845ac0 con 0x55fe1a840330 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbc437d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a835c90 con 0x55fe1a83f980 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a843030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a843030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35274/0 (socket says 172.21.15.3:35274) 2024-02-09T22:53:47.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1208569682 learned_addr learned my addr 172.21.15.3:0/1208569682 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a860050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.384+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a83f980 0x55fe1a848500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1208569682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a83f980 msgr2=0x55fe1a848500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a83f980 0x55fe1a848500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1208569682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 msgr2=0x55fe1a860050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a860050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1208569682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe1a742b30 con 0x55fe1a848eb0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a843030 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ffbb001ea40 tx=0x7ffbb0005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24edb7705195ff8d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffba0ff9640 1 -- 172.21.15.3:0/1208569682 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00042f0 con 0x55fe1a848eb0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffba0ff9640 1 -- 172.21.15.3:0/1208569682 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbb000b7e0 con 0x55fe1a848eb0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffba0ff9640 1 -- 172.21.15.3:0/1208569682 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb00114c0 con 0x55fe1a848eb0 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 -- 172.21.15.3:0/1208569682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 msgr2=0x55fe1a843030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a843030 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ffbb001ea40 tx=0x7ffbb0005d10 comp rx=0 tx=0).stop 2024-02-09T22:53:47.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 -- 172.21.15.3:0/1208569682 shutdown_connections 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a840330 0x55fe1a860050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a843030 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1208569682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a83f980 0x55fe1a848500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 -- 172.21.15.3:0/1208569682 >> 172.21.15.3:0/1208569682 conn(0x55fe1a83a050 msgr2=0x55fe1a83a430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.385+0000 7ffbc437d740 1 -- 172.21.15.3:0/1208569682 shutdown_connections 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 -- 172.21.15.3:0/1208569682 wait complete. 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 -- start start 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 Processor -- start 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a84e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a84eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a84f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a8507b0 con 0x55fe1a848eb0 2024-02-09T22:53:47.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a8af7e0 con 0x55fe1a83f980 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe1a8af950 con 0x55fe1a840330 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbc437d740 10 client.0 _ll_register_callbacks cb 0x7ffbb000b510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a84f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a84f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35290/0 (socket says 172.21.15.3:35290) 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1360880544 learned_addr learned my addr 172.21.15.3:0/1360880544 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a84e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.386+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a84eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1360880544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 msgr2=0x55fe1a84eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a84eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 msgr2=0x55fe1a84e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a84e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbba7fc640 1 -- 172.21.15.3:0/1360880544 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe1a7a6680 con 0x55fe1a848eb0 2024-02-09T22:53:47.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbba7fc640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a84f2e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c010 tx=0x7ffbb001e460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3da1b80795ffc2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffb9eff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffb9eff5640 1 -- 172.21.15.3:0/1360880544 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbb001e680 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffb9eff5640 1 -- 172.21.15.3:0/1360880544 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ffbb0000e40 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe1a845c00 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffb9eff5640 1 -- 172.21.15.3:0/1360880544 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffbb0003ec0 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 10 client.16410 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55fe1a768850 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffb9eff5640 1 -- 172.21.15.3:0/1360880544 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fe1a845c00 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffb9eff5640 1 -- 172.21.15.3:0/1360880544 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7ffbb00118a0 con 0x55fe1a848eb0 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 10 client.16410 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 10 client.16410 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 20 client.16410 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a8b0d40 0x55fe1a8b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 4 client.16410 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:53:47.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.387+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7ffbb0004180 con 0x55fe1a8b0d40 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a8b0d40 0x55fe1a8b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffb9eff5640 10 client.16410 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbbaffd640 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a8b0d40 0x55fe1a8b3160 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=763b5d9d8e12698c server_cookie=412954b57c4ca667 in_seq=0 out_seq=0 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffb9eff5640 1 -- 172.21.15.3:0/1360880544 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7ffbb0004180 con 0x55fe1a8b0d40 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffb9eff5640 10 client.16410 handle_command_reply: tid=0 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 client.16410 shutdown 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 msgr2=0x55fe1a84f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a84f2e0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7ffbb000c010 tx=0x7ffbb001e460 comp rx=0 tx=0).stop 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 shutdown_connections 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fe1a8b0d40 0x55fe1a8b3160 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fe1a848eb0 0x55fe1a84f2e0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fe1a840330 0x55fe1a84eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 --2- 172.21.15.3:0/1360880544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fe1a83f980 0x55fe1a84e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffb9eff5640 0 client.16410 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.388+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 >> 172.21.15.3:0/1360880544 conn(0x55fe1a83a050 msgr2=0x55fe1a860050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 shutdown_connections 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc437d740 1 -- 172.21.15.3:0/1360880544 wait complete. 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc437d740 20 client.16410 trim_cache size 0 max 16384 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:53:47.399 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:53:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ffba40a2230 msgr2=0x7ffba40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ffba40a2230 0x7ffba40a46f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ffbb40edae0 tx=0x7ffbb40b23a0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 msgr2=0x7ffbbc0fdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc0fdaf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ffbb4005940 tx=0x7ffbb402c650 comp rx=0 tx=0).stop 2024-02-09T22:53:47.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 shutdown_connections 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ffba40a2230 0x7ffba40a46f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ffbbc1026c0 0x7ffbbc0fdaf0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ffbbc101d20 0x7ffbbc0fd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 --2- 172.21.15.3:0/1050290812 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ffbbc100440 0x7ffbbc0fd070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 >> 172.21.15.3:0/1050290812 conn(0x7ffbbc077310 msgr2=0x7ffbbc069b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 shutdown_connections 2024-02-09T22:53:47.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.389+0000 7ffbc17ac640 1 -- 172.21.15.3:0/1050290812 wait complete. 2024-02-09T22:53:47.403 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:53:47.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:53:47.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 -- 172.21.15.3:0/937729279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 msgr2=0x7ff52c101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 --2- 172.21.15.3:0/937729279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c101820 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009930 tx=0x7ff51c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:47.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 -- 172.21.15.3:0/937729279 shutdown_connections 2024-02-09T22:53:47.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 --2- 172.21.15.3:0/937729279 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c101df0 0x7ff52c105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 --2- 172.21.15.3:0/937729279 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff52c106510 0x7ff52c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 --2- 172.21.15.3:0/937729279 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c101820 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 -- 172.21.15.3:0/937729279 >> 172.21.15.3:0/937729279 conn(0x7ff52c0fcce0 msgr2=0x7ff52c0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 -- 172.21.15.3:0/937729279 shutdown_connections 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 -- 172.21.15.3:0/937729279 wait complete. 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 Processor -- start 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.503+0000 7ff532913640 1 -- start start 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff52c106510 0x7ff52c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c197670 0x7ff52c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c198140 con 0x7ff52c101440 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c1982b0 con 0x7ff52c197670 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff52c198470 con 0x7ff52c106510 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35314/0 (socket says 172.21.15.3:35314) 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 -- 172.21.15.3:0/3334818041 learned_addr learned my addr 172.21.15.3:0/3334818041 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff530eb0640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c197670 0x7ff52c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52b7fe640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff52c106510 0x7ff52c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 -- 172.21.15.3:0/3334818041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff52c106510 msgr2=0x7ff52c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff52c106510 0x7ff52c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 -- 172.21.15.3:0/3334818041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c197670 msgr2=0x7ff52c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c197670 0x7ff52c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 -- 172.21.15.3:0/3334818041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff51c009560 con 0x7ff52c101440 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff530eb0640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c197670 0x7ff52c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff52bfff640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c19d5b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009230 tx=0x7ff51c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c004440 con 0x7ff52c101440 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff52c1a43b0 con 0x7ff52c101440 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff51c0052d0 con 0x7ff52c101440 2024-02-09T22:53:47.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.504+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff51c003d70 con 0x7ff52c101440 2024-02-09T22:53:47.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.505+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff52c1a4890 con 0x7ff52c101440 2024-02-09T22:53:47.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.506+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff52c102910 con 0x7ff52c101440 2024-02-09T22:53:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.506+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff51c005440 con 0x7ff52c101440 2024-02-09T22:53:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.507+0000 7ff5297fa640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff5140a2230 0x7ff5140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.507+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff51c0f0270 con 0x7ff52c101440 2024-02-09T22:53:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.507+0000 7ff52b7fe640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff5140a2230 0x7ff5140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.508+0000 7ff52b7fe640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff5140a2230 0x7ff5140a46f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff520005fd0 tx=0x7ff520005850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.512+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff51c0b9040 con 0x7ff52c101440 2024-02-09T22:53:47.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68087 3.81 MB/sec warmup 101 sec latency 2933.906 ms 2024-02-09T22:53:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.732+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff52c109740 con 0x7ff52c101440 2024-02-09T22:53:47.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.732+0000 7ff5297fa640 1 -- 172.21.15.3:0/3334818041 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11371 (secure 0 0 0) 0x7ff51c0bc700 con 0x7ff52c101440 2024-02-09T22:53:47.735 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:53:47.736 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:53:02.760086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:53:47.736 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 31 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff5140a2230 msgr2=0x7ff5140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff5140a2230 0x7ff5140a46f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff520005fd0 tx=0x7ff520005850 comp rx=0 tx=0).stop 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 msgr2=0x7ff52c19d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c19d5b0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7ff51c009230 tx=0x7ff51c004040 comp rx=0 tx=0).stop 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 shutdown_connections 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff5140a2230 0x7ff5140a46f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff52c197670 0x7ff52c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff52c106510 0x7ff52c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 --2- 172.21.15.3:0/3334818041 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff52c101440 0x7ff52c19d5b0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.734+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 >> 172.21.15.3:0/3334818041 conn(0x7ff52c0fcce0 msgr2=0x7ff52c0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.735+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 shutdown_connections 2024-02-09T22:53:47.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.735+0000 7ff532913640 1 -- 172.21.15.3:0/3334818041 wait complete. 2024-02-09T22:53:47.748 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:47.748 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.851+0000 7fcf26113640 1 -- 172.21.15.3:0/1568634817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2010b820 msgr2=0x7fcf2010dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.851+0000 7fcf26113640 1 --2- 172.21.15.3:0/1568634817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2010b820 0x7fcf2010dc50 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fcf18009ca0 tx=0x7fcf1802c180 comp rx=0 tx=0).stop 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- 172.21.15.3:0/1568634817 shutdown_connections 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 --2- 172.21.15.3:0/1568634817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf2010b820 0x7fcf2010dc50 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 --2- 172.21.15.3:0/1568634817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf20102d70 0x7fcf2010b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 --2- 172.21.15.3:0/1568634817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf20102450 0x7fcf20102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- 172.21.15.3:0/1568634817 >> 172.21.15.3:0/1568634817 conn(0x7fcf20077250 msgr2=0x7fcf20077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- 172.21.15.3:0/1568634817 shutdown_connections 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- 172.21.15.3:0/1568634817 wait complete. 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 Processor -- start 2024-02-09T22:53:47.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- start start 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf20102450 0x7fcf2019d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 0x7fcf2019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf2010b820 0x7fcf20197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20197ec0 con 0x7fcf20102d70 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf20198030 con 0x7fcf2010b820 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.852+0000 7fcf26113640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf201981f0 con 0x7fcf20102450 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 0x7fcf2019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 0x7fcf2019dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35332/0 (socket says 172.21.15.3:35332) 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 -- 172.21.15.3:0/498780907 learned_addr learned my addr 172.21.15.3:0/498780907 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf2010b820 0x7fcf20197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf20102450 0x7fcf2019d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 -- 172.21.15.3:0/498780907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf20102450 msgr2=0x7fcf2019d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf20102450 0x7fcf2019d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 -- 172.21.15.3:0/498780907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf2010b820 msgr2=0x7fcf20197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf2010b820 0x7fcf20197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 -- 172.21.15.3:0/498780907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf18009900 con 0x7fcf20102d70 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1effd640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 0x7fcf2019dbb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400b4a0 tx=0x7fcf1400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1cff9640 1 -- 172.21.15.3:0/498780907 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14010610 con 0x7fcf20102d70 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1cff9640 1 -- 172.21.15.3:0/498780907 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf14010c50 con 0x7fcf20102d70 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf201984d0 con 0x7fcf20102d70 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf1cff9640 1 -- 172.21.15.3:0/498780907 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14017e20 con 0x7fcf20102d70 2024-02-09T22:53:47.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.853+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf201a45f0 con 0x7fcf20102d70 2024-02-09T22:53:47.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.855+0000 7fcf1cff9640 1 -- 172.21.15.3:0/498780907 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fcf14017480 con 0x7fcf20102d70 2024-02-09T22:53:47.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 Processor -- start 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 -- start start 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x55925923bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259244d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592592453a0 0x559259245820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592591f7020 con 0x55925923b890 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592591aa320 con 0x55925923c240 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592591a7850 con 0x5592592453a0 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x55925923bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1cff9640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf080a2230 0x7fcf080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x55925923bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35340/0 (socket says 172.21.15.3:35340) 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2654043948 learned_addr learned my addr 172.21.15.3:0/2654043948 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1effd640 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592592453a0 0x559259245820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf080a2230 0x7fcf080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259244d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1cff9640 1 -- 172.21.15.3:0/498780907 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf140cd6b0 con 0x7fcf20102d70 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2654043948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592592453a0 msgr2=0x559259245820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592592453a0 0x559259245820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2654043948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 msgr2=0x559259244d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259244d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2654043948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559259165210 con 0x55925923b890 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.856+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259244d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.857+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x55925923bc70 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fcf180098d0 tx=0x7fcf18034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c141ea7c980e0f5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.857+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf080a2230 0x7fcf080a46f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf1804c000 tx=0x7fcf1804b000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.857+0000 7fcf05ffb640 1 -- 172.21.15.3:0/2654043948 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18003970 con 0x55925923b890 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.857+0000 7fcf05ffb640 1 -- 172.21.15.3:0/2654043948 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf18032c70 con 0x55925923b890 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.857+0000 7fcf05ffb640 1 -- 172.21.15.3:0/2654043948 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1804dbb0 con 0x55925923b890 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2654043948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 msgr2=0x55925923bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x55925923bc70 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fcf180098d0 tx=0x7fcf18034440 comp rx=0 tx=0).stop 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2654043948 shutdown_connections 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259244d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5592592453a0 0x559259245820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2654043948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x55925923bc70 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2654043948 >> 172.21.15.3:0/2654043948 conn(0x559259236100 msgr2=0x5592592364e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2654043948 shutdown_connections 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2654043948 wait complete. 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 -- start start 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 Processor -- start 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x559259248ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.858+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923c240 0x559259249420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592592453a0 0x559259249960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5592591a3030 con 0x55925923b890 2024-02-09T22:53:47.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559258e76430 con 0x5592592453a0 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559258e32e00 con 0x55925923c240 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 10 client.0 _ll_register_callbacks cb 0x559259198030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x559259248ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x559259248ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35348/0 (socket says 172.21.15.3:35348) 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 -- 172.21.15.3:0/2392808285 learned_addr learned my addr 172.21.15.3:0/2392808285 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1effd640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923c240 0x559259249420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 -- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923c240 msgr2=0x559259249420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923c240 0x559259249420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 -- 172.21.15.3:0/2392808285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592592453a0 msgr2=0x559259249960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592592453a0 0x559259249960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 -- 172.21.15.3:0/2392808285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2010e8d0 con 0x55925923b890 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1effd640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923c240 0x559259249420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x559259248ee0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000e1f0 tx=0x7fcf1000e6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d3c07230b024ef0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcefeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcefeffd640 1 -- 172.21.15.3:0/2392808285 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1000c070 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcefeffd640 1 -- 172.21.15.3:0/2392808285 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf10013610 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559258ea27e0 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcefeffd640 1 -- 172.21.15.3:0/2392808285 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf1000eee0 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 10 client.16428 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.859+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5592591a16b0 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcefeffd640 1 -- 172.21.15.3:0/2392808285 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559258ea27e0 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcefeffd640 1 -- 172.21.15.3:0/2392808285 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fcf1001ba10 con 0x55925923b890 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf28dc7740 10 client.16428 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf28dc7740 10 client.16428 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf28dc7740 20 client.16428 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559259259a10 0x55925925be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf28dc7740 4 client.16428 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:47.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559258edf720 con 0x559259259a10 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf1effd640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559259259a10 0x55925925be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcefeffd640 10 client.16428 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.860+0000 7fcf1effd640 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559259259a10 0x55925925be30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=af62634d8b441dfe server_cookie=651c229a9fda4a1 in_seq=0 out_seq=0 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcefeffd640 1 -- 172.21.15.3:0/2392808285 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559258edf720 con 0x559259259a10 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcefeffd640 10 client.16428 handle_command_reply: tid=0 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 client.16428 shutdown 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 msgr2=0x559259248ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x559259248ee0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fcf1000e1f0 tx=0x7fcf1000e6b0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 shutdown_connections 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559259259a10 0x55925925be30 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5592592453a0 0x559259249960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923c240 0x559259249420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcefeffd640 0 client.16428 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:47.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2392808285 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55925923b890 0x559259248ee0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 >> 172.21.15.3:0/2392808285 conn(0x559259236100 msgr2=0x559259237e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 shutdown_connections 2024-02-09T22:53:47.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.861+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2392808285 wait complete. 2024-02-09T22:53:47.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.862+0000 7fcf28dc7740 20 client.16428 trim_cache size 0 max 16384 2024-02-09T22:53:47.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 Processor -- start 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 -- start start 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x55925924b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259249780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x559259246790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559259128130 con 0x5592592453a0 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559259231a70 con 0x55925923c240 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559259247470 con 0x55925923b890 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.873+0000 7fcf1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x559259246790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x559259246790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35350/0 (socket says 172.21.15.3:35350) 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2252033325 learned_addr learned my addr 172.21.15.3:0/2252033325 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2252033325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 msgr2=0x55925924b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x55925924b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x55925924b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2252033325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 msgr2=0x559259249780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1effd640 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259249780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259249780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 -- 172.21.15.3:0/2252033325 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55925913dc90 con 0x5592592453a0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x55925924b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x559259246790 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fcf1802c800 tx=0x7fcf1802c180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c680f8a1608ccbd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf057fa640 1 -- 172.21.15.3:0/2252033325 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18034810 con 0x5592592453a0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf057fa640 1 -- 172.21.15.3:0/2252033325 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf18003970 con 0x5592592453a0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.874+0000 7fcf057fa640 1 -- 172.21.15.3:0/2252033325 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf180042a0 con 0x5592592453a0 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2252033325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 msgr2=0x559259246790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x559259246790 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fcf1802c800 tx=0x7fcf1802c180 comp rx=0 tx=0).stop 2024-02-09T22:53:47.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2252033325 shutdown_connections 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x559259249780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x559259246790 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/2252033325 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x55925924b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2252033325 >> 172.21.15.3:0/2252033325 conn(0x559259236100 msgr2=0x5592592607d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2252033325 shutdown_connections 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- 172.21.15.3:0/2252033325 wait complete. 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- start start 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 Processor -- start 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x559259250500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x55925924e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x55925924ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55925924f460 con 0x5592592453a0 2024-02-09T22:53:47.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55925924f5d0 con 0x55925923c240 2024-02-09T22:53:47.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.875+0000 7fcf28dc7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55925924f790 con 0x55925923b890 2024-02-09T22:53:47.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf28dc7740 10 client.0 _ll_register_callbacks cb 0x7fcf18037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:47.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x55925924ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x55925924ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35356/0 (socket says 172.21.15.3:35356) 2024-02-09T22:53:47.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 -- 172.21.15.3:0/1330079085 learned_addr learned my addr 172.21.15.3:0/1330079085 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:47.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1effd640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x55925924e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x559259250500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 -- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 msgr2=0x559259250500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x559259250500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 -- 172.21.15.3:0/1330079085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 msgr2=0x55925924e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x55925924e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 -- 172.21.15.3:0/1330079085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559259165210 con 0x5592592453a0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1effd640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x55925924e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf1ffff640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x55925924ef20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fcf18034f50 tx=0x7fcf180093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e983a6be5e65703b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcefeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcefeffd640 1 -- 172.21.15.3:0/1330079085 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf18056670 con 0x5592592453a0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcefeffd640 1 -- 172.21.15.3:0/1330079085 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf180567d0 con 0x5592592453a0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf18032c70 con 0x5592592453a0 2024-02-09T22:53:47.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcefeffd640 1 -- 172.21.15.3:0/1330079085 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf18056990 con 0x5592592453a0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf28dc7740 10 client.16434 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.876+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x559259163a70 con 0x5592592453a0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcefeffd640 1 -- 172.21.15.3:0/1330079085 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf18032c70 con 0x5592592453a0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcefeffd640 1 -- 172.21.15.3:0/1330079085 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fcf18042420 con 0x5592592453a0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf28dc7740 10 client.16434 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf28dc7740 10 client.16434 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf28dc7740 20 client.16434 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5592592b0ea0 0x5592592b32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf28dc7740 4 client.16434 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559258f2e890 con 0x5592592b0ea0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5592592b0ea0 0x5592592b32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcefeffd640 10 client.16434 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.877+0000 7fcf1f7fe640 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5592592b0ea0 0x5592592b32c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fa0f803fb71fa523 server_cookie=da8e309eeffedba2 in_seq=0 out_seq=0 2024-02-09T22:53:47.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcefeffd640 1 -- 172.21.15.3:0/1330079085 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559258f2e890 con 0x5592592b0ea0 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcefeffd640 10 client.16434 handle_command_reply: tid=0 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 client.16434 shutdown 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 msgr2=0x55925924ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x55925924ef20 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7fcf18034f50 tx=0x7fcf180093f0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 shutdown_connections 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5592592b0ea0 0x5592592b32c0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55925923c240 0x55925924e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5592592453a0 0x55925924ef20 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcefeffd640 0 client.16434 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 --2- 172.21.15.3:0/1330079085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55925923b890 0x559259250500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 >> 172.21.15.3:0/1330079085 conn(0x559259236100 msgr2=0x559259243c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 shutdown_connections 2024-02-09T22:53:47.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 1 -- 172.21.15.3:0/1330079085 wait complete. 2024-02-09T22:53:47.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf28dc7740 20 client.16434 trim_cache size 0 max 16384 2024-02-09T22:53:47.889 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf080a2230 msgr2=0x7fcf080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf080a2230 0x7fcf080a46f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcf1804c000 tx=0x7fcf1804b000 comp rx=0 tx=0).stop 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 msgr2=0x7fcf2019dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 0x7fcf2019dbb0 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400b4a0 tx=0x7fcf1400b960 comp rx=0 tx=0).stop 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 shutdown_connections 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf080a2230 0x7fcf080a46f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf2010b820 0x7fcf20197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf20102d70 0x7fcf2019dbb0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 --2- 172.21.15.3:0/498780907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf20102450 0x7fcf2019d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.878+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 >> 172.21.15.3:0/498780907 conn(0x7fcf20077250 msgr2=0x7fcf200fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.879+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 shutdown_connections 2024-02-09T22:53:47.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.879+0000 7fcf26113640 1 -- 172.21.15.3:0/498780907 wait complete. 2024-02-09T22:53:47.892 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:47.892 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 -- 172.21.15.3:0/75398960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 msgr2=0x7fba8010dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 --2- 172.21.15.3:0/75398960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba8010dc60 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7fba78009ca0 tx=0x7fba7802c180 comp rx=0 tx=0).stop 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 -- 172.21.15.3:0/75398960 shutdown_connections 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 --2- 172.21.15.3:0/75398960 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba8010dc60 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 --2- 172.21.15.3:0/75398960 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba80102d60 0x7fba8010b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 --2- 172.21.15.3:0/75398960 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba80102440 0x7fba80102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:47.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 -- 172.21.15.3:0/75398960 >> 172.21.15.3:0/75398960 conn(0x7fba80077300 msgr2=0x7fba80077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 -- 172.21.15.3:0/75398960 shutdown_connections 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.993+0000 7fba861e9640 1 -- 172.21.15.3:0/75398960 wait complete. 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 Processor -- start 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 -- start start 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba80102440 0x7fba8019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba80102d60 0x7fba8019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba801976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80197e30 con 0x7fba8010b810 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80197fa0 con 0x7fba80102440 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba861e9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba80198160 con 0x7fba80102d60 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba801976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba801976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35372/0 (socket says 172.21.15.3:35372) 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba7ffff640 1 -- 172.21.15.3:0/1424903776 learned_addr learned my addr 172.21.15.3:0/1424903776 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba80102440 0x7fba8019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.994+0000 7fba7effd640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba80102d60 0x7fba8019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7ffff640 1 -- 172.21.15.3:0/1424903776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba80102d60 msgr2=0x7fba8019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba80102d60 0x7fba8019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7ffff640 1 -- 172.21.15.3:0/1424903776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba80102440 msgr2=0x7fba8019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba80102440 0x7fba8019d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7ffff640 1 -- 172.21.15.3:0/1424903776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba78009900 con 0x7fba8010b810 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba801976a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fba7802c790 tx=0x7fba78034df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7cff9640 1 -- 172.21.15.3:0/1424903776 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78032790 con 0x7fba8010b810 2024-02-09T22:53:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba801983e0 con 0x7fba8010b810 2024-02-09T22:53:48.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7cff9640 1 -- 172.21.15.3:0/1424903776 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba78032dd0 con 0x7fba8010b810 2024-02-09T22:53:48.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba7cff9640 1 -- 172.21.15.3:0/1424903776 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba78040450 con 0x7fba8010b810 2024-02-09T22:53:48.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.995+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba801a4560 con 0x7fba8010b810 2024-02-09T22:53:48.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.997+0000 7fba7cff9640 1 -- 172.21.15.3:0/1424903776 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fba7800a040 con 0x7fba8010b810 2024-02-09T22:53:48.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.997+0000 7fba88dbe740 1 Processor -- start 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.997+0000 7fba88dbe740 1 -- start start 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670096de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700976990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7cff9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fba680a2230 0x7fba680a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009793b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba88dbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556700929020 con 0x556700976f60 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba88dbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567008dc320 con 0x55670096da30 2024-02-09T22:53:48.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba88dbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567008d9850 con 0x55670096e3e0 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7effd640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fba680a2230 0x7fba680a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009793b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700976990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700976990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:39186/0 (socket says 172.21.15.3:39186) 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009793b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35382/0 (socket says 172.21.15.3:35382) 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 -- 172.21.15.3:0/17945541 learned_addr learned my addr 172.21.15.3:0/17945541 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7cff9640 1 -- 172.21.15.3:0/1424903776 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba780f05c0 con 0x7fba8010b810 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7effd640 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670096de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 -- 172.21.15.3:0/17945541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 msgr2=0x55670096de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670096de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 -- 172.21.15.3:0/17945541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 msgr2=0x5567009793b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009793b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 -- 172.21.15.3:0/17945541 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556700897210 con 0x55670096e3e0 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7ffff640 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009793b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7effd640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fba680a2230 0x7fba680a46f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba80198ef0 tx=0x7fba740146c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.998+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700976990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba7000ac20 tx=0x7fba7000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7cd561cb46f8bb4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba65ffb640 1 -- 172.21.15.3:0/17945541 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba7000a720 con 0x55670096e3e0 2024-02-09T22:53:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba65ffb640 1 -- 172.21.15.3:0/17945541 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7000ae60 con 0x55670096e3e0 2024-02-09T22:53:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba65ffb640 1 -- 172.21.15.3:0/17945541 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba70010b30 con 0x55670096e3e0 2024-02-09T22:53:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 -- 172.21.15.3:0/17945541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 msgr2=0x556700976990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700976990 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba7000ac20 tx=0x7fba7000f790 comp rx=0 tx=0).stop 2024-02-09T22:53:48.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 -- 172.21.15.3:0/17945541 shutdown_connections 2024-02-09T22:53:48.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009793b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700976990 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 --2- 172.21.15.3:0/17945541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670096de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 -- 172.21.15.3:0/17945541 >> 172.21.15.3:0/17945541 conn(0x556700968100 msgr2=0x5567009684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 -- 172.21.15.3:0/17945541 shutdown_connections 2024-02-09T22:53:48.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:47.999+0000 7fba88dbe740 1 -- 172.21.15.3:0/17945541 wait complete. 2024-02-09T22:53:48.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 -- start start 2024-02-09T22:53:48.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 Processor -- start 2024-02-09T22:53:48.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700973f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700974470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009749b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567008d5030 con 0x556700976f60 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567005a8430 con 0x55670096da30 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556700564e00 con 0x55670096e3e0 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba88dbe740 10 client.0 _ll_register_callbacks cb 0x5567008ca030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009749b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009749b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35398/0 (socket says 172.21.15.3:35398) 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 -- 172.21.15.3:0/1001586353 learned_addr learned my addr 172.21.15.3:0/1001586353 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700973f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7effd640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700974470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 -- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 msgr2=0x556700974470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700974470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 -- 172.21.15.3:0/1001586353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 msgr2=0x556700973f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700973f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.000+0000 7fba7ffff640 1 -- 172.21.15.3:0/1001586353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7000a880 con 0x556700976f60 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009749b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fba780edeb0 tx=0x7fba7804b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f37f19426cd414a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba5effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba5effd640 1 -- 172.21.15.3:0/1001586353 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba780d5e20 con 0x556700976f60 2024-02-09T22:53:48.019 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba5effd640 1 -- 172.21.15.3:0/1001586353 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba780bd7f0 con 0x556700976f60 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5567005d47e0 con 0x556700976f60 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba5effd640 1 -- 172.21.15.3:0/1001586353 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba780bd0f0 con 0x556700976f60 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 10 client.16446 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5567008d36b0 con 0x556700976f60 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba5effd640 1 -- 172.21.15.3:0/1001586353 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba780d2070 con 0x556700976f60 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba5effd640 1 -- 172.21.15.3:0/1001586353 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fba780b9760 con 0x556700976f60 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 10 client.16446 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 10 client.16446 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:53:48.020 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 20 client.16446 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009872e0 0x556700989700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 4 client.16446 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.001+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556700611720 con 0x5567009872e0 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.002+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009872e0 0x556700989700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.002+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009872e0 0x556700989700 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=40d44808de3a750d server_cookie=f8104337071c637c in_seq=0 out_seq=0 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.002+0000 7fba5effd640 10 client.16446 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba5effd640 1 -- 172.21.15.3:0/1001586353 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556700611720 con 0x5567009872e0 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba5effd640 10 client.16446 handle_command_reply: tid=0 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 client.16446 shutdown 2024-02-09T22:53:48.021 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 msgr2=0x5567009749b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009749b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fba780edeb0 tx=0x7fba7804b000 comp rx=0 tx=0).stop 2024-02-09T22:53:48.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 shutdown_connections 2024-02-09T22:53:48.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009872e0 0x556700989700 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009749b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.022 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x556700974470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba5effd640 0 client.16446 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1001586353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700973f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 >> 172.21.15.3:0/1001586353 conn(0x556700968100 msgr2=0x556700969e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 shutdown_connections 2024-02-09T22:53:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 1 -- 172.21.15.3:0/1001586353 wait complete. 2024-02-09T22:53:48.023 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.003+0000 7fba88dbe740 20 client.16446 trim_cache size 0 max 16384 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 Processor -- start 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 -- start start 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700974480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 0x5567009749c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556700976f60 0x556700971190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670085a130 con 0x55670096e3e0 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556700968b50 con 0x55670096da30 2024-02-09T22:53:48.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba88dbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556700967c30 con 0x556700976f60 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 0x5567009749c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 0x5567009749c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35400/0 (socket says 172.21.15.3:35400) 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 -- 172.21.15.3:0/2596539587 learned_addr learned my addr 172.21.15.3:0/2596539587 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7ffff640 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556700976f60 0x556700971190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 -- 172.21.15.3:0/2596539587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556700976f60 msgr2=0x556700971190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556700976f60 0x556700971190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700974480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 -- 172.21.15.3:0/2596539587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 msgr2=0x556700974480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700974480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7effd640 1 -- 172.21.15.3:0/2596539587 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55670086fc90 con 0x55670096e3e0 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.015+0000 7fba7ffff640 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556700976f60 0x556700971190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba7effd640 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 0x5567009749c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a7a0 tx=0x7fba74009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78ed4b2f6212ce84 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba657fa640 1 -- 172.21.15.3:0/2596539587 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74026070 con 0x55670096e3e0 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba657fa640 1 -- 172.21.15.3:0/2596539587 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7401e040 con 0x55670096e3e0 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba657fa640 1 -- 172.21.15.3:0/2596539587 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba74006680 con 0x55670096e3e0 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 -- 172.21.15.3:0/2596539587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 msgr2=0x5567009749c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 0x5567009749c0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fba7400a7a0 tx=0x7fba74009000 comp rx=0 tx=0).stop 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 -- 172.21.15.3:0/2596539587 shutdown_connections 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556700976f60 0x556700971190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55670096e3e0 0x5567009749c0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 --2- 172.21.15.3:0/2596539587 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x556700974480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 -- 172.21.15.3:0/2596539587 >> 172.21.15.3:0/2596539587 conn(0x556700968100 msgr2=0x55670096f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 -- 172.21.15.3:0/2596539587 shutdown_connections 2024-02-09T22:53:48.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.016+0000 7fba88dbe740 1 -- 172.21.15.3:0/2596539587 wait complete. 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 -- start start 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 Processor -- start 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670097cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x55670097d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009dd7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670097da20 con 0x556700976f60 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55670097db90 con 0x55670096da30 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567009ddd60 con 0x55670096e3e0 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba88dbe740 10 client.0 _ll_register_callbacks cb 0x7fba7000a670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009dd7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009dd7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35414/0 (socket says 172.21.15.3:35414) 2024-02-09T22:53:48.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 -- 172.21.15.3:0/1441477553 learned_addr learned my addr 172.21.15.3:0/1441477553 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 -- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 msgr2=0x55670097d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x55670097d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 -- 172.21.15.3:0/1441477553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 msgr2=0x55670097cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670097cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.017+0000 7fba7ffff640 1 -- 172.21.15.3:0/1441477553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba7400e040 con 0x556700976f60 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba7ffff640 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009dd7f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fba7802c760 tx=0x7fba78043000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dcf25b8de39972 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba5effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba5effd640 1 -- 172.21.15.3:0/1441477553 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba780bd7f0 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba5effd640 1 -- 172.21.15.3:0/1441477553 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fba780d5e20 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fba7000ae60 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba5effd640 1 -- 172.21.15.3:0/1441477553 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fba7804e200 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 10 client.16452 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556700895a70 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba5effd640 1 -- 172.21.15.3:0/1441477553 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba7000ae60 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba5effd640 1 -- 172.21.15.3:0/1441477553 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fba780c5400 con 0x556700976f60 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 10 client.16452 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 10 client.16452 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 20 client.16452 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009df010 0x5567009e1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 4 client.16452 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:48.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556700660890 con 0x5567009df010 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.018+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009df010 0x5567009e1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba5effd640 10 client.16452 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba7f7fe640 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009df010 0x5567009e1450 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34d5dbf73561f58a server_cookie=6ea54cee9228022e in_seq=0 out_seq=0 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba5effd640 1 -- 172.21.15.3:0/1441477553 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556700660890 con 0x5567009df010 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba5effd640 10 client.16452 handle_command_reply: tid=0 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 client.16452 shutdown 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 msgr2=0x5567009dd7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009dd7f0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fba7802c760 tx=0x7fba78043000 comp rx=0 tx=0).stop 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 shutdown_connections 2024-02-09T22:53:48.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5567009df010 0x5567009e1450 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556700976f60 0x5567009dd7f0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55670096e3e0 0x55670097d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 --2- 172.21.15.3:0/1441477553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55670096da30 0x55670097cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba5effd640 0 client.16452 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:53:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 >> 172.21.15.3:0/1441477553 conn(0x556700968100 msgr2=0x5567009669b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 shutdown_connections 2024-02-09T22:53:48.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 1 -- 172.21.15.3:0/1441477553 wait complete. 2024-02-09T22:53:48.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.019+0000 7fba88dbe740 20 client.16452 trim_cache size 0 max 16384 2024-02-09T22:53:48.038 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:48.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fba680a2230 msgr2=0x7fba680a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fba680a2230 0x7fba680a46f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fba80198ef0 tx=0x7fba740146c0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 msgr2=0x7fba801976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba801976a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fba7802c790 tx=0x7fba78034df0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 shutdown_connections 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fba680a2230 0x7fba680a46f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fba8010b810 0x7fba801976a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fba80102d60 0x7fba8019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 --2- 172.21.15.3:0/1424903776 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fba80102440 0x7fba8019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 >> 172.21.15.3:0/1424903776 conn(0x7fba80077300 msgr2=0x7fba800fd1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 shutdown_connections 2024-02-09T22:53:48.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.020+0000 7fba861e9640 1 -- 172.21.15.3:0/1424903776 wait complete. 2024-02-09T22:53:48.042 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:48.042 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:53:48.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 -- 172.21.15.3:0/3335427792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 msgr2=0x7f04ac1065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 --2- 172.21.15.3:0/3335427792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac1065e0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009930 tx=0x7f04a002bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:48.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 -- 172.21.15.3:0/3335427792 shutdown_connections 2024-02-09T22:53:48.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 --2- 172.21.15.3:0/3335427792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04ac1000c0 0x7f04ac10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 --2- 172.21.15.3:0/3335427792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac0ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 --2- 172.21.15.3:0/3335427792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac1065e0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 -- 172.21.15.3:0/3335427792 >> 172.21.15.3:0/3335427792 conn(0x7f04ac076630 msgr2=0x7f04ac0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 -- 172.21.15.3:0/3335427792 shutdown_connections 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.146+0000 7f04b27c7640 1 -- 172.21.15.3:0/3335427792 wait complete. 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 Processor -- start 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 -- start start 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04ac1000c0 0x7f04ac19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac197740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac197ed0 con 0x7f04ac106200 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac198040 con 0x7f04ac1000c0 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b27c7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ac198200 con 0x7f04ac0ff700 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04b0d64640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac197740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04ab7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04ac1000c0 0x7f04ac19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04abfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac19d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:39222/0 (socket says 172.21.15.3:39222) 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.147+0000 7f04abfff640 1 -- 172.21.15.3:0/1833851749 learned_addr learned my addr 172.21.15.3:0/1833851749 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b0d64640 1 -- 172.21.15.3:0/1833851749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 msgr2=0x7f04ac19d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b0d64640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac19d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b0d64640 1 -- 172.21.15.3:0/1833851749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04ac1000c0 msgr2=0x7f04ac19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b0d64640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04ac1000c0 0x7f04ac19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b0d64640 1 -- 172.21.15.3:0/1833851749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a0009560 con 0x7f04ac106200 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04abfff640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b0d64640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac197740 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009d10 tx=0x7f04a4007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04a97fa640 1 -- 172.21.15.3:0/1833851749 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a4015070 con 0x7f04ac106200 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04a97fa640 1 -- 172.21.15.3:0/1833851749 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04a4009e60 con 0x7f04ac106200 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04ac198480 con 0x7f04ac106200 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04a97fa640 1 -- 172.21.15.3:0/1833851749 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a400ec00 con 0x7f04ac106200 2024-02-09T22:53:48.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.148+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac1009f0 con 0x7f04ac106200 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.150+0000 7f04a97fa640 1 -- 172.21.15.3:0/1833851749 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f04a4004a40 con 0x7f04ac106200 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.150+0000 7f04b5398740 1 Processor -- start 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 -- start start 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae38e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae41990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae41f60 0x56492ae443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492adf4020 con 0x56492ae393e0 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ada7320 con 0x56492ae38a30 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b5398740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ada4850 con 0x56492ae41f60 2024-02-09T22:53:48.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04a97fa640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04940a2230 0x7f04940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae41990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae41990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35440/0 (socket says 172.21.15.3:35440) 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 -- 172.21.15.3:0/4234714324 learned_addr learned my addr 172.21.15.3:0/4234714324 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b0d64640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04940a2230 0x7f04940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae38e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04a97fa640 1 -- 172.21.15.3:0/1833851749 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f04a40d1520 con 0x7f04ac106200 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae41f60 0x56492ae443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 -- 172.21.15.3:0/4234714324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae41f60 msgr2=0x56492ae443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae41f60 0x56492ae443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 -- 172.21.15.3:0/4234714324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 msgr2=0x56492ae38e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae38e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 -- 172.21.15.3:0/4234714324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56492ad62210 con 0x56492ae393e0 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae41f60 0x56492ae443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04abfff640 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae41990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009430 tx=0x7f04a002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=307eda32ee53e0c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.151+0000 7f04b0d64640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04940a2230 0x7f04940a46f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04a40ced50 tx=0x7f04a40936c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04927fc640 1 -- 172.21.15.3:0/4234714324 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00046c0 con 0x56492ae393e0 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04927fc640 1 -- 172.21.15.3:0/4234714324 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04a0004510 con 0x56492ae393e0 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04927fc640 1 -- 172.21.15.3:0/4234714324 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a00054e0 con 0x56492ae393e0 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 -- 172.21.15.3:0/4234714324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 msgr2=0x56492ae41990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae41990 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009430 tx=0x7f04a002c5d0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 -- 172.21.15.3:0/4234714324 shutdown_connections 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae41f60 0x56492ae443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae41990 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 --2- 172.21.15.3:0/4234714324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae38e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 -- 172.21.15.3:0/4234714324 >> 172.21.15.3:0/4234714324 conn(0x56492ae33100 msgr2=0x56492ae334e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 -- 172.21.15.3:0/4234714324 shutdown_connections 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.152+0000 7f04b5398740 1 -- 172.21.15.3:0/4234714324 wait complete. 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 -- start start 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 Processor -- start 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae3eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ada0030 con 0x56492ae41f60 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492aa73430 con 0x56492ae38a30 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492aa2fe00 con 0x56492ae393e0 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04b5398740 10 client.0 _ll_register_callbacks cb 0x56492ad95030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35448/0 (socket says 172.21.15.3:35448) 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04ab7fe640 1 -- 172.21.15.3:0/1982398782 learned_addr learned my addr 172.21.15.3:0/1982398782 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.153+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae3eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04ab7fe640 1 -- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 msgr2=0x56492ae3f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04abfff640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04ab7fe640 1 -- 172.21.15.3:0/1982398782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 msgr2=0x56492ae3eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae3eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04ab7fe640 1 -- 172.21.15.3:0/1982398782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04ac10e9e0 con 0x56492ae41f60 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3f970 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f049c011c70 tx=0x7f049c009aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e9b68c41f966c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f048b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f048b7fe640 1 -- 172.21.15.3:0/1982398782 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00fc50 con 0x56492ae41f60 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f048b7fe640 1 -- 172.21.15.3:0/1982398782 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f049c00fdb0 con 0x56492ae41f60 2024-02-09T22:53:48.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56492aa9f7e0 con 0x56492ae41f60 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f048b7fe640 1 -- 172.21.15.3:0/1982398782 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f049c004030 con 0x56492ae41f60 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 10 client.16464 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56492ad9e6b0 con 0x56492ae41f60 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f048b7fe640 1 -- 172.21.15.3:0/1982398782 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56492aa9f7e0 con 0x56492ae41f60 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f048b7fe640 1 -- 172.21.15.3:0/1982398782 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f049c02a450 con 0x56492ae41f60 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 10 client.16464 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 10 client.16464 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 20 client.16464 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492ae524b0 0x56492ae548d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 4 client.16464 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.154+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56492aadc720 con 0x56492ae524b0 2024-02-09T22:53:48.169 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.155+0000 7f04abfff640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492ae524b0 0x56492ae548d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.155+0000 7f04abfff640 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492ae524b0 0x56492ae548d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=75f2a090404e1c80 server_cookie=cbf2578b76dd1283 in_seq=0 out_seq=0 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.155+0000 7f048b7fe640 10 client.16464 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f048b7fe640 1 -- 172.21.15.3:0/1982398782 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56492aadc720 con 0x56492ae524b0 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f048b7fe640 10 client.16464 handle_command_reply: tid=0 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 client.16464 shutdown 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 msgr2=0x56492ae3f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3f970 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f049c011c70 tx=0x7f049c009aa0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 shutdown_connections 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492ae524b0 0x56492ae548d0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3f970 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.170 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 --2- 172.21.15.3:0/1982398782 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae3eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 >> 172.21.15.3:0/1982398782 conn(0x56492ae33100 msgr2=0x56492ae34e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f048b7fe640 0 client.16464 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:48.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 shutdown_connections 2024-02-09T22:53:48.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 1 -- 172.21.15.3:0/1982398782 wait complete. 2024-02-09T22:53:48.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.156+0000 7f04b5398740 20 client.16464 trim_cache size 0 max 16384 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 Processor -- start 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 -- start start 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae41620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ad25130 con 0x56492ae41f60 2024-02-09T22:53:48.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ae50d30 con 0x56492ae38a30 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04b5398740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ae38380 con 0x56492ae393e0 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35456/0 (socket says 172.21.15.3:35456) 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 -- 172.21.15.3:0/2272191467 learned_addr learned my addr 172.21.15.3:0/2272191467 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04abfff640 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae41620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 -- 172.21.15.3:0/2272191467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 msgr2=0x56492ae3a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 -- 172.21.15.3:0/2272191467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 msgr2=0x56492ae41620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae41620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.168+0000 7f04ab7fe640 1 -- 172.21.15.3:0/2272191467 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56492ad3ac90 con 0x56492ae41f60 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3c170 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f049c009f20 tx=0x7f049c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=819a3bfce349dcf8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f0491ffb640 1 -- 172.21.15.3:0/2272191467 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c011a70 con 0x56492ae41f60 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f0491ffb640 1 -- 172.21.15.3:0/2272191467 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f049c02aa80 con 0x56492ae41f60 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f0491ffb640 1 -- 172.21.15.3:0/2272191467 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049c00fe80 con 0x56492ae41f60 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 -- 172.21.15.3:0/2272191467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 msgr2=0x56492ae3c170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3c170 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f049c009f20 tx=0x7f049c001e50 comp rx=0 tx=0).stop 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 -- 172.21.15.3:0/2272191467 shutdown_connections 2024-02-09T22:53:48.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae41f60 0x56492ae3c170 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae393e0 0x56492ae3a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 --2- 172.21.15.3:0/2272191467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae38a30 0x56492ae41620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 -- 172.21.15.3:0/2272191467 >> 172.21.15.3:0/2272191467 conn(0x56492ae33100 msgr2=0x56492ae59210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 -- 172.21.15.3:0/2272191467 shutdown_connections 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.169+0000 7f04b5398740 1 -- 172.21.15.3:0/2272191467 wait complete. 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 -- start start 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 Processor -- start 2024-02-09T22:53:48.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae38a30 0x56492ae478c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae47e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae41f60 0x56492ae48340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492ae49860 con 0x56492ae393e0 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492aea89a0 con 0x56492ae41f60 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56492aea8b10 con 0x56492ae38a30 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04b5398740 10 client.0 _ll_register_callbacks cb 0x7f04a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae47e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04ab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae38a30 0x56492ae478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae47e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35462/0 (socket says 172.21.15.3:35462) 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 -- 172.21.15.3:0/278239 learned_addr learned my addr 172.21.15.3:0/278239 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae41f60 0x56492ae48340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 -- 172.21.15.3:0/278239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae38a30 msgr2=0x56492ae478c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae38a30 0x56492ae478c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 -- 172.21.15.3:0/278239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae41f60 msgr2=0x56492ae48340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae41f60 0x56492ae48340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04abfff640 1 -- 172.21.15.3:0/278239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56492ad9e6b0 con 0x56492ae393e0 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.170+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae41f60 0x56492ae48340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04abfff640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae47e00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f04a0005e50 tx=0x7f04a0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e563d6b765bf7354 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f048b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f048b7fe640 1 -- 172.21.15.3:0/278239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a004e070 con 0x56492ae393e0 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f048b7fe640 1 -- 172.21.15.3:0/278239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f04a0050b70 con 0x56492ae393e0 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04a0004510 con 0x56492ae393e0 2024-02-09T22:53:48.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f048b7fe640 1 -- 172.21.15.3:0/278239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04a0050d30 con 0x56492ae393e0 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 10 client.16470 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x56492ad60a70 con 0x56492ae393e0 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f048b7fe640 1 -- 172.21.15.3:0/278239 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f04a0004510 con 0x56492ae393e0 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f048b7fe640 1 -- 172.21.15.3:0/278239 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f04a003c050 con 0x56492ae393e0 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 10 client.16470 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 10 client.16470 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 20 client.16470 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492aea9ef0 0x56492aeac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 4 client.16470 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:48.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.171+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56492ab2b890 con 0x56492aea9ef0 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492aea9ef0 0x56492aeac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f048b7fe640 10 client.16470 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04ab7fe640 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492aea9ef0 0x56492aeac2e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5bf78ddd4328bf76 server_cookie=5ef9388cfbfbe7c6 in_seq=0 out_seq=0 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f048b7fe640 1 -- 172.21.15.3:0/278239 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56492ab2b890 con 0x56492aea9ef0 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f048b7fe640 10 client.16470 handle_command_reply: tid=0 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 client.16470 shutdown 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 msgr2=0x56492ae47e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae47e00 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f04a0005e50 tx=0x7f04a0005de0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 shutdown_connections 2024-02-09T22:53:48.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56492aea9ef0 0x56492aeac2e0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56492ae41f60 0x56492ae48340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56492ae393e0 0x56492ae47e00 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f048b7fe640 0 client.16470 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 --2- 172.21.15.3:0/278239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56492ae38a30 0x56492ae478c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 >> 172.21.15.3:0/278239 conn(0x56492ae33100 msgr2=0x56492ae3d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 shutdown_connections 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 1 -- 172.21.15.3:0/278239 wait complete. 2024-02-09T22:53:48.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.172+0000 7f04b5398740 20 client.16470 trim_cache size 0 max 16384 2024-02-09T22:53:48.190 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:48.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04940a2230 msgr2=0x7f04940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04940a2230 0x7f04940a46f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04a40ced50 tx=0x7f04a40936c0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 msgr2=0x7f04ac197740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac197740 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f04a4009d10 tx=0x7f04a4007590 comp rx=0 tx=0).stop 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 shutdown_connections 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04940a2230 0x7f04940a46f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f04ac1000c0 0x7f04ac19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f04ac106200 0x7f04ac197740 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 --2- 172.21.15.3:0/1833851749 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f04ac0ff700 0x7f04ac19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 >> 172.21.15.3:0/1833851749 conn(0x7f04ac076630 msgr2=0x7f04ac1041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 shutdown_connections 2024-02-09T22:53:48.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.173+0000 7f04b27c7640 1 -- 172.21.15.3:0/1833851749 wait complete. 2024-02-09T22:53:48.193 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:48.193 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 -- 172.21.15.3:0/1905794532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 msgr2=0x7f9060100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 --2- 172.21.15.3:0/1905794532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f9060100780 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f9050009930 tx=0x7f905002bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 -- 172.21.15.3:0/1905794532 shutdown_connections 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 --2- 172.21.15.3:0/1905794532 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 0x7f906010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 --2- 172.21.15.3:0/1905794532 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9060100d50 0x7f9060104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 --2- 172.21.15.3:0/1905794532 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f9060100780 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 -- 172.21.15.3:0/1905794532 >> 172.21.15.3:0/1905794532 conn(0x7f9060077300 msgr2=0x7f9060077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 -- 172.21.15.3:0/1905794532 shutdown_connections 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.300+0000 7f90667a7640 1 -- 172.21.15.3:0/1905794532 wait complete. 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 Processor -- start 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 -- start start 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f906019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 0x7f906019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9060197690 0x7f9060197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9060198160 con 0x7f90601003a0 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90601982d0 con 0x7f9060105420 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f90667a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9060198490 con 0x7f9060197690 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f906019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f906019d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35488/0 (socket says 172.21.15.3:35488) 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 -- 172.21.15.3:0/2771448383 learned_addr learned my addr 172.21.15.3:0/2771448383 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f9064d44640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9060197690 0x7f9060197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 0x7f906019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 -- 172.21.15.3:0/2771448383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9060197690 msgr2=0x7f9060197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9060197690 0x7f9060197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 -- 172.21.15.3:0/2771448383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 msgr2=0x7f906019db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 0x7f906019db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905ffff640 1 -- 172.21.15.3:0/2771448383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9050009560 con 0x7f90601003a0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.301+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 0x7f906019db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.302+0000 7f905ffff640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f906019d5d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9050005940 tx=0x7f90500042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.302+0000 7f905d7fa640 1 -- 172.21.15.3:0/2771448383 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90500044e0 con 0x7f90601003a0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.302+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90601a43d0 con 0x7f90601003a0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.302+0000 7f905d7fa640 1 -- 172.21.15.3:0/2771448383 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f905002cbc0 con 0x7f90601003a0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.302+0000 7f905d7fa640 1 -- 172.21.15.3:0/2771448383 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9050003e20 con 0x7f90601003a0 2024-02-09T22:53:48.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.302+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90601a48b0 con 0x7f90601003a0 2024-02-09T22:53:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.304+0000 7f905d7fa640 1 -- 172.21.15.3:0/2771448383 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f90500035f0 con 0x7f90601003a0 2024-02-09T22:53:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.304+0000 7f9069376740 1 Processor -- start 2024-02-09T22:53:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.304+0000 7f905d7fa640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f90480a2230 0x7f90480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.314 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.304+0000 7f9069376740 1 -- start start 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905d7fa640 1 -- 172.21.15.3:0/2771448383 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f90500f02a0 con 0x7f90601003a0 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9064d44640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f90480a2230 0x7f90480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9069376740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708eee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9069376740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708f7990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9069376740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 0x555e708fa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9069376740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e708aa020 con 0x555e708f7f60 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9069376740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e7085d320 con 0x555e708ef3e0 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9069376740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e7085a850 con 0x555e708eea30 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9064d44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 0x555e708fa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708eee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9064d44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 0x555e708fa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35504/0 (socket says 172.21.15.3:35504) 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9064d44640 1 -- 172.21.15.3:0/3426966121 learned_addr learned my addr 172.21.15.3:0/3426966121 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905f7fe640 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708f7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9064d44640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f90480a2230 0x7f90480a46f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f905800a850 tx=0x7f905800ac50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 -- 172.21.15.3:0/3426966121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 msgr2=0x555e708f7990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708f7990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 -- 172.21.15.3:0/3426966121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 msgr2=0x555e708fa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 0x555e708fa3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 -- 172.21.15.3:0/3426966121 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e70818210 con 0x555e708eea30 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f9064d44640 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 0x555e708fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f905ffff640 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708eee10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90500edb30 tx=0x7f90500bc730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eaaff2ade00ef32a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.305+0000 7f90467fc640 1 -- 172.21.15.3:0/3426966121 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90500c0f20 con 0x555e708eea30 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f90467fc640 1 -- 172.21.15.3:0/3426966121 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90500c5570 con 0x555e708eea30 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f90467fc640 1 -- 172.21.15.3:0/3426966121 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90500c5340 con 0x555e708eea30 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 -- 172.21.15.3:0/3426966121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 msgr2=0x555e708eee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708eee10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f90500edb30 tx=0x7f90500bc730 comp rx=0 tx=0).stop 2024-02-09T22:53:48.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 -- 172.21.15.3:0/3426966121 shutdown_connections 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708f7990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708f7f60 0x555e708fa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 --2- 172.21.15.3:0/3426966121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708eee10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 -- 172.21.15.3:0/3426966121 >> 172.21.15.3:0/3426966121 conn(0x555e708e9100 msgr2=0x555e708e94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 -- 172.21.15.3:0/3426966121 shutdown_connections 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.306+0000 7f9069376740 1 -- 172.21.15.3:0/3426966121 wait complete. 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 -- start start 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 Processor -- start 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708eea30 0x555e708f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f53d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e708f5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e70856030 con 0x555e708ef3e0 2024-02-09T22:53:48.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e70529430 con 0x555e708eea30 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e704e5e00 con 0x555e708f7f60 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9069376740 10 client.0 _ll_register_callbacks cb 0x555e7084b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35506/0 (socket says 172.21.15.3:35506) 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 -- 172.21.15.3:0/2120096651 learned_addr learned my addr 172.21.15.3:0/2120096651 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f9064d44640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e708f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905ffff640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708eea30 0x555e708f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 -- 172.21.15.3:0/2120096651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 msgr2=0x555e708f5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e708f5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 -- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708eea30 msgr2=0x555e708f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.307+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708eea30 0x555e708f4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f905f7fe640 1 -- 172.21.15.3:0/2120096651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90601a48b0 con 0x555e708ef3e0 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f905ffff640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708eea30 0x555e708f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f53d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f90540107f0 tx=0x7f9054010cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba947db9eb455257 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f903f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f903f7fe640 1 -- 172.21.15.3:0/2120096651 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054010ea0 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f903f7fe640 1 -- 172.21.15.3:0/2120096651 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f905400ed40 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e705557e0 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f903f7fe640 1 -- 172.21.15.3:0/2120096651 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9054021430 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 10 client.16482 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555e708546b0 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f903f7fe640 1 -- 172.21.15.3:0/2120096651 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555e705557e0 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f903f7fe640 1 -- 172.21.15.3:0/2120096651 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f905400d600 con 0x555e708ef3e0 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 10 client.16482 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 10 client.16482 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 20 client.16482 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e709082f0 0x555e7090a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 4 client.16482 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:48.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.308+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e70592720 con 0x555e709082f0 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.309+0000 7f905ffff640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e709082f0 0x555e7090a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.309+0000 7f905ffff640 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e709082f0 0x555e7090a710 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25da8ea5c82c9677 server_cookie=1f4c0f5f77dab1dc in_seq=0 out_seq=0 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.309+0000 7f903f7fe640 10 client.16482 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f903f7fe640 1 -- 172.21.15.3:0/2120096651 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e70592720 con 0x555e709082f0 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f903f7fe640 10 client.16482 handle_command_reply: tid=0 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 client.16482 shutdown 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 msgr2=0x555e708f53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f53d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f90540107f0 tx=0x7f9054010cb0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 shutdown_connections 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e709082f0 0x555e7090a710 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e708f5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f53d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f903f7fe640 0 client.16482 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 --2- 172.21.15.3:0/2120096651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708eea30 0x555e708f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 >> 172.21.15.3:0/2120096651 conn(0x555e708e9100 msgr2=0x555e708eae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 shutdown_connections 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 1 -- 172.21.15.3:0/2120096651 wait complete. 2024-02-09T22:53:48.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.310+0000 7f9069376740 20 client.16482 trim_cache size 0 max 16384 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 Processor -- start 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 -- start start 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708f75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708f7f60 0x555e708f2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e707db130 con 0x555e708ef3e0 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e705557e0 con 0x555e708f7f60 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9069376740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e708e4ba0 con 0x555e708eea30 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35516/0 (socket says 172.21.15.3:35516) 2024-02-09T22:53:48.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 -- 172.21.15.3:0/1577385574 learned_addr learned my addr 172.21.15.3:0/1577385574 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f9064d44640 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708f7f60 0x555e708f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 -- 172.21.15.3:0/1577385574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 msgr2=0x555e708f75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708f75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 -- 172.21.15.3:0/1577385574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708f7f60 msgr2=0x555e708f2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.322+0000 7f905f7fe640 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708f7f60 0x555e708f2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f905f7fe640 1 -- 172.21.15.3:0/1577385574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e707f0c90 con 0x555e708ef3e0 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f905f7fe640 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f5730 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f905400d930 tx=0x7f905400b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a6481cd8f7b1138 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9045ffb640 1 -- 172.21.15.3:0/1577385574 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9054010ea0 con 0x555e708ef3e0 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9045ffb640 1 -- 172.21.15.3:0/1577385574 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f905400e780 con 0x555e708ef3e0 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9045ffb640 1 -- 172.21.15.3:0/1577385574 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905400d690 con 0x555e708ef3e0 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 -- 172.21.15.3:0/1577385574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 msgr2=0x555e708f5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f5730 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f905400d930 tx=0x7f905400b520 comp rx=0 tx=0).stop 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 -- 172.21.15.3:0/1577385574 shutdown_connections 2024-02-09T22:53:48.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708f7f60 0x555e708f2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708ef3e0 0x555e708f5730 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 --2- 172.21.15.3:0/1577385574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708eea30 0x555e708f75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 -- 172.21.15.3:0/1577385574 >> 172.21.15.3:0/1577385574 conn(0x555e708e9100 msgr2=0x555e708f01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 -- 172.21.15.3:0/1577385574 shutdown_connections 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.323+0000 7f9069376740 1 -- 172.21.15.3:0/1577385574 wait complete. 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 -- start start 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 Processor -- start 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 0x555e708fe090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708fe5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e70958880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e70958dc0 con 0x555e708eea30 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e70958f00 con 0x555e708ef3e0 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e709590c0 con 0x555e708f7f60 2024-02-09T22:53:48.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f9069376740 10 client.0 _ll_register_callbacks cb 0x7f905002cd30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f905ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 0x555e708fe090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f905ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 0x555e708fe090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35530/0 (socket says 172.21.15.3:35530) 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f905ffff640 1 -- 172.21.15.3:0/2913827759 learned_addr learned my addr 172.21.15.3:0/2913827759 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.324+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708fe5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f905ffff640 1 -- 172.21.15.3:0/2913827759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 msgr2=0x555e70958880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f905ffff640 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e70958880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f905ffff640 1 -- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 msgr2=0x555e708fe5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f905ffff640 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708fe5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f905ffff640 1 -- 172.21.15.3:0/2913827759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e708546b0 con 0x555e708eea30 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f905ffff640 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 0x555e708fe090 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9050005c40 tx=0x7f9050003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466a11382179f103 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f903f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f903f7fe640 1 -- 172.21.15.3:0/2913827759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90500b9440 con 0x555e708eea30 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f903f7fe640 1 -- 172.21.15.3:0/2913827759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f90500b95a0 con 0x555e708eea30 2024-02-09T22:53:48.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f90500c5570 con 0x555e708eea30 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f903f7fe640 1 -- 172.21.15.3:0/2913827759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f90500c50e0 con 0x555e708eea30 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 10 client.16488 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x555e70816a70 con 0x555e708eea30 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f903f7fe640 1 -- 172.21.15.3:0/2913827759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f90500c5570 con 0x555e708eea30 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f903f7fe640 1 -- 172.21.15.3:0/2913827759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f905003c070 con 0x555e708eea30 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 10 client.16488 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 10 client.16488 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 20 client.16488 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e70960560 0x555e70962950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 4 client.16488 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.325+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e705e1890 con 0x555e70960560 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e70960560 0x555e70962950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f905f7fe640 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e70960560 0x555e70962950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=860b09fd654faf2b server_cookie=77137ce5e48d8edf in_seq=0 out_seq=0 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f903f7fe640 10 client.16488 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:48.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f903f7fe640 1 -- 172.21.15.3:0/2913827759 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e705e1890 con 0x555e70960560 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f903f7fe640 10 client.16488 handle_command_reply: tid=0 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f9069376740 1 client.16488 shutdown 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 msgr2=0x555e708fe090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.326+0000 7f9069376740 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 0x555e708fe090 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f9050005c40 tx=0x7f9050003920 comp rx=0 tx=0).stop 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 shutdown_connections 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e70960560 0x555e70962950 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e708ef3e0 0x555e708fe5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e708f7f60 0x555e70958880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 --2- 172.21.15.3:0/2913827759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e708eea30 0x555e708fe090 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f903f7fe640 0 client.16488 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 >> 172.21.15.3:0/2913827759 conn(0x555e708e9100 msgr2=0x555e708e79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 shutdown_connections 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 1 -- 172.21.15.3:0/2913827759 wait complete. 2024-02-09T22:53:48.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f9069376740 20 client.16488 trim_cache size 0 max 16384 2024-02-09T22:53:48.337 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f90480a2230 msgr2=0x7f90480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f90480a2230 0x7f90480a46f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f905800a850 tx=0x7f905800ac50 comp rx=0 tx=0).stop 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 msgr2=0x7f906019d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f906019d5d0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f9050005940 tx=0x7f90500042f0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 shutdown_connections 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f90480a2230 0x7f90480a46f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9060105420 0x7f906019db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9060197690 0x7f9060197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 --2- 172.21.15.3:0/2771448383 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f90601003a0 0x7f906019d5d0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 >> 172.21.15.3:0/2771448383 conn(0x7f9060077300 msgr2=0x7f9060069c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 shutdown_connections 2024-02-09T22:53:48.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.327+0000 7f90667a7640 1 -- 172.21.15.3:0/2771448383 wait complete. 2024-02-09T22:53:48.339 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:48 smithi003 ceph-mon[32678]: pgmap v79: 97 pgs: 97 active+clean; 39 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 205 op/s 2024-02-09T22:53:48.340 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:48 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3334818041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:53:48.341 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:53:48.341 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 -- 172.21.15.3:0/2625398586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4978105400 msgr2=0x7f497810cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 --2- 172.21.15.3:0/2625398586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4978105400 0x7f497810cfd0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f4970009cd0 tx=0x7f497002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 -- 172.21.15.3:0/2625398586 shutdown_connections 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 --2- 172.21.15.3:0/2625398586 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4978100ce0 0x7f4978104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 --2- 172.21.15.3:0/2625398586 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4978105400 0x7f497810cfd0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 --2- 172.21.15.3:0/2625398586 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4978100330 0x7f4978100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 -- 172.21.15.3:0/2625398586 >> 172.21.15.3:0/2625398586 conn(0x7f49780fbb90 msgr2=0x7f49780fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 -- 172.21.15.3:0/2625398586 shutdown_connections 2024-02-09T22:53:48.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 -- 172.21.15.3:0/2625398586 wait complete. 2024-02-09T22:53:48.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.442+0000 7f497d10c640 1 Processor -- start 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 -- start start 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4978100330 0x7f497819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4978105400 0x7f497819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 0x7f4978197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978198170 con 0x7f49781976a0 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49781982e0 con 0x7f4978105400 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497d10c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49781984a0 con 0x7f4978100330 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 0x7f4978197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 0x7f4978197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35554/0 (socket says 172.21.15.3:35554) 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 -- 172.21.15.3:0/2572897335 learned_addr learned my addr 172.21.15.3:0/2572897335 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497659c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4978105400 0x7f497819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f4976d9d640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4978100330 0x7f497819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 -- 172.21.15.3:0/2572897335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4978100330 msgr2=0x7f497819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4978100330 0x7f497819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 -- 172.21.15.3:0/2572897335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4978105400 msgr2=0x7f497819db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4978105400 0x7f497819db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.443+0000 7f497759e640 1 -- 172.21.15.3:0/2572897335 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4970009900 con 0x7f49781976a0 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.444+0000 7f497759e640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 0x7f4978197b20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4970004af0 tx=0x7f4970034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.444+0000 7f495ffff640 1 -- 172.21.15.3:0/2572897335 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700328c0 con 0x7f49781976a0 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.444+0000 7f495ffff640 1 -- 172.21.15.3:0/2572897335 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4970003e50 con 0x7f49781976a0 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.444+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49781a43e0 con 0x7f49781976a0 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.444+0000 7f495ffff640 1 -- 172.21.15.3:0/2572897335 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970004860 con 0x7f49781976a0 2024-02-09T22:53:48.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.444+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49781a4860 con 0x7f49781976a0 2024-02-09T22:53:48.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.446+0000 7f495ffff640 1 -- 172.21.15.3:0/2572897335 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f497000a040 con 0x7f49781976a0 2024-02-09T22:53:48.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.446+0000 7f497fdbf740 1 Processor -- start 2024-02-09T22:53:48.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.446+0000 7f497fdbf740 1 -- start start 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.446+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf824e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.446+0000 7f495ffff640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49600a2230 0x7f49600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.446+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8303b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497fdbf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf7e0020 con 0x5647cf82df60 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497fdbf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf793320 con 0x5647cf824a30 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497fdbf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf790850 con 0x5647cf8253e0 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8303b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f4976d9d640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49600a2230 0x7f49600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8303b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35570/0 (socket says 172.21.15.3:35570) 2024-02-09T22:53:48.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 -- 172.21.15.3:0/1783561049 learned_addr learned my addr 172.21.15.3:0/1783561049 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497659c640 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f495ffff640 1 -- 172.21.15.3:0/2572897335 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f49700f0ca0 con 0x7f49781976a0 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497659c640 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf824e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 -- 172.21.15.3:0/1783561049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 msgr2=0x5647cf82d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 -- 172.21.15.3:0/1783561049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 msgr2=0x5647cf824e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf824e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 -- 172.21.15.3:0/1783561049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647cf74e210 con 0x5647cf82df60 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497659c640 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f4976d9d640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49600a2230 0x7f49600a46f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4968009bd0 tx=0x7f4968005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f497759e640 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8303b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f49700ba140 tx=0x7f497002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44536d7f7127320a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f495d7fa640 1 -- 172.21.15.3:0/1783561049 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700bd0d0 con 0x5647cf82df60 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f495d7fa640 1 -- 172.21.15.3:0/1783561049 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49700d6680 con 0x5647cf82df60 2024-02-09T22:53:48.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.447+0000 7f495d7fa640 1 -- 172.21.15.3:0/1783561049 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970004040 con 0x5647cf82df60 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 -- 172.21.15.3:0/1783561049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 msgr2=0x5647cf8303b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8303b0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f49700ba140 tx=0x7f497002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 -- 172.21.15.3:0/1783561049 shutdown_connections 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8303b0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1783561049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf824e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 -- 172.21.15.3:0/1783561049 >> 172.21.15.3:0/1783561049 conn(0x5647cf81f100 msgr2=0x5647cf81f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 -- 172.21.15.3:0/1783561049 shutdown_connections 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 -- 172.21.15.3:0/1783561049 wait complete. 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.448+0000 7f497fdbf740 1 -- start start 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 Processor -- start 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf82aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf82b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf78c030 con 0x5647cf824a30 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf45f430 con 0x5647cf82df60 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf41be00 con 0x5647cf8253e0 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497fdbf740 10 client.0 _ll_register_callbacks cb 0x5647cf781030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf82aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf82aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35572/0 (socket says 172.21.15.3:35572) 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 -- 172.21.15.3:0/1134093050 learned_addr learned my addr 172.21.15.3:0/1134093050 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497659c640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497759e640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf82b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 -- 172.21.15.3:0/1134093050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 msgr2=0x5647cf82b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 -- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 msgr2=0x5647cf82b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf82b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 -- 172.21.15.3:0/1134093050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49781a4860 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f497759e640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf82b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f4976d9d640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf82aeb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4968003970 tx=0x7f4968004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd84053156d5ecb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f495aff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.449+0000 7f495aff5640 1 -- 172.21.15.3:0/1134093050 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496800ecc0 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f495aff5640 1 -- 172.21.15.3:0/1134093050 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f496800ee20 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5647cf48b7e0 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f495aff5640 1 -- 172.21.15.3:0/1134093050 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4968005020 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 10 client.16500 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5647cf78a6b0 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f495aff5640 1 -- 172.21.15.3:0/1134093050 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5647cf48b7e0 con 0x5647cf824a30 2024-02-09T22:53:48.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f495aff5640 1 -- 172.21.15.3:0/1134093050 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f4968024030 con 0x5647cf824a30 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 10 client.16500 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 10 client.16500 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 20 client.16500 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf83e350 0x5647cf840770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 4 client.16500 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5647cf4c8720 con 0x5647cf83e350 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.450+0000 7f497659c640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf83e350 0x5647cf840770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.451+0000 7f495aff5640 10 client.16500 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.451+0000 7f497659c640 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf83e350 0x5647cf840770 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1656838c3a5344ae server_cookie=115d1a2a294d619a in_seq=0 out_seq=0 2024-02-09T22:53:48.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.451+0000 7f495aff5640 1 -- 172.21.15.3:0/1134093050 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5647cf4c8720 con 0x5647cf83e350 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.451+0000 7f495aff5640 10 client.16500 handle_command_reply: tid=0 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.451+0000 7f497fdbf740 1 client.16500 shutdown 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 msgr2=0x5647cf82aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf82aeb0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4968003970 tx=0x7f4968004730 comp rx=0 tx=0).stop 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 shutdown_connections 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf83e350 0x5647cf840770 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf82b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf82b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 --2- 172.21.15.3:0/1134093050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf82aeb0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f495aff5640 0 client.16500 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:48.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 >> 172.21.15.3:0/1134093050 conn(0x5647cf81f100 msgr2=0x5647cf820e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 shutdown_connections 2024-02-09T22:53:48.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 1 -- 172.21.15.3:0/1134093050 wait complete. 2024-02-09T22:53:48.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.452+0000 7f497fdbf740 20 client.16500 trim_cache size 0 max 16384 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.463+0000 7f497fdbf740 1 Processor -- start 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 -- start start 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf82d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf826170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8266b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf711130 con 0x5647cf82df60 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf81fee0 con 0x5647cf824a30 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497fdbf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf83ad60 con 0x5647cf8253e0 2024-02-09T22:53:48.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8266b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8266b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35584/0 (socket says 172.21.15.3:35584) 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 -- 172.21.15.3:0/3775490523 learned_addr learned my addr 172.21.15.3:0/3775490523 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497659c640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf826170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f4976d9d640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf82d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 -- 172.21.15.3:0/3775490523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 msgr2=0x5647cf826170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf826170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 -- 172.21.15.3:0/3775490523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 msgr2=0x5647cf82d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf82d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 -- 172.21.15.3:0/3775490523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5647cf726c90 con 0x5647cf82df60 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497659c640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf826170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f4976d9d640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf82d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f497759e640 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8266b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f49700b9450 tx=0x7f4970004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766cbccee00e428e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f495cff9640 1 -- 172.21.15.3:0/3775490523 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700bd0d0 con 0x5647cf82df60 2024-02-09T22:53:48.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f495cff9640 1 -- 172.21.15.3:0/3775490523 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49700c18c0 con 0x5647cf82df60 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.464+0000 7f495cff9640 1 -- 172.21.15.3:0/3775490523 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700b9de0 con 0x5647cf82df60 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 -- 172.21.15.3:0/3775490523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 msgr2=0x5647cf8266b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8266b0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f49700b9450 tx=0x7f4970004300 comp rx=0 tx=0).stop 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 -- 172.21.15.3:0/3775490523 shutdown_connections 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf82df60 0x5647cf8266b0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf826170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3775490523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf824a30 0x5647cf82d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 -- 172.21.15.3:0/3775490523 >> 172.21.15.3:0/3775490523 conn(0x5647cf81f100 msgr2=0x5647cf81f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 -- 172.21.15.3:0/3775490523 shutdown_connections 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 -- 172.21.15.3:0/3775490523 wait complete. 2024-02-09T22:53:48.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 -- start start 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.465+0000 7f497fdbf740 1 Processor -- start 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf832dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf833300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf893830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf833a00 con 0x5647cf824a30 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf893d70 con 0x5647cf82df60 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5647cf893ee0 con 0x5647cf8253e0 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf832dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 10 client.0 _ll_register_callbacks cb 0x7f4968004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf832dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35590/0 (socket says 172.21.15.3:35590) 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 -- 172.21.15.3:0/3836419948 learned_addr learned my addr 172.21.15.3:0/3836419948 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497759e640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf893830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497659c640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf833300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 -- 172.21.15.3:0/3836419948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 msgr2=0x5647cf833300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf833300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 -- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 msgr2=0x5647cf893830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf893830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 -- 172.21.15.3:0/3836419948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5647cf74e210 con 0x5647cf824a30 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f4976d9d640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf832dc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4968004f70 tx=0x7f4968004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e464cab9990498e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f495aff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f495aff5640 1 -- 172.21.15.3:0/3836419948 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496800eac0 con 0x5647cf824a30 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f495aff5640 1 -- 172.21.15.3:0/3836419948 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f496802a020 con 0x5647cf824a30 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f49700d6680 con 0x5647cf824a30 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.466+0000 7f495aff5640 1 -- 172.21.15.3:0/3836419948 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f496800ec20 con 0x5647cf824a30 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 10 client.16506 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5647cf74ca70 con 0x5647cf824a30 2024-02-09T22:53:48.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f495aff5640 1 -- 172.21.15.3:0/3836419948 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49700d6680 con 0x5647cf824a30 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f495aff5640 1 -- 172.21.15.3:0/3836419948 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f4968024040 con 0x5647cf824a30 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 10 client.16506 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 10 client.16506 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 20 client.16506 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf895700 0x5647cf897b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 4 client.16506 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5647cf517890 con 0x5647cf895700 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497659c640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf895700 0x5647cf897b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f497659c640 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf895700 0x5647cf897b40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca642f02c5073e50 server_cookie=b45e5f7eee285dea in_seq=0 out_seq=0 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.467+0000 7f495aff5640 10 client.16506 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f495aff5640 1 -- 172.21.15.3:0/3836419948 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5647cf517890 con 0x5647cf895700 2024-02-09T22:53:48.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f495aff5640 10 client.16506 handle_command_reply: tid=0 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 client.16506 shutdown 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 msgr2=0x5647cf832dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf832dc0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f4968004f70 tx=0x7f4968004610 comp rx=0 tx=0).stop 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 shutdown_connections 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5647cf895700 0x5647cf897b40 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5647cf82df60 0x5647cf893830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5647cf8253e0 0x5647cf833300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 --2- 172.21.15.3:0/3836419948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5647cf824a30 0x5647cf832dc0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f495aff5640 0 client.16506 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:48.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 >> 172.21.15.3:0/3836419948 conn(0x5647cf81f100 msgr2=0x5647cf826170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 shutdown_connections 2024-02-09T22:53:48.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 1 -- 172.21.15.3:0/3836419948 wait complete. 2024-02-09T22:53:48.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497fdbf740 20 client.16506 trim_cache size 0 max 16384 2024-02-09T22:53:48.481 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49600a2230 msgr2=0x7f49600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49600a2230 0x7f49600a46f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4968009bd0 tx=0x7f4968005f90 comp rx=0 tx=0).stop 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 msgr2=0x7f4978197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 0x7f4978197b20 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f4970004af0 tx=0x7f4970034cc0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 shutdown_connections 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49600a2230 0x7f49600a46f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4978105400 0x7f497819db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f49781976a0 0x7f4978197b20 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 --2- 172.21.15.3:0/2572897335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4978100330 0x7f497819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:48.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.468+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 >> 172.21.15.3:0/2572897335 conn(0x7f49780fbb90 msgr2=0x7f49780fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:48.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.469+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 shutdown_connections 2024-02-09T22:53:48.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:48.469+0000 7f497d10c640 1 -- 172.21.15.3:0/2572897335 wait complete. 2024-02-09T22:53:48.484 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:48 smithi089 ceph-mon[36665]: pgmap v79: 97 pgs: 97 active+clean; 39 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 205 op/s 2024-02-09T22:53:48.485 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:48 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3334818041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:53:48.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68143 3.77 MB/sec warmup 102 sec latency 3353.553 ms 2024-02-09T22:53:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:48 smithi064 ceph-mon[36638]: pgmap v79: 97 pgs: 97 active+clean; 39 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.3 MiB/s wr, 205 op/s 2024-02-09T22:53:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:48 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3334818041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:53:49.488 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:53:49.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:53:49.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 -- 172.21.15.3:0/2788688802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a00ffac0 msgr2=0x7f14a00fff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 --2- 172.21.15.3:0/2788688802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a00ffac0 0x7f14a00fff20 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f1490009900 tx=0x7f149002bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 -- 172.21.15.3:0/2788688802 shutdown_connections 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 --2- 172.21.15.3:0/2788688802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a0100460 0x7f14a010c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 --2- 172.21.15.3:0/2788688802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a00ffac0 0x7f14a00fff20 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 --2- 172.21.15.3:0/2788688802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14a0104830 0x7f14a0104c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 -- 172.21.15.3:0/2788688802 >> 172.21.15.3:0/2788688802 conn(0x7f14a0075350 msgr2=0x7f14a0075760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 -- 172.21.15.3:0/2788688802 shutdown_connections 2024-02-09T22:53:49.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.605+0000 7f14a6681640 1 -- 172.21.15.3:0/2788688802 wait complete. 2024-02-09T22:53:49.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 Processor -- start 2024-02-09T22:53:49.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 -- start start 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a00ffac0 0x7f14a0101520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14a0100460 0x7f14a0101a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 0x7f14a0101fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01a5b70 con 0x7f14a0104830 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01a5ce0 con 0x7f14a00ffac0 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.606+0000 7f14a6681640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a01a5ea0 con 0x7f14a0100460 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 0x7f14a0101fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 0x7f14a0101fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35608/0 (socket says 172.21.15.3:35608) 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 -- 172.21.15.3:0/913180228 learned_addr learned my addr 172.21.15.3:0/913180228 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f149f7fe640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14a0100460 0x7f14a0101a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f149ffff640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a00ffac0 0x7f14a0101520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 -- 172.21.15.3:0/913180228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14a0100460 msgr2=0x7f14a0101a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14a0100460 0x7f14a0101a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 -- 172.21.15.3:0/913180228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a00ffac0 msgr2=0x7f14a0101520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a00ffac0 0x7f14a0101520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 -- 172.21.15.3:0/913180228 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490009560 con 0x7f14a0104830 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f149ffff640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a00ffac0 0x7f14a0101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a4c1e640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 0x7f14a0101fa0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f149800dd70 tx=0x7f149800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f149d7fa640 1 -- 172.21.15.3:0/913180228 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498004d20 con 0x7f14a0104830 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f149d7fa640 1 -- 172.21.15.3:0/913180228 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1498004e80 con 0x7f14a0104830 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a01a6180 con 0x7f14a0104830 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f149d7fa640 1 -- 172.21.15.3:0/913180228 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1498005650 con 0x7f14a0104830 2024-02-09T22:53:49.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.607+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a01a6640 con 0x7f14a0104830 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f149d7fa640 1 -- 172.21.15.3:0/913180228 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f149800ecd0 con 0x7f14a0104830 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 Processor -- start 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 -- start start 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017190 0x560155017570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x560155020150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 0x560155022a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560154f85000 con 0x560155020690 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560154f82530 con 0x560155017ab0 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a924e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560154f7dd10 con 0x560155017190 2024-02-09T22:53:49.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f149d7fa640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f148c0a2230 0x7f148c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f14a4c1e640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f148c0a2230 0x7f148c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 0x560155022a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 0x560155022a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35612/0 (socket says 172.21.15.3:35612) 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.610+0000 7f149f7fe640 1 -- 172.21.15.3:0/1375478201 learned_addr learned my addr 172.21.15.3:0/1375478201 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149d7fa640 1 -- 172.21.15.3:0/913180228 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14980ce6b0 con 0x7f14a0104830 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017190 0x560155017570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149ffff640 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x560155020150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 -- 172.21.15.3:0/1375478201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017190 msgr2=0x560155017570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017190 0x560155017570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 -- 172.21.15.3:0/1375478201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 msgr2=0x560155020150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x560155020150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 -- 172.21.15.3:0/1375478201 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560154f24210 con 0x560155020690 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f14a4c1e640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f148c0a2230 0x7f148c0a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f14980cbff0 tx=0x7f149806d1a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 0x560155022a80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1490005c40 tx=0x7f14900058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb69b416c3b9a7ac server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f14967fc640 1 -- 172.21.15.3:0/1375478201 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490004a00 con 0x560155020690 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f14967fc640 1 -- 172.21.15.3:0/1375478201 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1490004510 con 0x560155020690 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.611+0000 7f14967fc640 1 -- 172.21.15.3:0/1375478201 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490003dc0 con 0x560155020690 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 -- 172.21.15.3:0/1375478201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 msgr2=0x560155022a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 0x560155022a80 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f1490005c40 tx=0x7f14900058b0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 -- 172.21.15.3:0/1375478201 shutdown_connections 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x560155020150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155020690 0x560155022a80 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 --2- 172.21.15.3:0/1375478201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017190 0x560155017570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 -- 172.21.15.3:0/1375478201 >> 172.21.15.3:0/1375478201 conn(0x5601550118c0 msgr2=0x560155011ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 -- 172.21.15.3:0/1375478201 shutdown_connections 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 -- 172.21.15.3:0/1375478201 wait complete. 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.612+0000 7f14a924e740 1 -- start start 2024-02-09T22:53:49.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 Processor -- start 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x56015501dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 0x56015501e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560154f22e60 con 0x560155017190 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560154c514b0 con 0x560155017ab0 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560154f03670 con 0x560155020690 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f14a924e740 10 client.0 _ll_register_callbacks cb 0x560154f1b190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35618/0 (socket says 172.21.15.3:35618) 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 -- 172.21.15.3:0/3137093656 learned_addr learned my addr 172.21.15.3:0/3137093656 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149ffff640 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x56015501dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 -- 172.21.15.3:0/3137093656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 msgr2=0x56015501e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 0x56015501e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 -- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 msgr2=0x56015501dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x56015501dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 -- 172.21.15.3:0/3137093656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a010cfa0 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f149f7fe640 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501d730 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1490005940 tx=0x7f149002c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bac91548e420707 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.613+0000 7f147f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f147f7fe640 1 -- 172.21.15.3:0/3137093656 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490003da0 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f147f7fe640 1 -- 172.21.15.3:0/3137093656 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1490003f00 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560154c7d860 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f147f7fe640 1 -- 172.21.15.3:0/3137093656 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14900040c0 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 10 client.16518 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560154f7c390 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f147f7fe640 1 -- 172.21.15.3:0/3137093656 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560154c7d860 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f147f7fe640 1 -- 172.21.15.3:0/3137093656 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f149004f070 con 0x560155017190 2024-02-09T22:53:49.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 10 client.16518 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 10 client.16518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 20 client.16518 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560155030640 0x560155032a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 4 client.16518 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560154cce410 con 0x560155030640 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.614+0000 7f149ffff640 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560155030640 0x560155032a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.615+0000 7f149ffff640 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560155030640 0x560155032a60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26c4050083395eb5 server_cookie=7b199427ac371214 in_seq=0 out_seq=0 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.615+0000 7f147f7fe640 10 client.16518 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.615+0000 7f147f7fe640 1 -- 172.21.15.3:0/3137093656 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560154cce410 con 0x560155030640 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.615+0000 7f147f7fe640 10 client.16518 handle_command_reply: tid=0 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.615+0000 7f14a924e740 1 client.16518 shutdown 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 msgr2=0x56015501d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501d730 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f1490005940 tx=0x7f149002c8b0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 shutdown_connections 2024-02-09T22:53:49.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560155030640 0x560155032a60 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x56015501dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 0x56015501e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f147f7fe640 0 client.16518 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 --2- 172.21.15.3:0/3137093656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501d730 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 >> 172.21.15.3:0/3137093656 conn(0x5601550118c0 msgr2=0x560155013610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 shutdown_connections 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 1 -- 172.21.15.3:0/3137093656 wait complete. 2024-02-09T22:53:49.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.616+0000 7f14a924e740 20 client.16518 trim_cache size 0 max 16384 2024-02-09T22:53:49.626 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68143 3.73 MB/sec warmup 103 sec latency 1541.801 ms 2024-02-09T22:53:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 Processor -- start 2024-02-09T22:53:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 -- start start 2024-02-09T22:53:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x5601550274e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 0x560155027a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155016930 con 0x560155017190 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601550164b0 con 0x560155017ab0 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f14a924e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155037c30 con 0x560155020690 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35630/0 (socket says 172.21.15.3:35630) 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.631+0000 7f149f7fe640 1 -- 172.21.15.3:0/3453560831 learned_addr learned my addr 172.21.15.3:0/3453560831 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f149f7fe640 1 -- 172.21.15.3:0/3453560831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 msgr2=0x560155027a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f149f7fe640 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 0x560155027a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f149f7fe640 1 -- 172.21.15.3:0/3453560831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 msgr2=0x5601550274e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f149f7fe640 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x5601550274e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f149f7fe640 1 -- 172.21.15.3:0/3453560831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560154f18c70 con 0x560155017190 2024-02-09T22:53:49.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f149f7fe640 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501fe60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1490003870 tx=0x7f1490051680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59b4013a8aaaad05 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f1495ffb640 1 -- 172.21.15.3:0/3453560831 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149002c9e0 con 0x560155017190 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f1495ffb640 1 -- 172.21.15.3:0/3453560831 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1490051800 con 0x560155017190 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f1495ffb640 1 -- 172.21.15.3:0/3453560831 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14900043b0 con 0x560155017190 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f14a924e740 1 -- 172.21.15.3:0/3453560831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 msgr2=0x56015501fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.632+0000 7f14a924e740 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501fe60 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1490003870 tx=0x7f1490051680 comp rx=0 tx=0).stop 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- 172.21.15.3:0/3453560831 shutdown_connections 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155017ab0 0x5601550274e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155020690 0x560155027a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 --2- 172.21.15.3:0/3453560831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x56015501fe60 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- 172.21.15.3:0/3453560831 >> 172.21.15.3:0/3453560831 conn(0x5601550118c0 msgr2=0x560155010090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- 172.21.15.3:0/3453560831 shutdown_connections 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- 172.21.15.3:0/3453560831 wait complete. 2024-02-09T22:53:49.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- start start 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 Processor -- start 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x560155026870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017ab0 0x560155026db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155020690 0x560155080c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155081170 con 0x560155017190 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601550812e0 con 0x560155020690 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601550814a0 con 0x560155017ab0 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f14a924e740 10 client.0 _ll_register_callbacks cb 0x7f1490009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x560155026870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f149f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x560155026870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35642/0 (socket says 172.21.15.3:35642) 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.633+0000 7f149f7fe640 1 -- 172.21.15.3:0/1824130996 learned_addr learned my addr 172.21.15.3:0/1824130996 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149ffff640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017ab0 0x560155026db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155020690 0x560155080c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 -- 172.21.15.3:0/1824130996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017ab0 msgr2=0x560155026db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017ab0 0x560155026db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 -- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155020690 msgr2=0x560155080c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155020690 0x560155080c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 -- 172.21.15.3:0/1824130996 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560154f7c390 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f149f7fe640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x560155026870 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1490009850 tx=0x7f1490003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9439bfe465e5c93 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f147f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f147f7fe640 1 -- 172.21.15.3:0/1824130996 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149002c650 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f147f7fe640 1 -- 172.21.15.3:0/1824130996 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1490000e40 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1490004510 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f147f7fe640 1 -- 172.21.15.3:0/1824130996 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1490004340 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f14a924e740 10 client.16524 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.634+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x560154f3eac0 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f147f7fe640 1 -- 172.21.15.3:0/1824130996 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1490004510 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f147f7fe640 1 -- 172.21.15.3:0/1824130996 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f149006b070 con 0x560155017190 2024-02-09T22:53:49.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f14a924e740 10 client.16524 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f14a924e740 10 client.16524 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f14a924e740 20 client.16524 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f14a924e740 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5601550888c0 0x56015508ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f14a924e740 4 client.16524 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560154cba7b0 con 0x5601550888c0 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f149ffff640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5601550888c0 0x56015508ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f147f7fe640 10 client.16524 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.635+0000 7f149ffff640 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5601550888c0 0x56015508ace0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14056d6a908f7679 server_cookie=3fa079f202cb1c2a in_seq=0 out_seq=0 2024-02-09T22:53:49.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.636+0000 7f147f7fe640 1 -- 172.21.15.3:0/1824130996 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560154cba7b0 con 0x5601550888c0 2024-02-09T22:53:49.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.636+0000 7f147f7fe640 10 client.16524 handle_command_reply: tid=0 2024-02-09T22:53:49.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.636+0000 7f14a924e740 1 client.16524 shutdown 2024-02-09T22:53:49.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 msgr2=0x560155026870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x560155026870 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1490009850 tx=0x7f1490003f90 comp rx=0 tx=0).stop 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 shutdown_connections 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5601550888c0 0x56015508ace0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f147f7fe640 0 client.16524 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560155020690 0x560155080c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560155017ab0 0x560155026db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 --2- 172.21.15.3:0/1824130996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560155017190 0x560155026870 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 >> 172.21.15.3:0/1824130996 conn(0x5601550118c0 msgr2=0x560155010920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 shutdown_connections 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 1 -- 172.21.15.3:0/1824130996 wait complete. 2024-02-09T22:53:49.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a924e740 20 client.16524 trim_cache size 0 max 16384 2024-02-09T22:53:49.646 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:53:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:53:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "cf38b86c-d958-45a0-adaa-80b0c5de9e30", 2024-02-09T22:53:49.646 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:53:49.646 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f148c0a2230 msgr2=0x7f148c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f148c0a2230 0x7f148c0a46f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f14980cbff0 tx=0x7f149806d1a0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 msgr2=0x7f14a0101fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 0x7f14a0101fa0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f149800dd70 tx=0x7f149800b730 comp rx=0 tx=0).stop 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 shutdown_connections 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f148c0a2230 0x7f148c0a46f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14a0104830 0x7f14a0101fa0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14a0100460 0x7f14a0101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 --2- 172.21.15.3:0/913180228 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14a00ffac0 0x7f14a0101520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.637+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 >> 172.21.15.3:0/913180228 conn(0x7f14a0075350 msgr2=0x7f14a00fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.638+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 shutdown_connections 2024-02-09T22:53:49.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.638+0000 7f14a6681640 1 -- 172.21.15.3:0/913180228 wait complete. 2024-02-09T22:53:49.652 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:53:49.652 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.756+0000 7f96b0435640 1 -- 172.21.15.3:0/1503622485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a8101dd0 msgr2=0x7f96a8105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.756+0000 7f96b0435640 1 --2- 172.21.15.3:0/1503622485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a8101dd0 0x7f96a8105e10 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f9698009930 tx=0x7f969802bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- 172.21.15.3:0/1503622485 shutdown_connections 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 --2- 172.21.15.3:0/1503622485 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a81064f0 0x7f96a810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 --2- 172.21.15.3:0/1503622485 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a8101dd0 0x7f96a8105e10 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 --2- 172.21.15.3:0/1503622485 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96a8101420 0x7f96a8101800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- 172.21.15.3:0/1503622485 >> 172.21.15.3:0/1503622485 conn(0x7f96a80fcca0 msgr2=0x7f96a80ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- 172.21.15.3:0/1503622485 shutdown_connections 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- 172.21.15.3:0/1503622485 wait complete. 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 Processor -- start 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- start start 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a8101420 0x7f96a819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96a81064f0 0x7f96a819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 0x7f96a8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a81981c0 con 0x7f96a81976f0 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a8198330 con 0x7f96a8101420 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96b0435640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96a81984f0 con 0x7f96a81064f0 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96ae9d2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 0x7f96a8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96ae9d2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 0x7f96a8197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35674/0 (socket says 172.21.15.3:35674) 2024-02-09T22:53:49.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.757+0000 7f96ae9d2640 1 -- 172.21.15.3:0/988046298 learned_addr learned my addr 172.21.15.3:0/988046298 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96a81064f0 0x7f96a819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a8101420 0x7f96a819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae9d2640 1 -- 172.21.15.3:0/988046298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96a81064f0 msgr2=0x7f96a819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae9d2640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96a81064f0 0x7f96a819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae9d2640 1 -- 172.21.15.3:0/988046298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a8101420 msgr2=0x7f96a819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae9d2640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a8101420 0x7f96a819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae9d2640 1 -- 172.21.15.3:0/988046298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9698009560 con 0x7f96a81976f0 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a8101420 0x7f96a819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96ae9d2640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 0x7f96a8197b70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f96a400eaa0 tx=0x7f96a400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f969f7fe640 1 -- 172.21.15.3:0/988046298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a400cdb0 con 0x7f96a81976f0 2024-02-09T22:53:49.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f969f7fe640 1 -- 172.21.15.3:0/988046298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a4005020 con 0x7f96a81976f0 2024-02-09T22:53:49.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a81a4490 con 0x7f96a81976f0 2024-02-09T22:53:49.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f969f7fe640 1 -- 172.21.15.3:0/988046298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a40116b0 con 0x7f96a81976f0 2024-02-09T22:53:49.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.758+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a81a4920 con 0x7f96a81976f0 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.760+0000 7f969f7fe640 1 -- 172.21.15.3:0/988046298 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f96a4005520 con 0x7f96a81976f0 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 Processor -- start 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 -- start start 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c2980 0x55e5b21c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 0x55e5b21cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21ce300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b217dd00 con 0x55e5b21c3330 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b21322f0 con 0x55e5b21cbeb0 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96b3008740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b212f820 con 0x55e5b21c2980 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f969f7fe640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f96940a2230 0x7f96940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96ae1d1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 0x55e5b21cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96ae1d1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 0x55e5b21cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35678/0 (socket says 172.21.15.3:35678) 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96ae1d1640 1 -- 172.21.15.3:0/3630676984 learned_addr learned my addr 172.21.15.3:0/3630676984 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21ce300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96ae9d2640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f96940a2230 0x7f96940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.761+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c2980 0x55e5b21c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f969f7fe640 1 -- 172.21.15.3:0/988046298 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f96a40ce290 con 0x7f96a81976f0 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae1d1640 1 -- 172.21.15.3:0/3630676984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c2980 msgr2=0x55e5b21c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c2980 0x55e5b21c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae1d1640 1 -- 172.21.15.3:0/3630676984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 msgr2=0x55e5b21ce300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21ce300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae1d1640 1 -- 172.21.15.3:0/3630676984 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5b20ecff0 con 0x55e5b21c3330 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c2980 0x55e5b21c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 0x55e5b21cb8e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009d40 tx=0x7f96a00117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2b5c9fcaeeb5cfd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f96ae9d2640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f96940a2230 0x7f96940a46f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f96a40cbac0 tx=0x7f96a4090380 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f969cbfe640 1 -- 172.21.15.3:0/3630676984 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a0012650 con 0x55e5b21c3330 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f969cbfe640 1 -- 172.21.15.3:0/3630676984 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a0009e30 con 0x55e5b21c3330 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.762+0000 7f969cbfe640 1 -- 172.21.15.3:0/3630676984 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96a000fce0 con 0x55e5b21c3330 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- 172.21.15.3:0/3630676984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 msgr2=0x55e5b21cb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 0x55e5b21cb8e0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f96a0009d40 tx=0x7f96a00117b0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- 172.21.15.3:0/3630676984 shutdown_connections 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21ce300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c3330 0x55e5b21cb8e0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- 172.21.15.3:0/3630676984 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c2980 0x55e5b21c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- 172.21.15.3:0/3630676984 >> 172.21.15.3:0/3630676984 conn(0x55e5b21bd050 msgr2=0x55e5b21bd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- 172.21.15.3:0/3630676984 shutdown_connections 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- 172.21.15.3:0/3630676984 wait complete. 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- start start 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 Processor -- start 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21c8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21c3330 0x55e5b21c9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21cbeb0 0x55e5b21c9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b212b000 con 0x55e5b21c2980 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b1dfe430 con 0x55e5b21c3330 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.763+0000 7f96b3008740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b1dbae00 con 0x55e5b21cbeb0 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96b3008740 10 client.0 _ll_register_callbacks cb 0x55e5b211f7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21c8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21c8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35684/0 (socket says 172.21.15.3:35684) 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2082528611 learned_addr learned my addr 172.21.15.3:0/2082528611 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21c3330 0x55e5b21c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2082528611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21cbeb0 msgr2=0x55e5b21c9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21cbeb0 0x55e5b21c9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21c3330 msgr2=0x55e5b21c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21c3330 0x55e5b21c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2082528611 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a0009970 con 0x55e5b21c2980 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21c3330 0x55e5b21c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21c8e00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9698005ab0 tx=0x7f96980092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20f2fd90ddfee25d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f9691ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f9691ffb640 1 -- 172.21.15.3:0/2082528611 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f969802cc20 con 0x55e5b21c2980 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f9691ffb640 1 -- 172.21.15.3:0/2082528611 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9698047070 con 0x55e5b21c2980 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e5b1e2a7e0 con 0x55e5b21c2980 2024-02-09T22:53:49.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f9691ffb640 1 -- 172.21.15.3:0/2082528611 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9698004040 con 0x55e5b21c2980 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96b3008740 10 client.16536 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e5b2129680 con 0x55e5b21c2980 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.764+0000 7f9691ffb640 1 -- 172.21.15.3:0/2082528611 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e5b1e2a7e0 con 0x55e5b21c2980 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f9691ffb640 1 -- 172.21.15.3:0/2082528611 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f96980035f0 con 0x55e5b21c2980 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96b3008740 10 client.16536 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96b3008740 10 client.16536 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96b3008740 20 client.16536 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96b3008740 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b21dc380 0x55e5b21de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96b3008740 4 client.16536 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e5b1e67720 con 0x55e5b21dc380 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b21dc380 0x55e5b21de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f9691ffb640 10 client.16536 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.765+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b21dc380 0x55e5b21de7a0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd34d9203c663ac2 server_cookie=cccbbf6af5dc787d in_seq=0 out_seq=0 2024-02-09T22:53:49.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f9691ffb640 1 -- 172.21.15.3:0/2082528611 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e5b1e67720 con 0x55e5b21dc380 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f9691ffb640 10 client.16536 handle_command_reply: tid=0 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 client.16536 shutdown 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 msgr2=0x55e5b21c8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21c8e00 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f9698005ab0 tx=0x7f96980092f0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 shutdown_connections 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b21dc380 0x55e5b21de7a0 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21c3330 0x55e5b21c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21cbeb0 0x55e5b21c9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 --2- 172.21.15.3:0/2082528611 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21c8e00 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f9691ffb640 0 client.16536 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 >> 172.21.15.3:0/2082528611 conn(0x55e5b21bd050 msgr2=0x55e5b21beda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 shutdown_connections 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 1 -- 172.21.15.3:0/2082528611 wait complete. 2024-02-09T22:53:49.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.766+0000 7f96b3008740 20 client.16536 trim_cache size 0 max 16384 2024-02-09T22:53:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 Processor -- start 2024-02-09T22:53:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 -- start start 2024-02-09T22:53:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21cb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21c6080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b20affd0 con 0x55e5b21c2980 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b21bdaa0 con 0x55e5b21cbeb0 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96b3008740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b21c22d0 con 0x55e5b21c3330 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21cb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21cb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35686/0 (socket says 172.21.15.3:35686) 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2826024680 learned_addr learned my addr 172.21.15.3:0/2826024680 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2826024680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 msgr2=0x55e5b21c96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2826024680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 msgr2=0x55e5b21c6080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 -- 172.21.15.3:0/2826024680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e5b20c5b30 con 0x55e5b21c2980 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21cb530 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f969802c170 tx=0x7f9698005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc63ebd2b692447d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f9693fff640 1 -- 172.21.15.3:0/2826024680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9698003ca0 con 0x55e5b21c2980 2024-02-09T22:53:49.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f9693fff640 1 -- 172.21.15.3:0/2826024680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96980044e0 con 0x55e5b21c2980 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.782+0000 7f9693fff640 1 -- 172.21.15.3:0/2826024680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96980052b0 con 0x55e5b21c2980 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 -- 172.21.15.3:0/2826024680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 msgr2=0x55e5b21cb530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21cb530 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f969802c170 tx=0x7f9698005cb0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 -- 172.21.15.3:0/2826024680 shutdown_connections 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b21c6080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21c96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 --2- 172.21.15.3:0/2826024680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21cb530 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 -- 172.21.15.3:0/2826024680 >> 172.21.15.3:0/2826024680 conn(0x55e5b21bd050 msgr2=0x55e5b21e30e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 -- 172.21.15.3:0/2826024680 shutdown_connections 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 -- 172.21.15.3:0/2826024680 wait complete. 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 -- start start 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.783+0000 7f96b3008740 1 Processor -- start 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21d1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21d21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b2232870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b21d2940 con 0x55e5b21c2980 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b21d2a80 con 0x55e5b21cbeb0 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e5b2232db0 con 0x55e5b21c3330 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 10 client.0 _ll_register_callbacks cb 0x7f96a00096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21d1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21d1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35702/0 (socket says 172.21.15.3:35702) 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 -- 172.21.15.3:0/3951572603 learned_addr learned my addr 172.21.15.3:0/3951572603 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b2232870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21d21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 -- 172.21.15.3:0/3951572603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 msgr2=0x55e5b21d21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21d21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 -- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 msgr2=0x55e5b2232870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b2232870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 -- 172.21.15.3:0/3951572603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e5b2129680 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96ad9d0640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21d1c70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f969802ce00 tx=0x7f9698004e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab390fb7b18e7d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f9691ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f9691ffb640 1 -- 172.21.15.3:0/3951572603 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9698004940 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f9691ffb640 1 -- 172.21.15.3:0/3951572603 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9698047040 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f96a0009e30 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f9691ffb640 1 -- 172.21.15.3:0/3951572603 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9698004000 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 10 client.16542 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.784+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55e5b20eb850 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f9691ffb640 1 -- 172.21.15.3:0/3951572603 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96a0009e30 con 0x55e5b21c2980 2024-02-09T22:53:49.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f9691ffb640 1 -- 172.21.15.3:0/3951572603 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f969804c510 con 0x55e5b21c2980 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96b3008740 10 client.16542 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96b3008740 10 client.16542 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96b3008740 20 client.16542 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96b3008740 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b22341e0 0x55e5b2236600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96b3008740 4 client.16542 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e5b1eb6890 con 0x55e5b22341e0 2024-02-09T22:53:49.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b22341e0 0x55e5b2236600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f9691ffb640 10 client.16542 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.785+0000 7f96ae1d1640 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b22341e0 0x55e5b2236600 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d2e9389c4f8b0aa server_cookie=7123355ab87c77f4 in_seq=0 out_seq=0 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f9691ffb640 1 -- 172.21.15.3:0/3951572603 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55e5b1eb6890 con 0x55e5b22341e0 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f9691ffb640 10 client.16542 handle_command_reply: tid=0 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 client.16542 shutdown 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 msgr2=0x55e5b21d1c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21d1c70 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f969802ce00 tx=0x7f9698004e90 comp rx=0 tx=0).stop 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 shutdown_connections 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e5b22341e0 0x55e5b2236600 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e5b21cbeb0 0x55e5b2232870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e5b21c3330 0x55e5b21d21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 --2- 172.21.15.3:0/3951572603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e5b21c2980 0x55e5b21d1c70 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f9691ffb640 0 client.16542 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 >> 172.21.15.3:0/3951572603 conn(0x55e5b21bd050 msgr2=0x55e5b21bb960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 shutdown_connections 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 1 -- 172.21.15.3:0/3951572603 wait complete. 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b3008740 20 client.16542 trim_cache size 0 max 16384 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "cf38b86c-d958-45a0-adaa-80b0c5de9e30": { 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:53:49.797 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "cf38b86c-d958-45a0-adaa-80b0c5de9e30", 2024-02-09T22:53:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:53:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:53:49.798 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:53:49.798 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:53:49.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f96940a2230 msgr2=0x7f96940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f96940a2230 0x7f96940a46f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f96a40cbac0 tx=0x7f96a4090380 comp rx=0 tx=0).stop 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 msgr2=0x7f96a8197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 0x7f96a8197b70 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f96a400eaa0 tx=0x7f96a400ef60 comp rx=0 tx=0).stop 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 shutdown_connections 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f96940a2230 0x7f96940a46f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f96a81976f0 0x7f96a8197b70 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f96a81064f0 0x7f96a819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 --2- 172.21.15.3:0/988046298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f96a8101420 0x7f96a819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.786+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 >> 172.21.15.3:0/988046298 conn(0x7f96a80fcca0 msgr2=0x7f96a80fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.787+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 shutdown_connections 2024-02-09T22:53:49.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:53:49.787+0000 7f96b0435640 1 -- 172.21.15.3:0/988046298 wait complete. 2024-02-09T22:53:49.800 INFO:tasks.cephfs.filesystem:scrub status for tag:cf38b86c-d958-45a0-adaa-80b0c5de9e30 - {'path': '/', 'tag': 'cf38b86c-d958-45a0-adaa-80b0c5de9e30', 'options': 'recursive,force'} 2024-02-09T22:53:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: pgmap v80: 97 pgs: 97 active+clean; 39 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 834 KiB/s wr, 134 op/s 2024-02-09T22:53:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:53:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:50 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68211 3.70 MB/sec warmup 104 sec latency 1591.790 ms 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: pgmap v80: 97 pgs: 97 active+clean; 39 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 834 KiB/s wr, 134 op/s 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:50 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: pgmap v80: 97 pgs: 97 active+clean; 39 MiB data, 608 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 834 KiB/s wr, 134 op/s 2024-02-09T22:53:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:50.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:50 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:51 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:53:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:51 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:51.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:51 smithi089 ceph-mon[36665]: pgmap v81: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-02-09T22:53:51.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68211 3.66 MB/sec warmup 105 sec latency 1908.902 ms 2024-02-09T22:53:51.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:51 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:53:51.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:51 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:51.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:51 smithi003 ceph-mon[32678]: pgmap v81: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-02-09T22:53:51.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:51 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:53:51.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:51 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:53:51.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:51 smithi064 ceph-mon[36638]: pgmap v81: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.1 MiB/s wr, 163 op/s 2024-02-09T22:53:52.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 68211 3.63 MB/sec warmup 106 sec latency 2909.012 ms 2024-02-09T22:53:53.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 69861 3.66 MB/sec warmup 107 sec latency 3325.481 ms 2024-02-09T22:53:53.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:53 smithi003 ceph-mon[32678]: pgmap v82: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 756 KiB/s wr, 94 op/s 2024-02-09T22:53:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:53 smithi064 ceph-mon[36638]: pgmap v82: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 756 KiB/s wr, 94 op/s 2024-02-09T22:53:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:53 smithi089 ceph-mon[36665]: pgmap v82: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 756 KiB/s wr, 94 op/s 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 -- 172.21.15.3:0/4089955791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100380 msgr2=0x7f868c100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 --2- 172.21.15.3:0/4089955791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100380 0x7f868c100800 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f8674009930 tx=0x7f867402bc30 comp rx=0 tx=0).stop 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 -- 172.21.15.3:0/4089955791 shutdown_connections 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 --2- 172.21.15.3:0/4089955791 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100d40 0x7f868c10cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 --2- 172.21.15.3:0/4089955791 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100380 0x7f868c100800 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 --2- 172.21.15.3:0/4089955791 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f868c1062c0 0x7f868c1066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 -- 172.21.15.3:0/4089955791 >> 172.21.15.3:0/4089955791 conn(0x7f868c076ef0 msgr2=0x7f868c077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:54.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 -- 172.21.15.3:0/4089955791 shutdown_connections 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 -- 172.21.15.3:0/4089955791 wait complete. 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 Processor -- start 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.555+0000 7f8691363640 1 -- start start 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100380 0x7f868c196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 0x7f868c196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f868c1062c0 0x7f868c19afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c19b5c0 con 0x7f868c100d40 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c19b730 con 0x7f868c100380 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868c19b8f0 con 0x7f868c1062c0 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100380 0x7f868c196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 0x7f868c196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 0x7f868c196c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35716/0 (socket says 172.21.15.3:35716) 2024-02-09T22:53:54.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 -- 172.21.15.3:0/1954936913 learned_addr learned my addr 172.21.15.3:0/1954936913 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100380 0x7f868c196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43342/0 (socket says 172.21.15.3:43342) 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868b7fe640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f868c1062c0 0x7f868c19afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 -- 172.21.15.3:0/1954936913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f868c1062c0 msgr2=0x7f868c19afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f868c1062c0 0x7f868c19afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 -- 172.21.15.3:0/1954936913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100380 msgr2=0x7f868c196700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100380 0x7f868c196700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 -- 172.21.15.3:0/1954936913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8674009560 con 0x7f868c100d40 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f868a7fc640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 0x7f868c196c40 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8674009230 tx=0x7f867402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8683fff640 1 -- 172.21.15.3:0/1954936913 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674004350 con 0x7f868c100d40 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8683fff640 1 -- 172.21.15.3:0/1954936913 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86740044b0 con 0x7f868c100d40 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868c19bb70 con 0x7f868c100d40 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8683fff640 1 -- 172.21.15.3:0/1954936913 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674003d70 con 0x7f868c100d40 2024-02-09T22:53:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.556+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c069a20 con 0x7f868c100d40 2024-02-09T22:53:54.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.559+0000 7f8683fff640 1 -- 172.21.15.3:0/1954936913 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8674004df0 con 0x7f868c100d40 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.559+0000 7f869400f740 1 Processor -- start 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.559+0000 7f869400f740 1 -- start start 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f8683fff640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86780a22a0 0x7f86780a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f8683fff640 1 -- 172.21.15.3:0/1954936913 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f86740effe0 con 0x7f868c100d40 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868b7fe640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86780a22a0 0x7f86780a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f869400f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd75a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f869400f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd763260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f869400f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd765b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f869400f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd6c8a00 con 0x55eedd75a0b0 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f869400f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd6c5f30 con 0x55eedd75a9d0 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f869400f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd6c1710 con 0x55eedd7637a0 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd75a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd75a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35730/0 (socket says 172.21.15.3:35730) 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 -- 172.21.15.3:0/3513763813 learned_addr learned my addr 172.21.15.3:0/3513763813 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868b7fe640 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd765b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868a7fc640 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd763260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868b7fe640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86780a22a0 0x7f86780a4760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f868400ab90 tx=0x7f868400af90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 -- 172.21.15.3:0/3513763813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 msgr2=0x55eedd765b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd765b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 -- 172.21.15.3:0/3513763813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 msgr2=0x55eedd763260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd763260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868affd640 1 -- 172.21.15.3:0/3513763813 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eedd6863b0 con 0x55eedd75a0b0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.560+0000 7f868b7fe640 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd765b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.561+0000 7f868affd640 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd75a490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f867c009fd0 tx=0x7f867c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1073964fe789caf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.561+0000 7f86817fa640 1 -- 172.21.15.3:0/3513763813 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c013050 con 0x55eedd75a0b0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.561+0000 7f86817fa640 1 -- 172.21.15.3:0/3513763813 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f867c014040 con 0x55eedd75a0b0 2024-02-09T22:53:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.561+0000 7f86817fa640 1 -- 172.21.15.3:0/3513763813 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c018940 con 0x55eedd75a0b0 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.561+0000 7f869400f740 1 -- 172.21.15.3:0/3513763813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 msgr2=0x55eedd75a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.561+0000 7f869400f740 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd75a490 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f867c009fd0 tx=0x7f867c009500 comp rx=0 tx=0).stop 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- 172.21.15.3:0/3513763813 shutdown_connections 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd763260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd765b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 --2- 172.21.15.3:0/3513763813 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd75a490 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- 172.21.15.3:0/3513763813 >> 172.21.15.3:0/3513763813 conn(0x55eedd7549d0 msgr2=0x55eedd754db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- 172.21.15.3:0/3513763813 shutdown_connections 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- 172.21.15.3:0/3513763813 wait complete. 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- start start 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 Processor -- start 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd7607c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd75a9d0 0x55eedd760d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd7637a0 0x55eedd761240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd666b90 con 0x55eedd75a0b0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd395250 con 0x55eedd7637a0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd647200 con 0x55eedd75a9d0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f869400f740 10 client.0 _ll_register_callbacks cb 0x55eedd67d5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd7607c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd7607c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35742/0 (socket says 172.21.15.3:35742) 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f868affd640 1 -- 172.21.15.3:0/1448644083 learned_addr learned my addr 172.21.15.3:0/1448644083 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f868a7fc640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd75a9d0 0x55eedd760d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.562+0000 7f868b7fe640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd7637a0 0x55eedd761240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f868affd640 1 -- 172.21.15.3:0/1448644083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd75a9d0 msgr2=0x55eedd760d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f868affd640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd75a9d0 0x55eedd760d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f868affd640 1 -- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd7637a0 msgr2=0x55eedd761240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f868affd640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd7637a0 0x55eedd761240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f868affd640 1 -- 172.21.15.3:0/1448644083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867c009c80 con 0x55eedd75a0b0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f868affd640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd7607c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f867c00ee00 tx=0x7f867c00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10bf1be6a285da5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f86727fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f86727fc640 1 -- 172.21.15.3:0/1448644083 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c009220 con 0x55eedd75a0b0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eedd6e7ee0 con 0x55eedd75a0b0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f86727fc640 1 -- 172.21.15.3:0/1448644083 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f867c00be10 con 0x55eedd75a0b0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f86727fc640 1 -- 172.21.15.3:0/1448644083 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f867c013020 con 0x55eedd75a0b0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f869400f740 10 client.16554 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55eedd6bfd90 con 0x55eedd75a0b0 2024-02-09T22:53:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.563+0000 7f86727fc640 1 -- 172.21.15.3:0/1448644083 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55eedd6e7ee0 con 0x55eedd75a0b0 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f86727fc640 1 -- 172.21.15.3:0/1448644083 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f867c027030 con 0x55eedd75a0b0 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f869400f740 10 client.16554 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f869400f740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f869400f740 20 client.16554 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f869400f740 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd777c60 0x55eedd77a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f869400f740 4 client.16554 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eedd3c1600 con 0x55eedd777c60 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f868a7fc640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd777c60 0x55eedd77a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f868a7fc640 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd777c60 0x55eedd77a0a0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3d72400b3a00304 server_cookie=bb722c949bd3ef89 in_seq=0 out_seq=0 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.564+0000 7f86727fc640 10 client.16554 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f86727fc640 1 -- 172.21.15.3:0/1448644083 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55eedd3c1600 con 0x55eedd777c60 2024-02-09T22:53:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f86727fc640 10 client.16554 handle_command_reply: tid=0 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 client.16554 shutdown 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 msgr2=0x55eedd7607c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd7607c0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f867c00ee00 tx=0x7f867c00b7b0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 shutdown_connections 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd777c60 0x55eedd77a0a0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd7637a0 0x55eedd761240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f86727fc640 0 client.16554 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd75a9d0 0x55eedd760d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 --2- 172.21.15.3:0/1448644083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd7607c0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 >> 172.21.15.3:0/1448644083 conn(0x55eedd7549d0 msgr2=0x55eedd756720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 shutdown_connections 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 1 -- 172.21.15.3:0/1448644083 wait complete. 2024-02-09T22:53:54.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.565+0000 7f869400f740 20 client.16554 trim_cache size 0 max 16384 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 Processor -- start 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 -- start start 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd762ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd760d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd761250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd7748a0 con 0x55eedd75a0b0 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd754500 con 0x55eedd75a9d0 2024-02-09T22:53:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f869400f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd75b5a0 con 0x55eedd7637a0 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd762ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd761250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868a7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd760d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd761250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:39556/0 (socket says 172.21.15.3:39556) 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd762ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35754/0 (socket says 172.21.15.3:35754) 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868b7fe640 1 -- 172.21.15.3:0/3323255694 learned_addr learned my addr 172.21.15.3:0/3323255694 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.579+0000 7f868a7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd760d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:43372/0 (socket says 172.21.15.3:43372) 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f868a7fc640 1 -- 172.21.15.3:0/3323255694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 msgr2=0x55eedd761250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f868a7fc640 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd761250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f868a7fc640 1 -- 172.21.15.3:0/3323255694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 msgr2=0x55eedd762ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f868a7fc640 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd762ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f868a7fc640 1 -- 172.21.15.3:0/3323255694 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eedd65c820 con 0x55eedd75a9d0 2024-02-09T22:53:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f868a7fc640 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd760d10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f86740ed810 tx=0x7f86740bf040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c35fff1c999095 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f8680ff9640 1 -- 172.21.15.3:0/3323255694 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740035f0 con 0x55eedd75a9d0 2024-02-09T22:53:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f8680ff9640 1 -- 172.21.15.3:0/3323255694 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f86740bab50 con 0x55eedd75a9d0 2024-02-09T22:53:54.585 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.580+0000 7f8680ff9640 1 -- 172.21.15.3:0/3323255694 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86740c4dc0 con 0x55eedd75a9d0 2024-02-09T22:53:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- 172.21.15.3:0/3323255694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 msgr2=0x55eedd760d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd760d10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f86740ed810 tx=0x7f86740bf040 comp rx=0 tx=0).stop 2024-02-09T22:53:54.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- 172.21.15.3:0/3323255694 shutdown_connections 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd760d10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd761250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- 172.21.15.3:0/3323255694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd762ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- 172.21.15.3:0/3323255694 >> 172.21.15.3:0/3323255694 conn(0x55eedd7549d0 msgr2=0x55eedd75ba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- 172.21.15.3:0/3323255694 shutdown_connections 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- 172.21.15.3:0/3323255694 wait complete. 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- start start 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 Processor -- start 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd76a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd76a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd768620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd768c60 con 0x55eedd75a0b0 2024-02-09T22:53:54.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd768dd0 con 0x55eedd75a9d0 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eedd768f90 con 0x55eedd7637a0 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.581+0000 7f869400f740 10 client.0 _ll_register_callbacks cb 0x7f8674003ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd76a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd76a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35762/0 (socket says 172.21.15.3:35762) 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 -- 172.21.15.3:0/2524522167 learned_addr learned my addr 172.21.15.3:0/2524522167 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868b7fe640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd768620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868a7fc640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd76a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 -- 172.21.15.3:0/2524522167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 msgr2=0x55eedd768620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd768620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 -- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 msgr2=0x55eedd76a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd76a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 -- 172.21.15.3:0/2524522167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f868c069a20 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f868affd640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd76a160 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f867c00ee00 tx=0x7f867c005a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e666ec6748922e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f86727fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f86727fc640 1 -- 172.21.15.3:0/2524522167 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f867c013040 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f86727fc640 1 -- 172.21.15.3:0/2524522167 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f867c02d030 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f867c014040 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f86727fc640 1 -- 172.21.15.3:0/2524522167 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f867c00b820 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f869400f740 10 client.16560 fetch_fsmap learned FSMap version 31 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55eedd6826f0 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.582+0000 7f86727fc640 1 -- 172.21.15.3:0/2524522167 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f867c014040 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f86727fc640 1 -- 172.21.15.3:0/2524522167 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f867c027070 con 0x55eedd75a0b0 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f869400f740 10 client.16560 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f869400f740 10 client.16560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f869400f740 20 client.16560 populate_metadata read hostname 'smithi003' 2024-02-09T22:53:54.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f869400f740 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd7cf010 0x55eedd7d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:53:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f869400f740 4 client.16560 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:53:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55eedd4121b0 con 0x55eedd7cf010 2024-02-09T22:53:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f868a7fc640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd7cf010 0x55eedd7d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:53:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f868a7fc640 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd7cf010 0x55eedd7d1430 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a0c44038e5a325a server_cookie=28c0357ea2efdf2a in_seq=0 out_seq=0 2024-02-09T22:53:54.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.583+0000 7f86727fc640 10 client.16560 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.606+0000 7f86727fc640 1 -- 172.21.15.3:0/2524522167 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55eedd4121b0 con 0x55eedd7cf010 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.606+0000 7f86727fc640 10 client.16560 handle_command_reply: tid=0 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.606+0000 7f869400f740 1 client.16560 shutdown 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 msgr2=0x55eedd76a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd76a160 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f867c00ee00 tx=0x7f867c005a40 comp rx=0 tx=0).stop 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 shutdown_connections 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55eedd7cf010 0x55eedd7d1430 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55eedd75a9d0 0x55eedd76a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55eedd7637a0 0x55eedd768620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f86727fc640 0 client.16560 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 --2- 172.21.15.3:0/2524522167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55eedd75a0b0 0x55eedd76a160 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 >> 172.21.15.3:0/2524522167 conn(0x55eedd7549d0 msgr2=0x55eedd75f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 shutdown_connections 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 1 -- 172.21.15.3:0/2524522167 wait complete. 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f869400f740 20 client.16560 trim_cache size 0 max 16384 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:53:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:53:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:53:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86780a22a0 msgr2=0x7f86780a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86780a22a0 0x7f86780a4760 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f868400ab90 tx=0x7f868400af90 comp rx=0 tx=0).stop 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 msgr2=0x7f868c196c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 0x7f868c196c40 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f8674009230 tx=0x7f867402c6c0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 shutdown_connections 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86780a22a0 0x7f86780a4760 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f868c1062c0 0x7f868c19afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f868c100d40 0x7f868c196c40 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 --2- 172.21.15.3:0/1954936913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f868c100380 0x7f868c196700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.607+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 >> 172.21.15.3:0/1954936913 conn(0x7f868c076ef0 msgr2=0x7f868c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.608+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 shutdown_connections 2024-02-09T22:53:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:53:54.608+0000 7f8691363640 1 -- 172.21.15.3:0/1954936913 wait complete. 2024-02-09T22:53:54.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71639 3.74 MB/sec warmup 108 sec latency 9.209 ms 2024-02-09T22:53:55.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71685 3.70 MB/sec warmup 109 sec latency 976.902 ms 2024-02-09T22:53:56.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:56 smithi064 ceph-mon[36638]: pgmap v83: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 756 KiB/s wr, 94 op/s 2024-02-09T22:53:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:56 smithi089 ceph-mon[36665]: pgmap v83: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 756 KiB/s wr, 94 op/s 2024-02-09T22:53:56.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71685 3.67 MB/sec warmup 110 sec latency 1977.014 ms 2024-02-09T22:53:56.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:56 smithi003 ceph-mon[32678]: pgmap v83: 97 pgs: 97 active+clean; 39 MiB data, 590 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 756 KiB/s wr, 94 op/s 2024-02-09T22:53:57.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:57.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:57.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:57.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:57.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:57.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71685 3.64 MB/sec warmup 111 sec latency 2977.066 ms 2024-02-09T22:53:57.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:58.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:58 smithi064 ceph-mon[36638]: pgmap v84: 97 pgs: 97 active+clean; 38 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 968 KiB/s wr, 150 op/s 2024-02-09T22:53:58.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:53:58 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:53:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:58 smithi089 ceph-mon[36665]: pgmap v84: 97 pgs: 97 active+clean; 38 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 968 KiB/s wr, 150 op/s 2024-02-09T22:53:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:53:58 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:53:58.627 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71753 3.61 MB/sec warmup 112 sec latency 3396.064 ms 2024-02-09T22:53:58.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:58 smithi003 ceph-mon[32678]: pgmap v84: 97 pgs: 97 active+clean; 38 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 968 KiB/s wr, 150 op/s 2024-02-09T22:53:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:53:58 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:53:59.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71753 3.57 MB/sec warmup 113 sec latency 1534.645 ms 2024-02-09T22:54:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:00 smithi003 ceph-mon[32678]: pgmap v85: 97 pgs: 97 active+clean; 38 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 514 KiB/s wr, 85 op/s 2024-02-09T22:54:00.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:00 smithi064 ceph-mon[36638]: pgmap v85: 97 pgs: 97 active+clean; 38 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 514 KiB/s wr, 85 op/s 2024-02-09T22:54:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:00 smithi089 ceph-mon[36665]: pgmap v85: 97 pgs: 97 active+clean; 38 MiB data, 551 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 514 KiB/s wr, 85 op/s 2024-02-09T22:54:00.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 71753 3.54 MB/sec warmup 114 sec latency 2534.696 ms 2024-02-09T22:54:01.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 72658 3.55 MB/sec warmup 115 sec latency 2841.650 ms 2024-02-09T22:54:02.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:02 smithi064 ceph-mon[36638]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 552 KiB/s wr, 97 op/s 2024-02-09T22:54:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:02 smithi089 ceph-mon[36665]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 552 KiB/s wr, 97 op/s 2024-02-09T22:54:02.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 74932 3.64 MB/sec warmup 116 sec latency 10.119 ms 2024-02-09T22:54:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:02 smithi003 ceph-mon[32678]: pgmap v86: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 621 KiB/s rd, 552 KiB/s wr, 97 op/s 2024-02-09T22:54:03.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 76981 3.67 MB/sec warmup 117 sec latency 89.292 ms 2024-02-09T22:54:04.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:04 smithi064 ceph-mon[36638]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 250 KiB/s wr, 68 op/s 2024-02-09T22:54:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:04 smithi089 ceph-mon[36665]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 250 KiB/s wr, 68 op/s 2024-02-09T22:54:04.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78678 3.74 MB/sec warmup 118 sec latency 10.481 ms 2024-02-09T22:54:04.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:04 smithi003 ceph-mon[32678]: pgmap v87: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 250 KiB/s wr, 68 op/s 2024-02-09T22:54:05.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78809 3.71 MB/sec warmup 119 sec latency 914.343 ms 2024-02-09T22:54:06.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:06 smithi064 ceph-mon[36638]: pgmap v88: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 250 KiB/s wr, 68 op/s 2024-02-09T22:54:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:06 smithi089 ceph-mon[36665]: pgmap v88: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 250 KiB/s wr, 68 op/s 2024-02-09T22:54:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:06 smithi003 ceph-mon[32678]: pgmap v88: 97 pgs: 97 active+clean; 38 MiB data, 556 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 250 KiB/s wr, 68 op/s 2024-02-09T22:54:07.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78809 0.00 MB/sec execute 1 sec latency 2914.489 ms 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 -- 172.21.15.3:0/3245255644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c100340 msgr2=0x7f277c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 --2- 172.21.15.3:0/3245255644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c100340 0x7f277c100720 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f276c009930 tx=0x7f276c02bc20 comp rx=0 tx=0).stop 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 -- 172.21.15.3:0/3245255644 shutdown_connections 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 --2- 172.21.15.3:0/3245255644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f277c105410 0x7f277c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 --2- 172.21.15.3:0/3245255644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c100cf0 0x7f277c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 --2- 172.21.15.3:0/3245255644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c100340 0x7f277c100720 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 -- 172.21.15.3:0/3245255644 >> 172.21.15.3:0/3245255644 conn(0x7f277c077220 msgr2=0x7f277c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 -- 172.21.15.3:0/3245255644 shutdown_connections 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 -- 172.21.15.3:0/3245255644 wait complete. 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 Processor -- start 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.705+0000 7f2782647640 1 -- start start 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2782647640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f277c100340 0x7f277c198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2782647640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c105410 0x7f277c198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2782647640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 0x7f277c1a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2782647640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c1a3ce0 con 0x7f277c199060 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2782647640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c1a3e50 con 0x7f277c100340 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2782647640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f277c1a4010 con 0x7f277c105410 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 0x7f277c1a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 0x7f277c1a3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40150/0 (socket says 172.21.15.3:40150) 2024-02-09T22:54:07.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 -- 172.21.15.3:0/181667761 learned_addr learned my addr 172.21.15.3:0/181667761 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f277bfff640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f277c100340 0x7f277c198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f277b7fe640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c105410 0x7f277c198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 -- 172.21.15.3:0/181667761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c105410 msgr2=0x7f277c198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c105410 0x7f277c198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 -- 172.21.15.3:0/181667761 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f277c100340 msgr2=0x7f277c198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f277c100340 0x7f277c198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f2780be4640 1 -- 172.21.15.3:0/181667761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c009560 con 0x7f277c199060 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.706+0000 7f277b7fe640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c105410 0x7f277c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.707+0000 7f2780be4640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 0x7f277c1a3610 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f277400cce0 tx=0x7f277400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.707+0000 7f27797fa640 1 -- 172.21.15.3:0/181667761 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277400f450 con 0x7f277c199060 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.707+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f277c1a42f0 con 0x7f277c199060 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.707+0000 7f27797fa640 1 -- 172.21.15.3:0/181667761 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f277400fa90 con 0x7f277c199060 2024-02-09T22:54:07.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.707+0000 7f27797fa640 1 -- 172.21.15.3:0/181667761 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774018b90 con 0x7f277c199060 2024-02-09T22:54:07.713 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.707+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f277c1a47b0 con 0x7f277c199060 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.709+0000 7f27797fa640 1 -- 172.21.15.3:0/181667761 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f277400fc00 con 0x7f277c199060 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.709+0000 7f2785218740 1 Processor -- start 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.709+0000 7f2785218740 1 -- start start 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2785218740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61500b0 0x5599f6150490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2785218740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6159260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2785218740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 0x5599f615bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2785218740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f60bea00 con 0x5599f61597a0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2785218740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f60bbf30 con 0x5599f61500b0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2785218740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f60b7710 con 0x5599f61509d0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f27797fa640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27640a2220 0x7f27640a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 0x5599f615bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6159260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 0x5599f615bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40162/0 (socket says 172.21.15.3:40162) 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6159260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:45926/0 (socket says 172.21.15.3:45926) 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 -- 172.21.15.3:0/2226928912 learned_addr learned my addr 172.21.15.3:0/2226928912 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61500b0 0x5599f6150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f2780be4640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27640a2220 0x7f27640a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 -- 172.21.15.3:0/2226928912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 msgr2=0x5599f6159260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6159260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 -- 172.21.15.3:0/2226928912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61500b0 msgr2=0x5599f6150490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61500b0 0x5599f6150490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f27797fa640 1 -- 172.21.15.3:0/181667761 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f27740ce7e0 con 0x7f277c199060 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 -- 172.21.15.3:0/2226928912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599f607c3b0 con 0x5599f61597a0 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.710+0000 7f277b7fe640 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 0x5599f615bb90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2770013970 tx=0x7f277000b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b92484a8b4739ef4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2780be4640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27640a2220 0x7f27640a46e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2774097bd0 tx=0x7f277406b060 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f27627fc640 1 -- 172.21.15.3:0/2226928912 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700132d0 con 0x5599f61597a0 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f27627fc640 1 -- 172.21.15.3:0/2226928912 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2770019020 con 0x5599f61597a0 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f27627fc640 1 -- 172.21.15.3:0/2226928912 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770003850 con 0x5599f61597a0 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 -- 172.21.15.3:0/2226928912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 msgr2=0x5599f615bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 0x5599f615bb90 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f2770013970 tx=0x7f277000b4c0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 -- 172.21.15.3:0/2226928912 shutdown_connections 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61597a0 0x5599f615bb90 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6159260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 --2- 172.21.15.3:0/2226928912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61500b0 0x5599f6150490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 -- 172.21.15.3:0/2226928912 >> 172.21.15.3:0/2226928912 conn(0x5599f614a9d0 msgr2=0x5599f614adb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 -- 172.21.15.3:0/2226928912 shutdown_connections 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.711+0000 7f2785218740 1 -- 172.21.15.3:0/2226928912 wait complete. 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 -- start start 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 Processor -- start 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6156810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6156d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f6157290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f605cb90 con 0x5599f61500b0 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f5d8b250 con 0x5599f61597a0 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f603d200 con 0x5599f61509d0 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f2785218740 10 client.0 _ll_register_callbacks cb 0x5599f60735d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:07.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6156810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6156810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40170/0 (socket says 172.21.15.3:40170) 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.712+0000 7f277b7fe640 1 -- 172.21.15.3:0/3796659914 learned_addr learned my addr 172.21.15.3:0/3796659914 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277bfff640 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277b7fe640 1 -- 172.21.15.3:0/3796659914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 msgr2=0x5599f6156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277b7fe640 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277b7fe640 1 -- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 msgr2=0x5599f6157290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277b7fe640 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f6157290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277b7fe640 1 -- 172.21.15.3:0/3796659914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2770013650 con 0x5599f61500b0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f277b7fe640 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6156810 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f277000e720 tx=0x7f277000b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1ae0529491cf20 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f275b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f275b7fe640 1 -- 172.21.15.3:0/3796659914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277000eb70 con 0x5599f61500b0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f275b7fe640 1 -- 172.21.15.3:0/3796659914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2770000e40 con 0x5599f61500b0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599f60ddee0 con 0x5599f61500b0 2024-02-09T22:54:07.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f275b7fe640 1 -- 172.21.15.3:0/3796659914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2770005020 con 0x5599f61500b0 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f2785218740 10 client.16572 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5599f60b5d90 con 0x5599f61500b0 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.713+0000 7f275b7fe640 1 -- 172.21.15.3:0/3796659914 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5599f60ddee0 con 0x5599f61500b0 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f275b7fe640 1 -- 172.21.15.3:0/3796659914 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f2770004cd0 con 0x5599f61500b0 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f2785218740 10 client.16572 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f2785218740 10 client.16572 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f2785218740 20 client.16572 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f2785218740 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f6169970 0x5599f616bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f2785218740 4 client.16572 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599f5db7600 con 0x5599f6169970 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f277bfff640 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f6169970 0x5599f616bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f277bfff640 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f6169970 0x5599f616bd90 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4dc9f5a655168c05 server_cookie=9bce250286896595 in_seq=0 out_seq=0 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.714+0000 7f275b7fe640 10 client.16572 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:07.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f275b7fe640 1 -- 172.21.15.3:0/3796659914 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5599f5db7600 con 0x5599f6169970 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f275b7fe640 10 client.16572 handle_command_reply: tid=0 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 client.16572 shutdown 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 msgr2=0x5599f6156810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6156810 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f277000e720 tx=0x7f277000b410 comp rx=0 tx=0).stop 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 shutdown_connections 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f6169970 0x5599f616bd90 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f6157290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f6156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f275b7fe640 0 client.16572 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 --2- 172.21.15.3:0/3796659914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6156810 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 >> 172.21.15.3:0/3796659914 conn(0x5599f614a9d0 msgr2=0x5599f614c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 shutdown_connections 2024-02-09T22:54:07.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 1 -- 172.21.15.3:0/3796659914 wait complete. 2024-02-09T22:54:07.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.715+0000 7f2785218740 20 client.16572 trim_cache size 0 max 16384 2024-02-09T22:54:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 Processor -- start 2024-02-09T22:54:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 -- start start 2024-02-09T22:54:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6158f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f61550e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f6155620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f6169740 con 0x5599f61500b0 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f614bc00 con 0x5599f61597a0 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f2785218740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f6155b60 con 0x5599f61509d0 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40172/0 (socket says 172.21.15.3:40172) 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 -- 172.21.15.3:0/105882135 learned_addr learned my addr 172.21.15.3:0/105882135 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 -- 172.21.15.3:0/105882135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 msgr2=0x5599f61550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277bfff640 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f61550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f61550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 -- 172.21.15.3:0/105882135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 msgr2=0x5599f6155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f6155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 -- 172.21.15.3:0/105882135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5599f6052820 con 0x5599f61500b0 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277bfff640 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f61550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:07.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.729+0000 7f277b7fe640 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6158f40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2770006010 tx=0x7f2770013870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c62f20796740a9c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2761ffb640 1 -- 172.21.15.3:0/105882135 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277000eed0 con 0x5599f61500b0 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2761ffb640 1 -- 172.21.15.3:0/105882135 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f277000bb40 con 0x5599f61500b0 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2761ffb640 1 -- 172.21.15.3:0/105882135 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770005290 con 0x5599f61500b0 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 -- 172.21.15.3:0/105882135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 msgr2=0x5599f6158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6158f40 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f2770006010 tx=0x7f2770013870 comp rx=0 tx=0).stop 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 -- 172.21.15.3:0/105882135 shutdown_connections 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f6155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f61550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 --2- 172.21.15.3:0/105882135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6158f40 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 -- 172.21.15.3:0/105882135 >> 172.21.15.3:0/105882135 conn(0x5599f614a9d0 msgr2=0x5599f61706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 -- 172.21.15.3:0/105882135 shutdown_connections 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.730+0000 7f2785218740 1 -- 172.21.15.3:0/105882135 wait complete. 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 -- start start 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 Processor -- start 2024-02-09T22:54:07.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6160780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f615eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f615f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f615f780 con 0x5599f61500b0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f615f8f0 con 0x5599f61597a0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5599f61bfe60 con 0x5599f61509d0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f2785218740 10 client.0 _ll_register_callbacks cb 0x7f277000bee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6160780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f277b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6160780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:40180/0 (socket says 172.21.15.3:40180) 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f277b7fe640 1 -- 172.21.15.3:0/472795083 learned_addr learned my addr 172.21.15.3:0/472795083 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f277bfff640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f615eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.731+0000 7f277b7fe640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f615f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f277b7fe640 1 -- 172.21.15.3:0/472795083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 msgr2=0x5599f615eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f277b7fe640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f615eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f277b7fe640 1 -- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 msgr2=0x5599f615f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f277b7fe640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f615f0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f277b7fe640 1 -- 172.21.15.3:0/472795083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5599f60b5d90 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f277b7fe640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6160780 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f27700134b0 tx=0x7f2770013910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd59a7843cdd3c1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f275b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f275b7fe640 1 -- 172.21.15.3:0/472795083 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2770011c40 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f275b7fe640 1 -- 172.21.15.3:0/472795083 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2770000e40 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2770019020 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f275b7fe640 1 -- 172.21.15.3:0/472795083 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2770011da0 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f2785218740 10 client.16578 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5599f60786f0 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f275b7fe640 1 -- 172.21.15.3:0/472795083 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2770019020 con 0x5599f61500b0 2024-02-09T22:54:07.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.732+0000 7f275b7fe640 1 -- 172.21.15.3:0/472795083 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f277003b030 con 0x5599f61500b0 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f2785218740 10 client.16578 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f2785218740 10 client.16578 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f2785218740 20 client.16578 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f2785218740 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f61c10f0 0x5599f61c3510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f2785218740 4 client.16578 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5599f5e081b0 con 0x5599f61c10f0 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f277bfff640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f61c10f0 0x5599f61c3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f275b7fe640 10 client.16578 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:07.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.733+0000 7f277bfff640 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f61c10f0 0x5599f61c3510 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41f4c0cbcf692793 server_cookie=3fa198f795fde442 in_seq=0 out_seq=0 2024-02-09T22:54:07.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.764+0000 7f275b7fe640 1 -- 172.21.15.3:0/472795083 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5599f5e081b0 con 0x5599f61c10f0 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.764+0000 7f275b7fe640 10 client.16578 handle_command_reply: tid=0 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.764+0000 7f2785218740 1 client.16578 shutdown 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 msgr2=0x5599f6160780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6160780 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f27700134b0 tx=0x7f2770013910 comp rx=0 tx=0).stop 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 shutdown_connections 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5599f61c10f0 0x5599f61c3510 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5599f61597a0 0x5599f615f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f275b7fe640 0 client.16578 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5599f61509d0 0x5599f615eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 --2- 172.21.15.3:0/472795083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5599f61500b0 0x5599f6160780 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 >> 172.21.15.3:0/472795083 conn(0x5599f614a9d0 msgr2=0x5599f6154ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 shutdown_connections 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 1 -- 172.21.15.3:0/472795083 wait complete. 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.765+0000 7f2785218740 20 client.16578 trim_cache size 0 max 16384 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:54:07.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:54:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:54:07.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27640a2220 msgr2=0x7f27640a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27640a2220 0x7f27640a46e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2774097bd0 tx=0x7f277406b060 comp rx=0 tx=0).stop 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 msgr2=0x7f277c1a3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 0x7f277c1a3610 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f277400cce0 tx=0x7f277400a620 comp rx=0 tx=0).stop 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 shutdown_connections 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27640a2220 0x7f27640a46e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f277c199060 0x7f277c1a3610 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f277c105410 0x7f277c198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 --2- 172.21.15.3:0/181667761 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f277c100340 0x7f277c198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 >> 172.21.15.3:0/181667761 conn(0x7f277c077220 msgr2=0x7f277c0fe030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 shutdown_connections 2024-02-09T22:54:07.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:07.766+0000 7f2782647640 1 -- 172.21.15.3:0/181667761 wait complete. 2024-02-09T22:54:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:08 smithi089 ceph-mon[36665]: pgmap v89: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-02-09T22:54:08.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78877 0.08 MB/sec execute 2 sec latency 3330.936 ms 2024-02-09T22:54:08.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:08 smithi003 ceph-mon[32678]: pgmap v89: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-02-09T22:54:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:08 smithi064 ceph-mon[36638]: pgmap v89: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.1 MiB/s wr, 191 op/s 2024-02-09T22:54:09.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78877 0.05 MB/sec execute 3 sec latency 1538.930 ms 2024-02-09T22:54:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:10 smithi089 ceph-mon[36665]: pgmap v90: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 916 KiB/s wr, 135 op/s 2024-02-09T22:54:10.628 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78877 0.04 MB/sec execute 4 sec latency 2538.990 ms 2024-02-09T22:54:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:10 smithi003 ceph-mon[32678]: pgmap v90: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 916 KiB/s wr, 135 op/s 2024-02-09T22:54:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:10 smithi064 ceph-mon[36638]: pgmap v90: 97 pgs: 97 active+clean; 46 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 1002 KiB/s rd, 916 KiB/s wr, 135 op/s 2024-02-09T22:54:11.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78877 0.03 MB/sec execute 5 sec latency 3539.135 ms 2024-02-09T22:54:12.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:12 smithi089 ceph-mon[36665]: pgmap v91: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 984 KiB/s wr, 164 op/s 2024-02-09T22:54:12.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 78877 0.03 MB/sec execute 6 sec latency 4539.255 ms 2024-02-09T22:54:12.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:12 smithi003 ceph-mon[32678]: pgmap v91: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 984 KiB/s wr, 164 op/s 2024-02-09T22:54:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:12 smithi064 ceph-mon[36638]: pgmap v91: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 984 KiB/s wr, 164 op/s 2024-02-09T22:54:13.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 80422 0.95 MB/sec execute 7 sec latency 4952.662 ms 2024-02-09T22:54:14.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:14 smithi089 ceph-mon[36665]: pgmap v92: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 945 KiB/s wr, 152 op/s 2024-02-09T22:54:14.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82102 2.36 MB/sec execute 8 sec latency 10.032 ms 2024-02-09T22:54:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:14 smithi003 ceph-mon[32678]: pgmap v92: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 945 KiB/s wr, 152 op/s 2024-02-09T22:54:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:14 smithi064 ceph-mon[36638]: pgmap v92: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 945 KiB/s wr, 152 op/s 2024-02-09T22:54:15.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82302 2.11 MB/sec execute 9 sec latency 886.165 ms 2024-02-09T22:54:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:16 smithi089 ceph-mon[36665]: pgmap v93: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 945 KiB/s wr, 152 op/s 2024-02-09T22:54:16.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82302 1.90 MB/sec execute 10 sec latency 1886.283 ms 2024-02-09T22:54:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:16 smithi003 ceph-mon[32678]: pgmap v93: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 945 KiB/s wr, 152 op/s 2024-02-09T22:54:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:16 smithi064 ceph-mon[36638]: pgmap v93: 97 pgs: 97 active+clean; 39 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 945 KiB/s wr, 152 op/s 2024-02-09T22:54:17.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82302 1.73 MB/sec execute 11 sec latency 2886.396 ms 2024-02-09T22:54:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:18 smithi089 ceph-mon[36665]: pgmap v94: 97 pgs: 97 active+clean; 40 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 208 op/s 2024-02-09T22:54:18.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82426 1.61 MB/sec execute 12 sec latency 3304.431 ms 2024-02-09T22:54:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:18 smithi003 ceph-mon[32678]: pgmap v94: 97 pgs: 97 active+clean; 40 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 208 op/s 2024-02-09T22:54:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:18 smithi064 ceph-mon[36638]: pgmap v94: 97 pgs: 97 active+clean; 40 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.6 MiB/s wr, 208 op/s 2024-02-09T22:54:19.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82426 1.48 MB/sec execute 13 sec latency 1494.821 ms 2024-02-09T22:54:19.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:54:19.802 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 -- 172.21.15.3:0/990308846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208101430 msgr2=0x7f7208101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 --2- 172.21.15.3:0/990308846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208101430 0x7f7208101810 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f71f8009930 tx=0x7f71f802bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 -- 172.21.15.3:0/990308846 shutdown_connections 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 --2- 172.21.15.3:0/990308846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208101de0 0x7f7208105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 --2- 172.21.15.3:0/990308846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208106500 0x7f720810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 --2- 172.21.15.3:0/990308846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208101430 0x7f7208101810 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 -- 172.21.15.3:0/990308846 >> 172.21.15.3:0/990308846 conn(0x7f72080fccd0 msgr2=0x7f72080ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 -- 172.21.15.3:0/990308846 shutdown_connections 2024-02-09T22:54:19.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.915+0000 7f720c93a640 1 -- 172.21.15.3:0/990308846 wait complete. 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.916+0000 7f720c93a640 1 Processor -- start 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.916+0000 7f720c93a640 1 -- start start 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208101430 0x7f720819f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 0x7f720819fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208199950 0x7f7208199e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720819a340 con 0x7f7208106500 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720819a4b0 con 0x7f7208199950 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f720819a670 con 0x7f7208101430 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 0x7f720819fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 0x7f720819fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56794/0 (socket says 172.21.15.3:56794) 2024-02-09T22:54:19.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208101430 0x7f720819f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 -- 172.21.15.3:0/1833894443 learned_addr learned my addr 172.21.15.3:0/1833894443 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7206d9d640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208199950 0x7f7208199e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 -- 172.21.15.3:0/1833894443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208101430 msgr2=0x7f720819f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208101430 0x7f720819f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 -- 172.21.15.3:0/1833894443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208199950 msgr2=0x7f7208199e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208199950 0x7f7208199e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 -- 172.21.15.3:0/1833894443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f8009560 con 0x7f7208106500 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7206d9d640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208199950 0x7f7208199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f7205d9b640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 0x7f720819fce0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c970 tx=0x7f71fc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f71f77fe640 1 -- 172.21.15.3:0/1833894443 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc007bb0 con 0x7f7208106500 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f71f77fe640 1 -- 172.21.15.3:0/1833894443 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71fc007d10 con 0x7f7208106500 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72081a6660 con 0x7f7208106500 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.917+0000 7f71f77fe640 1 -- 172.21.15.3:0/1833894443 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc00f450 con 0x7f7208106500 2024-02-09T22:54:19.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.918+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72081a6b40 con 0x7f7208106500 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.920+0000 7f71f77fe640 1 -- 172.21.15.3:0/1833894443 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f71fc004f20 con 0x7f7208106500 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.920+0000 7f720f5e3740 1 Processor -- start 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 -- start start 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3d7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f71f77fe640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71f00a2230 0x7f71f00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3e08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc392d00 con 0x5575bc3e0eb0 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc3472f0 con 0x5575bc3d7980 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720f5e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc344820 con 0x5575bc3d8330 2024-02-09T22:54:19.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f720659c640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71f00a2230 0x7f71f00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7206d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7206d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e3300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56804/0 (socket says 172.21.15.3:56804) 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7206d9d640 1 -- 172.21.15.3:0/680950740 learned_addr learned my addr 172.21.15.3:0/680950740 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7205d9b640 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3e08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f71f77fe640 1 -- 172.21.15.3:0/1833894443 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f71fc0ce180 con 0x7f7208106500 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7206d9d640 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3d7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7205d9b640 1 -- 172.21.15.3:0/680950740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 msgr2=0x5575bc3d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7205d9b640 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3d7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7205d9b640 1 -- 172.21.15.3:0/680950740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 msgr2=0x5575bc3e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7205d9b640 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.921+0000 7f7205d9b640 1 -- 172.21.15.3:0/680950740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575bc301ff0 con 0x5575bc3d8330 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.922+0000 7f720659c640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71f00a2230 0x7f71f00a46f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f71f8005ab0 tx=0x7f71f8005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.922+0000 7f7205d9b640 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3e08e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0cb8a0 tx=0x7f71fc020000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d01a619d2796f455 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:19.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.922+0000 7f71f4ff9640 1 -- 172.21.15.3:0/680950740 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc09a4a0 con 0x5575bc3d8330 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.922+0000 7f71f4ff9640 1 -- 172.21.15.3:0/680950740 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71fc0a01b0 con 0x5575bc3d8330 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.922+0000 7f71f4ff9640 1 -- 172.21.15.3:0/680950740 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc09ff30 con 0x5575bc3d8330 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.922+0000 7f720f5e3740 1 -- 172.21.15.3:0/680950740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 msgr2=0x5575bc3e08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3e08e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71fc0cb8a0 tx=0x7f71fc020000 comp rx=0 tx=0).stop 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- 172.21.15.3:0/680950740 shutdown_connections 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3e08e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- 172.21.15.3:0/680950740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3d7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- 172.21.15.3:0/680950740 >> 172.21.15.3:0/680950740 conn(0x5575bc3d2050 msgr2=0x5575bc3d2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- 172.21.15.3:0/680950740 shutdown_connections 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- 172.21.15.3:0/680950740 wait complete. 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- start start 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 Processor -- start 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3ddde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 0x5575bc3de320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3e0eb0 0x5575bc3de860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc340000 con 0x5575bc3d8330 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc013430 con 0x5575bc3d7980 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bbfcfe00 con 0x5575bc3e0eb0 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f720f5e3740 10 client.0 _ll_register_callbacks cb 0x5575bc3347f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f7205d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 0x5575bc3de320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.923+0000 7f7205d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 0x5575bc3de320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56820/0 (socket says 172.21.15.3:56820) 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 -- 172.21.15.3:0/788059699 learned_addr learned my addr 172.21.15.3:0/788059699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7206d9d640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3e0eb0 0x5575bc3de860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f720659c640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3ddde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 -- 172.21.15.3:0/788059699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3e0eb0 msgr2=0x5575bc3de860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3e0eb0 0x5575bc3de860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 -- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 msgr2=0x5575bc3ddde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3ddde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 -- 172.21.15.3:0/788059699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72081a6b40 con 0x5575bc3d8330 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f7205d9b640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 0x5575bc3de320 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f71fc007590 tx=0x7f71fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=885995cbc18ec6ba server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f71edffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f71edffb640 1 -- 172.21.15.3:0/788059699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc0b7070 con 0x5575bc3d8330 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f71edffb640 1 -- 172.21.15.3:0/788059699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f71fc005600 con 0x5575bc3d8330 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575bc03f7e0 con 0x5575bc3d8330 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f71edffb640 1 -- 172.21.15.3:0/788059699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71fc05d020 con 0x5575bc3d8330 2024-02-09T22:54:19.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f720f5e3740 10 client.16590 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5575bc33e680 con 0x5575bc3d8330 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.924+0000 7f71edffb640 1 -- 172.21.15.3:0/788059699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5575bc03f7e0 con 0x5575bc3d8330 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f71edffb640 1 -- 172.21.15.3:0/788059699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f71fc0ba070 con 0x5575bc3d8330 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720f5e3740 10 client.16590 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720f5e3740 10 client.16590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720f5e3740 20 client.16590 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720f5e3740 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc3f0f60 0x5575bc3f3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720f5e3740 4 client.16590 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575bc07c720 con 0x5575bc3f0f60 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720659c640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc3f0f60 0x5575bc3f3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f720659c640 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc3f0f60 0x5575bc3f3380 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a41f08095217a323 server_cookie=e1f93dc5dcaba584 in_seq=0 out_seq=0 2024-02-09T22:54:19.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.925+0000 7f71edffb640 10 client.16590 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f71edffb640 1 -- 172.21.15.3:0/788059699 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5575bc07c720 con 0x5575bc3f0f60 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f71edffb640 10 client.16590 handle_command_reply: tid=0 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 client.16590 shutdown 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 msgr2=0x5575bc3de320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 0x5575bc3de320 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f71fc007590 tx=0x7f71fc0046c0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 shutdown_connections 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc3f0f60 0x5575bc3f3380 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3e0eb0 0x5575bc3de860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d8330 0x5575bc3de320 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f71edffb640 0 client.16590 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:19.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 --2- 172.21.15.3:0/788059699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d7980 0x5575bc3ddde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 >> 172.21.15.3:0/788059699 conn(0x5575bc3d2050 msgr2=0x5575bc3d3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:19.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 shutdown_connections 2024-02-09T22:54:19.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 1 -- 172.21.15.3:0/788059699 wait complete. 2024-02-09T22:54:19.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.926+0000 7f720f5e3740 20 client.16590 trim_cache size 0 max 16384 2024-02-09T22:54:19.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.941+0000 7f720f5e3740 1 Processor -- start 2024-02-09T22:54:19.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.941+0000 7f720f5e3740 1 -- start start 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.941+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 0x5575bc3e0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.941+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3de680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3e0eb0 0x5575bc3db080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720f5e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc3cd780 con 0x5575bc3d7980 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720f5e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc2c4fd0 con 0x5575bc3e0eb0 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720f5e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc3edee0 con 0x5575bc3d8330 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 0x5575bc3e0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 0x5575bc3e0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56830/0 (socket says 172.21.15.3:56830) 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 -- 172.21.15.3:0/2658979832 learned_addr learned my addr 172.21.15.3:0/2658979832 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 -- 172.21.15.3:0/2658979832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 msgr2=0x5575bc3de680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3de680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 -- 172.21.15.3:0/2658979832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3e0eb0 msgr2=0x5575bc3db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3e0eb0 0x5575bc3db080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 -- 172.21.15.3:0/2658979832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575bc2dab30 con 0x5575bc3d7980 2024-02-09T22:54:19.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f720659c640 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 0x5575bc3e0510 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f71f8036040 tx=0x7f71f803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a556efdfc7e36be4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:19.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f71effff640 1 -- 172.21.15.3:0/2658979832 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f8056070 con 0x5575bc3d7980 2024-02-09T22:54:19.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f71effff640 1 -- 172.21.15.3:0/2658979832 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71f8051070 con 0x5575bc3d7980 2024-02-09T22:54:19.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.942+0000 7f71effff640 1 -- 172.21.15.3:0/2658979832 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f80516b0 con 0x5575bc3d7980 2024-02-09T22:54:19.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- 172.21.15.3:0/2658979832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 msgr2=0x5575bc3e0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 0x5575bc3e0510 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f71f8036040 tx=0x7f71f803f000 comp rx=0 tx=0).stop 2024-02-09T22:54:19.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- 172.21.15.3:0/2658979832 shutdown_connections 2024-02-09T22:54:19.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3e0eb0 0x5575bc3db080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d8330 0x5575bc3de680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2658979832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3d7980 0x5575bc3e0510 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- 172.21.15.3:0/2658979832 >> 172.21.15.3:0/2658979832 conn(0x5575bc3d2050 msgr2=0x5575bc3d9100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- 172.21.15.3:0/2658979832 shutdown_connections 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- 172.21.15.3:0/2658979832 wait complete. 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- start start 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 Processor -- start 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d7980 0x5575bc3e69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d8330 0x5575bc3e6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc447470 con 0x5575bc3e0eb0 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc4475b0 con 0x5575bc3d8330 2024-02-09T22:54:19.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575bc4476f0 con 0x5575bc3d7980 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.943+0000 7f720f5e3740 10 client.0 _ll_register_callbacks cb 0x7f71fc007e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56836/0 (socket says 172.21.15.3:56836) 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 -- 172.21.15.3:0/2485164030 learned_addr learned my addr 172.21.15.3:0/2485164030 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7205d9b640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d8330 0x5575bc3e6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f720659c640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d7980 0x5575bc3e69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 -- 172.21.15.3:0/2485164030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d7980 msgr2=0x5575bc3e69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d7980 0x5575bc3e69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 -- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d8330 msgr2=0x5575bc3e6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d8330 0x5575bc3e6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 -- 172.21.15.3:0/2485164030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f720810e8e0 con 0x5575bc3e0eb0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f720659c640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d7980 0x5575bc3e69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f7206d9d640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e7420 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7200013c30 tx=0x7f7200011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8cd289f073317b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f71edffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:19.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f71edffb640 1 -- 172.21.15.3:0/2485164030 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200004d20 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f71edffb640 1 -- 172.21.15.3:0/2485164030 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7200011e40 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f71fc0a01b0 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f71edffb640 1 -- 172.21.15.3:0/2485164030 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7200004e80 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f720f5e3740 10 client.16596 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.944+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5575bc300850 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f71edffb640 1 -- 172.21.15.3:0/2485164030 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71fc0a01b0 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f71edffb640 1 -- 172.21.15.3:0/2485164030 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f7200010550 con 0x5575bc3e0eb0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720f5e3740 10 client.16596 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720f5e3740 10 client.16596 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720f5e3740 20 client.16596 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc4490c0 0x5575bc44b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720f5e3740 4 client.16596 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5575bc0cb890 con 0x5575bc4490c0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.945+0000 7f720659c640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc4490c0 0x5575bc44b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:19.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720659c640 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc4490c0 0x5575bc44b4e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d86d274c6aacf01a server_cookie=f8a5f6e14d5292df in_seq=0 out_seq=0 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f71edffb640 10 client.16596 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f71edffb640 1 -- 172.21.15.3:0/2485164030 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5575bc0cb890 con 0x5575bc4490c0 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f71edffb640 10 client.16596 handle_command_reply: tid=0 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 client.16596 shutdown 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 msgr2=0x5575bc3e7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e7420 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f7200013c30 tx=0x7f7200011810 comp rx=0 tx=0).stop 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 shutdown_connections 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5575bc4490c0 0x5575bc44b4e0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5575bc3d8330 0x5575bc3e6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5575bc3e0eb0 0x5575bc3e7420 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f71edffb640 0 client.16596 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 --2- 172.21.15.3:0/2485164030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5575bc3d7980 0x5575bc3e69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 >> 172.21.15.3:0/2485164030 conn(0x5575bc3d2050 msgr2=0x5575bc3d0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 shutdown_connections 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 1 -- 172.21.15.3:0/2485164030 wait complete. 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720f5e3740 20 client.16596 trim_cache size 0 max 16384 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:54:19.957 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:54:19.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71f00a2230 msgr2=0x7f71f00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720c93a640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71f00a2230 0x7f71f00a46f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f71f8005ab0 tx=0x7f71f8005fd0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 msgr2=0x7f720819fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:19.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720c93a640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 0x7f720819fce0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00c970 tx=0x7f71fc00ce30 comp rx=0 tx=0).stop 2024-02-09T22:54:19.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 shutdown_connections 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.946+0000 7f720c93a640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71f00a2230 0x7f71f00a46f0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.947+0000 7f720c93a640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7208199950 0x7f7208199e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.947+0000 7f720c93a640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7208106500 0x7f720819fce0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.947+0000 7f720c93a640 1 --2- 172.21.15.3:0/1833894443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7208101430 0x7f720819f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.947+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 >> 172.21.15.3:0/1833894443 conn(0x7f72080fccd0 msgr2=0x7f72080ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.947+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 shutdown_connections 2024-02-09T22:54:19.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:19.947+0000 7f720c93a640 1 -- 172.21.15.3:0/1833894443 wait complete. 2024-02-09T22:54:19.960 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:54:19.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 -- 172.21.15.3:0/893535177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc1065b0 msgr2=0x7f95fc10e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 --2- 172.21.15.3:0/893535177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc1065b0 0x7f95fc10e130 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f95f4009cd0 tx=0x7f95f402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 -- 172.21.15.3:0/893535177 shutdown_connections 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 --2- 172.21.15.3:0/893535177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc1065b0 0x7f95fc10e130 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 --2- 172.21.15.3:0/893535177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc100640 0x7f95fc105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 --2- 172.21.15.3:0/893535177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 -- 172.21.15.3:0/893535177 >> 172.21.15.3:0/893535177 conn(0x7f95fc076580 msgr2=0x7f95fc076990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 -- 172.21.15.3:0/893535177 shutdown_connections 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.060+0000 7f96036e1640 1 -- 172.21.15.3:0/893535177 wait complete. 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 Processor -- start 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 -- start start 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc1065b0 0x7f95fc101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc101680 0x7f95fc1a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1a5fc0 con 0x7f95fc0ffc90 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1a6130 con 0x7f95fc101680 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f96036e1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95fc1a62f0 con 0x7f95fc1065b0 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56864/0 (socket says 172.21.15.3:56864) 2024-02-09T22:54:20.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 -- 172.21.15.3:0/3765887076 learned_addr learned my addr 172.21.15.3:0/3765887076 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f9601c7e640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc101680 0x7f95fc1a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f9600c7c640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc1065b0 0x7f95fc101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 -- 172.21.15.3:0/3765887076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc1065b0 msgr2=0x7f95fc101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc1065b0 0x7f95fc101140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 -- 172.21.15.3:0/3765887076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc101680 msgr2=0x7f95fc1a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc101680 0x7f95fc1a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f960147d640 1 -- 172.21.15.3:0/3765887076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f4009900 con 0x7f95fc0ffc90 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.061+0000 7f9601c7e640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc101680 0x7f95fc1a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.062+0000 7f960147d640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100c00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f95f800e9f0 tx=0x7f95f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.062+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f8009700 con 0x7f95fc0ffc90 2024-02-09T22:54:20.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.062+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95fc1a65d0 con 0x7f95fc0ffc90 2024-02-09T22:54:20.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.062+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f800cd80 con 0x7f95fc0ffc90 2024-02-09T22:54:20.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.062+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f8018920 con 0x7f95fc0ffc90 2024-02-09T22:54:20.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.062+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95fc1a6b30 con 0x7f95fc0ffc90 2024-02-09T22:54:20.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.063+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95fc068500 con 0x7f95fc0ffc90 2024-02-09T22:54:20.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.064+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f95f8018a80 con 0x7f95fc0ffc90 2024-02-09T22:54:20.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.065+0000 7f95f27fc640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95e80a2260 0x7f95e80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.065+0000 7f9600c7c640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95e80a2260 0x7f95e80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.065+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f95f8014070 con 0x7f95fc0ffc90 2024-02-09T22:54:20.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.065+0000 7f9600c7c640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95e80a2260 0x7f95e80a4720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95fc102690 tx=0x7f95ec005be0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.069+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f95f8052d20 con 0x7f95fc0ffc90 2024-02-09T22:54:20.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.295+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95fc109760 con 0x7f95fc0ffc90 2024-02-09T22:54:20.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.296+0000 7f95f27fc640 1 -- 172.21.15.3:0/3765887076 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 31 v31) v1 ==== 94+0+11371 (secure 0 0 0) 0x7f95f8097200 con 0x7f95fc0ffc90 2024-02-09T22:54:20.299 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:54:20.299 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":31,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":31,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:53:02.760086+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:54:20.299 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 31 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95e80a2260 msgr2=0x7f95e80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95e80a2260 0x7f95e80a4720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f95fc102690 tx=0x7f95ec005be0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 msgr2=0x7f95fc100c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100c00 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f95f800e9f0 tx=0x7f95f800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 shutdown_connections 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95e80a2260 0x7f95e80a4720 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95fc101680 0x7f95fc1a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95fc1065b0 0x7f95fc101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 --2- 172.21.15.3:0/3765887076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95fc0ffc90 0x7f95fc100c00 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 >> 172.21.15.3:0/3765887076 conn(0x7f95fc076580 msgr2=0x7f95fc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 shutdown_connections 2024-02-09T22:54:20.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.298+0000 7f96036e1640 1 -- 172.21.15.3:0/3765887076 wait complete. 2024-02-09T22:54:20.311 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:20.311 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:54:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 -- 172.21.15.3:0/2170351978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 msgr2=0x7f9d20102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 --2- 172.21.15.3:0/2170351978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20102180 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009930 tx=0x7f9d0c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 -- 172.21.15.3:0/2170351978 shutdown_connections 2024-02-09T22:54:20.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 --2- 172.21.15.3:0/2170351978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d201026c0 0x7f9d2010d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 --2- 172.21.15.3:0/2170351978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20102180 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 --2- 172.21.15.3:0/2170351978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d20100440 0x7f9d20100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 -- 172.21.15.3:0/2170351978 >> 172.21.15.3:0/2170351978 conn(0x7f9d20077310 msgr2=0x7f9d20077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 -- 172.21.15.3:0/2170351978 shutdown_connections 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.413+0000 7f9d2701e640 1 -- 172.21.15.3:0/2170351978 wait complete. 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 Processor -- start 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 -- start start 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d20100440 0x7f9d20198710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20198c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d201026c0 0x7f9d201a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d201994b0 con 0x7f9d20101d20 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d201a3cd0 con 0x7f9d201026c0 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d2701e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d201a3e10 con 0x7f9d20100440 2024-02-09T22:54:20.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20198c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20198c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56874/0 (socket says 172.21.15.3:56874) 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 -- 172.21.15.3:0/1158757506 learned_addr learned my addr 172.21.15.3:0/1158757506 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d255bb640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d201026c0 0x7f9d201a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d24dba640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d20100440 0x7f9d20198710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 -- 172.21.15.3:0/1158757506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d20100440 msgr2=0x7f9d20198710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d20100440 0x7f9d20198710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 -- 172.21.15.3:0/1158757506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d201026c0 msgr2=0x7f9d201a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d201026c0 0x7f9d201a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 -- 172.21.15.3:0/1158757506 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d0c009560 con 0x7f9d20101d20 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d24dba640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d20100440 0x7f9d20198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.414+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20198c50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009230 tx=0x7f9d0c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.415+0000 7f9d1dffb640 1 -- 172.21.15.3:0/1158757506 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c004440 con 0x7f9d20101d20 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.415+0000 7f9d1dffb640 1 -- 172.21.15.3:0/1158757506 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c0052d0 con 0x7f9d20101d20 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.415+0000 7f9d1dffb640 1 -- 172.21.15.3:0/1158757506 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c003d70 con 0x7f9d20101d20 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.415+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d201a4010 con 0x7f9d20101d20 2024-02-09T22:54:20.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.415+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d201a4520 con 0x7f9d20101d20 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.417+0000 7f9d1dffb640 1 -- 172.21.15.3:0/1158757506 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9d0c005440 con 0x7f9d20101d20 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.417+0000 7f9d29bf0740 1 Processor -- start 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.417+0000 7f9d29bf0740 1 -- start start 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.417+0000 7f9d1dffb640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d100a2230 0x7f9d100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.417+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563cf8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.417+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d01990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d043b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563cb4020 con 0x556563cf93e0 2024-02-09T22:54:20.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563c67320 con 0x556563d01f60 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563c64850 con 0x556563cf8a30 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d01990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d255bb640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d100a2230 0x7f9d100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d01990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56888/0 (socket says 172.21.15.3:56888) 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 -- 172.21.15.3:0/4090916263 learned_addr learned my addr 172.21.15.3:0/4090916263 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1dffb640 1 -- 172.21.15.3:0/1158757506 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d0c0f0300 con 0x7f9d20101d20 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d24dba640 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563cf8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 -- 172.21.15.3:0/4090916263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 msgr2=0x556563cf8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563cf8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 -- 172.21.15.3:0/4090916263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 msgr2=0x556563d043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 -- 172.21.15.3:0/4090916263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556563c22210 con 0x556563cf93e0 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d255bb640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d100a2230 0x7f9d100a46f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9d180092f0 tx=0x7f9d18009890 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d01990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0b9230 tx=0x7f9d0c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f9ecc94c69e1d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d0affd640 1 -- 172.21.15.3:0/4090916263 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0cf620 con 0x556563cf93e0 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d0affd640 1 -- 172.21.15.3:0/4090916263 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c0cf440 con 0x556563cf93e0 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.418+0000 7f9d0affd640 1 -- 172.21.15.3:0/4090916263 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0c5570 con 0x556563cf93e0 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 -- 172.21.15.3:0/4090916263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 msgr2=0x556563d01990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d01990 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0b9230 tx=0x7f9d0c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 -- 172.21.15.3:0/4090916263 shutdown_connections 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d01990 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/4090916263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563cf8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 -- 172.21.15.3:0/4090916263 >> 172.21.15.3:0/4090916263 conn(0x556563cf3100 msgr2=0x556563cf34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 -- 172.21.15.3:0/4090916263 shutdown_connections 2024-02-09T22:54:20.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 -- 172.21.15.3:0/4090916263 wait complete. 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 -- start start 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.419+0000 7f9d29bf0740 1 Processor -- start 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf8a30 0x556563cfee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf93e0 0x556563cff3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cff8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563c60030 con 0x556563d01f60 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563933430 con 0x556563cf8a30 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565638efe00 con 0x556563cf93e0 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d29bf0740 10 client.0 _ll_register_callbacks cb 0x556563c55030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cff8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cff8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56902/0 (socket says 172.21.15.3:56902) 2024-02-09T22:54:20.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 -- 172.21.15.3:0/1814621078 learned_addr learned my addr 172.21.15.3:0/1814621078 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d24dba640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf8a30 0x556563cfee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf93e0 0x556563cff3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 -- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf93e0 msgr2=0x556563cff3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf93e0 0x556563cff3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 -- 172.21.15.3:0/1814621078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf8a30 msgr2=0x556563cfee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf8a30 0x556563cfee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d255bb640 1 -- 172.21.15.3:0/1814621078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d201a4520 con 0x556563d01f60 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d24dba640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf8a30 0x556563cfee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.420+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf93e0 0x556563cff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d255bb640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cff8e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9d180138b0 tx=0x7f9d18004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adba4f5a0e78bd68 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9cfffff640 1 -- 172.21.15.3:0/1814621078 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d180096a0 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9cfffff640 1 -- 172.21.15.3:0/1814621078 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d18019030 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55656395f7e0 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9cfffff640 1 -- 172.21.15.3:0/1814621078 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d1800fcf0 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 10 client.16614 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556563c5e6b0 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9cfffff640 1 -- 172.21.15.3:0/1814621078 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55656395f7e0 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9cfffff640 1 -- 172.21.15.3:0/1814621078 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9d18027060 con 0x556563d01f60 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 10 client.16614 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 10 client.16614 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 20 client.16614 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d14720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 4 client.16614 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.421+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55656399c720 con 0x556563d12300 2024-02-09T22:54:20.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.422+0000 7f9d24dba640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d14720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.422+0000 7f9d24dba640 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d14720 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e216e4ec91c6ebe9 server_cookie=5c702a191df89d4a in_seq=0 out_seq=0 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.422+0000 7f9cfffff640 10 client.16614 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9cfffff640 1 -- 172.21.15.3:0/1814621078 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55656399c720 con 0x556563d12300 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9cfffff640 10 client.16614 handle_command_reply: tid=0 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 client.16614 shutdown 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 msgr2=0x556563cff8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cff8e0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f9d180138b0 tx=0x7f9d18004ab0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 shutdown_connections 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d14720 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9cfffff640 0 client.16614 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cff8e0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf93e0 0x556563cff3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/1814621078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf8a30 0x556563cfee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 >> 172.21.15.3:0/1814621078 conn(0x556563cf3100 msgr2=0x556563cf4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 shutdown_connections 2024-02-09T22:54:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 1 -- 172.21.15.3:0/1814621078 wait complete. 2024-02-09T22:54:20.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.423+0000 7f9d29bf0740 20 client.16614 trim_cache size 0 max 16384 2024-02-09T22:54:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 Processor -- start 2024-02-09T22:54:20.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 -- start start 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d01590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf93e0 0x556563cff700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cfc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563be5130 con 0x556563d01f60 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55656399c720 con 0x556563cf93e0 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563cf8530 con 0x556563cf8a30 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cfc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cfc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56916/0 (socket says 172.21.15.3:56916) 2024-02-09T22:54:20.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 -- 172.21.15.3:0/3617315699 learned_addr learned my addr 172.21.15.3:0/3617315699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 -- 172.21.15.3:0/3617315699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 msgr2=0x556563d01590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d01590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 -- 172.21.15.3:0/3617315699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf93e0 msgr2=0x556563cff700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf93e0 0x556563cff700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.435+0000 7f9d255bb640 1 -- 172.21.15.3:0/3617315699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556563bfac90 con 0x556563d01f60 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d255bb640 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cfc0c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9d18000f80 tx=0x7f9d18005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb621c723eee5288 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/3617315699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800f730 con 0x556563d01f60 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/3617315699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d18029930 con 0x556563d01f60 2024-02-09T22:54:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/3617315699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18020b60 con 0x556563d01f60 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 -- 172.21.15.3:0/3617315699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 msgr2=0x556563cfc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cfc0c0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f9d18000f80 tx=0x7f9d18005cf0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 -- 172.21.15.3:0/3617315699 shutdown_connections 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563cf93e0 0x556563cff700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563d01f60 0x556563cfc0c0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/3617315699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d01590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 -- 172.21.15.3:0/3617315699 >> 172.21.15.3:0/3617315699 conn(0x556563cf3100 msgr2=0x556563d19080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 -- 172.21.15.3:0/3617315699 shutdown_connections 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.436+0000 7f9d29bf0740 1 -- 172.21.15.3:0/3617315699 wait complete. 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 -- start start 2024-02-09T22:54:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 Processor -- start 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d07ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d68810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563d08990 con 0x556563cf93e0 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563d68d50 con 0x556563d01f60 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556563d68e90 con 0x556563cf8a30 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d29bf0740 10 client.0 _ll_register_callbacks cb 0x7f9d0c0b8a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56922/0 (socket says 172.21.15.3:56922) 2024-02-09T22:54:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d1ffff640 1 -- 172.21.15.3:0/336688869 learned_addr learned my addr 172.21.15.3:0/336688869 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d255bb640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d68810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.437+0000 7f9d24dba640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d07ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d1ffff640 1 -- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 msgr2=0x556563d07ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d07ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d1ffff640 1 -- 172.21.15.3:0/336688869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 msgr2=0x556563d68810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d68810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d1ffff640 1 -- 172.21.15.3:0/336688869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556563c5e6b0 con 0x556563cf93e0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d255bb640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d1ffff640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d080e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009430 tx=0x7f9d0c0bce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f589655a48a3b23 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9cfffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9cfffff640 1 -- 172.21.15.3:0/336688869 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c0bd4d0 con 0x556563cf93e0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d0c0cf440 con 0x556563cf93e0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9cfffff640 1 -- 172.21.15.3:0/336688869 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d0c0db2e0 con 0x556563cf93e0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9cfffff640 1 -- 172.21.15.3:0/336688869 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d0c0c5570 con 0x556563cf93e0 2024-02-09T22:54:20.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 10 client.16620 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x556563c20a70 con 0x556563cf93e0 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9cfffff640 1 -- 172.21.15.3:0/336688869 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c086080 con 0x556563cf93e0 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9cfffff640 1 -- 172.21.15.3:0/336688869 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9d0c03c070 con 0x556563cf93e0 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 10 client.16620 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 10 client.16620 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 20 client.16620 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d6c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 4 client.16620 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.438+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5565639eb890 con 0x556563d12300 2024-02-09T22:54:20.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d24dba640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d6c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d24dba640 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d6c2d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=db0e42db1e591723 server_cookie=a6df700f12f0c9b2 in_seq=0 out_seq=0 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9cfffff640 10 client.16620 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9cfffff640 1 -- 172.21.15.3:0/336688869 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5565639eb890 con 0x556563d12300 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9cfffff640 10 client.16620 handle_command_reply: tid=0 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 client.16620 shutdown 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 msgr2=0x556563d080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d080e0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009430 tx=0x7f9d0c0bce40 comp rx=0 tx=0).stop 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 shutdown_connections 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x556563d12300 0x556563d6c2d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556563d01f60 0x556563d68810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556563cf93e0 0x556563d080e0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9cfffff640 0 client.16620 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 --2- 172.21.15.3:0/336688869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556563cf8a30 0x556563d07ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 >> 172.21.15.3:0/336688869 conn(0x556563cf3100 msgr2=0x556563cf19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 shutdown_connections 2024-02-09T22:54:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.439+0000 7f9d29bf0740 1 -- 172.21.15.3:0/336688869 wait complete. 2024-02-09T22:54:20.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d29bf0740 20 client.16620 trim_cache size 0 max 16384 2024-02-09T22:54:20.450 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d100a2230 msgr2=0x7f9d100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d100a2230 0x7f9d100a46f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9d180092f0 tx=0x7f9d18009890 comp rx=0 tx=0).stop 2024-02-09T22:54:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 msgr2=0x7f9d20198c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20198c50 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009230 tx=0x7f9d0c004040 comp rx=0 tx=0).stop 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 shutdown_connections 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d100a2230 0x7f9d100a46f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d201026c0 0x7f9d201a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d20101d20 0x7f9d20198c50 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 --2- 172.21.15.3:0/1158757506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d20100440 0x7f9d20198710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 >> 172.21.15.3:0/1158757506 conn(0x7f9d20077310 msgr2=0x7f9d20103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 shutdown_connections 2024-02-09T22:54:20.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.440+0000 7f9d2701e640 1 -- 172.21.15.3:0/1158757506 wait complete. 2024-02-09T22:54:20.454 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:20.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:54:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:20 smithi089 ceph-mon[36665]: pgmap v95: 97 pgs: 97 active+clean; 40 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 748 KiB/s wr, 85 op/s 2024-02-09T22:54:20.477 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:20 smithi003 ceph-mon[32678]: pgmap v95: 97 pgs: 97 active+clean; 40 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 748 KiB/s wr, 85 op/s 2024-02-09T22:54:20.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 -- 172.21.15.3:0/840428978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 msgr2=0x7f81c410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/840428978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c410cfd0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009cd0 tx=0x7f81bc02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 -- 172.21.15.3:0/840428978 shutdown_connections 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/840428978 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c410cfd0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/840428978 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c41013d0 0x7f81c4104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/840428978 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81c4100a20 0x7f81c4100e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 -- 172.21.15.3:0/840428978 >> 172.21.15.3:0/840428978 conn(0x7f81c4076630 msgr2=0x7f81c40fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 -- 172.21.15.3:0/840428978 shutdown_connections 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.556+0000 7f81cb0f4640 1 -- 172.21.15.3:0/840428978 wait complete. 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 Processor -- start 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 -- start start 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100a20 0x7f81c41985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c4198af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81c4199080 0x7f81c41a3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41a3cd0 con 0x7f81c4105400 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41a3e40 con 0x7f81c4199080 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81cb0f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81c41a4000 con 0x7f81c4100a20 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c4198af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c4198af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56942/0 (socket says 172.21.15.3:56942) 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81c3fff640 1 -- 172.21.15.3:0/268989480 learned_addr learned my addr 172.21.15.3:0/268989480 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81c8e90640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100a20 0x7f81c41985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.557+0000 7f81c9691640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81c4199080 0x7f81c41a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c3fff640 1 -- 172.21.15.3:0/268989480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100a20 msgr2=0x7f81c41985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c3fff640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100a20 0x7f81c41985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c3fff640 1 -- 172.21.15.3:0/268989480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81c4199080 msgr2=0x7f81c41a3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c3fff640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81c4199080 0x7f81c41a3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c3fff640 1 -- 172.21.15.3:0/268989480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81bc009900 con 0x7f81c4105400 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c3fff640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c4198af0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f81b0005fd0 tx=0x7f81b000dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c1ffb640 1 -- 172.21.15.3:0/268989480 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0009c30 con 0x7f81c4105400 2024-02-09T22:54:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c1ffb640 1 -- 172.21.15.3:0/268989480 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81b000de50 con 0x7f81c4105400 2024-02-09T22:54:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81c41a42e0 con 0x7f81c4105400 2024-02-09T22:54:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81c1ffb640 1 -- 172.21.15.3:0/268989480 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0010460 con 0x7f81c4105400 2024-02-09T22:54:20.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.558+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81c41a47f0 con 0x7f81c4105400 2024-02-09T22:54:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.560+0000 7f81c1ffb640 1 -- 172.21.15.3:0/268989480 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f81b00035f0 con 0x7f81c4105400 2024-02-09T22:54:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 Processor -- start 2024-02-09T22:54:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 -- start start 2024-02-09T22:54:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c1ffb640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f81b40a2230 0x7f81b40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x55854174be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541754990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 0x5585417573b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558541707020 con 0x558541754f60 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585416ba320 con 0x55854174c3e0 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81cdda1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585416b7850 con 0x55854174ba30 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f81b40a2230 0x7f81b40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c1ffb640 1 -- 172.21.15.3:0/268989480 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f81b00cdce0 con 0x7f81c4105400 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 0x5585417573b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 0x5585417573b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56954/0 (socket says 172.21.15.3:56954) 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 -- 172.21.15.3:0/1125625758 learned_addr learned my addr 172.21.15.3:0/1125625758 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c8e90640 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x55854174be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c3fff640 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541754990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 -- 172.21.15.3:0/1125625758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 msgr2=0x55854174be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x55854174be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 -- 172.21.15.3:0/1125625758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 msgr2=0x558541754990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541754990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.561+0000 7f81c9691640 1 -- 172.21.15.3:0/1125625758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558541675210 con 0x558541754f60 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81c9691640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f81b40a2230 0x7f81b40a46f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009430 tx=0x7f81bc032f80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81c9691640 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 0x5585417573b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f81bc02c700 tx=0x7f81bc0348b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27851bbd89aca91d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81aeffd640 1 -- 172.21.15.3:0/1125625758 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc034d10 con 0x558541754f60 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81aeffd640 1 -- 172.21.15.3:0/1125625758 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81bc034e70 con 0x558541754f60 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81aeffd640 1 -- 172.21.15.3:0/1125625758 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81bc004700 con 0x558541754f60 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 -- 172.21.15.3:0/1125625758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 msgr2=0x5585417573b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 0x5585417573b0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f81bc02c700 tx=0x7f81bc0348b0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 -- 172.21.15.3:0/1125625758 shutdown_connections 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541754990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558541754f60 0x5585417573b0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1125625758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x55854174be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.562+0000 7f81cdda1740 1 -- 172.21.15.3:0/1125625758 >> 172.21.15.3:0/1125625758 conn(0x558541746100 msgr2=0x5585417464e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 -- 172.21.15.3:0/1125625758 shutdown_connections 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 -- 172.21.15.3:0/1125625758 wait complete. 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 -- start start 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 Processor -- start 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x558541751f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 0x5585417524a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558541754f60 0x5585417529e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585416b3030 con 0x55854174c3e0 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558541386430 con 0x558541754f60 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558541342e00 con 0x55854174ba30 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81cdda1740 10 client.0 _ll_register_callbacks cb 0x5585416a8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 0x5585417524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 0x5585417524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56970/0 (socket says 172.21.15.3:56970) 2024-02-09T22:54:20.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.563+0000 7f81c3fff640 1 -- 172.21.15.3:0/2599612002 learned_addr learned my addr 172.21.15.3:0/2599612002 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c8e90640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x558541751f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c9691640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558541754f60 0x5585417529e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c3fff640 1 -- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 msgr2=0x558541751f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c3fff640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x558541751f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c3fff640 1 -- 172.21.15.3:0/2599612002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558541754f60 msgr2=0x5585417529e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c3fff640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558541754f60 0x5585417529e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c3fff640 1 -- 172.21.15.3:0/2599612002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81bc04a070 con 0x55854174c3e0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c9691640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558541754f60 0x5585417529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81c3fff640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 0x5585417524a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f81b009a3a0 tx=0x7f81b009a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4969aa63cabd7d29 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81a3fff640 1 -- 172.21.15.3:0/2599612002 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b0085070 con 0x55854174c3e0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81a3fff640 1 -- 172.21.15.3:0/2599612002 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f81b009a6a0 con 0x55854174c3e0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585413b27e0 con 0x55854174c3e0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81a3fff640 1 -- 172.21.15.3:0/2599612002 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81b009e7f0 con 0x55854174c3e0 2024-02-09T22:54:20.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81cdda1740 10 client.16632 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5585416b16b0 con 0x55854174c3e0 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.564+0000 7f81a3fff640 1 -- 172.21.15.3:0/2599612002 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5585413b27e0 con 0x55854174c3e0 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81a3fff640 1 -- 172.21.15.3:0/2599612002 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f81b0022020 con 0x55854174c3e0 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81cdda1740 10 client.16632 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81cdda1740 10 client.16632 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81cdda1740 20 client.16632 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81cdda1740 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558541765560 0x558541767980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81cdda1740 4 client.16632 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585413ef720 con 0x558541765560 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81c8e90640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558541765560 0x558541767980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81a3fff640 10 client.16632 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.565+0000 7f81c8e90640 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558541765560 0x558541767980 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3a1d5d4e533ce31a server_cookie=4db1e3b97cf5166a in_seq=0 out_seq=0 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81a3fff640 1 -- 172.21.15.3:0/2599612002 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5585413ef720 con 0x558541765560 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81a3fff640 10 client.16632 handle_command_reply: tid=0 2024-02-09T22:54:20.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 client.16632 shutdown 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 msgr2=0x5585417524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 0x5585417524a0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f81b009a3a0 tx=0x7f81b009a590 comp rx=0 tx=0).stop 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 shutdown_connections 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558541765560 0x558541767980 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558541754f60 0x5585417529e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81a3fff640 0 client.16632 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174c3e0 0x5585417524a0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 --2- 172.21.15.3:0/2599612002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55854174ba30 0x558541751f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 >> 172.21.15.3:0/2599612002 conn(0x558541746100 msgr2=0x558541747e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 shutdown_connections 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.566+0000 7f81cdda1740 1 -- 172.21.15.3:0/2599612002 wait complete. 2024-02-09T22:54:20.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.567+0000 7f81cdda1740 20 client.16632 trim_cache size 0 max 16384 2024-02-09T22:54:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 Processor -- start 2024-02-09T22:54:20.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 -- start start 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x5585417524d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541752a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 0x55854174f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558541638130 con 0x55854174ba30 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55854174ab90 con 0x55854174c3e0 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81cdda1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558541760f40 con 0x558541754f60 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x5585417524d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x5585417524d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56976/0 (socket says 172.21.15.3:56976) 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 -- 172.21.15.3:0/1712042994 learned_addr learned my addr 172.21.15.3:0/1712042994 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 -- 172.21.15.3:0/1712042994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 msgr2=0x55854174f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 0x55854174f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 -- 172.21.15.3:0/1712042994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 msgr2=0x558541752a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541752a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 -- 172.21.15.3:0/1712042994 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55854164dc90 con 0x55854174ba30 2024-02-09T22:54:20.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81c8e90640 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x5585417524d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x558541768220 tx=0x7f81b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96fd2235adafec08 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.579+0000 7f81ae7fc640 1 -- 172.21.15.3:0/1712042994 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b801e820 con 0x55854174ba30 2024-02-09T22:54:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81ae7fc640 1 -- 172.21.15.3:0/1712042994 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81b80268f0 con 0x55854174ba30 2024-02-09T22:54:20.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81ae7fc640 1 -- 172.21.15.3:0/1712042994 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b800f8c0 con 0x55854174ba30 2024-02-09T22:54:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 -- 172.21.15.3:0/1712042994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 msgr2=0x5585417524d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x5585417524d0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x558541768220 tx=0x7f81b80046c0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 -- 172.21.15.3:0/1712042994 shutdown_connections 2024-02-09T22:54:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x558541752a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 0x55854174f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 --2- 172.21.15.3:0/1712042994 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x5585417524d0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 -- 172.21.15.3:0/1712042994 >> 172.21.15.3:0/1712042994 conn(0x558541746100 msgr2=0x55854174d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 -- 172.21.15.3:0/1712042994 shutdown_connections 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 -- 172.21.15.3:0/1712042994 wait complete. 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 -- start start 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.580+0000 7f81cdda1740 1 Processor -- start 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x55854175a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x55854175ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 0x55854175b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55854175c870 con 0x55854174ba30 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585417bba60 con 0x55854174c3e0 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585417bbc20 con 0x558541754f60 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81cdda1740 10 client.0 _ll_register_callbacks cb 0x7f81bc037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x55854175a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x55854175a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56988/0 (socket says 172.21.15.3:56988) 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 -- 172.21.15.3:0/3300824359 learned_addr learned my addr 172.21.15.3:0/3300824359 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c3fff640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x55854175ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 -- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 msgr2=0x55854175b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 0x55854175b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 -- 172.21.15.3:0/3300824359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 msgr2=0x55854175ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x55854175ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 -- 172.21.15.3:0/3300824359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81b800a9e0 con 0x55854174ba30 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c3fff640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x55854175ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81c8e90640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x55854175a8d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f81b8005cd0 tx=0x7f81b801e920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d502957be54bda1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.581+0000 7f81a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81a3fff640 1 -- 172.21.15.3:0/3300824359 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81b8026c60 con 0x55854174ba30 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81a3fff640 1 -- 172.21.15.3:0/3300824359 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f81b8026dc0 con 0x55854174ba30 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f81bc034e70 con 0x55854174ba30 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81a3fff640 1 -- 172.21.15.3:0/3300824359 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81b800f5c0 con 0x55854174ba30 2024-02-09T22:54:20.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 10 client.16638 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x558541673a70 con 0x55854174ba30 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81a3fff640 1 -- 172.21.15.3:0/3300824359 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81b803b020 con 0x55854174ba30 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81a3fff640 1 -- 172.21.15.3:0/3300824359 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f81b801b040 con 0x55854174ba30 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 10 client.16638 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 10 client.16638 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 20 client.16638 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5585417bd490 0x5585417bf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 4 client.16638 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55854143e890 con 0x5585417bd490 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.582+0000 7f81c3fff640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5585417bd490 0x5585417bf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81c3fff640 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5585417bd490 0x5585417bf8b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8f14ba057730f466 server_cookie=76896bf795fa9de0 in_seq=0 out_seq=0 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81a3fff640 10 client.16638 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81a3fff640 1 -- 172.21.15.3:0/3300824359 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55854143e890 con 0x5585417bd490 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81a3fff640 10 client.16638 handle_command_reply: tid=0 2024-02-09T22:54:20.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 client.16638 shutdown 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 msgr2=0x55854175a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x55854175a8d0 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f81b8005cd0 tx=0x7f81b801e920 comp rx=0 tx=0).stop 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 shutdown_connections 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5585417bd490 0x5585417bf8b0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55854174c3e0 0x55854175ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81a3fff640 0 client.16638 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558541754f60 0x55854175b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 --2- 172.21.15.3:0/3300824359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55854174ba30 0x55854175a8d0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 >> 172.21.15.3:0/3300824359 conn(0x558541746100 msgr2=0x5585417449b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 shutdown_connections 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.583+0000 7f81cdda1740 1 -- 172.21.15.3:0/3300824359 wait complete. 2024-02-09T22:54:20.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cdda1740 20 client.16638 trim_cache size 0 max 16384 2024-02-09T22:54:20.593 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f81b40a2230 msgr2=0x7f81b40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f81b40a2230 0x7f81b40a46f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f81bc009430 tx=0x7f81bc032f80 comp rx=0 tx=0).stop 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 msgr2=0x7f81c4198af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c4198af0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f81b0005fd0 tx=0x7f81b000dd20 comp rx=0 tx=0).stop 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 shutdown_connections 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f81b40a2230 0x7f81b40a46f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f81c4199080 0x7f81c41a3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f81c4105400 0x7f81c4198af0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 --2- 172.21.15.3:0/268989480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f81c4100a20 0x7f81c41985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 >> 172.21.15.3:0/268989480 conn(0x7f81c4076630 msgr2=0x7f81c4103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 shutdown_connections 2024-02-09T22:54:20.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.584+0000 7f81cb0f4640 1 -- 172.21.15.3:0/268989480 wait complete. 2024-02-09T22:54:20.598 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:20.598 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:54:20.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82426 1.38 MB/sec execute 14 sec latency 2494.933 ms 2024-02-09T22:54:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:20 smithi064 ceph-mon[36638]: pgmap v95: 97 pgs: 97 active+clean; 40 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 748 KiB/s wr, 85 op/s 2024-02-09T22:54:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 -- 172.21.15.3:0/3297917467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038106500 msgr2=0x7fe03810e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/3297917467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038106500 0x7fe03810e0d0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fe030009cd0 tx=0x7fe03002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 -- 172.21.15.3:0/3297917467 shutdown_connections 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/3297917467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe038101de0 0x7fe038105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/3297917467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038106500 0x7fe03810e0d0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/3297917467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038101430 0x7fe038101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 -- 172.21.15.3:0/3297917467 >> 172.21.15.3:0/3297917467 conn(0x7fe0380fccd0 msgr2=0x7fe0380ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 -- 172.21.15.3:0/3297917467 shutdown_connections 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.732+0000 7fe03f9ac640 1 -- 172.21.15.3:0/3297917467 wait complete. 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 Processor -- start 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 -- start start 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 0x7fe03819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038106500 0x7fe03819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0381976d0 0x7fe038197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0381981a0 con 0x7fe038101430 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe038198310 con 0x7fe0381976d0 2024-02-09T22:54:20.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03f9ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0381984d0 con 0x7fe038106500 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03d748640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 0x7fe03819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03d748640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 0x7fe03819d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57010/0 (socket says 172.21.15.3:57010) 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03d748640 1 -- 172.21.15.3:0/1913586237 learned_addr learned my addr 172.21.15.3:0/1913586237 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.733+0000 7fe03cf47640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038106500 0x7fe03819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03d748640 1 -- 172.21.15.3:0/1913586237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038106500 msgr2=0x7fe03819db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03d748640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038106500 0x7fe03819db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03df49640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0381976d0 0x7fe038197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03d748640 1 -- 172.21.15.3:0/1913586237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0381976d0 msgr2=0x7fe038197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03d748640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0381976d0 0x7fe038197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03d748640 1 -- 172.21.15.3:0/1913586237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe030009900 con 0x7fe038101430 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03cf47640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038106500 0x7fe03819db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03d748640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 0x7fe03819d610 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe03400d770 tx=0x7fe03400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe02e7fc640 1 -- 172.21.15.3:0/1913586237 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe034009c30 con 0x7fe038101430 2024-02-09T22:54:20.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0381a4470 con 0x7fe038101430 2024-02-09T22:54:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe02e7fc640 1 -- 172.21.15.3:0/1913586237 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe03400de80 con 0x7fe038101430 2024-02-09T22:54:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe02e7fc640 1 -- 172.21.15.3:0/1913586237 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe034004e80 con 0x7fe038101430 2024-02-09T22:54:20.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.734+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0381a4950 con 0x7fe038101430 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.736+0000 7fe02e7fc640 1 -- 172.21.15.3:0/1913586237 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe034005020 con 0x7fe038101430 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 Processor -- start 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 -- start start 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9cfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe02e7fc640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0240a2230 0x7fe0240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9d8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9db3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a98b020 con 0x55873a9d8f60 2024-02-09T22:54:20.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a93e320 con 0x55873a9d03e0 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe04257f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a93b850 con 0x55873a9cfa30 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03d748640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0240a2230 0x7fe0240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9db3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9db3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57026/0 (socket says 172.21.15.3:57026) 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 -- 172.21.15.3:0/3788492966 learned_addr learned my addr 172.21.15.3:0/3788492966 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9cfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe02e7fc640 1 -- 172.21.15.3:0/1913586237 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe0340cdbd0 con 0x7fe038101430 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03cf47640 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9d8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 -- 172.21.15.3:0/3788492966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 msgr2=0x55873a9cfe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9cfe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 -- 172.21.15.3:0/3788492966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 msgr2=0x55873a9d8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9d8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03df49640 1 -- 172.21.15.3:0/3788492966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55873a8f9210 con 0x55873a9d8f60 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.737+0000 7fe03d748640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0240a2230 0x7fe0240a46f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe03400d740 tx=0x7fe034096170 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe03df49640 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9db3b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe0300098d0 tx=0x7fe030004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ce2cb6907a73f49 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe0237fe640 1 -- 172.21.15.3:0/3788492966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030034640 con 0x55873a9d8f60 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe0237fe640 1 -- 172.21.15.3:0/3788492966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe030032df0 con 0x55873a9d8f60 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe0237fe640 1 -- 172.21.15.3:0/3788492966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe030003c60 con 0x55873a9d8f60 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 -- 172.21.15.3:0/3788492966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 msgr2=0x55873a9db3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9db3b0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fe0300098d0 tx=0x7fe030004a40 comp rx=0 tx=0).stop 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 -- 172.21.15.3:0/3788492966 shutdown_connections 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9d8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9db3b0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 --2- 172.21.15.3:0/3788492966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9cfe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.738+0000 7fe04257f740 1 -- 172.21.15.3:0/3788492966 >> 172.21.15.3:0/3788492966 conn(0x55873a9ca100 msgr2=0x55873a9ca4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 -- 172.21.15.3:0/3788492966 shutdown_connections 2024-02-09T22:54:20.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 -- 172.21.15.3:0/3788492966 wait complete. 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 -- start start 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 Processor -- start 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9cfa30 0x55873a9d5f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9d8f60 0x55873a9d6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a937030 con 0x55873a9d03e0 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a60a430 con 0x55873a9cfa30 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a5c6e00 con 0x55873a9d8f60 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe04257f740 10 client.0 _ll_register_callbacks cb 0x55873a92c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe03cf47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe03cf47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57032/0 (socket says 172.21.15.3:57032) 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe03cf47640 1 -- 172.21.15.3:0/314185606 learned_addr learned my addr 172.21.15.3:0/314185606 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.739+0000 7fe03df49640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9cfa30 0x55873a9d5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03df49640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9d8f60 0x55873a9d6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03cf47640 1 -- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9d8f60 msgr2=0x55873a9d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03cf47640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9d8f60 0x55873a9d6980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03cf47640 1 -- 172.21.15.3:0/314185606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9cfa30 msgr2=0x55873a9d5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03cf47640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9cfa30 0x55873a9d5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03cf47640 1 -- 172.21.15.3:0/314185606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe03810e8e0 con 0x55873a9d03e0 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03df49640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9cfa30 0x55873a9d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe03cf47640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d6440 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe03806a990 tx=0x7fe028012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb53d49d2c0b2d6f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe020ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe020ff9640 1 -- 172.21.15.3:0/314185606 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028004230 con 0x55873a9d03e0 2024-02-09T22:54:20.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe020ff9640 1 -- 172.21.15.3:0/314185606 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe028004390 con 0x55873a9d03e0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55873a6367e0 con 0x55873a9d03e0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe020ff9640 1 -- 172.21.15.3:0/314185606 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe028010760 con 0x55873a9d03e0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe04257f740 10 client.16650 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55873a9356b0 con 0x55873a9d03e0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe020ff9640 1 -- 172.21.15.3:0/314185606 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55873a6367e0 con 0x55873a9d03e0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.740+0000 7fe020ff9640 1 -- 172.21.15.3:0/314185606 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fe028003c10 con 0x55873a9d03e0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe04257f740 10 client.16650 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe04257f740 10 client.16650 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe04257f740 20 client.16650 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe04257f740 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873a9e94b0 0x55873a9eb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe04257f740 4 client.16650 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55873a673720 con 0x55873a9e94b0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe03df49640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873a9e94b0 0x55873a9eb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe03df49640 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873a9e94b0 0x55873a9eb8d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a22b6c1018dc2029 server_cookie=893d382cff739c53 in_seq=0 out_seq=0 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.741+0000 7fe020ff9640 10 client.16650 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe020ff9640 1 -- 172.21.15.3:0/314185606 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55873a673720 con 0x55873a9e94b0 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe020ff9640 10 client.16650 handle_command_reply: tid=0 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 client.16650 shutdown 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 msgr2=0x55873a9d6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d6440 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fe03806a990 tx=0x7fe028012820 comp rx=0 tx=0).stop 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 shutdown_connections 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873a9e94b0 0x55873a9eb8d0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9d8f60 0x55873a9d6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d6440 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 --2- 172.21.15.3:0/314185606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9cfa30 0x55873a9d5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 >> 172.21.15.3:0/314185606 conn(0x55873a9ca100 msgr2=0x55873a9cbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe020ff9640 0 client.16650 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 shutdown_connections 2024-02-09T22:54:20.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 1 -- 172.21.15.3:0/314185606 wait complete. 2024-02-09T22:54:20.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.742+0000 7fe04257f740 20 client.16650 trim_cache size 0 max 16384 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 Processor -- start 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 -- start start 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9d8630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d8f60 0x55873a9d3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a8bc130 con 0x55873a9d03e0 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a9e7d80 con 0x55873a9d8f60 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe04257f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a9c5ab0 con 0x55873a9cfa30 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe03cf47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe03cf47640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d67a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57048/0 (socket says 172.21.15.3:57048) 2024-02-09T22:54:20.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.754+0000 7fe03cf47640 1 -- 172.21.15.3:0/2781134022 learned_addr learned my addr 172.21.15.3:0/2781134022 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03df49640 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d8f60 0x55873a9d3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03df49640 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9d8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03cf47640 1 -- 172.21.15.3:0/2781134022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 msgr2=0x55873a9d8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03cf47640 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9d8630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03cf47640 1 -- 172.21.15.3:0/2781134022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d8f60 msgr2=0x55873a9d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03cf47640 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d8f60 0x55873a9d3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03cf47640 1 -- 172.21.15.3:0/2781134022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55873a8d1c90 con 0x55873a9d03e0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03df49640 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d8f60 0x55873a9d3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe03cf47640 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d67a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe028005710 tx=0x7fe028005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ce14c87234adb8a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe022ffd640 1 -- 172.21.15.3:0/2781134022 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe02801b810 con 0x55873a9d03e0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe022ffd640 1 -- 172.21.15.3:0/2781134022 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe028012d20 con 0x55873a9d03e0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe022ffd640 1 -- 172.21.15.3:0/2781134022 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028010ac0 con 0x55873a9d03e0 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe04257f740 1 -- 172.21.15.3:0/2781134022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 msgr2=0x55873a9d67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe04257f740 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d67a0 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7fe028005710 tx=0x7fe028005740 comp rx=0 tx=0).stop 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe04257f740 1 -- 172.21.15.3:0/2781134022 shutdown_connections 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe04257f740 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d8f60 0x55873a9d3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.755+0000 7fe04257f740 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d03e0 0x55873a9d67a0 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 --2- 172.21.15.3:0/2781134022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9d8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- 172.21.15.3:0/2781134022 >> 172.21.15.3:0/2781134022 conn(0x55873a9ca100 msgr2=0x55873a9d47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- 172.21.15.3:0/2781134022 shutdown_connections 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- 172.21.15.3:0/2781134022 wait complete. 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- start start 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 Processor -- start 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9e0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9de570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9deab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a9df070 con 0x55873a9d8f60 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a9df1e0 con 0x55873a9d03e0 2024-02-09T22:54:20.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55873a9df3a0 con 0x55873a9cfa30 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe04257f740 10 client.0 _ll_register_callbacks cb 0x7fe030037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe03df49640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9deab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe03df49640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9deab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57054/0 (socket says 172.21.15.3:57054) 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe03df49640 1 -- 172.21.15.3:0/2148280311 learned_addr learned my addr 172.21.15.3:0/2148280311 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.756+0000 7fe03cf47640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9de570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9e0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 -- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 msgr2=0x55873a9e0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9e0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 -- 172.21.15.3:0/2148280311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 msgr2=0x55873a9de570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9de570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 -- 172.21.15.3:0/2148280311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55873a9356b0 con 0x55873a9d8f60 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe03df49640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9deab0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe030032650 tx=0x7fe030045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd4ace473f4d1c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe020ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe020ff9640 1 -- 172.21.15.3:0/2148280311 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0300327d0 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe020ff9640 1 -- 172.21.15.3:0/2148280311 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe030034a90 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe030032df0 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe020ff9640 1 -- 172.21.15.3:0/2148280311 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe030034c50 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe04257f740 10 client.16656 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55873a8f7a70 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe020ff9640 1 -- 172.21.15.3:0/2148280311 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe030032df0 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe020ff9640 1 -- 172.21.15.3:0/2148280311 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fe030064450 con 0x55873a9d8f60 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe04257f740 10 client.16656 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.757+0000 7fe04257f740 10 client.16656 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe04257f740 20 client.16656 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe04257f740 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873aa40cf0 0x55873aa430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe04257f740 4 client.16656 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55873a6c2890 con 0x55873aa40cf0 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe03cf47640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873aa40cf0 0x55873aa430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe03cf47640 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873aa40cf0 0x55873aa430e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=54c02477d2355038 server_cookie=b90390154dc571af in_seq=0 out_seq=0 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe020ff9640 10 client.16656 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe020ff9640 1 -- 172.21.15.3:0/2148280311 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55873a6c2890 con 0x55873aa40cf0 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe020ff9640 10 client.16656 handle_command_reply: tid=0 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.758+0000 7fe04257f740 1 client.16656 shutdown 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 msgr2=0x55873a9deab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9deab0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fe030032650 tx=0x7fe030045040 comp rx=0 tx=0).stop 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 shutdown_connections 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55873aa40cf0 0x55873aa430e0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55873a9d03e0 0x55873a9de570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55873a9d8f60 0x55873a9deab0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe020ff9640 0 client.16656 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 --2- 172.21.15.3:0/2148280311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55873a9cfa30 0x55873a9e0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 >> 172.21.15.3:0/2148280311 conn(0x55873a9ca100 msgr2=0x55873a9c89b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 shutdown_connections 2024-02-09T22:54:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 1 -- 172.21.15.3:0/2148280311 wait complete. 2024-02-09T22:54:20.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe04257f740 20 client.16656 trim_cache size 0 max 16384 2024-02-09T22:54:20.769 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0240a2230 msgr2=0x7fe0240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0240a2230 0x7fe0240a46f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe03400d740 tx=0x7fe034096170 comp rx=0 tx=0).stop 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 msgr2=0x7fe03819d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 0x7fe03819d610 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fe03400d770 tx=0x7fe03400dc30 comp rx=0 tx=0).stop 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 shutdown_connections 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0240a2230 0x7fe0240a46f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0381976d0 0x7fe038197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe038106500 0x7fe03819db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 --2- 172.21.15.3:0/1913586237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe038101430 0x7fe03819d610 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 >> 172.21.15.3:0/1913586237 conn(0x7fe0380fccd0 msgr2=0x7fe0380fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 shutdown_connections 2024-02-09T22:54:20.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.759+0000 7fe03f9ac640 1 -- 172.21.15.3:0/1913586237 wait complete. 2024-02-09T22:54:20.773 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:20.773 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:54:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 -- 172.21.15.3:0/2105766360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 msgr2=0x7efd98069120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 --2- 172.21.15.3:0/2105766360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd98069120 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7efd88009930 tx=0x7efd8802bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 -- 172.21.15.3:0/2105766360 shutdown_connections 2024-02-09T22:54:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 --2- 172.21.15.3:0/2105766360 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd980696f0 0x7efd98105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 --2- 172.21.15.3:0/2105766360 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 0x7efd9810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 --2- 172.21.15.3:0/2105766360 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd98069120 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 -- 172.21.15.3:0/2105766360 >> 172.21.15.3:0/2105766360 conn(0x7efd98077220 msgr2=0x7efd98077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 -- 172.21.15.3:0/2105766360 shutdown_connections 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.854+0000 7efd9da17640 1 -- 172.21.15.3:0/2105766360 wait complete. 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 Processor -- start 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 -- start start 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd9819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 0x7efd9819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd981976b0 0x7efd98197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd98198180 con 0x7efd98068d40 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd981982f0 con 0x7efd981976b0 2024-02-09T22:54:20.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd9da17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd981984b0 con 0x7efd98106510 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd9819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd9819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57066/0 (socket says 172.21.15.3:57066) 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd96ffd640 1 -- 172.21.15.3:0/1344378615 learned_addr learned my addr 172.21.15.3:0/1344378615 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.855+0000 7efd967fc640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 0x7efd9819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd977fe640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd981976b0 0x7efd98197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd96ffd640 1 -- 172.21.15.3:0/1344378615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 msgr2=0x7efd9819db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd96ffd640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 0x7efd9819db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd96ffd640 1 -- 172.21.15.3:0/1344378615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd981976b0 msgr2=0x7efd98197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd96ffd640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd981976b0 0x7efd98197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd96ffd640 1 -- 172.21.15.3:0/1344378615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd88009560 con 0x7efd98068d40 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd977fe640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd981976b0 0x7efd98197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd967fc640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 0x7efd9819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd96ffd640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd9819d5f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7efd88009230 tx=0x7efd88004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd9ca15640 1 -- 172.21.15.3:0/1344378615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88004440 con 0x7efd98068d40 2024-02-09T22:54:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd981a43f0 con 0x7efd98068d40 2024-02-09T22:54:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd9ca15640 1 -- 172.21.15.3:0/1344378615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd880052d0 con 0x7efd98068d40 2024-02-09T22:54:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd9ca15640 1 -- 172.21.15.3:0/1344378615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd88003d70 con 0x7efd98068d40 2024-02-09T22:54:20.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.856+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd981a4920 con 0x7efd98068d40 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 Processor -- start 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd9ca15640 1 -- 172.21.15.3:0/1344378615 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efd88005440 con 0x7efd98068d40 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 -- start start 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da237f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da238470 0x5558da245670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da247fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da1a6a00 con 0x5558da237b50 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da1a3f30 con 0x5558da238470 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efda05df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da19f710 con 0x5558da245bb0 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da237f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da237f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57076/0 (socket says 172.21.15.3:57076) 2024-02-09T22:54:20.868 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 -- 172.21.15.3:0/3746653931 learned_addr learned my addr 172.21.15.3:0/3746653931 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da238470 0x5558da245670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd967fc640 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da247fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd9ca15640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd800a2220 0x7efd800a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 -- 172.21.15.3:0/3746653931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 msgr2=0x5558da247fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da247fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 -- 172.21.15.3:0/3746653931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da238470 msgr2=0x5558da245670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da238470 0x5558da245670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 -- 172.21.15.3:0/3746653931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558da1643b0 con 0x5558da237b50 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd9ca15640 1 -- 172.21.15.3:0/1344378615 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efd880f0270 con 0x7efd98068d40 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.859+0000 7efd977fe640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd800a2220 0x7efd800a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efd977fe640 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da237f30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7efd9000e3a0 tx=0x7efd9000e860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddecbffaccb063af server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efd7dffb640 1 -- 172.21.15.3:0/3746653931 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90012950 con 0x5558da237b50 2024-02-09T22:54:20.869 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efd7dffb640 1 -- 172.21.15.3:0/3746653931 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd90012b30 con 0x5558da237b50 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efd7dffb640 1 -- 172.21.15.3:0/3746653931 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd90019670 con 0x5558da237b50 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efda05df740 1 -- 172.21.15.3:0/3746653931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 msgr2=0x5558da237f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efda05df740 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da237f30 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7efd9000e3a0 tx=0x7efd9000e860 comp rx=0 tx=0).stop 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.860+0000 7efd977fe640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd800a2220 0x7efd800a46e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd9001d000 tx=0x7efd90019d60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- 172.21.15.3:0/3746653931 shutdown_connections 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da238470 0x5558da245670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da247fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 --2- 172.21.15.3:0/3746653931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da237f30 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- 172.21.15.3:0/3746653931 >> 172.21.15.3:0/3746653931 conn(0x5558da2329d0 msgr2=0x5558da232db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- 172.21.15.3:0/3746653931 shutdown_connections 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- 172.21.15.3:0/3746653931 wait complete. 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- start start 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 Processor -- start 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da242ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.870 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da2430e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da245bb0 0x5558da243620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da144b90 con 0x5558da237b50 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d9e73250 con 0x5558da245bb0 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da125200 con 0x5558da238470 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efda05df740 10 client.0 _ll_register_callbacks cb 0x5558da15b5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.861+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da242ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da242ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57082/0 (socket says 172.21.15.3:57082) 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 -- 172.21.15.3:0/1378040718 learned_addr learned my addr 172.21.15.3:0/1378040718 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd967fc640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da2430e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd977fe640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da245bb0 0x5558da243620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 -- 172.21.15.3:0/1378040718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 msgr2=0x5558da2430e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da2430e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 -- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da245bb0 msgr2=0x5558da243620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da245bb0 0x5558da243620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.871 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 -- 172.21.15.3:0/1378040718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd90009b50 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd96ffd640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da242ba0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7efd880edb60 tx=0x7efd880f3970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b631c354216fcb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd76ffd640 1 -- 172.21.15.3:0/1378040718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd880c0070 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd76ffd640 1 -- 172.21.15.3:0/1378040718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efd880bd4a0 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5558da1c5ee0 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efd76ffd640 1 -- 172.21.15.3:0/1378040718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd880bc8e0 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efda05df740 10 client.16668 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.862+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5558da19dd90 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efd76ffd640 1 -- 172.21.15.3:0/1378040718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5558da1c5ee0 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efd76ffd640 1 -- 172.21.15.3:0/1378040718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7efd8803c070 con 0x5558da237b50 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efda05df740 10 client.16668 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efda05df740 10 client.16668 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efda05df740 20 client.16668 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.872 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efda05df740 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da255ca0 0x5558da2580c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efda05df740 4 client.16668 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5558d9e9f600 con 0x5558da255ca0 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efd967fc640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da255ca0 0x5558da2580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efd967fc640 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da255ca0 0x5558da2580c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f8fb9a8cb8ee088 server_cookie=a193c40452076815 in_seq=0 out_seq=0 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.863+0000 7efd76ffd640 10 client.16668 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efd76ffd640 1 -- 172.21.15.3:0/1378040718 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5558d9e9f600 con 0x5558da255ca0 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efd76ffd640 10 client.16668 handle_command_reply: tid=0 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 client.16668 shutdown 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 msgr2=0x5558da242ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da242ba0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7efd880edb60 tx=0x7efd880f3970 comp rx=0 tx=0).stop 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 shutdown_connections 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da255ca0 0x5558da2580c0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da245bb0 0x5558da243620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da2430e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efd76ffd640 0 client.16668 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:20.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 --2- 172.21.15.3:0/1378040718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da237b50 0x5558da242ba0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 >> 172.21.15.3:0/1378040718 conn(0x5558da2329d0 msgr2=0x5558da234720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 shutdown_connections 2024-02-09T22:54:20.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 1 -- 172.21.15.3:0/1378040718 wait complete. 2024-02-09T22:54:20.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.864+0000 7efda05df740 20 client.16668 trim_cache size 0 max 16384 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 -- 172.21.15.3:0/1353673080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 msgr2=0x7fda9410dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 --2- 172.21.15.3:0/1353673080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda9410dc50 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009ca0 tx=0x7fda8c02c180 comp rx=0 tx=0).stop 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 -- 172.21.15.3:0/1353673080 shutdown_connections 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 --2- 172.21.15.3:0/1353673080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda94102d70 0x7fda9410b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 --2- 172.21.15.3:0/1353673080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda9410dc50 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 --2- 172.21.15.3:0/1353673080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda94102450 0x7fda94102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 -- 172.21.15.3:0/1353673080 >> 172.21.15.3:0/1353673080 conn(0x7fda940772e0 msgr2=0x7fda940776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 -- 172.21.15.3:0/1353673080 shutdown_connections 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.877+0000 7fda9be76640 1 -- 172.21.15.3:0/1353673080 wait complete. 2024-02-09T22:54:20.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 Processor -- start 2024-02-09T22:54:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 -- start start 2024-02-09T22:54:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda94102450 0x7fda9419d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda94102d70 0x7fda9419d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda94197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 Processor -- start 2024-02-09T22:54:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda94197c60 con 0x7fda9410b820 2024-02-09T22:54:20.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda94197dd0 con 0x7fda94102d70 2024-02-09T22:54:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9be76640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda94197f90 con 0x7fda94102450 2024-02-09T22:54:20.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 -- start start 2024-02-09T22:54:20.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda94197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda94197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57100/0 (socket says 172.21.15.3:57100) 2024-02-09T22:54:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda99c12640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda94102450 0x7fda9419d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 -- 172.21.15.3:0/2585763091 learned_addr learned my addr 172.21.15.3:0/2585763091 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da2452d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da243440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da245bb0 0x5558da241b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda99411640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda94102d70 0x7fda9419d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da22e110 con 0x5558da245bb0 2024-02-09T22:54:20.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558d9e9f600 con 0x5558da237b50 2024-02-09T22:54:20.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efda05df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da2335a0 con 0x5558da238470 2024-02-09T22:54:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 -- 172.21.15.3:0/2585763091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda94102450 msgr2=0x7fda9419d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda94102450 0x7fda9419d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 -- 172.21.15.3:0/2585763091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda94102d70 msgr2=0x7fda9419d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda94102d70 0x7fda9419d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.878+0000 7fda9a413640 1 -- 172.21.15.3:0/2585763091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90009490 con 0x7fda9410b820 2024-02-09T22:54:20.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da2452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efd96ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da2452d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:56722/0 (socket says 172.21.15.3:56722) 2024-02-09T22:54:20.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efd96ffd640 1 -- 172.21.15.3:0/3077403497 learned_addr learned my addr 172.21.15.3:0/3077403497 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efd967fc640 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da243440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.878+0000 7efd977fe640 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da245bb0 0x5558da241b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.879+0000 7fda9a413640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda94197610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fda8c02c790 tx=0x7fda8c034670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd96ffd640 1 -- 172.21.15.3:0/3077403497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 msgr2=0x5558da243440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da243440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd96ffd640 1 -- 172.21.15.3:0/3077403497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da245bb0 msgr2=0x5558da241b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da245bb0 0x5558da241b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd96ffd640 1 -- 172.21.15.3:0/3077403497 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5558da13a820 con 0x5558da237b50 2024-02-09T22:54:20.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da2452d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd880bcd30 tx=0x7efd88003840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dae1984ab62cf8b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.879+0000 7fda8affd640 1 -- 172.21.15.3:0/2585763091 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c003b70 con 0x7fda9410b820 2024-02-09T22:54:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.879+0000 7fda8affd640 1 -- 172.21.15.3:0/2585763091 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda8c003cd0 con 0x7fda9410b820 2024-02-09T22:54:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.879+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda8c009900 con 0x7fda9410b820 2024-02-09T22:54:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.879+0000 7fda8affd640 1 -- 172.21.15.3:0/2585763091 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c0046c0 con 0x7fda9410b820 2024-02-09T22:54:20.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.879+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda941a4320 con 0x7fda9410b820 2024-02-09T22:54:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd7d7fa640 1 -- 172.21.15.3:0/3077403497 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd880c0030 con 0x5558da237b50 2024-02-09T22:54:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd7d7fa640 1 -- 172.21.15.3:0/3077403497 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd880ef030 con 0x5558da237b50 2024-02-09T22:54:20.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.879+0000 7efd7d7fa640 1 -- 172.21.15.3:0/3077403497 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd880bd000 con 0x5558da237b50 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 -- 172.21.15.3:0/3077403497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 msgr2=0x5558da2452d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da2452d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efd880bcd30 tx=0x7efd88003840 comp rx=0 tx=0).stop 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 -- 172.21.15.3:0/3077403497 shutdown_connections 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da245bb0 0x5558da241b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da238470 0x5558da243440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 --2- 172.21.15.3:0/3077403497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da2452d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 -- 172.21.15.3:0/3077403497 >> 172.21.15.3:0/3077403497 conn(0x5558da2329d0 msgr2=0x5558da241470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 -- 172.21.15.3:0/3077403497 shutdown_connections 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.880+0000 7efda05df740 1 -- 172.21.15.3:0/3077403497 wait complete. 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 -- start start 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 Processor -- start 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da24abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.892 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 0x5558da24b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da2a5130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da2a5670 con 0x5558da238470 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da2a57b0 con 0x5558da237b50 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5558da2a5970 con 0x5558da245bb0 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efda05df740 10 client.0 _ll_register_callbacks cb 0x7efd880045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.881+0000 7efd967fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 0x5558da24b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 0x5558da24b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57120/0 (socket says 172.21.15.3:57120) 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 -- 172.21.15.3:0/3881669262 learned_addr learned my addr 172.21.15.3:0/3881669262 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.893 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd977fe640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da2a5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda8affd640 1 -- 172.21.15.3:0/2585763091 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fda8c00a040 con 0x7fda9410b820 2024-02-09T22:54:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da24abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 -- 172.21.15.3:0/3881669262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 msgr2=0x5558da2a5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da2a5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 -- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 msgr2=0x5558da24abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.894 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da24abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 -- 172.21.15.3:0/3881669262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5558da19dd90 con 0x5558da238470 2024-02-09T22:54:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da24abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd977fe640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da2a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:20.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 Processor -- start 2024-02-09T22:54:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd967fc640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 0x5558da24b100 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7efd98198f00 tx=0x7efd8c01c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8699c8969a1b549d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.895 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd76ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 -- start start 2024-02-09T22:54:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd76ffd640 1 -- 172.21.15.3:0/3881669262 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8c017040 con 0x5558da238470 2024-02-09T22:54:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efd76ffd640 1 -- 172.21.15.3:0/3881669262 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efd8c003c20 con 0x5558da238470 2024-02-09T22:54:20.896 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.882+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd90012b30 con 0x5558da238470 2024-02-09T22:54:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532d33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d53283020 con 0x564d532c7a30 2024-02-09T22:54:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d53236320 con 0x564d532c83e0 2024-02-09T22:54:20.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.882+0000 7fda9ea48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d53233850 con 0x564d532d0f60 2024-02-09T22:54:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efd76ffd640 1 -- 172.21.15.3:0/3881669262 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd8c003de0 con 0x5558da238470 2024-02-09T22:54:20.898 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 10 client.16680 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.898 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57128/0 (socket says 172.21.15.3:57128) 2024-02-09T22:54:20.899 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5558da1606f0 con 0x5558da238470 2024-02-09T22:54:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 -- 172.21.15.3:0/1417318508 learned_addr learned my addr 172.21.15.3:0/1417318508 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda8affd640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda800a22b0 0x7fda800a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99c12640 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532d33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda8affd640 1 -- 172.21.15.3:0/2585763091 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda8c0f09d0 con 0x7fda9410b820 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda9a413640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda800a22b0 0x7fda800a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 -- 172.21.15.3:0/1417318508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 msgr2=0x564d532d33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532d33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 -- 172.21.15.3:0/1417318508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 msgr2=0x564d532d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efd76ffd640 1 -- 172.21.15.3:0/3881669262 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd90012b30 con 0x5558da238470 2024-02-09T22:54:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efd76ffd640 1 -- 172.21.15.3:0/3881669262 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7efd8c0243e0 con 0x5558da238470 2024-02-09T22:54:20.901 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 -- 172.21.15.3:0/1417318508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d531f1210 con 0x564d532c7a30 2024-02-09T22:54:20.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 10 client.16680 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 10 client.16680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 20 client.16680 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da2ad2f0 0x5558da2af710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 4 client.16680 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:54:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5558d9ef01b0 con 0x5558da2ad2f0 2024-02-09T22:54:20.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda99411640 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532c7e10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fda900135b0 tx=0x7fda90009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fa8480003a2406 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.883+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da2ad2f0 0x5558da2af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda7ffff640 1 -- 172.21.15.3:0/1417318508 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90013270 con 0x564d532c7a30 2024-02-09T22:54:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda7ffff640 1 -- 172.21.15.3:0/1417318508 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda90005500 con 0x564d532c7a30 2024-02-09T22:54:20.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.883+0000 7fda7ffff640 1 -- 172.21.15.3:0/1417318508 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9000f780 con 0x564d532c7a30 2024-02-09T22:54:20.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.884+0000 7efd76ffd640 10 client.16680 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:20.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.884+0000 7efd96ffd640 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da2ad2f0 0x5558da2af710 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4694fe482531a7cf server_cookie=19c53940df4f2be0 in_seq=0 out_seq=0 2024-02-09T22:54:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 -- 172.21.15.3:0/1417318508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 msgr2=0x564d532c7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532c7e10 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fda900135b0 tx=0x7fda90009890 comp rx=0 tx=0).stop 2024-02-09T22:54:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9a413640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda800a22b0 0x7fda800a4770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0b9ed0 tx=0x7fda8c02c180 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 -- 172.21.15.3:0/1417318508 shutdown_connections 2024-02-09T22:54:20.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532d33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 --2- 172.21.15.3:0/1417318508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532c7e10 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 -- 172.21.15.3:0/1417318508 >> 172.21.15.3:0/1417318508 conn(0x564d532c2100 msgr2=0x564d532c24e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 -- 172.21.15.3:0/1417318508 shutdown_connections 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.884+0000 7fda9ea48740 1 -- 172.21.15.3:0/1417318508 wait complete. 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 -- start start 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 Processor -- start 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d5322f030 con 0x564d532c7a30 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d52f02430 con 0x564d532c83e0 2024-02-09T22:54:20.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d52ebee00 con 0x564d532d0f60 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532cdfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57134/0 (socket says 172.21.15.3:57134) 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda99411640 1 -- 172.21.15.3:0/2955928091 learned_addr learned my addr 172.21.15.3:0/2955928091 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda99c12640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda9ea48740 10 client.0 _ll_register_callbacks cb 0x564d53224030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.885+0000 7fda99411640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99c12640 1 -- 172.21.15.3:0/2955928091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 msgr2=0x564d532cea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99c12640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99c12640 1 -- 172.21.15.3:0/2955928091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 msgr2=0x564d532cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99c12640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99c12640 1 -- 172.21.15.3:0/2955928091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda90009dd0 con 0x564d532c83e0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99411640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda99c12640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x564d532c87e0 tx=0x7fda84012b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=616662c38c24432a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:54:20.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda7d7fa640 1 -- 172.21.15.3:0/2955928091 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda84017070 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda7d7fa640 1 -- 172.21.15.3:0/2955928091 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda84004020 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d52f2e7e0 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda7d7fa640 1 -- 172.21.15.3:0/2955928091 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda840041e0 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda9ea48740 10 client.26051 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.886+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564d5322d6b0 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda7d7fa640 1 -- 172.21.15.3:0/2955928091 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564d52f2e7e0 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda7d7fa640 1 -- 172.21.15.3:0/2955928091 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fda84003a30 con 0x564d532c83e0 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda9ea48740 10 client.26051 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda9ea48740 10 client.26051 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda9ea48740 20 client.26051 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda9ea48740 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d532dfd00 0x564d532e2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda9ea48740 4 client.26051 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d52f6b720 con 0x564d532dfd00 2024-02-09T22:54:20.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.887+0000 7fda99411640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d532dfd00 0x564d532e2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda99411640 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d532dfd00 0x564d532e2120 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7e5fa9f4fba149b1 server_cookie=1b9e136b5fa1fd8e in_seq=0 out_seq=0 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda7d7fa640 10 client.26051 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda7d7fa640 1 -- 172.21.15.3:0/2955928091 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d52f6b720 con 0x564d532dfd00 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda7d7fa640 10 client.26051 handle_command_reply: tid=0 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda9ea48740 1 client.26051 shutdown 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 msgr2=0x564d532ce500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.888+0000 7fda9ea48740 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce500 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x564d532c87e0 tx=0x7fda84012b40 comp rx=0 tx=0).stop 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 shutdown_connections 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d532dfd00 0x564d532e2120 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce500 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 --2- 172.21.15.3:0/2955928091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532cdfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 >> 172.21.15.3:0/2955928091 conn(0x564d532c2100 msgr2=0x564d532c3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda7d7fa640 0 client.26051 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:20.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 shutdown_connections 2024-02-09T22:54:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 1 -- 172.21.15.3:0/2955928091 wait complete. 2024-02-09T22:54:20.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.889+0000 7fda9ea48740 20 client.26051 trim_cache size 0 max 16384 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.906+0000 7fda9ea48740 1 Processor -- start 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.906+0000 7fda9ea48740 1 -- start start 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.906+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d06f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda9ea48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d531b4130 con 0x564d532c7a30 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda9ea48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d532c2a40 con 0x564d532c83e0 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda9ea48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d532f1440 con 0x564d532d0f60 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57136/0 (socket says 172.21.15.3:57136) 2024-02-09T22:54:20.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 -- 172.21.15.3:0/539715620 learned_addr learned my addr 172.21.15.3:0/539715620 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99c12640 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 -- 172.21.15.3:0/539715620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 msgr2=0x564d532cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 -- 172.21.15.3:0/539715620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 msgr2=0x564d532ce860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 -- 172.21.15.3:0/539715620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d531c9c90 con 0x564d532c7a30 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99c12640 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda99411640 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d06f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fda90004cb0 tx=0x7fda90013400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5af23e03bfc35d0b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda7f7fe640 1 -- 172.21.15.3:0/539715620 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda900131d0 con 0x564d532c7a30 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda7f7fe640 1 -- 172.21.15.3:0/539715620 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda90004020 con 0x564d532c7a30 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.907+0000 7fda7f7fe640 1 -- 172.21.15.3:0/539715620 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda900039e0 con 0x564d532c7a30 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 -- 172.21.15.3:0/539715620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 msgr2=0x564d532d06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d06f0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fda90004cb0 tx=0x7fda90013400 comp rx=0 tx=0).stop 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 -- 172.21.15.3:0/539715620 shutdown_connections 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532ce860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d532cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 --2- 172.21.15.3:0/539715620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d06f0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 -- 172.21.15.3:0/539715620 >> 172.21.15.3:0/539715620 conn(0x564d532c2100 msgr2=0x564d532c9250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 -- 172.21.15.3:0/539715620 shutdown_connections 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 -- 172.21.15.3:0/539715620 wait complete. 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 -- start start 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.908+0000 7fda9ea48740 1 Processor -- start 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d53340340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d53340880 con 0x564d532c7a30 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d533409c0 con 0x564d532c83e0 2024-02-09T22:54:20.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d53340b00 con 0x564d532d0f60 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda9ea48740 10 client.0 _ll_register_callbacks cb 0x7fda900097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d71d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57146/0 (socket says 172.21.15.3:57146) 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 -- 172.21.15.3:0/784513604 learned_addr learned my addr 172.21.15.3:0/784513604 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99c12640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d53340340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 -- 172.21.15.3:0/784513604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 msgr2=0x564d53340340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d53340340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 -- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 msgr2=0x564d532d7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 -- 172.21.15.3:0/784513604 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda90027040 con 0x564d532c7a30 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99c12640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda99411640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d71d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fda90005d60 tx=0x7fda90013540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61f6d0ed9e4399f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:20.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.909+0000 7fda7d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda7d7fa640 1 -- 172.21.15.3:0/784513604 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda900043a0 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda7d7fa640 1 -- 172.21.15.3:0/784513604 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fda90004e50 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda90005500 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda7d7fa640 1 -- 172.21.15.3:0/784513604 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda90005010 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 10 client.16692 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564d531efa70 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda7d7fa640 1 -- 172.21.15.3:0/784513604 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda90041020 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda7d7fa640 1 -- 172.21.15.3:0/784513604 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fda9001d070 con 0x564d532c7a30 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 10 client.16692 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 10 client.16692 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 20 client.16692 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d53341eb0 0x564d533442a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 4 client.16692 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:20.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564d52fba890 con 0x564d53341eb0 2024-02-09T22:54:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efd76ffd640 1 -- 172.21.15.3:0/3881669262 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5558d9ef01b0 con 0x5558da2ad2f0 2024-02-09T22:54:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efd76ffd640 10 client.16680 handle_command_reply: tid=0 2024-02-09T22:54:20.922 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efda05df740 1 client.16680 shutdown 2024-02-09T22:54:20.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.910+0000 7fda99c12640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d53341eb0 0x564d533442a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 msgr2=0x5558da24b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efda05df740 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 0x5558da24b100 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7efd98198f00 tx=0x7efd8c01c6e0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.923 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 shutdown_connections 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.910+0000 7efda05df740 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5558da2ad2f0 0x5558da2af710 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5558da245bb0 0x5558da2a5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5558da238470 0x5558da24b100 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 1 --2- 172.21.15.3:0/3881669262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5558da237b50 0x5558da24abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 >> 172.21.15.3:0/3881669262 conn(0x5558da2329d0 msgr2=0x5558da2311a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd76ffd640 0 client.16680 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 shutdown_connections 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 1 -- 172.21.15.3:0/3881669262 wait complete. 2024-02-09T22:54:20.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efda05df740 20 client.16680 trim_cache size 0 max 16384 2024-02-09T22:54:20.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda7d7fa640 10 client.16692 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:20.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda99c12640 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d53341eb0 0x564d533442a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bf494fcd4f0a0b4e server_cookie=551d36658d4907be in_seq=0 out_seq=0 2024-02-09T22:54:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:54:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:54:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:54:20.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda7d7fa640 1 -- 172.21.15.3:0/784513604 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564d52fba890 con 0x564d53341eb0 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda7d7fa640 10 client.16692 handle_command_reply: tid=0 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 client.16692 shutdown 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 msgr2=0x564d532d71d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d71d0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fda90005d60 tx=0x7fda90013540 comp rx=0 tx=0).stop 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 shutdown_connections 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x564d53341eb0 0x564d533442a0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d532c83e0 0x564d532d7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d532d0f60 0x564d53340340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 --2- 172.21.15.3:0/784513604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d532c7a30 0x564d532d71d0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda7d7fa640 0 client.16692 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:20.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 >> 172.21.15.3:0/784513604 conn(0x564d532c2100 msgr2=0x564d532c09b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd800a2220 msgr2=0x7efd800a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd800a2220 0x7efd800a46e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7efd9001d000 tx=0x7efd90019d60 comp rx=0 tx=0).stop 2024-02-09T22:54:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 msgr2=0x7efd9819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd9819d5f0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7efd88009230 tx=0x7efd88004040 comp rx=0 tx=0).stop 2024-02-09T22:54:20.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 shutdown_connections 2024-02-09T22:54:20.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 1 -- 172.21.15.3:0/784513604 wait complete. 2024-02-09T22:54:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 shutdown_connections 2024-02-09T22:54:20.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd800a2220 0x7efd800a46e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd981976b0 0x7efd98197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd98106510 0x7efd9819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 --2- 172.21.15.3:0/1344378615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd98068d40 0x7efd9819d5f0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.911+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 >> 172.21.15.3:0/1344378615 conn(0x7efd98077220 msgr2=0x7efd980fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.911+0000 7fda9ea48740 20 client.16692 trim_cache size 0 max 16384 2024-02-09T22:54:20.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.912+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 shutdown_connections 2024-02-09T22:54:20.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:20.912+0000 7efd9da17640 1 -- 172.21.15.3:0/1344378615 wait complete. 2024-02-09T22:54:20.930 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda800a22b0 msgr2=0x7fda800a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda800a22b0 0x7fda800a4770 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0b9ed0 tx=0x7fda8c02c180 comp rx=0 tx=0).stop 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 msgr2=0x7fda94197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda94197610 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fda8c02c790 tx=0x7fda8c034670 comp rx=0 tx=0).stop 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 shutdown_connections 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fda800a22b0 0x7fda800a4770 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fda94102d70 0x7fda9419d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fda9410b820 0x7fda94197610 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 --2- 172.21.15.3:0/2585763091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fda94102450 0x7fda9419d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 >> 172.21.15.3:0/2585763091 conn(0x7fda940772e0 msgr2=0x7fda940fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 shutdown_connections 2024-02-09T22:54:20.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:20.912+0000 7fda9be76640 1 -- 172.21.15.3:0/2585763091 wait complete. 2024-02-09T22:54:20.932 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:20.932 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:54:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 -- 172.21.15.3:0/2966731136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 msgr2=0x7f9888100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 --2- 172.21.15.3:0/2966731136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888100710 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f987c009930 tx=0x7f987c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 -- 172.21.15.3:0/2966731136 shutdown_connections 2024-02-09T22:54:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 --2- 172.21.15.3:0/2966731136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 0x7f988810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 --2- 172.21.15.3:0/2966731136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888100ce0 0x7f9888104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 --2- 172.21.15.3:0/2966731136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888100710 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 -- 172.21.15.3:0/2966731136 >> 172.21.15.3:0/2966731136 conn(0x7f9888077250 msgr2=0x7f9888077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 -- 172.21.15.3:0/2966731136 shutdown_connections 2024-02-09T22:54:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 -- 172.21.15.3:0/2966731136 wait complete. 2024-02-09T22:54:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 Processor -- start 2024-02-09T22:54:21.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.034+0000 7f988cce1640 1 -- start start 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888198840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 0x7f9888198d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888199310 0x7f98881a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881a3f90 con 0x7f9888100330 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881a4100 con 0x7f9888105400 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98881a42c0 con 0x7f9888199310 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888198840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888198840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57168/0 (socket says 172.21.15.3:57168) 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 -- 172.21.15.3:0/3948093635 learned_addr learned my addr 172.21.15.3:0/3948093635 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888199310 0x7f98881a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 0x7f9888198d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 -- 172.21.15.3:0/3948093635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888199310 msgr2=0x7f98881a38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888199310 0x7f98881a38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 -- 172.21.15.3:0/3948093635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 msgr2=0x7f9888198d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 0x7f9888198d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 -- 172.21.15.3:0/3948093635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f987c009560 con 0x7f9888100330 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888199310 0x7f98881a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 0x7f9888198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988659c640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888198840 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f987c005940 tx=0x7f987c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f98777fe640 1 -- 172.21.15.3:0/3948093635 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0044e0 con 0x7f9888100330 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f98777fe640 1 -- 172.21.15.3:0/3948093635 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f987c02cbc0 con 0x7f9888100330 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98881a4540 con 0x7f9888100330 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.035+0000 7f98777fe640 1 -- 172.21.15.3:0/3948093635 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c003e20 con 0x7f9888100330 2024-02-09T22:54:21.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.036+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98881a4aa0 con 0x7f9888100330 2024-02-09T22:54:21.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.037+0000 7f98777fe640 1 -- 172.21.15.3:0/3948093635 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f987c0035f0 con 0x7f9888100330 2024-02-09T22:54:21.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 Processor -- start 2024-02-09T22:54:21.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f98777fe640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f98700a22b0 0x7f98700a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f98700a22b0 0x7f98700a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f98777fe640 1 -- 172.21.15.3:0/3948093635 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f987c0f0310 con 0x7f9888100330 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 -- start start 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01121cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e011222c0 0x564e0112adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e010dd020 con 0x564e01121910 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e01090320 con 0x564e0112b3c0 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988f98a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e0108d850 con 0x564e011222c0 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01121cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.038+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01121cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57170/0 (socket says 172.21.15.3:57170) 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 -- 172.21.15.3:0/181468177 learned_addr learned my addr 172.21.15.3:0/181468177 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:21.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f98700a22b0 0x7f98700a4770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9880009f20 tx=0x7f98800094f0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f9885d9b640 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e011222c0 0x564e0112adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 -- 172.21.15.3:0/181468177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e011222c0 msgr2=0x564e0112adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f9886d9d640 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e011222c0 0x564e0112adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 -- 172.21.15.3:0/181468177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 msgr2=0x564e0112b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 -- 172.21.15.3:0/181468177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e0104b210 con 0x564e01121910 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f988659c640 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01121cf0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f987c0bcdd0 tx=0x7f987c0bc860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=744a696d317741af server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f9874ff9640 1 -- 172.21.15.3:0/181468177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0c55e0 con 0x564e01121910 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f9874ff9640 1 -- 172.21.15.3:0/181468177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f987c0d0e90 con 0x564e01121910 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.039+0000 7f9874ff9640 1 -- 172.21.15.3:0/181468177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0bd2a0 con 0x564e01121910 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 -- 172.21.15.3:0/181468177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 msgr2=0x564e01121cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01121cf0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f987c0bcdd0 tx=0x7f987c0bc860 comp rx=0 tx=0).stop 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 -- 172.21.15.3:0/181468177 shutdown_connections 2024-02-09T22:54:21.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e011222c0 0x564e0112adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 --2- 172.21.15.3:0/181468177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01121cf0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 -- 172.21.15.3:0/181468177 >> 172.21.15.3:0/181468177 conn(0x564e0111c100 msgr2=0x564e0111c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 -- 172.21.15.3:0/181468177 shutdown_connections 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 -- 172.21.15.3:0/181468177 wait complete. 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 -- start start 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 Processor -- start 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.040+0000 7f988f98a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e01121910 0x564e0112ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 0x564e0112f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e01089030 con 0x564e011222c0 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e00d5c430 con 0x564e0112b3c0 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e00d18e00 con 0x564e01121910 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 10 client.0 _ll_register_callbacks cb 0x564e0107e030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:21.050 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 0x564e0112f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 0x564e0112f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57180/0 (socket says 172.21.15.3:57180) 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 -- 172.21.15.3:0/3328132403 learned_addr learned my addr 172.21.15.3:0/3328132403 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988659c640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e01121910 0x564e0112ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 -- 172.21.15.3:0/3328132403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e01121910 msgr2=0x564e0112ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e01121910 0x564e0112ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 -- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 msgr2=0x564e0112f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 -- 172.21.15.3:0/3328132403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98881a4aa0 con 0x564e011222c0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 0x564e0112f390 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f987800e2a0 tx=0x7f987800e760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=614cefde7f01750 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f986dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f986dffb640 1 -- 172.21.15.3:0/3328132403 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9878013660 con 0x564e011222c0 2024-02-09T22:54:21.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f986dffb640 1 -- 172.21.15.3:0/3328132403 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f98780137c0 con 0x564e011222c0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564e00d887e0 con 0x564e011222c0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f986dffb640 1 -- 172.21.15.3:0/3328132403 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9878013980 con 0x564e011222c0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 10 client.16704 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.041+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564e010876b0 con 0x564e011222c0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f986dffb640 1 -- 172.21.15.3:0/3328132403 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564e00d887e0 con 0x564e011222c0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f986dffb640 1 -- 172.21.15.3:0/3328132403 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f9878011070 con 0x564e011222c0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988f98a740 10 client.16704 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988f98a740 10 client.16704 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988f98a740 20 client.16704 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988f98a740 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e0113f7e0 0x564e01141c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988f98a740 4 client.16704 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564e00dc5720 con 0x564e0113f7e0 2024-02-09T22:54:21.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988659c640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e0113f7e0 0x564e01141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f988659c640 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e0113f7e0 0x564e01141c00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca0240cb8bd23d06 server_cookie=ebf0f033fe93cf2f in_seq=0 out_seq=0 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.042+0000 7f986dffb640 10 client.16704 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f986dffb640 1 -- 172.21.15.3:0/3328132403 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564e00dc5720 con 0x564e0113f7e0 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f986dffb640 10 client.16704 handle_command_reply: tid=0 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 client.16704 shutdown 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 msgr2=0x564e0112f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 0x564e0112f390 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f987800e2a0 tx=0x7f987800e760 comp rx=0 tx=0).stop 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 shutdown_connections 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e0113f7e0 0x564e01141c00 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e0112b3c0 0x564e0112f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f986dffb640 0 client.16704 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e011222c0 0x564e0112f390 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 --2- 172.21.15.3:0/3328132403 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e01121910 0x564e0112ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 >> 172.21.15.3:0/3328132403 conn(0x564e0111c100 msgr2=0x564e0111de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:21.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 shutdown_connections 2024-02-09T22:54:21.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 1 -- 172.21.15.3:0/3328132403 wait complete. 2024-02-09T22:54:21.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.043+0000 7f988f98a740 20 client.16704 trim_cache size 0 max 16384 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 Processor -- start 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 -- start start 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01131580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e0112ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e0112c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e0100e130 con 0x564e01121910 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e0113dee0 con 0x564e011222c0 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988f98a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e01129fc0 con 0x564e0112b3c0 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01131580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01131580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57196/0 (socket says 172.21.15.3:57196) 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 -- 172.21.15.3:0/1052122121 learned_addr learned my addr 172.21.15.3:0/1052122121 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f9886d9d640 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e0112c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 -- 172.21.15.3:0/1052122121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 msgr2=0x564e0112c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e0112c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 -- 172.21.15.3:0/1052122121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 msgr2=0x564e0112ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f9885d9b640 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e0112ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e0112ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f988659c640 1 -- 172.21.15.3:0/1052122121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564e01023c90 con 0x564e01121910 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.055+0000 7f9885d9b640 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e0112ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988659c640 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01131580 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b9810 tx=0x7f987c0bc8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91291eb4809b9e1e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f986ffff640 1 -- 172.21.15.3:0/1052122121 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0c0f90 con 0x564e01121910 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f986ffff640 1 -- 172.21.15.3:0/1052122121 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f987c0db070 con 0x564e01121910 2024-02-09T22:54:21.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f986ffff640 1 -- 172.21.15.3:0/1052122121 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0bd0a0 con 0x564e01121910 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 -- 172.21.15.3:0/1052122121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 msgr2=0x564e01131580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01131580 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f987c0b9810 tx=0x7f987c0bc8d0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 -- 172.21.15.3:0/1052122121 shutdown_connections 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e0112ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e0112c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 --2- 172.21.15.3:0/1052122121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01131580 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 -- 172.21.15.3:0/1052122121 >> 172.21.15.3:0/1052122121 conn(0x564e0111c100 msgr2=0x564e0112a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 -- 172.21.15.3:0/1052122121 shutdown_connections 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.056+0000 7f988f98a740 1 -- 172.21.15.3:0/1052122121 wait complete. 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 -- start start 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 Processor -- start 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01134f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e01135470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e01195d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e01135c80 con 0x564e01121910 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e01135df0 con 0x564e011222c0 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564e011962d0 con 0x564e0112b3c0 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988f98a740 10 client.0 _ll_register_callbacks cb 0x7f987c005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01134f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57208/0 (socket says 172.21.15.3:57208) 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 -- 172.21.15.3:0/3062910477 learned_addr learned my addr 172.21.15.3:0/3062910477 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f9886d9d640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e01195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e01135470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 -- 172.21.15.3:0/3062910477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 msgr2=0x564e01195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e01195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 -- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 msgr2=0x564e01135470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e01135470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.057+0000 7f988659c640 1 -- 172.21.15.3:0/3062910477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564e0104b210 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988659c640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01134f30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f987c0f02e0 tx=0x7f987c005750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b675973aea8506c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f986dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f986dffb640 1 -- 172.21.15.3:0/3062910477 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c0c0f90 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f986dffb640 1 -- 172.21.15.3:0/3062910477 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f987c0edc50 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f987c0d0e90 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f986dffb640 1 -- 172.21.15.3:0/3062910477 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7f987c0ede50 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 10 client.16710 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x564e01049a70 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f986dffb640 1 -- 172.21.15.3:0/3062910477 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f987c0d0e90 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f986dffb640 1 -- 172.21.15.3:0/3062910477 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7f987c0a3030 con 0x564e01121910 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 10 client.16710 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 10 client.16710 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 20 client.16710 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e011975a0 0x564e011999c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:21.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 4 client.16710 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.058+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564e00e14890 con 0x564e011975a0 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e011975a0 0x564e011999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f9885d9b640 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e011975a0 0x564e011999c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9d2507335f9dc38 server_cookie=ac83bd0147d06d6 in_seq=0 out_seq=0 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f986dffb640 10 client.16710 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f986dffb640 1 -- 172.21.15.3:0/3062910477 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564e00e14890 con 0x564e011975a0 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f986dffb640 10 client.16710 handle_command_reply: tid=0 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 client.16710 shutdown 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 msgr2=0x564e01134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01134f30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f987c0f02e0 tx=0x7f987c005750 comp rx=0 tx=0).stop 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 shutdown_connections 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564e011975a0 0x564e011999c0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564e011222c0 0x564e01135470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564e0112b3c0 0x564e01195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f986dffb640 0 client.16710 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 --2- 172.21.15.3:0/3062910477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564e01121910 0x564e01134f30 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 >> 172.21.15.3:0/3062910477 conn(0x564e0111c100 msgr2=0x564e0111a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.059+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 shutdown_connections 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988f98a740 1 -- 172.21.15.3:0/3062910477 wait complete. 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988f98a740 20 client.16710 trim_cache size 0 max 16384 2024-02-09T22:54:21.069 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f98700a22b0 msgr2=0x7f98700a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f98700a22b0 0x7f98700a4770 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9880009f20 tx=0x7f98800094f0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 msgr2=0x7f9888198840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888198840 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f987c005940 tx=0x7f987c0042f0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 shutdown_connections 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f98700a22b0 0x7f98700a4770 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9888105400 0x7f9888198d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9888199310 0x7f98881a38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 --2- 172.21.15.3:0/3948093635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9888100330 0x7f9888198840 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:21.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 >> 172.21.15.3:0/3948093635 conn(0x7f9888077250 msgr2=0x7f98880fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:21.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 shutdown_connections 2024-02-09T22:54:21.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:21.060+0000 7f988cce1640 1 -- 172.21.15.3:0/3948093635 wait complete. 2024-02-09T22:54:21.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:21 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3765887076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:54:21.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:21 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3765887076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:54:21.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 82426 1.29 MB/sec execute 15 sec latency 3495.093 ms 2024-02-09T22:54:21.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:21 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3765887076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:54:22.074 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:54:22.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:54:22.074 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.180+0000 7ff483393640 1 -- 172.21.15.3:0/974742000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c068810 msgr2=0x7ff47c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.180+0000 7ff483393640 1 --2- 172.21.15.3:0/974742000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c068810 0x7ff47c068bf0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7ff474009930 tx=0x7ff47402bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.180+0000 7ff483393640 1 -- 172.21.15.3:0/974742000 shutdown_connections 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.180+0000 7ff483393640 1 --2- 172.21.15.3:0/974742000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c0691c0 0x7ff47c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.180+0000 7ff483393640 1 --2- 172.21.15.3:0/974742000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c106570 0x7ff47c10e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.181+0000 7ff483393640 1 --2- 172.21.15.3:0/974742000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c068810 0x7ff47c068bf0 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.181+0000 7ff483393640 1 -- 172.21.15.3:0/974742000 >> 172.21.15.3:0/974742000 conn(0x7ff47c077230 msgr2=0x7ff47c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.181+0000 7ff483393640 1 -- 172.21.15.3:0/974742000 shutdown_connections 2024-02-09T22:54:22.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.181+0000 7ff483393640 1 -- 172.21.15.3:0/974742000 wait complete. 2024-02-09T22:54:22.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.181+0000 7ff483393640 1 Processor -- start 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.181+0000 7ff483393640 1 -- start start 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c068810 0x7ff47c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 0x7ff47c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197680 0x7ff47c197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c198150 con 0x7ff47c106570 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c1982c0 con 0x7ff47c197680 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c198480 con 0x7ff47c068810 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 0x7ff47c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 0x7ff47c19db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57228/0 (socket says 172.21.15.3:57228) 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 -- 172.21.15.3:0/2981902535 learned_addr learned my addr 172.21.15.3:0/2981902535 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48112f640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c068810 0x7ff47c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff481930640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197680 0x7ff47c197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 -- 172.21.15.3:0/2981902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c068810 msgr2=0x7ff47c19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c068810 0x7ff47c19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 -- 172.21.15.3:0/2981902535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197680 msgr2=0x7ff47c197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197680 0x7ff47c197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 -- 172.21.15.3:0/2981902535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff474009560 con 0x7ff47c106570 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff48092e640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 0x7ff47c19db00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e9f0 tx=0x7ff47000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff47a7fc640 1 -- 172.21.15.3:0/2981902535 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4700096a0 con 0x7ff47c106570 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff47a7fc640 1 -- 172.21.15.3:0/2981902535 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff47000ccc0 con 0x7ff47c106570 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c1a4420 con 0x7ff47c106570 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.182+0000 7ff47a7fc640 1 -- 172.21.15.3:0/2981902535 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff470018860 con 0x7ff47c106570 2024-02-09T22:54:22.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.183+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c1a4900 con 0x7ff47c106570 2024-02-09T22:54:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff47a7fc640 1 -- 172.21.15.3:0/2981902535 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff470010aa0 con 0x7ff47c106570 2024-02-09T22:54:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 Processor -- start 2024-02-09T22:54:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 -- start start 2024-02-09T22:54:22.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 0x5566ea5bf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5c8690 0x5566ea5caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea52d000 con 0x5566ea5c8690 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea52a530 con 0x5566ea5bf190 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.185+0000 7ff485f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea525d10 con 0x5566ea5bfab0 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48092e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5c8690 0x5566ea5caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff481930640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 0x5566ea5bf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46500/0 (socket says 172.21.15.3:46500) 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff481930640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 0x5566ea5bf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:56858/0 (socket says 172.21.15.3:56858) 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 -- 172.21.15.3:0/999903048 learned_addr learned my addr 172.21.15.3:0/999903048 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 -- 172.21.15.3:0/999903048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 msgr2=0x5566ea5bf570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 0x5566ea5bf570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 -- 172.21.15.3:0/999903048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5c8690 msgr2=0x5566ea5caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5c8690 0x5566ea5caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 -- 172.21.15.3:0/999903048 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566ea4cc210 con 0x5566ea5bfab0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff481930640 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 0x5566ea5bf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff47a7fc640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4640a2230 0x7ff4640a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff481930640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4640a2230 0x7ff4640a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff48112f640 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c8150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff474036040 tx=0x7ff47402c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb2e81a447cc28c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff47a7fc640 1 -- 172.21.15.3:0/2981902535 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff470014070 con 0x7ff47c106570 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff4637fe640 1 -- 172.21.15.3:0/999903048 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474004800 con 0x5566ea5bfab0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff4637fe640 1 -- 172.21.15.3:0/999903048 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff474004960 con 0x5566ea5bfab0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.186+0000 7ff4637fe640 1 -- 172.21.15.3:0/999903048 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4740050e0 con 0x5566ea5bfab0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 -- 172.21.15.3:0/999903048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 msgr2=0x5566ea5c8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff481930640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4640a2230 0x7ff4640a46f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009fd0 tx=0x7ff46c009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c8150 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7ff474036040 tx=0x7ff47402c5d0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 -- 172.21.15.3:0/999903048 shutdown_connections 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5c8690 0x5566ea5caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c8150 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 --2- 172.21.15.3:0/999903048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bf190 0x5566ea5bf570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 -- 172.21.15.3:0/999903048 >> 172.21.15.3:0/999903048 conn(0x5566ea5b98c0 msgr2=0x5566ea5b9ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 -- 172.21.15.3:0/999903048 shutdown_connections 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 -- 172.21.15.3:0/999903048 wait complete. 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.187+0000 7ff485f6f740 1 -- start start 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 Processor -- start 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bf190 0x5566ea5c5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea4cae60 con 0x5566ea5bfab0 2024-02-09T22:54:22.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea1f94b0 con 0x5566ea5c8690 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea4ab670 con 0x5566ea5bf190 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff485f6f740 10 client.0 _ll_register_callbacks cb 0x5566ea4c3190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57242/0 (socket says 172.21.15.3:57242) 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 -- 172.21.15.3:0/2511184194 learned_addr learned my addr 172.21.15.3:0/2511184194 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48112f640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bf190 0x5566ea5c5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff481930640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 -- 172.21.15.3:0/2511184194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bf190 msgr2=0x5566ea5c5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bf190 0x5566ea5c5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 -- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 msgr2=0x5566ea5c60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48092e640 1 -- 172.21.15.3:0/2511184194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c10e930 con 0x5566ea5bfab0 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.188+0000 7ff48112f640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bf190 0x5566ea5c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff48092e640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5b90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e9c0 tx=0x7ff47000cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adeb4d77aeab4aa1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff460ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff460ff9640 1 -- 172.21.15.3:0/2511184194 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47009a4e0 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff460ff9640 1 -- 172.21.15.3:0/2511184194 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff47009ae90 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5566ea225860 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff460ff9640 1 -- 172.21.15.3:0/2511184194 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff47009a640 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 10 client.16722 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5566ea524390 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff460ff9640 1 -- 172.21.15.3:0/2511184194 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5566ea225860 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff460ff9640 1 -- 172.21.15.3:0/2511184194 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7ff470099ee0 con 0x5566ea5bfab0 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 10 client.16722 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 10 client.16722 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 20 client.16722 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea5d8760 0x5566ea5dab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 4 client.16722 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:22.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.189+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5566ea276410 con 0x5566ea5d8760 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.190+0000 7ff48112f640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea5d8760 0x5566ea5dab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.190+0000 7ff48112f640 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea5d8760 0x5566ea5dab80 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e3767f1163500f5 server_cookie=661bef6acf986544 in_seq=0 out_seq=0 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.190+0000 7ff460ff9640 10 client.16722 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff460ff9640 1 -- 172.21.15.3:0/2511184194 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5566ea276410 con 0x5566ea5d8760 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff460ff9640 10 client.16722 handle_command_reply: tid=0 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 client.16722 shutdown 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 msgr2=0x5566ea5c5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5b90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e9c0 tx=0x7ff47000cef0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 shutdown_connections 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea5d8760 0x5566ea5dab80 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5b90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2511184194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bf190 0x5566ea5c5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 >> 172.21.15.3:0/2511184194 conn(0x5566ea5b98c0 msgr2=0x5566ea5bb610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff460ff9640 0 client.16722 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 shutdown_connections 2024-02-09T22:54:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 1 -- 172.21.15.3:0/2511184194 wait complete. 2024-02-09T22:54:22.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.191+0000 7ff485f6f740 20 client.16722 trim_cache size 0 max 16384 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 Processor -- start 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 -- start start 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea5c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea5bedf0 con 0x5566ea5bf190 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea5ba8c0 con 0x5566ea5c8690 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff485f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea5caa80 con 0x5566ea5bfab0 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff48112f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea5c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff48112f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea5c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57246/0 (socket says 172.21.15.3:57246) 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff48092e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff48112f640 1 -- 172.21.15.3:0/2302495684 learned_addr learned my addr 172.21.15.3:0/2302495684 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.207+0000 7ff481930640 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff481930640 1 -- 172.21.15.3:0/2302495684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 msgr2=0x5566ea5c5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff481930640 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff481930640 1 -- 172.21.15.3:0/2302495684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 msgr2=0x5566ea5c7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff481930640 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea5c7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff481930640 1 -- 172.21.15.3:0/2302495684 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566ea4c0c70 con 0x5566ea5c8690 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff48092e640 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.208+0000 7ff481930640 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c28f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0113a0 tx=0x7ff46c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=511cbc0aa6891455 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.210+0000 7ff462ffd640 1 -- 172.21.15.3:0/2302495684 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c024040 con 0x5566ea5c8690 2024-02-09T22:54:22.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.210+0000 7ff462ffd640 1 -- 172.21.15.3:0/2302495684 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff46c010e90 con 0x5566ea5c8690 2024-02-09T22:54:22.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.210+0000 7ff462ffd640 1 -- 172.21.15.3:0/2302495684 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c016070 con 0x5566ea5c8690 2024-02-09T22:54:22.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 -- 172.21.15.3:0/2302495684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 msgr2=0x5566ea5c28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c28f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0113a0 tx=0x7ff46c019000 comp rx=0 tx=0).stop 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 -- 172.21.15.3:0/2302495684 shutdown_connections 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5c8690 0x5566ea5c28f0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5bfab0 0x5566ea5c5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 --2- 172.21.15.3:0/2302495684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea5c7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 -- 172.21.15.3:0/2302495684 >> 172.21.15.3:0/2302495684 conn(0x5566ea5b98c0 msgr2=0x5566ea5c0970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 -- 172.21.15.3:0/2302495684 shutdown_connections 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.211+0000 7ff485f6f740 1 -- 172.21.15.3:0/2302495684 wait complete. 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 -- start start 2024-02-09T22:54:22.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 Processor -- start 2024-02-09T22:54:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea627fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bfab0 0x5566ea628520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5c8690 0x5566ea5ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea5cede0 con 0x5566ea5bf190 2024-02-09T22:54:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea5cef50 con 0x5566ea5bfab0 2024-02-09T22:54:22.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566ea5cf110 con 0x5566ea5c8690 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff485f6f740 10 client.0 _ll_register_callbacks cb 0x7ff46c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff48112f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea627fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff48112f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea627fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57262/0 (socket says 172.21.15.3:57262) 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff48112f640 1 -- 172.21.15.3:0/3924340785 learned_addr learned my addr 172.21.15.3:0/3924340785 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff481930640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5c8690 0x5566ea5ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.212+0000 7ff48092e640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bfab0 0x5566ea628520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff48112f640 1 -- 172.21.15.3:0/3924340785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5c8690 msgr2=0x5566ea5ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff48112f640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5c8690 0x5566ea5ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff48112f640 1 -- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bfab0 msgr2=0x5566ea628520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff48112f640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bfab0 0x5566ea628520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff48112f640 1 -- 172.21.15.3:0/3924340785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5566ea524390 con 0x5566ea5bf190 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff481930640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5c8690 0x5566ea5ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff48112f640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea627fe0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff474046000 tx=0x7ff47402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8170603febf99c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff460ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff460ff9640 1 -- 172.21.15.3:0/3924340785 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47404f040 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff460ff9640 1 -- 172.21.15.3:0/3924340785 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff474051d70 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff474004960 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff460ff9640 1 -- 172.21.15.3:0/3924340785 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff4740512c0 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff485f6f740 10 client.16728 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x7ff46c011050 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.213+0000 7ff460ff9640 1 -- 172.21.15.3:0/3924340785 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff474060020 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff460ff9640 1 -- 172.21.15.3:0/3924340785 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7ff47403c050 con 0x5566ea5bf190 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff485f6f740 10 client.16728 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff485f6f740 10 client.16728 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff485f6f740 20 client.16728 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff485f6f740 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea630c60 0x5566ea633080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff485f6f740 4 client.16728 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:54:22.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5566ea2627b0 con 0x5566ea630c60 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff48092e640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea630c60 0x5566ea633080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff460ff9640 10 client.16728 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.214+0000 7ff48092e640 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea630c60 0x5566ea633080 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=38c97cba0511f950 server_cookie=e64f38349cacdb0d in_seq=0 out_seq=0 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.215+0000 7ff460ff9640 1 -- 172.21.15.3:0/3924340785 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5566ea2627b0 con 0x5566ea630c60 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.215+0000 7ff460ff9640 10 client.16728 handle_command_reply: tid=0 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.215+0000 7ff485f6f740 1 client.16728 shutdown 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.215+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 msgr2=0x5566ea627fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.215+0000 7ff485f6f740 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea627fe0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7ff474046000 tx=0x7ff47402bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 shutdown_connections 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566ea630c60 0x5566ea633080 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566ea5bfab0 0x5566ea628520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff460ff9640 0 client.16728 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566ea5c8690 0x5566ea5ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 --2- 172.21.15.3:0/3924340785 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566ea5bf190 0x5566ea627fe0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 >> 172.21.15.3:0/3924340785 conn(0x5566ea5b98c0 msgr2=0x5566ea5b8090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 shutdown_connections 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 1 -- 172.21.15.3:0/3924340785 wait complete. 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff485f6f740 20 client.16728 trim_cache size 0 max 16384 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "f60472b5-ce26-4e2c-821a-78535db86c29", 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:54:22.223 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:54:22.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4640a2230 msgr2=0x7ff4640a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4640a2230 0x7ff4640a46f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009fd0 tx=0x7ff46c009500 comp rx=0 tx=0).stop 2024-02-09T22:54:22.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 msgr2=0x7ff47c19db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 0x7ff47c19db00 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7ff47000e9f0 tx=0x7ff47000eeb0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 shutdown_connections 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4640a2230 0x7ff4640a46f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197680 0x7ff47c197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c106570 0x7ff47c19db00 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 --2- 172.21.15.3:0/2981902535 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c068810 0x7ff47c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.216+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 >> 172.21.15.3:0/2981902535 conn(0x7ff47c077230 msgr2=0x7ff47c0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.217+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 shutdown_connections 2024-02-09T22:54:22.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.217+0000 7ff483393640 1 -- 172.21.15.3:0/2981902535 wait complete. 2024-02-09T22:54:22.232 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:54:22.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:54:22.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 -- 172.21.15.3:0/3978953740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102d70 msgr2=0x7fefc410b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 --2- 172.21.15.3:0/3978953740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102d70 0x7fefc410b250 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009900 tx=0x7fefb002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 -- 172.21.15.3:0/3978953740 shutdown_connections 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 --2- 172.21.15.3:0/3978953740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 0x7fefc410dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 --2- 172.21.15.3:0/3978953740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102d70 0x7fefc410b250 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 --2- 172.21.15.3:0/3978953740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102450 0x7fefc4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 -- 172.21.15.3:0/3978953740 >> 172.21.15.3:0/3978953740 conn(0x7fefc40772e0 msgr2=0x7fefc40776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 -- 172.21.15.3:0/3978953740 shutdown_connections 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 -- 172.21.15.3:0/3978953740 wait complete. 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.362+0000 7fefcafe2640 1 Processor -- start 2024-02-09T22:54:22.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 -- start start 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 0x7fefc419d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102d70 0x7fefc419db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 0x7fefc4197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4197e10 con 0x7fefc4102450 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4197f80 con 0x7fefc4102d70 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefcafe2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc4198140 con 0x7fefc410b820 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc8d7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 0x7fefc419d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc8d7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 0x7fefc419d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57290/0 (socket says 172.21.15.3:57290) 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102d70 0x7fefc419db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc957f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 0x7fefc4197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc8d7e640 1 -- 172.21.15.3:0/1432349898 learned_addr learned my addr 172.21.15.3:0/1432349898 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc3fff640 1 -- 172.21.15.3:0/1432349898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 msgr2=0x7fefc4197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc3fff640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 0x7fefc4197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc3fff640 1 -- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 msgr2=0x7fefc419d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc3fff640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 0x7fefc419d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc3fff640 1 -- 172.21.15.3:0/1432349898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0009560 con 0x7fefc4102d70 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 0x7fefc419d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.363+0000 7fefc957f640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 0x7fefc4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.364+0000 7fefc3fff640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102d70 0x7fefc419db00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fefb0036040 tx=0x7fefb002c720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.364+0000 7fefc1ffb640 1 -- 172.21.15.3:0/1432349898 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0004350 con 0x7fefc4102d70 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.364+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc41983c0 con 0x7fefc4102d70 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.364+0000 7fefc1ffb640 1 -- 172.21.15.3:0/1432349898 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefb00044b0 con 0x7fefc4102d70 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.364+0000 7fefc1ffb640 1 -- 172.21.15.3:0/1432349898 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0003d30 con 0x7fefc4102d70 2024-02-09T22:54:22.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.364+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc41a4540 con 0x7fefc4102d70 2024-02-09T22:54:22.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.366+0000 7fefc1ffb640 1 -- 172.21.15.3:0/1432349898 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fefb0004df0 con 0x7fefc4102d70 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefc1ffb640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fefb40a2230 0x7fefb40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fefb40a2230 0x7fefb40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefc1ffb640 1 -- 172.21.15.3:0/1432349898 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fefb00f0560 con 0x7fefc4102d70 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 Processor -- start 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 -- start start 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fefb40a2230 0x7fefb40a46f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fefb8011550 tx=0x7fefb8005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a55902ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590338e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559036300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558fe5d00 con 0x55a55902a980 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.367+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558f9a2f0 con 0x55a559033eb0 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558f97820 con 0x55a55902b330 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590338e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc8d7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a55902ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590338e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46568/0 (socket says 172.21.15.3:46568) 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 -- 172.21.15.3:0/4181612229 learned_addr learned my addr 172.21.15.3:0/4181612229 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc957f640 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559036300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 -- 172.21.15.3:0/4181612229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 msgr2=0x55a559036300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559036300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 -- 172.21.15.3:0/4181612229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 msgr2=0x55a55902ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a55902ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 -- 172.21.15.3:0/4181612229 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a558f54ff0 con 0x55a55902b330 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefc3fff640 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590338e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fefb00b94f0 tx=0x7fefb00bc9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9c8f5269989c7cc7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefaeffd640 1 -- 172.21.15.3:0/4181612229 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb00c11e0 con 0x55a55902b330 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefaeffd640 1 -- 172.21.15.3:0/4181612229 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefb00c5830 con 0x55a55902b330 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.368+0000 7fefaeffd640 1 -- 172.21.15.3:0/4181612229 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb00ee0f0 con 0x55a55902b330 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 -- 172.21.15.3:0/4181612229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 msgr2=0x55a5590338e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590338e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fefb00b94f0 tx=0x7fefb00bc9f0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 -- 172.21.15.3:0/4181612229 shutdown_connections 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559036300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590338e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/4181612229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a55902ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 -- 172.21.15.3:0/4181612229 >> 172.21.15.3:0/4181612229 conn(0x55a559025050 msgr2=0x55a559025430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 -- 172.21.15.3:0/4181612229 shutdown_connections 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.369+0000 7fefcdbbb740 1 -- 172.21.15.3:0/4181612229 wait complete. 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.370+0000 7fefcdbbb740 1 -- start start 2024-02-09T22:54:22.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.370+0000 7fefcdbbb740 1 Processor -- start 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.370+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a559030d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a559030d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a559030d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57306/0 (socket says 172.21.15.3:57306) 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 -- 172.21.15.3:0/3791248451 learned_addr learned my addr 172.21.15.3:0/3791248451 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590312b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a5590317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558f93000 con 0x55a55902a980 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558c66430 con 0x55a559033eb0 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558c22e00 con 0x55a55902b330 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc957f640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a5590317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefcdbbb740 10 client.0 _ll_register_callbacks cb 0x55a558f877f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:22.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc3fff640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590312b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 -- 172.21.15.3:0/3791248451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 msgr2=0x55a5590312b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590312b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 -- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 msgr2=0x55a5590317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a5590317f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc8d7e640 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc41a4540 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.371+0000 7fefc957f640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a5590317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a559030d70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fefb800fce0 tx=0x7fefb8013040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525216f14d799159 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefa3fff640 1 -- 172.21.15.3:0/3791248451 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb8021040 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefa3fff640 1 -- 172.21.15.3:0/3791248451 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fefb8027630 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a558c927e0 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefa3fff640 1 -- 172.21.15.3:0/3791248451 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefb800f660 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 10 client.16740 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a558f91680 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefa3fff640 1 -- 172.21.15.3:0/3791248451 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a558c927e0 con 0x55a55902a980 2024-02-09T22:54:22.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefa3fff640 1 -- 172.21.15.3:0/3791248451 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fefb80047a0 con 0x55a55902a980 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 10 client.16740 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 10 client.16740 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 20 client.16740 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.372+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a5590484e0 0x55a55904a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.373+0000 7fefcdbbb740 4 client.16740 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.373+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a558ccf720 con 0x55a5590484e0 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.373+0000 7fefc3fff640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a5590484e0 0x55a55904a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.373+0000 7fefc3fff640 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a5590484e0 0x55a55904a920 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f37116eebd70b9ef server_cookie=1269879952e51369 in_seq=0 out_seq=0 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.373+0000 7fefa3fff640 10 client.16740 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefa3fff640 1 -- 172.21.15.3:0/3791248451 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a558ccf720 con 0x55a5590484e0 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefa3fff640 10 client.16740 handle_command_reply: tid=0 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 client.16740 shutdown 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 msgr2=0x55a559030d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a559030d70 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fefb800fce0 tx=0x7fefb8013040 comp rx=0 tx=0).stop 2024-02-09T22:54:22.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 shutdown_connections 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a5590484e0 0x55a55904a920 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a5590317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a5590312b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3791248451 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902a980 0x55a559030d70 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 >> 172.21.15.3:0/3791248451 conn(0x55a559025050 msgr2=0x55a559026da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefa3fff640 0 client.16740 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 shutdown_connections 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.374+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3791248451 wait complete. 2024-02-09T22:54:22.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.375+0000 7fefcdbbb740 20 client.16740 trim_cache size 0 max 16384 2024-02-09T22:54:22.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:22 smithi003 ceph-mon[32678]: pgmap v96: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-02-09T22:54:22.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:22 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:54:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.391+0000 7fefcdbbb740 1 Processor -- start 2024-02-09T22:54:22.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.391+0000 7fefcdbbb740 1 -- start start 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a55902a980 0x55a55903efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a55903f500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a559033eb0 0x55a55902e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a558f17fd0 con 0x55a559033eb0 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a559033cb0 con 0x55a55902a980 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a559046c20 con 0x55a55902b330 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a55903f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc957f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a559033eb0 0x55a55902e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a55903f500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46586/0 (socket says 172.21.15.3:46586) 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 -- 172.21.15.3:0/3416490248 learned_addr learned my addr 172.21.15.3:0/3416490248 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a55902a980 0x55a55903efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 -- 172.21.15.3:0/3416490248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a55902a980 msgr2=0x55a55903efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a55902a980 0x55a55903efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 -- 172.21.15.3:0/3416490248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a559033eb0 msgr2=0x55a55902e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a559033eb0 0x55a55902e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 -- 172.21.15.3:0/3416490248 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a558f2db30 con 0x55a55902b330 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc957f640 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a559033eb0 0x55a55902e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefc3fff640 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a55903f500 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fefb00edea0 tx=0x7fefb0000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7aeeea6c4b070f35 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefae7fc640 1 -- 172.21.15.3:0/3416490248 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb00c11e0 con 0x55a55902b330 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefae7fc640 1 -- 172.21.15.3:0/3416490248 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefb00bb390 con 0x55a55902b330 2024-02-09T22:54:22.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.392+0000 7fefae7fc640 1 -- 172.21.15.3:0/3416490248 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb00c5600 con 0x55a55902b330 2024-02-09T22:54:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3416490248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 msgr2=0x55a55903f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a55903f500 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fefb00edea0 tx=0x7fefb0000bc0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3416490248 shutdown_connections 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a559033eb0 0x55a55902e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902b330 0x55a55903f500 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/3416490248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a55902a980 0x55a55903efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3416490248 >> 172.21.15.3:0/3416490248 conn(0x55a559025050 msgr2=0x55a55904f280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3416490248 shutdown_connections 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.393+0000 7fefcdbbb740 1 -- 172.21.15.3:0/3416490248 wait complete. 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 -- start start 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 Processor -- start 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902a980 0x55a559092990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 0x55a559092ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559093410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55909da10 con 0x55a55902b330 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55909db80 con 0x55a559033eb0 2024-02-09T22:54:22.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a55909dd40 con 0x55a55902a980 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefcdbbb740 10 client.0 _ll_register_callbacks cb 0x7fefb0005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 0x55a559092ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefc8d7e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902a980 0x55a559092990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefc3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 0x55a559092ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:57316/0 (socket says 172.21.15.3:57316) 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefc3fff640 1 -- 172.21.15.3:0/2324859446 learned_addr learned my addr 172.21.15.3:0/2324859446 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.394+0000 7fefc957f640 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559093410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefc3fff640 1 -- 172.21.15.3:0/2324859446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902a980 msgr2=0x55a559092990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefc3fff640 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902a980 0x55a559092990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefc3fff640 1 -- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 msgr2=0x55a559093410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefc3fff640 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559093410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefc3fff640 1 -- 172.21.15.3:0/2324859446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc00f9f0 con 0x55a55902b330 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefc3fff640 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 0x55a559092ed0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fefb00bb340 tx=0x7fefb00ee040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bce854ef7ebbd13 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefa3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:22.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefa3fff640 1 -- 172.21.15.3:0/2324859446 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb00b9700 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefa3fff640 1 -- 172.21.15.3:0/2324859446 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fefb00c11e0 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fefb00c5830 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefa3fff640 1 -- 172.21.15.3:0/2324859446 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefb00c53a0 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 10 client.16746 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x55a558f54ff0 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefa3fff640 1 -- 172.21.15.3:0/2324859446 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefb00c5830 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefa3fff640 1 -- 172.21.15.3:0/2324859446 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fefb003c070 con 0x55a55902b330 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 10 client.16746 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 10 client.16746 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 20 client.16746 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a55909f150 0x55a5590a1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 4 client.16746 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:54:22.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.395+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a558d1e890 con 0x55a55909f150 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a55909f150 0x55a5590a1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefc8d7e640 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a55909f150 0x55a5590a1540 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b85f02b8e9de3ec9 server_cookie=2fcf2907d3e1a8a7 in_seq=0 out_seq=0 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefa3fff640 10 client.16746 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefa3fff640 1 -- 172.21.15.3:0/2324859446 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55a558d1e890 con 0x55a55909f150 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefa3fff640 10 client.16746 handle_command_reply: tid=0 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 client.16746 shutdown 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 msgr2=0x55a559092ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 0x55a559092ed0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fefb00bb340 tx=0x7fefb00ee040 comp rx=0 tx=0).stop 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 shutdown_connections 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a55909f150 0x55a5590a1540 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a559033eb0 0x55a559093410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefa3fff640 0 client.16746 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a55902b330 0x55a559092ed0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.396+0000 7fefcdbbb740 1 --2- 172.21.15.3:0/2324859446 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a55902a980 0x55a559092990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 >> 172.21.15.3:0/2324859446 conn(0x55a559025050 msgr2=0x55a559023960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 shutdown_connections 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcdbbb740 1 -- 172.21.15.3:0/2324859446 wait complete. 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcdbbb740 20 client.16746 trim_cache size 0 max 16384 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "f60472b5-ce26-4e2c-821a-78535db86c29": { 2024-02-09T22:54:22.406 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:54:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "f60472b5-ce26-4e2c-821a-78535db86c29", 2024-02-09T22:54:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:54:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:54:22.407 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:54:22.407 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fefb40a2230 msgr2=0x7fefb40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fefb40a2230 0x7fefb40a46f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fefb8011550 tx=0x7fefb8005f90 comp rx=0 tx=0).stop 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102d70 msgr2=0x7fefc419db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102d70 0x7fefc419db00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fefb0036040 tx=0x7fefb002c720 comp rx=0 tx=0).stop 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 shutdown_connections 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fefb40a2230 0x7fefb40a46f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fefc4102d70 0x7fefc419db00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fefc410b820 0x7fefc4197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 --2- 172.21.15.3:0/1432349898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fefc4102450 0x7fefc419d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 >> 172.21.15.3:0/1432349898 conn(0x7fefc40772e0 msgr2=0x7fefc40fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 shutdown_connections 2024-02-09T22:54:22.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:22.397+0000 7fefcafe2640 1 -- 172.21.15.3:0/1432349898 wait complete. 2024-02-09T22:54:22.412 INFO:tasks.cephfs.filesystem:scrub status for tag:f60472b5-ce26-4e2c-821a-78535db86c29 - {'path': '/', 'tag': 'f60472b5-ce26-4e2c-821a-78535db86c29', 'options': 'recursive,force'} 2024-02-09T22:54:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:22 smithi089 ceph-mon[36665]: pgmap v96: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-02-09T22:54:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:22 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:54:22.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 83328 1.51 MB/sec execute 16 sec latency 4110.905 ms 2024-02-09T22:54:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:22 smithi064 ceph-mon[36638]: pgmap v96: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 1.0 MiB/s wr, 109 op/s 2024-02-09T22:54:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:22 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:54:22.685 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:54:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:54:22.225+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000208 [...5,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/WORD/ auth{3=1,4=1} v12485 ap=2 f(v209 m2024-02-09T22:54:14.652404+0000 3=3+0) n(v10 rc2024-02-09T22:54:14.652404+0000 b780288 4=3+1) old_inodes=2 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@1218} mcw={3=AsLsXsFs,4=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d51f5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000208:[<0x10000000009/WORD v1606>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x10000000208:[<0x10000000009/WORD v12485>,<0x10000000008/~dmtmp v3995>,<0x10000000007/client0 v210>,<0x10000000006/clients v196>,<0x10000000005/tmp v199>,<0x100000001fb/client.0 v184>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v203>,<0x10000000001/sv_1 v214>,<0x10000000000/qa v214>,<0x1/volumes v194>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:54:13.326420+0000 5=5+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:54:14.652404+0000 b780288 4=3+1)","memory_value.dirstat":"f(v209 m2024-02-09T22:54:14.652404+0000 3=3+0)","memory_value.rstat":"n(v10 rc2024-02-09T22:54:14.652404+0000 b780288 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:54:22.686 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:54:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:54:22.232+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000225 [...5,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM/ auth{1=1} v12491 ap=2 f(v533 m2024-02-09T22:54:14.698403+0000 3=3+0) n(v11 rc2024-02-09T22:54:14.698403+0000 b780288 4=3+1) old_inodes=2 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@2817} mcw={1=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d5398580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000225:[<0x10000000009/PM v1478>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x10000000225:[<0x10000000009/PM v12491>,<0x10000000008/~dmtmp v4001>,<0x10000000007/client0 v212>,<0x10000000006/clients v198>,<0x10000000005/tmp v201>,<0x100000001fb/client.0 v186>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v205>,<0x10000000001/sv_1 v216>,<0x10000000000/qa v216>,<0x1/volumes v196>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:54:14.628404+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:54:14.698403+0000 b780288 4=3+1)","memory_value.dirstat":"f(v533 m2024-02-09T22:54:14.698403+0000 3=3+0)","memory_value.rstat":"n(v11 rc2024-02-09T22:54:14.698403+0000 b780288 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:54:22.686 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:54:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:54:22.235+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023e [...5,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS/ auth{4=1} v12487 ap=2 f(v104 m2024-02-09T22:54:14.724403+0000 1=1+0) n(v10 rc2024-02-09T22:54:14.724403+0000 b260096 2=1+1) old_inodes=2 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@603} mcw={4=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d53d5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000023e:[<0x10000000009/ACCESS v1492>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x1000000023e:[<0x10000000009/ACCESS v12487>,<0x10000000008/~dmtmp v4001>,<0x10000000007/client0 v212>,<0x10000000006/clients v198>,<0x10000000005/tmp v201>,<0x100000001fb/client.0 v186>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v205>,<0x10000000001/sv_1 v216>,<0x10000000000/qa v216>,<0x1/volumes v196>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:54:14.109410+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:54:14.724403+0000 b260096 2=1+1)","memory_value.dirstat":"f(v104 m2024-02-09T22:54:14.724403+0000 1=1+0)","memory_value.rstat":"n(v10 rc2024-02-09T22:54:14.724403+0000 b260096 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:54:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:54:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:23.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:54:23.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:54:23.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:23 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:54:23.629 INFO:tasks.workunit.client.0.smithi003.stdout: 1 85604 2.23 MB/sec execute 17 sec latency 14.526 ms 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:54:23.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:23 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:54:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:54:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: Scrub error on inode 0x10000000208 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/WORD) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.b log and `damage ls` output for details 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:54:23.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:23 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:54:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:24 smithi089 ceph-mon[36665]: pgmap v97: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 987 KiB/s wr, 80 op/s 2024-02-09T22:54:24.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 85843 2.12 MB/sec execute 18 sec latency 804.944 ms 2024-02-09T22:54:24.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:24 smithi003 ceph-mon[32678]: pgmap v97: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 987 KiB/s wr, 80 op/s 2024-02-09T22:54:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:24 smithi064 ceph-mon[36638]: pgmap v97: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 987 KiB/s wr, 80 op/s 2024-02-09T22:54:25.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 85843 2.01 MB/sec execute 19 sec latency 1805.000 ms 2024-02-09T22:54:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:26 smithi089 ceph-mon[36665]: pgmap v98: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 987 KiB/s wr, 80 op/s 2024-02-09T22:54:26.488 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:26 smithi064 ceph-mon[36638]: pgmap v98: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 987 KiB/s wr, 80 op/s 2024-02-09T22:54:26.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 85843 1.90 MB/sec execute 20 sec latency 2805.101 ms 2024-02-09T22:54:26.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:26 smithi003 ceph-mon[32678]: pgmap v98: 97 pgs: 97 active+clean; 41 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 761 KiB/s rd, 987 KiB/s wr, 80 op/s 2024-02-09T22:54:27.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 85843 1.81 MB/sec execute 21 sec latency 3805.195 ms 2024-02-09T22:54:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:28 smithi089 ceph-mon[36665]: pgmap v99: 97 pgs: 97 active+clean; 42 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-02-09T22:54:28.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 87392 1.99 MB/sec execute 22 sec latency 4217.921 ms 2024-02-09T22:54:28.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:28 smithi003 ceph-mon[32678]: pgmap v99: 97 pgs: 97 active+clean; 42 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-02-09T22:54:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:28 smithi064 ceph-mon[36638]: pgmap v99: 97 pgs: 97 active+clean; 42 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-02-09T22:54:29.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 89200 2.48 MB/sec execute 23 sec latency 32.405 ms 2024-02-09T22:54:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:30 smithi089 ceph-mon[36665]: pgmap v100: 97 pgs: 97 active+clean; 42 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 543 KiB/s wr, 82 op/s 2024-02-09T22:54:30.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 89392 2.38 MB/sec execute 24 sec latency 892.184 ms 2024-02-09T22:54:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:30 smithi003 ceph-mon[32678]: pgmap v100: 97 pgs: 97 active+clean; 42 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 543 KiB/s wr, 82 op/s 2024-02-09T22:54:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:30 smithi064 ceph-mon[36638]: pgmap v100: 97 pgs: 97 active+clean; 42 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 543 KiB/s wr, 82 op/s 2024-02-09T22:54:31.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 89392 2.29 MB/sec execute 25 sec latency 1892.269 ms 2024-02-09T22:54:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:31 smithi003 ceph-mon[32678]: pgmap v101: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-02-09T22:54:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:31 smithi064 ceph-mon[36638]: pgmap v101: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-02-09T22:54:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:31 smithi089 ceph-mon[36665]: pgmap v101: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 139 op/s 2024-02-09T22:54:32.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 89392 2.20 MB/sec execute 26 sec latency 2892.325 ms 2024-02-09T22:54:32.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:54:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:54:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:54:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:54:32.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:54:32.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:54:32.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:54:32.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:54:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:54:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:54:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:54:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:54:33.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 91030 2.37 MB/sec execute 27 sec latency 3304.228 ms 2024-02-09T22:54:33.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:33 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:54:33.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:33 smithi003 ceph-mon[32678]: pgmap v102: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 KiB/s wr, 115 op/s 2024-02-09T22:54:33.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:33 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:54:33.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:33 smithi064 ceph-mon[36638]: pgmap v102: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 KiB/s wr, 115 op/s 2024-02-09T22:54:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:33 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:54:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:33 smithi089 ceph-mon[36665]: pgmap v102: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 KiB/s wr, 115 op/s 2024-02-09T22:54:34.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 -- 172.21.15.3:0/2379815739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 msgr2=0x7fa18c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 --2- 172.21.15.3:0/2379815739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c10cfe0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa184009c10 tx=0x7fa18402bef0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 -- 172.21.15.3:0/2379815739 shutdown_connections 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 --2- 172.21.15.3:0/2379815739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c10cfe0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 --2- 172.21.15.3:0/2379815739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100cf0 0x7fa18c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 --2- 172.21.15.3:0/2379815739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa18c100340 0x7fa18c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 -- 172.21.15.3:0/2379815739 >> 172.21.15.3:0/2379815739 conn(0x7fa18c077220 msgr2=0x7fa18c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 -- 172.21.15.3:0/2379815739 shutdown_connections 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.001+0000 7fa1916f4640 1 -- 172.21.15.3:0/2379815739 wait complete. 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 Processor -- start 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 -- start start 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100340 0x7fa18c19d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa18c197690 0x7fa18c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c198160 con 0x7fa18c105410 2024-02-09T22:54:34.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c1982d0 con 0x7fa18c197690 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa1916f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c198490 con 0x7fa18c100340 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa18a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100340 0x7fa18c19d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa18a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c19dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51050/0 (socket says 172.21.15.3:51050) 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa18a7fc640 1 -- 172.21.15.3:0/4219998938 learned_addr learned my addr 172.21.15.3:0/4219998938 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.002+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa18c197690 0x7fa18c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18affd640 1 -- 172.21.15.3:0/4219998938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa18c197690 msgr2=0x7fa18c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18affd640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa18c197690 0x7fa18c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18affd640 1 -- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 msgr2=0x7fa18c19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18affd640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18affd640 1 -- 172.21.15.3:0/4219998938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa180009560 con 0x7fa18c100340 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:34.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa18affd640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100340 0x7fa18c19d580 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9f0 tx=0x7fa17c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa173fff640 1 -- 172.21.15.3:0/4219998938 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c009640 con 0x7fa18c100340 2024-02-09T22:54:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa184009840 con 0x7fa18c100340 2024-02-09T22:54:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa173fff640 1 -- 172.21.15.3:0/4219998938 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa17c00ccc0 con 0x7fa18c100340 2024-02-09T22:54:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa173fff640 1 -- 172.21.15.3:0/4219998938 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c018790 con 0x7fa18c100340 2024-02-09T22:54:34.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.003+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c1a4780 con 0x7fa18c100340 2024-02-09T22:54:34.015 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.005+0000 7fa173fff640 1 -- 172.21.15.3:0/4219998938 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa17c0109a0 con 0x7fa18c100340 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.005+0000 7fa1943a2740 1 Processor -- start 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.005+0000 7fa1943a2740 1 -- start start 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa173fff640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1740a2220 0x7fa1740a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d7f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d88260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1943a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1ceda00 con 0x5608c1d7f0b0 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1943a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1ceaf30 con 0x5608c1d7f9d0 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1943a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1ce6710 con 0x5608c1d887a0 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1740a2220 0x7fa1740a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.016 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa173fff640 1 -- 172.21.15.3:0/4219998938 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa17c014070 con 0x7fa18c100340 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d7f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18a7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d88260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d7f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51060/0 (socket says 172.21.15.3:51060) 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 -- 172.21.15.3:0/4188301231 learned_addr learned my addr 172.21.15.3:0/4188301231 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 -- 172.21.15.3:0/4188301231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 msgr2=0x5608c1d8ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 -- 172.21.15.3:0/4188301231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 msgr2=0x5608c1d88260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d88260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 -- 172.21.15.3:0/4188301231 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608c1cab3b0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d88260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18affd640 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d7f490 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa17c097300 tx=0x7fa17c09ad70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf18819e676cf83c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1740a2220 0x7fa1740a46e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa184009370 tx=0x7fa18402ceb0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.006+0000 7fa1717fa640 1 -- 172.21.15.3:0/4188301231 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c086d20 con 0x5608c1d7f0b0 2024-02-09T22:54:34.017 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1717fa640 1 -- 172.21.15.3:0/4188301231 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa17c09b5a0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1717fa640 1 -- 172.21.15.3:0/4188301231 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c07e9d0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 -- 172.21.15.3:0/4188301231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 msgr2=0x5608c1d7f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d7f490 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa17c097300 tx=0x7fa17c09ad70 comp rx=0 tx=0).stop 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 -- 172.21.15.3:0/4188301231 shutdown_connections 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d88260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4188301231 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d7f490 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 -- 172.21.15.3:0/4188301231 >> 172.21.15.3:0/4188301231 conn(0x5608c1d799d0 msgr2=0x5608c1d79db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 -- 172.21.15.3:0/4188301231 shutdown_connections 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.007+0000 7fa1943a2740 1 -- 172.21.15.3:0/4188301231 wait complete. 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 -- start start 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 Processor -- start 2024-02-09T22:54:34.018 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f0b0 0x5608c1d857e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d85d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 0x5608c1d86260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1c8bb90 con 0x5608c1d887a0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c19ba250 con 0x5608c1d7f9d0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1c6c200 con 0x5608c1d7f0b0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa1943a2740 10 client.0 _ll_register_callbacks cb 0x5608c1ca25d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 0x5608c1d86260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f0b0 0x5608c1d857e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 0x5608c1d86260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51068/0 (socket says 172.21.15.3:51068) 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18a7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d85d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 -- 172.21.15.3:0/1788410048 learned_addr learned my addr 172.21.15.3:0/1788410048 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 -- 172.21.15.3:0/1788410048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f0b0 msgr2=0x5608c1d857e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f0b0 0x5608c1d857e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.019 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 -- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 msgr2=0x5608c1d85d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d85d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18b7fe640 1 -- 172.21.15.3:0/1788410048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c1a4780 con 0x5608c1d887a0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.008+0000 7fa18affd640 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f0b0 0x5608c1d857e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 0x5608c1d86260 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa184004a60 tx=0x7fa18402bef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ce9d37602f39a6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa16eff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa16eff5640 1 -- 172.21.15.3:0/1788410048 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1840047a0 con 0x5608c1d887a0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5608c1d0cee0 con 0x5608c1d887a0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa16eff5640 1 -- 172.21.15.3:0/1788410048 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa184043450 con 0x5608c1d887a0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa16eff5640 1 -- 172.21.15.3:0/1788410048 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa184043610 con 0x5608c1d887a0 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 10 client.16758 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:34.020 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5608c1ce4d90 con 0x5608c1d887a0 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa16eff5640 1 -- 172.21.15.3:0/1788410048 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5608c1d0cee0 con 0x5608c1d887a0 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa16eff5640 1 -- 172.21.15.3:0/1788410048 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa184034700 con 0x5608c1d887a0 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 10 client.16758 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 10 client.16758 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 20 client.16758 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1d98ba0 0x5608c1d9afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 4 client.16758 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.009+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5608c19e6600 con 0x5608c1d98ba0 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.010+0000 7fa18affd640 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1d98ba0 0x5608c1d9afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.021 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.010+0000 7fa18affd640 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1d98ba0 0x5608c1d9afc0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f49017c66dd853b server_cookie=4fbace21a37c1290 in_seq=0 out_seq=0 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.010+0000 7fa16eff5640 10 client.16758 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.010+0000 7fa16eff5640 1 -- 172.21.15.3:0/1788410048 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5608c19e6600 con 0x5608c1d98ba0 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.010+0000 7fa16eff5640 10 client.16758 handle_command_reply: tid=0 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 client.16758 shutdown 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 msgr2=0x5608c1d86260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 0x5608c1d86260 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa184004a60 tx=0x7fa18402bef0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 shutdown_connections 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1d98ba0 0x5608c1d9afc0 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d85d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.022 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d887a0 0x5608c1d86260 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 --2- 172.21.15.3:0/1788410048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f0b0 0x5608c1d857e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 >> 172.21.15.3:0/1788410048 conn(0x5608c1d799d0 msgr2=0x5608c1d7b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa16eff5640 0 client.16758 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 shutdown_connections 2024-02-09T22:54:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 1 -- 172.21.15.3:0/1788410048 wait complete. 2024-02-09T22:54:34.023 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.011+0000 7fa1943a2740 20 client.16758 trim_cache size 0 max 16384 2024-02-09T22:54:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 Processor -- start 2024-02-09T22:54:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 -- start start 2024-02-09T22:54:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d87f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f9d0 0x5608c1d86080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d887a0 0x5608c1d9f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1d75110 con 0x5608c1d7f0b0 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1d0cee0 con 0x5608c1d887a0 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa1943a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1d7aa20 con 0x5608c1d7f9d0 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d87f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d87f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51082/0 (socket says 172.21.15.3:51082) 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 -- 172.21.15.3:0/4251634551 learned_addr learned my addr 172.21.15.3:0/4251634551 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f9d0 0x5608c1d86080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d887a0 0x5608c1d9f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 -- 172.21.15.3:0/4251634551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f9d0 msgr2=0x5608c1d86080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f9d0 0x5608c1d86080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 -- 172.21.15.3:0/4251634551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d887a0 msgr2=0x5608c1d9f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d887a0 0x5608c1d9f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18affd640 1 -- 172.21.15.3:0/4251634551 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5608c1c81820 con 0x5608c1d7f0b0 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f9d0 0x5608c1d86080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.024+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d887a0 0x5608c1d9f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa18affd640 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d87f10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa17c000c00 tx=0x7fa17c009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=981540dbd657bb23 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa170ff9640 1 -- 172.21.15.3:0/4251634551 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c09a800 con 0x5608c1d7f0b0 2024-02-09T22:54:34.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa170ff9640 1 -- 172.21.15.3:0/4251634551 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa17c09eff0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa170ff9640 1 -- 172.21.15.3:0/4251634551 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c0ce030 con 0x5608c1d7f0b0 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 -- 172.21.15.3:0/4251634551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 msgr2=0x5608c1d87f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d87f10 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa17c000c00 tx=0x7fa17c009f20 comp rx=0 tx=0).stop 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 -- 172.21.15.3:0/4251634551 shutdown_connections 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d887a0 0x5608c1d9f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d7f9d0 0x5608c1d86080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 --2- 172.21.15.3:0/4251634551 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d87f10 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 -- 172.21.15.3:0/4251634551 >> 172.21.15.3:0/4251634551 conn(0x5608c1d799d0 msgr2=0x5608c1d80b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 -- 172.21.15.3:0/4251634551 shutdown_connections 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.025+0000 7fa1943a2740 1 -- 172.21.15.3:0/4251634551 wait complete. 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 -- start start 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 Processor -- start 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d8f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.034 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d8d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1d8e010 con 0x5608c1d7f0b0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1d8e180 con 0x5608c1d7f9d0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608c1d8e340 con 0x5608c1d887a0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa1943a2740 10 client.0 _ll_register_callbacks cb 0x7fa180005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d8f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d8f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51096/0 (socket says 172.21.15.3:51096) 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 -- 172.21.15.3:0/3739776517 learned_addr learned my addr 172.21.15.3:0/3739776517 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d8d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 -- 172.21.15.3:0/3739776517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 msgr2=0x5608c1d8d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 -- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 msgr2=0x5608c1d8d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d8d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18affd640 1 -- 172.21.15.3:0/3739776517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5608c1cab3b0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.026+0000 7fa18b7fe640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa18affd640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d8f780 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0a3110 tx=0x7fa17c0cbef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262ccbdbf546b60 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa16eff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa16eff5640 1 -- 172.21.15.3:0/3739776517 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c098f40 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa16eff5640 1 -- 172.21.15.3:0/3739776517 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa17c0a3640 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa17c09b5a0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa16eff5640 1 -- 172.21.15.3:0/3739776517 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=31) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa17c0662e0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 10 client.16764 fetch_fsmap learned FSMap version 31 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=31,monmap=4+}) v3 -- 0x5608c1ca76f0 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa16eff5640 1 -- 172.21.15.3:0/3739776517 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa17c0a7080 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa16eff5640 1 -- 172.21.15.3:0/3739776517 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 31) v1 ==== 7906+0+0 (secure 0 0 0) 0x7fa17c082040 con 0x5608c1d7f0b0 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 10 client.16764 fetch_fsmap finished waiting for FSMap version 31 2024-02-09T22:54:34.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 10 client.16764 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 20 client.16764 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1def330 0x5608c1df1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 4 client.16764 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.027+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5608c1a371b0 con 0x5608c1def330 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.028+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1def330 0x5608c1df1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.028+0000 7fa16eff5640 10 client.16764 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:34.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.028+0000 7fa18a7fc640 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1def330 0x5608c1df1750 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc5c2aa65c7afa27 server_cookie=137dc3da7d93c24e in_seq=0 out_seq=0 2024-02-09T22:54:34.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa16eff5640 1 -- 172.21.15.3:0/3739776517 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5608c1a371b0 con 0x5608c1def330 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa16eff5640 10 client.16764 handle_command_reply: tid=0 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 client.16764 shutdown 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 msgr2=0x5608c1d8f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d8f780 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0a3110 tx=0x7fa17c0cbef0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 shutdown_connections 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5608c1def330 0x5608c1df1750 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5608c1d7f9d0 0x5608c1d8d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5608c1d887a0 0x5608c1d8d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa16eff5640 0 client.16764 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 --2- 172.21.15.3:0/3739776517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5608c1d7f0b0 0x5608c1d8f780 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 >> 172.21.15.3:0/3739776517 conn(0x5608c1d799d0 msgr2=0x5608c1d9f900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 shutdown_connections 2024-02-09T22:54:34.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 1 -- 172.21.15.3:0/3739776517 wait complete. 2024-02-09T22:54:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.057+0000 7fa1943a2740 20 client.16764 trim_cache size 0 max 16384 2024-02-09T22:54:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:54:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:54:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:54:34.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:54:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1740a2220 msgr2=0x7fa1740a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1740a2220 0x7fa1740a46e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa184009370 tx=0x7fa18402ceb0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100340 msgr2=0x7fa18c19d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100340 0x7fa18c19d580 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fa17c00e9f0 tx=0x7fa17c00edf0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 shutdown_connections 2024-02-09T22:54:34.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1740a2220 0x7fa1740a46e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa18c197690 0x7fa18c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa18c105410 0x7fa18c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 --2- 172.21.15.3:0/4219998938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa18c100340 0x7fa18c19d580 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 >> 172.21.15.3:0/4219998938 conn(0x7fa18c077220 msgr2=0x7fa18c068e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 shutdown_connections 2024-02-09T22:54:34.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:34.058+0000 7fa1916f4640 1 -- 172.21.15.3:0/4219998938 wait complete. 2024-02-09T22:54:34.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92734 2.72 MB/sec execute 28 sec latency 17.954 ms 2024-02-09T22:54:35.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92936 2.63 MB/sec execute 29 sec latency 865.898 ms 2024-02-09T22:54:36.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:36 smithi064 ceph-mon[36638]: pgmap v103: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 KiB/s wr, 115 op/s 2024-02-09T22:54:36.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:36 smithi089 ceph-mon[36665]: pgmap v103: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 KiB/s wr, 115 op/s 2024-02-09T22:54:36.630 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92936 2.54 MB/sec execute 30 sec latency 1865.963 ms 2024-02-09T22:54:36.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:36 smithi003 ceph-mon[32678]: pgmap v103: 97 pgs: 97 active+clean; 44 MiB data, 644 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 727 KiB/s wr, 115 op/s 2024-02-09T22:54:37.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92936 2.46 MB/sec execute 31 sec latency 2866.106 ms 2024-02-09T22:54:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:38 smithi064 ceph-mon[36638]: pgmap v104: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 972 KiB/s wr, 177 op/s 2024-02-09T22:54:38.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:38 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:38 smithi089 ceph-mon[36665]: pgmap v104: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 972 KiB/s wr, 177 op/s 2024-02-09T22:54:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:38 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:38.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92992 2.39 MB/sec execute 32 sec latency 3282.953 ms 2024-02-09T22:54:38.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:38 smithi003 ceph-mon[32678]: pgmap v104: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 972 KiB/s wr, 177 op/s 2024-02-09T22:54:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:38 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:39.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92992 2.32 MB/sec execute 33 sec latency 1541.670 ms 2024-02-09T22:54:40.438 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:40 smithi064 ceph-mon[36638]: pgmap v105: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 735 KiB/s wr, 119 op/s 2024-02-09T22:54:40.480 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:40 smithi089 ceph-mon[36665]: pgmap v105: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 735 KiB/s wr, 119 op/s 2024-02-09T22:54:40.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92992 2.25 MB/sec execute 34 sec latency 2541.797 ms 2024-02-09T22:54:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:40 smithi003 ceph-mon[32678]: pgmap v105: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 735 KiB/s wr, 119 op/s 2024-02-09T22:54:41.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92992 2.19 MB/sec execute 35 sec latency 3541.954 ms 2024-02-09T22:54:42.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:42 smithi064 ceph-mon[36638]: pgmap v106: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-02-09T22:54:42.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:42 smithi089 ceph-mon[36665]: pgmap v106: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-02-09T22:54:42.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 92992 2.12 MB/sec execute 36 sec latency 4542.090 ms 2024-02-09T22:54:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:42 smithi003 ceph-mon[32678]: pgmap v106: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 147 op/s 2024-02-09T22:54:43.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 94700 2.25 MB/sec execute 37 sec latency 4951.204 ms 2024-02-09T22:54:44.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:44 smithi089 ceph-mon[36665]: pgmap v107: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 774 KiB/s wr, 90 op/s 2024-02-09T22:54:44.631 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96436 2.51 MB/sec execute 38 sec latency 11.610 ms 2024-02-09T22:54:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:44 smithi003 ceph-mon[32678]: pgmap v107: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 774 KiB/s wr, 90 op/s 2024-02-09T22:54:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:44 smithi064 ceph-mon[36638]: pgmap v107: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 774 KiB/s wr, 90 op/s 2024-02-09T22:54:45.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96539 2.45 MB/sec execute 39 sec latency 952.760 ms 2024-02-09T22:54:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:46 smithi089 ceph-mon[36665]: pgmap v108: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 774 KiB/s wr, 90 op/s 2024-02-09T22:54:46.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96539 2.39 MB/sec execute 40 sec latency 1952.878 ms 2024-02-09T22:54:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:46 smithi003 ceph-mon[32678]: pgmap v108: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 774 KiB/s wr, 90 op/s 2024-02-09T22:54:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:46 smithi064 ceph-mon[36638]: pgmap v108: 97 pgs: 97 active+clean; 42 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 774 KiB/s wr, 90 op/s 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 -- 172.21.15.3:0/1675242760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 msgr2=0x7f1d140fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1675242760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d140fff60 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1d04009930 tx=0x7f1d0402bcd0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 -- 172.21.15.3:0/1675242760 shutdown_connections 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1675242760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d141004a0 0x7f1d1410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1675242760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d140fff60 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1675242760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d14107420 0x7f1d14107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 -- 172.21.15.3:0/1675242760 >> 172.21.15.3:0/1675242760 conn(0x7f1d140753d0 msgr2=0x7f1d140757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 -- 172.21.15.3:0/1675242760 shutdown_connections 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.155+0000 7f1d19df7640 1 -- 172.21.15.3:0/1675242760 wait complete. 2024-02-09T22:54:47.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 Processor -- start 2024-02-09T22:54:47.161 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 -- start start 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d1419d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d141004a0 0x7f1d1419db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d14107420 0x7f1d14197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14197e20 con 0x7f1d140ffae0 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14197f90 con 0x7f1d141004a0 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.156+0000 7f1d19df7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d14198150 con 0x7f1d14107420 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d1419d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d1419d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35476/0 (socket says 172.21.15.3:35476) 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 -- 172.21.15.3:0/1961696427 learned_addr learned my addr 172.21.15.3:0/1961696427 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d13fff640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d14107420 0x7f1d14197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d141004a0 0x7f1d1419db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 -- 172.21.15.3:0/1961696427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d14107420 msgr2=0x7f1d14197780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d14107420 0x7f1d14197780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.162 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 -- 172.21.15.3:0/1961696427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d141004a0 msgr2=0x7f1d1419db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d141004a0 0x7f1d1419db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 -- 172.21.15.3:0/1961696427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d04009560 con 0x7f1d140ffae0 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d141004a0 0x7f1d1419db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d1419d610 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800e9f0 tx=0x7f1d0800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d10ff9640 1 -- 172.21.15.3:0/1961696427 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08009700 con 0x7f1d140ffae0 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d10ff9640 1 -- 172.21.15.3:0/1961696427 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d0800cd80 con 0x7f1d140ffae0 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d10ff9640 1 -- 172.21.15.3:0/1961696427 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08018920 con 0x7f1d140ffae0 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d14198430 con 0x7f1d140ffae0 2024-02-09T22:54:47.163 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.157+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d141a4640 con 0x7f1d140ffae0 2024-02-09T22:54:47.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d10ff9640 1 -- 172.21.15.3:0/1961696427 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1d08018a80 con 0x7f1d140ffae0 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 Processor -- start 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 -- start start 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d10ff9640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1cfc0a2220 0x7f1cfc0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91e5f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91f3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d13fff640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1cfc0a2220 0x7f1cfc0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d10ff9640 1 -- 172.21.15.3:0/1961696427 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1d08014070 con 0x7f1d140ffae0 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 0x559fa91f5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa9154a00 con 0x559fa91f3bb0 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa9151f30 con 0x559fa91e6470 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.160+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa914d710 con 0x559fa91e5b50 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d13fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 0x559fa91f5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.169 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91e5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d13fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 0x559fa91f5fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35482/0 (socket says 172.21.15.3:35482) 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d13fff640 1 -- 172.21.15.3:0/1245827365 learned_addr learned my addr 172.21.15.3:0/1245827365 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91f3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 -- 172.21.15.3:0/1245827365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 msgr2=0x559fa91f3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91f3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 -- 172.21.15.3:0/1245827365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 msgr2=0x559fa91f5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 0x559fa91f5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 -- 172.21.15.3:0/1245827365 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fa91123b0 con 0x559fa91e5b50 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d13fff640 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 0x559fa91f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d13fff640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1cfc0a2220 0x7f1cfc0a46e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c013cf0 tx=0x7f1d0c009f10 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91e5f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d080cbba0 tx=0x7f1d0809a860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=98b519b2d0826eed server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.161+0000 7f1cf9ffb640 1 -- 172.21.15.3:0/1245827365 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d080a35e0 con 0x559fa91e5b50 2024-02-09T22:54:47.170 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1cf9ffb640 1 -- 172.21.15.3:0/1245827365 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d080aee90 con 0x559fa91e5b50 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1cf9ffb640 1 -- 172.21.15.3:0/1245827365 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0809a290 con 0x559fa91e5b50 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1245827365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 msgr2=0x559fa91e5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91e5f30 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f1d080cbba0 tx=0x7f1d0809a860 comp rx=0 tx=0).stop 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1245827365 shutdown_connections 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91f3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91f3bb0 0x559fa91f5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1245827365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91e5f30 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1245827365 >> 172.21.15.3:0/1245827365 conn(0x559fa91e09d0 msgr2=0x559fa91e0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1245827365 shutdown_connections 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.162+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1245827365 wait complete. 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 -- start start 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 Processor -- start 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91f0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.171 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91f3bb0 0x559fa91f1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa90f2b90 con 0x559fa91e6470 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa8e21250 con 0x559fa91f3bb0 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa90d3200 con 0x559fa91e5b50 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d1c9ce740 10 client.0 _ll_register_callbacks cb 0x559fa91095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91f10e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35494/0 (socket says 172.21.15.3:35494) 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d12ffd640 1 -- 172.21.15.3:0/3666495632 learned_addr learned my addr 172.21.15.3:0/3666495632 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.163+0000 7f1d137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91f0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d13fff640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91f3bb0 0x559fa91f1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d12ffd640 1 -- 172.21.15.3:0/3666495632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 msgr2=0x559fa91f0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91f0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d12ffd640 1 -- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91f3bb0 msgr2=0x559fa91f1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.172 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91f3bb0 0x559fa91f1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d12ffd640 1 -- 172.21.15.3:0/3666495632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d141a4640 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d137fe640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91f10e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1d04005ec0 tx=0x7f1d040048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f69e1afe31a0767 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1cf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/3666495632 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0402cea0 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559fa9173ee0 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/3666495632 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1d04004240 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/3666495632 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d04004400 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d1c9ce740 10 client.16776 fetch_fsmap learned FSMap version 32 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.164+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559fa914bd90 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/3666495632 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559fa9173ee0 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/3666495632 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 32) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1d0403c040 con 0x559fa91e6470 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d1c9ce740 10 client.16776 fetch_fsmap finished waiting for FSMap version 32 2024-02-09T22:54:47.173 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d1c9ce740 10 client.16776 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d1c9ce740 20 client.16776 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa9203e20 0x559fa9206260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d1c9ce740 4 client.16776 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fa8e4d600 con 0x559fa9203e20 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.165+0000 7f1d137fe640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa9203e20 0x559fa9206260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.166+0000 7f1d137fe640 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa9203e20 0x559fa9206260 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98825ac14a21a86a server_cookie=357973a61b4a1a22 in_seq=0 out_seq=0 2024-02-09T22:54:47.174 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.166+0000 7f1cf2ffd640 10 client.16776 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:47.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.166+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/3666495632 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559fa8e4d600 con 0x559fa9203e20 2024-02-09T22:54:47.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.166+0000 7f1cf2ffd640 10 client.16776 handle_command_reply: tid=0 2024-02-09T22:54:47.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.166+0000 7f1d1c9ce740 1 client.16776 shutdown 2024-02-09T22:54:47.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 msgr2=0x559fa91f10e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91f10e0 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f1d04005ec0 tx=0x7f1d040048f0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.175 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 shutdown_connections 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa9203e20 0x559fa9206260 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91f3bb0 0x559fa91f1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91f10e0 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1cf2ffd640 0 client.16776 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/3666495632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91e5b50 0x559fa91f0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 >> 172.21.15.3:0/3666495632 conn(0x559fa91e09d0 msgr2=0x559fa91e2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 shutdown_connections 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/3666495632 wait complete. 2024-02-09T22:54:47.176 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.167+0000 7f1d1c9ce740 20 client.16776 trim_cache size 0 max 16384 2024-02-09T22:54:47.184 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 Processor -- start 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 -- start start 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 0x559fa91f3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91dbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa91e55f0 con 0x559fa91e5b50 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa91ec7a0 con 0x559fa91e6470 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa91e0240 con 0x559fa91f3bb0 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 0x559fa91f3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 0x559fa91f3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35498/0 (socket says 172.21.15.3:35498) 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.180+0000 7f1d137fe640 1 -- 172.21.15.3:0/642525824 learned_addr learned my addr 172.21.15.3:0/642525824 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d13fff640 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d137fe640 1 -- 172.21.15.3:0/642525824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 msgr2=0x559fa91f1440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.185 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d137fe640 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d137fe640 1 -- 172.21.15.3:0/642525824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 msgr2=0x559fa91dbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d137fe640 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91dbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d137fe640 1 -- 172.21.15.3:0/642525824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fa90e8820 con 0x559fa91e5b50 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d137fe640 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 0x559fa91f3260 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009380 tx=0x7f1d08010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5255a867a21c561 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1cf97fa640 1 -- 172.21.15.3:0/642525824 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d080b8070 con 0x559fa91e5b50 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1cf97fa640 1 -- 172.21.15.3:0/642525824 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d080a35e0 con 0x559fa91e5b50 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1cf97fa640 1 -- 172.21.15.3:0/642525824 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d080b9230 con 0x559fa91e5b50 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/642525824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 msgr2=0x559fa91f3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.181+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 0x559fa91f3260 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f1d08009380 tx=0x7f1d08010f50 comp rx=0 tx=0).stop 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/642525824 shutdown_connections 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e6470 0x559fa91dbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f1440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/642525824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e5b50 0x559fa91f3260 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.186 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/642525824 >> 172.21.15.3:0/642525824 conn(0x559fa91e09d0 msgr2=0x559fa91e0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/642525824 shutdown_connections 2024-02-09T22:54:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/642525824 wait complete. 2024-02-09T22:54:47.187 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- start start 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 Processor -- start 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e5b50 0x559fa91fa810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91fad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa91f94d0 con 0x559fa91e6470 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa91f9640 con 0x559fa91e5b50 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.182+0000 7f1d1c9ce740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fa91f9800 con 0x559fa91f3bb0 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d1c9ce740 10 client.0 _ll_register_callbacks cb 0x7f1d0c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91fad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.190 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91fad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35514/0 (socket says 172.21.15.3:35514) 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 -- 172.21.15.3:0/1087794727 learned_addr learned my addr 172.21.15.3:0/1087794727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d13fff640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e5b50 0x559fa91fa810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 -- 172.21.15.3:0/1087794727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 msgr2=0x559fa91f8e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f8e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 -- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e5b50 msgr2=0x559fa91fa810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e5b50 0x559fa91fa810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 -- 172.21.15.3:0/1087794727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fa91123b0 con 0x559fa91e6470 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d12ffd640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91fad50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1d04005ab0 tx=0x7f1d04009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7e75333c89a87b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1cf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/1087794727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0402cea0 con 0x559fa91e6470 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/1087794727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1d04000e40 con 0x559fa91e6470 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1d080aee90 con 0x559fa91e6470 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/1087794727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d040035f0 con 0x559fa91e6470 2024-02-09T22:54:47.191 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d1c9ce740 10 client.16782 fetch_fsmap learned FSMap version 32 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.183+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) v3 -- 0x559fa910e6f0 con 0x559fa91e6470 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/1087794727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1d080aee90 con 0x559fa91e6470 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/1087794727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 32) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1d0403c040 con 0x559fa91e6470 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d1c9ce740 10 client.16782 fetch_fsmap finished waiting for FSMap version 32 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d1c9ce740 10 client.16782 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d1c9ce740 20 client.16782 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa925b500 0x559fa925d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d1c9ce740 4 client.16782 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559fa8e9e1b0 con 0x559fa925b500 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa925b500 0x559fa925d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1cf2ffd640 10 client.16782 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:47.192 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.184+0000 7f1d137fe640 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa925b500 0x559fa925d920 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1318c5b332879f6 server_cookie=22700f24cc24b4ac in_seq=0 out_seq=0 2024-02-09T22:54:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1cf2ffd640 1 -- 172.21.15.3:0/1087794727 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559fa8e9e1b0 con 0x559fa925b500 2024-02-09T22:54:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1cf2ffd640 10 client.16782 handle_command_reply: tid=0 2024-02-09T22:54:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 client.16782 shutdown 2024-02-09T22:54:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 msgr2=0x559fa91fad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91fad50 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f1d04005ab0 tx=0x7f1d04009800 comp rx=0 tx=0).stop 2024-02-09T22:54:47.220 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 shutdown_connections 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559fa925b500 0x559fa925d920 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559fa91f3bb0 0x559fa91f8e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559fa91e6470 0x559fa91fad50 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1cf2ffd640 0 client.16782 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 --2- 172.21.15.3:0/1087794727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559fa91e5b50 0x559fa91fa810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 >> 172.21.15.3:0/1087794727 conn(0x559fa91e09d0 msgr2=0x559fa920abc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 shutdown_connections 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.215+0000 7f1d1c9ce740 1 -- 172.21.15.3:0/1087794727 wait complete. 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d1c9ce740 20 client.16782 trim_cache size 0 max 16384 2024-02-09T22:54:47.221 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:54:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:54:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:54:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:54:47.222 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1cfc0a2220 msgr2=0x7f1cfc0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1cfc0a2220 0x7f1cfc0a46e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1d0c013cf0 tx=0x7f1d0c009f10 comp rx=0 tx=0).stop 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 msgr2=0x7f1d1419d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d1419d610 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f1d0800e9f0 tx=0x7f1d0800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 shutdown_connections 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1cfc0a2220 0x7f1cfc0a46e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1d141004a0 0x7f1d1419db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1d14107420 0x7f1d14197780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 --2- 172.21.15.3:0/1961696427 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1d140ffae0 0x7f1d1419d610 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 >> 172.21.15.3:0/1961696427 conn(0x7f1d140753d0 msgr2=0x7f1d140fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 shutdown_connections 2024-02-09T22:54:47.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:54:47.216+0000 7f1d19df7640 1 -- 172.21.15.3:0/1961696427 wait complete. 2024-02-09T22:54:47.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96539 2.33 MB/sec execute 41 sec latency 2952.928 ms 2024-02-09T22:54:48.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:48 smithi089 ceph-mon[36665]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 996 KiB/s wr, 134 op/s 2024-02-09T22:54:48.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:48 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:48.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96607 2.28 MB/sec execute 42 sec latency 3363.394 ms 2024-02-09T22:54:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:48 smithi003 ceph-mon[32678]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 996 KiB/s wr, 134 op/s 2024-02-09T22:54:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:48 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:48 smithi064 ceph-mon[36638]: pgmap v109: 97 pgs: 97 active+clean; 44 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 996 KiB/s wr, 134 op/s 2024-02-09T22:54:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:48 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:49.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:49 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:49.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:49 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:49.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96607 2.23 MB/sec execute 43 sec latency 1543.793 ms 2024-02-09T22:54:49.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:49 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:50 smithi089 ceph-mon[36665]: pgmap v110: 97 pgs: 97 active+clean; 44 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 751 KiB/s wr, 72 op/s 2024-02-09T22:54:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:50 smithi003 ceph-mon[32678]: pgmap v110: 97 pgs: 97 active+clean; 44 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 751 KiB/s wr, 72 op/s 2024-02-09T22:54:50.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96607 2.18 MB/sec execute 44 sec latency 2543.839 ms 2024-02-09T22:54:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:50 smithi064 ceph-mon[36638]: pgmap v110: 97 pgs: 97 active+clean; 44 MiB data, 669 MiB used, 1.0 TiB / 1.0 TiB avail; 557 KiB/s rd, 751 KiB/s wr, 72 op/s 2024-02-09T22:54:51.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96607 2.13 MB/sec execute 45 sec latency 3543.932 ms 2024-02-09T22:54:52.413 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:54:52.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:54:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:52 smithi089 ceph-mon[36665]: pgmap v111: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 -- 172.21.15.3:0/3004808140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c107930 msgr2=0x7fd99c10e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 --2- 172.21.15.3:0/3004808140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c107930 0x7fd99c10e2d0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fd994009cd0 tx=0x7fd99402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 -- 172.21.15.3:0/3004808140 shutdown_connections 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 --2- 172.21.15.3:0/3004808140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c107930 0x7fd99c10e2d0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 --2- 172.21.15.3:0/3004808140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c102df0 0x7fd99c107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 --2- 172.21.15.3:0/3004808140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c102440 0x7fd99c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 -- 172.21.15.3:0/3004808140 >> 172.21.15.3:0/3004808140 conn(0x7fd99c077310 msgr2=0x7fd99c077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 -- 172.21.15.3:0/3004808140 shutdown_connections 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.527+0000 7fd9a173a640 1 -- 172.21.15.3:0/3004808140 wait complete. 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 Processor -- start 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 -- start start 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 0x7fd99c100090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c107930 0x7fd99c1005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c100b10 0x7fd99c1a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c1a6170 con 0x7fd99c102440 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c1a62e0 con 0x7fd99c107930 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd9a173a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99c1a64a0 con 0x7fd99c100b10 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd99affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 0x7fd99c100090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd99affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 0x7fd99c100090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35536/0 (socket says 172.21.15.3:35536) 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.528+0000 7fd99affd640 1 -- 172.21.15.3:0/2535070096 learned_addr learned my addr 172.21.15.3:0/2535070096 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd9927fc640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c107930 0x7fd99c1005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c100b10 0x7fd99c1a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99affd640 1 -- 172.21.15.3:0/2535070096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c100b10 msgr2=0x7fd99c1a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99affd640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c100b10 0x7fd99c1a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99affd640 1 -- 172.21.15.3:0/2535070096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c107930 msgr2=0x7fd99c1005d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99affd640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c107930 0x7fd99c1005d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99affd640 1 -- 172.21.15.3:0/2535070096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd994009900 con 0x7fd99c102440 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c100b10 0x7fd99c1a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd99affd640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 0x7fd99c100090 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00d950 tx=0x7fd98c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd998ff9640 1 -- 172.21.15.3:0/2535070096 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c009a30 con 0x7fd99c102440 2024-02-09T22:54:52.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd998ff9640 1 -- 172.21.15.3:0/2535070096 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd98c009b90 con 0x7fd99c102440 2024-02-09T22:54:52.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c1a6780 con 0x7fd99c102440 2024-02-09T22:54:52.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd998ff9640 1 -- 172.21.15.3:0/2535070096 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c004a00 con 0x7fd99c102440 2024-02-09T22:54:52.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.529+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c1a6c90 con 0x7fd99c102440 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.531+0000 7fd998ff9640 1 -- 172.21.15.3:0/2535070096 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd98c009d00 con 0x7fd99c102440 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.531+0000 7fd9a43e3740 1 Processor -- start 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.531+0000 7fd9a43e3740 1 -- start start 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef54d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef60300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24ef0fd00 con 0x55a24ef55330 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24eec42f0 con 0x55a24ef5deb0 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24eec1820 con 0x55a24ef54980 2024-02-09T22:54:52.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35538/0 (socket says 172.21.15.3:35538) 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 -- 172.21.15.3:0/3394869606 learned_addr learned my addr 172.21.15.3:0/3394869606 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd998ff9640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd9880a2230 0x7fd9880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd9927fc640 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef60300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef54d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 -- 172.21.15.3:0/3394869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 msgr2=0x55a24ef54d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef54d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 -- 172.21.15.3:0/3394869606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 msgr2=0x55a24ef60300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef60300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99affd640 1 -- 172.21.15.3:0/3394869606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a24ee7eff0 con 0x55a24ef55330 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef54d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:52.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.532+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd9880a2230 0x7fd9880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.533+0000 7fd998ff9640 1 -- 172.21.15.3:0/2535070096 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd98c0cdb60 con 0x7fd99c102440 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.533+0000 7fd99affd640 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5d8e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd98c096f50 tx=0x7fd98c021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6e6fc6f11f48de3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.533+0000 7fd9917fa640 1 -- 172.21.15.3:0/3394869606 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c005680 con 0x55a24ef55330 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.533+0000 7fd9917fa640 1 -- 172.21.15.3:0/3394869606 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd98c0d1030 con 0x55a24ef55330 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.533+0000 7fd9917fa640 1 -- 172.21.15.3:0/3394869606 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c09ac80 con 0x55a24ef55330 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- 172.21.15.3:0/3394869606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 msgr2=0x55a24ef5d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd9880a2230 0x7fd9880a46f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd994033000 tx=0x7fd994009860 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5d8e0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7fd98c096f50 tx=0x7fd98c021000 comp rx=0 tx=0).stop 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- 172.21.15.3:0/3394869606 shutdown_connections 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef60300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5d8e0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/3394869606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef54d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- 172.21.15.3:0/3394869606 >> 172.21.15.3:0/3394869606 conn(0x55a24ef4f050 msgr2=0x55a24ef4f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- 172.21.15.3:0/3394869606 shutdown_connections 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- 172.21.15.3:0/3394869606 wait complete. 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- start start 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 Processor -- start 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef5ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef5b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24eebd000 con 0x55a24ef55330 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24eb90430 con 0x55a24ef5deb0 2024-02-09T22:54:52.547 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24eb4ce00 con 0x55a24ef54980 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.534+0000 7fd9a43e3740 10 client.0 _ll_register_callbacks cb 0x55a24eeb17f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5b240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35552/0 (socket says 172.21.15.3:35552) 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 -- 172.21.15.3:0/349005299 learned_addr learned my addr 172.21.15.3:0/349005299 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd99affd640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef5ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef5b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 -- 172.21.15.3:0/349005299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 msgr2=0x55a24ef5ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef5ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 -- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 msgr2=0x55a24ef5b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef5b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 -- 172.21.15.3:0/349005299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c1a6c90 con 0x55a24ef55330 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9927fc640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5b240 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd98400bdd0 tx=0x7fd9840117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b6e909421ff091 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9827fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9827fc640 1 -- 172.21.15.3:0/349005299 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984012650 con 0x55a24ef55330 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9827fc640 1 -- 172.21.15.3:0/349005299 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd9840127b0 con 0x55a24ef55330 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a24ebbc7e0 con 0x55a24ef55330 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9827fc640 1 -- 172.21.15.3:0/349005299 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd984011ec0 con 0x55a24ef55330 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9a43e3740 10 client.16794 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:52.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.535+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a24eebb680 con 0x55a24ef55330 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9827fc640 1 -- 172.21.15.3:0/349005299 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a24ebbc7e0 con 0x55a24ef55330 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9827fc640 1 -- 172.21.15.3:0/349005299 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd98401bc20 con 0x55a24ef55330 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9a43e3740 10 client.16794 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9a43e3740 10 client.16794 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9a43e3740 20 client.16794 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24ef6e370 0x55a24ef70790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9a43e3740 4 client.16794 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a24ebf9720 con 0x55a24ef6e370 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd99affd640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24ef6e370 0x55a24ef70790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd9827fc640 10 client.16794 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.536+0000 7fd99affd640 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24ef6e370 0x55a24ef70790 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8917fe3a6809c494 server_cookie=78fdfed8f2d67a14 in_seq=0 out_seq=0 2024-02-09T22:54:52.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9827fc640 1 -- 172.21.15.3:0/349005299 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a24ebf9720 con 0x55a24ef6e370 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9827fc640 10 client.16794 handle_command_reply: tid=0 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 client.16794 shutdown 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 msgr2=0x55a24ef5b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5b240 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fd98400bdd0 tx=0x7fd9840117b0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 shutdown_connections 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24ef6e370 0x55a24ef70790 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef5b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef55330 0x55a24ef5b240 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9827fc640 0 client.16794 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/349005299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef54980 0x55a24ef5ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 >> 172.21.15.3:0/349005299 conn(0x55a24ef4f050 msgr2=0x55a24ef50da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 shutdown_connections 2024-02-09T22:54:52.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 1 -- 172.21.15.3:0/349005299 wait complete. 2024-02-09T22:54:52.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.537+0000 7fd9a43e3740 20 client.16794 trim_cache size 0 max 16384 2024-02-09T22:54:52.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.552+0000 7fd9a43e3740 1 Processor -- start 2024-02-09T22:54:52.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.552+0000 7fd9a43e3740 1 -- start start 2024-02-09T22:54:52.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 0x55a24ef5b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef5b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef57f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24ef4a780 con 0x55a24ef54980 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24ee41fd0 con 0x55a24ef5deb0 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24ebbc7e0 con 0x55a24ef55330 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 0x55a24ef5b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 0x55a24ef5b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35560/0 (socket says 172.21.15.3:35560) 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 -- 172.21.15.3:0/186853347 learned_addr learned my addr 172.21.15.3:0/186853347 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef57f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9927fc640 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef5b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 -- 172.21.15.3:0/186853347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 msgr2=0x55a24ef5b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef5b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 -- 172.21.15.3:0/186853347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 msgr2=0x55a24ef57f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef57f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 -- 172.21.15.3:0/186853347 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a24ee57b30 con 0x55a24ef54980 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd9927fc640 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef5b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.553+0000 7fd99affd640 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 0x55a24ef5b250 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd98c0969e0 tx=0x7fd98c096c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc45ddef2cc78aa3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd990ff9640 1 -- 172.21.15.3:0/186853347 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0cde20 con 0x55a24ef54980 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd990ff9640 1 -- 172.21.15.3:0/186853347 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd98c005680 con 0x55a24ef54980 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd990ff9640 1 -- 172.21.15.3:0/186853347 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98c09e6d0 con 0x55a24ef54980 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 -- 172.21.15.3:0/186853347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 msgr2=0x55a24ef5b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 0x55a24ef5b250 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fd98c0969e0 tx=0x7fd98c096c70 comp rx=0 tx=0).stop 2024-02-09T22:54:52.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 -- 172.21.15.3:0/186853347 shutdown_connections 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef5deb0 0x55a24ef57f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef5b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/186853347 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef54980 0x55a24ef5b250 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 -- 172.21.15.3:0/186853347 >> 172.21.15.3:0/186853347 conn(0x55a24ef4f050 msgr2=0x55a24ef750f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 -- 172.21.15.3:0/186853347 shutdown_connections 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.554+0000 7fd9a43e3740 1 -- 172.21.15.3:0/186853347 wait complete. 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 -- start start 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 Processor -- start 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef54980 0x55a24ef63930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef63e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 0x55a24ef643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24efc4880 con 0x55a24ef5deb0 2024-02-09T22:54:52.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24efc49c0 con 0x55a24ef54980 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a24efc4b00 con 0x55a24ef55330 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9a43e3740 10 client.0 _ll_register_callbacks cb 0x7fd98c0d1540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 0x55a24ef643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 0x55a24ef643b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35574/0 (socket says 172.21.15.3:35574) 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 -- 172.21.15.3:0/4047321275 learned_addr learned my addr 172.21.15.3:0/4047321275 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99affd640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef54980 0x55a24ef63930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd9927fc640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef63e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 -- 172.21.15.3:0/4047321275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 msgr2=0x55a24ef63e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef63e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 -- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef54980 msgr2=0x55a24ef63930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef54980 0x55a24ef63930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.555+0000 7fd99b7fe640 1 -- 172.21.15.3:0/4047321275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a24ee7eff0 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd99b7fe640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 0x55a24ef643b0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd994009cd0 tx=0x7fd9940095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6986bfda754a7da server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9827fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9827fc640 1 -- 172.21.15.3:0/4047321275 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd994003d80 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9827fc640 1 -- 172.21.15.3:0/4047321275 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd994045070 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd98c0d1030 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9827fc640 1 -- 172.21.15.3:0/4047321275 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd994034e90 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 10 client.16800 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55a24ee7d850 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9827fc640 1 -- 172.21.15.3:0/4047321275 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd98c0d1030 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9827fc640 1 -- 172.21.15.3:0/4047321275 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd99403e810 con 0x55a24ef5deb0 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 10 client.16800 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:52.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 10 client.16800 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 20 client.16800 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24efc6370 0x55a24efc8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 4 client.16800 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.556+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a24ec48890 con 0x55a24efc6370 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd99affd640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24efc6370 0x55a24efc8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd99affd640 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24efc6370 0x55a24efc8790 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c92d46945257d16 server_cookie=10dd1fb90727296d in_seq=0 out_seq=0 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9827fc640 10 client.16800 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9827fc640 1 -- 172.21.15.3:0/4047321275 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a24ec48890 con 0x55a24efc6370 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9827fc640 10 client.16800 handle_command_reply: tid=0 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 client.16800 shutdown 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 msgr2=0x55a24ef643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 0x55a24ef643b0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fd994009cd0 tx=0x7fd9940095c0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 shutdown_connections 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a24efc6370 0x55a24efc8790 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a24ef5deb0 0x55a24ef643b0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a24ef55330 0x55a24ef63e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9827fc640 0 client.16800 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 --2- 172.21.15.3:0/4047321275 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a24ef54980 0x55a24ef63930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 >> 172.21.15.3:0/4047321275 conn(0x55a24ef4f050 msgr2=0x55a24ef4d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 shutdown_connections 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.557+0000 7fd9a43e3740 1 -- 172.21.15.3:0/4047321275 wait complete. 2024-02-09T22:54:52.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a43e3740 20 client.16800 trim_cache size 0 max 16384 2024-02-09T22:54:52.570 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:54:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:54:52.570 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:54:52.570 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:54:52.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd9880a2230 msgr2=0x7fd9880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd9880a2230 0x7fd9880a46f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fd994033000 tx=0x7fd994009860 comp rx=0 tx=0).stop 2024-02-09T22:54:52.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 msgr2=0x7fd99c100090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 0x7fd99c100090 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fd98c00d950 tx=0x7fd98c00de10 comp rx=0 tx=0).stop 2024-02-09T22:54:52.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 shutdown_connections 2024-02-09T22:54:52.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd9880a2230 0x7fd9880a46f0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd99c107930 0x7fd99c1005d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd99c100b10 0x7fd99c1a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 --2- 172.21.15.3:0/2535070096 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd99c102440 0x7fd99c100090 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 >> 172.21.15.3:0/2535070096 conn(0x7fd99c077310 msgr2=0x7fd99c0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 shutdown_connections 2024-02-09T22:54:52.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.558+0000 7fd9a173a640 1 -- 172.21.15.3:0/2535070096 wait complete. 2024-02-09T22:54:52.572 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:52 smithi003 ceph-mon[32678]: pgmap v111: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-02-09T22:54:52.573 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:54:52.574 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:54:52.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 96607 2.08 MB/sec execute 46 sec latency 4543.996 ms 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 -- 172.21.15.3:0/3481941435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a410db80 msgr2=0x7fe3a410ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/3481941435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a410db80 0x7fe3a410ff70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fe394009ca0 tx=0x7fe39402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 -- 172.21.15.3:0/3481941435 shutdown_connections 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/3481941435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3a41008f0 0x7fe3a410d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/3481941435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a410db80 0x7fe3a410ff70 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/3481941435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a40fffd0 0x7fe3a41003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 -- 172.21.15.3:0/3481941435 >> 172.21.15.3:0/3481941435 conn(0x7fe3a40772b0 msgr2=0x7fe3a40776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 -- 172.21.15.3:0/3481941435 shutdown_connections 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.672+0000 7fe3a89a1640 1 -- 172.21.15.3:0/3481941435 wait complete. 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 Processor -- start 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 -- start start 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 0x7fe3a419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3a41008f0 0x7fe3a419fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a410db80 0x7fe3a4199860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a4199eb0 con 0x7fe3a40fffd0 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a419a020 con 0x7fe3a41008f0 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.673+0000 7fe3a89a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a419a1e0 con 0x7fe3a410db80 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 0x7fe3a419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 0x7fe3a419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35594/0 (socket says 172.21.15.3:35594) 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 -- 172.21.15.3:0/1793127498 learned_addr learned my addr 172.21.15.3:0/1793127498 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:52.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a1d9b640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3a41008f0 0x7fe3a419fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a2d9d640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a410db80 0x7fe3a4199860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 -- 172.21.15.3:0/1793127498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a410db80 msgr2=0x7fe3a4199860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a410db80 0x7fe3a4199860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 -- 172.21.15.3:0/1793127498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3a41008f0 msgr2=0x7fe3a419fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3a41008f0 0x7fe3a419fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 -- 172.21.15.3:0/1793127498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe394009900 con 0x7fe3a40fffd0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a259c640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 0x7fe3a419f6b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00d770 tx=0x7fe39c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c009c30 con 0x7fe3a40fffd0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe39c00de80 con 0x7fe3a40fffd0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a419a4c0 con 0x7fe3a40fffd0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe39c004e80 con 0x7fe3a40fffd0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.674+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a41a6750 con 0x7fe3a40fffd0 2024-02-09T22:54:52.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.675+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a40680b0 con 0x7fe3a40fffd0 2024-02-09T22:54:52.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.676+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe39c005020 con 0x7fe3a40fffd0 2024-02-09T22:54:52.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.677+0000 7fe3937fe640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe38c0a22b0 0x7fe38c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:52.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.677+0000 7fe3a1d9b640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe38c0a22b0 0x7fe38c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:52.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.677+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe39c0cdba0 con 0x7fe3a40fffd0 2024-02-09T22:54:52.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.677+0000 7fe3a1d9b640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe38c0a22b0 0x7fe38c0a4770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe398005fd0 tx=0x7fe398005850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:52.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.681+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe39c096800 con 0x7fe3a40fffd0 2024-02-09T22:54:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:52 smithi064 ceph-mon[36638]: pgmap v111: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 1.1 MiB/s wr, 104 op/s 2024-02-09T22:54:52.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.901+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3a4108d10 con 0x7fe3a40fffd0 2024-02-09T22:54:52.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.902+0000 7fe3937fe640 1 -- 172.21.15.3:0/1793127498 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+11363 (secure 0 0 0) 0x7fe39c099ec0 con 0x7fe3a40fffd0 2024-02-09T22:54:52.906 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:54:52.907 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":34,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:54:47.762608+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:54:52.907 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 34 2024-02-09T22:54:52.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe38c0a22b0 msgr2=0x7fe38c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe38c0a22b0 0x7fe38c0a4770 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe398005fd0 tx=0x7fe398005850 comp rx=0 tx=0).stop 2024-02-09T22:54:52.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 msgr2=0x7fe3a419f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 0x7fe3a419f6b0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fe39c00d770 tx=0x7fe39c00dc30 comp rx=0 tx=0).stop 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 shutdown_connections 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe38c0a22b0 0x7fe38c0a4770 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe3a41008f0 0x7fe3a419fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe3a410db80 0x7fe3a4199860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 --2- 172.21.15.3:0/1793127498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe3a40fffd0 0x7fe3a419f6b0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 >> 172.21.15.3:0/1793127498 conn(0x7fe3a40772b0 msgr2=0x7fe3a410f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 shutdown_connections 2024-02-09T22:54:52.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:52.904+0000 7fe3a89a1640 1 -- 172.21.15.3:0/1793127498 wait complete. 2024-02-09T22:54:52.919 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:52.919 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:54:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/1055825763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac105420 msgr2=0x7fc3ac10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/1055825763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac105420 0x7fc3ac10cff0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fc39c009cd0 tx=0x7fc39c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/1055825763 shutdown_connections 2024-02-09T22:54:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/1055825763 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac105420 0x7fc3ac10cff0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/1055825763 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3ac100d50 0x7fc3ac104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/1055825763 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/1055825763 >> 172.21.15.3:0/1055825763 conn(0x7fc3ac077300 msgr2=0x7fc3ac077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/1055825763 shutdown_connections 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/1055825763 wait complete. 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 Processor -- start 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.020+0000 7fc3b0f0f640 1 -- start start 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac19d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac105420 0x7fc3ac19db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3ac1976c0 0x7fc3ac197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac198190 con 0x7fc3ac1003a0 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac198300 con 0x7fc3ac105420 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3ac1984c0 con 0x7fc3ac1976c0 2024-02-09T22:54:53.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac19d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac19d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35612/0 (socket says 172.21.15.3:35612) 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2791298944 learned_addr learned my addr 172.21.15.3:0/2791298944 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3ac1976c0 0x7fc3ac197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac105420 0x7fc3ac19db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2791298944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3ac1976c0 msgr2=0x7fc3ac197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3ac1976c0 0x7fc3ac197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2791298944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac105420 msgr2=0x7fc3ac19db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac105420 0x7fc3ac19db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2791298944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c009900 con 0x7fc3ac1003a0 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac105420 0x7fc3ac19db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:53.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac19d600 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400d950 tx=0x7fc3a400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc39b7fe640 1 -- 172.21.15.3:0/2791298944 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4009c30 con 0x7fc3ac1003a0 2024-02-09T22:54:53.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3ac1a4460 con 0x7fc3ac1003a0 2024-02-09T22:54:53.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc39b7fe640 1 -- 172.21.15.3:0/2791298944 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3a4009d90 con 0x7fc3ac1003a0 2024-02-09T22:54:53.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc39b7fe640 1 -- 172.21.15.3:0/2791298944 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a400f7e0 con 0x7fc3ac1003a0 2024-02-09T22:54:53.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.021+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac1a4940 con 0x7fc3ac1003a0 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc39b7fe640 1 -- 172.21.15.3:0/2791298944 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc3a4010ad0 con 0x7fc3ac1003a0 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 Processor -- start 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 -- start start 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17ca30 0x55dfec17ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 0x55dfec185990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec1883b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec138020 con 0x55dfec17d3e0 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec0eb320 con 0x55dfec17ca30 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec0e8850 con 0x55dfec185f60 2024-02-09T22:54:53.036 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.024+0000 7fc39b7fe640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc3940a2230 0x7fc3940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 0x55dfec185990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 0x55dfec185990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35624/0 (socket says 172.21.15.3:35624) 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 -- 172.21.15.3:0/908080874 learned_addr learned my addr 172.21.15.3:0/908080874 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc3940a2230 0x7fc3940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17ca30 0x55dfec17ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec1883b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc39b7fe640 1 -- 172.21.15.3:0/2791298944 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc3a40cddc0 con 0x7fc3ac1003a0 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 -- 172.21.15.3:0/908080874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 msgr2=0x55dfec1883b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec1883b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 -- 172.21.15.3:0/908080874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17ca30 msgr2=0x55dfec17ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17ca30 0x55dfec17ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 -- 172.21.15.3:0/908080874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfec0a6210 con 0x55dfec17d3e0 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec1883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:53.037 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc3940a2230 0x7fc3940a46f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc39c004af0 tx=0x7fc39c032ec0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 0x55dfec185990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4096c40 tx=0x7fc3a406c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17bc87f2563b4f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc398ff9640 1 -- 172.21.15.3:0/908080874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a409a140 con 0x55dfec17d3e0 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc398ff9640 1 -- 172.21.15.3:0/908080874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3a409aee0 con 0x55dfec17d3e0 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.025+0000 7fc398ff9640 1 -- 172.21.15.3:0/908080874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a40cb7c0 con 0x55dfec17d3e0 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/908080874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 msgr2=0x55dfec185990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 0x55dfec185990 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4096c40 tx=0x7fc3a406c7a0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/908080874 shutdown_connections 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec1883b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17d3e0 0x55dfec185990 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/908080874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17ca30 0x55dfec17ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/908080874 >> 172.21.15.3:0/908080874 conn(0x55dfec177100 msgr2=0x55dfec1774e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/908080874 shutdown_connections 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/908080874 wait complete. 2024-02-09T22:54:53.038 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.026+0000 7fc3b3bb7740 1 -- start start 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 Processor -- start 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec182fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec1834e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec183a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec0e4030 con 0x55dfec185f60 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfebdb7430 con 0x55dfec17d3e0 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfebd73e00 con 0x55dfec17ca30 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3b3bb7740 10 client.0 _ll_register_callbacks cb 0x55dfec0d9030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec183a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec183a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35628/0 (socket says 172.21.15.3:35628) 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 -- 172.21.15.3:0/3864274132 learned_addr learned my addr 172.21.15.3:0/3864274132 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec182fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec1834e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.039 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 -- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 msgr2=0x55dfec182fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec182fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 -- 172.21.15.3:0/3864274132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 msgr2=0x55dfec1834e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec1834e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 -- 172.21.15.3:0/3864274132 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3ac1a4940 con 0x55dfec185f60 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec182fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec1834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec183a20 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc39c036000 tx=0x7fc39c045040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6770002bb678016b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.027+0000 7fc391ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc391ffb640 1 -- 172.21.15.3:0/3864274132 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c034e90 con 0x55dfec185f60 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc391ffb640 1 -- 172.21.15.3:0/3864274132 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc39c004040 con 0x55dfec185f60 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dfebde37e0 con 0x55dfec185f60 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc391ffb640 1 -- 172.21.15.3:0/3864274132 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c004200 con 0x55dfec185f60 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 10 client.16818 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.040 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55dfec0e26b0 con 0x55dfec185f60 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc391ffb640 1 -- 172.21.15.3:0/3864274132 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55dfebde37e0 con 0x55dfec185f60 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc391ffb640 1 -- 172.21.15.3:0/3864274132 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc39c04d9b0 con 0x55dfec185f60 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 10 client.16818 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 10 client.16818 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 20 client.16818 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec196600 0x55dfec198a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 4 client.16818 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dfebe20720 con 0x55dfec196600 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.028+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec196600 0x55dfec198a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.029+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec196600 0x55dfec198a40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=186580117de0543c server_cookie=ce5ecced2a47e7ae in_seq=0 out_seq=0 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.029+0000 7fc391ffb640 10 client.16818 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc391ffb640 1 -- 172.21.15.3:0/3864274132 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55dfebe20720 con 0x55dfec196600 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc391ffb640 10 client.16818 handle_command_reply: tid=0 2024-02-09T22:54:53.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 client.16818 shutdown 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 msgr2=0x55dfec183a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec183a20 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7fc39c036000 tx=0x7fc39c045040 comp rx=0 tx=0).stop 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 shutdown_connections 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec196600 0x55dfec198a40 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec1834e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec183a20 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/3864274132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec182fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc391ffb640 0 client.16818 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 >> 172.21.15.3:0/3864274132 conn(0x55dfec177100 msgr2=0x55dfec178e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 shutdown_connections 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/3864274132 wait complete. 2024-02-09T22:54:53.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.030+0000 7fc3b3bb7740 20 client.16818 trim_cache size 0 max 16384 2024-02-09T22:54:53.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.042+0000 7fc3b3bb7740 1 Processor -- start 2024-02-09T22:54:53.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.042+0000 7fc3b3bb7740 1 -- start start 2024-02-09T22:54:53.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 0x55dfec1856d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec183840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec19d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.052 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec069130 con 0x55dfec17ca30 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec17eb80 con 0x55dfec17d3e0 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec17d170 con 0x55dfec185f60 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 0x55dfec1856d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 0x55dfec1856d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35638/0 (socket says 172.21.15.3:35638) 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2632197615 learned_addr learned my addr 172.21.15.3:0/2632197615 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec183840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec19d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2632197615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 msgr2=0x55dfec19d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec19d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2632197615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 msgr2=0x55dfec183840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec183840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 -- 172.21.15.3:0/2632197615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dfec07ec90 con 0x55dfec17ca30 2024-02-09T22:54:53.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 0x55dfec1856d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc3a409a4e0 tx=0x7fc3a4010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c1ee4872fe026a3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc393fff640 1 -- 172.21.15.3:0/2632197615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a409a860 con 0x55dfec17ca30 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc393fff640 1 -- 172.21.15.3:0/2632197615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3a40950d0 con 0x55dfec17ca30 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.043+0000 7fc393fff640 1 -- 172.21.15.3:0/2632197615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4096e50 con 0x55dfec17ca30 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/2632197615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 msgr2=0x55dfec1856d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 0x55dfec1856d0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc3a409a4e0 tx=0x7fc3a4010e70 comp rx=0 tx=0).stop 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/2632197615 shutdown_connections 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec183840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec185f60 0x55dfec19d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/2632197615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec17ca30 0x55dfec1856d0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/2632197615 >> 172.21.15.3:0/2632197615 conn(0x55dfec177100 msgr2=0x55dfec17d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/2632197615 shutdown_connections 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/2632197615 wait complete. 2024-02-09T22:54:53.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.044+0000 7fc3b3bb7740 1 -- start start 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 Processor -- start 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec18acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec18b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec1ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec18b9b0 con 0x55dfec185f60 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec18bb20 con 0x55dfec17d3e0 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dfec1ec060 con 0x55dfec17ca30 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3b3bb7740 10 client.0 _ll_register_callbacks cb 0x7fc3a4010d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec1ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aa59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec18acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec1ebb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35652/0 (socket says 172.21.15.3:35652) 2024-02-09T22:54:53.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 -- 172.21.15.3:0/874068385 learned_addr learned my addr 172.21.15.3:0/874068385 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3a9d9b640 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec18b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 -- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 msgr2=0x55dfec18acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec18acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 -- 172.21.15.3:0/874068385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 msgr2=0x55dfec18b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec18b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 -- 172.21.15.3:0/874068385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dfec0a6210 con 0x55dfec185f60 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.045+0000 7fc3aad9d640 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec1ebb20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc39c035fc0 tx=0x7fc39c032520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f615a03c335692e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc391ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc391ffb640 1 -- 172.21.15.3:0/874068385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c04d620 con 0x55dfec185f60 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc391ffb640 1 -- 172.21.15.3:0/874068385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc39c034540 con 0x55dfec185f60 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3a409aee0 con 0x55dfec185f60 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc391ffb640 1 -- 172.21.15.3:0/874068385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c04d780 con 0x55dfec185f60 2024-02-09T22:54:53.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 10 client.16824 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55dfec0a4a70 con 0x55dfec185f60 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc391ffb640 1 -- 172.21.15.3:0/874068385 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc3a409aee0 con 0x55dfec185f60 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc391ffb640 1 -- 172.21.15.3:0/874068385 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc39c057450 con 0x55dfec185f60 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 10 client.16824 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 10 client.16824 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 20 client.16824 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec1ed5a0 0x55dfec1ef990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 4 client.16824 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.046+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dfebe6f890 con 0x55dfec1ed5a0 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec1ed5a0 0x55dfec1ef990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc391ffb640 10 client.16824 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3aa59c640 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec1ed5a0 0x55dfec1ef990 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2e9a616ff87d2d7a server_cookie=7cfb8d0b59e57045 in_seq=0 out_seq=0 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc391ffb640 1 -- 172.21.15.3:0/874068385 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dfebe6f890 con 0x55dfec1ed5a0 2024-02-09T22:54:53.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc391ffb640 10 client.16824 handle_command_reply: tid=0 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 client.16824 shutdown 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 msgr2=0x55dfec1ebb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec1ebb20 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fc39c035fc0 tx=0x7fc39c032520 comp rx=0 tx=0).stop 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 shutdown_connections 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55dfec1ed5a0 0x55dfec1ef990 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55dfec17d3e0 0x55dfec18b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55dfec185f60 0x55dfec1ebb20 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 --2- 172.21.15.3:0/874068385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55dfec17ca30 0x55dfec18acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc391ffb640 0 client.16824 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 >> 172.21.15.3:0/874068385 conn(0x55dfec177100 msgr2=0x55dfec19d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 shutdown_connections 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.047+0000 7fc3b3bb7740 1 -- 172.21.15.3:0/874068385 wait complete. 2024-02-09T22:54:53.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b3bb7740 20 client.16824 trim_cache size 0 max 16384 2024-02-09T22:54:53.059 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:53.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc3940a2230 msgr2=0x7fc3940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc3940a2230 0x7fc3940a46f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fc39c004af0 tx=0x7fc39c032ec0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 msgr2=0x7fc3ac19d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac19d600 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400d950 tx=0x7fc3a400de10 comp rx=0 tx=0).stop 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 shutdown_connections 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc3940a2230 0x7fc3940a46f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc3ac105420 0x7fc3ac19db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc3ac1976c0 0x7fc3ac197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 --2- 172.21.15.3:0/2791298944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc3ac1003a0 0x7fc3ac19d600 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 >> 172.21.15.3:0/2791298944 conn(0x7fc3ac077300 msgr2=0x7fc3ac069cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 shutdown_connections 2024-02-09T22:54:53.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.048+0000 7fc3b0f0f640 1 -- 172.21.15.3:0/2791298944 wait complete. 2024-02-09T22:54:53.064 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:53.064 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:54:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2638618764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c1014b0 msgr2=0x7f6c9c104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2638618764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c1014b0 0x7f6c9c104c90 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6c90009930 tx=0x7f6c9002bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2638618764 shutdown_connections 2024-02-09T22:54:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2638618764 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c9c105370 0x7f6c9c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2638618764 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c1014b0 0x7f6c9c104c90 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2638618764 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c100b00 0x7f6c9c100ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2638618764 >> 172.21.15.3:0/2638618764 conn(0x7f6c9c076630 msgr2=0x7f6c9c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2638618764 shutdown_connections 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.171+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2638618764 wait complete. 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 Processor -- start 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 -- start start 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 0x7f6c9c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c105370 0x7f6c9c19db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c9c1976a0 0x7f6c9c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c198170 con 0x7f6c9c100b00 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1982e0 con 0x7f6c9c1976a0 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6ca1e15640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9c1984a0 con 0x7f6c9c105370 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 0x7f6c9c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 0x7f6c9c19d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35674/0 (socket says 172.21.15.3:35674) 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2790996564 learned_addr learned my addr 172.21.15.3:0/2790996564 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9affd640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c105370 0x7f6c9c19db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c9c1976a0 0x7f6c9c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2790996564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c105370 msgr2=0x7f6c9c19db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c105370 0x7f6c9c19db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2790996564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c9c1976a0 msgr2=0x7f6c9c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c9c1976a0 0x7f6c9c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.172+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2790996564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c90009560 con 0x7f6c9c100b00 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.173+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 0x7f6c9c19d5e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e9f0 tx=0x7f6c8c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.173+0000 7f6c98ff9640 1 -- 172.21.15.3:0/2790996564 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c009720 con 0x7f6c9c100b00 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.173+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9c1a4440 con 0x7f6c9c100b00 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.173+0000 7f6c98ff9640 1 -- 172.21.15.3:0/2790996564 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c00cd80 con 0x7f6c9c100b00 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.173+0000 7f6c98ff9640 1 -- 172.21.15.3:0/2790996564 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c010690 con 0x7f6c9c100b00 2024-02-09T22:54:53.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.173+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c1a4920 con 0x7f6c9c100b00 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.175+0000 7f6c98ff9640 1 -- 172.21.15.3:0/2790996564 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6c8c0046c0 con 0x7f6c9c100b00 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.175+0000 7f6ca49e1740 1 Processor -- start 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.175+0000 7f6ca49e1740 1 -- start start 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.175+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dba30 0x55ad132dbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dc3e0 0x55ad132e4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad132e73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad13297020 con 0x55ad132e4f60 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad1324a320 con 0x55ad132dc3e0 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad13247850 con 0x55ad132dba30 2024-02-09T22:54:53.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c98ff9640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6c840a2230 0x7f6c840a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad132e73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dc3e0 0x55ad132e4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad132e73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35678/0 (socket says 172.21.15.3:35678) 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 -- 172.21.15.3:0/2549967928 learned_addr learned my addr 172.21.15.3:0/2549967928 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dba30 0x55ad132dbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6c840a2230 0x7f6c840a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c98ff9640 1 -- 172.21.15.3:0/2790996564 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6c8c014070 con 0x7f6c9c100b00 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 -- 172.21.15.3:0/2549967928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dba30 msgr2=0x55ad132dbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dba30 0x55ad132dbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 -- 172.21.15.3:0/2549967928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dc3e0 msgr2=0x55ad132e4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dc3e0 0x55ad132e4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 -- 172.21.15.3:0/2549967928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad13205210 con 0x55ad132e4f60 2024-02-09T22:54:53.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dc3e0 0x55ad132e4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9affd640 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad132e73b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005ab0 tx=0x7f6c9002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dc100619cfe0b91 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6c840a2230 0x7f6c840a46f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c94005fd0 tx=0x7f6c94009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.176+0000 7f6c81ffb640 1 -- 172.21.15.3:0/2549967928 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90004740 con 0x55ad132e4f60 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6c81ffb640 1 -- 172.21.15.3:0/2549967928 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c900048a0 con 0x55ad132e4f60 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6c81ffb640 1 -- 172.21.15.3:0/2549967928 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90004ff0 con 0x55ad132e4f60 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2549967928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 msgr2=0x55ad132e73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad132e73b0 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6c90005ab0 tx=0x7f6c9002c5d0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2549967928 shutdown_connections 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dc3e0 0x55ad132e4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad132e73b0 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2549967928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dba30 0x55ad132dbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2549967928 >> 172.21.15.3:0/2549967928 conn(0x55ad132d6100 msgr2=0x55ad132d64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2549967928 shutdown_connections 2024-02-09T22:54:53.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.177+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2549967928 wait complete. 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 -- start start 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 Processor -- start 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132e1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 0x55ad132e23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132e4f60 0x55ad132e28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad13243030 con 0x55ad132dc3e0 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad12f16430 con 0x55ad132dba30 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad12ed2e00 con 0x55ad132e4f60 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6ca49e1740 10 client.0 _ll_register_callbacks cb 0x55ad13238030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 0x55ad132e23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 0x55ad132e23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35694/0 (socket says 172.21.15.3:35694) 2024-02-09T22:54:53.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 -- 172.21.15.3:0/1147194439 learned_addr learned my addr 172.21.15.3:0/1147194439 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132e1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132e4f60 0x55ad132e28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 -- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132e4f60 msgr2=0x55ad132e28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132e4f60 0x55ad132e28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 -- 172.21.15.3:0/1147194439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 msgr2=0x55ad132e1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132e1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.178+0000 7f6c9affd640 1 -- 172.21.15.3:0/1147194439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9c0ff780 con 0x55ad132dc3e0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c9affd640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 0x55ad132e23a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6c9002c3e0 tx=0x7f6c90004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b501d791c3dacd10 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c7affd640 1 -- 172.21.15.3:0/1147194439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c90005200 con 0x55ad132dc3e0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c7affd640 1 -- 172.21.15.3:0/1147194439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6c9002cbe0 con 0x55ad132dc3e0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad12f427e0 con 0x55ad132dc3e0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c7affd640 1 -- 172.21.15.3:0/1147194439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c90005360 con 0x55ad132dc3e0 2024-02-09T22:54:53.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 10 client.16836 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55ad132416b0 con 0x55ad132dc3e0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c7affd640 1 -- 172.21.15.3:0/1147194439 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ad12f427e0 con 0x55ad132dc3e0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6c7affd640 1 -- 172.21.15.3:0/1147194439 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6c9003c050 con 0x55ad132dc3e0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 10 client.16836 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 10 client.16836 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 20 client.16836 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad132f50e0 0x55ad132f7500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 4 client.16836 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.179+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad12f7f720 con 0x55ad132f50e0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.180+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad132f50e0 0x55ad132f7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.180+0000 7f6c7affd640 10 client.16836 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.180+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad132f50e0 0x55ad132f7500 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c48db80e5629b826 server_cookie=d84fa8694a38195e in_seq=0 out_seq=0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6c7affd640 1 -- 172.21.15.3:0/1147194439 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad12f7f720 con 0x55ad132f50e0 2024-02-09T22:54:53.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6c7affd640 10 client.16836 handle_command_reply: tid=0 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 client.16836 shutdown 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 msgr2=0x55ad132e23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 0x55ad132e23a0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f6c9002c3e0 tx=0x7f6c90004300 comp rx=0 tx=0).stop 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 shutdown_connections 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad132f50e0 0x55ad132f7500 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132e4f60 0x55ad132e28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6c7affd640 0 client.16836 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dc3e0 0x55ad132e23a0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1147194439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132e1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 >> 172.21.15.3:0/1147194439 conn(0x55ad132d6100 msgr2=0x55ad132d7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 shutdown_connections 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1147194439 wait complete. 2024-02-09T22:54:53.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.181+0000 7f6ca49e1740 20 client.16836 trim_cache size 0 max 16384 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.193+0000 7f6ca49e1740 1 Processor -- start 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.193+0000 7f6ca49e1740 1 -- start start 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 0x55ad132e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132df0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132e4f60 0x55ad132df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad131c8130 con 0x55ad132dba30 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad132dad50 con 0x55ad132e4f60 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad132e03e0 con 0x55ad132dc3e0 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 0x55ad132e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 0x55ad132e4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35706/0 (socket says 172.21.15.3:35706) 2024-02-09T22:54:53.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2458637226 learned_addr learned my addr 172.21.15.3:0/2458637226 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9affd640 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132df0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132e4f60 0x55ad132df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2458637226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 msgr2=0x55ad132df0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132df0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2458637226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132e4f60 msgr2=0x55ad132df600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132e4f60 0x55ad132df600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 -- 172.21.15.3:0/2458637226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad131ddc90 con 0x55ad132dba30 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9affd640 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132df0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 0x55ad132e4590 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0cbbd0 tx=0x7f6c8c0c5710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf383b2daca0046 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c817fa640 1 -- 172.21.15.3:0/2458637226 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c09ef60 con 0x55ad132dba30 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c817fa640 1 -- 172.21.15.3:0/2458637226 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c8c09d040 con 0x55ad132dba30 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.194+0000 7f6c817fa640 1 -- 172.21.15.3:0/2458637226 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c8c0a3120 con 0x55ad132dba30 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2458637226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 msgr2=0x55ad132e4590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 0x55ad132e4590 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c0cbbd0 tx=0x7f6c8c0c5710 comp rx=0 tx=0).stop 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2458637226 shutdown_connections 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132e4f60 0x55ad132df600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132df0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/2458637226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132dba30 0x55ad132e4590 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2458637226 >> 172.21.15.3:0/2458637226 conn(0x55ad132d6100 msgr2=0x55ad132e0730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2458637226 shutdown_connections 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 -- 172.21.15.3:0/2458637226 wait complete. 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.195+0000 7f6ca49e1740 1 -- start start 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 Processor -- start 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132eb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132eb810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad133456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad13345c30 con 0x55ad132e4f60 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad13345d70 con 0x55ad132dba30 2024-02-09T22:54:53.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad13345f30 con 0x55ad132dc3e0 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6ca49e1740 10 client.0 _ll_register_callbacks cb 0x7f6c8c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad133456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad133456f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35722/0 (socket says 172.21.15.3:35722) 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 -- 172.21.15.3:0/1576163276 learned_addr learned my addr 172.21.15.3:0/1576163276 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132eb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9affd640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132eb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 -- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 msgr2=0x55ad132eb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132eb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 -- 172.21.15.3:0/1576163276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 msgr2=0x55ad132eb2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132eb2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 -- 172.21.15.3:0/1576163276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad132416b0 con 0x55ad132e4f60 2024-02-09T22:54:53.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c9bfff640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad133456f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400fde0 tx=0x7f6c94004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9441163c295ac13 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.196+0000 7f6c7affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6c7affd640 1 -- 172.21.15.3:0/1576163276 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94025040 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6c7affd640 1 -- 172.21.15.3:0/1576163276 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6c94010e00 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad132e1c90 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6c7affd640 1 -- 172.21.15.3:0/1576163276 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c94004b20 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 10 client.16842 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x7f6c8c0b0e70 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6c7affd640 1 -- 172.21.15.3:0/1576163276 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ad132e1c90 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6c7affd640 1 -- 172.21.15.3:0/1576163276 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6c9402a480 con 0x55ad132e4f60 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 10 client.16842 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 10 client.16842 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 20 client.16842 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad1334d8a0 0x55ad1334fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 4 client.16842 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x7f6c900048a0 con 0x55ad1334d8a0 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.197+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad1334d8a0 0x55ad1334fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6c9b7fe640 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad1334d8a0 0x55ad1334fcc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f7e6b256b60e3f3 server_cookie=1df908a2196806df in_seq=0 out_seq=0 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6c7affd640 10 client.16842 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6c7affd640 1 -- 172.21.15.3:0/1576163276 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f6c900048a0 con 0x55ad1334d8a0 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6c7affd640 10 client.16842 handle_command_reply: tid=0 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 client.16842 shutdown 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 msgr2=0x55ad133456f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad133456f0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f6c9400fde0 tx=0x7f6c94004ab0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 shutdown_connections 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad1334d8a0 0x55ad1334fcc0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad132e4f60 0x55ad133456f0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad132dc3e0 0x55ad132eb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6c7affd640 0 client.16842 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:54:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 --2- 172.21.15.3:0/1576163276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad132dba30 0x55ad132eb2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 >> 172.21.15.3:0/1576163276 conn(0x55ad132d6100 msgr2=0x55ad132e1e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 shutdown_connections 2024-02-09T22:54:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 1 -- 172.21.15.3:0/1576163276 wait complete. 2024-02-09T22:54:53.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.198+0000 7f6ca49e1740 20 client.16842 trim_cache size 0 max 16384 2024-02-09T22:54:53.209 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6c840a2230 msgr2=0x7f6c840a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6c840a2230 0x7f6c840a46f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f6c94005fd0 tx=0x7f6c94009500 comp rx=0 tx=0).stop 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 msgr2=0x7f6c9c19d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 0x7f6c9c19d5e0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6c8c00e9f0 tx=0x7f6c8c00edf0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 shutdown_connections 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6c840a2230 0x7f6c840a46f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6c9c1976a0 0x7f6c9c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6c9c105370 0x7f6c9c19db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 --2- 172.21.15.3:0/2790996564 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6c9c100b00 0x7f6c9c19d5e0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 >> 172.21.15.3:0/2790996564 conn(0x7f6c9c076630 msgr2=0x7f6c9c103510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 shutdown_connections 2024-02-09T22:54:53.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.199+0000 7f6ca1e15640 1 -- 172.21.15.3:0/2790996564 wait complete. 2024-02-09T22:54:53.214 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:53.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 -- 172.21.15.3:0/4083396560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f4102d70 msgr2=0x7f29f410b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/4083396560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f4102d70 0x7f29f410b250 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f29e4009900 tx=0x7f29e402bcd0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 -- 172.21.15.3:0/4083396560 shutdown_connections 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/4083396560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29f410b820 0x7f29f410dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/4083396560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f4102d70 0x7f29f410b250 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/4083396560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29f4102450 0x7f29f4102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 -- 172.21.15.3:0/4083396560 >> 172.21.15.3:0/4083396560 conn(0x7f29f40772e0 msgr2=0x7f29f40776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 -- 172.21.15.3:0/4083396560 shutdown_connections 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 -- 172.21.15.3:0/4083396560 wait complete. 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 Processor -- start 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.318+0000 7f29fb9e2640 1 -- start start 2024-02-09T22:54:53.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29f4102450 0x7f29f41000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29f4102d70 0x7f29f41005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 0x7f29f4100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f41a5c30 con 0x7f29f410b820 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f41a5da0 con 0x7f29f4102d70 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f41a5f60 con 0x7f29f4102450 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 0x7f29f4100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f977e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29f4102450 0x7f29f41000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 0x7f29f4100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35746/0 (socket says 172.21.15.3:35746) 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f8f7d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29f4102d70 0x7f29f41005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 -- 172.21.15.3:0/219203030 learned_addr learned my addr 172.21.15.3:0/219203030 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 -- 172.21.15.3:0/219203030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29f4102450 msgr2=0x7f29f41000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29f4102450 0x7f29f41000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 -- 172.21.15.3:0/219203030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29f4102d70 msgr2=0x7f29f41005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29f4102d70 0x7f29f41005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 -- 172.21.15.3:0/219203030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e4009560 con 0x7f29f410b820 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 0x7f29f4100b20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00ea60 tx=0x7f29ec00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29ea7fc640 1 -- 172.21.15.3:0/219203030 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec00ce40 con 0x7f29f410b820 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29ea7fc640 1 -- 172.21.15.3:0/219203030 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29ec005020 con 0x7f29f410b820 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29ea7fc640 1 -- 172.21.15.3:0/219203030 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec010690 con 0x7f29f410b820 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f41a6240 con 0x7f29f410b820 2024-02-09T22:54:53.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.319+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f41a66d0 con 0x7f29f410b820 2024-02-09T22:54:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.321+0000 7f29ea7fc640 1 -- 172.21.15.3:0/219203030 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f29ec005500 con 0x7f29f410b820 2024-02-09T22:54:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 Processor -- start 2024-02-09T22:54:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 -- start start 2024-02-09T22:54:53.332 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 0x561dc0a07e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29ea7fc640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f29e00a2230 0x7f29e00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a10990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc09c3020 con 0x561dc0a07a30 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0976320 con 0x561dc0a10f60 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0973850 con 0x561dc0a083e0 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29f8f7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 0x561dc0a07e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29f977e640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f29e00a2230 0x7f29e00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29f8f7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 0x561dc0a07e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35764/0 (socket says 172.21.15.3:35764) 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29f8f7d640 1 -- 172.21.15.3:0/2854305700 learned_addr learned my addr 172.21.15.3:0/2854305700 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29ea7fc640 1 -- 172.21.15.3:0/219203030 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f29ec014070 con 0x7f29f410b820 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.322+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a10990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.333 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f8f7d640 1 -- 172.21.15.3:0/2854305700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 msgr2=0x561dc0a10990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a10990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f8f7d640 1 -- 172.21.15.3:0/2854305700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 msgr2=0x561dc0a133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f8f7d640 1 -- 172.21.15.3:0/2854305700 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561dc0931210 con 0x561dc0a07a30 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f977e640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f29e00a2230 0x7f29e00a46f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009bd0 tx=0x7f29f0005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 0x561dc0a07e10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c210 tx=0x7f29e4005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c6af0566fe61b4e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29df7fe640 1 -- 172.21.15.3:0/2854305700 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4004ae0 con 0x561dc0a07a30 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29df7fe640 1 -- 172.21.15.3:0/2854305700 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29e4004030 con 0x561dc0a07a30 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.323+0000 7f29df7fe640 1 -- 172.21.15.3:0/2854305700 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4005390 con 0x561dc0a07a30 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 -- 172.21.15.3:0/2854305700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 msgr2=0x561dc0a07e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 0x561dc0a07e10 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f29e402c210 tx=0x7f29e4005b50 comp rx=0 tx=0).stop 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 -- 172.21.15.3:0/2854305700 shutdown_connections 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a10990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/2854305700 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a07a30 0x561dc0a07e10 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 -- 172.21.15.3:0/2854305700 >> 172.21.15.3:0/2854305700 conn(0x561dc0a02100 msgr2=0x561dc0a024e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 -- 172.21.15.3:0/2854305700 shutdown_connections 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 -- 172.21.15.3:0/2854305700 wait complete. 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 -- start start 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.324+0000 7f29fe5b8740 1 Processor -- start 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a0df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a0e450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a0e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc096f030 con 0x561dc0a10f60 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0642430 con 0x561dc0a07a30 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc05fee00 con 0x561dc0a083e0 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 10 client.0 _ll_register_callbacks cb 0x561dc0964030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a0e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a0e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35766/0 (socket says 172.21.15.3:35766) 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1246419288 learned_addr learned my addr 172.21.15.3:0/1246419288 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f977e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a0df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a0e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 msgr2=0x561dc0a0e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a0e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1246419288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 msgr2=0x561dc0a0df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a0df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1246419288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f410e8d0 con 0x561dc0a10f60 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a0e990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f29ec018000 tx=0x7f29ec09b700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffc55c01ed416ac server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29dcff9640 1 -- 172.21.15.3:0/1246419288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec0a36e0 con 0x561dc0a10f60 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29dcff9640 1 -- 172.21.15.3:0/1246419288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f29ec0aef90 con 0x561dc0a10f60 2024-02-09T22:54:53.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.325+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561dc066e7e0 con 0x561dc0a10f60 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29dcff9640 1 -- 172.21.15.3:0/1246419288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29ec0af150 con 0x561dc0a10f60 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 10 client.16854 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x561dc096d6b0 con 0x561dc0a10f60 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29dcff9640 1 -- 172.21.15.3:0/1246419288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29ec0a7030 con 0x561dc0a10f60 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29dcff9640 1 -- 172.21.15.3:0/1246419288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f29ec01f060 con 0x561dc0a10f60 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 10 client.16854 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 10 client.16854 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 20 client.16854 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a21500 0x561dc0a23920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 4 client.16854 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561dc06ab720 con 0x561dc0a21500 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29f977e640 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a21500 0x561dc0a23920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29dcff9640 10 client.16854 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:53.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.326+0000 7f29f977e640 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a21500 0x561dc0a23920 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce8ee96556c1cc35 server_cookie=96dcf09b2aff85be in_seq=0 out_seq=0 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29dcff9640 1 -- 172.21.15.3:0/1246419288 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561dc06ab720 con 0x561dc0a21500 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29dcff9640 10 client.16854 handle_command_reply: tid=0 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 client.16854 shutdown 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 msgr2=0x561dc0a0e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a0e990 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f29ec018000 tx=0x7f29ec09b700 comp rx=0 tx=0).stop 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 shutdown_connections 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a21500 0x561dc0a23920 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a0e990 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a0e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29dcff9640 0 client.16854 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1246419288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a0df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 >> 172.21.15.3:0/1246419288 conn(0x561dc0a02100 msgr2=0x561dc0a03e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 shutdown_connections 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1246419288 wait complete. 2024-02-09T22:54:53.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.327+0000 7f29fe5b8740 20 client.16854 trim_cache size 0 max 16384 2024-02-09T22:54:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.339+0000 7f29fe5b8740 1 Processor -- start 2024-02-09T22:54:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.339+0000 7f29fe5b8740 1 -- start start 2024-02-09T22:54:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a07a30 0x561dc0a10640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 0x561dc0a0e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a0b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc08f4130 con 0x561dc0a083e0 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0a019c0 con 0x561dc0a10f60 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0a06ac0 con 0x561dc0a07a30 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 0x561dc0a0e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f977e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a07a30 0x561dc0a10640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a0b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 0x561dc0a0e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35770/0 (socket says 172.21.15.3:35770) 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f977e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a07a30 0x561dc0a10640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:55278/0 (socket says 172.21.15.3:55278) 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 -- 172.21.15.3:0/355863775 learned_addr learned my addr 172.21.15.3:0/355863775 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 -- 172.21.15.3:0/355863775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a07a30 msgr2=0x561dc0a10640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a07a30 0x561dc0a10640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 -- 172.21.15.3:0/355863775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 msgr2=0x561dc0a0b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a0b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 -- 172.21.15.3:0/355863775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561dc0909c90 con 0x561dc0a083e0 2024-02-09T22:54:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 0x561dc0a0e7b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x561dc0a07500 tx=0x7f29e4004f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc99c66534fd1f15 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29deffd640 1 -- 172.21.15.3:0/355863775 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e40043a0 con 0x561dc0a083e0 2024-02-09T22:54:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29deffd640 1 -- 172.21.15.3:0/355863775 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29e4004df0 con 0x561dc0a083e0 2024-02-09T22:54:53.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.340+0000 7f29deffd640 1 -- 172.21.15.3:0/355863775 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e4005540 con 0x561dc0a083e0 2024-02-09T22:54:53.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 -- 172.21.15.3:0/355863775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 msgr2=0x561dc0a0e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 0x561dc0a0e7b0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x561dc0a07500 tx=0x7f29e4004f70 comp rx=0 tx=0).stop 2024-02-09T22:54:53.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 -- 172.21.15.3:0/355863775 shutdown_connections 2024-02-09T22:54:53.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a10f60 0x561dc0a0b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a083e0 0x561dc0a0e7b0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/355863775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a07a30 0x561dc0a10640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 -- 172.21.15.3:0/355863775 >> 172.21.15.3:0/355863775 conn(0x561dc0a02100 msgr2=0x561dc0a28280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 -- 172.21.15.3:0/355863775 shutdown_connections 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 -- 172.21.15.3:0/355863775 wait complete. 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 -- start start 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.341+0000 7f29fe5b8740 1 Processor -- start 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a18380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a16770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a16da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0a17440 con 0x561dc0a10f60 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0a175b0 con 0x561dc0a07a30 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561dc0a77a10 con 0x561dc0a083e0 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29fe5b8740 10 client.0 _ll_register_callbacks cb 0x7f29f0005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a16da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a16da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35782/0 (socket says 172.21.15.3:35782) 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1740240410 learned_addr learned my addr 172.21.15.3:0/1740240410 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f977e640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a18380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f8f7d640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a16770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 msgr2=0x561dc0a16770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a16770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1740240410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 msgr2=0x561dc0a18380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a18380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 -- 172.21.15.3:0/1740240410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561dc0931210 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f977e640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a18380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29f9f7f640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a16da0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0058b0 tx=0x7f29ec0a3500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d2f0eeac471a35 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.342+0000 7f29dcff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29dcff9640 1 -- 172.21.15.3:0/1740240410 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29ec09a8c0 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29dcff9640 1 -- 172.21.15.3:0/1740240410 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f29ec0b3c70 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561dc06fa890 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29dcff9640 1 -- 172.21.15.3:0/1740240410 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29ec005190 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 10 client.16860 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x561dc092fa70 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29dcff9640 1 -- 172.21.15.3:0/1740240410 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561dc06fa890 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29dcff9640 1 -- 172.21.15.3:0/1740240410 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f29ec01f070 con 0x561dc0a10f60 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 10 client.16860 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 10 client.16860 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:54:53.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 20 client.16860 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a78ed0 0x561dc0a7b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 4 client.16860 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561dc061a700 con 0x561dc0a78ed0 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.343+0000 7f29f977e640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a78ed0 0x561dc0a7b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29dcff9640 10 client.16860 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29f977e640 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a78ed0 0x561dc0a7b2c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c04fff6006d14781 server_cookie=2702d26f38804c66 in_seq=0 out_seq=0 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29dcff9640 1 -- 172.21.15.3:0/1740240410 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561dc061a700 con 0x561dc0a78ed0 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29dcff9640 10 client.16860 handle_command_reply: tid=0 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 client.16860 shutdown 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 msgr2=0x561dc0a16da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a16da0 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f29ec0058b0 tx=0x7f29ec0a3500 comp rx=0 tx=0).stop 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 shutdown_connections 2024-02-09T22:54:53.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x561dc0a78ed0 0x561dc0a7b2c0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561dc0a10f60 0x561dc0a16da0 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561dc0a083e0 0x561dc0a16770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29dcff9640 0 client.16860 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 --2- 172.21.15.3:0/1740240410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561dc0a07a30 0x561dc0a18380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 >> 172.21.15.3:0/1740240410 conn(0x561dc0a02100 msgr2=0x561dc0a009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 shutdown_connections 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 1 -- 172.21.15.3:0/1740240410 wait complete. 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.344+0000 7f29fe5b8740 20 client.16860 trim_cache size 0 max 16384 2024-02-09T22:54:53.357 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f29e00a2230 msgr2=0x7f29e00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f29e00a2230 0x7f29e00a46f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009bd0 tx=0x7f29f0005f90 comp rx=0 tx=0).stop 2024-02-09T22:54:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 msgr2=0x7f29f4100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 0x7f29f4100b20 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f29ec00ea60 tx=0x7f29ec00ef20 comp rx=0 tx=0).stop 2024-02-09T22:54:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 shutdown_connections 2024-02-09T22:54:53.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f29e00a2230 0x7f29e00a46f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f29f4102d70 0x7f29f41005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f29f410b820 0x7f29f4100b20 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 --2- 172.21.15.3:0/219203030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f29f4102450 0x7f29f41000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 >> 172.21.15.3:0/219203030 conn(0x7f29f40772e0 msgr2=0x7f29f40fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 shutdown_connections 2024-02-09T22:54:53.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.345+0000 7f29fb9e2640 1 -- 172.21.15.3:0/219203030 wait complete. 2024-02-09T22:54:53.360 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:53.360 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:54:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 -- 172.21.15.3:0/3524314996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae64105400 msgr2=0x7fae6410cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 --2- 172.21.15.3:0/3524314996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae64105400 0x7fae6410cfd0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fae5c009930 tx=0x7fae5c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 -- 172.21.15.3:0/3524314996 shutdown_connections 2024-02-09T22:54:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 --2- 172.21.15.3:0/3524314996 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae64100ce0 0x7fae64104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 --2- 172.21.15.3:0/3524314996 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae64105400 0x7fae6410cfd0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 --2- 172.21.15.3:0/3524314996 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae64100330 0x7fae64100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 -- 172.21.15.3:0/3524314996 >> 172.21.15.3:0/3524314996 conn(0x7fae64077250 msgr2=0x7fae64077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 -- 172.21.15.3:0/3524314996 shutdown_connections 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 -- 172.21.15.3:0/3524314996 wait complete. 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.468+0000 7fae689c9640 1 Processor -- start 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 -- start start 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae64100330 0x7fae6410d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae64105400 0x7fae6410ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 0x7fae641a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641a5fe0 con 0x7fae6410e330 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641a6150 con 0x7fae64100330 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae689c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae641a6310 con 0x7fae64105400 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 0x7fae641a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 0x7fae641a5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35798/0 (socket says 172.21.15.3:35798) 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 -- 172.21.15.3:0/3346115814 learned_addr learned my addr 172.21.15.3:0/3346115814 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae6259c640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae64100330 0x7fae6410d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae61d9b640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae64105400 0x7fae6410ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 -- 172.21.15.3:0/3346115814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae64105400 msgr2=0x7fae6410ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae64105400 0x7fae6410ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 -- 172.21.15.3:0/3346115814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae64100330 msgr2=0x7fae6410d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae64100330 0x7fae6410d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 -- 172.21.15.3:0/3346115814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae5c009560 con 0x7fae6410e330 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.469+0000 7fae62d9d640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 0x7fae641a5a70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fae5c005940 tx=0x7fae5c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.470+0000 7fae5b7fe640 1 -- 172.21.15.3:0/3346115814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c004350 con 0x7fae6410e330 2024-02-09T22:54:53.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.470+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae641a6590 con 0x7fae6410e330 2024-02-09T22:54:53.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.470+0000 7fae5b7fe640 1 -- 172.21.15.3:0/3346115814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae5c0044b0 con 0x7fae6410e330 2024-02-09T22:54:53.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.470+0000 7fae5b7fe640 1 -- 172.21.15.3:0/3346115814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae5c003cc0 con 0x7fae6410e330 2024-02-09T22:54:53.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.470+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae641a6af0 con 0x7fae6410e330 2024-02-09T22:54:53.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:53 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1793127498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:54:53.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae5b7fe640 1 -- 172.21.15.3:0/3346115814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fae5c005580 con 0x7fae6410e330 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 Processor -- start 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 -- start start 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4638a30 0x5643c4638e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4641990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4641f60 0x5643c46443b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c45f4020 con 0x5643c4638a30 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c45a7320 con 0x5643c4641f60 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6b595740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c45a4850 con 0x5643c46393e0 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae5b7fe640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae500a2230 0x7fae500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.472+0000 7fae6259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4641990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae61d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4638a30 0x5643c4638e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4641990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:55316/0 (socket says 172.21.15.3:55316) 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 -- 172.21.15.3:0/1568270237 learned_addr learned my addr 172.21.15.3:0/1568270237 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae62d9d640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae500a2230 0x7fae500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae61d9b640 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4641f60 0x5643c46443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae5b7fe640 1 -- 172.21.15.3:0/3346115814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fae5c0f0080 con 0x7fae6410e330 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 -- 172.21.15.3:0/1568270237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4641f60 msgr2=0x5643c46443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4641f60 0x5643c46443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 -- 172.21.15.3:0/1568270237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4638a30 msgr2=0x5643c4638e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4638a30 0x5643c4638e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 -- 172.21.15.3:0/1568270237 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643c4562210 con 0x5643c46393e0 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae6259c640 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4641990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fae540114f0 tx=0x7fae540119b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94764a41f233a0e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae62d9d640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae500a2230 0x7fae500a46f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0ed9c0 tx=0x7fae5c03ee80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae58ff9640 1 -- 172.21.15.3:0/1568270237 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54004d20 con 0x5643c46393e0 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae58ff9640 1 -- 172.21.15.3:0/1568270237 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae54004e80 con 0x5643c46393e0 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.473+0000 7fae58ff9640 1 -- 172.21.15.3:0/1568270237 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54005280 con 0x5643c46393e0 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 -- 172.21.15.3:0/1568270237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 msgr2=0x5643c4641990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4641990 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fae540114f0 tx=0x7fae540119b0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 -- 172.21.15.3:0/1568270237 shutdown_connections 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4641f60 0x5643c46443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4641990 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 --2- 172.21.15.3:0/1568270237 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4638a30 0x5643c4638e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 -- 172.21.15.3:0/1568270237 >> 172.21.15.3:0/1568270237 conn(0x5643c4633100 msgr2=0x5643c46334e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 -- 172.21.15.3:0/1568270237 shutdown_connections 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 -- 172.21.15.3:0/1568270237 wait complete. 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 -- start start 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.474+0000 7fae6b595740 1 Processor -- start 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c463ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c463f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c463f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c45a0030 con 0x5643c4641f60 2024-02-09T22:54:53.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c4273430 con 0x5643c4638a30 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c422fe00 con 0x5643c46393e0 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 10 client.0 _ll_register_callbacks cb 0x5643c4595030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c463f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c463f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35812/0 (socket says 172.21.15.3:35812) 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c463f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 -- 172.21.15.3:0/967334340 learned_addr learned my addr 172.21.15.3:0/967334340 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c463ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 -- 172.21.15.3:0/967334340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 msgr2=0x5643c463f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c463f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 -- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 msgr2=0x5643c463ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c463ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 -- 172.21.15.3:0/967334340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae54009c80 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae61d9b640 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c463f990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fae4c012bd0 tx=0x7fae4c00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b179f712b6beb39 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae49ffb640 1 -- 172.21.15.3:0/967334340 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c00eca0 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae49ffb640 1 -- 172.21.15.3:0/967334340 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fae4c00ee00 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.475+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643c42dc720 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae49ffb640 1 -- 172.21.15.3:0/967334340 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae4c005020 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 10 client.16872 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5643c459e6b0 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae49ffb640 1 -- 172.21.15.3:0/967334340 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5643c42dc720 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae49ffb640 1 -- 172.21.15.3:0/967334340 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fae4c02a450 con 0x5643c4641f60 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 10 client.16872 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 10 client.16872 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 20 client.16872 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c4652440 0x5643c4654860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 4 client.16872 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643c432b890 con 0x5643c4652440 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.476+0000 7fae6259c640 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c4652440 0x5643c4654860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.477+0000 7fae6259c640 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c4652440 0x5643c4654860 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=efcc08d139814785 server_cookie=efe1b123d4e6689a in_seq=0 out_seq=0 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.477+0000 7fae49ffb640 10 client.16872 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.477+0000 7fae49ffb640 1 -- 172.21.15.3:0/967334340 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5643c432b890 con 0x5643c4652440 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.477+0000 7fae49ffb640 10 client.16872 handle_command_reply: tid=0 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.477+0000 7fae6b595740 1 client.16872 shutdown 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 msgr2=0x5643c463f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c463f990 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fae4c012bd0 tx=0x7fae4c00bb40 comp rx=0 tx=0).stop 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 shutdown_connections 2024-02-09T22:54:53.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c4652440 0x5643c4654860 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c463f990 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c463f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 --2- 172.21.15.3:0/967334340 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c463ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 >> 172.21.15.3:0/967334340 conn(0x5643c4633100 msgr2=0x5643c4634e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae49ffb640 0 client.16872 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 shutdown_connections 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 1 -- 172.21.15.3:0/967334340 wait complete. 2024-02-09T22:54:53.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.478+0000 7fae6b595740 20 client.16872 trim_cache size 0 max 16384 2024-02-09T22:54:53.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 Processor -- start 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 -- start start 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c4641640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 0x5643c463c190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c4641f60 0x5643c463c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c4525130 con 0x5643c46393e0 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c4652210 con 0x5643c4638a30 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6b595740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c4633d00 con 0x5643c4641f60 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 0x5643c463c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 0x5643c463c190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35816/0 (socket says 172.21.15.3:35816) 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae6259c640 1 -- 172.21.15.3:0/2961817719 learned_addr learned my addr 172.21.15.3:0/2961817719 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae61d9b640 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c4641640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.490+0000 7fae61d9b640 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c4641f60 0x5643c463c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6259c640 1 -- 172.21.15.3:0/2961817719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c4641f60 msgr2=0x5643c463c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6259c640 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c4641f60 0x5643c463c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6259c640 1 -- 172.21.15.3:0/2961817719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 msgr2=0x5643c4641640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6259c640 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c4641640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6259c640 1 -- 172.21.15.3:0/2961817719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643c453ac90 con 0x5643c46393e0 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae61d9b640 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c4641f60 0x5643c463c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6259c640 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 0x5643c463c190 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fae540114c0 tx=0x7fae54009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367609f0a535ac2e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae4bfff640 1 -- 172.21.15.3:0/2961817719 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54029070 con 0x5643c46393e0 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae4bfff640 1 -- 172.21.15.3:0/2961817719 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae54004710 con 0x5643c46393e0 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae4bfff640 1 -- 172.21.15.3:0/2961817719 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae54003bd0 con 0x5643c46393e0 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6b595740 1 -- 172.21.15.3:0/2961817719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 msgr2=0x5643c463c190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.491+0000 7fae6b595740 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 0x5643c463c190 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fae540114c0 tx=0x7fae54009be0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- 172.21.15.3:0/2961817719 shutdown_connections 2024-02-09T22:54:53.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c4641f60 0x5643c463c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c46393e0 0x5643c463c190 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 --2- 172.21.15.3:0/2961817719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c4641640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- 172.21.15.3:0/2961817719 >> 172.21.15.3:0/2961817719 conn(0x5643c4633100 msgr2=0x5643c463a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- 172.21.15.3:0/2961817719 shutdown_connections 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- 172.21.15.3:0/2961817719 wait complete. 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- start start 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 Processor -- start 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c46483c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4648900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c46a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c46a1e70 con 0x5643c4641f60 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c46a1fe0 con 0x5643c4638a30 2024-02-09T22:54:53.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643c46a21a0 con 0x5643c46393e0 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6b595740 10 client.0 _ll_register_callbacks cb 0x7fae54009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae61d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c46483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae61d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c46a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae61d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c46a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35822/0 (socket says 172.21.15.3:35822) 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae61d9b640 1 -- 172.21.15.3:0/723149533 learned_addr learned my addr 172.21.15.3:0/723149533 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.492+0000 7fae6259c640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4648900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae61d9b640 1 -- 172.21.15.3:0/723149533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 msgr2=0x5643c4648900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae61d9b640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4648900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae61d9b640 1 -- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 msgr2=0x5643c46483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae61d9b640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c46483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae61d9b640 1 -- 172.21.15.3:0/723149533 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643c4562210 con 0x5643c4641f60 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6259c640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4648900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae61d9b640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c46a1930 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fae4c005fd0 tx=0x7fae4c00b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ae221127c84dbf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae49ffb640 1 -- 172.21.15.3:0/723149533 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae4c00b870 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae49ffb640 1 -- 172.21.15.3:0/723149533 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fae4c000e40 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643c429f7e0 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae49ffb640 1 -- 172.21.15.3:0/723149533 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae4c00be80 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6b595740 10 client.16878 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x5643c4560a70 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae49ffb640 1 -- 172.21.15.3:0/723149533 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5643c429f7e0 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae49ffb640 1 -- 172.21.15.3:0/723149533 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fae4c00eca0 con 0x5643c4641f60 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6b595740 10 client.16878 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6b595740 10 client.16878 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.493+0000 7fae6b595740 20 client.16878 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c46aabd0 0x5643c46acff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 4 client.16878 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:53.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5643c424b700 con 0x5643c46aabd0 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6259c640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c46aabd0 0x5643c46acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae49ffb640 10 client.16878 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6259c640 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c46aabd0 0x5643c46acff0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cd85dd7169067a9d server_cookie=23430ba60057f774 in_seq=0 out_seq=0 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae49ffb640 1 -- 172.21.15.3:0/723149533 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5643c424b700 con 0x5643c46aabd0 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae49ffb640 10 client.16878 handle_command_reply: tid=0 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 client.16878 shutdown 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 msgr2=0x5643c46a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c46a1930 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fae4c005fd0 tx=0x7fae4c00b410 comp rx=0 tx=0).stop 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 shutdown_connections 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5643c46aabd0 0x5643c46acff0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643c4641f60 0x5643c46a1930 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643c46393e0 0x5643c4648900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 --2- 172.21.15.3:0/723149533 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643c4638a30 0x5643c46483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae49ffb640 0 client.16878 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:54:53.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 >> 172.21.15.3:0/723149533 conn(0x5643c4633100 msgr2=0x5643c46319b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.494+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 shutdown_connections 2024-02-09T22:54:53.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae6b595740 1 -- 172.21.15.3:0/723149533 wait complete. 2024-02-09T22:54:53.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae6b595740 20 client.16878 trim_cache size 0 max 16384 2024-02-09T22:54:53.509 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:53.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae500a2230 msgr2=0x7fae500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae500a2230 0x7fae500a46f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fae5c0ed9c0 tx=0x7fae5c03ee80 comp rx=0 tx=0).stop 2024-02-09T22:54:53.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 msgr2=0x7fae641a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 0x7fae641a5a70 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fae5c005940 tx=0x7fae5c02c650 comp rx=0 tx=0).stop 2024-02-09T22:54:53.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 shutdown_connections 2024-02-09T22:54:53.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae500a2230 0x7fae500a46f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae6410e330 0x7fae641a5a70 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae64105400 0x7fae6410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 --2- 172.21.15.3:0/3346115814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae64100330 0x7fae6410d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 >> 172.21.15.3:0/3346115814 conn(0x7fae64077250 msgr2=0x7fae640fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 shutdown_connections 2024-02-09T22:54:53.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.495+0000 7fae689c9640 1 -- 172.21.15.3:0/3346115814 wait complete. 2024-02-09T22:54:53.512 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:53 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1793127498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:54:53.513 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:54:53.513 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 -- 172.21.15.3:0/2119235374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 msgr2=0x7ff06010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 --2- 172.21.15.3:0/2119235374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff06010e090 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7ff058009cd0 tx=0x7ff05802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 -- 172.21.15.3:0/2119235374 shutdown_connections 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 --2- 172.21.15.3:0/2119235374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0600ff6a0 0x7ff0600ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 --2- 172.21.15.3:0/2119235374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff06010e090 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 --2- 172.21.15.3:0/2119235374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff060106200 0x7ff0601065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 -- 172.21.15.3:0/2119235374 >> 172.21.15.3:0/2119235374 conn(0x7ff060076630 msgr2=0x7ff0600fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 -- 172.21.15.3:0/2119235374 shutdown_connections 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 -- 172.21.15.3:0/2119235374 wait complete. 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 Processor -- start 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.619+0000 7ff0658eb640 1 -- start start 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff0658eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0600ff6a0 0x7ff060100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff0658eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff060100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff0658eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff060106200 0x7ff060101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff0658eb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0601a5c10 con 0x7ff060100060 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff0658eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0601a5d80 con 0x7ff060106200 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff0658eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0601a5f40 con 0x7ff0600ff6a0 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff060100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0600ff6a0 0x7ff060100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff060100dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35840/0 (socket says 172.21.15.3:35840) 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05e7fc640 1 -- 172.21.15.3:0/620933067 learned_addr learned my addr 172.21.15.3:0/620933067 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff060106200 0x7ff060101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 -- 172.21.15.3:0/620933067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff060106200 msgr2=0x7ff060101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff060106200 0x7ff060101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 -- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 msgr2=0x7ff060100dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff060100dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 -- 172.21.15.3:0/620933067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff058009900 con 0x7ff0600ff6a0 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff060100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.620+0000 7ff05effd640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0600ff6a0 0x7ff060100880 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff05000e9f0 tx=0x7ff05000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.621+0000 7ff0648e9640 1 -- 172.21.15.3:0/620933067 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05000ccc0 con 0x7ff0600ff6a0 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.621+0000 7ff0648e9640 1 -- 172.21.15.3:0/620933067 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff05000ce20 con 0x7ff0600ff6a0 2024-02-09T22:54:53.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.621+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0601a6220 con 0x7ff0600ff6a0 2024-02-09T22:54:53.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.621+0000 7ff0648e9640 1 -- 172.21.15.3:0/620933067 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff050018920 con 0x7ff0600ff6a0 2024-02-09T22:54:53.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.621+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0601a6730 con 0x7ff0600ff6a0 2024-02-09T22:54:53.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff0648e9640 1 -- 172.21.15.3:0/620933067 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff050018a80 con 0x7ff0600ff6a0 2024-02-09T22:54:53.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 Processor -- start 2024-02-09T22:54:53.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 -- start start 2024-02-09T22:54:53.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff0648e9640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff04c0a2260 0x7ff04c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 0x565535df9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535dfa3e0 0x565535e02990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535db5020 con 0x565535dfa3e0 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535d68320 con 0x565535e02f60 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.623+0000 7ff068599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535d65850 con 0x565535df9a30 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff04c0a2260 0x7ff04c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 0x565535df9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535dfa3e0 0x565535e02990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 0x565535df9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:55372/0 (socket says 172.21.15.3:55372) 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 -- 172.21.15.3:0/244516740 learned_addr learned my addr 172.21.15.3:0/244516740 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff0648e9640 1 -- 172.21.15.3:0/620933067 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff050014070 con 0x7ff0600ff6a0 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 -- 172.21.15.3:0/244516740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 msgr2=0x565535e053b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e053b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 -- 172.21.15.3:0/244516740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535dfa3e0 msgr2=0x565535e02990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535dfa3e0 0x565535e02990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 -- 172.21.15.3:0/244516740 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565535d23210 con 0x565535df9a30 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535dfa3e0 0x565535e02990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05effd640 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 0x565535df9e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff0500cba80 tx=0x7ff05000cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffe851e6db14ba68 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff04c0a2260 0x7ff04c0a4720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff058005cc0 tx=0x7ff058009820 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff055ffb640 1 -- 172.21.15.3:0/244516740 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05009ad70 con 0x565535df9a30 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff055ffb640 1 -- 172.21.15.3:0/244516740 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff05009ee10 con 0x565535df9a30 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.624+0000 7ff055ffb640 1 -- 172.21.15.3:0/244516740 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05001ce50 con 0x565535df9a30 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 -- 172.21.15.3:0/244516740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 msgr2=0x565535df9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 0x565535df9e10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff0500cba80 tx=0x7ff05000cf90 comp rx=0 tx=0).stop 2024-02-09T22:54:53.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 -- 172.21.15.3:0/244516740 shutdown_connections 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535dfa3e0 0x565535e02990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 --2- 172.21.15.3:0/244516740 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535df9a30 0x565535df9e10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 -- 172.21.15.3:0/244516740 >> 172.21.15.3:0/244516740 conn(0x565535df4100 msgr2=0x565535df44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 -- 172.21.15.3:0/244516740 shutdown_connections 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 -- 172.21.15.3:0/244516740 wait complete. 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.625+0000 7ff068599740 1 -- start start 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 Processor -- start 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535dfffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535dfa3e0 0x565535e00500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535e02f60 0x565535e00a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535d61030 con 0x565535df9a30 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535a34430 con 0x565535dfa3e0 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5655359f0e00 con 0x565535e02f60 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535dfffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff068599740 10 client.0 _ll_register_callbacks cb 0x565535d56030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535dfffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35872/0 (socket says 172.21.15.3:35872) 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 -- 172.21.15.3:0/3336601075 learned_addr learned my addr 172.21.15.3:0/3336601075 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535e02f60 0x565535e00a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535dfa3e0 0x565535e00500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 -- 172.21.15.3:0/3336601075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535e02f60 msgr2=0x565535e00a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535e02f60 0x565535e00a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 -- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535dfa3e0 msgr2=0x565535e00500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535dfa3e0 0x565535e00500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05effd640 1 -- 172.21.15.3:0/3336601075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0601a6730 con 0x565535df9a30 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.626+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535dfa3e0 0x565535e00500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff05effd640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535dfffc0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff0500a2f30 tx=0x7ff0500cb9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d6ebf960e1ee2f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff03effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff03effd640 1 -- 172.21.15.3:0/3336601075 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0500cbea0 con 0x565535df9a30 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff03effd640 1 -- 172.21.15.3:0/3336601075 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff05009ad70 con 0x565535df9a30 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x565535a607e0 con 0x565535df9a30 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff03effd640 1 -- 172.21.15.3:0/3336601075 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0500a3460 con 0x565535df9a30 2024-02-09T22:54:53.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 10 client.16890 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x565535d5f6b0 con 0x565535df9a30 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff03effd640 1 -- 172.21.15.3:0/3336601075 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0500b7020 con 0x565535df9a30 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff03effd640 1 -- 172.21.15.3:0/3336601075 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff050085070 con 0x565535df9a30 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 10 client.16890 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 10 client.16890 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 20 client.16890 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e13290 0x565535e15680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 4 client.16890 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x565535a9d720 con 0x565535e13290 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.627+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e13290 0x565535e15680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e13290 0x565535e15680 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a0bbc3508b3e2b0 server_cookie=671910eea0eaa270 in_seq=0 out_seq=0 2024-02-09T22:54:53.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff03effd640 10 client.16890 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:53.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 98307 2.19 MB/sec execute 47 sec latency 4953.597 ms 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff03effd640 1 -- 172.21.15.3:0/3336601075 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x565535a9d720 con 0x565535e13290 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff03effd640 10 client.16890 handle_command_reply: tid=0 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff068599740 1 client.16890 shutdown 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 msgr2=0x565535dfffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.628+0000 7ff068599740 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535dfffc0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7ff0500a2f30 tx=0x7ff0500cb9c0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 shutdown_connections 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e13290 0x565535e15680 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535dfa3e0 0x565535e00500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535e02f60 0x565535e00a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff03effd640 0 client.16890 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 --2- 172.21.15.3:0/3336601075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535dfffc0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 >> 172.21.15.3:0/3336601075 conn(0x565535df4100 msgr2=0x565535df5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 shutdown_connections 2024-02-09T22:54:53.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 1 -- 172.21.15.3:0/3336601075 wait complete. 2024-02-09T22:54:53.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.629+0000 7ff068599740 20 client.16890 trim_cache size 0 max 16384 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 Processor -- start 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 -- start start 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e026f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e00510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e00a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535ce6130 con 0x565535df9a30 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535e00f90 con 0x565535e02f60 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff068599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535df9530 con 0x565535dfa3e0 2024-02-09T22:54:53.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff05effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e026f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff05effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e026f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35874/0 (socket says 172.21.15.3:35874) 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff05effd640 1 -- 172.21.15.3:0/2672340170 learned_addr learned my addr 172.21.15.3:0/2672340170 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.641+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e00510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e00a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05effd640 1 -- 172.21.15.3:0/2672340170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 msgr2=0x565535e00510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05effd640 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e00510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05effd640 1 -- 172.21.15.3:0/2672340170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 msgr2=0x565535e00a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05effd640 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e00a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05effd640 1 -- 172.21.15.3:0/2672340170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x565535cfbc90 con 0x565535df9a30 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff05effd640 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e026f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff050000c00 tx=0x7ff05000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e200d0a4bf043702 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff0557fa640 1 -- 172.21.15.3:0/2672340170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05009ad70 con 0x565535df9a30 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff0557fa640 1 -- 172.21.15.3:0/2672340170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0500b7020 con 0x565535df9a30 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.642+0000 7ff0557fa640 1 -- 172.21.15.3:0/2672340170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05009a1c0 con 0x565535df9a30 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 -- 172.21.15.3:0/2672340170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 msgr2=0x565535e026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e026f0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7ff050000c00 tx=0x7ff05000cf90 comp rx=0 tx=0).stop 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 -- 172.21.15.3:0/2672340170 shutdown_connections 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e00a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e00510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 --2- 172.21.15.3:0/2672340170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e026f0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 -- 172.21.15.3:0/2672340170 >> 172.21.15.3:0/2672340170 conn(0x565535df4100 msgr2=0x565535dfb2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 -- 172.21.15.3:0/2672340170 shutdown_connections 2024-02-09T22:54:53.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 -- 172.21.15.3:0/2672340170 wait complete. 2024-02-09T22:54:53.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 -- start start 2024-02-09T22:54:53.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.643+0000 7ff068599740 1 Processor -- start 2024-02-09T22:54:53.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e09c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e0a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e081b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535e08770 con 0x565535df9a30 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535e088e0 con 0x565535e02f60 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x565535e08aa0 con 0x565535dfa3e0 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 10 client.0 _ll_register_callbacks cb 0x7ff050010e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e09c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e09c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35880/0 (socket says 172.21.15.3:35880) 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e0a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 -- 172.21.15.3:0/3985560699 learned_addr learned my addr 172.21.15.3:0/3985560699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05f7fe640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e081b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 -- 172.21.15.3:0/3985560699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 msgr2=0x565535e0a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e0a150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 -- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 msgr2=0x565535e081b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e081b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 -- 172.21.15.3:0/3985560699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565535d5f6b0 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e0a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff05effd640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e09c10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0500971d0 tx=0x7ff050097010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efcc10c2bd9427d4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff03effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff03effd640 1 -- 172.21.15.3:0/3985560699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05007f070 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff03effd640 1 -- 172.21.15.3:0/3985560699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff0500b8080 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.644+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff05009ee10 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff03effd640 1 -- 172.21.15.3:0/3985560699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff05009ad70 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 10 client.16896 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x565535d21a70 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff03effd640 1 -- 172.21.15.3:0/3985560699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff05009ee10 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff03effd640 1 -- 172.21.15.3:0/3985560699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff050082020 con 0x565535df9a30 2024-02-09T22:54:53.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 10 client.16896 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 10 client.16896 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 20 client.16896 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e6a500 0x565535e6c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 4 client.16896 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x565535aec890 con 0x565535e6a500 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e6a500 0x565535e6c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff05e7fc640 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e6a500 0x565535e6c8f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ea19f2d272c5aa9d server_cookie=5cf060c9df06cb9 in_seq=0 out_seq=0 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.645+0000 7ff03effd640 10 client.16896 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff03effd640 1 -- 172.21.15.3:0/3985560699 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x565535aec890 con 0x565535e6a500 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff03effd640 10 client.16896 handle_command_reply: tid=0 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 client.16896 shutdown 2024-02-09T22:54:53.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 msgr2=0x565535e09c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e09c10 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0500971d0 tx=0x7ff050097010 comp rx=0 tx=0).stop 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 shutdown_connections 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x565535e6a500 0x565535e6c8f0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x565535e02f60 0x565535e081b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x565535dfa3e0 0x565535e0a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 --2- 172.21.15.3:0/3985560699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x565535df9a30 0x565535e09c10 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 >> 172.21.15.3:0/3985560699 conn(0x565535df4100 msgr2=0x565535df5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff03effd640 0 client.16896 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 shutdown_connections 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 1 -- 172.21.15.3:0/3985560699 wait complete. 2024-02-09T22:54:53.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff068599740 20 client.16896 trim_cache size 0 max 16384 2024-02-09T22:54:53.658 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:54:53.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff04c0a2260 msgr2=0x7ff04c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff0658eb640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff04c0a2260 0x7ff04c0a4720 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff058005cc0 tx=0x7ff058009820 comp rx=0 tx=0).stop 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0600ff6a0 msgr2=0x7ff060100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.646+0000 7ff0658eb640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0600ff6a0 0x7ff060100880 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff05000e9f0 tx=0x7ff05000eeb0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 shutdown_connections 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff04c0a2260 0x7ff04c0a4720 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff060106200 0x7ff060101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff060100060 0x7ff060100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 --2- 172.21.15.3:0/620933067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0600ff6a0 0x7ff060100880 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 >> 172.21.15.3:0/620933067 conn(0x7ff060076630 msgr2=0x7ff060104160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 shutdown_connections 2024-02-09T22:54:53.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:53.647+0000 7ff0658eb640 1 -- 172.21.15.3:0/620933067 wait complete. 2024-02-09T22:54:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:53 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1793127498' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:54:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:54 smithi089 ceph-mon[36665]: pgmap v112: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 601 KiB/s wr, 75 op/s 2024-02-09T22:54:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:54 smithi003 ceph-mon[32678]: pgmap v112: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 601 KiB/s wr, 75 op/s 2024-02-09T22:54:54.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100028 2.39 MB/sec execute 48 sec latency 56.109 ms 2024-02-09T22:54:54.662 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:54:54.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:54:54.663 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:54:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:54 smithi064 ceph-mon[36638]: pgmap v112: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 601 KiB/s wr, 75 op/s 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 -- 172.21.15.3:0/2077010964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40100ce0 msgr2=0x7fbf40104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 --2- 172.21.15.3:0/2077010964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40100ce0 0x7fbf40104d10 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009930 tx=0x7fbf2802bc30 comp rx=0 tx=0).stop 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 -- 172.21.15.3:0/2077010964 shutdown_connections 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 --2- 172.21.15.3:0/2077010964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf401053f0 0x7fbf4010cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 --2- 172.21.15.3:0/2077010964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40100ce0 0x7fbf40104d10 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 --2- 172.21.15.3:0/2077010964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf40100330 0x7fbf40100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 -- 172.21.15.3:0/2077010964 >> 172.21.15.3:0/2077010964 conn(0x7fbf400771d0 msgr2=0x7fbf400775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 -- 172.21.15.3:0/2077010964 shutdown_connections 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 -- 172.21.15.3:0/2077010964 wait complete. 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 Processor -- start 2024-02-09T22:54:54.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.767+0000 7fbf454b5640 1 -- start start 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf40100330 0x7fbf40101db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf401053f0 0x7fbf401022f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 0x7fbf401a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf401a6000 con 0x7fbf40102830 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf401a6170 con 0x7fbf40100330 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf401a6330 con 0x7fbf401053f0 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 0x7fbf401a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 0x7fbf401a5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35884/0 (socket says 172.21.15.3:35884) 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/739116669 learned_addr learned my addr 172.21.15.3:0/739116669 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf401053f0 0x7fbf401022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3effd640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf40100330 0x7fbf40101db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/739116669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf401053f0 msgr2=0x7fbf401022f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf401053f0 0x7fbf401022f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/739116669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf40100330 msgr2=0x7fbf40101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf40100330 0x7fbf40101db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/739116669 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28009560 con 0x7fbf40102830 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3effd640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf40100330 0x7fbf40101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 0x7fbf401a5a30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000cad0 tx=0x7fbf3000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf37fff640 1 -- 172.21.15.3:0/739116669 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000ae20 con 0x7fbf40102830 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf401a6610 con 0x7fbf40102830 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf37fff640 1 -- 172.21.15.3:0/739116669 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf30005020 con 0x7fbf40102830 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.768+0000 7fbf37fff640 1 -- 172.21.15.3:0/739116669 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3000f6b0 con 0x7fbf40102830 2024-02-09T22:54:54.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.769+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf401a6aa0 con 0x7fbf40102830 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.771+0000 7fbf37fff640 1 -- 172.21.15.3:0/739116669 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fbf30005520 con 0x7fbf40102830 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.771+0000 7fbf48166740 1 Processor -- start 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.771+0000 7fbf48166740 1 -- start start 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf37fff640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbf2c0a2260 0x7fbf2c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbf2c0a2260 0x7fbf2c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf48166740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff6a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf48166740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff73150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf48166740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff75a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf48166740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9fed8000 con 0x564d9ff6a190 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf48166740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9fed5530 con 0x564d9ff73690 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf48166740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9fed0d10 con 0x564d9ff6aab0 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf37fff640 1 -- 172.21.15.3:0/739116669 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbf300ce020 con 0x7fbf40102830 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff6a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff6a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35892/0 (socket says 172.21.15.3:35892) 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 -- 172.21.15.3:0/3265319856 learned_addr learned my addr 172.21.15.3:0/3265319856 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff75a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff73150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 -- 172.21.15.3:0/3265319856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 msgr2=0x564d9ff73150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff73150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 -- 172.21.15.3:0/3265319856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 msgr2=0x564d9ff75a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff75a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3effd640 1 -- 172.21.15.3:0/3265319856 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d9fe77210 con 0x564d9ff6a190 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbf2c0a2260 0x7fbf2c0a4720 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf28000c00 tx=0x7fbf280047e0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.772+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff75a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.773+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff6a570 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009d40 tx=0x7fbf380117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de4f0b03b57fafae server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.773+0000 7fbf357fa640 1 -- 172.21.15.3:0/3265319856 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf38012650 con 0x564d9ff6a190 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.773+0000 7fbf357fa640 1 -- 172.21.15.3:0/3265319856 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf38009e30 con 0x564d9ff6a190 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.773+0000 7fbf357fa640 1 -- 172.21.15.3:0/3265319856 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3800fce0 con 0x564d9ff6a190 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 -- 172.21.15.3:0/3265319856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 msgr2=0x564d9ff6a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff6a570 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fbf38009d40 tx=0x7fbf380117b0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 -- 172.21.15.3:0/3265319856 shutdown_connections 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff75a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff73150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 --2- 172.21.15.3:0/3265319856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff6a570 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 -- 172.21.15.3:0/3265319856 >> 172.21.15.3:0/3265319856 conn(0x564d9ff648c0 msgr2=0x564d9ff64ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 -- 172.21.15.3:0/3265319856 shutdown_connections 2024-02-09T22:54:54.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 -- 172.21.15.3:0/3265319856 wait complete. 2024-02-09T22:54:54.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 -- start start 2024-02-09T22:54:54.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.774+0000 7fbf48166740 1 Processor -- start 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ff706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ff70be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff71120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9fe75e60 con 0x564d9ff73690 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9fba44b0 con 0x564d9ff6aab0 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9fe56670 con 0x564d9ff6a190 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 10 client.0 _ll_register_callbacks cb 0x564d9fe6e190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff71120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff71120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35898/0 (socket says 172.21.15.3:35898) 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/3057901647 learned_addr learned my addr 172.21.15.3:0/3057901647 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ff706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ff70be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/3057901647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 msgr2=0x564d9ff706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ff706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 msgr2=0x564d9ff70be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ff70be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/3057901647 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf38009970 con 0x564d9ff73690 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ff70be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff71120 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbf30005cc0 tx=0x7fbf3009b2a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8423ae5277697a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf267fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf267fc640 1 -- 172.21.15.3:0/3057901647 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf30005390 con 0x564d9ff73690 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf267fc640 1 -- 172.21.15.3:0/3057901647 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbf300a2d50 con 0x564d9ff73690 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d9fbd0860 con 0x564d9ff73690 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf267fc640 1 -- 172.21.15.3:0/3057901647 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf300a2f10 con 0x564d9ff73690 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 10 client.16908 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:54.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.775+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x564d9fecf390 con 0x564d9ff73690 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf267fc640 1 -- 172.21.15.3:0/3057901647 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf300b7080 con 0x564d9ff73690 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf267fc640 1 -- 172.21.15.3:0/3057901647 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf30016030 con 0x564d9ff73690 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf48166740 10 client.16908 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf48166740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf48166740 20 client.16908 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf48166740 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ff836c0 0x564d9ff85ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf48166740 4 client.16908 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d9fc21410 con 0x564d9ff836c0 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ff836c0 0x564d9ff85ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf267fc640 10 client.16908 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.776+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ff836c0 0x564d9ff85ab0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac097b1b94bf661b server_cookie=90099a608cc860af in_seq=0 out_seq=0 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf267fc640 1 -- 172.21.15.3:0/3057901647 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d9fc21410 con 0x564d9ff836c0 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf267fc640 10 client.16908 handle_command_reply: tid=0 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 client.16908 shutdown 2024-02-09T22:54:54.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 msgr2=0x564d9ff71120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff71120 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7fbf30005cc0 tx=0x7fbf3009b2a0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 shutdown_connections 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ff836c0 0x564d9ff85ab0 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ff70be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff71120 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf267fc640 0 client.16908 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 --2- 172.21.15.3:0/3057901647 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ff706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 >> 172.21.15.3:0/3057901647 conn(0x564d9ff648c0 msgr2=0x564d9ff66610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 shutdown_connections 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 1 -- 172.21.15.3:0/3057901647 wait complete. 2024-02-09T22:54:54.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.777+0000 7fbf48166740 20 client.16908 trim_cache size 0 max 16384 2024-02-09T22:54:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 Processor -- start 2024-02-09T22:54:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 -- start start 2024-02-09T22:54:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff72dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff6d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff6de60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9ff64720 con 0x564d9ff6a190 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9ff600a0 con 0x564d9ff73690 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.792+0000 7fbf48166740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9ff69df0 con 0x564d9ff6aab0 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff72dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff72dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35912/0 (socket says 172.21.15.3:35912) 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 -- 172.21.15.3:0/3577932255 learned_addr learned my addr 172.21.15.3:0/3577932255 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff6d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff6de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 -- 172.21.15.3:0/3577932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 msgr2=0x564d9ff6d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff6d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 -- 172.21.15.3:0/3577932255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 msgr2=0x564d9ff6de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff6de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 -- 172.21.15.3:0/3577932255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d9fe6bc70 con 0x564d9ff6a190 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff6de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:54.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf3effd640 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff72dd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800f8f0 tx=0x7fbf3800f920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4a94d025b5c5752 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf34ff9640 1 -- 172.21.15.3:0/3577932255 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3802b070 con 0x564d9ff6a190 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf34ff9640 1 -- 172.21.15.3:0/3577932255 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf38009530 con 0x564d9ff6a190 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.793+0000 7fbf34ff9640 1 -- 172.21.15.3:0/3577932255 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf380056e0 con 0x564d9ff6a190 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 -- 172.21.15.3:0/3577932255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 msgr2=0x564d9ff72dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff72dd0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fbf3800f8f0 tx=0x7fbf3800f920 comp rx=0 tx=0).stop 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 -- 172.21.15.3:0/3577932255 shutdown_connections 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff73690 0x564d9ff6de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6aab0 0x564d9ff6d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 --2- 172.21.15.3:0/3577932255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff6a190 0x564d9ff72dd0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 -- 172.21.15.3:0/3577932255 >> 172.21.15.3:0/3577932255 conn(0x564d9ff648c0 msgr2=0x564d9ff6b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 -- 172.21.15.3:0/3577932255 shutdown_connections 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 -- 172.21.15.3:0/3577932255 wait complete. 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 -- start start 2024-02-09T22:54:54.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.794+0000 7fbf48166740 1 Processor -- start 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ffd40e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ffd4620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff79990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9ff79f50 con 0x564d9ff73690 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9ff7a0c0 con 0x564d9ff6aab0 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d9ff7a280 con 0x564d9ff6a190 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 10 client.0 _ll_register_callbacks cb 0x7fbf380096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff79990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff79990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35924/0 (socket says 172.21.15.3:35924) 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/663262136 learned_addr learned my addr 172.21.15.3:0/663262136 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3effd640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ffd40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ffd4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/663262136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 msgr2=0x564d9ffd40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ffd40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 msgr2=0x564d9ffd4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ffd4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 -- 172.21.15.3:0/663262136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d9fe77210 con 0x564d9ff73690 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3e7fc640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ffd4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf3f7fe640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff79990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbf300cbda0 tx=0x7fbf30005920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4adfd6f4afb49393 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf267fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf267fc640 1 -- 172.21.15.3:0/663262136 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf3009b140 con 0x564d9ff73690 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf267fc640 1 -- 172.21.15.3:0/663262136 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbf30099e90 con 0x564d9ff73690 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf38009e30 con 0x564d9ff73690 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf267fc640 1 -- 172.21.15.3:0/663262136 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf3009ec10 con 0x564d9ff73690 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 10 client.16914 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.795+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x564d9fe91ac0 con 0x564d9ff73690 2024-02-09T22:54:54.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf267fc640 1 -- 172.21.15.3:0/663262136 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf38009e30 con 0x564d9ff73690 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf267fc640 1 -- 172.21.15.3:0/663262136 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf30016040 con 0x564d9ff73690 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf48166740 10 client.16914 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf48166740 10 client.16914 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf48166740 20 client.16914 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf48166740 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ffdbba0 0x564d9ffddfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf48166740 4 client.16914 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564d9fc0d7b0 con 0x564d9ffdbba0 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf3effd640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ffdbba0 0x564d9ffddfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf267fc640 10 client.16914 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.796+0000 7fbf3effd640 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ffdbba0 0x564d9ffddfc0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71ef3fac5cbf6a7a server_cookie=d630f2c4f6646824 in_seq=0 out_seq=0 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf267fc640 1 -- 172.21.15.3:0/663262136 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564d9fc0d7b0 con 0x564d9ffdbba0 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf267fc640 10 client.16914 handle_command_reply: tid=0 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 client.16914 shutdown 2024-02-09T22:54:54.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 msgr2=0x564d9ff79990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff79990 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fbf300cbda0 tx=0x7fbf30005920 comp rx=0 tx=0).stop 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 shutdown_connections 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d9ffdbba0 0x564d9ffddfc0 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d9ff6aab0 0x564d9ffd4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d9ff73690 0x564d9ff79990 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf267fc640 0 client.16914 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 --2- 172.21.15.3:0/663262136 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d9ff6a190 0x564d9ffd40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 >> 172.21.15.3:0/663262136 conn(0x564d9ff648c0 msgr2=0x564d9ff70640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 shutdown_connections 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 1 -- 172.21.15.3:0/663262136 wait complete. 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.797+0000 7fbf48166740 20 client.16914 trim_cache size 0 max 16384 2024-02-09T22:54:54.807 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:54:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:54:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "945a2c88-61d4-4d09-8448-46842d1c1cfd", 2024-02-09T22:54:54.808 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:54:54.808 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbf2c0a2260 msgr2=0x7fbf2c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbf2c0a2260 0x7fbf2c0a4720 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf28000c00 tx=0x7fbf280047e0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 msgr2=0x7fbf401a5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 0x7fbf401a5a30 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fbf3000cad0 tx=0x7fbf3000cf90 comp rx=0 tx=0).stop 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 shutdown_connections 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbf2c0a2260 0x7fbf2c0a4720 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbf40102830 0x7fbf401a5a30 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbf401053f0 0x7fbf401022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 --2- 172.21.15.3:0/739116669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbf40100330 0x7fbf40101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 >> 172.21.15.3:0/739116669 conn(0x7fbf400771d0 msgr2=0x7fbf40069cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 shutdown_connections 2024-02-09T22:54:54.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.798+0000 7fbf454b5640 1 -- 172.21.15.3:0/739116669 wait complete. 2024-02-09T22:54:54.814 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:54:54.814 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:54:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 -- 172.21.15.3:0/600949670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0ff6a0 msgr2=0x7efc3c0ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 --2- 172.21.15.3:0/600949670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0ff6a0 0x7efc3c0ffb20 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7efc30009900 tx=0x7efc3002bc80 comp rx=0 tx=0).stop 2024-02-09T22:54:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 -- 172.21.15.3:0/600949670 shutdown_connections 2024-02-09T22:54:54.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 --2- 172.21.15.3:0/600949670 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c100060 0x7efc3c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 --2- 172.21.15.3:0/600949670 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c0ff6a0 0x7efc3c0ffb20 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 --2- 172.21.15.3:0/600949670 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc3c1031b0 0x7efc3c103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.919+0000 7efc4417a640 1 -- 172.21.15.3:0/600949670 >> 172.21.15.3:0/600949670 conn(0x7efc3c076620 msgr2=0x7efc3c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 -- 172.21.15.3:0/600949670 shutdown_connections 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 -- 172.21.15.3:0/600949670 wait complete. 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 Processor -- start 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 -- start start 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c0ff6a0 0x7efc3c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc3c100060 0x7efc3c19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 0x7efc3c1977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c197e90 con 0x7efc3c1031b0 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c198000 con 0x7efc3c100060 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc4417a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc3c1981c0 con 0x7efc3c0ff6a0 2024-02-09T22:54:54.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc42717640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 0x7efc3c1977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc42717640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 0x7efc3c1977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35950/0 (socket says 172.21.15.3:35950) 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc42717640 1 -- 172.21.15.3:0/2674309248 learned_addr learned my addr 172.21.15.3:0/2674309248 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.920+0000 7efc41f16640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c0ff6a0 0x7efc3c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc41715640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc3c100060 0x7efc3c19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc42717640 1 -- 172.21.15.3:0/2674309248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c0ff6a0 msgr2=0x7efc3c19d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc42717640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c0ff6a0 0x7efc3c19d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc42717640 1 -- 172.21.15.3:0/2674309248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc3c100060 msgr2=0x7efc3c19dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc42717640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc3c100060 0x7efc3c19dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc42717640 1 -- 172.21.15.3:0/2674309248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc30009560 con 0x7efc3c1031b0 2024-02-09T22:54:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc42717640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 0x7efc3c1977f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00eaa0 tx=0x7efc2c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc3affd640 1 -- 172.21.15.3:0/2674309248 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c00cdb0 con 0x7efc3c1031b0 2024-02-09T22:54:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc3affd640 1 -- 172.21.15.3:0/2674309248 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc2c005020 con 0x7efc3c1031b0 2024-02-09T22:54:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc3c1984a0 con 0x7efc3c1031b0 2024-02-09T22:54:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc3affd640 1 -- 172.21.15.3:0/2674309248 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0116b0 con 0x7efc3c1031b0 2024-02-09T22:54:54.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.921+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c1a46b0 con 0x7efc3c1031b0 2024-02-09T22:54:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.923+0000 7efc3affd640 1 -- 172.21.15.3:0/2674309248 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efc2c005520 con 0x7efc3c1031b0 2024-02-09T22:54:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 Processor -- start 2024-02-09T22:54:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 -- start start 2024-02-09T22:54:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc3affd640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc240a2230 0x7efc240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3a980 0x55b993d3ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3b330 0x55b993d438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993d46300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993cf5d00 con 0x55b993d43eb0 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993caa2f0 con 0x55b993d3a980 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc46e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993ca7820 con 0x55b993d3b330 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc41715640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc240a2230 0x7efc240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc42717640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993d46300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc3affd640 1 -- 172.21.15.3:0/2674309248 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efc2c0ce290 con 0x7efc3c1031b0 2024-02-09T22:54:54.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc42717640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993d46300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35966/0 (socket says 172.21.15.3:35966) 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc42717640 1 -- 172.21.15.3:0/2223331177 learned_addr learned my addr 172.21.15.3:0/2223331177 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc41f16640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3a980 0x55b993d3ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.924+0000 7efc41715640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3b330 0x55b993d438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc42717640 1 -- 172.21.15.3:0/2223331177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3b330 msgr2=0x55b993d438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc42717640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3b330 0x55b993d438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc42717640 1 -- 172.21.15.3:0/2223331177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3a980 msgr2=0x55b993d3ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc42717640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3a980 0x55b993d3ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc42717640 1 -- 172.21.15.3:0/2223331177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b993c64ff0 con 0x55b993d43eb0 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc41f16640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3a980 0x55b993d3ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc41715640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3b330 0x55b993d438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc42717640 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993d46300 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7efc2c097160 tx=0x7efc2c0970d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5e5ed53bcfc094 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc41715640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc240a2230 0x7efc240a46f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efc30006010 tx=0x7efc30005b50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc23fff640 1 -- 172.21.15.3:0/2223331177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c09ad00 con 0x55b993d43eb0 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc23fff640 1 -- 172.21.15.3:0/2223331177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc2c09eda0 con 0x55b993d43eb0 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.925+0000 7efc23fff640 1 -- 172.21.15.3:0/2223331177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0cbe30 con 0x55b993d43eb0 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- 172.21.15.3:0/2223331177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 msgr2=0x55b993d46300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993d46300 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7efc2c097160 tx=0x7efc2c0970d0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- 172.21.15.3:0/2223331177 shutdown_connections 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993d46300 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3b330 0x55b993d438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2223331177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3a980 0x55b993d3ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- 172.21.15.3:0/2223331177 >> 172.21.15.3:0/2223331177 conn(0x55b993d35050 msgr2=0x55b993d35430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- 172.21.15.3:0/2223331177 shutdown_connections 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- 172.21.15.3:0/2223331177 wait complete. 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- start start 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 Processor -- start 2024-02-09T22:54:54.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d40de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 0x55b993d41320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d43eb0 0x55b993d41860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993ca3000 con 0x55b993d3b330 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993976430 con 0x55b993d43eb0 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.926+0000 7efc46e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993932e00 con 0x55b993d3a980 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc46e2b740 10 client.0 _ll_register_callbacks cb 0x55b993c977f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41715640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 0x55b993d41320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d40de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d40de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:55502/0 (socket says 172.21.15.3:55502) 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41715640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 0x55b993d41320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35972/0 (socket says 172.21.15.3:35972) 2024-02-09T22:54:54.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 -- 172.21.15.3:0/629783874 learned_addr learned my addr 172.21.15.3:0/629783874 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc42717640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d43eb0 0x55b993d41860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 -- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d43eb0 msgr2=0x55b993d41860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d43eb0 0x55b993d41860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 -- 172.21.15.3:0/629783874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 msgr2=0x55b993d41320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 0x55b993d41320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 -- 172.21.15.3:0/629783874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc3c1a46b0 con 0x55b993d3a980 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc42717640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d43eb0 0x55b993d41860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41715640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 0x55b993d41320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc41f16640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d40de0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efc3400bf80 tx=0x7efc34011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab731740742d7733 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc217fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc217fa640 1 -- 172.21.15.3:0/629783874 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34016070 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc217fa640 1 -- 172.21.15.3:0/629783874 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efc34004020 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b9939a27e0 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc217fa640 1 -- 172.21.15.3:0/629783874 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc340041e0 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc46e2b740 10 client.26293 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.927+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55b993ca1680 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc217fa640 1 -- 172.21.15.3:0/629783874 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc34027020 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc217fa640 1 -- 172.21.15.3:0/629783874 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7efc34003850 con 0x55b993d3a980 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc46e2b740 10 client.26293 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc46e2b740 10 client.26293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc46e2b740 20 client.26293 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:54.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc46e2b740 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993d54420 0x55b993d56840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc46e2b740 4 client.26293 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9939df720 con 0x55b993d54420 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc41715640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993d54420 0x55b993d56840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc217fa640 10 client.26293 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.928+0000 7efc41715640 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993d54420 0x55b993d56840 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fc3891f7db28ce7 server_cookie=e2db9683f1828758 in_seq=0 out_seq=0 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc217fa640 1 -- 172.21.15.3:0/629783874 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b9939df720 con 0x55b993d54420 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc217fa640 10 client.26293 handle_command_reply: tid=0 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 client.26293 shutdown 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 msgr2=0x55b993d40de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d40de0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efc3400bf80 tx=0x7efc34011810 comp rx=0 tx=0).stop 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 shutdown_connections 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993d54420 0x55b993d56840 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d43eb0 0x55b993d41860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3b330 0x55b993d41320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc217fa640 0 client.26293 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 --2- 172.21.15.3:0/629783874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d40de0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 >> 172.21.15.3:0/629783874 conn(0x55b993d35050 msgr2=0x55b993d36da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 shutdown_connections 2024-02-09T22:54:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 1 -- 172.21.15.3:0/629783874 wait complete. 2024-02-09T22:54:54.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.929+0000 7efc46e2b740 20 client.26293 trim_cache size 0 max 16384 2024-02-09T22:54:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.944+0000 7efc46e2b740 1 Processor -- start 2024-02-09T22:54:54.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.944+0000 7efc46e2b740 1 -- start start 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 0x55b993d34070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d43eb0 0x55b993d4b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc46e2b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993c27fd0 con 0x55b993d3a980 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc46e2b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993d39d00 con 0x55b993d3b330 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc46e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993d5ba90 con 0x55b993d43eb0 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 0x55b993d34070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 0x55b993d34070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35984/0 (socket says 172.21.15.3:35984) 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 -- 172.21.15.3:0/2634013928 learned_addr learned my addr 172.21.15.3:0/2634013928 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 -- 172.21.15.3:0/2634013928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d43eb0 msgr2=0x55b993d4b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d43eb0 0x55b993d4b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 -- 172.21.15.3:0/2634013928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 msgr2=0x55b993d4aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 -- 172.21.15.3:0/2634013928 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b993c3db30 con 0x55b993d3a980 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc41f16640 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 0x55b993d34070 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7efc34026040 tx=0x7efc3400bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62dfde0923567744 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc237fe640 1 -- 172.21.15.3:0/2634013928 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc34016040 con 0x55b993d3a980 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc237fe640 1 -- 172.21.15.3:0/2634013928 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc34027070 con 0x55b993d3a980 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.945+0000 7efc237fe640 1 -- 172.21.15.3:0/2634013928 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc340042e0 con 0x55b993d3a980 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 -- 172.21.15.3:0/2634013928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 msgr2=0x55b993d34070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 0x55b993d34070 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7efc34026040 tx=0x7efc3400bb70 comp rx=0 tx=0).stop 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 -- 172.21.15.3:0/2634013928 shutdown_connections 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d43eb0 0x55b993d4b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 --2- 172.21.15.3:0/2634013928 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d3a980 0x55b993d34070 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 -- 172.21.15.3:0/2634013928 >> 172.21.15.3:0/2634013928 conn(0x55b993d35050 msgr2=0x55b993d33960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 -- 172.21.15.3:0/2634013928 shutdown_connections 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 -- 172.21.15.3:0/2634013928 wait complete. 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 -- start start 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.946+0000 7efc46e2b740 1 Processor -- start 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d4a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993da49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993da4ef0 con 0x55b993d43eb0 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993da5060 con 0x55b993d3b330 2024-02-09T22:54:54.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b993da5220 con 0x55b993d3a980 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 10 client.0 _ll_register_callbacks cb 0x7efc2c097020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993da49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993da49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35992/0 (socket says 172.21.15.3:35992) 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc41f16640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d4a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 -- 172.21.15.3:0/1380745872 learned_addr learned my addr 172.21.15.3:0/1380745872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc41715640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 -- 172.21.15.3:0/1380745872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 msgr2=0x55b993d4a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d4a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 -- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 msgr2=0x55b993d4a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 -- 172.21.15.3:0/1380745872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b993ca1680 con 0x55b993d43eb0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc41f16640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d4a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc41715640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc42717640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993da49b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0976d0 tx=0x7efc2c005890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece812c53cb40116 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc217fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc217fa640 1 -- 172.21.15.3:0/1380745872 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc2c0053a0 con 0x55b993d43eb0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc217fa640 1 -- 172.21.15.3:0/1380745872 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efc2c09af60 con 0x55b993d43eb0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.947+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efc2c09eda0 con 0x55b993d43eb0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc217fa640 1 -- 172.21.15.3:0/1380745872 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7efc2c0cbe30 con 0x55b993d43eb0 2024-02-09T22:54:54.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 10 client.16932 fetch_fsmap learned FSMap version 34 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55b993c63850 con 0x55b993d43eb0 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc217fa640 1 -- 172.21.15.3:0/1380745872 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc2c09eda0 con 0x55b993d43eb0 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc217fa640 1 -- 172.21.15.3:0/1380745872 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 34) v1 ==== 7890+0+0 (secure 0 0 0) 0x7efc2c018030 con 0x55b993d43eb0 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 10 client.16932 fetch_fsmap finished waiting for FSMap version 34 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 10 client.16932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 20 client.16932 populate_metadata read hostname 'smithi003' 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993dac5d0 0x55b993dae9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 4 client.16932 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b993a2e890 con 0x55b993dac5d0 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc41f16640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993dac5d0 0x55b993dae9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc217fa640 10 client.16932 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.948+0000 7efc41f16640 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993dac5d0 0x55b993dae9f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2de6880bbc41e834 server_cookie=7c9fdf4e557d8045 in_seq=0 out_seq=0 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc217fa640 1 -- 172.21.15.3:0/1380745872 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b993a2e890 con 0x55b993dac5d0 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc217fa640 10 client.16932 handle_command_reply: tid=0 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 client.16932 shutdown 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 msgr2=0x55b993da49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993da49b0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7efc2c0976d0 tx=0x7efc2c005890 comp rx=0 tx=0).stop 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 shutdown_connections 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b993dac5d0 0x55b993dae9f0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b993d3b330 0x55b993d4a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b993d43eb0 0x55b993da49b0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc217fa640 0 client.16932 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 --2- 172.21.15.3:0/1380745872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b993d3a980 0x55b993d4a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 >> 172.21.15.3:0/1380745872 conn(0x55b993d35050 msgr2=0x55b993d341f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 shutdown_connections 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 1 -- 172.21.15.3:0/1380745872 wait complete. 2024-02-09T22:54:54.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc46e2b740 20 client.16932 trim_cache size 0 max 16384 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "945a2c88-61d4-4d09-8448-46842d1c1cfd": { 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "945a2c88-61d4-4d09-8448-46842d1c1cfd", 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:54:54.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:54:54.964 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:54:54.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc240a2230 msgr2=0x7efc240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc240a2230 0x7efc240a46f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7efc30006010 tx=0x7efc30005b50 comp rx=0 tx=0).stop 2024-02-09T22:54:54.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 msgr2=0x7efc3c1977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:54:54.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 0x7efc3c1977f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7efc2c00eaa0 tx=0x7efc2c00ef60 comp rx=0 tx=0).stop 2024-02-09T22:54:54.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 shutdown_connections 2024-02-09T22:54:54.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efc240a2230 0x7efc240a46f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efc3c100060 0x7efc3c19dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efc3c1031b0 0x7efc3c1977f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 --2- 172.21.15.3:0/2674309248 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efc3c0ff6a0 0x7efc3c19d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:54:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.949+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 >> 172.21.15.3:0/2674309248 conn(0x7efc3c076620 msgr2=0x7efc3c1053e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:54:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.950+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 shutdown_connections 2024-02-09T22:54:54.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:54:54.950+0000 7efc4417a640 1 -- 172.21.15.3:0/2674309248 wait complete. 2024-02-09T22:54:54.967 INFO:tasks.cephfs.filesystem:scrub status for tag:945a2c88-61d4-4d09-8448-46842d1c1cfd - {'path': '/', 'tag': '945a2c88-61d4-4d09-8448-46842d1c1cfd', 'options': 'recursive,force'} 2024-02-09T22:54:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:55.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:55.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:55 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100084 2.34 MB/sec execute 49 sec latency 765.259 ms 2024-02-09T22:54:55.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:55.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:55.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:55 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:55.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:55.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:55.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:55.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:55 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: pgmap v113: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 601 KiB/s wr, 75 op/s 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:54:56.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.588 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:56 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: pgmap v113: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 601 KiB/s wr, 75 op/s 2024-02-09T22:54:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:54:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:54:56.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:56 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100084 2.29 MB/sec execute 50 sec latency 1765.295 ms 2024-02-09T22:54:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: pgmap v113: 97 pgs: 97 active+clean; 44 MiB data, 679 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 601 KiB/s wr, 75 op/s 2024-02-09T22:54:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:54:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:54:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:56.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:56 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:54:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:57 smithi003 ceph-mon[32678]: pgmap v114: 97 pgs: 97 active+clean; 46 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 820 KiB/s wr, 126 op/s 2024-02-09T22:54:57.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100084 2.25 MB/sec execute 51 sec latency 2765.383 ms 2024-02-09T22:54:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:57 smithi064 ceph-mon[36638]: pgmap v114: 97 pgs: 97 active+clean; 46 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 820 KiB/s wr, 126 op/s 2024-02-09T22:54:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:57 smithi089 ceph-mon[36665]: pgmap v114: 97 pgs: 97 active+clean; 46 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 820 KiB/s wr, 126 op/s 2024-02-09T22:54:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:54:58 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:58.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100152 2.21 MB/sec execute 52 sec latency 3182.040 ms 2024-02-09T22:54:58.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:54:58 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:58.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:54:58 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:54:59.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100152 2.17 MB/sec execute 53 sec latency 1538.558 ms 2024-02-09T22:55:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.314+0000 7f06a6883640 1 -- 172.21.15.3:0/1030874262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a0107420 msgr2=0x7f06a0107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.314+0000 7f06a6883640 1 --2- 172.21.15.3:0/1030874262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a0107420 0x7f06a0107800 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f0690009930 tx=0x7f069002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 -- 172.21.15.3:0/1030874262 shutdown_connections 2024-02-09T22:55:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 --2- 172.21.15.3:0/1030874262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06a01004a0 0x7f06a010e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.319 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 --2- 172.21.15.3:0/1030874262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06a00ffae0 0x7f06a00fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 --2- 172.21.15.3:0/1030874262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a0107420 0x7f06a0107800 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 -- 172.21.15.3:0/1030874262 >> 172.21.15.3:0/1030874262 conn(0x7f06a00753d0 msgr2=0x7f06a00757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 -- 172.21.15.3:0/1030874262 shutdown_connections 2024-02-09T22:55:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 -- 172.21.15.3:0/1030874262 wait complete. 2024-02-09T22:55:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 Processor -- start 2024-02-09T22:55:00.320 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.315+0000 7f06a6883640 1 -- start start 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a6883640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 0x7f06a019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a6883640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06a01004a0 0x7f06a019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a6883640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06a0107420 0x7f06a01976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a6883640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0197e50 con 0x7f06a00ffae0 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a6883640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0197fc0 con 0x7f06a0107420 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a6883640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0198180 con 0x7f06a01004a0 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 0x7f06a019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 0x7f06a019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32898/0 (socket says 172.21.15.3:32898) 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 -- 172.21.15.3:0/3322151339 learned_addr learned my addr 172.21.15.3:0/3322151339 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f06a4e20640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06a0107420 0x7f06a01976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069f7fe640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06a01004a0 0x7f06a019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 -- 172.21.15.3:0/3322151339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06a01004a0 msgr2=0x7f06a019db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.322 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06a01004a0 0x7f06a019db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 -- 172.21.15.3:0/3322151339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06a0107420 msgr2=0x7f06a01976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06a0107420 0x7f06a01976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 -- 172.21.15.3:0/3322151339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0690009560 con 0x7f06a00ffae0 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.316+0000 7f069ffff640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 0x7f06a019d600 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0690009230 tx=0x7f0690004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.317+0000 7f069d7fa640 1 -- 172.21.15.3:0/3322151339 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0690004440 con 0x7f06a00ffae0 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.317+0000 7f069d7fa640 1 -- 172.21.15.3:0/3322151339 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06900052d0 con 0x7f06a00ffae0 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.317+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a0198400 con 0x7f06a00ffae0 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.317+0000 7f069d7fa640 1 -- 172.21.15.3:0/3322151339 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0690003de0 con 0x7f06a00ffae0 2024-02-09T22:55:00.323 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.317+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a01a4580 con 0x7f06a00ffae0 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.319+0000 7f069d7fa640 1 -- 172.21.15.3:0/3322151339 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0690005440 con 0x7f06a00ffae0 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.319+0000 7f06a944b740 1 Processor -- start 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.319+0000 7f06a944b740 1 -- start start 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a944b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750a8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a944b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a944b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a944b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564275016a00 con 0x5642750a89d0 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a944b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564275013f30 con 0x5642750a80b0 2024-02-09T22:55:00.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a944b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56427500f710 con 0x5642750b17a0 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069d7fa640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06880a2220 0x7f06880a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b1260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32910/0 (socket says 172.21.15.3:32910) 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 -- 172.21.15.3:0/4043684190 learned_addr learned my addr 172.21.15.3:0/4043684190 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a4e20640 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750a8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069f7fe640 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f06a4e20640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06880a2220 0x7f06880a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069d7fa640 1 -- 172.21.15.3:0/3322151339 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f06900f07d0 con 0x7f06a00ffae0 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 -- 172.21.15.3:0/4043684190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 msgr2=0x5642750b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 -- 172.21.15.3:0/4043684190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 msgr2=0x5642750a8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750a8490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.320+0000 7f069ffff640 1 -- 172.21.15.3:0/4043684190 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564274fd43b0 con 0x5642750a89d0 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f069ffff640 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b1260 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f069004e000 tx=0x7f06900038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297d9292c04e03b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f06a4e20640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06880a2220 0x7f06880a46e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0698005fd0 tx=0x7f06980099c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f06867fc640 1 -- 172.21.15.3:0/4043684190 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06900bdc30 con 0x5642750a89d0 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f06867fc640 1 -- 172.21.15.3:0/4043684190 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06900bd2a0 con 0x5642750a89d0 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f06867fc640 1 -- 172.21.15.3:0/4043684190 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06900ee170 con 0x5642750a89d0 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f06a944b740 1 -- 172.21.15.3:0/4043684190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 msgr2=0x5642750b1260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.321+0000 7f06a944b740 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b1260 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f069004e000 tx=0x7f06900038b0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- 172.21.15.3:0/4043684190 shutdown_connections 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b1260 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 --2- 172.21.15.3:0/4043684190 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750a8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- 172.21.15.3:0/4043684190 >> 172.21.15.3:0/4043684190 conn(0x5642750a29d0 msgr2=0x5642750a2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- 172.21.15.3:0/4043684190 shutdown_connections 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- 172.21.15.3:0/4043684190 wait complete. 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- start start 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 Processor -- start 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750ae760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750aeca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750af1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564274fb4b90 con 0x5642750a89d0 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564274ce3250 con 0x5642750a80b0 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.322+0000 7f06a944b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564274f95200 con 0x5642750b17a0 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f06a944b740 10 client.0 _ll_register_callbacks cb 0x564274fcb5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750aeca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750aeca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32926/0 (socket says 172.21.15.3:32926) 2024-02-09T22:55:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 -- 172.21.15.3:0/418908317 learned_addr learned my addr 172.21.15.3:0/418908317 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f06a4e20640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750af1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069ffff640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750ae760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 -- 172.21.15.3:0/418908317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 msgr2=0x5642750af1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750af1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 -- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 msgr2=0x5642750ae760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750ae760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 -- 172.21.15.3:0/418908317 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a01a4580 con 0x5642750a89d0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069ffff640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f069f7fe640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750aeca0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f069400b950 tx=0x7f069400be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2ac5be5d8dd0d8b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f067f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f067f7fe640 1 -- 172.21.15.3:0/418908317 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694016070 con 0x5642750a89d0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f067f7fe640 1 -- 172.21.15.3:0/418908317 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0694003e80 con 0x5642750a89d0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.323+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564275035ee0 con 0x5642750a89d0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f067f7fe640 1 -- 172.21.15.3:0/418908317 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0694005020 con 0x5642750a89d0 2024-02-09T22:55:00.335 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 10 client.16944 fetch_fsmap learned FSMap version 35 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x56427500dd90 con 0x5642750a89d0 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f067f7fe640 1 -- 172.21.15.3:0/418908317 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564275035ee0 con 0x5642750a89d0 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f067f7fe640 1 -- 172.21.15.3:0/418908317 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f069401b470 con 0x5642750a89d0 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 10 client.16944 fetch_fsmap finished waiting for FSMap version 35 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 10 client.16944 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 20 client.16944 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5642750c1770 0x5642750c3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 4 client.16944 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564274d0f600 con 0x5642750c1770 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.324+0000 7f069ffff640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5642750c1770 0x5642750c3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.325+0000 7f069ffff640 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5642750c1770 0x5642750c3bb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b69084a0b9d62f server_cookie=6ea073fe95be7cc9 in_seq=0 out_seq=0 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.325+0000 7f067f7fe640 10 client.16944 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:00.336 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f067f7fe640 1 -- 172.21.15.3:0/418908317 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564274d0f600 con 0x5642750c1770 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f067f7fe640 10 client.16944 handle_command_reply: tid=0 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 client.16944 shutdown 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 msgr2=0x5642750aeca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750aeca0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f069400b950 tx=0x7f069400be10 comp rx=0 tx=0).stop 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 shutdown_connections 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5642750c1770 0x5642750c3bb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750b17a0 0x5642750af1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750aeca0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f067f7fe640 0 client.16944 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 --2- 172.21.15.3:0/418908317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750a80b0 0x5642750ae760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 >> 172.21.15.3:0/418908317 conn(0x5642750a29d0 msgr2=0x5642750a4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:00.337 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 shutdown_connections 2024-02-09T22:55:00.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 1 -- 172.21.15.3:0/418908317 wait complete. 2024-02-09T22:55:00.338 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.326+0000 7f06a944b740 20 client.16944 trim_cache size 0 max 16384 2024-02-09T22:55:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 Processor -- start 2024-02-09T22:55:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 -- start start 2024-02-09T22:55:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 0x5642750af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a89d0 0x5642750ad030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.348 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x5642750ad570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56427509cf10 con 0x5642750a80b0 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642750bd160 con 0x5642750b17a0 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.339+0000 7f06a944b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5642750be220 con 0x5642750a89d0 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 0x5642750af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 0x5642750af000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32932/0 (socket says 172.21.15.3:32932) 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 -- 172.21.15.3:0/555102200 learned_addr learned my addr 172.21.15.3:0/555102200 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 -- 172.21.15.3:0/555102200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a89d0 msgr2=0x5642750ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069f7fe640 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a89d0 0x5642750ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f06a4e20640 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x5642750ad570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a89d0 0x5642750ad030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 -- 172.21.15.3:0/555102200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 msgr2=0x5642750ad570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x5642750ad570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 -- 172.21.15.3:0/555102200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564274faa820 con 0x5642750a80b0 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f06a4e20640 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x5642750ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f069ffff640 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 0x5642750af000 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f06900bcfa0 tx=0x7f0690000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f8234649ca0cb3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f0685ffb640 1 -- 172.21.15.3:0/555102200 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06900c13f0 con 0x5642750a80b0 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f0685ffb640 1 -- 172.21.15.3:0/555102200 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06900c5a40 con 0x5642750a80b0 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.340+0000 7f0685ffb640 1 -- 172.21.15.3:0/555102200 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06900b9a90 con 0x5642750a80b0 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 -- 172.21.15.3:0/555102200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 msgr2=0x5642750af000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 0x5642750af000 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f06900bcfa0 tx=0x7f0690000bc0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 -- 172.21.15.3:0/555102200 shutdown_connections 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x5642750ad570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a89d0 0x5642750ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 --2- 172.21.15.3:0/555102200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a80b0 0x5642750af000 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 -- 172.21.15.3:0/555102200 >> 172.21.15.3:0/555102200 conn(0x5642750a29d0 msgr2=0x5642750c8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 -- 172.21.15.3:0/555102200 shutdown_connections 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 -- 172.21.15.3:0/555102200 wait complete. 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 -- start start 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.341+0000 7f06a944b740 1 Processor -- start 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a80b0 0x5642750b63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x564275116cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564275117210 con 0x5642750a89d0 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564275117350 con 0x5642750b17a0 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564275117490 con 0x5642750a80b0 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 10 client.0 _ll_register_callbacks cb 0x7f06940042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32948/0 (socket says 172.21.15.3:32948) 2024-02-09T22:55:00.351 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 -- 172.21.15.3:0/1094965341 learned_addr learned my addr 172.21.15.3:0/1094965341 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069ffff640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a80b0 0x5642750b63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a4e20640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x564275116cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 -- 172.21.15.3:0/1094965341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a80b0 msgr2=0x5642750b63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a80b0 0x5642750b63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 -- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 msgr2=0x564275116cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x564275116cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 -- 172.21.15.3:0/1094965341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564274fd43b0 con 0x5642750a89d0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f069f7fe640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b68e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0694004280 tx=0x7f0694003f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b960df182846352 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f067f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f067f7fe640 1 -- 172.21.15.3:0/1094965341 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0694016070 con 0x5642750a89d0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f067f7fe640 1 -- 172.21.15.3:0/1094965341 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0694000e40 con 0x5642750a89d0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.342+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f06900bd2a0 con 0x5642750a89d0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f067f7fe640 1 -- 172.21.15.3:0/1094965341 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7f069401b670 con 0x5642750a89d0 2024-02-09T22:55:00.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 10 client.16950 fetch_fsmap learned FSMap version 35 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) v3 -- 0x564274fd06f0 con 0x5642750a89d0 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f067f7fe640 1 -- 172.21.15.3:0/1094965341 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06900bd2a0 con 0x5642750a89d0 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f067f7fe640 1 -- 172.21.15.3:0/1094965341 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 35) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f0694036a20 con 0x5642750a89d0 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 10 client.16950 fetch_fsmap finished waiting for FSMap version 35 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 10 client.16950 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 20 client.16950 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564275118ba0 0x56427511afc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 4 client.16950 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564274d601b0 con 0x564275118ba0 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.343+0000 7f069ffff640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564275118ba0 0x56427511afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.344+0000 7f069ffff640 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564275118ba0 0x56427511afc0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=510b41bfafe75995 server_cookie=7441169fe96629be in_seq=0 out_seq=0 2024-02-09T22:55:00.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.344+0000 7f067f7fe640 10 client.16950 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.367+0000 7f067f7fe640 1 -- 172.21.15.3:0/1094965341 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564274d601b0 con 0x564275118ba0 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.367+0000 7f067f7fe640 10 client.16950 handle_command_reply: tid=0 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.367+0000 7f06a944b740 1 client.16950 shutdown 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 msgr2=0x5642750b68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b68e0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0694004280 tx=0x7f0694003f00 comp rx=0 tx=0).stop 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 shutdown_connections 2024-02-09T22:55:00.372 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564275118ba0 0x56427511afc0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5642750b17a0 0x564275116cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5642750a89d0 0x5642750b68e0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f067f7fe640 0 client.16950 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 --2- 172.21.15.3:0/1094965341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5642750a80b0 0x5642750b63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 >> 172.21.15.3:0/1094965341 conn(0x5642750a29d0 msgr2=0x5642750acfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 shutdown_connections 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 1 -- 172.21.15.3:0/1094965341 wait complete. 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a944b740 20 client.16950 trim_cache size 0 max 16384 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:55:00.373 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:55:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:55:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:55:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06880a2220 msgr2=0x7f06880a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06880a2220 0x7f06880a46e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f0698005fd0 tx=0x7f06980099c0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 msgr2=0x7f06a019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 0x7f06a019d600 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f0690009230 tx=0x7f0690004040 comp rx=0 tx=0).stop 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 shutdown_connections 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06880a2220 0x7f06880a46e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f06a0107420 0x7f06a01976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f06a01004a0 0x7f06a019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 --2- 172.21.15.3:0/3322151339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f06a00ffae0 0x7f06a019d600 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.368+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 >> 172.21.15.3:0/3322151339 conn(0x7f06a00753d0 msgr2=0x7f06a00fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.369+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 shutdown_connections 2024-02-09T22:55:00.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:00.369+0000 7f06a6883640 1 -- 172.21.15.3:0/3322151339 wait complete. 2024-02-09T22:55:00.434 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:00 smithi003 ceph-mon[32678]: pgmap v115: 97 pgs: 97 active+clean; 46 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 598 KiB/s wr, 82 op/s 2024-02-09T22:55:00.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:00 smithi064 ceph-mon[36638]: pgmap v115: 97 pgs: 97 active+clean; 46 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 598 KiB/s wr, 82 op/s 2024-02-09T22:55:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:00 smithi089 ceph-mon[36665]: pgmap v115: 97 pgs: 97 active+clean; 46 MiB data, 691 MiB used, 1.0 TiB / 1.0 TiB avail; 542 KiB/s rd, 598 KiB/s wr, 82 op/s 2024-02-09T22:55:00.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100152 2.13 MB/sec execute 54 sec latency 2538.686 ms 2024-02-09T22:55:01.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 100814 2.17 MB/sec execute 55 sec latency 2922.717 ms 2024-02-09T22:55:02.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:02 smithi064 ceph-mon[36638]: pgmap v116: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 880 KiB/s wr, 103 op/s 2024-02-09T22:55:02.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:02.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:02.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:02.436 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:02 smithi089 ceph-mon[36665]: pgmap v116: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 880 KiB/s wr, 103 op/s 2024-02-09T22:55:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:02 smithi003 ceph-mon[32678]: pgmap v116: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 880 KiB/s wr, 103 op/s 2024-02-09T22:55:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:02.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103227 2.38 MB/sec execute 56 sec latency 22.755 ms 2024-02-09T22:55:03.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:03 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:03.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:03 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:03 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:03.632 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103632 2.35 MB/sec execute 57 sec latency 559.337 ms 2024-02-09T22:55:04.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:04 smithi064 ceph-mon[36638]: pgmap v117: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 501 KiB/s wr, 71 op/s 2024-02-09T22:55:04.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:04 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:55:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:04 smithi089 ceph-mon[36665]: pgmap v117: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 501 KiB/s wr, 71 op/s 2024-02-09T22:55:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:04 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:55:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:04 smithi003 ceph-mon[32678]: pgmap v117: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 501 KiB/s wr, 71 op/s 2024-02-09T22:55:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:04 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:55:04.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103632 2.31 MB/sec execute 58 sec latency 1559.377 ms 2024-02-09T22:55:05.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103632 2.27 MB/sec execute 59 sec latency 2559.463 ms 2024-02-09T22:55:06.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:06 smithi064 ceph-mon[36638]: pgmap v118: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 501 KiB/s wr, 71 op/s 2024-02-09T22:55:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:06 smithi089 ceph-mon[36665]: pgmap v118: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 501 KiB/s wr, 71 op/s 2024-02-09T22:55:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:06 smithi003 ceph-mon[32678]: pgmap v118: 97 pgs: 97 active+clean; 46 MiB data, 692 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 501 KiB/s wr, 71 op/s 2024-02-09T22:55:06.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103632 2.23 MB/sec execute 60 sec latency 3559.555 ms 2024-02-09T22:55:07.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 103632 2.19 MB/sec execute 61 sec latency 4559.657 ms 2024-02-09T22:55:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:08 smithi089 ceph-mon[36665]: pgmap v119: 97 pgs: 97 active+clean; 48 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:08 smithi003 ceph-mon[32678]: pgmap v119: 97 pgs: 97 active+clean; 48 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:08.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 105300 2.27 MB/sec execute 62 sec latency 4969.080 ms 2024-02-09T22:55:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:08 smithi064 ceph-mon[36638]: pgmap v119: 97 pgs: 97 active+clean; 48 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:09.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107012 2.42 MB/sec execute 63 sec latency 9.551 ms 2024-02-09T22:55:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:10 smithi089 ceph-mon[36665]: pgmap v120: 97 pgs: 97 active+clean; 48 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 885 KiB/s wr, 117 op/s 2024-02-09T22:55:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:10 smithi003 ceph-mon[32678]: pgmap v120: 97 pgs: 97 active+clean; 48 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 885 KiB/s wr, 117 op/s 2024-02-09T22:55:10.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107117 2.39 MB/sec execute 64 sec latency 938.705 ms 2024-02-09T22:55:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:10 smithi064 ceph-mon[36638]: pgmap v120: 97 pgs: 97 active+clean; 48 MiB data, 710 MiB used, 1.0 TiB / 1.0 TiB avail; 573 KiB/s rd, 885 KiB/s wr, 117 op/s 2024-02-09T22:55:11.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107117 2.35 MB/sec execute 65 sec latency 1938.843 ms 2024-02-09T22:55:12.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:12 smithi089 ceph-mon[36665]: pgmap v121: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 173 op/s 2024-02-09T22:55:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:12 smithi003 ceph-mon[32678]: pgmap v121: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 173 op/s 2024-02-09T22:55:12.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107117 2.32 MB/sec execute 66 sec latency 2938.959 ms 2024-02-09T22:55:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:12 smithi064 ceph-mon[36638]: pgmap v121: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 173 op/s 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 -- 172.21.15.3:0/3352166922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 msgr2=0x7f616c10fe90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 --2- 172.21.15.3:0/3352166922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c10fe90 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f6164009ca0 tx=0x7f616402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 -- 172.21.15.3:0/3352166922 shutdown_connections 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 --2- 172.21.15.3:0/3352166922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c10fe90 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 --2- 172.21.15.3:0/3352166922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c069d80 0x7f616c06a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 --2- 172.21.15.3:0/3352166922 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c1021b0 0x7f616c069840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 -- 172.21.15.3:0/3352166922 >> 172.21.15.3:0/3352166922 conn(0x7f616c076ef0 msgr2=0x7f616c077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 -- 172.21.15.3:0/3352166922 shutdown_connections 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.466+0000 7f6173f83640 1 -- 172.21.15.3:0/3352166922 wait complete. 2024-02-09T22:55:13.472 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 Processor -- start 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 -- start start 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c069d80 0x7f616c19f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1021b0 0x7f616c19fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c1997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c199e80 con 0x7f616c10da80 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c199ff0 con 0x7f616c069d80 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6173f83640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c19a1b0 con 0x7f616c1021b0 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6172520640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c1997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6172520640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c1997e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50802/0 (socket says 172.21.15.3:50802) 2024-02-09T22:55:13.473 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.467+0000 7f6172520640 1 -- 172.21.15.3:0/671861886 learned_addr learned my addr 172.21.15.3:0/671861886 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6171d1f640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c069d80 0x7f616c19f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f617151e640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1021b0 0x7f616c19fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6172520640 1 -- 172.21.15.3:0/671861886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1021b0 msgr2=0x7f616c19fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6172520640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1021b0 0x7f616c19fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6172520640 1 -- 172.21.15.3:0/671861886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c069d80 msgr2=0x7f616c19f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6172520640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c069d80 0x7f616c19f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6172520640 1 -- 172.21.15.3:0/671861886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164009900 con 0x7f616c10da80 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f617151e640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1021b0 0x7f616c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6171d1f640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c069d80 0x7f616c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6172520640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c1997e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6164004af0 tx=0x7f6164034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6162ffd640 1 -- 172.21.15.3:0/671861886 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61640328c0 con 0x7f616c10da80 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6162ffd640 1 -- 172.21.15.3:0/671861886 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6164003e50 con 0x7f616c10da80 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c19a430 con 0x7f616c10da80 2024-02-09T22:55:13.474 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6162ffd640 1 -- 172.21.15.3:0/671861886 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6164004860 con 0x7f616c10da80 2024-02-09T22:55:13.475 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.468+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c1a66a0 con 0x7f616c10da80 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.470+0000 7f6162ffd640 1 -- 172.21.15.3:0/671861886 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f616400a040 con 0x7f616c10da80 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 Processor -- start 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 -- start start 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6162ffd640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2220 0x7f61580a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0253490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e025c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e025eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.482 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e01c1a00 con 0x5652e02530b0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e01bef30 con 0x5652e025c7a0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6176c31740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e01ba710 con 0x5652e02539d0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f617151e640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2220 0x7f61580a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0253490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.471+0000 7f6162ffd640 1 -- 172.21.15.3:0/671861886 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f61640f0da0 con 0x7f616c10da80 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0253490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50806/0 (socket says 172.21.15.3:50806) 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 -- 172.21.15.3:0/1246879952 learned_addr learned my addr 172.21.15.3:0/1246879952 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f617151e640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e025c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6172520640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e025eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 -- 172.21.15.3:0/1246879952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 msgr2=0x5652e025c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.483 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e025c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 -- 172.21.15.3:0/1246879952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 msgr2=0x5652e025eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e025eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 -- 172.21.15.3:0/1246879952 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652e017f3b0 con 0x5652e02530b0 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f617151e640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e025c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6172520640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e025eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f617151e640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2220 0x7f61580a46e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f615c0058c0 tx=0x7f615c014830 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6171d1f640 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0253490 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f6168005fd0 tx=0x7f616800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df8ed92a386efb97 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6157fff640 1 -- 172.21.15.3:0/1246879952 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168005020 con 0x5652e02530b0 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6157fff640 1 -- 172.21.15.3:0/1246879952 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6168005660 con 0x5652e02530b0 2024-02-09T22:55:13.484 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.472+0000 7f6157fff640 1 -- 172.21.15.3:0/1246879952 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61680116d0 con 0x5652e02530b0 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 -- 172.21.15.3:0/1246879952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 msgr2=0x5652e0253490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0253490 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f6168005fd0 tx=0x7f616800e6c0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 -- 172.21.15.3:0/1246879952 shutdown_connections 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e025eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e025c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 --2- 172.21.15.3:0/1246879952 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0253490 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 -- 172.21.15.3:0/1246879952 >> 172.21.15.3:0/1246879952 conn(0x5652e024d9d0 msgr2=0x5652e024ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 -- 172.21.15.3:0/1246879952 shutdown_connections 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 -- 172.21.15.3:0/1246879952 wait complete. 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.473+0000 7f6176c31740 1 -- start start 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 Processor -- start 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0259730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0259c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.485 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e025a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e015fb90 con 0x5652e02530b0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652dfe8e250 con 0x5652e02539d0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e0140200 con 0x5652e025c7a0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6176c31740 10 client.0 _ll_register_callbacks cb 0x5652e01765d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0259730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0259730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50814/0 (socket says 172.21.15.3:50814) 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 -- 172.21.15.3:0/2327841759 learned_addr learned my addr 172.21.15.3:0/2327841759 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6172520640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e025a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f617151e640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0259c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 -- 172.21.15.3:0/2327841759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 msgr2=0x5652e025a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e025a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 -- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 msgr2=0x5652e0259c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0259c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f6171d1f640 1 -- 172.21.15.3:0/2327841759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6168009900 con 0x5652e02530b0 2024-02-09T22:55:13.486 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.474+0000 7f617151e640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0259c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6171d1f640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0259730 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6168009f90 tx=0x7f6168009fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299fa9913bece4e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f61557fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f61557fa640 1 -- 172.21.15.3:0/2327841759 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168021070 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f61557fa640 1 -- 172.21.15.3:0/2327841759 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f61680050b0 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5652e01e0ee0 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f61557fa640 1 -- 172.21.15.3:0/2327841759 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6168005270 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 10 client.16962 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5652e01b8d90 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f61557fa640 1 -- 172.21.15.3:0/2327841759 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5652e01e0ee0 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f61557fa640 1 -- 172.21.15.3:0/2327841759 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f616800c070 con 0x5652e02530b0 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 10 client.16962 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 10 client.16962 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 20 client.16962 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e026c880 0x5652e026ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.487 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 4 client.16962 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.475+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5652dfeba600 con 0x5652e026c880 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.476+0000 7f617151e640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e026c880 0x5652e026ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.476+0000 7f61557fa640 10 client.16962 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.476+0000 7f617151e640 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e026c880 0x5652e026ecc0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=62e0cbe31d51c649 server_cookie=7c6280c05c5d80fa in_seq=0 out_seq=0 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f61557fa640 1 -- 172.21.15.3:0/2327841759 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5652dfeba600 con 0x5652e026c880 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f61557fa640 10 client.16962 handle_command_reply: tid=0 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 client.16962 shutdown 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 msgr2=0x5652e0259730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0259730 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f6168009f90 tx=0x7f6168009fc0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 shutdown_connections 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e026c880 0x5652e026ecc0 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0259c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e025a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.488 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 --2- 172.21.15.3:0/2327841759 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e0259730 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f61557fa640 0 client.16962 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.477+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 >> 172.21.15.3:0/2327841759 conn(0x5652e024d9d0 msgr2=0x5652e024f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.478+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 shutdown_connections 2024-02-09T22:55:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.478+0000 7f6176c31740 1 -- 172.21.15.3:0/2327841759 wait complete. 2024-02-09T22:55:13.489 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.478+0000 7f6176c31740 20 client.16962 trim_cache size 0 max 16384 2024-02-09T22:55:13.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 Processor -- start 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 -- start start 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e025be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e0259fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e0258000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e02545a0 con 0x5652e02530b0 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e0257c80 con 0x5652e025c7a0 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6176c31740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e024e350 con 0x5652e02539d0 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e025be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e025be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50826/0 (socket says 172.21.15.3:50826) 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 -- 172.21.15.3:0/3853908421 learned_addr learned my addr 172.21.15.3:0/3853908421 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:13.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6172520640 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e0258000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f617151e640 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e0259fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 -- 172.21.15.3:0/3853908421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 msgr2=0x5652e0259fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e0259fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 -- 172.21.15.3:0/3853908421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 msgr2=0x5652e0258000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e0258000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.491+0000 7f6171d1f640 1 -- 172.21.15.3:0/3853908421 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652e0155820 con 0x5652e02530b0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6171d1f640 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e025be60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f61680095d0 tx=0x7f61680097e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75abf5c5827adc89 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f61577fe640 1 -- 172.21.15.3:0/3853908421 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168021040 con 0x5652e02530b0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f61577fe640 1 -- 172.21.15.3:0/3853908421 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61680092a0 con 0x5652e02530b0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f61577fe640 1 -- 172.21.15.3:0/3853908421 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168004020 con 0x5652e02530b0 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 -- 172.21.15.3:0/3853908421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 msgr2=0x5652e025be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e025be60 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f61680095d0 tx=0x7f61680097e0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 -- 172.21.15.3:0/3853908421 shutdown_connections 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e025c7a0 0x5652e0258000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e02539d0 0x5652e0259fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 --2- 172.21.15.3:0/3853908421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e025be60 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 -- 172.21.15.3:0/3853908421 >> 172.21.15.3:0/3853908421 conn(0x5652e024d9d0 msgr2=0x5652e0273660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 -- 172.21.15.3:0/3853908421 shutdown_connections 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.492+0000 7f6176c31740 1 -- 172.21.15.3:0/3853908421 wait complete. 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 -- start start 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 Processor -- start 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e02616d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0261c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e02bbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e02bc2f0 con 0x5652e02530b0 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e02bc430 con 0x5652e02539d0 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652e02bc5f0 con 0x5652e025c7a0 2024-02-09T22:55:13.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6176c31740 10 client.0 _ll_register_callbacks cb 0x7f6168009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e02616d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6171d1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e02616d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50828/0 (socket says 172.21.15.3:50828) 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6171d1f640 1 -- 172.21.15.3:0/295140430 learned_addr learned my addr 172.21.15.3:0/295140430 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f617151e640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0261c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.493+0000 7f6172520640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e02bbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6171d1f640 1 -- 172.21.15.3:0/295140430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 msgr2=0x5652e02bbdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6171d1f640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e02bbdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6171d1f640 1 -- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 msgr2=0x5652e0261c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6171d1f640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0261c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.502 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6171d1f640 1 -- 172.21.15.3:0/295140430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652e01b8d90 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f617151e640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0261c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6171d1f640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e02616d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6168009bf0 tx=0x7f61680052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a183f317359fbd2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f61557fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f61557fa640 1 -- 172.21.15.3:0/295140430 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6168021070 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f61557fa640 1 -- 172.21.15.3:0/295140430 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6168000e40 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6168005660 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f61557fa640 1 -- 172.21.15.3:0/295140430 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61680054d0 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 10 client.16968 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5652e017b6f0 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f61557fa640 1 -- 172.21.15.3:0/295140430 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6168005660 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f61557fa640 1 -- 172.21.15.3:0/295140430 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f616800c040 con 0x5652e02530b0 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 10 client.16968 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 10 client.16968 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:13.503 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 20 client.16968 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e02c3980 0x5652e02c5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.494+0000 7f6176c31740 4 client.16968 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:55:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.495+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5652dff0b1b0 con 0x5652e02c3980 2024-02-09T22:55:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.495+0000 7f617151e640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e02c3980 0x5652e02c5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.495+0000 7f61557fa640 10 client.16968 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:13.504 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.495+0000 7f617151e640 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e02c3980 0x5652e02c5d70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5500359dadacd30 server_cookie=676386a650ad9047 in_seq=0 out_seq=0 2024-02-09T22:55:13.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.527+0000 7f61557fa640 1 -- 172.21.15.3:0/295140430 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5652dff0b1b0 con 0x5652e02c3980 2024-02-09T22:55:13.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.527+0000 7f61557fa640 10 client.16968 handle_command_reply: tid=0 2024-02-09T22:55:13.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.527+0000 7f6176c31740 1 client.16968 shutdown 2024-02-09T22:55:13.531 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 msgr2=0x5652e02616d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e02616d0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f6168009bf0 tx=0x7f61680052e0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 shutdown_connections 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5652e02c3980 0x5652e02c5d70 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5652e02539d0 0x5652e0261c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5652e025c7a0 0x5652e02bbdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 --2- 172.21.15.3:0/295140430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5652e02530b0 0x5652e02616d0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f61557fa640 0 client.16968 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 >> 172.21.15.3:0/295140430 conn(0x5652e024d9d0 msgr2=0x5652e0257f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 shutdown_connections 2024-02-09T22:55:13.532 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 1 -- 172.21.15.3:0/295140430 wait complete. 2024-02-09T22:55:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.528+0000 7f6176c31740 20 client.16968 trim_cache size 0 max 16384 2024-02-09T22:55:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:55:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:55:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:55:13.533 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2220 msgr2=0x7f61580a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2220 0x7f61580a46e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f615c0058c0 tx=0x7f615c014830 comp rx=0 tx=0).stop 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 msgr2=0x7f616c1997e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c1997e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f6164004af0 tx=0x7f6164034cc0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 shutdown_connections 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f61580a2220 0x7f61580a46e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f616c10da80 0x7f616c1997e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.534 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f616c1021b0 0x7f616c19fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 --2- 172.21.15.3:0/671861886 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f616c069d80 0x7f616c19f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 >> 172.21.15.3:0/671861886 conn(0x7f616c076ef0 msgr2=0x7f616c0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 shutdown_connections 2024-02-09T22:55:13.535 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:13.529+0000 7f6173f83640 1 -- 172.21.15.3:0/671861886 wait complete. 2024-02-09T22:55:13.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107241 2.29 MB/sec execute 67 sec latency 3357.847 ms 2024-02-09T22:55:14.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:14 smithi089 ceph-mon[36665]: pgmap v122: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-02-09T22:55:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:14 smithi003 ceph-mon[32678]: pgmap v122: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-02-09T22:55:14.633 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107241 2.25 MB/sec execute 68 sec latency 1496.935 ms 2024-02-09T22:55:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:14 smithi064 ceph-mon[36638]: pgmap v122: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-02-09T22:55:15.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107241 2.22 MB/sec execute 69 sec latency 2496.990 ms 2024-02-09T22:55:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:16 smithi089 ceph-mon[36665]: pgmap v123: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-02-09T22:55:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:16 smithi003 ceph-mon[32678]: pgmap v123: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-02-09T22:55:16.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107241 2.19 MB/sec execute 70 sec latency 3497.079 ms 2024-02-09T22:55:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:16 smithi064 ceph-mon[36638]: pgmap v123: 97 pgs: 97 active+clean; 50 MiB data, 737 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 152 op/s 2024-02-09T22:55:17.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 107241 2.16 MB/sec execute 71 sec latency 4497.190 ms 2024-02-09T22:55:18.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:18 smithi089 ceph-mon[36665]: pgmap v124: 97 pgs: 97 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 178 op/s 2024-02-09T22:55:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:18 smithi003 ceph-mon[32678]: pgmap v124: 97 pgs: 97 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 178 op/s 2024-02-09T22:55:18.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 108851 2.22 MB/sec execute 72 sec latency 4905.890 ms 2024-02-09T22:55:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:18 smithi064 ceph-mon[36638]: pgmap v124: 97 pgs: 97 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 178 op/s 2024-02-09T22:55:19.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 110582 2.35 MB/sec execute 73 sec latency 12.812 ms 2024-02-09T22:55:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:20 smithi089 ceph-mon[36665]: pgmap v125: 97 pgs: 97 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 565 KiB/s wr, 81 op/s 2024-02-09T22:55:20.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:20 smithi003 ceph-mon[32678]: pgmap v125: 97 pgs: 97 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 565 KiB/s wr, 81 op/s 2024-02-09T22:55:20.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 110667 2.32 MB/sec execute 74 sec latency 955.359 ms 2024-02-09T22:55:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:20 smithi064 ceph-mon[36638]: pgmap v125: 97 pgs: 97 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 565 KiB/s wr, 81 op/s 2024-02-09T22:55:21.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 110667 2.29 MB/sec execute 75 sec latency 1955.464 ms 2024-02-09T22:55:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:22 smithi089 ceph-mon[36665]: pgmap v126: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 134 op/s 2024-02-09T22:55:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:22 smithi003 ceph-mon[32678]: pgmap v126: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 134 op/s 2024-02-09T22:55:22.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 110667 2.26 MB/sec execute 76 sec latency 2955.531 ms 2024-02-09T22:55:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:22 smithi064 ceph-mon[36638]: pgmap v126: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 134 op/s 2024-02-09T22:55:23.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:23 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:55:23.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:23 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:55:23.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 111871 2.30 MB/sec execute 77 sec latency 3365.219 ms 2024-02-09T22:55:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:23 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:55:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:24 smithi089 ceph-mon[36665]: pgmap v127: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 543 KiB/s wr, 78 op/s 2024-02-09T22:55:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:55:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:55:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:55:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:24 smithi003 ceph-mon[32678]: pgmap v127: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 543 KiB/s wr, 78 op/s 2024-02-09T22:55:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:55:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:55:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:55:24.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 113958 2.45 MB/sec execute 78 sec latency 29.042 ms 2024-02-09T22:55:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:24 smithi064 ceph-mon[36638]: pgmap v127: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 543 KiB/s wr, 78 op/s 2024-02-09T22:55:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:55:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:55:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:55:24.968 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:55:24.968 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 -- 172.21.15.3:0/245255158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a8105410 msgr2=0x7fb0a810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/245255158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a8105410 0x7fb0a810cfe0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0009c10 tx=0x7fb0a002bef0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 -- 172.21.15.3:0/245255158 shutdown_connections 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/245255158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0a8100cf0 0x7fb0a8104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/245255158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a8105410 0x7fb0a810cfe0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/245255158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0a8100340 0x7fb0a8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 -- 172.21.15.3:0/245255158 >> 172.21.15.3:0/245255158 conn(0x7fb0a80772e0 msgr2=0x7fb0a80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 -- 172.21.15.3:0/245255158 shutdown_connections 2024-02-09T22:55:25.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.077+0000 7fb0aec8d640 1 -- 172.21.15.3:0/245255158 wait complete. 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 Processor -- start 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 -- start start 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0a8100340 0x7fb0a819d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0a8105410 0x7fb0a819dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 0x7fb0a8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8198180 con 0x7fb0a81976b0 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a81982f0 con 0x7fb0a8100340 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aec8d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a81984b0 con 0x7fb0a8105410 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 0x7fb0a8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0a8100340 0x7fb0a819d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 0x7fb0a8197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59796/0 (socket says 172.21.15.3:59796) 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 -- 172.21.15.3:0/1938226766 learned_addr learned my addr 172.21.15.3:0/1938226766 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0a8105410 0x7fb0a819dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 -- 172.21.15.3:0/1938226766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0a8105410 msgr2=0x7fb0a819dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0a8105410 0x7fb0a819dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 -- 172.21.15.3:0/1938226766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0a8100340 msgr2=0x7fb0a819d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0a8100340 0x7fb0a819d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 -- 172.21.15.3:0/1938226766 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a0009840 con 0x7fb0a81976b0 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.078+0000 7fb0ad22a640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 0x7fb0a8197b30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0036040 tx=0x7fb0a002cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.079+0000 7fb0a5ffb640 1 -- 172.21.15.3:0/1938226766 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a0003970 con 0x7fb0a81976b0 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.079+0000 7fb0a5ffb640 1 -- 172.21.15.3:0/1938226766 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb0a0004040 con 0x7fb0a81976b0 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.079+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a81a43a0 con 0x7fb0a81976b0 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.079+0000 7fb0a5ffb640 1 -- 172.21.15.3:0/1938226766 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0a003d690 con 0x7fb0a81976b0 2024-02-09T22:55:25.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.079+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a81a4870 con 0x7fb0a81976b0 2024-02-09T22:55:25.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.081+0000 7fb0a5ffb640 1 -- 172.21.15.3:0/1938226766 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb0a0003ad0 con 0x7fb0a81976b0 2024-02-09T22:55:25.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.081+0000 7fb0b185b740 1 Processor -- start 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 -- start start 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b7980 0x559a881b7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 0x559a881c08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881c0eb0 0x559a881c3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88172d00 con 0x559a881b8330 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a881272f0 con 0x559a881b7980 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0b185b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88124820 con 0x559a881c0eb0 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 0x559a881c08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 0x559a881c08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59806/0 (socket says 172.21.15.3:59806) 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 -- 172.21.15.3:0/995482988 learned_addr learned my addr 172.21.15.3:0/995482988 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b7980 0x559a881b7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881c0eb0 0x559a881c3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0a5ffb640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb0980a22b0 0x7fb0980a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0ad22a640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb0980a22b0 0x7fb0980a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 -- 172.21.15.3:0/995482988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881c0eb0 msgr2=0x559a881c3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881c0eb0 0x559a881c3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 -- 172.21.15.3:0/995482988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b7980 msgr2=0x559a881b7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b7980 0x559a881b7d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 -- 172.21.15.3:0/995482988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a880e1ff0 con 0x559a881b8330 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b7980 0x559a881b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0a5ffb640 1 -- 172.21.15.3:0/1938226766 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb0a00f03a0 con 0x7fb0a81976b0 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.082+0000 7fb0aca29640 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 0x559a881c08e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009ce0 tx=0x7fb09c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbdc2c1f183bff8f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.083+0000 7fb092ffd640 1 -- 172.21.15.3:0/995482988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c019630 con 0x559a881b8330 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.083+0000 7fb092ffd640 1 -- 172.21.15.3:0/995482988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb09c019c70 con 0x559a881b8330 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.083+0000 7fb092ffd640 1 -- 172.21.15.3:0/995482988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c00ead0 con 0x559a881b8330 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.083+0000 7fb0ad22a640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb0980a22b0 0x7fb0980a4770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00edce0 tx=0x7fb0a002bef0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.083+0000 7fb0b185b740 1 -- 172.21.15.3:0/995482988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 msgr2=0x559a881c08e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.083+0000 7fb0b185b740 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 0x559a881c08e0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009ce0 tx=0x7fb09c011810 comp rx=0 tx=0).stop 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 -- 172.21.15.3:0/995482988 shutdown_connections 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881c0eb0 0x559a881c3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881b8330 0x559a881c08e0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 --2- 172.21.15.3:0/995482988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b7980 0x559a881b7d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 -- 172.21.15.3:0/995482988 >> 172.21.15.3:0/995482988 conn(0x559a881b2050 msgr2=0x559a881b2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 -- 172.21.15.3:0/995482988 shutdown_connections 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 -- 172.21.15.3:0/995482988 wait complete. 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 -- start start 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.084+0000 7fb0b185b740 1 Processor -- start 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881bddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881be830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88120000 con 0x559a881c0eb0 2024-02-09T22:55:25.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a87df3430 con 0x559a881b8330 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a87dafe00 con 0x559a881b7980 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0b185b740 10 client.0 _ll_register_callbacks cb 0x559a881147f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881bddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881bddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:36958/0 (socket says 172.21.15.3:36958) 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 -- 172.21.15.3:0/807891410 learned_addr learned my addr 172.21.15.3:0/807891410 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881be830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 -- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 msgr2=0x559a881be2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 -- 172.21.15.3:0/807891410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 msgr2=0x559a881be830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881be830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 -- 172.21.15.3:0/807891410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c009910 con 0x559a881b7980 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0aca29640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881bddb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb094012d60 tx=0x7fb094009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ae6f6ae76ffbc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.085+0000 7fb087fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb087fff640 1 -- 172.21.15.3:0/807891410 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb094005220 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb087fff640 1 -- 172.21.15.3:0/807891410 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb09400fe60 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a87e1f7e0 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb087fff640 1 -- 172.21.15.3:0/807891410 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb094005380 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 10 client.26338 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x559a8811e680 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb087fff640 1 -- 172.21.15.3:0/807891410 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb09402e020 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb087fff640 1 -- 172.21.15.3:0/807891410 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb094021550 con 0x559a881b7980 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 10 client.26338 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 10 client.26338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 20 client.26338 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a881d13d0 0x559a881d37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 4 client.26338 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a87e5c720 con 0x559a881d13d0 2024-02-09T22:55:25.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.086+0000 7fb0aca29640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a881d13d0 0x559a881d37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0aca29640 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a881d13d0 0x559a881d37f0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67143888a4c7e2ef server_cookie=268dbbadac03788 in_seq=0 out_seq=0 2024-02-09T22:55:25.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb087fff640 10 client.26338 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb087fff640 1 -- 172.21.15.3:0/807891410 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559a87e5c720 con 0x559a881d13d0 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb087fff640 10 client.26338 handle_command_reply: tid=0 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 client.26338 shutdown 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 msgr2=0x559a881bddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881bddb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb094012d60 tx=0x7fb094009a20 comp rx=0 tx=0).stop 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 shutdown_connections 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a881d13d0 0x559a881d37f0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881be830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 --2- 172.21.15.3:0/807891410 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881bddb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 >> 172.21.15.3:0/807891410 conn(0x559a881b2050 msgr2=0x559a881b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.087+0000 7fb087fff640 0 client.26338 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:25.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.088+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 shutdown_connections 2024-02-09T22:55:25.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.088+0000 7fb0b185b740 1 -- 172.21.15.3:0/807891410 wait complete. 2024-02-09T22:55:25.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.088+0000 7fb0b185b740 20 client.26338 trim_cache size 0 max 16384 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 Processor -- start 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 -- start start 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881bb010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a881ad780 con 0x559a881c0eb0 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a880a4fd0 con 0x559a881b8330 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0b185b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a881b72d0 con 0x559a881b7980 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0a7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881bb010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:33262/0 (socket says 172.21.15.3:33262) 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 -- 172.21.15.3:0/3088336437 learned_addr learned my addr 172.21.15.3:0/3088336437 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 -- 172.21.15.3:0/3088336437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 msgr2=0x559a881c04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 -- 172.21.15.3:0/3088336437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 msgr2=0x559a881bb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881bb010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.103+0000 7fb0aca29640 1 -- 172.21.15.3:0/3088336437 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a880bab30 con 0x559a881b8330 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0aca29640 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be650 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009ce0 tx=0x7fb09c009f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dc67c17ace08722 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0927fc640 1 -- 172.21.15.3:0/3088336437 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c029070 con 0x559a881b8330 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0927fc640 1 -- 172.21.15.3:0/3088336437 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb09c011d10 con 0x559a881b8330 2024-02-09T22:55:25.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0927fc640 1 -- 172.21.15.3:0/3088336437 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c019690 con 0x559a881b8330 2024-02-09T22:55:25.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 -- 172.21.15.3:0/3088336437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 msgr2=0x559a881be650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be650 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009ce0 tx=0x7fb09c009f70 comp rx=0 tx=0).stop 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 -- 172.21.15.3:0/3088336437 shutdown_connections 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881be650 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a881bb010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 --2- 172.21.15.3:0/3088336437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 -- 172.21.15.3:0/3088336437 >> 172.21.15.3:0/3088336437 conn(0x559a881b2050 msgr2=0x559a881d8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 -- 172.21.15.3:0/3088336437 shutdown_connections 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.104+0000 7fb0b185b740 1 -- 172.21.15.3:0/3088336437 wait complete. 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 -- start start 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 Processor -- start 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881c7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a882278e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a881c7b80 con 0x559a881c0eb0 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88227e20 con 0x559a881b8330 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a88227f60 con 0x559a881b7980 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0b185b740 10 client.0 _ll_register_callbacks cb 0x7fb09c0097c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0a7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0a7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a882278e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0aca29640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:33270/0 (socket says 172.21.15.3:33270) 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0aca29640 1 -- 172.21.15.3:0/508542770 learned_addr learned my addr 172.21.15.3:0/508542770 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.105+0000 7fb0aca29640 1 -- 172.21.15.3:0/508542770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 msgr2=0x559a881c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0aca29640 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0aca29640 1 -- 172.21.15.3:0/508542770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 msgr2=0x559a882278e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0aca29640 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a882278e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0aca29640 1 -- 172.21.15.3:0/508542770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a880e1ff0 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0aca29640 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881c7480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00ed70 tx=0x7fb09c00e7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65d5d378db077b08 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb087fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb087fff640 1 -- 172.21.15.3:0/508542770 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb09c029040 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb09c019c70 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb087fff640 1 -- 172.21.15.3:0/508542770 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb09c005230 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb087fff640 1 -- 172.21.15.3:0/508542770 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb09c0053f0 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0b185b740 10 client.26297 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x559a880e0850 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb087fff640 1 -- 172.21.15.3:0/508542770 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb09c019c70 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb087fff640 1 -- 172.21.15.3:0/508542770 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb09c0048e0 con 0x559a881b8330 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0b185b740 10 client.26297 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0b185b740 10 client.26297 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.106+0000 7fb0b185b740 20 client.26297 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a882293d0 0x559a8822b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 4 client.26297 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559a87eab890 con 0x559a882293d0 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a882293d0 0x559a8822b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0a7fff640 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a882293d0 0x559a8822b7f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26745738a6785c9 server_cookie=b302ff74c7997ba2 in_seq=0 out_seq=0 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb087fff640 10 client.26297 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb087fff640 1 -- 172.21.15.3:0/508542770 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559a87eab890 con 0x559a882293d0 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb087fff640 10 client.26297 handle_command_reply: tid=0 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 client.26297 shutdown 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 msgr2=0x559a881c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881c7480 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb09c00ed70 tx=0x7fb09c00e7f0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 shutdown_connections 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559a882293d0 0x559a8822b7f0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a881b8330 0x559a881c7480 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a881c0eb0 0x559a882278e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 --2- 172.21.15.3:0/508542770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a881b7980 0x559a881c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb087fff640 0 client.26297 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 >> 172.21.15.3:0/508542770 conn(0x559a881b2050 msgr2=0x559a881b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 shutdown_connections 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.107+0000 7fb0b185b740 1 -- 172.21.15.3:0/508542770 wait complete. 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0b185b740 20 client.26297 trim_cache size 0 max 16384 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:55:25.117 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:55:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb0980a22b0 msgr2=0x7fb0980a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb0980a22b0 0x7fb0980a4770 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb0a00edce0 tx=0x7fb0a002bef0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 msgr2=0x7fb0a8197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 0x7fb0a8197b30 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fb0a0036040 tx=0x7fb0a002cac0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 shutdown_connections 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb0980a22b0 0x7fb0980a4770 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb0a81976b0 0x7fb0a8197b30 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb0a8105410 0x7fb0a819dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 --2- 172.21.15.3:0/1938226766 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb0a8100340 0x7fb0a819d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 >> 172.21.15.3:0/1938226766 conn(0x7fb0a80772e0 msgr2=0x7fb0a80fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 shutdown_connections 2024-02-09T22:55:25.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.108+0000 7fb0aec8d640 1 -- 172.21.15.3:0/1938226766 wait complete. 2024-02-09T22:55:25.124 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:55:25.124 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 -- 172.21.15.3:0/4042629425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68100d00 msgr2=0x7f3e68104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/4042629425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68100d00 0x7f3e68104d40 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f3e50009930 tx=0x7f3e5002bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 -- 172.21.15.3:0/4042629425 shutdown_connections 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/4042629425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68105420 0x7f3e6810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/4042629425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68100d00 0x7f3e68104d40 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/4042629425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e68100350 0x7f3e68100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.225+0000 7f3e6ca55640 1 -- 172.21.15.3:0/4042629425 >> 172.21.15.3:0/4042629425 conn(0x7f3e680772b0 msgr2=0x7f3e680776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.226+0000 7f3e6ca55640 1 -- 172.21.15.3:0/4042629425 shutdown_connections 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.226+0000 7f3e6ca55640 1 -- 172.21.15.3:0/4042629425 wait complete. 2024-02-09T22:55:25.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.226+0000 7f3e6ca55640 1 Processor -- start 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.226+0000 7f3e6ca55640 1 -- start start 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e68100350 0x7f3e6819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 0x7f3e6819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68197700 0x7f3e68197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e681981d0 con 0x7f3e68105420 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e68198340 con 0x7f3e68100350 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e68198500 con 0x7f3e68197700 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 0x7f3e6819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68197700 0x7f3e68197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e5e59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e68100350 0x7f3e6819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 0x7f3e6819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59846/0 (socket says 172.21.15.3:59846) 2024-02-09T22:55:25.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6659c640 1 -- 172.21.15.3:0/220249711 learned_addr learned my addr 172.21.15.3:0/220249711 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 -- 172.21.15.3:0/220249711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e68100350 msgr2=0x7f3e6819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e68100350 0x7f3e6819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 -- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 msgr2=0x7f3e6819db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 0x7f3e6819db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 -- 172.21.15.3:0/220249711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e50009560 con 0x7f3e68197700 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6659c640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 0x7f3e6819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e66d9d640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68197700 0x7f3e68197b80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009fd0 tx=0x7f3e6000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e6000ce20 con 0x7f3e68197700 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e681a4450 con 0x7f3e68197700 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.227+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e60005020 con 0x7f3e68197700 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.228+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e60010690 con 0x7f3e68197700 2024-02-09T22:55:25.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.228+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e681a48e0 con 0x7f3e68197700 2024-02-09T22:55:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.228+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e68101820 con 0x7f3e68197700 2024-02-09T22:55:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.229+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3e60005500 con 0x7f3e68197700 2024-02-09T22:55:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.230+0000 7f3e5ffff640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3e540a2230 0x7f3e540a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.230+0000 7f3e5e59c640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3e540a2230 0x7f3e540a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.230+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3e60014070 con 0x7f3e68197700 2024-02-09T22:55:25.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.230+0000 7f3e5e59c640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3e540a2230 0x7f3e540a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009bd0 tx=0x7f3e58005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.235+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3e60097410 con 0x7f3e68197700 2024-02-09T22:55:25.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.477+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e681086b0 con 0x7f3e68197700 2024-02-09T22:55:25.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.478+0000 7f3e5ffff640 1 -- 172.21.15.3:0/220249711 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f3e60005190 con 0x7f3e68197700 2024-02-09T22:55:25.481 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:55:25.482 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:55:02.762674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:55:25.482 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 36 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3e540a2230 msgr2=0x7f3e540a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3e540a2230 0x7f3e540a46f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f3e58009bd0 tx=0x7f3e58005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68197700 msgr2=0x7f3e68197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68197700 0x7f3e68197b80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3e60009fd0 tx=0x7f3e6000ef10 comp rx=0 tx=0).stop 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 shutdown_connections 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3e540a2230 0x7f3e540a46f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3e68197700 0x7f3e68197b80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3e68105420 0x7f3e6819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 --2- 172.21.15.3:0/220249711 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3e68100350 0x7f3e6819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 >> 172.21.15.3:0/220249711 conn(0x7f3e680772b0 msgr2=0x7f3e680fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 shutdown_connections 2024-02-09T22:55:25.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.481+0000 7f3e6ca55640 1 -- 172.21.15.3:0/220249711 wait complete. 2024-02-09T22:55:25.495 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:25.495 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:55:25.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 -- 172.21.15.3:0/2786561798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c140fffd0 msgr2=0x7f9c141003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/2786561798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c140fffd0 0x7f9c141003b0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009860 tx=0x7f9c0402bc00 comp rx=0 tx=0).stop 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 -- 172.21.15.3:0/2786561798 shutdown_connections 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/2786561798 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 0x7f9c1410d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/2786561798 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c1410db80 0x7f9c1410ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/2786561798 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c140fffd0 0x7f9c141003b0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 -- 172.21.15.3:0/2786561798 >> 172.21.15.3:0/2786561798 conn(0x7f9c140772e0 msgr2=0x7f9c140776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 -- 172.21.15.3:0/2786561798 shutdown_connections 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 -- 172.21.15.3:0/2786561798 wait complete. 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.600+0000 7f9c19cd6640 1 Processor -- start 2024-02-09T22:55:25.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 -- start start 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c140fffd0 0x7f9c1419f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 0x7f9c1419fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 0x7f9c14199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14199f80 con 0x7f9c1410db80 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1419a0f0 con 0x7f9c141008f0 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c19cd6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1419a2b0 con 0x7f9c140fffd0 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 0x7f9c14199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 0x7f9c14199930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59864/0 (socket says 172.21.15.3:59864) 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c12ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 0x7f9c1419fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c140fffd0 0x7f9c1419f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 -- 172.21.15.3:0/1863324709 learned_addr learned my addr 172.21.15.3:0/1863324709 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 -- 172.21.15.3:0/1863324709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c140fffd0 msgr2=0x7f9c1419f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c140fffd0 0x7f9c1419f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 -- 172.21.15.3:0/1863324709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 msgr2=0x7f9c1419fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 0x7f9c1419fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 -- 172.21.15.3:0/1863324709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c04009490 con 0x7f9c1410db80 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 0x7f9c1419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c137fe640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c140fffd0 0x7f9c1419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.601+0000 7f9c13fff640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 0x7f9c14199930 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00ee80 tx=0x7f9c0c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.602+0000 7f9c10ff9640 1 -- 172.21.15.3:0/1863324709 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0099a0 con 0x7f9c1410db80 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.602+0000 7f9c10ff9640 1 -- 172.21.15.3:0/1863324709 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c0c003970 con 0x7f9c1410db80 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.602+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1419a590 con 0x7f9c1410db80 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.602+0000 7f9c10ff9640 1 -- 172.21.15.3:0/1863324709 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c004780 con 0x7f9c1410db80 2024-02-09T22:55:25.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.602+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c141a67d0 con 0x7f9c1410db80 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.604+0000 7f9c10ff9640 1 -- 172.21.15.3:0/1863324709 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9c0c00a040 con 0x7f9c1410db80 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.604+0000 7f9c1c981740 1 Processor -- start 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 -- start start 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36aa30 0x55a14a36ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 0x55a14a373990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a373f60 0x55a14a3763b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a326020 con 0x55a14a373f60 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a2d9320 con 0x55a14a36aa30 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c1c981740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a2d6850 con 0x55a14a36b3e0 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c10ff9640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bfc0a2230 0x7f9bfc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 0x55a14a373990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a373f60 0x55a14a3763b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 0x55a14a373990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:37034/0 (socket says 172.21.15.3:37034) 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 -- 172.21.15.3:0/3297199712 learned_addr learned my addr 172.21.15.3:0/3297199712 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36aa30 0x55a14a36ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c13fff640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bfc0a2230 0x7f9bfc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c10ff9640 1 -- 172.21.15.3:0/1863324709 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9c0c0cdec0 con 0x7f9c1410db80 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 -- 172.21.15.3:0/3297199712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36aa30 msgr2=0x55a14a36ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36aa30 0x55a14a36ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 -- 172.21.15.3:0/3297199712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a373f60 msgr2=0x55a14a3763b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a373f60 0x55a14a3763b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.605+0000 7f9c137fe640 1 -- 172.21.15.3:0/3297199712 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a14a294210 con 0x55a14a36b3e0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.606+0000 7f9c137fe640 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 0x55a14a373990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009860 tx=0x7f9c040058d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=243da1da03a3ff7d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.606+0000 7f9c13fff640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bfc0a2230 0x7f9bfc0a46f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c09a540 tx=0x7f9c0c009b00 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.606+0000 7f9bf9ffb640 1 -- 172.21.15.3:0/3297199712 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c04004400 con 0x55a14a36b3e0 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.606+0000 7f9bf9ffb640 1 -- 172.21.15.3:0/3297199712 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c04034920 con 0x55a14a36b3e0 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.606+0000 7f9bf9ffb640 1 -- 172.21.15.3:0/3297199712 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040035f0 con 0x55a14a36b3e0 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- 172.21.15.3:0/3297199712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 msgr2=0x55a14a373990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 0x55a14a373990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009860 tx=0x7f9c040058d0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- 172.21.15.3:0/3297199712 shutdown_connections 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a373f60 0x55a14a3763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36b3e0 0x55a14a373990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- 172.21.15.3:0/3297199712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36aa30 0x55a14a36ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- 172.21.15.3:0/3297199712 >> 172.21.15.3:0/3297199712 conn(0x55a14a365100 msgr2=0x55a14a3654e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- 172.21.15.3:0/3297199712 shutdown_connections 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- 172.21.15.3:0/3297199712 wait complete. 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- start start 2024-02-09T22:55:25.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 Processor -- start 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 0x55a14a370eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36b3e0 0x55a14a3713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a373f60 0x55a14a371930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.607+0000 7f9c1c981740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a2d2030 con 0x55a14a36aa30 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c1c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a149fa5430 con 0x55a14a36b3e0 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c1c981740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a149f61e00 con 0x55a14a373f60 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c1c981740 10 client.0 _ll_register_callbacks cb 0x55a14a2c7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 0x55a14a370eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 0x55a14a370eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59886/0 (socket says 172.21.15.3:59886) 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 -- 172.21.15.3:0/988518514 learned_addr learned my addr 172.21.15.3:0/988518514 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c137fe640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36b3e0 0x55a14a3713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a373f60 0x55a14a371930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 -- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a373f60 msgr2=0x55a14a371930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a373f60 0x55a14a371930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 -- 172.21.15.3:0/988518514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36b3e0 msgr2=0x55a14a3713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36b3e0 0x55a14a3713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 -- 172.21.15.3:0/988518514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14110b00 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c137fe640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36b3e0 0x55a14a3713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 0x55a14a370eb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f9c08004640 tx=0x7f9c08004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2423420a80bc76 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9bf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/988518514 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c08005020 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/988518514 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9c08005180 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a149fd17e0 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/988518514 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c08005340 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c1c981740 10 client.17004 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.608+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55a14a2d06b0 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/988518514 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a149fd17e0 con 0x55a14a36aa30 2024-02-09T22:55:25.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/988518514 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c0800e7b0 con 0x55a14a36aa30 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c1c981740 10 client.17004 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c1c981740 10 client.17004 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c1c981740 20 client.17004 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c1c981740 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a384520 0x55a14a386940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c1c981740 4 client.17004 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a14a00e720 con 0x55a14a384520 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c137fe640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a384520 0x55a14a386940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9c137fe640 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a384520 0x55a14a386940 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4fec9112a89fbf6d server_cookie=2771fb11ba6d3351 in_seq=0 out_seq=0 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.609+0000 7f9bf2ffd640 10 client.17004 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/988518514 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a14a00e720 con 0x55a14a384520 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9bf2ffd640 10 client.17004 handle_command_reply: tid=0 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 client.17004 shutdown 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 msgr2=0x55a14a370eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 0x55a14a370eb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f9c08004640 tx=0x7f9c08004730 comp rx=0 tx=0).stop 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 shutdown_connections 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a384520 0x55a14a386940 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a36b3e0 0x55a14a3713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a373f60 0x55a14a371930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9bf2ffd640 0 client.17004 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 --2- 172.21.15.3:0/988518514 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36aa30 0x55a14a370eb0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.610+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 >> 172.21.15.3:0/988518514 conn(0x55a14a365100 msgr2=0x55a14a366e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.611+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 shutdown_connections 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.611+0000 7f9c1c981740 1 -- 172.21.15.3:0/988518514 wait complete. 2024-02-09T22:55:25.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.611+0000 7f9c1c981740 20 client.17004 trim_cache size 0 max 16384 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 Processor -- start 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 -- start start 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a3735e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a36e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a36e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a257130 con 0x55a14a36b3e0 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a36a380 con 0x55a14a373f60 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c1c981740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a360ab0 con 0x55a14a36aa30 2024-02-09T22:55:25.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a36e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c12ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a3735e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a36e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59898/0 (socket says 172.21.15.3:59898) 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 -- 172.21.15.3:0/953174099 learned_addr learned my addr 172.21.15.3:0/953174099 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a36e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 -- 172.21.15.3:0/953174099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 msgr2=0x55a14a3735e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a3735e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 -- 172.21.15.3:0/953174099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 msgr2=0x55a14a36e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a36e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.624+0000 7f9c137fe640 1 -- 172.21.15.3:0/953174099 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a14a26cc90 con 0x55a14a36b3e0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c137fe640 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a36e130 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009830 tx=0x7f9c04041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a57d4708f1a5d49 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9bf97fa640 1 -- 172.21.15.3:0/953174099 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0404f040 con 0x55a14a36b3e0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9bf97fa640 1 -- 172.21.15.3:0/953174099 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c04051040 con 0x55a14a36b3e0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9bf97fa640 1 -- 172.21.15.3:0/953174099 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c040035f0 con 0x55a14a36b3e0 2024-02-09T22:55:25.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 -- 172.21.15.3:0/953174099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 msgr2=0x55a14a36e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a36e130 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f9c04009830 tx=0x7f9c04041040 comp rx=0 tx=0).stop 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 -- 172.21.15.3:0/953174099 shutdown_connections 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a36e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a36e130 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 --2- 172.21.15.3:0/953174099 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a3735e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.625+0000 7f9c1c981740 1 -- 172.21.15.3:0/953174099 >> 172.21.15.3:0/953174099 conn(0x55a14a365100 msgr2=0x55a14a38b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 -- 172.21.15.3:0/953174099 shutdown_connections 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 -- 172.21.15.3:0/953174099 wait complete. 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 -- start start 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 Processor -- start 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a379aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a379fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a37a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a3daac0 con 0x55a14a36b3e0 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a3dac30 con 0x55a14a373f60 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a14a3dadf0 con 0x55a14a36aa30 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c1c981740 10 client.0 _ll_register_callbacks cb 0x7f9c04037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a379fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a379fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59912/0 (socket says 172.21.15.3:59912) 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c137fe640 1 -- 172.21.15.3:0/839184872 learned_addr learned my addr 172.21.15.3:0/839184872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a379aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.626+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a37a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c137fe640 1 -- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 msgr2=0x55a14a379aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c137fe640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a379aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c137fe640 1 -- 172.21.15.3:0/839184872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 msgr2=0x55a14a37a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c137fe640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a37a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c137fe640 1 -- 172.21.15.3:0/839184872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a14a294210 con 0x55a14a36b3e0 2024-02-09T22:55:25.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c137fe640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a379fe0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9c0402e010 tx=0x7f9c0402bc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=340bd6229a004628 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9bf2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/839184872 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0404f040 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c04034920 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/839184872 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9c04004280 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/839184872 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c04004440 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 10 client.17010 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55a14a292a70 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/839184872 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c0406d020 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/839184872 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9c040035f0 con 0x55a14a36b3e0 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 10 client.17010 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 10 client.17010 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:55:25.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 20 client.17010 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.627+0000 7f9c1c981740 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a3dc220 0x55a14a3de640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 4 client.17010 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a14a05d890 con 0x55a14a3dc220 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a3dc220 0x55a14a3de640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9bf2ffd640 10 client.17010 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c12ffd640 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a3dc220 0x55a14a3de640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fd0f5c2a829e397f server_cookie=c55d734fdf0a2a0 in_seq=0 out_seq=0 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9bf2ffd640 1 -- 172.21.15.3:0/839184872 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a14a05d890 con 0x55a14a3dc220 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9bf2ffd640 10 client.17010 handle_command_reply: tid=0 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 client.17010 shutdown 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 msgr2=0x55a14a379fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a379fe0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f9c0402e010 tx=0x7f9c0402bc00 comp rx=0 tx=0).stop 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 shutdown_connections 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55a14a3dc220 0x55a14a3de640 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a14a373f60 0x55a14a37a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a14a36b3e0 0x55a14a379fe0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 --2- 172.21.15.3:0/839184872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a14a36aa30 0x55a14a379aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.628+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 >> 172.21.15.3:0/839184872 conn(0x55a14a365100 msgr2=0x55a14a3639b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9bf2ffd640 0 client.17010 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 shutdown_connections 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c1c981740 1 -- 172.21.15.3:0/839184872 wait complete. 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c1c981740 20 client.17010 trim_cache size 0 max 16384 2024-02-09T22:55:25.638 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:25.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bfc0a2230 msgr2=0x7f9bfc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bfc0a2230 0x7f9bfc0a46f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c09a540 tx=0x7f9c0c009b00 comp rx=0 tx=0).stop 2024-02-09T22:55:25.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 msgr2=0x7f9c14199930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.629+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 0x7f9c14199930 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c00ee80 tx=0x7f9c0c00c740 comp rx=0 tx=0).stop 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 shutdown_connections 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9bfc0a2230 0x7f9bfc0a46f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9c141008f0 0x7f9c1419fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9c1410db80 0x7f9c14199930 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 --2- 172.21.15.3:0/1863324709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9c140fffd0 0x7f9c1419f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 >> 172.21.15.3:0/1863324709 conn(0x7f9c140772e0 msgr2=0x7f9c1410f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 shutdown_connections 2024-02-09T22:55:25.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.630+0000 7f9c19cd6640 1 -- 172.21.15.3:0/1863324709 wait complete. 2024-02-09T22:55:25.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 116416 2.56 MB/sec execute 79 sec latency 11.292 ms 2024-02-09T22:55:25.644 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:25.644 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 -- 172.21.15.3:0/1901809178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 msgr2=0x7f2188100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 --2- 172.21.15.3:0/1901809178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f2188100710 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7f2178009930 tx=0x7f217802bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 -- 172.21.15.3:0/1901809178 shutdown_connections 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 --2- 172.21.15.3:0/1901809178 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2188100ce0 0x7f2188104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 --2- 172.21.15.3:0/1901809178 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2188105400 0x7f218810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 --2- 172.21.15.3:0/1901809178 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f2188100710 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.752+0000 7f218ca1c640 1 -- 172.21.15.3:0/1901809178 >> 172.21.15.3:0/1901809178 conn(0x7f2188077250 msgr2=0x7f2188077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 -- 172.21.15.3:0/1901809178 shutdown_connections 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 -- 172.21.15.3:0/1901809178 wait complete. 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 Processor -- start 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 -- start start 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f21881986a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2188105400 0x7f2188198be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2188199170 0x7f21881a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21881a3df0 con 0x7f2188100330 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21881a3f60 con 0x7f2188105400 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.753+0000 7f218ca1c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21881a4120 con 0x7f2188199170 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f21881986a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f21881986a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59928/0 (socket says 172.21.15.3:59928) 2024-02-09T22:55:25.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 -- 172.21.15.3:0/3069495727 learned_addr learned my addr 172.21.15.3:0/3069495727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f2185d9b640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2188105400 0x7f2188198be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f2186d9d640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2188199170 0x7f21881a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 -- 172.21.15.3:0/3069495727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2188199170 msgr2=0x7f21881a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2188199170 0x7f21881a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 -- 172.21.15.3:0/3069495727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2188105400 msgr2=0x7f2188198be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2188105400 0x7f2188198be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 -- 172.21.15.3:0/3069495727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2178009560 con 0x7f2188100330 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218659c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f21881986a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f2178000c00 tx=0x7f217802cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f21777fe640 1 -- 172.21.15.3:0/3069495727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178004350 con 0x7f2188100330 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f21777fe640 1 -- 172.21.15.3:0/3069495727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21780044b0 con 0x7f2188100330 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21881a43a0 con 0x7f2188100330 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f21777fe640 1 -- 172.21.15.3:0/3069495727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2178003d00 con 0x7f2188100330 2024-02-09T22:55:25.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.754+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21881a48b0 con 0x7f2188100330 2024-02-09T22:55:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.757+0000 7f21777fe640 1 -- 172.21.15.3:0/3069495727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f2178004df0 con 0x7f2188100330 2024-02-09T22:55:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.757+0000 7f218f5f3740 1 Processor -- start 2024-02-09T22:55:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.757+0000 7f218f5f3740 1 -- start start 2024-02-09T22:55:25.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.757+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b5ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.757+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5f3e0 0x55c981b67990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b6a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f218f5f3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981b1a020 con 0x55c981b5ea30 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f218f5f3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981acd320 con 0x55c981b5f3e0 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f218f5f3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981aca850 con 0x55c981b67f60 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f21777fe640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21700a2230 0x7f21700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2186d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b5ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f218659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5f3e0 0x55c981b67990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2186d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b5ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59942/0 (socket says 172.21.15.3:59942) 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2186d9d640 1 -- 172.21.15.3:0/2477535013 learned_addr learned my addr 172.21.15.3:0/2477535013 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b6a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2186d9d640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21700a2230 0x7f21700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 -- 172.21.15.3:0/2477535013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5f3e0 msgr2=0x55c981b67990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5f3e0 0x55c981b67990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f21777fe640 1 -- 172.21.15.3:0/3069495727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f21780f0680 con 0x7f2188100330 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 -- 172.21.15.3:0/2477535013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 msgr2=0x55c981b5ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b5ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 -- 172.21.15.3:0/2477535013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c981a88210 con 0x55c981b67f60 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2186d9d640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21700a2230 0x7f21700a46f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21800099e0 tx=0x7f2180009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2185d9b640 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b6a3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f217c005d00 tx=0x7f217c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e7f2db0e5273649 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2174ff9640 1 -- 172.21.15.3:0/2477535013 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c015850 con 0x55c981b67f60 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2174ff9640 1 -- 172.21.15.3:0/2477535013 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f217c019070 con 0x55c981b67f60 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.758+0000 7f2174ff9640 1 -- 172.21.15.3:0/2477535013 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f217c020dd0 con 0x55c981b67f60 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 -- 172.21.15.3:0/2477535013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 msgr2=0x55c981b6a3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b6a3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f217c005d00 tx=0x7f217c0107a0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 -- 172.21.15.3:0/2477535013 shutdown_connections 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5f3e0 0x55c981b67990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b6a3b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 --2- 172.21.15.3:0/2477535013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b5ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 -- 172.21.15.3:0/2477535013 >> 172.21.15.3:0/2477535013 conn(0x55c981b59100 msgr2=0x55c981b594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 -- 172.21.15.3:0/2477535013 shutdown_connections 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.759+0000 7f218f5f3740 1 -- 172.21.15.3:0/2477535013 wait complete. 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 -- start start 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 Processor -- start 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b64ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5f3e0 0x55c981b65410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b65950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981ac6030 con 0x55c981b5ea30 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981799430 con 0x55c981b67f60 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981755e00 con 0x55c981b5f3e0 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218f5f3740 10 client.0 _ll_register_callbacks cb 0x55c981abb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b64ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b64ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59958/0 (socket says 172.21.15.3:59958) 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 -- 172.21.15.3:0/887471247 learned_addr learned my addr 172.21.15.3:0/887471247 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f2185d9b640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5f3e0 0x55c981b65410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f2186d9d640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b65950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 -- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5f3e0 msgr2=0x55c981b65410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5f3e0 0x55c981b65410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 -- 172.21.15.3:0/887471247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 msgr2=0x55c981b65950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b65950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 -- 172.21.15.3:0/887471247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f217c00b8c0 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f218659c640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b64ed0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f21780edf10 tx=0x7f217808d280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f757ded57a00dd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.760+0000 7f216dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f216dffb640 1 -- 172.21.15.3:0/887471247 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780c1220 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f216dffb640 1 -- 172.21.15.3:0/887471247 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f21780bda00 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c9817c57e0 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f216dffb640 1 -- 172.21.15.3:0/887471247 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21780ee230 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 10 client.17022 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55c981ac46b0 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f216dffb640 1 -- 172.21.15.3:0/887471247 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c9817c57e0 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f216dffb640 1 -- 172.21.15.3:0/887471247 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f217803c070 con 0x55c981b5ea30 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 10 client.17022 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 10 client.17022 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:55:25.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 20 client.17022 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981b780c0 0x55c981b7a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 4 client.17022 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.761+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c981802720 con 0x55c981b780c0 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.762+0000 7f2185d9b640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981b780c0 0x55c981b7a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.762+0000 7f216dffb640 10 client.17022 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.762+0000 7f2185d9b640 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981b780c0 0x55c981b7a4b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d336b8db8e6d2b17 server_cookie=5d109f2668401ab0 in_seq=0 out_seq=0 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f216dffb640 1 -- 172.21.15.3:0/887471247 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c981802720 con 0x55c981b780c0 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f216dffb640 10 client.17022 handle_command_reply: tid=0 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 client.17022 shutdown 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 msgr2=0x55c981b64ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b64ed0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f21780edf10 tx=0x7f217808d280 comp rx=0 tx=0).stop 2024-02-09T22:55:25.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 shutdown_connections 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981b780c0 0x55c981b7a4b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b65950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5f3e0 0x55c981b65410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 --2- 172.21.15.3:0/887471247 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5ea30 0x55c981b64ed0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 >> 172.21.15.3:0/887471247 conn(0x55c981b59100 msgr2=0x55c981b5ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f216dffb640 0 client.17022 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 shutdown_connections 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 1 -- 172.21.15.3:0/887471247 wait complete. 2024-02-09T22:55:25.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.763+0000 7f218f5f3740 20 client.17022 trim_cache size 0 max 16384 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 Processor -- start 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 -- start start 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5ea30 0x55c981b67600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b65770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b7ee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981a4b130 con 0x55c981b5f3e0 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981b73ea0 con 0x55c981b5ea30 2024-02-09T22:55:25.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218f5f3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981b65cb0 con 0x55c981b67f60 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f2185d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b65770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f2185d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b65770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59964/0 (socket says 172.21.15.3:59964) 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f2185d9b640 1 -- 172.21.15.3:0/4184632076 learned_addr learned my addr 172.21.15.3:0/4184632076 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218659c640 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5ea30 0x55c981b67600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f2186d9d640 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b7ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218659c640 1 -- 172.21.15.3:0/4184632076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 msgr2=0x55c981b7ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218659c640 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b7ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.775+0000 7f218659c640 1 -- 172.21.15.3:0/4184632076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 msgr2=0x55c981b65770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f218659c640 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b65770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f218659c640 1 -- 172.21.15.3:0/4184632076 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c981a60c90 con 0x55c981b5ea30 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f2185d9b640 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b65770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f218659c640 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5ea30 0x55c981b67600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21780b9710 tx=0x7f2178045f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef3fafca3f60f5ef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f216ffff640 1 -- 172.21.15.3:0/4184632076 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780ee230 con 0x55c981b5ea30 2024-02-09T22:55:25.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f216ffff640 1 -- 172.21.15.3:0/4184632076 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21780bc840 con 0x55c981b5ea30 2024-02-09T22:55:25.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.776+0000 7f216ffff640 1 -- 172.21.15.3:0/4184632076 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780bd390 con 0x55c981b5ea30 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 -- 172.21.15.3:0/4184632076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5ea30 msgr2=0x55c981b67600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5ea30 0x55c981b67600 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f21780b9710 tx=0x7f2178045f90 comp rx=0 tx=0).stop 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 -- 172.21.15.3:0/4184632076 shutdown_connections 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b67f60 0x55c981b7ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b65770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 --2- 172.21.15.3:0/4184632076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b5ea30 0x55c981b67600 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 -- 172.21.15.3:0/4184632076 >> 172.21.15.3:0/4184632076 conn(0x55c981b59100 msgr2=0x55c981b594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 -- 172.21.15.3:0/4184632076 shutdown_connections 2024-02-09T22:55:25.785 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 -- 172.21.15.3:0/4184632076 wait complete. 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 -- start start 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 Processor -- start 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.777+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 0x55c981b6f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b6d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b6da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981b6df90 con 0x55c981b5f3e0 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981b6e100 con 0x55c981b67f60 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c981b6e270 con 0x55c981b5ea30 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 10 client.0 _ll_register_callbacks cb 0x7f217c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 0x55c981b6f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f2185d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b6d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.786 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 0x55c981b6f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:37124/0 (socket says 172.21.15.3:37124) 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 -- 172.21.15.3:0/1689714880 learned_addr learned my addr 172.21.15.3:0/1689714880 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f2185d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b6d510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59972/0 (socket says 172.21.15.3:59972) 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f2186d9d640 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b6da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 -- 172.21.15.3:0/1689714880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 msgr2=0x55c981b6da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b6da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 -- 172.21.15.3:0/1689714880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 msgr2=0x55c981b6d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b6d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 -- 172.21.15.3:0/1689714880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c981a88210 con 0x55c981b5ea30 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218659c640 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 0x55c981b6f170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f21780bd000 tx=0x7f217802c450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29f4e67ff2becf56 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f216dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f216dffb640 1 -- 172.21.15.3:0/1689714880 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21780bd370 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f217c019070 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.778+0000 7f216dffb640 1 -- 172.21.15.3:0/1689714880 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f21780c5a10 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f216dffb640 1 -- 172.21.15.3:0/1689714880 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2178082ea0 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 10 client.26386 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7f217c00b8c0 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f216dffb640 1 -- 172.21.15.3:0/1689714880 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2178086020 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f216dffb640 1 -- 172.21.15.3:0/1689714880 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f217803c070 con 0x55c981b5ea30 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 10 client.26386 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 10 client.26386 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 20 client.26386 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981bcf600 0x55c981bd19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 4 client.26386 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c981851890 con 0x55c981bcf600 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.779+0000 7f2185d9b640 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981bcf600 0x55c981bd19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f216dffb640 10 client.26386 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f2185d9b640 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981bcf600 0x55c981bd19f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4ad72656bad0abde server_cookie=a4ba222add4d1af in_seq=0 out_seq=0 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f216dffb640 1 -- 172.21.15.3:0/1689714880 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c981851890 con 0x55c981bcf600 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f216dffb640 10 client.26386 handle_command_reply: tid=0 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 client.26386 shutdown 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 msgr2=0x55c981b6f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 0x55c981b6f170 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f21780bd000 tx=0x7f217802c450 comp rx=0 tx=0).stop 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 shutdown_connections 2024-02-09T22:55:25.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55c981bcf600 0x55c981bd19f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c981b67f60 0x55c981b6da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c981b5f3e0 0x55c981b6d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f216dffb640 0 client.26386 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 --2- 172.21.15.3:0/1689714880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c981b5ea30 0x55c981b6f170 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 >> 172.21.15.3:0/1689714880 conn(0x55c981b59100 msgr2=0x55c981b7ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 shutdown_connections 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 1 -- 172.21.15.3:0/1689714880 wait complete. 2024-02-09T22:55:25.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218f5f3740 20 client.26386 trim_cache size 0 max 16384 2024-02-09T22:55:25.791 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:25.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21700a2230 msgr2=0x7f21700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218ca1c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21700a2230 0x7f21700a46f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f21800099e0 tx=0x7f2180009500 comp rx=0 tx=0).stop 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 msgr2=0x7f21881986a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.780+0000 7f218ca1c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f21881986a0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f2178000c00 tx=0x7f217802cd80 comp rx=0 tx=0).stop 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 shutdown_connections 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21700a2230 0x7f21700a46f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2188105400 0x7f2188198be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2188199170 0x7f21881a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 --2- 172.21.15.3:0/3069495727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2188100330 0x7f21881986a0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 >> 172.21.15.3:0/3069495727 conn(0x7f2188077250 msgr2=0x7f21880fe1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 shutdown_connections 2024-02-09T22:55:25.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.781+0000 7f218ca1c640 1 -- 172.21.15.3:0/3069495727 wait complete. 2024-02-09T22:55:25.795 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:25.795 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 -- 172.21.15.3:0/2784247159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148101430 msgr2=0x7ff148101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2784247159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148101430 0x7ff148101810 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7ff138009cd0 tx=0x7ff13802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 -- 172.21.15.3:0/2784247159 shutdown_connections 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2784247159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff148106500 0x7ff14810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2784247159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148101de0 0x7ff148105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2784247159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148101430 0x7ff148101810 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 -- 172.21.15.3:0/2784247159 >> 172.21.15.3:0/2784247159 conn(0x7ff1480fccd0 msgr2=0x7ff1480ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 -- 172.21.15.3:0/2784247159 shutdown_connections 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.903+0000 7ff14f38c640 1 -- 172.21.15.3:0/2784247159 wait complete. 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 Processor -- start 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 -- start start 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff148101430 0x7ff14819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 0x7ff14819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148197740 0x7ff148197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148198210 con 0x7ff148106500 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148198380 con 0x7ff148101430 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff148198540 con 0x7ff148197740 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 0x7ff14819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 0x7ff14819db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59990/0 (socket says 172.21.15.3:59990) 2024-02-09T22:55:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 -- 172.21.15.3:0/2819047050 learned_addr learned my addr 172.21.15.3:0/2819047050 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14d128640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff148101430 0x7ff14819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14d929640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148197740 0x7ff148197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 -- 172.21.15.3:0/2819047050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148197740 msgr2=0x7ff148197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148197740 0x7ff148197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 -- 172.21.15.3:0/2819047050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff148101430 msgr2=0x7ff14819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff148101430 0x7ff14819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 -- 172.21.15.3:0/2819047050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff138009900 con 0x7ff148106500 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14d929640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148197740 0x7ff148197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14c927640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 0x7ff14819db70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00e9f0 tx=0x7ff13c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff1467fc640 1 -- 172.21.15.3:0/2819047050 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c0096c0 con 0x7ff148106500 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff1467fc640 1 -- 172.21.15.3:0/2819047050 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff13c005500 con 0x7ff148106500 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.904+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1481a4490 con 0x7ff148106500 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.905+0000 7ff1467fc640 1 -- 172.21.15.3:0/2819047050 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff13c010430 con 0x7ff148106500 2024-02-09T22:55:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.905+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1481a4970 con 0x7ff148106500 2024-02-09T22:55:25.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.907+0000 7ff1467fc640 1 -- 172.21.15.3:0/2819047050 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff13c005020 con 0x7ff148106500 2024-02-09T22:55:25.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.907+0000 7ff151f64740 1 Processor -- start 2024-02-09T22:55:25.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.907+0000 7ff151f64740 1 -- start start 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff151f64740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4ace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff151f64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff151f64740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff151f64740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd468020 con 0x55f1bd4aca30 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff151f64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd41b320 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff151f64740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd418850 con 0x55f1bd4b5f60 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff1467fc640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1300a22b0 0x7ff1300a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d929640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:33408/0 (socket says 172.21.15.3:33408) 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d929640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60006/0 (socket says 172.21.15.3:60006) 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d929640 1 -- 172.21.15.3:0/1944092765 learned_addr learned my addr 172.21.15.3:0/1944092765 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.918 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d929640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1300a22b0 0x7ff1300a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14c927640 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff1467fc640 1 -- 172.21.15.3:0/2819047050 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff13c014070 con 0x7ff148106500 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 -- 172.21.15.3:0/1944092765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4b5f60 msgr2=0x55f1bd4b83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 -- 172.21.15.3:0/1944092765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 msgr2=0x55f1bd4ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d128640 1 -- 172.21.15.3:0/1944092765 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1bd3d6210 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.908+0000 7ff14d929640 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff14d929640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1300a22b0 0x7ff1300a4770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff1400129b0 tx=0x7ff140005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff14d128640 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b5990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff13802c700 tx=0x7ff138005f70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f3e2f339a230bcc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff12f7fe640 1 -- 172.21.15.3:0/1944092765 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138034440 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.919 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff12f7fe640 1 -- 172.21.15.3:0/1944092765 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1380345a0 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff12f7fe640 1 -- 172.21.15.3:0/1944092765 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138003b70 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff151f64740 1 -- 172.21.15.3:0/1944092765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 msgr2=0x55f1bd4b5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.909+0000 7ff151f64740 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b5990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff13802c700 tx=0x7ff138005f70 comp rx=0 tx=0).stop 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- 172.21.15.3:0/1944092765 shutdown_connections 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b5990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 --2- 172.21.15.3:0/1944092765 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- 172.21.15.3:0/1944092765 >> 172.21.15.3:0/1944092765 conn(0x55f1bd4a7100 msgr2=0x55f1bd4a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- 172.21.15.3:0/1944092765 shutdown_connections 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- 172.21.15.3:0/1944092765 wait complete. 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- start start 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 Processor -- start 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd414030 con 0x55f1bd4b5f60 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd0e7430 con 0x55f1bd4aca30 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd0a3e00 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.910+0000 7ff151f64740 10 client.0 _ll_register_callbacks cb 0x55f1bd409030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d929640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:33412/0 (socket says 172.21.15.3:33412) 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14c927640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 -- 172.21.15.3:0/1735257093 learned_addr learned my addr 172.21.15.3:0/1735257093 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 -- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 msgr2=0x55f1bd4b3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 -- 172.21.15.3:0/1735257093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 msgr2=0x55f1bd4b39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b39c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 -- 172.21.15.3:0/1735257093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff14810e8e0 con 0x55f1bd4aca30 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d929640 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff14d128640 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b2f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff13802c630 tx=0x7ff138032960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49ecb64c7fb72a0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff12cff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff12cff9640 1 -- 172.21.15.3:0/1735257093 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138052070 con 0x55f1bd4aca30 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff12cff9640 1 -- 172.21.15.3:0/1735257093 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff138034ca0 con 0x55f1bd4aca30 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1bd1137e0 con 0x55f1bd4aca30 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff12cff9640 1 -- 172.21.15.3:0/1735257093 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff138034440 con 0x55f1bd4aca30 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff151f64740 10 client.26351 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.911+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55f1bd4126b0 con 0x55f1bd4aca30 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff12cff9640 1 -- 172.21.15.3:0/1735257093 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f1bd1137e0 con 0x55f1bd4aca30 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff12cff9640 1 -- 172.21.15.3:0/1735257093 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff138055030 con 0x55f1bd4aca30 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff151f64740 10 client.26351 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff151f64740 10 client.26351 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff151f64740 20 client.26351 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff151f64740 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd4c60f0 0x55f1bd4c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff151f64740 4 client.26351 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1bd150720 con 0x55f1bd4c60f0 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff14c927640 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd4c60f0 0x55f1bd4c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff12cff9640 10 client.26351 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.912+0000 7ff14c927640 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd4c60f0 0x55f1bd4c84e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eeed19dfe4cae1f7 server_cookie=251b97a7d8990f1 in_seq=0 out_seq=0 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff12cff9640 1 -- 172.21.15.3:0/1735257093 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f1bd150720 con 0x55f1bd4c60f0 2024-02-09T22:55:25.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff12cff9640 10 client.26351 handle_command_reply: tid=0 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 client.26351 shutdown 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 msgr2=0x55f1bd4b2f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b2f40 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ff13802c630 tx=0x7ff138032960 comp rx=0 tx=0).stop 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 shutdown_connections 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd4c60f0 0x55f1bd4c84e0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4b39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 --2- 172.21.15.3:0/1735257093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b2f40 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 >> 172.21.15.3:0/1735257093 conn(0x55f1bd4a7100 msgr2=0x55f1bd4a8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff12cff9640 0 client.26351 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 shutdown_connections 2024-02-09T22:55:25.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 1 -- 172.21.15.3:0/1735257093 wait complete. 2024-02-09T22:55:25.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.913+0000 7ff151f64740 20 client.26351 trim_cache size 0 max 16384 2024-02-09T22:55:25.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 Processor -- start 2024-02-09T22:55:25.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 -- start start 2024-02-09T22:55:25.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4ae250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd4a28b0 con 0x55f1bd4b5f60 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd399130 con 0x55f1bd4aca30 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.925+0000 7ff151f64740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd4a2a40 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d128640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4ae250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14c927640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d128640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:33418/0 (socket says 172.21.15.3:33418) 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d128640 1 -- 172.21.15.3:0/673987327 learned_addr learned my addr 172.21.15.3:0/673987327 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 -- 172.21.15.3:0/673987327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 msgr2=0x55f1bd4b37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 -- 172.21.15.3:0/673987327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 msgr2=0x55f1bd4b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 -- 172.21.15.3:0/673987327 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1bd3aec90 con 0x55f1bd4b5f60 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d128640 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14c927640 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff14d929640 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4ae250 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff14000d710 tx=0x7ff1400096d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7791b83221629758 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff12effd640 1 -- 172.21.15.3:0/673987327 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140025070 con 0x55f1bd4b5f60 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff12effd640 1 -- 172.21.15.3:0/673987327 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff14000dec0 con 0x55f1bd4b5f60 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.926+0000 7ff12effd640 1 -- 172.21.15.3:0/673987327 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff140004e90 con 0x55f1bd4b5f60 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 -- 172.21.15.3:0/673987327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 msgr2=0x55f1bd4ae250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4ae250 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7ff14000d710 tx=0x7ff1400096d0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 -- 172.21.15.3:0/673987327 shutdown_connections 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4ae250 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4b37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 --2- 172.21.15.3:0/673987327 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4aca30 0x55f1bd4b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 -- 172.21.15.3:0/673987327 >> 172.21.15.3:0/673987327 conn(0x55f1bd4a7100 msgr2=0x55f1bd4a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 -- 172.21.15.3:0/673987327 shutdown_connections 2024-02-09T22:55:25.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 -- 172.21.15.3:0/673987327 wait complete. 2024-02-09T22:55:25.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 -- start start 2024-02-09T22:55:25.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.927+0000 7ff151f64740 1 Processor -- start 2024-02-09T22:55:25.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4bb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4bbe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4bc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd51c5a0 con 0x55f1bd4aca30 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd51c6e0 con 0x55f1bd4b5f60 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1bd51c820 con 0x55f1bd4ad3e0 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 10 client.0 _ll_register_callbacks cb 0x7ff140005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4bb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4bb8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60038/0 (socket says 172.21.15.3:60038) 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 -- 172.21.15.3:0/3950451409 learned_addr learned my addr 172.21.15.3:0/3950451409 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14c927640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4bbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d929640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4bc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 -- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 msgr2=0x55f1bd4bbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4bbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 -- 172.21.15.3:0/3950451409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4b5f60 msgr2=0x55f1bd4bc370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4bc370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 -- 172.21.15.3:0/3950451409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1bd4126b0 con 0x55f1bd4aca30 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d929640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff14d128640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4bb8f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff13802c630 tx=0x7ff138009860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c271d33474140ec server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff12cff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff12cff9640 1 -- 172.21.15.3:0/3950451409 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff138052070 con 0x55f1bd4aca30 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff12cff9640 1 -- 172.21.15.3:0/3950451409 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff138004710 con 0x55f1bd4aca30 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.928+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1bd19f890 con 0x55f1bd4aca30 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff12cff9640 1 -- 172.21.15.3:0/3950451409 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff138034440 con 0x55f1bd4aca30 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 10 client.17046 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:25.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7ff13c09d040 con 0x55f1bd4aca30 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff12cff9640 1 -- 172.21.15.3:0/3950451409 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f1bd19f890 con 0x55f1bd4aca30 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff12cff9640 1 -- 172.21.15.3:0/3950451409 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff13805e070 con 0x55f1bd4aca30 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 10 client.17046 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 10 client.17046 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 20 client.17046 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd51db20 0x55f1bd51ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 4 client.17046 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f1bd0bf700 con 0x55f1bd51db20 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.929+0000 7ff14c927640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd51db20 0x55f1bd51ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff12cff9640 10 client.17046 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff14c927640 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd51db20 0x55f1bd51ff40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b65e0d66d821571 server_cookie=d8927c518ff912b2 in_seq=0 out_seq=0 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff12cff9640 1 -- 172.21.15.3:0/3950451409 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f1bd0bf700 con 0x55f1bd51db20 2024-02-09T22:55:25.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff12cff9640 10 client.17046 handle_command_reply: tid=0 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 client.17046 shutdown 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 msgr2=0x55f1bd4bb8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4bb8f0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7ff13802c630 tx=0x7ff138009860 comp rx=0 tx=0).stop 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 shutdown_connections 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f1bd51db20 0x55f1bd51ff40 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1bd4b5f60 0x55f1bd4bc370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff12cff9640 0 client.17046 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1bd4ad3e0 0x55f1bd4bbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 --2- 172.21.15.3:0/3950451409 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1bd4aca30 0x55f1bd4bb8f0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 >> 172.21.15.3:0/3950451409 conn(0x55f1bd4a7100 msgr2=0x55f1bd4ae250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 shutdown_connections 2024-02-09T22:55:25.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 1 -- 172.21.15.3:0/3950451409 wait complete. 2024-02-09T22:55:25.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.930+0000 7ff151f64740 20 client.17046 trim_cache size 0 max 16384 2024-02-09T22:55:25.941 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1300a22b0 msgr2=0x7ff1300a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1300a22b0 0x7ff1300a4770 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff1400129b0 tx=0x7ff140005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 msgr2=0x7ff14819db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 0x7ff14819db70 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7ff13c00e9f0 tx=0x7ff13c00eeb0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 shutdown_connections 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1300a22b0 0x7ff1300a4770 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff148197740 0x7ff148197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff148106500 0x7ff14819db70 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 --2- 172.21.15.3:0/2819047050 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff148101430 0x7ff14819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 >> 172.21.15.3:0/2819047050 conn(0x7ff1480fccd0 msgr2=0x7ff1480fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 shutdown_connections 2024-02-09T22:55:25.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:25.931+0000 7ff14f38c640 1 -- 172.21.15.3:0/2819047050 wait complete. 2024-02-09T22:55:25.944 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:25.945 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:55:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 -- 172.21.15.3:0/3072910189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 msgr2=0x7f015c102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 --2- 172.21.15.3:0/3072910189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c102830 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f0150009900 tx=0x7f015002bcd0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 -- 172.21.15.3:0/3072910189 shutdown_connections 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 --2- 172.21.15.3:0/3072910189 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f015c102d70 0x7f015c10b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 --2- 172.21.15.3:0/3072910189 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f015c10b820 0x7f015c10dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 --2- 172.21.15.3:0/3072910189 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c102830 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 -- 172.21.15.3:0/3072910189 >> 172.21.15.3:0/3072910189 conn(0x7f015c077220 msgr2=0x7f015c077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 -- 172.21.15.3:0/3072910189 shutdown_connections 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.048+0000 7f0162e9d640 1 -- 172.21.15.3:0/3072910189 wait complete. 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 Processor -- start 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 -- start start 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f015c102d70 0x7f015c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f015c10b820 0x7f015c197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c197e10 con 0x7f015c102450 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c197f80 con 0x7f015c102d70 2024-02-09T22:55:26.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0162e9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f015c198140 con 0x7f015c10b820 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60052/0 (socket says 172.21.15.3:60052) 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 -- 172.21.15.3:0/314623955 learned_addr learned my addr 172.21.15.3:0/314623955 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f016143a640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f015c10b820 0x7f015c197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f015bfff640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f015c102d70 0x7f015c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 -- 172.21.15.3:0/314623955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f015c10b820 msgr2=0x7f015c197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f015c10b820 0x7f015c197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 -- 172.21.15.3:0/314623955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f015c102d70 msgr2=0x7f015c19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f015c102d70 0x7f015c19db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.049+0000 7f0160c39640 1 -- 172.21.15.3:0/314623955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0150009560 con 0x7f015c102450 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.050+0000 7f0160c39640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c19d5c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f0150005940 tx=0x7f015002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.050+0000 7f0159ffb640 1 -- 172.21.15.3:0/314623955 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150004350 con 0x7f015c102450 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.050+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f015c1983c0 con 0x7f015c102450 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.050+0000 7f0159ffb640 1 -- 172.21.15.3:0/314623955 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01500044b0 con 0x7f015c102450 2024-02-09T22:55:26.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.050+0000 7f0159ffb640 1 -- 172.21.15.3:0/314623955 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0150003d30 con 0x7f015c102450 2024-02-09T22:55:26.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.050+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f015c1a4540 con 0x7f015c102450 2024-02-09T22:55:26.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.052+0000 7f0159ffb640 1 -- 172.21.15.3:0/314623955 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0150004df0 con 0x7f015c102450 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.052+0000 7f0165a74740 1 Processor -- start 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 -- start start 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0159ffb640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f014c0a2230 0x7f014c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c84097de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 0x55c840986990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f016143a640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f014c0a2230 0x7f014c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c840986f60 0x55c8409893b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c840939020 con 0x55c84097e3e0 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8408ec320 con 0x55c840986f60 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0165a74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8408e9850 con 0x55c84097da30 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0159ffb640 1 -- 172.21.15.3:0/314623955 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f01500f0600 con 0x7f015c102450 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 0x55c840986990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 0x55c840986990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60062/0 (socket says 172.21.15.3:60062) 2024-02-09T22:55:26.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 -- 172.21.15.3:0/3650551554 learned_addr learned my addr 172.21.15.3:0/3650551554 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0160c39640 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c84097de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f016143a640 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c840986f60 0x55c8409893b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 -- 172.21.15.3:0/3650551554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 msgr2=0x55c84097de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c84097de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 -- 172.21.15.3:0/3650551554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c840986f60 msgr2=0x55c8409893b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c840986f60 0x55c8409893b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 -- 172.21.15.3:0/3650551554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8408a7210 con 0x55c84097e3e0 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f016143a640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f014c0a2230 0x7f014c0a46f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f015c06aa10 tx=0x7f0154013ae0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f015bfff640 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 0x55c840986990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0148009c90 tx=0x7f0148011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3a8531811311901 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.053+0000 7f0146ffd640 1 -- 172.21.15.3:0/3650551554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0148009e40 con 0x55c84097e3e0 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0146ffd640 1 -- 172.21.15.3:0/3650551554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01480198f0 con 0x55c84097e3e0 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0146ffd640 1 -- 172.21.15.3:0/3650551554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01480189f0 con 0x55c84097e3e0 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 -- 172.21.15.3:0/3650551554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 msgr2=0x55c840986990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 0x55c840986990 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f0148009c90 tx=0x7f0148011810 comp rx=0 tx=0).stop 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 -- 172.21.15.3:0/3650551554 shutdown_connections 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c840986f60 0x55c8409893b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c84097e3e0 0x55c840986990 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 --2- 172.21.15.3:0/3650551554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c84097de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 -- 172.21.15.3:0/3650551554 >> 172.21.15.3:0/3650551554 conn(0x55c840978100 msgr2=0x55c8409784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 -- 172.21.15.3:0/3650551554 shutdown_connections 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.054+0000 7f0165a74740 1 -- 172.21.15.3:0/3650551554 wait complete. 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 -- start start 2024-02-09T22:55:26.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 Processor -- start 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c840983e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097e3e0 0x55c8409843c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c840984900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8408e5030 con 0x55c840986f60 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8405b8430 con 0x55c84097e3e0 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c840574e00 con 0x55c84097da30 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0165a74740 10 client.0 _ll_register_callbacks cb 0x55c8408da030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f016143a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c840984900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f016143a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c840984900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60064/0 (socket says 172.21.15.3:60064) 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f016143a640 1 -- 172.21.15.3:0/1799786076 learned_addr learned my addr 172.21.15.3:0/1799786076 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f015bfff640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097e3e0 0x55c8409843c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f0160c39640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c840983e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.055+0000 7f016143a640 1 -- 172.21.15.3:0/1799786076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 msgr2=0x55c840983e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f016143a640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c840983e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f016143a640 1 -- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097e3e0 msgr2=0x55c8409843c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f016143a640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097e3e0 0x55c8409843c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f016143a640 1 -- 172.21.15.3:0/1799786076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01480098f0 con 0x55c840986f60 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f0160c39640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c840983e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f016143a640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c840984900 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0154012420 tx=0x7f0154004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1332d65a90629b01 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f013bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f013bfff640 1 -- 172.21.15.3:0/1799786076 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154004340 con 0x55c840986f60 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f013bfff640 1 -- 172.21.15.3:0/1799786076 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f01540044a0 con 0x55c840986f60 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c8405e47e0 con 0x55c840986f60 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f013bfff640 1 -- 172.21.15.3:0/1799786076 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0154004660 con 0x55c840986f60 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f0165a74740 10 client.17058 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55c8408e36b0 con 0x55c840986f60 2024-02-09T22:55:26.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f013bfff640 1 -- 172.21.15.3:0/1799786076 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c8405e47e0 con 0x55c840986f60 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.056+0000 7f013bfff640 1 -- 172.21.15.3:0/1799786076 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0154027060 con 0x55c840986f60 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0165a74740 10 client.17058 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0165a74740 10 client.17058 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0165a74740 20 client.17058 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0165a74740 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409975d0 0x55c8409999f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0165a74740 4 client.17058 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c840621720 con 0x55c8409975d0 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0160c39640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409975d0 0x55c8409999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f013bfff640 10 client.17058 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.057+0000 7f0160c39640 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409975d0 0x55c8409999f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e31751649367afa0 server_cookie=7eaef6c126da22a0 in_seq=0 out_seq=0 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f013bfff640 1 -- 172.21.15.3:0/1799786076 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c840621720 con 0x55c8409975d0 2024-02-09T22:55:26.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f013bfff640 10 client.17058 handle_command_reply: tid=0 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 client.17058 shutdown 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 msgr2=0x55c840984900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c840984900 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f0154012420 tx=0x7f0154004ab0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 shutdown_connections 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409975d0 0x55c8409999f0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097e3e0 0x55c8409843c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c840984900 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f013bfff640 0 client.17058 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 --2- 172.21.15.3:0/1799786076 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097da30 0x55c840983e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 >> 172.21.15.3:0/1799786076 conn(0x55c840978100 msgr2=0x55c840979e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 shutdown_connections 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 1 -- 172.21.15.3:0/1799786076 wait complete. 2024-02-09T22:55:26.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.058+0000 7f0165a74740 20 client.17058 trim_cache size 0 max 16384 2024-02-09T22:55:26.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 Processor -- start 2024-02-09T22:55:26.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 -- start start 2024-02-09T22:55:26.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c8409865b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c840984720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c84097f170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c84086a130 con 0x55c840986f60 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c840621720 con 0x55c84097da30 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0165a74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8405e47e0 con 0x55c84097e3e0 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c84097f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c84097f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60074/0 (socket says 172.21.15.3:60074) 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 -- 172.21.15.3:0/1082406342 learned_addr learned my addr 172.21.15.3:0/1082406342 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f0160c39640 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c8409865b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f015bfff640 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c840984720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 -- 172.21.15.3:0/1082406342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 msgr2=0x55c840984720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c840984720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 -- 172.21.15.3:0/1082406342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 msgr2=0x55c8409865b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c8409865b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 -- 172.21.15.3:0/1082406342 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c84087fc90 con 0x55c840986f60 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.071+0000 7f016143a640 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c84097f170 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f015400f3e0 tx=0x7f0154012150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e1c2695bc145bd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f01467fc640 1 -- 172.21.15.3:0/1082406342 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540206e0 con 0x55c840986f60 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f01467fc640 1 -- 172.21.15.3:0/1082406342 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0154020840 con 0x55c840986f60 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f01467fc640 1 -- 172.21.15.3:0/1082406342 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01540047c0 con 0x55c840986f60 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 -- 172.21.15.3:0/1082406342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 msgr2=0x55c84097f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c84097f170 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f015400f3e0 tx=0x7f0154012150 comp rx=0 tx=0).stop 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 -- 172.21.15.3:0/1082406342 shutdown_connections 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c84097f170 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c840984720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 --2- 172.21.15.3:0/1082406342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c8409865b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 -- 172.21.15.3:0/1082406342 >> 172.21.15.3:0/1082406342 conn(0x55c840978100 msgr2=0x55c8409784e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 -- 172.21.15.3:0/1082406342 shutdown_connections 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 -- 172.21.15.3:0/1082406342 wait complete. 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.072+0000 7f0165a74740 1 -- start start 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 Processor -- start 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c84098c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c84098c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c8409e6b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8409e7050 con 0x55c840986f60 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8409e7190 con 0x55c84097da30 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8409e7350 con 0x55c84097e3e0 2024-02-09T22:55:26.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0165a74740 10 client.0 _ll_register_callbacks cb 0x7f01480097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c8409e6b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c8409e6b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60086/0 (socket says 172.21.15.3:60086) 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f015bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c84098c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f0160c39640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c84098c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 -- 172.21.15.3:0/1033366525 learned_addr learned my addr 172.21.15.3:0/1033366525 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 -- 172.21.15.3:0/1033366525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 msgr2=0x55c84098c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c84098c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 -- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 msgr2=0x55c84098c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c84098c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 -- 172.21.15.3:0/1033366525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8408e36b0 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f016143a640 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c8409e6b10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0154012080 tx=0x7f015401e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13164201e86a7e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.073+0000 7f013bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f013bfff640 1 -- 172.21.15.3:0/1033366525 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0154020690 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f013bfff640 1 -- 172.21.15.3:0/1033366525 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0154003c80 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f01480198f0 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f013bfff640 1 -- 172.21.15.3:0/1033366525 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f01540209b0 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 10 client.17064 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55c8408a5a70 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f013bfff640 1 -- 172.21.15.3:0/1033366525 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f01480198f0 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f013bfff640 1 -- 172.21.15.3:0/1033366525 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0154027070 con 0x55c840986f60 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 10 client.17064 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 10 client.17064 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:55:26.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 20 client.17064 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409ee6b0 0x55c8409f0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 4 client.17064 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c840670890 con 0x55c8409ee6b0 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.074+0000 7f0160c39640 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409ee6b0 0x55c8409f0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f013bfff640 10 client.17064 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0160c39640 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409ee6b0 0x55c8409f0aa0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ce624acff4b57450 server_cookie=764193363d4b32cd in_seq=0 out_seq=0 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f013bfff640 1 -- 172.21.15.3:0/1033366525 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c840670890 con 0x55c8409ee6b0 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f013bfff640 10 client.17064 handle_command_reply: tid=0 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 client.17064 shutdown 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 msgr2=0x55c8409e6b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c8409e6b10 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f0154012080 tx=0x7f015401e000 comp rx=0 tx=0).stop 2024-02-09T22:55:26.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 shutdown_connections 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c8409ee6b0 0x55c8409f0aa0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c840986f60 0x55c8409e6b10 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c84097e3e0 0x55c84098c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f013bfff640 0 client.17064 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 --2- 172.21.15.3:0/1033366525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c84097da30 0x55c84098c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 >> 172.21.15.3:0/1033366525 conn(0x55c840978100 msgr2=0x55c84097f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 shutdown_connections 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 1 -- 172.21.15.3:0/1033366525 wait complete. 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.075+0000 7f0165a74740 20 client.17064 trim_cache size 0 max 16384 2024-02-09T22:55:26.086 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f014c0a2230 msgr2=0x7f014c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f014c0a2230 0x7f014c0a46f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f015c06aa10 tx=0x7f0154013ae0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 msgr2=0x7f015c19d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c19d5c0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f0150005940 tx=0x7f015002c710 comp rx=0 tx=0).stop 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 shutdown_connections 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f014c0a2230 0x7f014c0a46f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f015c102d70 0x7f015c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f015c10b820 0x7f015c197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 --2- 172.21.15.3:0/314623955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f015c102450 0x7f015c19d5c0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 >> 172.21.15.3:0/314623955 conn(0x7f015c077220 msgr2=0x7f015c0fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 shutdown_connections 2024-02-09T22:55:26.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.076+0000 7f0162e9d640 1 -- 172.21.15.3:0/314623955 wait complete. 2024-02-09T22:55:26.090 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:26.090 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/2034509895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 msgr2=0x7f8d78109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/2034509895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d78109870 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009930 tx=0x7f8d6802bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/2034509895 shutdown_connections 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/2034509895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 0x7f8d78103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/2034509895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 0x7f8d781102c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/2034509895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d78109870 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/2034509895 >> 172.21.15.3:0/2034509895 conn(0x7f8d780feea0 msgr2=0x7f8d781012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/2034509895 shutdown_connections 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/2034509895 wait complete. 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 Processor -- start 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.194+0000 7f8d7cd3e640 1 -- start start 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7cd3e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 0x7f8d7819f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7cd3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 0x7f8d7819fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7cd3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d781998d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7cd3e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d78199f20 con 0x7f8d78109490 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7cd3e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7819a090 con 0x7f8d781033f0 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7cd3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7819a250 con 0x7f8d78103db0 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d781998d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d781998d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60108/0 (socket says 172.21.15.3:60108) 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 -- 172.21.15.3:0/3400045271 learned_addr learned my addr 172.21.15.3:0/3400045271 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 0x7f8d7819fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7659c640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 0x7f8d7819f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 -- 172.21.15.3:0/3400045271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 msgr2=0x7f8d7819fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 0x7f8d7819fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 -- 172.21.15.3:0/3400045271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 msgr2=0x7f8d7819f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 0x7f8d7819f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 -- 172.21.15.3:0/3400045271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d68009560 con 0x7f8d78109490 2024-02-09T22:55:26.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d7659c640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 0x7f8d7819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 0x7f8d7819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d781998d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000eaa0 tx=0x7f8d7000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d6f7fe640 1 -- 172.21.15.3:0/3400045271 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7000cdb0 con 0x7f8d78109490 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.195+0000 7f8d6f7fe640 1 -- 172.21.15.3:0/3400045271 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d70005020 con 0x7f8d78109490 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.196+0000 7f8d6f7fe640 1 -- 172.21.15.3:0/3400045271 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d700116b0 con 0x7f8d78109490 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.196+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d7819a530 con 0x7f8d78109490 2024-02-09T22:55:26.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.196+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d781a67c0 con 0x7f8d78109490 2024-02-09T22:55:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.198+0000 7f8d6f7fe640 1 -- 172.21.15.3:0/3400045271 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8d70005520 con 0x7f8d78109490 2024-02-09T22:55:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.198+0000 7f8d7f9eb740 1 Processor -- start 2024-02-09T22:55:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.198+0000 7f8d7f9eb740 1 -- start start 2024-02-09T22:55:26.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.198+0000 7f8d6f7fe640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8d640a2230 0x7f8d640a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.198+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 0x55e947ef9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.198+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f02990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947f02f60 0x55e947f053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8d640a2230 0x7f8d640a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947eb5020 con 0x55e947f02f60 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947e68320 con 0x55e947efa3e0 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947e65850 con 0x55e947ef9a30 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d6f7fe640 1 -- 172.21.15.3:0/3400045271 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8d700ce1d0 con 0x7f8d78109490 2024-02-09T22:55:26.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 0x55e947ef9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 0x55e947ef9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:37274/0 (socket says 172.21.15.3:37274) 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 -- 172.21.15.3:0/4273810491 learned_addr learned my addr 172.21.15.3:0/4273810491 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947f02f60 0x55e947f053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 -- 172.21.15.3:0/4273810491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 msgr2=0x55e947f02990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f02990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 -- 172.21.15.3:0/4273810491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947f02f60 msgr2=0x55e947f053b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947f02f60 0x55e947f053b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 -- 172.21.15.3:0/4273810491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e947e23210 con 0x55e947ef9a30 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8d640a2230 0x7f8d640a46f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8d78101560 tx=0x7f8d60011860 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d7659c640 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 0x55e947ef9e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009930 tx=0x7f8d68004a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f1b15f07a0f1ca3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d6cff9640 1 -- 172.21.15.3:0/4273810491 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6802c7e0 con 0x55e947ef9a30 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d6cff9640 1 -- 172.21.15.3:0/4273810491 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d68004510 con 0x55e947ef9a30 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.199+0000 7f8d6cff9640 1 -- 172.21.15.3:0/4273810491 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d680053b0 con 0x55e947ef9a30 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/4273810491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 msgr2=0x55e947ef9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 0x55e947ef9e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009930 tx=0x7f8d68004a70 comp rx=0 tx=0).stop 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/4273810491 shutdown_connections 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f02990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947f02f60 0x55e947f053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/4273810491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947ef9a30 0x55e947ef9e10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/4273810491 >> 172.21.15.3:0/4273810491 conn(0x55e947ef4100 msgr2=0x55e947ef44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/4273810491 shutdown_connections 2024-02-09T22:55:26.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/4273810491 wait complete. 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.200+0000 7f8d7f9eb740 1 -- start start 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 Processor -- start 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947ef9a30 0x55e947effe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 0x55e947f00390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f008d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947e61030 con 0x55e947efa3e0 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947b34430 con 0x55e947ef9a30 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947af0e00 con 0x55e947f02f60 2024-02-09T22:55:26.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7f9eb740 10 client.0 _ll_register_callbacks cb 0x55e947e56030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 0x55e947f00390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 0x55e947f00390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60126/0 (socket says 172.21.15.3:60126) 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 -- 172.21.15.3:0/3786724282 learned_addr learned my addr 172.21.15.3:0/3786724282 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d7659c640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947ef9a30 0x55e947effe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f008d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 -- 172.21.15.3:0/3786724282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 msgr2=0x55e947f008d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f008d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 -- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947ef9a30 msgr2=0x55e947effe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947ef9a30 0x55e947effe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 -- 172.21.15.3:0/3786724282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d78110bc0 con 0x55e947efa3e0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 0x55e947f00390 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8d60003820 tx=0x7f8d60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f977f8ca81015c10 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.201+0000 7f8d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d5dffb640 1 -- 172.21.15.3:0/3786724282 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d6000fa50 con 0x55e947efa3e0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d5dffb640 1 -- 172.21.15.3:0/3786724282 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8d6000fbb0 con 0x55e947efa3e0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e947b607e0 con 0x55e947efa3e0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d5dffb640 1 -- 172.21.15.3:0/3786724282 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d6000fd70 con 0x55e947efa3e0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 10 client.17076 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55e947e5f6b0 con 0x55e947efa3e0 2024-02-09T22:55:26.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d5dffb640 1 -- 172.21.15.3:0/3786724282 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e947b607e0 con 0x55e947efa3e0 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d5dffb640 1 -- 172.21.15.3:0/3786724282 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8d600305f0 con 0x55e947efa3e0 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 10 client.17076 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 20 client.17076 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f13490 0x55e947f158b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 4 client.17076 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e947b9d720 con 0x55e947f13490 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.202+0000 7f8d7659c640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f13490 0x55e947f158b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.203+0000 7f8d7659c640 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f13490 0x55e947f158b0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92a73ff97a178c98 server_cookie=c7f1f92b1e766503 in_seq=0 out_seq=0 2024-02-09T22:55:26.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.203+0000 7f8d5dffb640 10 client.17076 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.203+0000 7f8d5dffb640 1 -- 172.21.15.3:0/3786724282 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e947b9d720 con 0x55e947f13490 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.203+0000 7f8d5dffb640 10 client.17076 handle_command_reply: tid=0 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.203+0000 7f8d7f9eb740 1 client.17076 shutdown 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 msgr2=0x55e947f00390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 0x55e947f00390 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8d60003820 tx=0x7f8d60004730 comp rx=0 tx=0).stop 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 shutdown_connections 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f13490 0x55e947f158b0 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f008d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d5dffb640 0 client.17076 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947efa3e0 0x55e947f00390 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/3786724282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947ef9a30 0x55e947effe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 >> 172.21.15.3:0/3786724282 conn(0x55e947ef4100 msgr2=0x55e947ef5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 shutdown_connections 2024-02-09T22:55:26.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/3786724282 wait complete. 2024-02-09T22:55:26.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.204+0000 7f8d7f9eb740 20 client.17076 trim_cache size 0 max 16384 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 Processor -- start 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 -- start start 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f02580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f006f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947efd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947de6130 con 0x55e947ef9a30 2024-02-09T22:55:26.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947f0ff00 con 0x55e947efa3e0 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947efba60 con 0x55e947f02f60 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f02580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f02580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60134/0 (socket says 172.21.15.3:60134) 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d7659c640 1 -- 172.21.15.3:0/2356008600 learned_addr learned my addr 172.21.15.3:0/2356008600 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947efd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.216+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d7659c640 1 -- 172.21.15.3:0/2356008600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 msgr2=0x55e947efd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d7659c640 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947efd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d7659c640 1 -- 172.21.15.3:0/2356008600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 msgr2=0x55e947f006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d7659c640 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d7659c640 1 -- 172.21.15.3:0/2356008600 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e947dfbc90 con 0x55e947ef9a30 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d7659c640 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f02580 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c780 tx=0x7f8d68041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31e028421f90b2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d5ffff640 1 -- 172.21.15.3:0/2356008600 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68054070 con 0x55e947ef9a30 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d5ffff640 1 -- 172.21.15.3:0/2356008600 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d68004a70 con 0x55e947ef9a30 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.217+0000 7f8d5ffff640 1 -- 172.21.15.3:0/2356008600 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68050330 con 0x55e947ef9a30 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/2356008600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 msgr2=0x55e947f02580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f02580 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f8d6802c780 tx=0x7f8d68041040 comp rx=0 tx=0).stop 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/2356008600 shutdown_connections 2024-02-09T22:55:26.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947efd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/2356008600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f02580 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/2356008600 >> 172.21.15.3:0/2356008600 conn(0x55e947ef4100 msgr2=0x55e947f1a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/2356008600 shutdown_connections 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/2356008600 wait complete. 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- start start 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 Processor -- start 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f08890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f08dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f09310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947f0a830 con 0x55e947ef9a30 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947f699a0 con 0x55e947efa3e0 2024-02-09T22:55:26.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e947f69b10 con 0x55e947f02f60 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7f9eb740 10 client.0 _ll_register_callbacks cb 0x7f8d68009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f08890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f08890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60140/0 (socket says 172.21.15.3:60140) 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d7659c640 1 -- 172.21.15.3:0/17248394 learned_addr learned my addr 172.21.15.3:0/17248394 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f08dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.218+0000 7f8d76d9d640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f09310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7659c640 1 -- 172.21.15.3:0/17248394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 msgr2=0x55e947f09310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7659c640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f09310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7659c640 1 -- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 msgr2=0x55e947f08dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7659c640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f08dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7659c640 1 -- 172.21.15.3:0/17248394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e947e23210 con 0x55e947ef9a30 2024-02-09T22:55:26.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7659c640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f08890 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000c00 tx=0x7f8d68005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29401f24bc795ba9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d5dffb640 1 -- 172.21.15.3:0/17248394 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d68054040 con 0x55e947ef9a30 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d5dffb640 1 -- 172.21.15.3:0/17248394 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8d6804e070 con 0x55e947ef9a30 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d68004510 con 0x55e947ef9a30 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d5dffb640 1 -- 172.21.15.3:0/17248394 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d68005400 con 0x55e947ef9a30 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7f9eb740 10 client.17082 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55e947e21a70 con 0x55e947ef9a30 2024-02-09T22:55:26.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d5dffb640 1 -- 172.21.15.3:0/17248394 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d68067080 con 0x55e947ef9a30 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.219+0000 7f8d5dffb640 1 -- 172.21.15.3:0/17248394 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8d6803c050 con 0x55e947ef9a30 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 10 client.17082 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 20 client.17082 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f6aec0 0x55e947f6d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 4 client.17082 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e947bec890 con 0x55e947f6aec0 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f6aec0 0x55e947f6d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d5dffb640 10 client.17082 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d6ffff640 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f6aec0 0x55e947f6d2e0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47fadf79aee1160d server_cookie=5eb81007916a2212 in_seq=0 out_seq=0 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d5dffb640 1 -- 172.21.15.3:0/17248394 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e947bec890 con 0x55e947f6aec0 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d5dffb640 10 client.17082 handle_command_reply: tid=0 2024-02-09T22:55:26.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 client.17082 shutdown 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 msgr2=0x55e947f08890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f08890 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f8d68000c00 tx=0x7f8d68005e50 comp rx=0 tx=0).stop 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 shutdown_connections 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e947f6aec0 0x55e947f6d2e0 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e947efa3e0 0x55e947f08dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e947f02f60 0x55e947f09310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 --2- 172.21.15.3:0/17248394 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e947ef9a30 0x55e947f08890 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d5dffb640 0 client.17082 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.220+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 >> 172.21.15.3:0/17248394 conn(0x55e947ef4100 msgr2=0x55e947ef29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 shutdown_connections 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7f9eb740 1 -- 172.21.15.3:0/17248394 wait complete. 2024-02-09T22:55:26.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7f9eb740 20 client.17082 trim_cache size 0 max 16384 2024-02-09T22:55:26.235 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:26.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8d640a2230 msgr2=0x7f8d640a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8d640a2230 0x7f8d640a46f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8d78101560 tx=0x7f8d60011860 comp rx=0 tx=0).stop 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 msgr2=0x7f8d781998d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d781998d0 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000eaa0 tx=0x7f8d7000ef60 comp rx=0 tx=0).stop 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 shutdown_connections 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8d640a2230 0x7f8d640a46f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8d78109490 0x7f8d781998d0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8d78103db0 0x7f8d7819fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 --2- 172.21.15.3:0/3400045271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8d781033f0 0x7f8d7819f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 >> 172.21.15.3:0/3400045271 conn(0x7f8d780feea0 msgr2=0x7f8d780ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 shutdown_connections 2024-02-09T22:55:26.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:26.221+0000 7f8d7cd3e640 1 -- 172.21.15.3:0/3400045271 wait complete. 2024-02-09T22:55:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:26 smithi003 ceph-mon[32678]: pgmap v128: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 543 KiB/s wr, 78 op/s 2024-02-09T22:55:26.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:26 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/220249711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:55:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:26 smithi089 ceph-mon[36665]: pgmap v128: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 543 KiB/s wr, 78 op/s 2024-02-09T22:55:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:26 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/220249711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:55:26.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:26 smithi064 ceph-mon[36638]: pgmap v128: 97 pgs: 97 active+clean; 48 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 543 KiB/s wr, 78 op/s 2024-02-09T22:55:26.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:26 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/220249711' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:55:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 -- 172.21.15.3:0/2221280860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100d40 msgr2=0x7fcd2410cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 --2- 172.21.15.3:0/2221280860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100d40 0x7fcd2410cf80 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009cd0 tx=0x7fcd1c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 -- 172.21.15.3:0/2221280860 shutdown_connections 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 --2- 172.21.15.3:0/2221280860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd24100380 0x7fcd24100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 --2- 172.21.15.3:0/2221280860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100d40 0x7fcd2410cf80 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 --2- 172.21.15.3:0/2221280860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd24105130 0x7fcd24105510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 -- 172.21.15.3:0/2221280860 >> 172.21.15.3:0/2221280860 conn(0x7fcd24076f80 msgr2=0x7fcd24077390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 -- 172.21.15.3:0/2221280860 shutdown_connections 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.617+0000 7fcd298a7640 1 -- 172.21.15.3:0/2221280860 wait complete. 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 Processor -- start 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 -- start start 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 0x7fcd2419c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd24100d40 0x7fcd2419c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd24105130 0x7fcd24196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24196c70 con 0x7fcd24100380 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24196de0 con 0x7fcd24100d40 2024-02-09T22:55:26.623 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd298a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd24196fa0 con 0x7fcd24105130 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 0x7fcd2419c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 0x7fcd2419c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54388/0 (socket says 172.21.15.3:54388) 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 -- 172.21.15.3:0/219309614 learned_addr learned my addr 172.21.15.3:0/219309614 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd24100d40 0x7fcd2419c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd237fe640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd24105130 0x7fcd24196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 -- 172.21.15.3:0/219309614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd24105130 msgr2=0x7fcd24196620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd24105130 0x7fcd24196620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 -- 172.21.15.3:0/219309614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd24100d40 msgr2=0x7fcd2419c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd24100d40 0x7fcd2419c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.618+0000 7fcd22ffd640 1 -- 172.21.15.3:0/219309614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c009900 con 0x7fcd24100380 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.619+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 0x7fcd2419c420 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400d950 tx=0x7fcd1400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.619+0000 7fcd288a5640 1 -- 172.21.15.3:0/219309614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14009a30 con 0x7fcd24100380 2024-02-09T22:55:26.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.619+0000 7fcd288a5640 1 -- 172.21.15.3:0/219309614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd14009b90 con 0x7fcd24100380 2024-02-09T22:55:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.619+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd24197280 con 0x7fcd24100380 2024-02-09T22:55:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.619+0000 7fcd288a5640 1 -- 172.21.15.3:0/219309614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd14004a00 con 0x7fcd24100380 2024-02-09T22:55:26.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.619+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd241a34c0 con 0x7fcd24100380 2024-02-09T22:55:26.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.621+0000 7fcd288a5640 1 -- 172.21.15.3:0/219309614 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fcd14009d00 con 0x7fcd24100380 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.621+0000 7fcd2c482740 1 Processor -- start 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.621+0000 7fcd2c482740 1 -- start start 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd288a5640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd100a2220 0x7fcd100a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 0x5565f9dc7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 0x5565f9dd3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9dd5da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd2c482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9d36a00 con 0x5565f9dc7b90 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd2c482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9d33f30 con 0x5565f9dd39b0 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd2c482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9d2f710 con 0x5565f9dc84b0 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd237fe640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd100a2220 0x7fcd100a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 0x5565f9dc7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 0x5565f9dd3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 0x5565f9dc7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54398/0 (socket says 172.21.15.3:54398) 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 0x5565f9dd3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:45992/0 (socket says 172.21.15.3:45992) 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 -- 172.21.15.3:0/1715310637 learned_addr learned my addr 172.21.15.3:0/1715310637 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd237fe640 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9dd5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd288a5640 1 -- 172.21.15.3:0/219309614 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd140cda70 con 0x7fcd24100380 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 -- 172.21.15.3:0/1715310637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 msgr2=0x5565f9dd3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 0x5565f9dd3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 -- 172.21.15.3:0/1715310637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 msgr2=0x5565f9dd5da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9dd5da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 -- 172.21.15.3:0/1715310637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565f9cf43b0 con 0x5565f9dc7b90 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 0x5565f9dd3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:26.633 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 0x5565f9dc7f70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fcd140a2700 tx=0x7fcd1409a360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfd93b431e217dfe server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd237fe640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd100a2220 0x7fcd100a46e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006010 tx=0x7fcd1c009360 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd19ffb640 1 -- 172.21.15.3:0/1715310637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1409ab90 con 0x5565f9dc7b90 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd19ffb640 1 -- 172.21.15.3:0/1715310637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd1409a490 con 0x5565f9dc7b90 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.622+0000 7fcd19ffb640 1 -- 172.21.15.3:0/1715310637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd140cb470 con 0x5565f9dc7b90 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 -- 172.21.15.3:0/1715310637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 msgr2=0x5565f9dc7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 0x5565f9dc7f70 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fcd140a2700 tx=0x7fcd1409a360 comp rx=0 tx=0).stop 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 -- 172.21.15.3:0/1715310637 shutdown_connections 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9dd5da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc84b0 0x5565f9dd3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1715310637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc7b90 0x5565f9dc7f70 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 -- 172.21.15.3:0/1715310637 >> 172.21.15.3:0/1715310637 conn(0x5565f9dc29d0 msgr2=0x5565f9dc2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 -- 172.21.15.3:0/1715310637 shutdown_connections 2024-02-09T22:55:26.634 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 -- 172.21.15.3:0/1715310637 wait complete. 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.623+0000 7fcd2c482740 1 -- start start 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 Processor -- start 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd0940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc84b0 0x5565f9dd0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 0x5565f9dd13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9cd4b90 con 0x5565f9dd39b0 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9a03250 con 0x5565f9dc84b0 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9cb5200 con 0x5565f9dc7b90 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd2c482740 10 client.0 _ll_register_callbacks cb 0x5565f9ceb5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 0x5565f9dd13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 0x5565f9dd13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54408/0 (socket says 172.21.15.3:54408) 2024-02-09T22:55:26.635 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 -- 172.21.15.3:0/2102170592 learned_addr learned my addr 172.21.15.3:0/2102170592 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc84b0 0x5565f9dd0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 -- 172.21.15.3:0/2102170592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 msgr2=0x5565f9dd0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 -- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc84b0 msgr2=0x5565f9dd0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc84b0 0x5565f9dd0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 -- 172.21.15.3:0/2102170592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd241a34c0 con 0x5565f9dd39b0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd237fe640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 0x5565f9dd13c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009cd0 tx=0x7fcd1c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fa6192c04e7299c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.624+0000 7fcd02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd02ffd640 1 -- 172.21.15.3:0/2102170592 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c032ac0 con 0x5565f9dd39b0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565f9d55ee0 con 0x5565f9dd39b0 2024-02-09T22:55:26.636 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd02ffd640 1 -- 172.21.15.3:0/2102170592 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd1c003970 con 0x5565f9dd39b0 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd02ffd640 1 -- 172.21.15.3:0/2102170592 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd1c0324d0 con 0x5565f9dd39b0 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 10 client.17094 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5565f9d2dd90 con 0x5565f9dd39b0 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd02ffd640 1 -- 172.21.15.3:0/2102170592 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5565f9d55ee0 con 0x5565f9dd39b0 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd02ffd640 1 -- 172.21.15.3:0/2102170592 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcd1c03e040 con 0x5565f9dd39b0 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 10 client.17094 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 10 client.17094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 20 client.17094 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9de3b70 0x5565f9de5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 4 client.17094 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565f9a2f600 con 0x5565f9de3b70 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.625+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9de3b70 0x5565f9de5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.637 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.626+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9de3b70 0x5565f9de5f90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fddffde501af69be server_cookie=d94fddd42f9d4c53 in_seq=0 out_seq=0 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.626+0000 7fcd02ffd640 10 client.17094 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.626+0000 7fcd02ffd640 1 -- 172.21.15.3:0/2102170592 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5565f9a2f600 con 0x5565f9de3b70 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.626+0000 7fcd02ffd640 10 client.17094 handle_command_reply: tid=0 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.626+0000 7fcd2c482740 1 client.17094 shutdown 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 msgr2=0x5565f9dd13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 0x5565f9dd13c0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c009cd0 tx=0x7fcd1c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 shutdown_connections 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9de3b70 0x5565f9de5f90 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc84b0 0x5565f9dd0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dd39b0 0x5565f9dd13c0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd02ffd640 0 client.17094 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 --2- 172.21.15.3:0/2102170592 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.638 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 >> 172.21.15.3:0/2102170592 conn(0x5565f9dc29d0 msgr2=0x5565f9dc4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 shutdown_connections 2024-02-09T22:55:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 1 -- 172.21.15.3:0/2102170592 wait complete. 2024-02-09T22:55:26.639 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.627+0000 7fcd2c482740 20 client.17094 trim_cache size 0 max 16384 2024-02-09T22:55:26.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 117812 2.63 MB/sec execute 80 sec latency 81.755 ms 2024-02-09T22:55:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 Processor -- start 2024-02-09T22:55:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 -- start start 2024-02-09T22:55:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc7b90 0x5565f9dcf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.647 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dd39b0 0x5565f9dc18b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9d55ee0 con 0x5565f9dc84b0 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9dc37d0 con 0x5565f9dc7b90 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd2c482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9de05e0 con 0x5565f9dd39b0 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54416/0 (socket says 172.21.15.3:54416) 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 -- 172.21.15.3:0/403397260 learned_addr learned my addr 172.21.15.3:0/403397260 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 -- 172.21.15.3:0/403397260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dd39b0 msgr2=0x5565f9dc18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd237fe640 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dd39b0 0x5565f9dc18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc7b90 0x5565f9dcf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dd39b0 0x5565f9dc18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 -- 172.21.15.3:0/403397260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc7b90 msgr2=0x5565f9dcf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.648 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc7b90 0x5565f9dcf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd1bfff640 1 -- 172.21.15.3:0/403397260 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565f9cca820 con 0x5565f9dc84b0 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.640+0000 7fcd237fe640 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dd39b0 0x5565f9dc18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd11e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c009bf0 tx=0x7fcd0c010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7479c6627bb97fb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd197fa640 1 -- 172.21.15.3:0/403397260 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c004030 con 0x5565f9dc84b0 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd197fa640 1 -- 172.21.15.3:0/403397260 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd0c004190 con 0x5565f9dc84b0 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd197fa640 1 -- 172.21.15.3:0/403397260 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c01b860 con 0x5565f9dc84b0 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 -- 172.21.15.3:0/403397260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 msgr2=0x5565f9dd11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd11e0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c009bf0 tx=0x7fcd0c010800 comp rx=0 tx=0).stop 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 -- 172.21.15.3:0/403397260 shutdown_connections 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dd39b0 0x5565f9dc18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd11e0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.649 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 --2- 172.21.15.3:0/403397260 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dc7b90 0x5565f9dcf230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 -- 172.21.15.3:0/403397260 >> 172.21.15.3:0/403397260 conn(0x5565f9dc29d0 msgr2=0x5565f9dc11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 -- 172.21.15.3:0/403397260 shutdown_connections 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.641+0000 7fcd2c482740 1 -- 172.21.15.3:0/403397260 wait complete. 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 -- start start 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 Processor -- start 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd9f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9e341a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9e346e0 con 0x5565f9dc84b0 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9e34820 con 0x5565f9dd39b0 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9e349e0 con 0x5565f9dc7b90 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd2c482740 10 client.0 _ll_register_callbacks cb 0x7fcd0c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.650 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd9f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54426/0 (socket says 172.21.15.3:54426) 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 -- 172.21.15.3:0/1749008814 learned_addr learned my addr 172.21.15.3:0/1749008814 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 -- 172.21.15.3:0/1749008814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 msgr2=0x5565f9dd99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 -- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 msgr2=0x5565f9e341a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9e341a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.642+0000 7fcd1bfff640 1 -- 172.21.15.3:0/1749008814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd0c009960 con 0x5565f9dc84b0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd1bfff640 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd9f00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c01e040 tx=0x7fcd0c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e37a17493b4a001 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd02ffd640 1 -- 172.21.15.3:0/1749008814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c003ac0 con 0x5565f9dc84b0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd02ffd640 1 -- 172.21.15.3:0/1749008814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcd0c005030 con 0x5565f9dc84b0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd1409a490 con 0x5565f9dc84b0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd02ffd640 1 -- 172.21.15.3:0/1749008814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd0c0051f0 con 0x5565f9dc84b0 2024-02-09T22:55:26.651 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 10 client.17100 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5565f9cf06f0 con 0x5565f9dc84b0 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd02ffd640 1 -- 172.21.15.3:0/1749008814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd1409a490 con 0x5565f9dc84b0 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd02ffd640 1 -- 172.21.15.3:0/1749008814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcd0c01b680 con 0x5565f9dc84b0 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 10 client.17100 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 10 client.17100 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 20 client.17100 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9e3c0f0 0x5565f9e3e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 4 client.17100 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5565f9a801b0 con 0x5565f9e3c0f0 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.643+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9e3c0f0 0x5565f9e3e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.644+0000 7fcd02ffd640 10 client.17100 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.652 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.644+0000 7fcd22ffd640 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9e3c0f0 0x5565f9e3e510 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355c971dce18c2b5 server_cookie=686f6b61b429a20d in_seq=0 out_seq=0 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.671+0000 7fcd02ffd640 1 -- 172.21.15.3:0/1749008814 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5565f9a801b0 con 0x5565f9e3c0f0 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.671+0000 7fcd02ffd640 10 client.17100 handle_command_reply: tid=0 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.671+0000 7fcd2c482740 1 client.17100 shutdown 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 msgr2=0x5565f9dd9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd9f00 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c01e040 tx=0x7fcd0c009da0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 shutdown_connections 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5565f9e3c0f0 0x5565f9e3e510 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565f9dd39b0 0x5565f9e341a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565f9dc84b0 0x5565f9dd9f00 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd02ffd640 0 client.17100 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 --2- 172.21.15.3:0/1749008814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565f9dc7b90 0x5565f9dd99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 >> 172.21.15.3:0/1749008814 conn(0x5565f9dc29d0 msgr2=0x5565f9dc1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 shutdown_connections 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 1 -- 172.21.15.3:0/1749008814 wait complete. 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd2c482740 20 client.17100 trim_cache size 0 max 16384 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:55:26.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:55:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:55:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd100a2220 msgr2=0x7fcd100a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd298a7640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd100a2220 0x7fcd100a46e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcd1c006010 tx=0x7fcd1c009360 comp rx=0 tx=0).stop 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 msgr2=0x7fcd2419c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd298a7640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 0x7fcd2419c420 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7fcd1400d950 tx=0x7fcd1400de10 comp rx=0 tx=0).stop 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.672+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 shutdown_connections 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcd100a2220 0x7fcd100a46e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcd24100d40 0x7fcd2419c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcd24105130 0x7fcd24196620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 --2- 172.21.15.3:0/219309614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcd24100380 0x7fcd2419c420 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 >> 172.21.15.3:0/219309614 conn(0x7fcd24076f80 msgr2=0x7fcd240fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 shutdown_connections 2024-02-09T22:55:26.678 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:26.673+0000 7fcd298a7640 1 -- 172.21.15.3:0/219309614 wait complete. 2024-02-09T22:55:27.238 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:55:27.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:55:27.239 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:55:27.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/147141225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 msgr2=0x7fd19c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/147141225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10cfc0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fd194009cd0 tx=0x7fd19402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/147141225 shutdown_connections 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/147141225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c100ce0 0x7fd19c104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/147141225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10cfc0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/147141225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd19c100330 0x7fd19c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/147141225 >> 172.21.15.3:0/147141225 conn(0x7fd19c0771d0 msgr2=0x7fd19c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/147141225 shutdown_connections 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.360+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/147141225 wait complete. 2024-02-09T22:55:27.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 Processor -- start 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 -- start start 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd19c100330 0x7fd19c10d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c10e320 0x7fd19c1a5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1a5f50 con 0x7fd19c1053f0 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1a60c0 con 0x7fd19c10e320 2024-02-09T22:55:27.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd1a0aa7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19c1a6280 con 0x7fd19c100330 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54444/0 (socket says 172.21.15.3:54444) 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.361+0000 7fd191d9b640 1 -- 172.21.15.3:0/2080249282 learned_addr learned my addr 172.21.15.3:0/2080249282 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd19c100330 0x7fd19c10d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c10e320 0x7fd19c1a5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd191d9b640 1 -- 172.21.15.3:0/2080249282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd19c100330 msgr2=0x7fd19c10d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd19c100330 0x7fd19c10d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd191d9b640 1 -- 172.21.15.3:0/2080249282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c10e320 msgr2=0x7fd19c1a5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c10e320 0x7fd19c1a5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd191d9b640 1 -- 172.21.15.3:0/2080249282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd194009900 con 0x7fd19c1053f0 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c10e320 0x7fd19c1a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10dde0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fd18400e9c0 tx=0x7fd18400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd193fff640 1 -- 172.21.15.3:0/2080249282 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184009700 con 0x7fd19c1053f0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19c1a6560 con 0x7fd19c1053f0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd193fff640 1 -- 172.21.15.3:0/2080249282 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd18400cd80 con 0x7fd19c1053f0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd193fff640 1 -- 172.21.15.3:0/2080249282 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184010430 con 0x7fd19c1053f0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.362+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19c1a6a70 con 0x7fd19c1053f0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.364+0000 7fd1a3684740 1 Processor -- start 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 -- start start 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd1c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd25150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd27a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cc8a000 con 0x55fa0cd1cab0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cc87530 con 0x55fa0cd25690 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd1a3684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cc82d10 con 0x55fa0cd1c190 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd25150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd25150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54452/0 (socket says 172.21.15.3:54452) 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 -- 172.21.15.3:0/4075817872 learned_addr learned my addr 172.21.15.3:0/4075817872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd1c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd193fff640 1 -- 172.21.15.3:0/2080249282 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd184005020 con 0x7fd19c1053f0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 -- 172.21.15.3:0/4075817872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 msgr2=0x55fa0cd1c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd1c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 -- 172.21.15.3:0/4075817872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 msgr2=0x55fa0cd27a80 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd27a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.365+0000 7fd19a59c640 1 -- 172.21.15.3:0/4075817872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa0cc29210 con 0x55fa0cd1cab0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd19a59c640 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd25150 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00b760 tx=0x7fd18c00bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221d51e95d648ea4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd190d99640 1 -- 172.21.15.3:0/4075817872 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c003dd0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd190d99640 1 -- 172.21.15.3:0/4075817872 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd18c00be70 con 0x55fa0cd1cab0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd190d99640 1 -- 172.21.15.3:0/4075817872 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c011440 con 0x55fa0cd1cab0 2024-02-09T22:55:27.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd193fff640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd1880a2260 0x7fd1880a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd1880a2260 0x7fd1880a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.366+0000 7fd193fff640 1 -- 172.21.15.3:0/2080249282 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd184014070 con 0x7fd19c1053f0 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- 172.21.15.3:0/4075817872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 msgr2=0x55fa0cd25150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd1880a2260 0x7fd1880a4720 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd1940098d0 tx=0x7fd194004a40 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd25150 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fd18c00b760 tx=0x7fd18c00bc20 comp rx=0 tx=0).stop 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- 172.21.15.3:0/4075817872 shutdown_connections 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd27a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd25150 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- 172.21.15.3:0/4075817872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd1c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- 172.21.15.3:0/4075817872 >> 172.21.15.3:0/4075817872 conn(0x55fa0cd168c0 msgr2=0x55fa0cd16ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- 172.21.15.3:0/4075817872 shutdown_connections 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- 172.21.15.3:0/4075817872 wait complete. 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- start start 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 Processor -- start 2024-02-09T22:55:27.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd226c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd22c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd23140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cc27e60 con 0x55fa0cd1cab0 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0c9564b0 con 0x55fa0cd1c190 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cc08670 con 0x55fa0cd25690 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.367+0000 7fd1a3684740 10 client.0 _ll_register_callbacks cb 0x55fa0cc20190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd22c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd22c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54456/0 (socket says 172.21.15.3:54456) 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 -- 172.21.15.3:0/2247888964 learned_addr learned my addr 172.21.15.3:0/2247888964 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd23140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd226c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 -- 172.21.15.3:0/2247888964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 msgr2=0x55fa0cd23140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd23140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 -- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 msgr2=0x55fa0cd226c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd226c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 -- 172.21.15.3:0/2247888964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd18c004370 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd22c00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd1840cbbe0 tx=0x7fd18409a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f97f79905a11e754 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd181ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd181ffb640 1 -- 172.21.15.3:0/2247888964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1840a35c0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd181ffb640 1 -- 172.21.15.3:0/2247888964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd1840052d0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa0c982860 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd181ffb640 1 -- 172.21.15.3:0/2247888964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1840054b0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd1a3684740 10 client.17112 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fa0cc81390 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.368+0000 7fd181ffb640 1 -- 172.21.15.3:0/2247888964 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd184064080 con 0x55fa0cd1cab0 2024-02-09T22:55:27.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd181ffb640 1 -- 172.21.15.3:0/2247888964 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd184066120 con 0x55fa0cd1cab0 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd1a3684740 10 client.17112 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd1a3684740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd1a3684740 20 client.17112 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd357e0 0x55fa0cd37bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd1a3684740 4 client.17112 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa0c9d3410 con 0x55fa0cd357e0 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd357e0 0x55fa0cd37bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd357e0 0x55fa0cd37bd0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f098c1d5b9815d server_cookie=b5fea49c69da729c in_seq=0 out_seq=0 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.369+0000 7fd181ffb640 10 client.17112 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd181ffb640 1 -- 172.21.15.3:0/2247888964 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fa0c9d3410 con 0x55fa0cd357e0 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd181ffb640 10 client.17112 handle_command_reply: tid=0 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 client.17112 shutdown 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 msgr2=0x55fa0cd22c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd22c00 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7fd1840cbbe0 tx=0x7fd18409a840 comp rx=0 tx=0).stop 2024-02-09T22:55:27.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 shutdown_connections 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd357e0 0x55fa0cd37bd0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd23140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd22c00 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2247888964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd226c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd181ffb640 0 client.17112 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 >> 172.21.15.3:0/2247888964 conn(0x55fa0cd168c0 msgr2=0x55fa0cd18610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 shutdown_connections 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 1 -- 172.21.15.3:0/2247888964 wait complete. 2024-02-09T22:55:27.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.370+0000 7fd1a3684740 20 client.17112 trim_cache size 0 max 16384 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 Processor -- start 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 -- start start 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd24df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd1d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd1df10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cd120d0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.385+0000 7fd1a3684740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cd17b20 con 0x55fa0cd25690 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd1a3684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cd3ce80 con 0x55fa0cd1c190 2024-02-09T22:55:27.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd1d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd1d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54466/0 (socket says 172.21.15.3:54466) 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 -- 172.21.15.3:0/3414973399 learned_addr learned my addr 172.21.15.3:0/3414973399 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd19a59c640 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd24df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd1df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 -- 172.21.15.3:0/3414973399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 msgr2=0x55fa0cd24df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd24df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 -- 172.21.15.3:0/3414973399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 msgr2=0x55fa0cd1df10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd1df10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 -- 172.21.15.3:0/3414973399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa0cc1dc70 con 0x55fa0cd1cab0 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd1df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd191d9b640 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd1d9d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd18400c6a0 tx=0x7fd184099330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84f0008f3de0bf8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd183fff640 1 -- 172.21.15.3:0/3414973399 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1840970c0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd183fff640 1 -- 172.21.15.3:0/3414973399 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fa0c982860 con 0x55fa0cd1cab0 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.386+0000 7fd183fff640 1 -- 172.21.15.3:0/3414973399 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd184066210 con 0x55fa0cd1cab0 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- 172.21.15.3:0/3414973399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 msgr2=0x55fa0cd1d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd1d9d0 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd18400c6a0 tx=0x7fd184099330 comp rx=0 tx=0).stop 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- 172.21.15.3:0/3414973399 shutdown_connections 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd25690 0x55fa0cd1df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd1d9d0 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- 172.21.15.3:0/3414973399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd1c190 0x55fa0cd24df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- 172.21.15.3:0/3414973399 >> 172.21.15.3:0/3414973399 conn(0x55fa0cd168c0 msgr2=0x55fa0cd1ce90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- 172.21.15.3:0/3414973399 shutdown_connections 2024-02-09T22:55:27.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- 172.21.15.3:0/3414973399 wait complete. 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- start start 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 Processor -- start 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd86220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd86760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd2ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cd2bff0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cd2c160 con 0x55fa0cd1c190 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa0cd2c320 con 0x55fa0cd25690 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd1a3684740 10 client.0 _ll_register_callbacks cb 0x7fd18c004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd86760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd191d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd86760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54480/0 (socket says 172.21.15.3:54480) 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd191d9b640 1 -- 172.21.15.3:0/2372503277 learned_addr learned my addr 172.21.15.3:0/2372503277 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.387+0000 7fd19ad9d640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd2ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd191d9b640 1 -- 172.21.15.3:0/2372503277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 msgr2=0x55fa0cd2ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd2ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd86220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd191d9b640 1 -- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 msgr2=0x55fa0cd86220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd86220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd191d9b640 1 -- 172.21.15.3:0/2372503277 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa0cc81390 con 0x55fa0cd1cab0 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd191d9b640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd86760 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd18409adb0 tx=0x7fd184004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8c0ae2000d0868b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd181ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd181ffb640 1 -- 172.21.15.3:0/2372503277 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1840970c0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd181ffb640 1 -- 172.21.15.3:0/2372503277 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd1840a35c0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa0cd224f0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd181ffb640 1 -- 172.21.15.3:0/2372503277 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1840a3130 con 0x55fa0cd1cab0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 10 client.17118 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fa0cc43ac0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd181ffb640 1 -- 172.21.15.3:0/2372503277 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fa0cd224f0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd181ffb640 1 -- 172.21.15.3:0/2372503277 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd184099ce0 con 0x55fa0cd1cab0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 10 client.17118 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 20 client.17118 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd8dc20 0x55fa0cd90010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 4 client.17118 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.388+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x7fd18c00be70 con 0x55fa0cd8dc20 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.389+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd8dc20 0x55fa0cd90010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.389+0000 7fd19a59c640 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd8dc20 0x55fa0cd90010 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e99d8063203d8762 server_cookie=3b50557b40f973f in_seq=0 out_seq=0 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.389+0000 7fd181ffb640 10 client.17118 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd181ffb640 1 -- 172.21.15.3:0/2372503277 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7fd18c00be70 con 0x55fa0cd8dc20 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd181ffb640 10 client.17118 handle_command_reply: tid=0 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 client.17118 shutdown 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 msgr2=0x55fa0cd86760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd86760 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd18409adb0 tx=0x7fd184004950 comp rx=0 tx=0).stop 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 shutdown_connections 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa0cd8dc20 0x55fa0cd90010 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd181ffb640 0 client.17118 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa0cd25690 0x55fa0cd2ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa0cd1cab0 0x55fa0cd86760 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 --2- 172.21.15.3:0/2372503277 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa0cd1c190 0x55fa0cd86220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 >> 172.21.15.3:0/2372503277 conn(0x55fa0cd168c0 msgr2=0x55fa0cd1d9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 shutdown_connections 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 1 -- 172.21.15.3:0/2372503277 wait complete. 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.390+0000 7fd1a3684740 20 client.17118 trim_cache size 0 max 16384 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "6326f8ba-11eb-4c2d-9cbd-78f9237fdba8", 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:55:27.403 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:55:27.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd1880a2260 msgr2=0x7fd1880a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd1880a2260 0x7fd1880a4720 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd1940098d0 tx=0x7fd194004a40 comp rx=0 tx=0).stop 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 msgr2=0x7fd19c10dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10dde0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fd18400e9c0 tx=0x7fd18400ee80 comp rx=0 tx=0).stop 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 shutdown_connections 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd1880a2260 0x7fd1880a4720 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd19c10e320 0x7fd19c1a5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd19c1053f0 0x7fd19c10dde0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 --2- 172.21.15.3:0/2080249282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd19c100330 0x7fd19c10d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 >> 172.21.15.3:0/2080249282 conn(0x7fd19c0771d0 msgr2=0x7fd19c0fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 shutdown_connections 2024-02-09T22:55:27.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.391+0000 7fd1a0aa7640 1 -- 172.21.15.3:0/2080249282 wait complete. 2024-02-09T22:55:27.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:55:27.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 -- 172.21.15.3:0/3750719719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa584105410 msgr2=0x7fa58410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 --2- 172.21.15.3:0/3750719719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa584105410 0x7fa58410cfe0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7fa574009cd0 tx=0x7fa57402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 -- 172.21.15.3:0/3750719719 shutdown_connections 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 --2- 172.21.15.3:0/3750719719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa584105410 0x7fa58410cfe0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 --2- 172.21.15.3:0/3750719719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100cf0 0x7fa584104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 --2- 172.21.15.3:0/3750719719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa584100340 0x7fa584100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 -- 172.21.15.3:0/3750719719 >> 172.21.15.3:0/3750719719 conn(0x7fa5840772e0 msgr2=0x7fa5840776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 -- 172.21.15.3:0/3750719719 shutdown_connections 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 -- 172.21.15.3:0/3750719719 wait complete. 2024-02-09T22:55:27.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 Processor -- start 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.519+0000 7fa58b593640 1 -- start start 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100340 0x7fa58419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa584105410 0x7fa58419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5841976b0 0x7fa584197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa584198180 con 0x7fa5841976b0 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5841982f0 con 0x7fa584105410 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5841984b0 con 0x7fa584100340 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5841976b0 0x7fa584197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58932f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100340 0x7fa58419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58932f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100340 0x7fa58419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46110/0 (socket says 172.21.15.3:46110) 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58932f640 1 -- 172.21.15.3:0/867969208 learned_addr learned my addr 172.21.15.3:0/867969208 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa588b2e640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa584105410 0x7fa58419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 -- 172.21.15.3:0/867969208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100340 msgr2=0x7fa58419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100340 0x7fa58419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 -- 172.21.15.3:0/867969208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa584105410 msgr2=0x7fa58419db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa584105410 0x7fa58419db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 -- 172.21.15.3:0/867969208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa574009900 con 0x7fa5841976b0 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa589b30640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5841976b0 0x7fa584197b30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa57402c630 tx=0x7fa574034a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa5827fc640 1 -- 172.21.15.3:0/867969208 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5740328b0 con 0x7fa5841976b0 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5841a43f0 con 0x7fa5841976b0 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa5827fc640 1 -- 172.21.15.3:0/867969208 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa574003e50 con 0x7fa5841976b0 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa5827fc640 1 -- 172.21.15.3:0/867969208 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574004860 con 0x7fa5841976b0 2024-02-09T22:55:27.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.520+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5841a4870 con 0x7fa5841976b0 2024-02-09T22:55:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.523+0000 7fa5827fc640 1 -- 172.21.15.3:0/867969208 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa57400a040 con 0x7fa5841976b0 2024-02-09T22:55:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.523+0000 7fa5827fc640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa56c0a2230 0x7fa56c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.523+0000 7fa58e16b740 1 Processor -- start 2024-02-09T22:55:27.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.523+0000 7fa5827fc640 1 -- 172.21.15.3:0/867969208 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa5740f0b00 con 0x7fa5841976b0 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.523+0000 7fa58932f640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa56c0a2230 0x7fa56c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 -- start start 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ed5d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed6330 0x563f58ede8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58edeeb0 0x563f58ee1300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58e90d00 con 0x563f58edeeb0 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58e452f0 con 0x563f58ed6330 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58e16b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58e42820 con 0x563f58ed5980 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ed5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ed5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46120/0 (socket says 172.21.15.3:46120) 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 -- 172.21.15.3:0/2925676659 learned_addr learned my addr 172.21.15.3:0/2925676659 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa589b30640 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58edeeb0 0x563f58ee1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa56c0a2230 0x7fa56c0a46f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009ce0 tx=0x7fa57c005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa588b2e640 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed6330 0x563f58ede8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 -- 172.21.15.3:0/2925676659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed6330 msgr2=0x563f58ede8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed6330 0x563f58ede8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 -- 172.21.15.3:0/2925676659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58edeeb0 msgr2=0x563f58ee1300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58edeeb0 0x563f58ee1300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 -- 172.21.15.3:0/2925676659 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f58dffff0 con 0x563f58ed5980 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.524+0000 7fa58932f640 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ed5d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009fa0 tx=0x7fa57c014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb104d124a2a4074 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.525+0000 7fa56b7fe640 1 -- 172.21.15.3:0/2925676659 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c01dc30 con 0x563f58ed5980 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.525+0000 7fa56b7fe640 1 -- 172.21.15.3:0/2925676659 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa57c01dd90 con 0x563f58ed5980 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.525+0000 7fa56b7fe640 1 -- 172.21.15.3:0/2925676659 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c0253f0 con 0x563f58ed5980 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- 172.21.15.3:0/2925676659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 msgr2=0x563f58ed5d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ed5d60 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009fa0 tx=0x7fa57c014000 comp rx=0 tx=0).stop 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- 172.21.15.3:0/2925676659 shutdown_connections 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed6330 0x563f58ede8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58edeeb0 0x563f58ee1300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- 172.21.15.3:0/2925676659 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ed5d60 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- 172.21.15.3:0/2925676659 >> 172.21.15.3:0/2925676659 conn(0x563f58ed0050 msgr2=0x563f58ed0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- 172.21.15.3:0/2925676659 shutdown_connections 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- 172.21.15.3:0/2925676659 wait complete. 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- start start 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 Processor -- start 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed5980 0x563f58edbdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58edc310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58edeeb0 0x563f58edc850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58e3e000 con 0x563f58ed6330 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.526+0000 7fa58e16b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58b11430 con 0x563f58ed5980 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa58e16b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58acde00 con 0x563f58edeeb0 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa58e16b740 10 client.0 _ll_register_callbacks cb 0x563f58e327f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:27.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa588b2e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58edc310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58edeeb0 0x563f58edc850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa58932f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed5980 0x563f58edbdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa588b2e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58edc310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54532/0 (socket says 172.21.15.3:54532) 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa588b2e640 1 -- 172.21.15.3:0/13164370 learned_addr learned my addr 172.21.15.3:0/13164370 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 -- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed5980 msgr2=0x563f58edbdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed5980 0x563f58edbdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 -- 172.21.15.3:0/13164370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 msgr2=0x563f58edc310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58edc310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 -- 172.21.15.3:0/13164370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57c017050 con 0x563f58edeeb0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa588b2e640 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58edc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa589b30640 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58edeeb0 0x563f58edc850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa5740ee390 tx=0x7fa5740bd2e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=833f4ee4d5ae0db8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa568ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa568ff9640 1 -- 172.21.15.3:0/13164370 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5740c1780 con 0x563f58edeeb0 2024-02-09T22:55:27.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa568ff9640 1 -- 172.21.15.3:0/13164370 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa5740c5dd0 con 0x563f58edeeb0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.527+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f58b3d7e0 con 0x563f58edeeb0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa568ff9640 1 -- 172.21.15.3:0/13164370 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5740d6150 con 0x563f58edeeb0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 10 client.26485 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x563f58e3c680 con 0x563f58edeeb0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa568ff9640 1 -- 172.21.15.3:0/13164370 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563f58b3d7e0 con 0x563f58edeeb0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa568ff9640 1 -- 172.21.15.3:0/13164370 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa5740c5940 con 0x563f58edeeb0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 10 client.26485 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 10 client.26485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 20 client.26485 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58ef14b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 4 client.26485 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f58b7a720 con 0x563f58eef0c0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.528+0000 7fa58932f640 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58ef14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.529+0000 7fa568ff9640 10 client.26485 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.529+0000 7fa58932f640 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58ef14b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a6764c38a7387bf server_cookie=70903a1fd770e8f5 in_seq=0 out_seq=0 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.529+0000 7fa568ff9640 1 -- 172.21.15.3:0/13164370 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563f58b7a720 con 0x563f58eef0c0 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.529+0000 7fa568ff9640 10 client.26485 handle_command_reply: tid=0 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.529+0000 7fa58e16b740 1 client.26485 shutdown 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58edeeb0 msgr2=0x563f58edc850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58edeeb0 0x563f58edc850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa5740ee390 tx=0x7fa5740bd2e0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 shutdown_connections 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58ef14b0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58edeeb0 0x563f58edc850 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa568ff9640 0 client.26485 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58edc310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 --2- 172.21.15.3:0/13164370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58ed5980 0x563f58edbdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 >> 172.21.15.3:0/13164370 conn(0x563f58ed0050 msgr2=0x563f58ed1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 shutdown_connections 2024-02-09T22:55:27.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 1 -- 172.21.15.3:0/13164370 wait complete. 2024-02-09T22:55:27.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.530+0000 7fa58e16b740 20 client.26485 trim_cache size 0 max 16384 2024-02-09T22:55:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 Processor -- start 2024-02-09T22:55:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 -- start start 2024-02-09T22:55:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ecf070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58ee5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58ee6420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58dc2fd0 con 0x563f58ed6330 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58ee17c0 con 0x563f58edeeb0 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.547+0000 7fa58e16b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58ea5b00 con 0x563f58ed5980 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58ee5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58ee5ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54548/0 (socket says 172.21.15.3:54548) 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 -- 172.21.15.3:0/3273052811 learned_addr learned my addr 172.21.15.3:0/3273052811 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa589b30640 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58ee6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa58932f640 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ecf070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 -- 172.21.15.3:0/3273052811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 msgr2=0x563f58ecf070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ecf070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 -- 172.21.15.3:0/3273052811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 msgr2=0x563f58ee6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58ee6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 -- 172.21.15.3:0/3273052811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f58dd8b30 con 0x563f58ed6330 2024-02-09T22:55:27.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa588b2e640 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58ee5ee0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa5780125c0 tx=0x7fa578005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9fb16e67f794ba server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa56affd640 1 -- 172.21.15.3:0/3273052811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578020070 con 0x563f58ed6330 2024-02-09T22:55:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa56affd640 1 -- 172.21.15.3:0/3273052811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa578018040 con 0x563f58ed6330 2024-02-09T22:55:27.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.548+0000 7fa56affd640 1 -- 172.21.15.3:0/3273052811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa578017050 con 0x563f58ed6330 2024-02-09T22:55:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 -- 172.21.15.3:0/3273052811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 msgr2=0x563f58ee5ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58ee5ee0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fa5780125c0 tx=0x7fa578005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 -- 172.21.15.3:0/3273052811 shutdown_connections 2024-02-09T22:55:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58ee6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed6330 0x563f58ee5ee0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 --2- 172.21.15.3:0/3273052811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed5980 0x563f58ecf070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 -- 172.21.15.3:0/3273052811 >> 172.21.15.3:0/3273052811 conn(0x563f58ed0050 msgr2=0x563f58ece960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 -- 172.21.15.3:0/3273052811 shutdown_connections 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 -- 172.21.15.3:0/3273052811 wait complete. 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 -- start start 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.549+0000 7fa58e16b740 1 Processor -- start 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 0x563f58ee5260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed6330 0x563f58ee57a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58f3f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58f3fae0 con 0x563f58ed5980 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58f3fc20 con 0x563f58edeeb0 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f58f3fde0 con 0x563f58ed6330 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58e16b740 10 client.0 _ll_register_callbacks cb 0x7fa57c0099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:27.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 0x563f58ee5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 0x563f58ee5260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54562/0 (socket says 172.21.15.3:54562) 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 -- 172.21.15.3:0/656708129 learned_addr learned my addr 172.21.15.3:0/656708129 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa588b2e640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed6330 0x563f58ee57a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa589b30640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58f3f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 -- 172.21.15.3:0/656708129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed6330 msgr2=0x563f58ee57a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed6330 0x563f58ee57a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 -- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 msgr2=0x563f58f3f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58f3f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 -- 172.21.15.3:0/656708129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa578009e20 con 0x563f58ed5980 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa58932f640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 0x563f58ee5260 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009770 tx=0x7fa57c009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483072302870e285 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.550+0000 7fa568ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa568ff9640 1 -- 172.21.15.3:0/656708129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57c036040 con 0x563f58ed5980 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa57c01dd90 con 0x563f58ed5980 2024-02-09T22:55:27.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa568ff9640 1 -- 172.21.15.3:0/656708129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa57c004740 con 0x563f58ed5980 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa568ff9640 1 -- 172.21.15.3:0/656708129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa57c004900 con 0x563f58ed5980 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 10 client.17136 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x563f58dfe850 con 0x563f58ed5980 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa568ff9640 1 -- 172.21.15.3:0/656708129 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa57c04a020 con 0x563f58ed5980 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa568ff9640 1 -- 172.21.15.3:0/656708129 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa57c0305c0 con 0x563f58ed5980 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 10 client.17136 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 20 client.17136 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58f496a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 4 client.17136 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563f58bc9890 con 0x563f58eef0c0 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.551+0000 7fa588b2e640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58f496a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:27.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa588b2e640 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58f496a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f98e95961a0399 server_cookie=495f067c55caa5b3 in_seq=0 out_seq=0 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa568ff9640 10 client.17136 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa568ff9640 1 -- 172.21.15.3:0/656708129 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563f58bc9890 con 0x563f58eef0c0 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa568ff9640 10 client.17136 handle_command_reply: tid=0 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 client.17136 shutdown 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 msgr2=0x563f58ee5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 0x563f58ee5260 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009770 tx=0x7fa57c009700 comp rx=0 tx=0).stop 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 shutdown_connections 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f58eef0c0 0x563f58f496a0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f58edeeb0 0x563f58f3f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f58ed6330 0x563f58ee57a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 --2- 172.21.15.3:0/656708129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f58ed5980 0x563f58ee5260 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 >> 172.21.15.3:0/656708129 conn(0x563f58ed0050 msgr2=0x563f58ecf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa568ff9640 0 client.17136 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 shutdown_connections 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 1 -- 172.21.15.3:0/656708129 wait complete. 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.552+0000 7fa58e16b740 20 client.17136 trim_cache size 0 max 16384 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: "6326f8ba-11eb-4c2d-9cbd-78f9237fdba8": { 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "6326f8ba-11eb-4c2d-9cbd-78f9237fdba8", 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:55:27.562 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:55:27.563 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:55:27.563 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:55:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa56c0a2230 msgr2=0x7fa56c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa56c0a2230 0x7fa56c0a46f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa57c009ce0 tx=0x7fa57c005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:27.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5841976b0 msgr2=0x7fa584197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5841976b0 0x7fa584197b30 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fa57402c630 tx=0x7fa574034a90 comp rx=0 tx=0).stop 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 shutdown_connections 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa56c0a2230 0x7fa56c0a46f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa584105410 0x7fa58419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5841976b0 0x7fa584197b30 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 --2- 172.21.15.3:0/867969208 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa584100340 0x7fa58419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 >> 172.21.15.3:0/867969208 conn(0x7fa5840772e0 msgr2=0x7fa5840fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 shutdown_connections 2024-02-09T22:55:27.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:27.553+0000 7fa58b593640 1 -- 172.21.15.3:0/867969208 wait complete. 2024-02-09T22:55:27.567 INFO:tasks.cephfs.filesystem:scrub status for tag:6326f8ba-11eb-4c2d-9cbd-78f9237fdba8 - {'path': '/', 'tag': '6326f8ba-11eb-4c2d-9cbd-78f9237fdba8', 'options': 'recursive,force'} 2024-02-09T22:55:27.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 118226 2.62 MB/sec execute 81 sec latency 870.230 ms 2024-02-09T22:55:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: pgmap v129: 97 pgs: 97 active+clean; 52 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 895 KiB/s wr, 137 op/s 2024-02-09T22:55:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:55:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:28 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: pgmap v129: 97 pgs: 97 active+clean; 52 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 895 KiB/s wr, 137 op/s 2024-02-09T22:55:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:55:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:28 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 120874 2.79 MB/sec execute 82 sec latency 50.204 ms 2024-02-09T22:55:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: pgmap v129: 97 pgs: 97 active+clean; 52 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 895 KiB/s wr, 137 op/s 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:55:28.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:28.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:28 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:55:29.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 121424 2.77 MB/sec execute 83 sec latency 464.563 ms 2024-02-09T22:55:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:30 smithi089 ceph-mon[36665]: pgmap v130: 97 pgs: 97 active+clean; 52 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 826 KiB/s wr, 112 op/s 2024-02-09T22:55:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:30 smithi003 ceph-mon[32678]: pgmap v130: 97 pgs: 97 active+clean; 52 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 826 KiB/s wr, 112 op/s 2024-02-09T22:55:30.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 121424 2.73 MB/sec execute 84 sec latency 1464.600 ms 2024-02-09T22:55:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:30 smithi064 ceph-mon[36638]: pgmap v130: 97 pgs: 97 active+clean; 52 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 826 KiB/s wr, 112 op/s 2024-02-09T22:55:31.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 121424 2.70 MB/sec execute 85 sec latency 2464.709 ms 2024-02-09T22:55:32.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:32 smithi089 ceph-mon[36665]: pgmap v131: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-02-09T22:55:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:32 smithi003 ceph-mon[32678]: pgmap v131: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-02-09T22:55:32.634 INFO:tasks.workunit.client.0.smithi003.stdout: 1 121424 2.67 MB/sec execute 86 sec latency 3464.760 ms 2024-02-09T22:55:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:32 smithi064 ceph-mon[36638]: pgmap v131: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-02-09T22:55:33.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 123040 2.72 MB/sec execute 87 sec latency 3880.176 ms 2024-02-09T22:55:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:34 smithi089 ceph-mon[36665]: pgmap v132: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:34.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:34 smithi003 ceph-mon[32678]: pgmap v132: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:34.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124736 2.82 MB/sec execute 88 sec latency 11.462 ms 2024-02-09T22:55:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:34 smithi064 ceph-mon[36638]: pgmap v132: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:35.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124905 2.79 MB/sec execute 89 sec latency 896.100 ms 2024-02-09T22:55:36.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:36 smithi089 ceph-mon[36665]: pgmap v133: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:36 smithi003 ceph-mon[32678]: pgmap v133: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:36.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124905 2.76 MB/sec execute 90 sec latency 1896.190 ms 2024-02-09T22:55:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:36 smithi064 ceph-mon[36638]: pgmap v133: 97 pgs: 97 active+clean; 52 MiB data, 760 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 168 op/s 2024-02-09T22:55:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:37.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124905 2.73 MB/sec execute 91 sec latency 2896.294 ms 2024-02-09T22:55:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:38 smithi003 ceph-mon[32678]: pgmap v134: 97 pgs: 97 active+clean; 54 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-02-09T22:55:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:38 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:55:38.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124973 2.70 MB/sec execute 92 sec latency 3304.511 ms 2024-02-09T22:55:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:38 smithi064 ceph-mon[36638]: pgmap v134: 97 pgs: 97 active+clean; 54 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-02-09T22:55:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:38 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:55:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:38 smithi089 ceph-mon[36665]: pgmap v134: 97 pgs: 97 active+clean; 54 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.6 MiB/s wr, 221 op/s 2024-02-09T22:55:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:38 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:55:39.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124973 2.67 MB/sec execute 93 sec latency 1540.732 ms 2024-02-09T22:55:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 -- 172.21.15.3:0/2335462166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 msgr2=0x7fc69410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.772 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 --2- 172.21.15.3:0/2335462166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc69410e0b0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009cd0 tx=0x7fc68c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 -- 172.21.15.3:0/2335462166 shutdown_connections 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 --2- 172.21.15.3:0/2335462166 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 0x7fc6940ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 --2- 172.21.15.3:0/2335462166 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc69410e0b0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 --2- 172.21.15.3:0/2335462166 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc694106220 0x7fc694106600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 -- 172.21.15.3:0/2335462166 >> 172.21.15.3:0/2335462166 conn(0x7fc694076650 msgr2=0x7fc6940fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 -- 172.21.15.3:0/2335462166 shutdown_connections 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.769+0000 7fc699963640 1 -- 172.21.15.3:0/2335462166 wait complete. 2024-02-09T22:55:39.773 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 Processor -- start 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 -- start start 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 0x7fc6941008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc694100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc694106220 0x7fc694101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6941a5d30 con 0x7fc694100080 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6941a5ea0 con 0x7fc6940ff6c0 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc699963640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6941a6060 con 0x7fc694106220 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc6927fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc694100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.775 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc6927fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc694100de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60380/0 (socket says 172.21.15.3:60380) 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc6927fc640 1 -- 172.21.15.3:0/269515478 learned_addr learned my addr 172.21.15.3:0/269515478 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.770+0000 7fc692ffd640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 0x7fc6941008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6937fe640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc694106220 0x7fc694101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6927fc640 1 -- 172.21.15.3:0/269515478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc694106220 msgr2=0x7fc694101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6927fc640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc694106220 0x7fc694101320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6927fc640 1 -- 172.21.15.3:0/269515478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 msgr2=0x7fc6941008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6927fc640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 0x7fc6941008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6927fc640 1 -- 172.21.15.3:0/269515478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c009900 con 0x7fc694100080 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc692ffd640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 0x7fc6941008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc6927fc640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc694100de0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d950 tx=0x7fc68800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc698961640 1 -- 172.21.15.3:0/269515478 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688009c30 con 0x7fc694100080 2024-02-09T22:55:39.776 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6941a6340 con 0x7fc694100080 2024-02-09T22:55:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc698961640 1 -- 172.21.15.3:0/269515478 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc688009d90 con 0x7fc694100080 2024-02-09T22:55:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc698961640 1 -- 172.21.15.3:0/269515478 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68800f7e0 con 0x7fc694100080 2024-02-09T22:55:39.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.771+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6941a68a0 con 0x7fc694100080 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.773+0000 7fc698961640 1 -- 172.21.15.3:0/269515478 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc688010ad0 con 0x7fc694100080 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 Processor -- start 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 -- start start 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc698961640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc67c0a2220 0x7fc67c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a080b0 0x560450a08490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 0x560450a11260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a13b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450976a00 con 0x560450a089d0 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450973f30 con 0x560450a117a0 2024-02-09T22:55:39.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc69c60c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56045096f710 con 0x560450a080b0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6937fe640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc67c0a2220 0x7fc67c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc698961640 1 -- 172.21.15.3:0/269515478 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc6880cde40 con 0x7fc694100080 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 0x560450a11260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 0x560450a11260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60394/0 (socket says 172.21.15.3:60394) 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 -- 172.21.15.3:0/2554459645 learned_addr learned my addr 172.21.15.3:0/2554459645 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6937fe640 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a13b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc692ffd640 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a080b0 0x560450a08490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 -- 172.21.15.3:0/2554459645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a080b0 msgr2=0x560450a08490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a080b0 0x560450a08490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 -- 172.21.15.3:0/2554459645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 msgr2=0x560450a13b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a13b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.774+0000 7fc6927fc640 1 -- 172.21.15.3:0/2554459645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604509343b0 con 0x560450a089d0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc6927fc640 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 0x560450a11260 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc688000c00 tx=0x7fc688096bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49514dbd4c49816f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:39.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc6937fe640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc67c0a2220 0x7fc67c0a46e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc68c005e30 tx=0x7fc68c005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc679ffb640 1 -- 172.21.15.3:0/2554459645 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc688022b90 con 0x560450a089d0 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc679ffb640 1 -- 172.21.15.3:0/2554459645 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc68809af00 con 0x560450a089d0 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc679ffb640 1 -- 172.21.15.3:0/2554459645 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6880a2fa0 con 0x560450a089d0 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 -- 172.21.15.3:0/2554459645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 msgr2=0x560450a11260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 0x560450a11260 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fc688000c00 tx=0x7fc688096bc0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 -- 172.21.15.3:0/2554459645 shutdown_connections 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a13b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a089d0 0x560450a11260 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 --2- 172.21.15.3:0/2554459645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a080b0 0x560450a08490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.775+0000 7fc69c60c740 1 -- 172.21.15.3:0/2554459645 >> 172.21.15.3:0/2554459645 conn(0x560450a029d0 msgr2=0x560450a02db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 -- 172.21.15.3:0/2554459645 shutdown_connections 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 -- 172.21.15.3:0/2554459645 wait complete. 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 -- start start 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 Processor -- start 2024-02-09T22:55:39.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a0e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a0f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450914b90 con 0x560450a080b0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450643250 con 0x560450a117a0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604508f5200 con 0x560450a089d0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc69c60c740 10 client.0 _ll_register_callbacks cb 0x56045092b5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.776+0000 7fc692ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a0e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a0e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60396/0 (socket says 172.21.15.3:60396) 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 -- 172.21.15.3:0/3326922037 learned_addr learned my addr 172.21.15.3:0/3326922037 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc6927fc640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc6937fe640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a0f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 -- 172.21.15.3:0/3326922037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 msgr2=0x560450a0eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 -- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 msgr2=0x560450a0f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a0f2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 -- 172.21.15.3:0/3326922037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6941a68a0 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc692ffd640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a0e860 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc684010500 tx=0x7fc684010900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2542537035df17e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc672ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc672ffd640 1 -- 172.21.15.3:0/3326922037 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684014640 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc672ffd640 1 -- 172.21.15.3:0/3326922037 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc6840147a0 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560450995ee0 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc672ffd640 1 -- 172.21.15.3:0/3326922037 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc684014960 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc69c60c740 10 client.17148 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.777+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x56045096dd90 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc672ffd640 1 -- 172.21.15.3:0/3326922037 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560450995ee0 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc672ffd640 1 -- 172.21.15.3:0/3326922037 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc684012d00 con 0x560450a080b0 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc69c60c740 10 client.17148 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:39.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc69c60c740 10 client.17148 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc69c60c740 20 client.17148 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a21af0 0x560450a23f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc69c60c740 4 client.17148 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56045066f600 con 0x560450a21af0 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.778+0000 7fc6927fc640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a21af0 0x560450a23f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc672ffd640 10 client.17148 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc6927fc640 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a21af0 0x560450a23f10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a41195646caae37 server_cookie=e56bb694ef94c389 in_seq=0 out_seq=0 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc672ffd640 1 -- 172.21.15.3:0/3326922037 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56045066f600 con 0x560450a21af0 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc672ffd640 10 client.17148 handle_command_reply: tid=0 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 client.17148 shutdown 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 msgr2=0x560450a0e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a0e860 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fc684010500 tx=0x7fc684010900 comp rx=0 tx=0).stop 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 shutdown_connections 2024-02-09T22:55:39.788 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a21af0 0x560450a23f10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a0f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc672ffd640 0 client.17148 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3326922037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a0e860 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.779+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 >> 172.21.15.3:0/3326922037 conn(0x560450a029d0 msgr2=0x560450a04720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.780+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 shutdown_connections 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.780+0000 7fc69c60c740 1 -- 172.21.15.3:0/3326922037 wait complete. 2024-02-09T22:55:39.789 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.780+0000 7fc69c60c740 20 client.17148 trim_cache size 0 max 16384 2024-02-09T22:55:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 Processor -- start 2024-02-09T22:55:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 -- start start 2024-02-09T22:55:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a080b0 0x560450a10f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 0x560450a0bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450a07870 con 0x560450a117a0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604509fe2a0 con 0x560450a080b0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc69c60c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450a0cdd0 con 0x560450a089d0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 0x560450a0bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 0x560450a0bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60410/0 (socket says 172.21.15.3:60410) 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 -- 172.21.15.3:0/3474365242 learned_addr learned my addr 172.21.15.3:0/3474365242 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc692ffd640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a080b0 0x560450a10f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6927fc640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 -- 172.21.15.3:0/3474365242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 msgr2=0x560450a0f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 -- 172.21.15.3:0/3474365242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a080b0 msgr2=0x560450a10f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a080b0 0x560450a10f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6937fe640 1 -- 172.21.15.3:0/3474365242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56045090a820 con 0x560450a117a0 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc692ffd640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a080b0 0x560450a10f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:39.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.793+0000 7fc6927fc640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc6937fe640 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 0x560450a0bb00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009cd0 tx=0x7fc68c049000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=217bac37f6e0d42f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc6797fa640 1 -- 172.21.15.3:0/3474365242 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c0324c0 con 0x560450a117a0 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc6797fa640 1 -- 172.21.15.3:0/3474365242 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc68c034c90 con 0x560450a117a0 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc6797fa640 1 -- 172.21.15.3:0/3474365242 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c0042a0 con 0x560450a117a0 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 -- 172.21.15.3:0/3474365242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 msgr2=0x560450a0bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 0x560450a0bb00 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009cd0 tx=0x7fc68c049000 comp rx=0 tx=0).stop 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 -- 172.21.15.3:0/3474365242 shutdown_connections 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a117a0 0x560450a0bb00 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a0f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 --2- 172.21.15.3:0/3474365242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a080b0 0x560450a10f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 -- 172.21.15.3:0/3474365242 >> 172.21.15.3:0/3474365242 conn(0x560450a029d0 msgr2=0x560450a0d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.794+0000 7fc69c60c740 1 -- 172.21.15.3:0/3474365242 shutdown_connections 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 -- 172.21.15.3:0/3474365242 wait complete. 2024-02-09T22:55:39.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 -- start start 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 Processor -- start 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a17170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a176b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a17c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450a781c0 con 0x560450a080b0 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450a78330 con 0x560450a117a0 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560450a784f0 con 0x560450a089d0 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc69c60c740 10 client.0 _ll_register_callbacks cb 0x7fc684007520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc692ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a17170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc692ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a17170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60418/0 (socket says 172.21.15.3:60418) 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.795+0000 7fc692ffd640 1 -- 172.21.15.3:0/1226218872 learned_addr learned my addr 172.21.15.3:0/1226218872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc692ffd640 1 -- 172.21.15.3:0/1226218872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 msgr2=0x560450a176b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc6927fc640 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a176b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc692ffd640 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a176b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc692ffd640 1 -- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 msgr2=0x560450a17c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:39.803 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc692ffd640 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a17c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc692ffd640 1 -- 172.21.15.3:0/1226218872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc68c049070 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc692ffd640 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a17170 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc684006e10 tx=0x7fc684006aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afadeb4e47b1f196 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc672ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc672ffd640 1 -- 172.21.15.3:0/1226218872 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc684013c50 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc672ffd640 1 -- 172.21.15.3:0/1226218872 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc684000e40 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc68809af00 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc672ffd640 1 -- 172.21.15.3:0/1226218872 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc684013db0 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc69c60c740 10 client.17154 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5604509306f0 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.796+0000 7fc672ffd640 1 -- 172.21.15.3:0/1226218872 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc68809af00 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc672ffd640 1 -- 172.21.15.3:0/1226218872 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc68401d680 con 0x560450a080b0 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc69c60c740 10 client.17154 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:39.804 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc69c60c740 10 client.17154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc69c60c740 20 client.17154 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc69c60c740 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a79ba0 0x560450a7bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc69c60c740 4 client.17154 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5604506c01b0 con 0x560450a79ba0 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc6927fc640 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a79ba0 0x560450a7bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc672ffd640 10 client.17154 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:39.805 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.797+0000 7fc6927fc640 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a79ba0 0x560450a7bfe0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a855a5467c7826 server_cookie=7cc915b4f83dad90 in_seq=0 out_seq=0 2024-02-09T22:55:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc672ffd640 1 -- 172.21.15.3:0/1226218872 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5604506c01b0 con 0x560450a79ba0 2024-02-09T22:55:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc672ffd640 10 client.17154 handle_command_reply: tid=0 2024-02-09T22:55:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 client.17154 shutdown 2024-02-09T22:55:39.830 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 msgr2=0x560450a17170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a17170 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fc684006e10 tx=0x7fc684006aa0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 shutdown_connections 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560450a79ba0 0x560450a7bfe0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560450a117a0 0x560450a17c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560450a089d0 0x560450a176b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc672ffd640 0 client.17154 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 --2- 172.21.15.3:0/1226218872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560450a080b0 0x560450a17170 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 >> 172.21.15.3:0/1226218872 conn(0x560450a029d0 msgr2=0x560450a011a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 shutdown_connections 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 1 -- 172.21.15.3:0/1226218872 wait complete. 2024-02-09T22:55:39.831 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.827+0000 7fc69c60c740 20 client.17154 trim_cache size 0 max 16384 2024-02-09T22:55:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:55:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:55:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:55:39.832 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc67c0a2220 msgr2=0x7fc67c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc67c0a2220 0x7fc67c0a46e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc68c005e30 tx=0x7fc68c005fd0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 msgr2=0x7fc694100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc694100de0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fc68800d950 tx=0x7fc68800de10 comp rx=0 tx=0).stop 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 shutdown_connections 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc67c0a2220 0x7fc67c0a46e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc694106220 0x7fc694101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc694100080 0x7fc694100de0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 --2- 172.21.15.3:0/269515478 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc6940ff6c0 0x7fc6941008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 >> 172.21.15.3:0/269515478 conn(0x7fc694076650 msgr2=0x7fc6941041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 shutdown_connections 2024-02-09T22:55:39.833 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:39.828+0000 7fc699963640 1 -- 172.21.15.3:0/269515478 wait complete. 2024-02-09T22:55:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:40 smithi003 ceph-mon[32678]: pgmap v135: 97 pgs: 97 active+clean; 54 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T22:55:40.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124973 2.65 MB/sec execute 94 sec latency 2540.837 ms 2024-02-09T22:55:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:40 smithi064 ceph-mon[36638]: pgmap v135: 97 pgs: 97 active+clean; 54 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T22:55:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:40 smithi089 ceph-mon[36665]: pgmap v135: 97 pgs: 97 active+clean; 54 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T22:55:41.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:41 smithi003 ceph-mon[32678]: pgmap v136: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-02-09T22:55:41.635 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124973 2.62 MB/sec execute 95 sec latency 3540.961 ms 2024-02-09T22:55:41.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:41 smithi064 ceph-mon[36638]: pgmap v136: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-02-09T22:55:41.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:41 smithi089 ceph-mon[36665]: pgmap v136: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.3 MiB/s wr, 186 op/s 2024-02-09T22:55:42.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 124973 2.59 MB/sec execute 96 sec latency 4541.102 ms 2024-02-09T22:55:43.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 126629 2.63 MB/sec execute 97 sec latency 4950.053 ms 2024-02-09T22:55:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:43 smithi064 ceph-mon[36638]: pgmap v137: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 563 KiB/s wr, 77 op/s 2024-02-09T22:55:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:43 smithi089 ceph-mon[36665]: pgmap v137: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 563 KiB/s wr, 77 op/s 2024-02-09T22:55:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:43 smithi003 ceph-mon[32678]: pgmap v137: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 563 KiB/s wr, 77 op/s 2024-02-09T22:55:44.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128341 2.73 MB/sec execute 98 sec latency 9.999 ms 2024-02-09T22:55:45.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128449 2.70 MB/sec execute 99 sec latency 941.385 ms 2024-02-09T22:55:46.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:46 smithi064 ceph-mon[36638]: pgmap v138: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 563 KiB/s wr, 77 op/s 2024-02-09T22:55:46.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:46 smithi089 ceph-mon[36665]: pgmap v138: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 563 KiB/s wr, 77 op/s 2024-02-09T22:55:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:46 smithi003 ceph-mon[32678]: pgmap v138: 97 pgs: 97 active+clean; 51 MiB data, 793 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 563 KiB/s wr, 77 op/s 2024-02-09T22:55:46.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128449 2.68 MB/sec execute 100 sec latency 1941.465 ms 2024-02-09T22:55:47.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128449 2.65 MB/sec execute 101 sec latency 2941.549 ms 2024-02-09T22:55:48.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:48 smithi089 ceph-mon[36665]: pgmap v139: 97 pgs: 97 active+clean; 53 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-02-09T22:55:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:48 smithi003 ceph-mon[32678]: pgmap v139: 97 pgs: 97 active+clean; 53 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-02-09T22:55:48.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128518 2.63 MB/sec execute 102 sec latency 3353.038 ms 2024-02-09T22:55:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:48 smithi064 ceph-mon[36638]: pgmap v139: 97 pgs: 97 active+clean; 53 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-02-09T22:55:49.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128518 2.60 MB/sec execute 103 sec latency 1543.509 ms 2024-02-09T22:55:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:50 smithi089 ceph-mon[36665]: pgmap v140: 97 pgs: 97 active+clean; 53 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 560 KiB/s wr, 77 op/s 2024-02-09T22:55:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:50 smithi003 ceph-mon[32678]: pgmap v140: 97 pgs: 97 active+clean; 53 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 560 KiB/s wr, 77 op/s 2024-02-09T22:55:50.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128518 2.58 MB/sec execute 104 sec latency 2543.644 ms 2024-02-09T22:55:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:50 smithi064 ceph-mon[36638]: pgmap v140: 97 pgs: 97 active+clean; 53 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 556 KiB/s rd, 560 KiB/s wr, 77 op/s 2024-02-09T22:55:51.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128518 2.55 MB/sec execute 105 sec latency 3543.771 ms 2024-02-09T22:55:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:52 smithi089 ceph-mon[36665]: pgmap v141: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 605 KiB/s wr, 89 op/s 2024-02-09T22:55:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:52 smithi003 ceph-mon[32678]: pgmap v141: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 605 KiB/s wr, 89 op/s 2024-02-09T22:55:52.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 128518 2.53 MB/sec execute 106 sec latency 4543.831 ms 2024-02-09T22:55:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:52 smithi064 ceph-mon[36638]: pgmap v141: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 591 KiB/s rd, 605 KiB/s wr, 89 op/s 2024-02-09T22:55:52.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 -- 172.21.15.3:0/479087262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed4100cf0 msgr2=0x7f1ed4104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 --2- 172.21.15.3:0/479087262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed4100cf0 0x7f1ed4104d30 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009930 tx=0x7f1ec402bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 -- 172.21.15.3:0/479087262 shutdown_connections 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 --2- 172.21.15.3:0/479087262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed4105410 0x7f1ed410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 --2- 172.21.15.3:0/479087262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed4100cf0 0x7f1ed4104d30 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 --2- 172.21.15.3:0/479087262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ed4100340 0x7f1ed4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 -- 172.21.15.3:0/479087262 >> 172.21.15.3:0/479087262 conn(0x7f1ed4077220 msgr2=0x7f1ed4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 -- 172.21.15.3:0/479087262 shutdown_connections 2024-02-09T22:55:52.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 -- 172.21.15.3:0/479087262 wait complete. 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.922+0000 7f1edc584640 1 Processor -- start 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 -- start start 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed4100340 0x7f1ed410d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ed4105410 0x7f1ed410de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 0x7f1ed41a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed41a6030 con 0x7f1ed410e340 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed41a61a0 con 0x7f1ed4105410 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edc584640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ed41a6360 con 0x7f1ed4100340 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 0x7f1ed41a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 0x7f1ed41a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39848/0 (socket says 172.21.15.3:39848) 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 -- 172.21.15.3:0/558764479 learned_addr learned my addr 172.21.15.3:0/558764479 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ed4105410 0x7f1ed410de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1eda320640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed4100340 0x7f1ed410d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 -- 172.21.15.3:0/558764479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed4100340 msgr2=0x7f1ed410d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed4100340 0x7f1ed410d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 -- 172.21.15.3:0/558764479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ed4105410 msgr2=0x7f1ed410de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ed4105410 0x7f1ed410de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 -- 172.21.15.3:0/558764479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec4009560 con 0x7f1ed410e340 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.923+0000 7f1edab21640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 0x7f1ed41a5a60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00eb10 tx=0x7f1ecc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.924+0000 7f1ecb7fe640 1 -- 172.21.15.3:0/558764479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc004b20 con 0x7f1ed410e340 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.924+0000 7f1ecb7fe640 1 -- 172.21.15.3:0/558764479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ecc004c80 con 0x7f1ed410e340 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.924+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ed41a6640 con 0x7f1ed410e340 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.924+0000 7f1ecb7fe640 1 -- 172.21.15.3:0/558764479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ecc010650 con 0x7f1ed410e340 2024-02-09T22:55:52.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.924+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed41a6b20 con 0x7f1ed410e340 2024-02-09T22:55:52.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1ecb7fe640 1 -- 172.21.15.3:0/558764479 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1ecc005020 con 0x7f1ed410e340 2024-02-09T22:55:52.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 Processor -- start 2024-02-09T22:55:52.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 -- start start 2024-02-09T22:55:52.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99de490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de9d0 0x555ac99e7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99e9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac994ca00 con 0x555ac99e77a0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac9949f30 con 0x555ac99de9d0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1edf22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac9945710 con 0x555ac99de0b0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1ed9b1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99e9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.927+0000 7f1ecb7fe640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1ec00a2250 0x7f1ec00a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ed9b1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99e9b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39856/0 (socket says 172.21.15.3:39856) 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de9d0 0x555ac99e7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ed9b1f640 1 -- 172.21.15.3:0/1527711335 learned_addr learned my addr 172.21.15.3:0/1527711335 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99de490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1edab21640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1ec00a2250 0x7f1ec00a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ecb7fe640 1 -- 172.21.15.3:0/558764479 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1ecc014070 con 0x7f1ed410e340 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 -- 172.21.15.3:0/1527711335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 msgr2=0x555ac99de490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99de490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 -- 172.21.15.3:0/1527711335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 msgr2=0x555ac99e9b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99e9b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 -- 172.21.15.3:0/1527711335 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac990a3b0 con 0x555ac99de9d0 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99de490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1eda320640 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de9d0 0x555ac99e7260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009c20 tx=0x7f1ed0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=912c4db6a2953ab5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1edab21640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1ec00a2250 0x7f1ec00a4710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc09ac50 tx=0x7f1ecc085a80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ec8bfe640 1 -- 172.21.15.3:0/1527711335 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0004050 con 0x555ac99de9d0 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ec8bfe640 1 -- 172.21.15.3:0/1527711335 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ed0005500 con 0x555ac99de9d0 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.928+0000 7f1ec8bfe640 1 -- 172.21.15.3:0/1527711335 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0004c30 con 0x555ac99de9d0 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 -- 172.21.15.3:0/1527711335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de9d0 msgr2=0x555ac99e7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de9d0 0x555ac99e7260 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0009c20 tx=0x7f1ed0011810 comp rx=0 tx=0).stop 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 -- 172.21.15.3:0/1527711335 shutdown_connections 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de9d0 0x555ac99e7260 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99e9b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1527711335 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99de490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 -- 172.21.15.3:0/1527711335 >> 172.21.15.3:0/1527711335 conn(0x555ac99d89d0 msgr2=0x555ac99d8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 -- 172.21.15.3:0/1527711335 shutdown_connections 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.929+0000 7f1edf22a740 1 -- 172.21.15.3:0/1527711335 wait complete. 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 -- start start 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 Processor -- start 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99e47f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99e77a0 0x555ac99e5270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac98eab90 con 0x555ac99de9d0 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac9619250 con 0x555ac99de0b0 2024-02-09T22:55:52.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac98cb200 con 0x555ac99e77a0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1edf22a740 10 client.0 _ll_register_callbacks cb 0x555ac99015d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39870/0 (socket says 172.21.15.3:39870) 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 -- 172.21.15.3:0/2656143875 learned_addr learned my addr 172.21.15.3:0/2656143875 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99e47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99e77a0 0x555ac99e5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 -- 172.21.15.3:0/2656143875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99e77a0 msgr2=0x555ac99e5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99e77a0 0x555ac99e5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 -- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 msgr2=0x555ac99e47f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99e47f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.930+0000 7f1eda320640 1 -- 172.21.15.3:0/2656143875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ed0009850 con 0x555ac99de9d0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1eda320640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e4d30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0004910 tx=0x7f1ed0004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24637ed8eeac5e5a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1ebdffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1ebdffb640 1 -- 172.21.15.3:0/2656143875 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0003a00 con 0x555ac99de9d0 2024-02-09T22:55:52.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1ebdffb640 1 -- 172.21.15.3:0/2656143875 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1ed00046c0 con 0x555ac99de9d0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ac996bee0 con 0x555ac99de9d0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1ebdffb640 1 -- 172.21.15.3:0/2656143875 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ed0003b60 con 0x555ac99de9d0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 10 client.17166 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x555ac9943d90 con 0x555ac99de9d0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1ebdffb640 1 -- 172.21.15.3:0/2656143875 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555ac996bee0 con 0x555ac99de9d0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1ebdffb640 1 -- 172.21.15.3:0/2656143875 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1ed0021040 con 0x555ac99de9d0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 10 client.17166 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 10 client.17166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 20 client.17166 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac99f7850 0x555ac99f9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 4 client.17166 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.931+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ac9645600 con 0x555ac99f7850 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.932+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac99f7850 0x555ac99f9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.932+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac99f7850 0x555ac99f9c70 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d48a3056a6b1a701 server_cookie=453dff173d75cf45 in_seq=0 out_seq=0 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.932+0000 7f1ebdffb640 10 client.17166 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1ebdffb640 1 -- 172.21.15.3:0/2656143875 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f1ec404fa90 con 0x555ac99f7850 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1ebdffb640 10 client.17166 handle_command_reply: tid=0 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 client.17166 shutdown 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 msgr2=0x555ac99e4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e4d30 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0004910 tx=0x7f1ed0004430 comp rx=0 tx=0).stop 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 shutdown_connections 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac99f7850 0x555ac99f9c70 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99e77a0 0x555ac99e5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e4d30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1ebdffb640 0 client.17166 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 --2- 172.21.15.3:0/2656143875 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99e47f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 >> 172.21.15.3:0/2656143875 conn(0x555ac99d89d0 msgr2=0x555ac99da720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:52.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 shutdown_connections 2024-02-09T22:55:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 1 -- 172.21.15.3:0/2656143875 wait complete. 2024-02-09T22:55:52.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.933+0000 7f1edf22a740 20 client.17166 trim_cache size 0 max 16384 2024-02-09T22:55:52.953 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 Processor -- start 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 -- start start 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99e6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99e77a0 0x555ac99e1a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac99d99d0 con 0x555ac99de9d0 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac99d2ec0 con 0x555ac99e77a0 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.946+0000 7f1edf22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac99d42a0 con 0x555ac99de0b0 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e5090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39872/0 (socket says 172.21.15.3:39872) 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 -- 172.21.15.3:0/1835459462 learned_addr learned my addr 172.21.15.3:0/1835459462 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99e6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99e77a0 0x555ac99e1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 -- 172.21.15.3:0/1835459462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 msgr2=0x555ac99e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99e6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 -- 172.21.15.3:0/1835459462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99e77a0 msgr2=0x555ac99e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99e77a0 0x555ac99e1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 -- 172.21.15.3:0/1835459462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac98e0820 con 0x555ac99de9d0 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1eda320640 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e5090 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0011d10 tx=0x7f1ed0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b93b032cbdbc827 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1ebffff640 1 -- 172.21.15.3:0/1835459462 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed00044a0 con 0x555ac99de9d0 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1ebffff640 1 -- 172.21.15.3:0/1835459462 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ed0009a70 con 0x555ac99de9d0 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.947+0000 7f1ebffff640 1 -- 172.21.15.3:0/1835459462 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ed0003e90 con 0x555ac99de9d0 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- 172.21.15.3:0/1835459462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 msgr2=0x555ac99e5090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e5090 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f1ed0011d10 tx=0x7f1ed0011810 comp rx=0 tx=0).stop 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- 172.21.15.3:0/1835459462 shutdown_connections 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99e77a0 0x555ac99e1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99de9d0 0x555ac99e5090 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1835459462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de0b0 0x555ac99e6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- 172.21.15.3:0/1835459462 >> 172.21.15.3:0/1835459462 conn(0x555ac99d89d0 msgr2=0x555ac99e30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- 172.21.15.3:0/1835459462 shutdown_connections 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- 172.21.15.3:0/1835459462 wait complete. 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- start start 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 Processor -- start 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99ee700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de9d0 0x555ac99ecaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99ed030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac99ed5c0 con 0x555ac99e77a0 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac99ed730 con 0x555ac99de0b0 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac99ed8f0 con 0x555ac99de9d0 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.948+0000 7f1edf22a740 10 client.0 _ll_register_callbacks cb 0x7f1ecc0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:52.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99ed030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99ed030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:39880/0 (socket says 172.21.15.3:39880) 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1eda320640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de9d0 0x555ac99ecaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 -- 172.21.15.3:0/1346043638 learned_addr learned my addr 172.21.15.3:0/1346043638 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 -- 172.21.15.3:0/1346043638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de9d0 msgr2=0x555ac99ecaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de9d0 0x555ac99ecaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 -- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 msgr2=0x555ac99ee700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99ee700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 -- 172.21.15.3:0/1346043638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac9943d90 con 0x555ac99e77a0 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ed9b1f640 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99ed030 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009230 tx=0x7f1ec4005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf1da9536c5c929 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ebdffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ebdffb640 1 -- 172.21.15.3:0/1346043638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ec402ccb0 con 0x555ac99e77a0 2024-02-09T22:55:52.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ebdffb640 1 -- 172.21.15.3:0/1346043638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1ec4004040 con 0x555ac99e77a0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ed0005500 con 0x555ac99e77a0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1ebdffb640 1 -- 172.21.15.3:0/1346043638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ec4004200 con 0x555ac99e77a0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1edf22a740 10 client.17172 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.949+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x555ac99066f0 con 0x555ac99e77a0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1ebdffb640 1 -- 172.21.15.3:0/1346043638 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ed0005500 con 0x555ac99e77a0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1ebdffb640 1 -- 172.21.15.3:0/1346043638 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1ec4005050 con 0x555ac99e77a0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1edf22a740 10 client.17172 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1edf22a740 10 client.17172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1edf22a740 20 client.17172 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac9a4f390 0x555ac9a517b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1edf22a740 4 client.17172 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555ac96961b0 con 0x555ac9a4f390 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1eda320640 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac9a4f390 0x555ac9a517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:52.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1ebdffb640 10 client.17172 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:52.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.950+0000 7f1eda320640 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac9a4f390 0x555ac9a517b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8cb07cd67ae8355c server_cookie=4822e3a534948d49 in_seq=0 out_seq=0 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.977+0000 7f1ebdffb640 1 -- 172.21.15.3:0/1346043638 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555ac96961b0 con 0x555ac9a4f390 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.977+0000 7f1ebdffb640 10 client.17172 handle_command_reply: tid=0 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.977+0000 7f1edf22a740 1 client.17172 shutdown 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 msgr2=0x555ac99ed030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99ed030 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f1ec4009230 tx=0x7f1ec4005e70 comp rx=0 tx=0).stop 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 shutdown_connections 2024-02-09T22:55:52.981 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555ac9a4f390 0x555ac9a517b0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555ac99e77a0 0x555ac99ed030 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555ac99de9d0 0x555ac99ecaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 --2- 172.21.15.3:0/1346043638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555ac99de0b0 0x555ac99ee700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1ebdffb640 0 client.17172 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 >> 172.21.15.3:0/1346043638 conn(0x555ac99d89d0 msgr2=0x555ac99d71a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 shutdown_connections 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 1 -- 172.21.15.3:0/1346043638 wait complete. 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edf22a740 20 client.17172 trim_cache size 0 max 16384 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:55:52.982 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:55:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:55:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:55:52.983 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1ec00a2250 msgr2=0x7f1ec00a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1ec00a2250 0x7f1ec00a4710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc09ac50 tx=0x7f1ecc085a80 comp rx=0 tx=0).stop 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 msgr2=0x7f1ed41a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 0x7f1ed41a5a60 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc00eb10 tx=0x7f1ecc00c2d0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 shutdown_connections 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1ec00a2250 0x7f1ec00a4710 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1ed4105410 0x7f1ed410de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1ed410e340 0x7f1ed41a5a60 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 --2- 172.21.15.3:0/558764479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1ed4100340 0x7f1ed410d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 >> 172.21.15.3:0/558764479 conn(0x7f1ed4077220 msgr2=0x7f1ed40fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.978+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 shutdown_connections 2024-02-09T22:55:52.984 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:55:52.979+0000 7f1edc584640 1 -- 172.21.15.3:0/558764479 wait complete. 2024-02-09T22:55:53.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 130182 2.57 MB/sec execute 107 sec latency 4950.355 ms 2024-02-09T22:55:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:54 smithi089 ceph-mon[36665]: pgmap v142: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 514 KiB/s wr, 65 op/s 2024-02-09T22:55:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:54 smithi003 ceph-mon[32678]: pgmap v142: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 514 KiB/s wr, 65 op/s 2024-02-09T22:55:54.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 131998 2.66 MB/sec execute 108 sec latency 10.277 ms 2024-02-09T22:55:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:54 smithi064 ceph-mon[36638]: pgmap v142: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 514 KiB/s wr, 65 op/s 2024-02-09T22:55:55.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 134612 2.76 MB/sec execute 109 sec latency 11.413 ms 2024-02-09T22:55:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:56 smithi089 ceph-mon[36665]: pgmap v143: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 514 KiB/s wr, 65 op/s 2024-02-09T22:55:56.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:56 smithi064 ceph-mon[36638]: pgmap v143: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 514 KiB/s wr, 65 op/s 2024-02-09T22:55:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:56 smithi003 ceph-mon[32678]: pgmap v143: 97 pgs: 97 active+clean; 53 MiB data, 755 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 514 KiB/s wr, 65 op/s 2024-02-09T22:55:56.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 135543 2.78 MB/sec execute 110 sec latency 237.021 ms 2024-02-09T22:55:57.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:55:57.569 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:55:57.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 135543 2.76 MB/sec execute 111 sec latency 1237.071 ms 2024-02-09T22:55:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 -- 172.21.15.3:0/366231056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408102df0 msgr2=0x7fb408107220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/366231056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408102df0 0x7fb408107220 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0009930 tx=0x7fb3f002bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 -- 172.21.15.3:0/366231056 shutdown_connections 2024-02-09T22:55:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/366231056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408107930 0x7fb40810e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/366231056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408102df0 0x7fb408107220 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/366231056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb408102440 0x7fb408102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.683+0000 7fb40e8d0640 1 -- 172.21.15.3:0/366231056 >> 172.21.15.3:0/366231056 conn(0x7fb408077310 msgr2=0x7fb408077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.684+0000 7fb40e8d0640 1 -- 172.21.15.3:0/366231056 shutdown_connections 2024-02-09T22:55:57.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.684+0000 7fb40e8d0640 1 -- 172.21.15.3:0/366231056 wait complete. 2024-02-09T22:55:57.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.684+0000 7fb40e8d0640 1 Processor -- start 2024-02-09T22:55:57.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.684+0000 7fb40e8d0640 1 -- start start 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb408102440 0x7fb40819d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 0x7fb40819dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408197820 0x7fb408197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4081982d0 con 0x7fb408107930 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb408198440 con 0x7fb408102440 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4081985b0 con 0x7fb408197820 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 0x7fb40819dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 0x7fb40819dbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42448/0 (socket says 172.21.15.3:42448) 2024-02-09T22:55:57.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 -- 172.21.15.3:0/1192919905 learned_addr learned my addr 172.21.15.3:0/1192919905 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb407fff640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb408102440 0x7fb40819d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40ce6d640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408197820 0x7fb408197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 -- 172.21.15.3:0/1192919905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408197820 msgr2=0x7fb408197cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408197820 0x7fb408197cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 -- 172.21.15.3:0/1192919905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb408102440 msgr2=0x7fb40819d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb408102440 0x7fb40819d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 -- 172.21.15.3:0/1192919905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f0009560 con 0x7fb408107930 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4077fe640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 0x7fb40819dbf0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0000c00 tx=0x7fb3f0005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4057fa640 1 -- 172.21.15.3:0/1192919905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0004350 con 0x7fb408107930 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb4057fa640 1 -- 172.21.15.3:0/1192919905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3f00044b0 con 0x7fb408107930 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.685+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4081a4500 con 0x7fb408107930 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.686+0000 7fb4057fa640 1 -- 172.21.15.3:0/1192919905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f0003cd0 con 0x7fb408107930 2024-02-09T22:55:57.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.686+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4081a49e0 con 0x7fb408107930 2024-02-09T22:55:57.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb4057fa640 1 -- 172.21.15.3:0/1192919905 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb3f0004df0 con 0x7fb408107930 2024-02-09T22:55:57.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 Processor -- start 2024-02-09T22:55:57.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 -- start start 2024-02-09T22:55:57.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb4057fa640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3f40a2230 0x7fb3f40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 0x55fcd9626d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9627330 0x55fcd962f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd962feb0 0x55fcd9632300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd95e1d00 con 0x55fcd962feb0 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd95962f0 con 0x55fcd9627330 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.688+0000 7fb411576740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9593820 con 0x55fcd9626980 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb40ce6d640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3f40a2230 0x7fb3f40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb4057fa640 1 -- 172.21.15.3:0/1192919905 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3f00f00c0 con 0x7fb408107930 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb40ce6d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd962feb0 0x55fcd9632300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 0x55fcd9626d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb40ce6d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd962feb0 0x55fcd9632300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42464/0 (socket says 172.21.15.3:42464) 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 0x55fcd9626d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41618/0 (socket says 172.21.15.3:41618) 2024-02-09T22:55:57.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb40ce6d640 1 -- 172.21.15.3:0/50266856 learned_addr learned my addr 172.21.15.3:0/50266856 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb4077fe640 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9627330 0x55fcd962f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 -- 172.21.15.3:0/50266856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9627330 msgr2=0x55fcd962f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9627330 0x55fcd962f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 -- 172.21.15.3:0/50266856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd962feb0 msgr2=0x55fcd9632300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd962feb0 0x55fcd9632300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 -- 172.21.15.3:0/50266856 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcd9550ff0 con 0x55fcd9626980 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb40ce6d640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3f40a2230 0x7fb3f40a46f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb400009fd0 tx=0x7fb400009da0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb407fff640 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 0x55fcd9626d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009c20 tx=0x7fb3f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a21da2d606eb61c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.689+0000 7fb3fe7fc640 1 -- 172.21.15.3:0/50266856 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8004050 con 0x55fcd9626980 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb3fe7fc640 1 -- 172.21.15.3:0/50266856 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3f8005500 con 0x55fcd9626980 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb3fe7fc640 1 -- 172.21.15.3:0/50266856 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8004d30 con 0x55fcd9626980 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 -- 172.21.15.3:0/50266856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 msgr2=0x55fcd9626d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 0x55fcd9626d60 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009c20 tx=0x7fb3f8011810 comp rx=0 tx=0).stop 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 -- 172.21.15.3:0/50266856 shutdown_connections 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9627330 0x55fcd962f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd962feb0 0x55fcd9632300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 --2- 172.21.15.3:0/50266856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9626980 0x55fcd9626d60 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 -- 172.21.15.3:0/50266856 >> 172.21.15.3:0/50266856 conn(0x55fcd9621050 msgr2=0x55fcd9621430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 -- 172.21.15.3:0/50266856 shutdown_connections 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.690+0000 7fb411576740 1 -- 172.21.15.3:0/50266856 wait complete. 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 -- start start 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 Processor -- start 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd962cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd962d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd962d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd958f000 con 0x55fcd9626980 2024-02-09T22:55:57.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9262430 con 0x55fcd962feb0 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd921ee00 con 0x55fcd9627330 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb411576740 10 client.0 _ll_register_callbacks cb 0x55fcd95837f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd962cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd962cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42470/0 (socket says 172.21.15.3:42470) 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 -- 172.21.15.3:0/216342130 learned_addr learned my addr 172.21.15.3:0/216342130 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb40ce6d640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd962d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb4077fe640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd962d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 -- 172.21.15.3:0/216342130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 msgr2=0x55fcd962d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd962d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 -- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 msgr2=0x55fcd962d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd962d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.691+0000 7fb407fff640 1 -- 172.21.15.3:0/216342130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f8009850 con 0x55fcd9626980 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb407fff640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd962cde0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8004a10 tx=0x7fb3f8004500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=290a6de9b342ef64 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb3e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb3e77fe640 1 -- 172.21.15.3:0/216342130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8003a00 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb3e77fe640 1 -- 172.21.15.3:0/216342130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3f80047c0 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fcd928e7e0 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb3e77fe640 1 -- 172.21.15.3:0/216342130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3f8003b60 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 10 client.17184 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fcd958d680 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb3e77fe640 1 -- 172.21.15.3:0/216342130 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fcd928e7e0 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb3e77fe640 1 -- 172.21.15.3:0/216342130 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3f8024540 con 0x55fcd9626980 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 10 client.17184 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 10 client.17184 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 20 client.17184 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9640300 0x55fcd9642720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 4 client.17184 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:57.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.692+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fcd92cb720 con 0x55fcd9640300 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.693+0000 7fb4077fe640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9640300 0x55fcd9642720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.693+0000 7fb3e77fe640 10 client.17184 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.693+0000 7fb4077fe640 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9640300 0x55fcd9642720 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9bec228be7f20bf3 server_cookie=6969da9423749f27 in_seq=0 out_seq=0 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb3e77fe640 1 -- 172.21.15.3:0/216342130 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fcd92cb720 con 0x55fcd9640300 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb3e77fe640 10 client.17184 handle_command_reply: tid=0 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 client.17184 shutdown 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 msgr2=0x55fcd962cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd962cde0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8004a10 tx=0x7fb3f8004500 comp rx=0 tx=0).stop 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 shutdown_connections 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9640300 0x55fcd9642720 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd962d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd962d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb3e77fe640 0 client.17184 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:57.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 --2- 172.21.15.3:0/216342130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd962cde0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 >> 172.21.15.3:0/216342130 conn(0x55fcd9621050 msgr2=0x55fcd9622da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 shutdown_connections 2024-02-09T22:55:57.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 1 -- 172.21.15.3:0/216342130 wait complete. 2024-02-09T22:55:57.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.694+0000 7fb411576740 20 client.17184 trim_cache size 0 max 16384 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.709+0000 7fb411576740 1 Processor -- start 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 -- start start 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9626980 0x55fcd962d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 0x55fcd962d890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd962feb0 0x55fcd962a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9513fd0 con 0x55fcd9627330 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd963cec0 con 0x55fcd9626980 2024-02-09T22:55:57.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb411576740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9622050 con 0x55fcd962feb0 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 0x55fcd962d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 0x55fcd962d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42480/0 (socket says 172.21.15.3:42480) 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 -- 172.21.15.3:0/2238884568 learned_addr learned my addr 172.21.15.3:0/2238884568 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb407fff640 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9626980 0x55fcd962d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 -- 172.21.15.3:0/2238884568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd962feb0 msgr2=0x55fcd962a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd962feb0 0x55fcd962a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 -- 172.21.15.3:0/2238884568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9626980 msgr2=0x55fcd962d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9626980 0x55fcd962d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 -- 172.21.15.3:0/2238884568 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fcd9529b30 con 0x55fcd9627330 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb4077fe640 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 0x55fcd962d890 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00b8ff0 tx=0x7fb3f002cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ab87db781ab9761 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb3fdffb640 1 -- 172.21.15.3:0/2238884568 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00035f0 con 0x55fcd9627330 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb3fdffb640 1 -- 172.21.15.3:0/2238884568 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3f0005600 con 0x55fcd9627330 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.710+0000 7fb3fdffb640 1 -- 172.21.15.3:0/2238884568 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f00c5330 con 0x55fcd9627330 2024-02-09T22:55:57.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 -- 172.21.15.3:0/2238884568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 msgr2=0x55fcd962d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 0x55fcd962d890 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7fb3f00b8ff0 tx=0x7fb3f002cda0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 -- 172.21.15.3:0/2238884568 shutdown_connections 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd962feb0 0x55fcd962a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9627330 0x55fcd962d890 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 --2- 172.21.15.3:0/2238884568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd9626980 0x55fcd962d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 -- 172.21.15.3:0/2238884568 >> 172.21.15.3:0/2238884568 conn(0x55fcd9621050 msgr2=0x55fcd9628100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 -- 172.21.15.3:0/2238884568 shutdown_connections 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 -- 172.21.15.3:0/2238884568 wait complete. 2024-02-09T22:55:57.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.711+0000 7fb411576740 1 -- start start 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 Processor -- start 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd9635f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd96364c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd9696850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9636bc0 con 0x55fcd9626980 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9696d90 con 0x55fcd962feb0 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fcd9696ed0 con 0x55fcd9627330 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb411576740 10 client.0 _ll_register_callbacks cb 0x7fb3f80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd9635f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd9635f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42494/0 (socket says 172.21.15.3:42494) 2024-02-09T22:55:57.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 -- 172.21.15.3:0/3828218752 learned_addr learned my addr 172.21.15.3:0/3828218752 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb40ce6d640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd9696850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb4077fe640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd96364c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 -- 172.21.15.3:0/3828218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 msgr2=0x55fcd96364c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd96364c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 -- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 msgr2=0x55fcd9696850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd9696850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 -- 172.21.15.3:0/3828218752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4081a49e0 con 0x55fcd9626980 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb407fff640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd9635f80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009250 tx=0x7fb3f8024d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d8bc0464de65e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.712+0000 7fb3e77fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb3e77fe640 1 -- 172.21.15.3:0/3828218752 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8011d10 con 0x55fcd9626980 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb3e77fe640 1 -- 172.21.15.3:0/3828218752 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3f8000e40 con 0x55fcd9626980 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3f8005500 con 0x55fcd9626980 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb3e77fe640 1 -- 172.21.15.3:0/3828218752 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3f8009a70 con 0x55fcd9626980 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 10 client.17190 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:57.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55fcd954f850 con 0x55fcd9626980 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb3e77fe640 1 -- 172.21.15.3:0/3828218752 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3f8005500 con 0x55fcd9626980 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb3e77fe640 1 -- 172.21.15.3:0/3828218752 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3f8004a40 con 0x55fcd9626980 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 10 client.17190 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 20 client.17190 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9698380 0x55fcd969a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 4 client.17190 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fcd931a890 con 0x55fcd9698380 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.713+0000 7fb4077fe640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9698380 0x55fcd969a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb3e77fe640 10 client.17190 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb4077fe640 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9698380 0x55fcd969a7a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=345aa4e58495d3d7 server_cookie=fad38f2bc36eba24 in_seq=0 out_seq=0 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb3e77fe640 1 -- 172.21.15.3:0/3828218752 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fcd931a890 con 0x55fcd9698380 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb3e77fe640 10 client.17190 handle_command_reply: tid=0 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 client.17190 shutdown 2024-02-09T22:55:57.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 msgr2=0x55fcd9635f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd9635f80 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7fb3f8009250 tx=0x7fb3f8024d40 comp rx=0 tx=0).stop 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 shutdown_connections 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fcd9698380 0x55fcd969a7a0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fcd962feb0 0x55fcd9696850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fcd9627330 0x55fcd96364c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 --2- 172.21.15.3:0/3828218752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fcd9626980 0x55fcd9635f80 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb3e77fe640 0 client.17190 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 >> 172.21.15.3:0/3828218752 conn(0x55fcd9621050 msgr2=0x55fcd961f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 shutdown_connections 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 1 -- 172.21.15.3:0/3828218752 wait complete. 2024-02-09T22:55:57.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.714+0000 7fb411576740 20 client.17190 trim_cache size 0 max 16384 2024-02-09T22:55:57.724 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:55:57.724 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:55:57.724 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:55:57.724 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3f40a2230 msgr2=0x7fb3f40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3f40a2230 0x7fb3f40a46f0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb400009fd0 tx=0x7fb400009da0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 msgr2=0x7fb40819dbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 0x7fb40819dbf0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7fb3f0000c00 tx=0x7fb3f0005e70 comp rx=0 tx=0).stop 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 shutdown_connections 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3f40a2230 0x7fb3f40a46f0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb408197820 0x7fb408197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb408107930 0x7fb40819dbf0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 --2- 172.21.15.3:0/1192919905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb408102440 0x7fb40819d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 >> 172.21.15.3:0/1192919905 conn(0x7fb408077310 msgr2=0x7fb4080ff210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 shutdown_connections 2024-02-09T22:55:57.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.715+0000 7fb40e8d0640 1 -- 172.21.15.3:0/1192919905 wait complete. 2024-02-09T22:55:57.729 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:55:57.729 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 -- 172.21.15.3:0/1330614391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14105420 msgr2=0x7fef1410cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 --2- 172.21.15.3:0/1330614391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14105420 0x7fef1410cff0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7fef0c009cd0 tx=0x7fef0c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 -- 172.21.15.3:0/1330614391 shutdown_connections 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 --2- 172.21.15.3:0/1330614391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14105420 0x7fef1410cff0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 --2- 172.21.15.3:0/1330614391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef14100d00 0x7fef14104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 --2- 172.21.15.3:0/1330614391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14100350 0x7fef14100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 -- 172.21.15.3:0/1330614391 >> 172.21.15.3:0/1330614391 conn(0x7fef14077370 msgr2=0x7fef14077780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 -- 172.21.15.3:0/1330614391 shutdown_connections 2024-02-09T22:55:57.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 -- 172.21.15.3:0/1330614391 wait complete. 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 Processor -- start 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.829+0000 7fef19664640 1 -- start start 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef14100350 0x7fef14198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14105420 0x7fef14198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 0x7fef141a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef141a3dd0 con 0x7fef14199150 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef141a3f40 con 0x7fef14105420 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef141a4100 con 0x7fef14100350 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 0x7fef141a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 0x7fef141a3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42512/0 (socket says 172.21.15.3:42512) 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 -- 172.21.15.3:0/2876910113 learned_addr learned my addr 172.21.15.3:0/2876910113 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef12ffd640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef14100350 0x7fef14198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef0a7fc640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14105420 0x7fef14198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 -- 172.21.15.3:0/2876910113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef14100350 msgr2=0x7fef14198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef14100350 0x7fef14198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 -- 172.21.15.3:0/2876910113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14105420 msgr2=0x7fef14198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14105420 0x7fef14198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 -- 172.21.15.3:0/2876910113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c009900 con 0x7fef14199150 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef0a7fc640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14105420 0x7fef14198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef137fe640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 0x7fef141a3700 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fef0c004af0 tx=0x7fef0c034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c0328c0 con 0x7fef14199150 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fef0c003e50 con 0x7fef14199150 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.830+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef141a4380 con 0x7fef14199150 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.831+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0c004860 con 0x7fef14199150 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.831+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef141a4830 con 0x7fef14199150 2024-02-09T22:55:57.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.831+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef14101820 con 0x7fef14199150 2024-02-09T22:55:57.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.832+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fef0c00a040 con 0x7fef14199150 2024-02-09T22:55:57.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.833+0000 7fef10ff9640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fef000a2230 0x7fef000a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:57.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.833+0000 7fef12ffd640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fef000a2230 0x7fef000a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:57.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.833+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fef0c0f1230 con 0x7fef14199150 2024-02-09T22:55:57.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.834+0000 7fef12ffd640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fef000a2230 0x7fef000a46f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef04009b60 tx=0x7fef04005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:57.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:57.837+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fef0c0ba0c0 con 0x7fef14199150 2024-02-09T22:55:58.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.075+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef14108750 con 0x7fef14199150 2024-02-09T22:55:58.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.076+0000 7fef10ff9640 1 -- 172.21.15.3:0/2876910113 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+11361 (secure 0 0 0) 0x7fef0c039090 con 0x7fef14199150 2024-02-09T22:55:58.079 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:55:58.079 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:55:02.762674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:55:58.080 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 36 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fef000a2230 msgr2=0x7fef000a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fef000a2230 0x7fef000a46f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fef04009b60 tx=0x7fef04005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 msgr2=0x7fef141a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 0x7fef141a3700 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fef0c004af0 tx=0x7fef0c034cc0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 shutdown_connections 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fef000a2230 0x7fef000a46f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fef14105420 0x7fef14198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fef14199150 0x7fef141a3700 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 --2- 172.21.15.3:0/2876910113 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fef14100350 0x7fef14198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 >> 172.21.15.3:0/2876910113 conn(0x7fef14077370 msgr2=0x7fef140fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 shutdown_connections 2024-02-09T22:55:58.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.078+0000 7fef19664640 1 -- 172.21.15.3:0/2876910113 wait complete. 2024-02-09T22:55:58.093 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:58.093 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:55:58.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 -- 172.21.15.3:0/228621217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069d50 msgr2=0x7f147410e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 --2- 172.21.15.3:0/228621217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069d50 0x7f147410e0b0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f146c009ca0 tx=0x7f146c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 -- 172.21.15.3:0/228621217 shutdown_connections 2024-02-09T22:55:58.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 --2- 172.21.15.3:0/228621217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069390 0x7f1474069810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 --2- 172.21.15.3:0/228621217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069d50 0x7f147410e0b0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 --2- 172.21.15.3:0/228621217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741061e0 0x7f14741065c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 -- 172.21.15.3:0/228621217 >> 172.21.15.3:0/228621217 conn(0x7f1474077fa0 msgr2=0x7f14740783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 -- 172.21.15.3:0/228621217 shutdown_connections 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.207+0000 7f147c1d7640 1 -- 172.21.15.3:0/228621217 wait complete. 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 Processor -- start 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 -- start start 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 0x7f147419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069d50 0x7f147419d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741061e0 0x7f14741975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474197c30 con 0x7f1474069390 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474197da0 con 0x7f1474069d50 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147c1d7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1474197f60 con 0x7f14741061e0 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 0x7f147419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 0x7f147419d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42528/0 (socket says 172.21.15.3:42528) 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 -- 172.21.15.3:0/1714279672 learned_addr learned my addr 172.21.15.3:0/1714279672 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479772640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069d50 0x7f147419d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f147a774640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741061e0 0x7f14741975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 -- 172.21.15.3:0/1714279672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741061e0 msgr2=0x7f14741975e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741061e0 0x7f14741975e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 -- 172.21.15.3:0/1714279672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069d50 msgr2=0x7f147419d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.208+0000 7f1479f73640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069d50 0x7f147419d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.209+0000 7f1479f73640 1 -- 172.21.15.3:0/1714279672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c009900 con 0x7f1474069390 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.209+0000 7f1479772640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069d50 0x7f147419d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.209+0000 7f1479f73640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 0x7f147419d430 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f147000e9f0 tx=0x7f147000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.209+0000 7f146affd640 1 -- 172.21.15.3:0/1714279672 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470009720 con 0x7f1474069390 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.209+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1474198240 con 0x7f1474069390 2024-02-09T22:55:58.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.209+0000 7f146affd640 1 -- 172.21.15.3:0/1714279672 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f147000cd80 con 0x7f1474069390 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.210+0000 7f146affd640 1 -- 172.21.15.3:0/1714279672 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1470010690 con 0x7f1474069390 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.210+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14741a44d0 con 0x7f1474069390 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.211+0000 7f146affd640 1 -- 172.21.15.3:0/1714279672 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f14700046c0 con 0x7f1474069390 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.212+0000 7f146affd640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14600a2230 0x7f14600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.212+0000 7f1479772640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14600a2230 0x7f14600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.212+0000 7f146affd640 1 -- 172.21.15.3:0/1714279672 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1470014070 con 0x7f1474069390 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.212+0000 7f147edad740 1 Processor -- start 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.212+0000 7f1479772640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14600a2230 0x7f14600a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f14740785a0 tx=0x7f1464005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.212+0000 7f147edad740 1 -- start start 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147edad740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d8fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147edad740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 0x563261d98990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147edad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d98f60 0x563261d9b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147edad740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261d4b020 con 0x563261d8fa30 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147edad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261cfe320 con 0x563261d98f60 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147edad740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261cfb850 con 0x563261d903e0 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d8fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 0x563261d98990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d8fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42544/0 (socket says 172.21.15.3:42544) 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 0x563261d98990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41698/0 (socket says 172.21.15.3:41698) 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 -- 172.21.15.3:0/3880851781 learned_addr learned my addr 172.21.15.3:0/3880851781 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f147a774640 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d98f60 0x563261d9b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 -- 172.21.15.3:0/3880851781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 msgr2=0x563261d98990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 0x563261d98990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 -- 172.21.15.3:0/3880851781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d98f60 msgr2=0x563261d9b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d98f60 0x563261d9b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 -- 172.21.15.3:0/3880851781 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563261cb9210 con 0x563261d8fa30 2024-02-09T22:55:58.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479772640 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 0x563261d98990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f1479f73640 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d8fe10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1470097240 tx=0x7f147009a800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e00f1a76feec01b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.213+0000 7f145ffff640 1 -- 172.21.15.3:0/3880851781 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14700a3580 con 0x563261d8fa30 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f145ffff640 1 -- 172.21.15.3:0/3880851781 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14700cbdc0 con 0x563261d8fa30 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f145ffff640 1 -- 172.21.15.3:0/3880851781 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f147009a450 con 0x563261d8fa30 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 -- 172.21.15.3:0/3880851781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 msgr2=0x563261d8fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d8fe10 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f1470097240 tx=0x7f147009a800 comp rx=0 tx=0).stop 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 -- 172.21.15.3:0/3880851781 shutdown_connections 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d98f60 0x563261d9b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d903e0 0x563261d98990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 --2- 172.21.15.3:0/3880851781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d8fe10 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 -- 172.21.15.3:0/3880851781 >> 172.21.15.3:0/3880851781 conn(0x563261d8a100 msgr2=0x563261d8a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 -- 172.21.15.3:0/3880851781 shutdown_connections 2024-02-09T22:55:58.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.214+0000 7f147edad740 1 -- 172.21.15.3:0/3880851781 wait complete. 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 -- start start 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 Processor -- start 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d8fa30 0x563261d95f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 0x563261d964d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d96a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261cf7030 con 0x563261d903e0 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632619ca430 con 0x563261d8fa30 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261986e00 con 0x563261d98f60 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147edad740 10 client.0 _ll_register_callbacks cb 0x563261cec030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f1479772640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 0x563261d964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f1479772640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 0x563261d964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42548/0 (socket says 172.21.15.3:42548) 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f147a774640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d96a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f1479772640 1 -- 172.21.15.3:0/609860391 learned_addr learned my addr 172.21.15.3:0/609860391 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.215+0000 7f1479f73640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d8fa30 0x563261d95f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f1479772640 1 -- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 msgr2=0x563261d96a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f1479772640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d96a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f1479772640 1 -- 172.21.15.3:0/609860391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d8fa30 msgr2=0x563261d95f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f1479772640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d8fa30 0x563261d95f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f1479772640 1 -- 172.21.15.3:0/609860391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14741a44d0 con 0x563261d903e0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f147a774640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d96a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f1479772640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 0x563261d964d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1464003940 tx=0x7f1464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eda1391c30e21f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f145d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f145d7fa640 1 -- 172.21.15.3:0/609860391 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400fd10 con 0x563261d903e0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f145d7fa640 1 -- 172.21.15.3:0/609860391 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f146400fe70 con 0x563261d903e0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5632619f67e0 con 0x563261d903e0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f145d7fa640 1 -- 172.21.15.3:0/609860391 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1464028440 con 0x563261d903e0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f147edad740 10 client.17208 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x563261cf56b0 con 0x563261d903e0 2024-02-09T22:55:58.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.216+0000 7f145d7fa640 1 -- 172.21.15.3:0/609860391 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5632619f67e0 con 0x563261d903e0 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f145d7fa640 1 -- 172.21.15.3:0/609860391 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f14640306c0 con 0x563261d903e0 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f147edad740 10 client.17208 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f147edad740 10 client.17208 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f147edad740 20 client.17208 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f147edad740 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261da9590 0x563261dab9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f147edad740 4 client.17208 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563261a33720 con 0x563261da9590 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f1479f73640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261da9590 0x563261dab9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f145d7fa640 10 client.17208 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.217+0000 7f1479f73640 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261da9590 0x563261dab9b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=85196582784c79f server_cookie=51ebf320eb9f852b in_seq=0 out_seq=0 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f145d7fa640 1 -- 172.21.15.3:0/609860391 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563261a33720 con 0x563261da9590 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f145d7fa640 10 client.17208 handle_command_reply: tid=0 2024-02-09T22:55:58.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 client.17208 shutdown 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 msgr2=0x563261d964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 0x563261d964d0 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f1464003940 tx=0x7f1464004730 comp rx=0 tx=0).stop 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 shutdown_connections 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261da9590 0x563261dab9b0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d96a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d903e0 0x563261d964d0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f145d7fa640 0 client.17208 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 --2- 172.21.15.3:0/609860391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d8fa30 0x563261d95f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 >> 172.21.15.3:0/609860391 conn(0x563261d8a100 msgr2=0x563261d8be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 shutdown_connections 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 1 -- 172.21.15.3:0/609860391 wait complete. 2024-02-09T22:55:58.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.218+0000 7f147edad740 20 client.17208 trim_cache size 0 max 16384 2024-02-09T22:55:58.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 Processor -- start 2024-02-09T22:55:58.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 -- start start 2024-02-09T22:55:58.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d986c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d96830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d912b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261c7c130 con 0x563261d8fa30 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261d89f60 con 0x563261d903e0 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.231+0000 7f147edad740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261db0bc0 con 0x563261d98f60 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f147a774640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d912b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42556/0 (socket says 172.21.15.3:42556) 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 -- 172.21.15.3:0/3702977575 learned_addr learned my addr 172.21.15.3:0/3702977575 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479772640 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d96830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 -- 172.21.15.3:0/3702977575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 msgr2=0x563261d912b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d912b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 -- 172.21.15.3:0/3702977575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 msgr2=0x563261d96830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d96830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 -- 172.21.15.3:0/3702977575 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563261c91c90 con 0x563261d8fa30 2024-02-09T22:55:58.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f147a774640 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d912b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f1479f73640 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d986c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f147009ad70 tx=0x7f1470004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6faa1d0ed40cce1f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f145f7fe640 1 -- 172.21.15.3:0/3702977575 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14700a3580 con 0x563261d8fa30 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f145f7fe640 1 -- 172.21.15.3:0/3702977575 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f147009ef30 con 0x563261d8fa30 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.232+0000 7f145f7fe640 1 -- 172.21.15.3:0/3702977575 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14700acb80 con 0x563261d8fa30 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 -- 172.21.15.3:0/3702977575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 msgr2=0x563261d986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d986c0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f147009ad70 tx=0x7f1470004b30 comp rx=0 tx=0).stop 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 -- 172.21.15.3:0/3702977575 shutdown_connections 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d96830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d98f60 0x563261d912b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 --2- 172.21.15.3:0/3702977575 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d8fa30 0x563261d986c0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 -- 172.21.15.3:0/3702977575 >> 172.21.15.3:0/3702977575 conn(0x563261d8a100 msgr2=0x563261d907c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 -- 172.21.15.3:0/3702977575 shutdown_connections 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 -- 172.21.15.3:0/3702977575 wait complete. 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 -- start start 2024-02-09T22:55:58.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.233+0000 7f147edad740 1 Processor -- start 2024-02-09T22:55:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d8fa30 0x563261da01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d9e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 0x563261d9ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261d9f060 con 0x563261d98f60 2024-02-09T22:55:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261d9f1a0 con 0x563261d903e0 2024-02-09T22:55:58.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563261d9f360 con 0x563261d8fa30 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147edad740 10 client.0 _ll_register_callbacks cb 0x7f147000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147a774640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 0x563261d9ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479f73640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d8fa30 0x563261da01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479772640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d9e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147a774640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 0x563261d9ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42570/0 (socket says 172.21.15.3:42570) 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147a774640 1 -- 172.21.15.3:0/3326822964 learned_addr learned my addr 172.21.15.3:0/3326822964 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479f73640 1 -- 172.21.15.3:0/3326822964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 msgr2=0x563261d9e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479f73640 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d9e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479f73640 1 -- 172.21.15.3:0/3326822964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 msgr2=0x563261d9ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479f73640 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 0x563261d9ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f1479f73640 1 -- 172.21.15.3:0/3326822964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147410e920 con 0x563261d8fa30 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.234+0000 7f147a774640 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 0x563261d9ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f1479f73640 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d8fa30 0x563261da01f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f14700977b0 tx=0x7f14700cbbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a83cddcfc71e552 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f145d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f145d7fa640 1 -- 172.21.15.3:0/3326822964 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14700a3580 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f145d7fa640 1 -- 172.21.15.3:0/3326822964 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14700990e0 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f14700cbdc0 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f145d7fa640 1 -- 172.21.15.3:0/3326822964 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1470097450 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f147edad740 10 client.26551 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7f14700b7040 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f145d7fa640 1 -- 172.21.15.3:0/3326822964 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14700cbdc0 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.235+0000 7f145d7fa640 1 -- 172.21.15.3:0/3326822964 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f147001f060 con 0x563261d8fa30 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f147edad740 10 client.26551 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f147edad740 10 client.26551 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f147edad740 20 client.26551 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f147edad740 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261e00b50 0x563261e02f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f147edad740 4 client.26551 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563261a82890 con 0x563261e00b50 2024-02-09T22:55:58.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f1479772640 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261e00b50 0x563261e02f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f1479772640 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261e00b50 0x563261e02f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=f6d77b281e52e0e5 server_cookie=644820f865ba1ecb in_seq=0 out_seq=0 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.236+0000 7f145d7fa640 10 client.26551 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f145d7fa640 1 -- 172.21.15.3:0/3326822964 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563261a82890 con 0x563261e00b50 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f145d7fa640 10 client.26551 handle_command_reply: tid=0 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 client.26551 shutdown 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d8fa30 msgr2=0x563261da01f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d8fa30 0x563261da01f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f14700977b0 tx=0x7f14700cbbf0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 shutdown_connections 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x563261e00b50 0x563261e02f40 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563261d903e0 0x563261d9e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f145d7fa640 0 client.26551 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563261d98f60 0x563261d9ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 --2- 172.21.15.3:0/3326822964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563261d8fa30 0x563261da01f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 >> 172.21.15.3:0/3326822964 conn(0x563261d8a100 msgr2=0x563261d912b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 shutdown_connections 2024-02-09T22:55:58.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 1 -- 172.21.15.3:0/3326822964 wait complete. 2024-02-09T22:55:58.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.239+0000 7f147edad740 20 client.26551 trim_cache size 0 max 16384 2024-02-09T22:55:58.246 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14600a2230 msgr2=0x7f14600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14600a2230 0x7f14600a46f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f14740785a0 tx=0x7f1464005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 msgr2=0x7f147419d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 0x7f147419d430 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f147000e9f0 tx=0x7f147000edf0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 shutdown_connections 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14600a2230 0x7f14600a46f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1474069d50 0x7f147419d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14741061e0 0x7f14741975e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 --2- 172.21.15.3:0/1714279672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1474069390 0x7f147419d430 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 >> 172.21.15.3:0/1714279672 conn(0x7f1474077fa0 msgr2=0x7f14740fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 shutdown_connections 2024-02-09T22:55:58.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.240+0000 7f147c1d7640 1 -- 172.21.15.3:0/1714279672 wait complete. 2024-02-09T22:55:58.253 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:58.253 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:55:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 -- 172.21.15.3:0/564144192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc100340 msgr2=0x7fb3dc100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 --2- 172.21.15.3:0/564144192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc100340 0x7fb3dc100720 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009930 tx=0x7fb3d802bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 -- 172.21.15.3:0/564144192 shutdown_connections 2024-02-09T22:55:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 --2- 172.21.15.3:0/564144192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3dc100cf0 0x7fb3dc104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 --2- 172.21.15.3:0/564144192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc105410 0x7fb3dc10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 --2- 172.21.15.3:0/564144192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc100340 0x7fb3dc100720 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 -- 172.21.15.3:0/564144192 >> 172.21.15.3:0/564144192 conn(0x7fb3dc0772e0 msgr2=0x7fb3dc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 -- 172.21.15.3:0/564144192 shutdown_connections 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.364+0000 7fb3e3488640 1 -- 172.21.15.3:0/564144192 wait complete. 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 Processor -- start 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 -- start start 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3dc100340 0x7fb3dc19f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 0x7fb3dc19fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc199880 0x7fb3dc199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc19a270 con 0x7fb3dc105410 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc19a3e0 con 0x7fb3dc100340 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e3488640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3dc19a5a0 con 0x7fb3dc199880 2024-02-09T22:55:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e0a23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 0x7fb3dc19fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e1a25640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc199880 0x7fb3dc199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e0a23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 0x7fb3dc19fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42598/0 (socket says 172.21.15.3:42598) 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1647223672 learned_addr learned my addr 172.21.15.3:0/1647223672 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e1224640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3dc100340 0x7fb3dc19f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1647223672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc199880 msgr2=0x7fb3dc199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.365+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc199880 0x7fb3dc199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1647223672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3dc100340 msgr2=0x7fb3dc19f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3dc100340 0x7fb3dc19f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1647223672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d8009560 con 0x7fb3dc105410 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 0x7fb3dc19fc10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc005fd0 tx=0x7fb3cc00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3d27fc640 1 -- 172.21.15.3:0/1647223672 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc009c30 con 0x7fb3dc105410 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3dc1a6590 con 0x7fb3dc105410 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3d27fc640 1 -- 172.21.15.3:0/1647223672 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc009d90 con 0x7fb3dc105410 2024-02-09T22:55:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3d27fc640 1 -- 172.21.15.3:0/1647223672 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc010460 con 0x7fb3dc105410 2024-02-09T22:55:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.366+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc1a6aa0 con 0x7fb3dc105410 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.368+0000 7fb3d27fc640 1 -- 172.21.15.3:0/1647223672 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb3cc003f20 con 0x7fb3dc105410 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 Processor -- start 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 -- start start 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3d27fc640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3c80a2230 0x7fb3c80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6ab6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab73e0 0x562bf6abf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6abff60 0x562bf6ac23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6a72020 con 0x562bf6ab6a30 2024-02-09T22:55:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6a25320 con 0x562bf6ab73e0 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e6069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6a22850 con 0x562bf6abff60 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3d27fc640 1 -- 172.21.15.3:0/1647223672 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3cc0cdce0 con 0x7fb3dc105410 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3c80a2230 0x7fb3c80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e1224640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6ab6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e1224640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6ab6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42612/0 (socket says 172.21.15.3:42612) 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e1224640 1 -- 172.21.15.3:0/2817592671 learned_addr learned my addr 172.21.15.3:0/2817592671 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.369+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6abff60 0x562bf6ac23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab73e0 0x562bf6abf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1224640 1 -- 172.21.15.3:0/2817592671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6abff60 msgr2=0x562bf6ac23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1224640 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6abff60 0x562bf6ac23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1224640 1 -- 172.21.15.3:0/2817592671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab73e0 msgr2=0x562bf6abf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1224640 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab73e0 0x562bf6abf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1224640 1 -- 172.21.15.3:0/2817592671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bf69e0210 con 0x562bf6ab6a30 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3c80a2230 0x7fb3c80a46f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4007740 tx=0x7fb3d4007d30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3e1224640 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6ab6e10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8036040 tx=0x7fb3d8005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff01c3b19dc04b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3c77fe640 1 -- 172.21.15.3:0/2817592671 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8004a40 con 0x562bf6ab6a30 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3c77fe640 1 -- 172.21.15.3:0/2817592671 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d8004030 con 0x562bf6ab6a30 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.370+0000 7fb3c77fe640 1 -- 172.21.15.3:0/2817592671 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80052f0 con 0x562bf6ab6a30 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 -- 172.21.15.3:0/2817592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 msgr2=0x562bf6ab6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6ab6e10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8036040 tx=0x7fb3d8005e80 comp rx=0 tx=0).stop 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 -- 172.21.15.3:0/2817592671 shutdown_connections 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab73e0 0x562bf6abf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6abff60 0x562bf6ac23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2817592671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6ab6e10 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 -- 172.21.15.3:0/2817592671 >> 172.21.15.3:0/2817592671 conn(0x562bf6ab1100 msgr2=0x562bf6ab14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 -- 172.21.15.3:0/2817592671 shutdown_connections 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 -- 172.21.15.3:0/2817592671 wait complete. 2024-02-09T22:55:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 -- start start 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.371+0000 7fb3e6069740 1 Processor -- start 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6abceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6abd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6abd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6abd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e1224640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6abceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6a1e030 con 0x562bf6ab6a30 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6abd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41742/0 (socket says 172.21.15.3:41742) 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1494824719 learned_addr learned my addr 172.21.15.3:0/1494824719 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e1224640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6abceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42614/0 (socket says 172.21.15.3:42614) 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf66f1430 con 0x562bf6abff60 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf66ade00 con 0x562bf6ab73e0 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e6069740 10 client.0 _ll_register_callbacks cb 0x562bf6a13030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6abd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1494824719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 msgr2=0x562bf6abd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6abd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1494824719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 msgr2=0x562bf6abceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6abceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 -- 172.21.15.3:0/1494824719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3dc10d7f0 con 0x562bf6ab73e0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6abd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e1224640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6abceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6abd3f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc096b60 tx=0x7fb3cc09ae00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf542753fe8ac1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.372+0000 7fb3c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/1494824719 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0a2ea0 con 0x562bf6ab73e0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/1494824719 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3cc0cb6e0 con 0x562bf6ab73e0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bf671d7e0 con 0x562bf6ab73e0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/1494824719 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3cc0cb8a0 con 0x562bf6ab73e0 2024-02-09T22:55:58.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3e6069740 10 client.26560 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x562bf6a1c6b0 con 0x562bf6ab73e0 2024-02-09T22:55:58.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/1494824719 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc0b3250 con 0x562bf6ab73e0 2024-02-09T22:55:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.373+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/1494824719 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3cc022020 con 0x562bf6ab73e0 2024-02-09T22:55:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e6069740 10 client.26560 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e6069740 10 client.26560 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:55:58.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e6069740 20 client.26560 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e6069740 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6ad0460 0x562bf6ad2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e6069740 4 client.26560 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bf671d7e0 con 0x562bf6ad0460 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e1224640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6ad0460 0x562bf6ad2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3c4ff9640 10 client.26560 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.374+0000 7fb3e1224640 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6ad0460 0x562bf6ad2920 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8e5eac8a19aadb78 server_cookie=59d8734b3b925915 in_seq=0 out_seq=0 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/1494824719 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562bf671d7e0 con 0x562bf6ad0460 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3c4ff9640 10 client.26560 handle_command_reply: tid=0 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 client.26560 shutdown 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 msgr2=0x562bf6abd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6abd3f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc096b60 tx=0x7fb3cc09ae00 comp rx=0 tx=0).stop 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 shutdown_connections 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6ad0460 0x562bf6ad2920 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6abd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6abd3f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 --2- 172.21.15.3:0/1494824719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab6a30 0x562bf6abceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3c4ff9640 0 client.26560 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 >> 172.21.15.3:0/1494824719 conn(0x562bf6ab1100 msgr2=0x562bf6ab2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 shutdown_connections 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 1 -- 172.21.15.3:0/1494824719 wait complete. 2024-02-09T22:55:58.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.375+0000 7fb3e6069740 20 client.26560 trim_cache size 0 max 16384 2024-02-09T22:55:58.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:58 smithi003 ceph-mon[32678]: pgmap v144: 97 pgs: 97 active+clean; 56 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:55:58.384 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:55:58 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2876910113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 Processor -- start 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 -- start start 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 0x562bf6abf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6ad72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 0x562bf6aba150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6aac8b0 con 0x562bf6abff60 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf69a3130 con 0x562bf6ab6a30 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e6069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf671d810 con 0x562bf6ab73e0 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e1a25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 0x562bf6aba150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e1224640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 0x562bf6abf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e1224640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 0x562bf6abf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52712/0 (socket says 172.21.15.3:52712) 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e1a25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 0x562bf6aba150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42622/0 (socket says 172.21.15.3:42622) 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e1224640 1 -- 172.21.15.3:0/2391086602 learned_addr learned my addr 172.21.15.3:0/2391086602 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.387+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6ad72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1a25640 1 -- 172.21.15.3:0/2391086602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 msgr2=0x562bf6ad72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6ad72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1a25640 1 -- 172.21.15.3:0/2391086602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 msgr2=0x562bf6abf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 0x562bf6abf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1a25640 1 -- 172.21.15.3:0/2391086602 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bf69b8c90 con 0x562bf6abff60 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6ad72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1224640 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 0x562bf6abf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 0x562bf6aba150 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40142f0 tx=0x7fb3d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449b08a1aa3c7027 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3c6ffd640 1 -- 172.21.15.3:0/2391086602 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4012de0 con 0x562bf6abff60 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3c6ffd640 1 -- 172.21.15.3:0/2391086602 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d401a040 con 0x562bf6abff60 2024-02-09T22:55:58.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.388+0000 7fb3c6ffd640 1 -- 172.21.15.3:0/2391086602 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d4005740 con 0x562bf6abff60 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 -- 172.21.15.3:0/2391086602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 msgr2=0x562bf6aba150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 0x562bf6aba150 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fb3d40142f0 tx=0x7fb3d4004ab0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 -- 172.21.15.3:0/2391086602 shutdown_connections 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6abff60 0x562bf6aba150 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab73e0 0x562bf6ad72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 --2- 172.21.15.3:0/2391086602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6ab6a30 0x562bf6abf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 -- 172.21.15.3:0/2391086602 >> 172.21.15.3:0/2391086602 conn(0x562bf6ab1100 msgr2=0x562bf6ab77c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 -- 172.21.15.3:0/2391086602 shutdown_connections 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 -- 172.21.15.3:0/2391086602 wait complete. 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 -- start start 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.389+0000 7fb3e6069740 1 Processor -- start 2024-02-09T22:55:58.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab6a30 0x562bf6b20f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 0x562bf6b21440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6ac6220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6ac67e0 con 0x562bf6ab73e0 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6ac6950 con 0x562bf6abff60 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bf6ac6b10 con 0x562bf6ab6a30 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 10 client.0 _ll_register_callbacks cb 0x7fb3d8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 0x562bf6b21440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 0x562bf6b21440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42628/0 (socket says 172.21.15.3:42628) 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 -- 172.21.15.3:0/3794185384 learned_addr learned my addr 172.21.15.3:0/3794185384 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e1224640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab6a30 0x562bf6b20f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e1a25640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6ac6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 -- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab6a30 msgr2=0x562bf6b20f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab6a30 0x562bf6b20f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 -- 172.21.15.3:0/3794185384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 msgr2=0x562bf6ac6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6ac6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 -- 172.21.15.3:0/3794185384 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bf69e0210 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e0a23640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 0x562bf6b21440 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc09a8c0 tx=0x7fb3cc09e910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2045ee439d240e6a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/3794185384 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3cc0b3250 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/3794185384 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3cc0a2ea0 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.390+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3d8004030 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/3794185384 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3cc09a240 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 10 client.17232 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x562bf6a1c6b0 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/3794185384 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3cc07d020 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/3794185384 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3cc022070 con 0x562bf6ab73e0 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 10 client.17232 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 10 client.17232 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:55:58.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 20 client.17232 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6b28a10 0x562bf6b2ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 4 client.17232 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562bf675a720 con 0x562bf6b28a10 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.391+0000 7fb3e1224640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6b28a10 0x562bf6b2ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3c4ff9640 10 client.17232 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e1224640 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6b28a10 0x562bf6b2ae30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=987bbbc064da2f19 server_cookie=963b91ef68e82360 in_seq=0 out_seq=0 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3c4ff9640 1 -- 172.21.15.3:0/3794185384 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562bf675a720 con 0x562bf6b28a10 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3c4ff9640 10 client.17232 handle_command_reply: tid=0 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 client.17232 shutdown 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 msgr2=0x562bf6b21440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 0x562bf6b21440 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc09a8c0 tx=0x7fb3cc09e910 comp rx=0 tx=0).stop 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 shutdown_connections 2024-02-09T22:55:58.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x562bf6b28a10 0x562bf6b2ae30 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bf6abff60 0x562bf6ac6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3c4ff9640 0 client.17232 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bf6ab73e0 0x562bf6b21440 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 --2- 172.21.15.3:0/3794185384 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bf6ab6a30 0x562bf6b20f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 >> 172.21.15.3:0/3794185384 conn(0x562bf6ab1100 msgr2=0x562bf6ad72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 shutdown_connections 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 1 -- 172.21.15.3:0/3794185384 wait complete. 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e6069740 20 client.17232 trim_cache size 0 max 16384 2024-02-09T22:55:58.401 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:58.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.392+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3c80a2230 msgr2=0x7fb3c80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3c80a2230 0x7fb3c80a46f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb3d4007740 tx=0x7fb3d4007d30 comp rx=0 tx=0).stop 2024-02-09T22:55:58.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 msgr2=0x7fb3dc19fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 0x7fb3dc19fc10 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fb3cc005fd0 tx=0x7fb3cc00dde0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 shutdown_connections 2024-02-09T22:55:58.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3c80a2230 0x7fb3c80a46f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3dc199880 0x7fb3dc199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3dc105410 0x7fb3dc19fc10 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 --2- 172.21.15.3:0/1647223672 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3dc100340 0x7fb3dc19f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 >> 172.21.15.3:0/1647223672 conn(0x7fb3dc0772e0 msgr2=0x7fb3dc0fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 shutdown_connections 2024-02-09T22:55:58.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.393+0000 7fb3e3488640 1 -- 172.21.15.3:0/1647223672 wait complete. 2024-02-09T22:55:58.406 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:58.407 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:55:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:58 smithi089 ceph-mon[36665]: pgmap v144: 97 pgs: 97 active+clean; 56 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:55:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:55:58 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2876910113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:55:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 -- 172.21.15.3:0/2372794338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 msgr2=0x7fbe6810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 --2- 172.21.15.3:0/2372794338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6810cfe0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009cd0 tx=0x7fbe5802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 -- 172.21.15.3:0/2372794338 shutdown_connections 2024-02-09T22:55:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 --2- 172.21.15.3:0/2372794338 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100cf0 0x7fbe68104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 --2- 172.21.15.3:0/2372794338 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6810cfe0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 --2- 172.21.15.3:0/2372794338 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe68100340 0x7fbe68100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 -- 172.21.15.3:0/2372794338 >> 172.21.15.3:0/2372794338 conn(0x7fbe680772e0 msgr2=0x7fbe680776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 -- 172.21.15.3:0/2372794338 shutdown_connections 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.516+0000 7fbe70772640 1 -- 172.21.15.3:0/2372794338 wait complete. 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 Processor -- start 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 -- start start 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100340 0x7fbe6819d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6819db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe68197690 0x7fbe68197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68198160 con 0x7fbe68105410 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe681982d0 con 0x7fbe68100340 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe70772640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe68198490 con 0x7fbe68197690 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6819db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6819db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42658/0 (socket says 172.21.15.3:42658) 2024-02-09T22:55:58.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 -- 172.21.15.3:0/4176330810 learned_addr learned my addr 172.21.15.3:0/4176330810 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6e50e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100340 0x7fbe6819d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe68197690 0x7fbe68197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 -- 172.21.15.3:0/4176330810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe68197690 msgr2=0x7fbe68197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe68197690 0x7fbe68197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 -- 172.21.15.3:0/4176330810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100340 msgr2=0x7fbe6819d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100340 0x7fbe6819d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6dd0d640 1 -- 172.21.15.3:0/4176330810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe58009900 con 0x7fbe68105410 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.517+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100340 0x7fbe6819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.518+0000 7fbe6dd0d640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6819db10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00e9c0 tx=0x7fbe5c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.518+0000 7fbe677fe640 1 -- 172.21.15.3:0/4176330810 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c009700 con 0x7fbe68105410 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.518+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe681a4430 con 0x7fbe68105410 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.518+0000 7fbe677fe640 1 -- 172.21.15.3:0/4176330810 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c00cd80 con 0x7fbe68105410 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.518+0000 7fbe677fe640 1 -- 172.21.15.3:0/4176330810 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c010430 con 0x7fbe68105410 2024-02-09T22:55:58.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.518+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe681a4960 con 0x7fbe68105410 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.520+0000 7fbe677fe640 1 -- 172.21.15.3:0/4176330810 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fbe5c005000 con 0x7fbe68105410 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.520+0000 7fbe73416740 1 Processor -- start 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 -- start start 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff06cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff0fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff103c0 0x56466ff10820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe677fe640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbe500a2230 0x7fbe500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fec2020 con 0x56466ff06910 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fe75320 con 0x56466ff103c0 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe73416740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fe72850 con 0x56466ff072c0 2024-02-09T22:55:58.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff06cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6dd0d640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbe500a2230 0x7fbe500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff06cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42662/0 (socket says 172.21.15.3:42662) 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2139109101 learned_addr learned my addr 172.21.15.3:0/2139109101 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe677fe640 1 -- 172.21.15.3:0/4176330810 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbe5c014070 con 0x7fbe68105410 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff103c0 0x56466ff10820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff0fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2139109101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 msgr2=0x56466ff0fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff0fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2139109101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff103c0 msgr2=0x56466ff10820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff103c0 0x56466ff10820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2139109101 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56466fe30210 con 0x56466ff06910 2024-02-09T22:55:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.521+0000 7fbe6dd0d640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbe500a2230 0x7fbe500a46f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c09adb0 tx=0x7fbe5c004a50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.522+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff06cf0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbe58004af0 tx=0x7fbe58009820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f6cca6fa10bf164 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.522+0000 7fbe64ff9640 1 -- 172.21.15.3:0/2139109101 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58034440 con 0x56466ff06910 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.522+0000 7fbe64ff9640 1 -- 172.21.15.3:0/2139109101 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe580345a0 con 0x56466ff06910 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.522+0000 7fbe64ff9640 1 -- 172.21.15.3:0/2139109101 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58003bd0 con 0x56466ff06910 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.522+0000 7fbe73416740 1 -- 172.21.15.3:0/2139109101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 msgr2=0x56466ff06cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.522+0000 7fbe73416740 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff06cf0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7fbe58004af0 tx=0x7fbe58009820 comp rx=0 tx=0).stop 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- 172.21.15.3:0/2139109101 shutdown_connections 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff103c0 0x56466ff10820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff0fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 --2- 172.21.15.3:0/2139109101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff06cf0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- 172.21.15.3:0/2139109101 >> 172.21.15.3:0/2139109101 conn(0x56466ff01100 msgr2=0x56466ff014e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- 172.21.15.3:0/2139109101 shutdown_connections 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- 172.21.15.3:0/2139109101 wait complete. 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- start start 2024-02-09T22:55:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 Processor -- start 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff13e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff143d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff14910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fe6e030 con 0x56466ff06910 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fb41430 con 0x56466ff072c0 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fafde00 con 0x56466ff103c0 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe73416740 10 client.0 _ll_register_callbacks cb 0x56466fe63030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff13e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff13e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42664/0 (socket says 172.21.15.3:42664) 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2062061549 learned_addr learned my addr 172.21.15.3:0/2062061549 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.523+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff143d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff14910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 msgr2=0x56466ff14910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff14910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2062061549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 msgr2=0x56466ff143d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff143d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/2062061549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe6810d7f0 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff13e90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009ca0 tx=0x7fbe58032a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20af7375bacd254a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe4dffb640 1 -- 172.21.15.3:0/2062061549 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58034ea0 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe4dffb640 1 -- 172.21.15.3:0/2062061549 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbe58003970 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56466fb6d7e0 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe4dffb640 1 -- 172.21.15.3:0/2062061549 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe58003b30 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe73416740 10 client.17244 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x56466fe6c6b0 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.524+0000 7fbe4dffb640 1 -- 172.21.15.3:0/2062061549 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56466fb6d7e0 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe4dffb640 1 -- 172.21.15.3:0/2062061549 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe5804e450 con 0x56466ff06910 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe73416740 10 client.17244 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe73416740 10 client.17244 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:55:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe73416740 20 client.17244 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe73416740 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff24600 0x56466ff269f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe73416740 4 client.17244 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56466fbaa720 con 0x56466ff24600 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff24600 0x56466ff269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff24600 0x56466ff269f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=657e1212d9b26132 server_cookie=352fa674c1f4b6ad in_seq=0 out_seq=0 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.525+0000 7fbe4dffb640 10 client.17244 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe4dffb640 1 -- 172.21.15.3:0/2062061549 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56466fbaa720 con 0x56466ff24600 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe4dffb640 10 client.17244 handle_command_reply: tid=0 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 client.17244 shutdown 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 msgr2=0x56466ff13e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff13e90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009ca0 tx=0x7fbe58032a60 comp rx=0 tx=0).stop 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 shutdown_connections 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff24600 0x56466ff269f0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff143d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff14910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 --2- 172.21.15.3:0/2062061549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff13e90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe4dffb640 0 client.17244 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 >> 172.21.15.3:0/2062061549 conn(0x56466ff01100 msgr2=0x56466ff02e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 shutdown_connections 2024-02-09T22:55:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 1 -- 172.21.15.3:0/2062061549 wait complete. 2024-02-09T22:55:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.526+0000 7fbe73416740 20 client.17244 trim_cache size 0 max 16384 2024-02-09T22:55:58.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 Processor -- start 2024-02-09T22:55:58.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 -- start start 2024-02-09T22:55:58.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff165c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff0e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466fdf3130 con 0x56466ff06910 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466ff213b0 con 0x56466ff072c0 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.538+0000 7fbe73416740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466ff01ed0 con 0x56466ff103c0 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff165c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff165c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42666/0 (socket says 172.21.15.3:42666) 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/3957983849 learned_addr learned my addr 172.21.15.3:0/3957983849 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff0e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/3957983849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 msgr2=0x56466ff117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff117b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/3957983849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 msgr2=0x56466ff0e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff0e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 -- 172.21.15.3:0/3957983849 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56466fe08c90 con 0x56466ff06910 2024-02-09T22:55:58.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff165c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbe58032a30 tx=0x7fbe5802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=144709a1b6770548 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe4ffff640 1 -- 172.21.15.3:0/3957983849 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58032b70 con 0x56466ff06910 2024-02-09T22:55:58.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe4ffff640 1 -- 172.21.15.3:0/3957983849 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe58003e50 con 0x56466ff06910 2024-02-09T22:55:58.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.539+0000 7fbe4ffff640 1 -- 172.21.15.3:0/3957983849 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe58004040 con 0x56466ff06910 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 -- 172.21.15.3:0/3957983849 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 msgr2=0x56466ff165c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff165c0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fbe58032a30 tx=0x7fbe5802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 -- 172.21.15.3:0/3957983849 shutdown_connections 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff072c0 0x56466ff0e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff103c0 0x56466ff117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 --2- 172.21.15.3:0/3957983849 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff06910 0x56466ff165c0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 -- 172.21.15.3:0/3957983849 >> 172.21.15.3:0/3957983849 conn(0x56466ff01100 msgr2=0x56466feff9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 -- 172.21.15.3:0/3957983849 shutdown_connections 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 -- 172.21.15.3:0/3957983849 wait complete. 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 -- start start 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.540+0000 7fbe73416740 1 Processor -- start 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff06910 0x56466ff74e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff753a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff103c0 0x56466ff1a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466ff1ab40 con 0x56466ff103c0 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466ff1acb0 con 0x56466ff06910 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56466ff1ae70 con 0x56466ff072c0 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe73416740 10 client.0 _ll_register_callbacks cb 0x7fbe58037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff753a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6ed0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff103c0 0x56466ff1a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff753a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41812/0 (socket says 172.21.15.3:41812) 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 -- 172.21.15.3:0/3914101158 learned_addr learned my addr 172.21.15.3:0/3914101158 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff06910 0x56466ff74e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 -- 172.21.15.3:0/3914101158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff06910 msgr2=0x56466ff74e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff06910 0x56466ff74e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 -- 172.21.15.3:0/3914101158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff103c0 msgr2=0x56466ff1a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff103c0 0x56466ff1a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 -- 172.21.15.3:0/3914101158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56466fe6c6b0 con 0x56466ff072c0 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff06910 0x56466ff74e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe6e50e640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff753a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009fd0 tx=0x7fbe6001e760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2311d7024459658 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe4dffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.541+0000 7fbe4dffb640 1 -- 172.21.15.3:0/3914101158 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe60017030 con 0x56466ff072c0 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe580345a0 con 0x56466ff072c0 2024-02-09T22:55:58.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe4dffb640 1 -- 172.21.15.3:0/3914101158 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbe60005020 con 0x56466ff072c0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe4dffb640 1 -- 172.21.15.3:0/3914101158 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe600051e0 con 0x56466ff072c0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 10 client.26584 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x56466fe2ea70 con 0x56466ff072c0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe4dffb640 1 -- 172.21.15.3:0/3914101158 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe580345a0 con 0x56466ff072c0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe4dffb640 1 -- 172.21.15.3:0/3914101158 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbe600104c0 con 0x56466ff072c0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 10 client.26584 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 10 client.26584 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 20 client.26584 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff7cd40 0x56466ff7f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 4 client.26584 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56466fbf9890 con 0x56466ff7cd40 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.542+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff7cd40 0x56466ff7f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe6ed0f640 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff7cd40 0x56466ff7f160 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=99f388079f03d3c6 server_cookie=ffc570c977cd8f3d in_seq=0 out_seq=0 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe4dffb640 10 client.26584 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:58.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe4dffb640 1 -- 172.21.15.3:0/3914101158 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56466fbf9890 con 0x56466ff7cd40 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe4dffb640 10 client.26584 handle_command_reply: tid=0 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 client.26584 shutdown 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 msgr2=0x56466ff753a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff753a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fbe60009fd0 tx=0x7fbe6001e760 comp rx=0 tx=0).stop 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 shutdown_connections 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56466ff7cd40 0x56466ff7f160 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56466ff103c0 0x56466ff1a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56466ff072c0 0x56466ff753a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe4dffb640 0 client.26584 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 --2- 172.21.15.3:0/3914101158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56466ff06910 0x56466ff74e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 >> 172.21.15.3:0/3914101158 conn(0x56466ff01100 msgr2=0x56466ff146c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 shutdown_connections 2024-02-09T22:55:58.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 1 -- 172.21.15.3:0/3914101158 wait complete. 2024-02-09T22:55:58.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe73416740 20 client.26584 trim_cache size 0 max 16384 2024-02-09T22:55:58.553 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbe500a2230 msgr2=0x7fbe500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbe500a2230 0x7fbe500a46f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c09adb0 tx=0x7fbe5c004a50 comp rx=0 tx=0).stop 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 msgr2=0x7fbe6819db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6819db10 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00e9c0 tx=0x7fbe5c00ee80 comp rx=0 tx=0).stop 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 shutdown_connections 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbe500a2230 0x7fbe500a46f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbe68197690 0x7fbe68197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbe68105410 0x7fbe6819db10 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.543+0000 7fbe70772640 1 --2- 172.21.15.3:0/4176330810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbe68100340 0x7fbe6819d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.544+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 >> 172.21.15.3:0/4176330810 conn(0x7fbe680772e0 msgr2=0x7fbe68068840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.544+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 shutdown_connections 2024-02-09T22:55:58.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.544+0000 7fbe70772640 1 -- 172.21.15.3:0/4176330810 wait complete. 2024-02-09T22:55:58.558 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:58.558 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:55:58.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 137189 2.79 MB/sec execute 112 sec latency 1661.135 ms 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.664+0000 7f80b07dd640 1 -- 172.21.15.3:0/1154672085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a8102d70 msgr2=0x7f80a810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.664+0000 7f80b07dd640 1 --2- 172.21.15.3:0/1154672085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a8102d70 0x7f80a810b250 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f8098009900 tx=0x7f809802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- 172.21.15.3:0/1154672085 shutdown_connections 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 --2- 172.21.15.3:0/1154672085 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80a810b820 0x7f80a810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 --2- 172.21.15.3:0/1154672085 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a8102d70 0x7f80a810b250 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 --2- 172.21.15.3:0/1154672085 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102450 0x7f80a8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- 172.21.15.3:0/1154672085 >> 172.21.15.3:0/1154672085 conn(0x7f80a80772e0 msgr2=0x7f80a80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- 172.21.15.3:0/1154672085 shutdown_connections 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- 172.21.15.3:0/1154672085 wait complete. 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 Processor -- start 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- start start 2024-02-09T22:55:58.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80a8102450 0x7f80a819d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102d70 0x7f80a819dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 0x7f80a8197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8197df0 con 0x7f80a810b820 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8197f60 con 0x7f80a8102450 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.665+0000 7f80b07dd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80a8198120 con 0x7f80a8102d70 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 0x7f80a8197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 0x7f80a8197660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42680/0 (socket says 172.21.15.3:42680) 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 -- 172.21.15.3:0/33474751 learned_addr learned my addr 172.21.15.3:0/33474751 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80add78640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102d70 0x7f80a819dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80ae579640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80a8102450 0x7f80a819d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 -- 172.21.15.3:0/33474751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102d70 msgr2=0x7f80a819dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102d70 0x7f80a819dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 -- 172.21.15.3:0/33474751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80a8102450 msgr2=0x7f80a819d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80a8102450 0x7f80a819d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 -- 172.21.15.3:0/33474751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8098009560 con 0x7f80a810b820 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80add78640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102d70 0x7f80a819dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80aed7a640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 0x7f80a8197660 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f80a000ea60 tx=0x7f80a000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f809f7fe640 1 -- 172.21.15.3:0/33474751 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a000ce40 con 0x7f80a810b820 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f809f7fe640 1 -- 172.21.15.3:0/33474751 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a0005020 con 0x7f80a810b820 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80a8198400 con 0x7f80a810b820 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f809f7fe640 1 -- 172.21.15.3:0/33474751 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0010690 con 0x7f80a810b820 2024-02-09T22:55:58.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.666+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a81a4520 con 0x7f80a810b820 2024-02-09T22:55:58.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.669+0000 7f809f7fe640 1 -- 172.21.15.3:0/33474751 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f80a0005500 con 0x7f80a810b820 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.669+0000 7f809f7fe640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f80940a2230 0x7f80940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.669+0000 7f80b33a8740 1 Processor -- start 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.669+0000 7f80ae579640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f80940a2230 0x7f80940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.669+0000 7f809f7fe640 1 -- 172.21.15.3:0/33474751 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f80a0014070 con 0x7f80a810b820 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 -- start start 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68ef0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efdda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68f00780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68eac020 con 0x55da68ef07b0 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68e5f320 con 0x55da68efe370 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80b33a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68e5c850 con 0x55da68ef1160 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f80940a2230 0x7f80940a46f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009bd0 tx=0x7f80a4005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68ef0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68ef0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42686/0 (socket says 172.21.15.3:42686) 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 -- 172.21.15.3:0/2391860737 learned_addr learned my addr 172.21.15.3:0/2391860737 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80add78640 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efdda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80aed7a640 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68f00780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 -- 172.21.15.3:0/2391860737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 msgr2=0x55da68efdda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efdda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 -- 172.21.15.3:0/2391860737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 msgr2=0x55da68f00780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68f00780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 -- 172.21.15.3:0/2391860737 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da68e1a210 con 0x55da68ef07b0 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.670+0000 7f80ae579640 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68ef0b90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f80a4005bc0 tx=0x7f80a4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b52e8ffd334c4de server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f809cff9640 1 -- 172.21.15.3:0/2391860737 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4005020 con 0x55da68ef07b0 2024-02-09T22:55:58.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f809cff9640 1 -- 172.21.15.3:0/2391860737 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a4021070 con 0x55da68ef07b0 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f809cff9640 1 -- 172.21.15.3:0/2391860737 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a40266e0 con 0x55da68ef07b0 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 -- 172.21.15.3:0/2391860737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 msgr2=0x55da68ef0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68ef0b90 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f80a4005bc0 tx=0x7f80a4014000 comp rx=0 tx=0).stop 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 -- 172.21.15.3:0/2391860737 shutdown_connections 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68f00780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efdda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 --2- 172.21.15.3:0/2391860737 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68ef0b90 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.671+0000 7f80b33a8740 1 -- 172.21.15.3:0/2391860737 >> 172.21.15.3:0/2391860737 conn(0x55da68eeb100 msgr2=0x55da68eeb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 -- 172.21.15.3:0/2391860737 shutdown_connections 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 -- 172.21.15.3:0/2391860737 wait complete. 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 -- start start 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 Processor -- start 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68efb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efb860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68efbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68e58030 con 0x55da68efe370 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68b2b430 con 0x55da68ef07b0 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68ae7e00 con 0x55da68ef1160 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80b33a8740 10 client.0 _ll_register_callbacks cb 0x55da68e4d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80aed7a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68efbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80add78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efb860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80add78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efb860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41846/0 (socket says 172.21.15.3:41846) 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80add78640 1 -- 172.21.15.3:0/4166583799 learned_addr learned my addr 172.21.15.3:0/4166583799 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.672+0000 7f80ae579640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68efb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80aed7a640 1 -- 172.21.15.3:0/4166583799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 msgr2=0x55da68efb860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80aed7a640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efb860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80aed7a640 1 -- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 msgr2=0x55da68efb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80aed7a640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68efb320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80aed7a640 1 -- 172.21.15.3:0/4166583799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80a4017070 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80add78640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80ae579640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68efb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80aed7a640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68efbda0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f80a00cbc40 tx=0x7f80a009b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57400be57e9b40ef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f8091ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f8091ffb640 1 -- 172.21.15.3:0/4166583799 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a00a3420 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f8091ffb640 1 -- 172.21.15.3:0/4166583799 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f80a00a36b0 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55da68b577e0 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f8091ffb640 1 -- 172.21.15.3:0/4166583799 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80a009a360 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80b33a8740 10 client.17262 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55da68e566b0 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.673+0000 7f8091ffb640 1 -- 172.21.15.3:0/4166583799 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55da68b577e0 con 0x55da68efe370 2024-02-09T22:55:58.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f8091ffb640 1 -- 172.21.15.3:0/4166583799 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f80a009e040 con 0x55da68efe370 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80b33a8740 10 client.17262 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80b33a8740 10 client.17262 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80b33a8740 20 client.17262 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f0e920 0x55da68f10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80b33a8740 4 client.17262 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55da68b94720 con 0x55da68f0e920 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80ae579640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f0e920 0x55da68f10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f8091ffb640 10 client.17262 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.674+0000 7f80ae579640 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f0e920 0x55da68f10d40 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bb80401cd3e42ce7 server_cookie=8399f72953fdbf62 in_seq=0 out_seq=0 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f8091ffb640 1 -- 172.21.15.3:0/4166583799 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55da68b94720 con 0x55da68f0e920 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f8091ffb640 10 client.17262 handle_command_reply: tid=0 2024-02-09T22:55:58.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 client.17262 shutdown 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 msgr2=0x55da68efbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68efbda0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f80a00cbc40 tx=0x7f80a009b6d0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 shutdown_connections 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f0e920 0x55da68f10d40 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68efbda0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efb860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f8091ffb640 0 client.17262 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4166583799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68efb320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 >> 172.21.15.3:0/4166583799 conn(0x55da68eeb100 msgr2=0x55da68eece50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 shutdown_connections 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 1 -- 172.21.15.3:0/4166583799 wait complete. 2024-02-09T22:55:58.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.675+0000 7f80b33a8740 20 client.17262 trim_cache size 0 max 16384 2024-02-09T22:55:58.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:58 smithi064 ceph-mon[36638]: pgmap v144: 97 pgs: 97 active+clean; 56 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:55:58.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:55:58 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2876910113' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 Processor -- start 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 -- start start 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68efda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68ef9bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68ddd130 con 0x55da68ef07b0 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68b94720 con 0x55da68efe370 2024-02-09T22:55:58.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80b33a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68b577e0 con 0x55da68ef1160 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80ae579640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68efda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80ae579640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68efda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42708/0 (socket says 172.21.15.3:42708) 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80add78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80ae579640 1 -- 172.21.15.3:0/1303014289 learned_addr learned my addr 172.21.15.3:0/1303014289 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.688+0000 7f80aed7a640 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68ef9bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f80ae579640 1 -- 172.21.15.3:0/1303014289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 msgr2=0x55da68efbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f80ae579640 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f80ae579640 1 -- 172.21.15.3:0/1303014289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 msgr2=0x55da68ef9bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f80ae579640 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68ef9bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f80ae579640 1 -- 172.21.15.3:0/1303014289 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55da68df2c90 con 0x55da68ef07b0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f80ae579640 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68efda50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009a90 tx=0x7f80a4029040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78a24c4a6bf5f105 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f8093fff640 1 -- 172.21.15.3:0/1303014289 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a402c040 con 0x55da68ef07b0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f8093fff640 1 -- 172.21.15.3:0/1303014289 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a40035f0 con 0x55da68ef07b0 2024-02-09T22:55:58.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.689+0000 7f8093fff640 1 -- 172.21.15.3:0/1303014289 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a4005020 con 0x55da68ef07b0 2024-02-09T22:55:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- 172.21.15.3:0/1303014289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 msgr2=0x55da68efda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68efda50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009a90 tx=0x7f80a4029040 comp rx=0 tx=0).stop 2024-02-09T22:55:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- 172.21.15.3:0/1303014289 shutdown_connections 2024-02-09T22:55:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68efe370 0x55da68ef9bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68efbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- 172.21.15.3:0/1303014289 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68ef07b0 0x55da68efda50 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- 172.21.15.3:0/1303014289 >> 172.21.15.3:0/1303014289 conn(0x55da68eeb100 msgr2=0x55da68ee6180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- 172.21.15.3:0/1303014289 shutdown_connections 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- 172.21.15.3:0/1303014289 wait complete. 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- start start 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 Processor -- start 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68f05110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68f05650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68f037b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68f03d70 con 0x55da68efe370 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68f03ee0 con 0x55da68ef07b0 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.690+0000 7f80b33a8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55da68f040a0 con 0x55da68ef1160 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80b33a8740 10 client.0 _ll_register_callbacks cb 0x7f80a40099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68f037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80add78640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68f05650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68f037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42720/0 (socket says 172.21.15.3:42720) 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 -- 172.21.15.3:0/4003187063 learned_addr learned my addr 172.21.15.3:0/4003187063 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80ae579640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68f05110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 -- 172.21.15.3:0/4003187063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 msgr2=0x55da68f05650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68f05650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 -- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 msgr2=0x55da68f05110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68f05110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 -- 172.21.15.3:0/4003187063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55da68e1a210 con 0x55da68efe370 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80add78640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68f05650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80aed7a640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68f037b0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f80a00991c0 tx=0x7f80a009a150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bf371060736123d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f8091ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f8091ffb640 1 -- 172.21.15.3:0/4003187063 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80a0005190 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f8091ffb640 1 -- 172.21.15.3:0/4003187063 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f80a00a36b0 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80a4021070 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f8091ffb640 1 -- 172.21.15.3:0/4003187063 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80a009aa50 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80b33a8740 10 client.17268 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.691+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55da68e18a70 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f8091ffb640 1 -- 172.21.15.3:0/4003187063 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80a0089080 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f8091ffb640 1 -- 172.21.15.3:0/4003187063 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f80a009e030 con 0x55da68efe370 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80b33a8740 10 client.17268 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80b33a8740 10 client.17268 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80b33a8740 20 client.17268 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f65b30 0x55da68f67f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80b33a8740 4 client.17268 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55da68be3890 con 0x55da68f65b30 2024-02-09T22:55:58.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80ae579640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f65b30 0x55da68f67f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f80ae579640 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f65b30 0x55da68f67f50 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=967c506d1d196c69 server_cookie=8b81bd01373a7fee in_seq=0 out_seq=0 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f8091ffb640 10 client.17268 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f8091ffb640 1 -- 172.21.15.3:0/4003187063 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55da68be3890 con 0x55da68f65b30 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.692+0000 7f8091ffb640 10 client.17268 handle_command_reply: tid=0 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 client.17268 shutdown 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 msgr2=0x55da68f037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68f037b0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f80a00991c0 tx=0x7f80a009a150 comp rx=0 tx=0).stop 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 shutdown_connections 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55da68f65b30 0x55da68f67f50 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55da68efe370 0x55da68f037b0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55da68ef1160 0x55da68f05650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 --2- 172.21.15.3:0/4003187063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55da68ef07b0 0x55da68f05110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 >> 172.21.15.3:0/4003187063 conn(0x55da68eeb100 msgr2=0x55da68ef9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f8091ffb640 0 client.17268 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:55:58.702 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 shutdown_connections 2024-02-09T22:55:58.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 1 -- 172.21.15.3:0/4003187063 wait complete. 2024-02-09T22:55:58.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b33a8740 20 client.17268 trim_cache size 0 max 16384 2024-02-09T22:55:58.703 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f80940a2230 msgr2=0x7f80940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f80940a2230 0x7f80940a46f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f80a4009bd0 tx=0x7f80a4005f90 comp rx=0 tx=0).stop 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 msgr2=0x7f80a8197660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 0x7f80a8197660 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f80a000ea60 tx=0x7f80a000ef20 comp rx=0 tx=0).stop 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 shutdown_connections 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f80940a2230 0x7f80940a46f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f80a810b820 0x7f80a8197660 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f80a8102d70 0x7f80a819dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 --2- 172.21.15.3:0/33474751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f80a8102450 0x7f80a819d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.693+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 >> 172.21.15.3:0/33474751 conn(0x7f80a80772e0 msgr2=0x7f80a80fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.694+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 shutdown_connections 2024-02-09T22:55:58.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.694+0000 7f80b07dd640 1 -- 172.21.15.3:0/33474751 wait complete. 2024-02-09T22:55:58.707 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:55:58.708 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.814+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/813164982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84105410 msgr2=0x7f6d8410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.814+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/813164982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84105410 0x7f6d8410cfe0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c009cd0 tx=0x7f6d7c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.814+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/813164982 shutdown_connections 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.814+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/813164982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84105410 0x7f6d8410cfe0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.814+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/813164982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d84100cf0 0x7f6d84104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.814+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/813164982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d84100340 0x7f6d84100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/813164982 >> 172.21.15.3:0/813164982 conn(0x7f6d840772e0 msgr2=0x7f6d840776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/813164982 shutdown_connections 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/813164982 wait complete. 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 Processor -- start 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- start start 2024-02-09T22:55:58.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 0x7f6d841986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d84105410 0x7f6d84198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d84199160 0x7f6d841a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841a3dd0 con 0x7f6d84100340 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841a3f40 con 0x7f6d84105410 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d8b1f6640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d841a4100 con 0x7f6d84199160 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d88f92640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 0x7f6d841986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d88f92640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 0x7f6d841986e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42740/0 (socket says 172.21.15.3:42740) 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d88f92640 1 -- 172.21.15.3:0/4102058520 learned_addr learned my addr 172.21.15.3:0/4102058520 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.815+0000 7f6d89793640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d84199160 0x7f6d841a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d83fff640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d84105410 0x7f6d84198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d88f92640 1 -- 172.21.15.3:0/4102058520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d84199160 msgr2=0x7f6d841a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d88f92640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d84199160 0x7f6d841a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d88f92640 1 -- 172.21.15.3:0/4102058520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d84105410 msgr2=0x7f6d84198c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d88f92640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d84105410 0x7f6d84198c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d88f92640 1 -- 172.21.15.3:0/4102058520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c009900 con 0x7f6d84100340 2024-02-09T22:55:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d88f92640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 0x7f6d841986e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800e9f0 tx=0x7f6d7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d81ffb640 1 -- 172.21.15.3:0/4102058520 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78009700 con 0x7f6d84100340 2024-02-09T22:55:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d81ffb640 1 -- 172.21.15.3:0/4102058520 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d7800cd80 con 0x7f6d84100340 2024-02-09T22:55:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d841a43e0 con 0x7f6d84100340 2024-02-09T22:55:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d81ffb640 1 -- 172.21.15.3:0/4102058520 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d78018920 con 0x7f6d84100340 2024-02-09T22:55:58.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.816+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d841a4940 con 0x7f6d84100340 2024-02-09T22:55:58.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.819+0000 7f6d81ffb640 1 -- 172.21.15.3:0/4102058520 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6d78018a80 con 0x7f6d84100340 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.819+0000 7f6d8ddc5740 1 Processor -- start 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 -- start start 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 0x55f356d45e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d4e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d513b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356d01020 con 0x55f356d45a30 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356cb4320 con 0x55f356d463e0 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356cb1850 con 0x55f356d4ef60 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d81ffb640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d740a22e0 0x7f6d740a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d89793640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 0x55f356d45e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d89793640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 0x55f356d45e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42748/0 (socket says 172.21.15.3:42748) 2024-02-09T22:55:58.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d89793640 1 -- 172.21.15.3:0/1688563687 learned_addr learned my addr 172.21.15.3:0/1688563687 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.820+0000 7f6d88f92640 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d4e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d81ffb640 1 -- 172.21.15.3:0/4102058520 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6d78014070 con 0x7f6d84100340 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 -- 172.21.15.3:0/1688563687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 msgr2=0x55f356d513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 -- 172.21.15.3:0/1688563687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 msgr2=0x55f356d4e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d4e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 -- 172.21.15.3:0/1688563687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f356c6f210 con 0x55f356d45a30 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d83fff640 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d88f92640 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d4e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 0x55f356d45e10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c02c700 tx=0x7f6d7c032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea30d200ad96b800 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d89793640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d740a22e0 0x7f6d740a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.821+0000 7f6d6effd640 1 -- 172.21.15.3:0/1688563687 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c034440 con 0x55f356d45a30 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d6effd640 1 -- 172.21.15.3:0/1688563687 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d7c0326b0 con 0x55f356d45a30 2024-02-09T22:55:58.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d6effd640 1 -- 172.21.15.3:0/1688563687 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c0042c0 con 0x55f356d45a30 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/1688563687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 msgr2=0x55f356d45e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 0x55f356d45e10 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c02c700 tx=0x7f6d7c032d70 comp rx=0 tx=0).stop 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d89793640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d740a22e0 0x7f6d740a47a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c047000 tx=0x7f6d7c004620 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/1688563687 shutdown_connections 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d4e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/1688563687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d45a30 0x55f356d45e10 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.822+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/1688563687 >> 172.21.15.3:0/1688563687 conn(0x55f356d40100 msgr2=0x55f356d404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.823+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/1688563687 shutdown_connections 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.823+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/1688563687 wait complete. 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.823+0000 7f6d8ddc5740 1 -- start start 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.823+0000 7f6d8ddc5740 1 Processor -- start 2024-02-09T22:55:58.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d4bee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 0x55f356d4c960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356cad030 con 0x55f356d463e0 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356980430 con 0x55f356d4ef60 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f35693ce00 con 0x55f356d45a30 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d8ddc5740 10 client.0 _ll_register_callbacks cb 0x55f356ca2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d89793640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 0x55f356d4c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42754/0 (socket says 172.21.15.3:42754) 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 -- 172.21.15.3:0/968106462 learned_addr learned my addr 172.21.15.3:0/968106462 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d89793640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 0x55f356d4c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52856/0 (socket says 172.21.15.3:52856) 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d88f92640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d4bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 -- 172.21.15.3:0/968106462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 msgr2=0x55f356d4bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d4bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 -- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 msgr2=0x55f356d4c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 0x55f356d4c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d83fff640 1 -- 172.21.15.3:0/968106462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8410d7f0 con 0x55f356d463e0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.824+0000 7f6d89793640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 0x55f356d4c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d83fff640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c420 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6d70012700 tx=0x7f6d70012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5c6816adf9e95b3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d63fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d63fff640 1 -- 172.21.15.3:0/968106462 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70017070 con 0x55f356d463e0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f3569ac7e0 con 0x55f356d463e0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d63fff640 1 -- 172.21.15.3:0/968106462 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6d70004050 con 0x55f356d463e0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d63fff640 1 -- 172.21.15.3:0/968106462 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d70004210 con 0x55f356d463e0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d8ddc5740 10 client.17280 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55f356cab6b0 con 0x55f356d463e0 2024-02-09T22:55:58.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d63fff640 1 -- 172.21.15.3:0/968106462 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f3569ac7e0 con 0x55f356d463e0 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.825+0000 7f6d63fff640 1 -- 172.21.15.3:0/968106462 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6d70003a00 con 0x55f356d463e0 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d8ddc5740 10 client.17280 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d8ddc5740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d8ddc5740 20 client.17280 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356d63780 0x55f356d65ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d8ddc5740 4 client.17280 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3569e9720 con 0x55f356d63780 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d88f92640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356d63780 0x55f356d65ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d88f92640 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356d63780 0x55f356d65ba0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e029bf1e2c28a5 server_cookie=7cb295a46e343734 in_seq=0 out_seq=0 2024-02-09T22:55:58.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.826+0000 7f6d63fff640 10 client.17280 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:58.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d63fff640 1 -- 172.21.15.3:0/968106462 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f3569e9720 con 0x55f356d63780 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d63fff640 10 client.17280 handle_command_reply: tid=0 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 client.17280 shutdown 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 msgr2=0x55f356d4c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c420 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f6d70012700 tx=0x7f6d70012bc0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 shutdown_connections 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356d63780 0x55f356d65ba0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d4ef60 0x55f356d4c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c420 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/968106462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d4bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d63fff640 0 client.17280 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.828+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 >> 172.21.15.3:0/968106462 conn(0x55f356d40100 msgr2=0x55f356d41e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.829+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 shutdown_connections 2024-02-09T22:55:58.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.829+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/968106462 wait complete. 2024-02-09T22:55:58.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.829+0000 7f6d8ddc5740 20 client.17280 trim_cache size 0 max 16384 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 Processor -- start 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 -- start start 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d45a30 0x55f356d4e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d6a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356c32130 con 0x55f356d463e0 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356d44ac0 con 0x55f356d45a30 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356d3bab0 con 0x55f356d4ef60 2024-02-09T22:55:58.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42758/0 (socket says 172.21.15.3:42758) 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d83fff640 1 -- 172.21.15.3:0/3350351907 learned_addr learned my addr 172.21.15.3:0/3350351907 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d89793640 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d6a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.845+0000 7f6d88f92640 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d45a30 0x55f356d4e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d83fff640 1 -- 172.21.15.3:0/3350351907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 msgr2=0x55f356d6a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d83fff640 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d6a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d83fff640 1 -- 172.21.15.3:0/3350351907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d45a30 msgr2=0x55f356d4e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d83fff640 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d45a30 0x55f356d4e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d83fff640 1 -- 172.21.15.3:0/3350351907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f356c47c90 con 0x55f356d463e0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d88f92640 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d45a30 0x55f356d4e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d83fff640 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c7a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6d70005bc0 tx=0x7f6d70009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4bd5e7cc61a5fc8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d6e7fc640 1 -- 172.21.15.3:0/3350351907 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70017070 con 0x55f356d463e0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d6e7fc640 1 -- 172.21.15.3:0/3350351907 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d70025520 con 0x55f356d463e0 2024-02-09T22:55:58.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.846+0000 7f6d6e7fc640 1 -- 172.21.15.3:0/3350351907 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d70025780 con 0x55f356d463e0 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3350351907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 msgr2=0x55f356d4c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c7a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6d70005bc0 tx=0x7f6d70009f20 comp rx=0 tx=0).stop 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3350351907 shutdown_connections 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d4ef60 0x55f356d6a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d463e0 0x55f356d4c7a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3350351907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d45a30 0x55f356d4e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3350351907 >> 172.21.15.3:0/3350351907 conn(0x55f356d40100 msgr2=0x55f356d3b180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3350351907 shutdown_connections 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3350351907 wait complete. 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 -- start start 2024-02-09T22:55:58.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.847+0000 7f6d8ddc5740 1 Processor -- start 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d58dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d59310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 0x55f356d59850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356db9cd0 con 0x55f356d4ef60 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356db9e10 con 0x55f356d463e0 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f356db9f50 con 0x55f356d45a30 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d8ddc5740 10 client.0 _ll_register_callbacks cb 0x7f6d7c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 0x55f356d59850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d88f92640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d58dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 0x55f356d59850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42774/0 (socket says 172.21.15.3:42774) 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d83fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d59310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.856 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 -- 172.21.15.3:0/3693487079 learned_addr learned my addr 172.21.15.3:0/3693487079 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 -- 172.21.15.3:0/3693487079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 msgr2=0x55f356d58dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d58dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 -- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 msgr2=0x55f356d59310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d59310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 -- 172.21.15.3:0/3693487079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f356cab6b0 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d89793640 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 0x55f356d59850 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c02c6d0 tx=0x7f6d7c050000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73fb9df448918564 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d63fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.848+0000 7f6d63fff640 1 -- 172.21.15.3:0/3693487079 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d7c04f040 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d63fff640 1 -- 172.21.15.3:0/3693487079 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6d7c05b040 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d7c0326b0 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d63fff640 1 -- 172.21.15.3:0/3693487079 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d7c034440 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 10 client.17286 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55f356c6da70 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d63fff640 1 -- 172.21.15.3:0/3693487079 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d7c0326b0 con 0x55f356d4ef60 2024-02-09T22:55:58.857 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d63fff640 1 -- 172.21.15.3:0/3693487079 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6d7c04a9f0 con 0x55f356d4ef60 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 10 client.17286 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 20 client.17286 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356dbb400 0x55f356dbd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 4 client.17286 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f356a38890 con 0x55f356dbb400 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.849+0000 7f6d88f92640 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356dbb400 0x55f356dbd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d63fff640 10 client.17286 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d88f92640 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356dbb400 0x55f356dbd820 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8651d402b35928b server_cookie=ff10ae6ac41703b3 in_seq=0 out_seq=0 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d63fff640 1 -- 172.21.15.3:0/3693487079 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f356a38890 con 0x55f356dbb400 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d63fff640 10 client.17286 handle_command_reply: tid=0 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 client.17286 shutdown 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 msgr2=0x55f356d59850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.858 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 0x55f356d59850 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c02c6d0 tx=0x7f6d7c050000 comp rx=0 tx=0).stop 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 shutdown_connections 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f356dbb400 0x55f356dbd820 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f356d463e0 0x55f356d59310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f356d4ef60 0x55f356d59850 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d63fff640 0 client.17286 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 --2- 172.21.15.3:0/3693487079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f356d45a30 0x55f356d58dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 >> 172.21.15.3:0/3693487079 conn(0x55f356d40100 msgr2=0x55f356d6a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 shutdown_connections 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 1 -- 172.21.15.3:0/3693487079 wait complete. 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.850+0000 7f6d8ddc5740 20 client.17286 trim_cache size 0 max 16384 2024-02-09T22:55:58.859 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:55:58.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d740a22e0 msgr2=0x7f6d740a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d740a22e0 0x7f6d740a47a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f6d7c047000 tx=0x7f6d7c004620 comp rx=0 tx=0).stop 2024-02-09T22:55:58.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 msgr2=0x7f6d841986e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 0x7f6d841986e0 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6d7800e9f0 tx=0x7f6d7800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 shutdown_connections 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d740a22e0 0x7f6d740a47a0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6d84105410 0x7f6d84198c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6d84199160 0x7f6d841a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 --2- 172.21.15.3:0/4102058520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6d84100340 0x7f6d841986e0 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 >> 172.21.15.3:0/4102058520 conn(0x7f6d840772e0 msgr2=0x7f6d840fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 shutdown_connections 2024-02-09T22:55:58.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:58.851+0000 7f6d8b1f6640 1 -- 172.21.15.3:0/4102058520 wait complete. 2024-02-09T22:55:59.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 138920 2.87 MB/sec execute 113 sec latency 12.536 ms 2024-02-09T22:55:59.866 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:55:59.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:55:59.866 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.971+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/4272708232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8100310 msgr2=0x7fa1f81006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.971+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/4272708232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8100310 0x7fa1f81006f0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4009930 tx=0x7fa1f402bc30 comp rx=0 tx=0).stop 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/4272708232 shutdown_connections 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/4272708232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f8100cc0 0x7fa1f8104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/4272708232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f81053e0 0x7fa1f810cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/4272708232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8100310 0x7fa1f81006f0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/4272708232 >> 172.21.15.3:0/4272708232 conn(0x7fa1f8077230 msgr2=0x7fa1f8077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/4272708232 shutdown_connections 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/4272708232 wait complete. 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 Processor -- start 2024-02-09T22:55:59.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- start start 2024-02-09T22:55:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f8100310 0x7fa1f819f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f81053e0 0x7fa1f819fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 0x7fa1f8199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f819a300 con 0x7fa1f8199910 2024-02-09T22:55:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f819a470 con 0x7fa1f81053e0 2024-02-09T22:55:59.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.972+0000 7fa1ff3e9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f819a630 con 0x7fa1f8100310 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 0x7fa1f8199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 0x7fa1f8199dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42792/0 (socket says 172.21.15.3:42792) 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 -- 172.21.15.3:0/421004774 learned_addr learned my addr 172.21.15.3:0/421004774 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fc984640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f81053e0 0x7fa1f819fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd185640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f8100310 0x7fa1f819f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 -- 172.21.15.3:0/421004774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f8100310 msgr2=0x7fa1f819f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f8100310 0x7fa1f819f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 -- 172.21.15.3:0/421004774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f81053e0 msgr2=0x7fa1f819fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f81053e0 0x7fa1f819fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 -- 172.21.15.3:0/421004774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f4009560 con 0x7fa1f8199910 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd185640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f8100310 0x7fa1f819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1fd986640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 0x7fa1f8199dc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000eaa0 tx=0x7fa1f000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:59.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1ee7fc640 1 -- 172.21.15.3:0/421004774 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f000cdb0 con 0x7fa1f8199910 2024-02-09T22:55:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1ee7fc640 1 -- 172.21.15.3:0/421004774 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f0005020 con 0x7fa1f8199910 2024-02-09T22:55:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f81a6620 con 0x7fa1f8199910 2024-02-09T22:55:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1ee7fc640 1 -- 172.21.15.3:0/421004774 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f00116b0 con 0x7fa1f8199910 2024-02-09T22:55:59.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.973+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f81a6b30 con 0x7fa1f8199910 2024-02-09T22:55:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa1ee7fc640 1 -- 172.21.15.3:0/421004774 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa1f0005520 con 0x7fa1f8199910 2024-02-09T22:55:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 Processor -- start 2024-02-09T22:55:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 -- start start 2024-02-09T22:55:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263190 0x558181263570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x55818126c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818126ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581811d1000 con 0x558181263ab0 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581811ce530 con 0x55818126c690 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.976+0000 7fa201fbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581811c9d10 con 0x558181263190 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x55818126c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x55818126c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42798/0 (socket says 172.21.15.3:42798) 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 -- 172.21.15.3:0/2991416727 learned_addr learned my addr 172.21.15.3:0/2991416727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1ee7fc640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1e40a2230 0x7fa1e40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fc984640 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818126ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fc984640 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263190 0x558181263570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd986640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1e40a2230 0x7fa1e40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 -- 172.21.15.3:0/2991416727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263190 msgr2=0x558181263570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263190 0x558181263570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1ee7fc640 1 -- 172.21.15.3:0/421004774 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa1f00ce390 con 0x7fa1f8199910 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 -- 172.21.15.3:0/2991416727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 msgr2=0x55818126ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818126ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 -- 172.21.15.3:0/2991416727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558181170210 con 0x558181263ab0 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fc984640 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818126ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd185640 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x55818126c150 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4006010 tx=0x7fa1f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1606325189c616c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1fd986640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1e40a2230 0x7fa1e40a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00cbbc0 tx=0x7fa1f00662a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1e37fe640 1 -- 172.21.15.3:0/2991416727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4004ab0 con 0x558181263ab0 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1e37fe640 1 -- 172.21.15.3:0/2991416727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f4004030 con 0x558181263ab0 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.977+0000 7fa1e37fe640 1 -- 172.21.15.3:0/2991416727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4005380 con 0x558181263ab0 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 -- 172.21.15.3:0/2991416727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 msgr2=0x55818126c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x55818126c150 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4006010 tx=0x7fa1f40046c0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 -- 172.21.15.3:0/2991416727 shutdown_connections 2024-02-09T22:55:59.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818126ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x55818126c150 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 --2- 172.21.15.3:0/2991416727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263190 0x558181263570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 -- 172.21.15.3:0/2991416727 >> 172.21.15.3:0/2991416727 conn(0x55818125d8c0 msgr2=0x55818125dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 -- 172.21.15.3:0/2991416727 shutdown_connections 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.978+0000 7fa201fbe740 1 -- 172.21.15.3:0/2991416727 wait complete. 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 -- start start 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 Processor -- start 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812696b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x558181269bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55818126c690 0x55818126a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818116ee60 con 0x55818126c690 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558180e9d4b0 con 0x558181263190 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55818114f670 con 0x558181263ab0 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa201fbe740 10 client.0 _ll_register_callbacks cb 0x558181167190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:55:59.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa1fc984640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55818126c690 0x55818126a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa1fd185640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x558181269bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa1fd185640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x558181269bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41960/0 (socket says 172.21.15.3:41960) 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa1fd185640 1 -- 172.21.15.3:0/1149353643 learned_addr learned my addr 172.21.15.3:0/1149353643 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa1fc984640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55818126c690 0x55818126a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42806/0 (socket says 172.21.15.3:42806) 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.979+0000 7fa1fc984640 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812696b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1fd185640 1 -- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 msgr2=0x5581812696b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1fd185640 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812696b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1fd185640 1 -- 172.21.15.3:0/1149353643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55818126c690 msgr2=0x55818126a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1fd185640 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55818126c690 0x55818126a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1fd185640 1 -- 172.21.15.3:0/1149353643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f810d7c0 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1fd185640 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x558181269bf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4000c00 tx=0x7fa1f4009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81000e91d17afa6c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/1149353643 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f402cbf0 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/1149353643 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa1f4005080 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558180ec9860 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/1149353643 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1f4005240 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa201fbe740 10 client.26629 fetch_fsmap learned FSMap version 36 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5581811c8390 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.980+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/1149353643 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558180ec9860 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/1149353643 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa1f403c040 con 0x558181263ab0 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa201fbe740 10 client.26629 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:55:59.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa201fbe740 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa201fbe740 20 client.26629 populate_metadata read hostname 'smithi003' 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa201fbe740 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55818127c300 0x55818127e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa201fbe740 4 client.26629 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558180f1a410 con 0x55818127c300 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa1fc984640 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55818127c300 0x55818127e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa1fc984640 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55818127c300 0x55818127e720 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8257f56bb00544d1 server_cookie=8d7d78761aeb90c9 in_seq=0 out_seq=0 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.981+0000 7fa1e0ff9640 10 client.26629 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/1149353643 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558180f1a410 con 0x55818127c300 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa1e0ff9640 10 client.26629 handle_command_reply: tid=0 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 client.26629 shutdown 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 msgr2=0x558181269bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x558181269bf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa1f4000c00 tx=0x7fa1f4009260 comp rx=0 tx=0).stop 2024-02-09T22:55:59.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 shutdown_connections 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55818127c300 0x55818127e720 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55818126c690 0x55818126a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa1e0ff9640 0 client.26629 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x558181269bf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 --2- 172.21.15.3:0/1149353643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812696b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 >> 172.21.15.3:0/1149353643 conn(0x55818125d8c0 msgr2=0x55818125f610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 shutdown_connections 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 1 -- 172.21.15.3:0/1149353643 wait complete. 2024-02-09T22:55:59.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.982+0000 7fa201fbe740 20 client.26629 trim_cache size 0 max 16384 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.997+0000 7fa201fbe740 1 Processor -- start 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.997+0000 7fa201fbe740 1 -- start start 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 0x558181269f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x55818125c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818125cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa201fbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558181262e60 con 0x558181263190 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa201fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581812791c0 con 0x55818126c690 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa201fbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558181283930 con 0x558181263ab0 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 0x558181269f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 0x558181269f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42810/0 (socket says 172.21.15.3:42810) 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 -- 172.21.15.3:0/4250825508 learned_addr learned my addr 172.21.15.3:0/4250825508 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 -- 172.21.15.3:0/4250825508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 msgr2=0x55818125c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x55818125c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 -- 172.21.15.3:0/4250825508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 msgr2=0x55818125cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818125cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 -- 172.21.15.3:0/4250825508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558181164c70 con 0x558181263190 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1fc984640 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 0x558181269f50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fa1f819b100 tx=0x7fa1e80047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cdc7d4debdceade server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1e2ffd640 1 -- 172.21.15.3:0/4250825508 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e8016e30 con 0x558181263190 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1e2ffd640 1 -- 172.21.15.3:0/4250825508 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e800c020 con 0x558181263190 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.998+0000 7fa1e2ffd640 1 -- 172.21.15.3:0/4250825508 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e80207a0 con 0x558181263190 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 -- 172.21.15.3:0/4250825508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 msgr2=0x558181269f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 0x558181269f50 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fa1f819b100 tx=0x7fa1e80047d0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 -- 172.21.15.3:0/4250825508 shutdown_connections 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55818126c690 0x55818125cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558181263ab0 0x55818125c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 --2- 172.21.15.3:0/4250825508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263190 0x558181269f50 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 -- 172.21.15.3:0/4250825508 >> 172.21.15.3:0/4250825508 conn(0x55818125d8c0 msgr2=0x55818125c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 -- 172.21.15.3:0/4250825508 shutdown_connections 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 -- 172.21.15.3:0/4250825508 wait complete. 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 -- start start 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:55:59.999+0000 7fa201fbe740 1 Processor -- start 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x5581812c8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55818126c690 0x5581812c9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581812d3990 con 0x558181263ab0 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581812d3b00 con 0x558181263190 2024-02-09T22:56:00.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581812d3cc0 con 0x55818126c690 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 10 client.0 _ll_register_callbacks cb 0x7fa1f4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x5581812c8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x5581812c8ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42824/0 (socket says 172.21.15.3:42824) 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 -- 172.21.15.3:0/548783553 learned_addr learned my addr 172.21.15.3:0/548783553 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fc984640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fc984640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55818126c690 0x5581812c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 -- 172.21.15.3:0/548783553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55818126c690 msgr2=0x5581812c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55818126c690 0x5581812c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 -- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 msgr2=0x5581812c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 -- 172.21.15.3:0/548783553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e800c410 con 0x558181263ab0 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1fd185640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x5581812c8ce0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40092a0 tx=0x7fa1f4005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4333210896e546c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1e0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/548783553 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1f4034a50 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/548783553 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa1f4000e40 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.000+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1f4004030 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/548783553 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1f40046c0 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 10 client.17304 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55818118aac0 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/548783553 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1f4004030 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/548783553 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa1f403c040 con 0x558181263ab0 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 10 client.17304 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 20 client.17304 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:00.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581812d50f0 0x5581812d7510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 4 client.17304 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558180f067b0 con 0x5581812d50f0 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.001+0000 7fa1fc984640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581812d50f0 0x5581812d7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.002+0000 7fa1e0ff9640 10 client.17304 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.002+0000 7fa1fc984640 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581812d50f0 0x5581812d7510 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bbf1706ee0561b0f server_cookie=161063cb43ce539c in_seq=0 out_seq=0 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa1e0ff9640 1 -- 172.21.15.3:0/548783553 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558180f067b0 con 0x5581812d50f0 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa1e0ff9640 10 client.17304 handle_command_reply: tid=0 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 client.17304 shutdown 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 msgr2=0x5581812c8ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x5581812c8ce0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7fa1f40092a0 tx=0x7fa1f4005de0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 shutdown_connections 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581812d50f0 0x5581812d7510 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55818126c690 0x5581812c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558181263ab0 0x5581812c8ce0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 --2- 172.21.15.3:0/548783553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558181263190 0x5581812c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa1e0ff9640 0 client.17304 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 >> 172.21.15.3:0/548783553 conn(0x55818125d8c0 msgr2=0x55818125c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.003+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 shutdown_connections 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa201fbe740 1 -- 172.21.15.3:0/548783553 wait complete. 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa201fbe740 20 client.17304 trim_cache size 0 max 16384 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "281e9fef-9dfa-462b-9157-95c6d919c26a", 2024-02-09T22:56:00.014 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:56:00.015 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:56:00.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1e40a2230 msgr2=0x7fa1e40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1e40a2230 0x7fa1e40a46f0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa1f00cbbc0 tx=0x7fa1f00662a0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 msgr2=0x7fa1f8199dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 0x7fa1f8199dc0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000eaa0 tx=0x7fa1f000ef60 comp rx=0 tx=0).stop 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 shutdown_connections 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1e40a2230 0x7fa1e40a46f0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f81053e0 0x7fa1f819fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f8199910 0x7fa1f8199dc0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 --2- 172.21.15.3:0/421004774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f8100310 0x7fa1f819f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 >> 172.21.15.3:0/421004774 conn(0x7fa1f8077230 msgr2=0x7fa1f80fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 shutdown_connections 2024-02-09T22:56:00.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.004+0000 7fa1ff3e9640 1 -- 172.21.15.3:0/421004774 wait complete. 2024-02-09T22:56:00.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:56:00.019 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.142+0000 7f0536ac9640 1 -- 172.21.15.3:0/392322895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530100d00 msgr2=0x7f0530104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.142+0000 7f0536ac9640 1 --2- 172.21.15.3:0/392322895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530100d00 0x7f0530104d40 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f051c009930 tx=0x7f051c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 -- 172.21.15.3:0/392322895 shutdown_connections 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 --2- 172.21.15.3:0/392322895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0530105420 0x7f053010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 --2- 172.21.15.3:0/392322895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530100d00 0x7f0530104d40 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 --2- 172.21.15.3:0/392322895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0530100350 0x7f0530100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 -- 172.21.15.3:0/392322895 >> 172.21.15.3:0/392322895 conn(0x7f0530077250 msgr2=0x7f0530077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 -- 172.21.15.3:0/392322895 shutdown_connections 2024-02-09T22:56:00.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.143+0000 7f0536ac9640 1 -- 172.21.15.3:0/392322895 wait complete. 2024-02-09T22:56:00.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 Processor -- start 2024-02-09T22:56:00.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 -- start start 2024-02-09T22:56:00.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0530100350 0x7f05301965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530105420 0x7f0530196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f053019ae20 0x7f053019b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053019b810 con 0x7f0530105420 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053019b980 con 0x7f053019ae20 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0536ac9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053019baf0 con 0x7f0530100350 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f052ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530105420 0x7f0530196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0534865640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0530100350 0x7f05301965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f052ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530105420 0x7f0530196af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42828/0 (socket says 172.21.15.3:42828) 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0535066640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f053019ae20 0x7f053019b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f052ffff640 1 -- 172.21.15.3:0/4181862038 learned_addr learned my addr 172.21.15.3:0/4181862038 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0535066640 1 -- 172.21.15.3:0/4181862038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0530100350 msgr2=0x7f05301965b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0535066640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0530100350 0x7f05301965b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0535066640 1 -- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530105420 msgr2=0x7f0530196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0535066640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530105420 0x7f0530196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.144+0000 7f0535066640 1 -- 172.21.15.3:0/4181862038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c009560 con 0x7f053019ae20 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.145+0000 7f0535066640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f053019ae20 0x7f053019b2d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0528009fd0 tx=0x7f052800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.145+0000 7f052dffb640 1 -- 172.21.15.3:0/4181862038 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052800ce20 con 0x7f053019ae20 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.145+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05300692f0 con 0x7f053019ae20 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.145+0000 7f052dffb640 1 -- 172.21.15.3:0/4181862038 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0528005020 con 0x7f053019ae20 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.145+0000 7f052dffb640 1 -- 172.21.15.3:0/4181862038 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0528010690 con 0x7f053019ae20 2024-02-09T22:56:00.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.145+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05300697a0 con 0x7f053019ae20 2024-02-09T22:56:00.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.148+0000 7f052dffb640 1 -- 172.21.15.3:0/4181862038 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0528005500 con 0x7f053019ae20 2024-02-09T22:56:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f052dffb640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f05200a2230 0x7f05200a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 Processor -- start 2024-02-09T22:56:00.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 -- start start 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 0x55ee12f94d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0534865640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f05200a2230 0x7f05200a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12fa0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12f4fd00 con 0x55ee12f9deb0 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12f042f0 con 0x55ee12f94980 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0539780740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12f01820 con 0x55ee12f95330 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0534865640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 0x55ee12f94d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0535066640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12fa0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0534865640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 0x55ee12f94d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52944/0 (socket says 172.21.15.3:52944) 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f0534865640 1 -- 172.21.15.3:0/2807070091 learned_addr learned my addr 172.21.15.3:0/2807070091 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.149+0000 7f052dffb640 1 -- 172.21.15.3:0/4181862038 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0528014070 con 0x7f053019ae20 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f052ffff640 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 -- 172.21.15.3:0/2807070091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 msgr2=0x55ee12f9d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 -- 172.21.15.3:0/2807070091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 msgr2=0x55ee12fa0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12fa0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 -- 172.21.15.3:0/2807070091 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee12ebeff0 con 0x55ee12f94980 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f052ffff640 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 0x55ee12f94d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f052401bc70 tx=0x7f05240127c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d79118118f4c43b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f0534865640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f05200a2230 0x7f05200a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0524012cc0 tx=0x7f0524014000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f051affd640 1 -- 172.21.15.3:0/2807070091 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052400b910 con 0x55ee12f94980 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f051affd640 1 -- 172.21.15.3:0/2807070091 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0524018040 con 0x55ee12f94980 2024-02-09T22:56:00.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.150+0000 7f051affd640 1 -- 172.21.15.3:0/2807070091 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05240047a0 con 0x55ee12f94980 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 -- 172.21.15.3:0/2807070091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 msgr2=0x55ee12f94d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 0x55ee12f94d60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f052401bc70 tx=0x7f05240127c0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 -- 172.21.15.3:0/2807070091 shutdown_connections 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12fa0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 --2- 172.21.15.3:0/2807070091 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f94980 0x55ee12f94d60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 -- 172.21.15.3:0/2807070091 >> 172.21.15.3:0/2807070091 conn(0x55ee12f8f050 msgr2=0x55ee12f8f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 -- 172.21.15.3:0/2807070091 shutdown_connections 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.151+0000 7f0539780740 1 -- 172.21.15.3:0/2807070091 wait complete. 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.152+0000 7f0539780740 1 -- start start 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.152+0000 7f0539780740 1 Processor -- start 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.152+0000 7f0539780740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f94980 0x55ee12f9aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.152+0000 7f0539780740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.152+0000 7f0539780740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 0x55ee12f9b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0539780740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12efd000 con 0x55ee12f94980 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0539780740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12bd0430 con 0x55ee12f9deb0 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0539780740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12b8ce00 con 0x55ee12f95330 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0535066640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 0x55ee12f9b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0535066640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 0x55ee12f9b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52950/0 (socket says 172.21.15.3:52950) 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f052ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0535066640 1 -- 172.21.15.3:0/1098813602 learned_addr learned my addr 172.21.15.3:0/1098813602 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f94980 0x55ee12f9aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0539780740 10 client.0 _ll_register_callbacks cb 0x55ee12ef17f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 -- 172.21.15.3:0/1098813602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 msgr2=0x55ee12f9b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 -- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 msgr2=0x55ee12f9b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 0x55ee12f9b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 -- 172.21.15.3:0/1098813602 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f052401b8a0 con 0x55ee12f94980 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0535066640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 0x55ee12f9b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f0534865640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f94980 0x55ee12f9aea0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f052400b3d0 tx=0x7f052400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a7f4cc62d17dcd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.153+0000 7f050ffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.154+0000 7f050ffff640 1 -- 172.21.15.3:0/1098813602 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05240055b0 con 0x55ee12f94980 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.154+0000 7f050ffff640 1 -- 172.21.15.3:0/1098813602 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f052400ed30 con 0x55ee12f94980 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.154+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ee12bfc7e0 con 0x55ee12f94980 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.154+0000 7f050ffff640 1 -- 172.21.15.3:0/1098813602 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0524004650 con 0x55ee12f94980 2024-02-09T22:56:00.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.154+0000 7f0539780740 10 client.17316 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.154+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55ee12efb680 con 0x55ee12f94980 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f050ffff640 1 -- 172.21.15.3:0/1098813602 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ee12bfc7e0 con 0x55ee12f94980 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f050ffff640 1 -- 172.21.15.3:0/1098813602 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0524022030 con 0x55ee12f94980 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f0539780740 10 client.17316 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f0539780740 10 client.17316 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f0539780740 20 client.17316 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f0539780740 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee12fae3a0 0x55ee12fb07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f0539780740 4 client.17316 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee12c39720 con 0x55ee12fae3a0 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.155+0000 7f052ffff640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee12fae3a0 0x55ee12fb07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f052ffff640 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee12fae3a0 0x55ee12fb07c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f3fc3ba0999203f server_cookie=de719bf86faab3fe in_seq=0 out_seq=0 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f050ffff640 10 client.17316 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f050ffff640 1 -- 172.21.15.3:0/1098813602 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ee12c39720 con 0x55ee12fae3a0 2024-02-09T22:56:00.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f050ffff640 10 client.17316 handle_command_reply: tid=0 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 client.17316 shutdown 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f94980 msgr2=0x55ee12f9aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f94980 0x55ee12f9aea0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f052400b3d0 tx=0x7f052400be60 comp rx=0 tx=0).stop 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 shutdown_connections 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee12fae3a0 0x55ee12fb07c0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f9deb0 0x55ee12f9b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f95330 0x55ee12f9b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 --2- 172.21.15.3:0/1098813602 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f94980 0x55ee12f9aea0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 >> 172.21.15.3:0/1098813602 conn(0x55ee12f8f050 msgr2=0x55ee12f90da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.156+0000 7f050ffff640 0 client.17316 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.157+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 shutdown_connections 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.157+0000 7f0539780740 1 -- 172.21.15.3:0/1098813602 wait complete. 2024-02-09T22:56:00.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.157+0000 7f0539780740 20 client.17316 trim_cache size 0 max 16384 2024-02-09T22:56:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 Processor -- start 2024-02-09T22:56:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 -- start start 2024-02-09T22:56:00.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12f96200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12f96740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12f997d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12e81fd0 con 0x55ee12f9deb0 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12bfc7e0 con 0x55ee12f95330 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0539780740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12fa0a70 con 0x55ee12f94980 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0535066640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12f997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0535066640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12f997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42852/0 (socket says 172.21.15.3:42852) 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0535066640 1 -- 172.21.15.3:0/1805566560 learned_addr learned my addr 172.21.15.3:0/1805566560 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f0534865640 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12f96200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.173+0000 7f052ffff640 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12f96740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f0535066640 1 -- 172.21.15.3:0/1805566560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 msgr2=0x55ee12f96200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f0535066640 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12f96200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f0535066640 1 -- 172.21.15.3:0/1805566560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 msgr2=0x55ee12f96740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f0535066640 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12f96740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f0535066640 1 -- 172.21.15.3:0/1805566560 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee12e97b30 con 0x55ee12f9deb0 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f052ffff640 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12f96740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f0535066640 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12f997d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f05280cc490 tx=0x7f05280cbe60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea5f6ea515382d3d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f051a7fc640 1 -- 172.21.15.3:0/1805566560 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052809b130 con 0x55ee12f9deb0 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f051a7fc640 1 -- 172.21.15.3:0/1805566560 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f052809f280 con 0x55ee12f9deb0 2024-02-09T22:56:00.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.174+0000 7f051a7fc640 1 -- 172.21.15.3:0/1805566560 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052809a580 con 0x55ee12f9deb0 2024-02-09T22:56:00.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 -- 172.21.15.3:0/1805566560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 msgr2=0x55ee12f997d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12f997d0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f05280cc490 tx=0x7f05280cbe60 comp rx=0 tx=0).stop 2024-02-09T22:56:00.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 -- 172.21.15.3:0/1805566560 shutdown_connections 2024-02-09T22:56:00.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12f96740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12f997d0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 --2- 172.21.15.3:0/1805566560 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12f96200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 -- 172.21.15.3:0/1805566560 >> 172.21.15.3:0/1805566560 conn(0x55ee12f8f050 msgr2=0x55ee12fb5160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 -- 172.21.15.3:0/1805566560 shutdown_connections 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 -- 172.21.15.3:0/1805566560 wait complete. 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 -- start start 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.175+0000 7f0539780740 1 Processor -- start 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12fa3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12fa3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12ffd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12ffde60 con 0x55ee12f9deb0 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12ffdfd0 con 0x55ee12f95330 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee12ffe190 con 0x55ee12f94980 2024-02-09T22:56:00.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0539780740 10 client.0 _ll_register_callbacks cb 0x7f052401b4f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12ffd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0534865640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12fa3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f052ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12fa3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12ffd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42860/0 (socket says 172.21.15.3:42860) 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f052ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12fa3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52978/0 (socket says 172.21.15.3:52978) 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 -- 172.21.15.3:0/51841420 learned_addr learned my addr 172.21.15.3:0/51841420 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0534865640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12fa3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42010/0 (socket says 172.21.15.3:42010) 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 -- 172.21.15.3:0/51841420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 msgr2=0x55ee12fa3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12fa3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 -- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 msgr2=0x55ee12fa3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12fa3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 -- 172.21.15.3:0/51841420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f053010d800 con 0x55ee12f9deb0 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f052ffff640 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12fa3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:00.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f0535066640 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12ffd920 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0528097b00 tx=0x7f05280a38d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01c13750615e4c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.176+0000 7f050ffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f050ffff640 1 -- 172.21.15.3:0/51841420 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052809b1b0 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f050ffff640 1 -- 172.21.15.3:0/51841420 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f052809a580 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0524018040 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f050ffff640 1 -- 172.21.15.3:0/51841420 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f052809a740 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 10 client.17322 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x7f05300697a0 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f050ffff640 1 -- 172.21.15.3:0/51841420 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0524018040 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f050ffff640 1 -- 172.21.15.3:0/51841420 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f052801f070 con 0x55ee12f9deb0 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 10 client.17322 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 10 client.17322 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 20 client.17322 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee13005590 0x55ee13007980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:00.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 4 client.17322 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ee12c88890 con 0x55ee13005590 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.177+0000 7f0534865640 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee13005590 0x55ee13007980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0534865640 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee13005590 0x55ee13007980 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90c33f9ec20120ff server_cookie=1d6673aeecd05d65 in_seq=0 out_seq=0 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f050ffff640 10 client.17322 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f050ffff640 1 -- 172.21.15.3:0/51841420 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ee12c88890 con 0x55ee13005590 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f050ffff640 10 client.17322 handle_command_reply: tid=0 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 client.17322 shutdown 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 msgr2=0x55ee12ffd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12ffd920 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f0528097b00 tx=0x7f05280a38d0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 shutdown_connections 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ee13005590 0x55ee13007980 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f050ffff640 0 client.17322 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:00.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ee12f95330 0x55ee12fa3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ee12f9deb0 0x55ee12ffd920 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 --2- 172.21.15.3:0/51841420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ee12f94980 0x55ee12fa3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 >> 172.21.15.3:0/51841420 conn(0x55ee12f8f050 msgr2=0x55ee12f96200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 shutdown_connections 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 1 -- 172.21.15.3:0/51841420 wait complete. 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.178+0000 7f0539780740 20 client.17322 trim_cache size 0 max 16384 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:56:00.188 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout: "281e9fef-9dfa-462b-9157-95c6d919c26a": { 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "281e9fef-9dfa-462b-9157-95c6d919c26a", 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:56:00.189 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:56:00.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f05200a2230 msgr2=0x7f05200a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f05200a2230 0x7f05200a46f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f0524012cc0 tx=0x7f0524014000 comp rx=0 tx=0).stop 2024-02-09T22:56:00.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f053019ae20 msgr2=0x7f053019b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f053019ae20 0x7f053019b2d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0528009fd0 tx=0x7f052800ef10 comp rx=0 tx=0).stop 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 shutdown_connections 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f05200a2230 0x7f05200a46f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f053019ae20 0x7f053019b2d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0530105420 0x7f0530196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 --2- 172.21.15.3:0/4181862038 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0530100350 0x7f05301965b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 >> 172.21.15.3:0/4181862038 conn(0x7f0530077250 msgr2=0x7f05300fdee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 shutdown_connections 2024-02-09T22:56:00.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:00.179+0000 7f0536ac9640 1 -- 172.21.15.3:0/4181862038 wait complete. 2024-02-09T22:56:00.194 INFO:tasks.cephfs.filesystem:scrub status for tag:281e9fef-9dfa-462b-9157-95c6d919c26a - {'path': '/', 'tag': '281e9fef-9dfa-462b-9157-95c6d919c26a', 'options': 'recursive,force'} 2024-02-09T22:56:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: pgmap v145: 97 pgs: 97 active+clean; 56 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 642 KiB/s wr, 87 op/s 2024-02-09T22:56:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:00.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:00.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:00.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:00 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: pgmap v145: 97 pgs: 97 active+clean; 56 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 642 KiB/s wr, 87 op/s 2024-02-09T22:56:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:00.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:00.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:00.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:00 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.636 INFO:tasks.workunit.client.0.smithi003.stdout: 1 139546 2.87 MB/sec execute 114 sec latency 329.079 ms 2024-02-09T22:56:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: pgmap v145: 97 pgs: 97 active+clean; 56 MiB data, 762 MiB used, 1.0 TiB / 1.0 TiB avail; 959 KiB/s rd, 642 KiB/s wr, 87 op/s 2024-02-09T22:56:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:00.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:00 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:01 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:01 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142166 2.99 MB/sec execute 115 sec latency 12.489 ms 2024-02-09T22:56:01.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:56:01.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:01.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:01.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:01.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:01.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:01 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:02 smithi089 ceph-mon[36665]: pgmap v146: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 207 op/s 2024-02-09T22:56:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:02 smithi003 ceph-mon[32678]: pgmap v146: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 207 op/s 2024-02-09T22:56:02.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142577 2.97 MB/sec execute 116 sec latency 565.510 ms 2024-02-09T22:56:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:02 smithi064 ceph-mon[36638]: pgmap v146: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.7 MiB/s wr, 207 op/s 2024-02-09T22:56:03.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142633 2.94 MB/sec execute 117 sec latency 983.035 ms 2024-02-09T22:56:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:04 smithi089 ceph-mon[36665]: pgmap v147: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-02-09T22:56:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:04 smithi003 ceph-mon[32678]: pgmap v147: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-02-09T22:56:04.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142633 2.92 MB/sec execute 118 sec latency 1546.795 ms 2024-02-09T22:56:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:04 smithi064 ceph-mon[36638]: pgmap v147: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-02-09T22:56:05.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142633 2.89 MB/sec execute 119 sec latency 2546.868 ms 2024-02-09T22:56:06.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 -- 172.21.15.3:0/3036426871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0841004a0 msgr2=0x7fe08410e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 --2- 172.21.15.3:0/3036426871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0841004a0 0x7fe08410e0a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fe07c009cd0 tx=0x7fe07c02c180 comp rx=0 tx=0).stop 2024-02-09T22:56:06.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 -- 172.21.15.3:0/3036426871 shutdown_connections 2024-02-09T22:56:06.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 --2- 172.21.15.3:0/3036426871 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0840ffae0 0x7fe0840fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 --2- 172.21.15.3:0/3036426871 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0841004a0 0x7fe08410e0a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 --2- 172.21.15.3:0/3036426871 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe084107420 0x7fe084107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 -- 172.21.15.3:0/3036426871 >> 172.21.15.3:0/3036426871 conn(0x7fe0840753d0 msgr2=0x7fe0840757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:06.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 -- 172.21.15.3:0/3036426871 shutdown_connections 2024-02-09T22:56:06.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.078+0000 7fe089fbc640 1 -- 172.21.15.3:0/3036426871 wait complete. 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 Processor -- start 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 -- start start 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 0x7fe084100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0841004a0 0x7fe084101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe084107420 0x7fe084101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0841a5c10 con 0x7fe0840ffae0 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0841a5d80 con 0x7fe0841004a0 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe089fbc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0841a5f40 con 0x7fe084107420 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 0x7fe084100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.084 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 0x7fe084100cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42872/0 (socket says 172.21.15.3:42872) 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.079+0000 7fe0837fe640 1 -- 172.21.15.3:0/1395820693 learned_addr learned my addr 172.21.15.3:0/1395820693 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe083fff640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe084107420 0x7fe084101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0841004a0 0x7fe084101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe0837fe640 1 -- 172.21.15.3:0/1395820693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe084107420 msgr2=0x7fe084101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe084107420 0x7fe084101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe0837fe640 1 -- 172.21.15.3:0/1395820693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0841004a0 msgr2=0x7fe084101200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0841004a0 0x7fe084101200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe0837fe640 1 -- 172.21.15.3:0/1395820693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07c009900 con 0x7fe0840ffae0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 0x7fe084100cc0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe07400c970 tx=0x7fe07400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe080ff9640 1 -- 172.21.15.3:0/1395820693 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074007bb0 con 0x7fe0840ffae0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe080ff9640 1 -- 172.21.15.3:0/1395820693 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe074007d10 con 0x7fe0840ffae0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0841a6220 con 0x7fe0840ffae0 2024-02-09T22:56:06.085 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe080ff9640 1 -- 172.21.15.3:0/1395820693 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe07400f450 con 0x7fe0840ffae0 2024-02-09T22:56:06.086 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.080+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0841a66e0 con 0x7fe0840ffae0 2024-02-09T22:56:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.082+0000 7fe080ff9640 1 -- 172.21.15.3:0/1395820693 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe074004f70 con 0x7fe0840ffae0 2024-02-09T22:56:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 Processor -- start 2024-02-09T22:56:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 -- start start 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe080ff9640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe06c0a2220 0x7fe06c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b8394f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b8394f9d0 0x563b83958260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b839587a0 0x563b8395ab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b838bda00 con 0x563b839587a0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b838baf30 con 0x563b8394f0b0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b838b6710 con 0x563b8394f9d0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe083fff640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe06c0a2220 0x7fe06c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe080ff9640 1 -- 172.21.15.3:0/1395820693 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe0740cddc0 con 0x7fe0840ffae0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe083fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b839587a0 0x563b8395ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b8394f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b8394f9d0 0x563b83958260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b8394f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53016/0 (socket says 172.21.15.3:53016) 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.083+0000 7fe0837fe640 1 -- 172.21.15.3:0/1462238489 learned_addr learned my addr 172.21.15.3:0/1462238489 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe0837fe640 1 -- 172.21.15.3:0/1462238489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b8394f9d0 msgr2=0x563b83958260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b8394f9d0 0x563b83958260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe0837fe640 1 -- 172.21.15.3:0/1462238489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b839587a0 msgr2=0x563b8395ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b839587a0 0x563b8395ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe0837fe640 1 -- 172.21.15.3:0/1462238489 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b8387b3b0 con 0x563b8394f0b0 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe083fff640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe06c0a2220 0x7fe06c0a46e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe07c006010 tx=0x7fe07c032550 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b8394f490 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe0740a29f0 tx=0x7fe07409a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bbdd27a8794db33 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe069ffb640 1 -- 172.21.15.3:0/1462238489 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074005600 con 0x563b8394f0b0 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe069ffb640 1 -- 172.21.15.3:0/1462238489 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe07409ae80 con 0x563b8394f0b0 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.084+0000 7fe069ffb640 1 -- 172.21.15.3:0/1462238489 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0740a2f90 con 0x563b8394f0b0 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1462238489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 msgr2=0x563b8394f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b8394f490 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe0740a29f0 tx=0x7fe07409a660 comp rx=0 tx=0).stop 2024-02-09T22:56:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1462238489 shutdown_connections 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b839587a0 0x563b8395ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b8394f9d0 0x563b83958260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1462238489 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b8394f490 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1462238489 >> 172.21.15.3:0/1462238489 conn(0x563b839499d0 msgr2=0x563b83949db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1462238489 shutdown_connections 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1462238489 wait complete. 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.085+0000 7fe08cb8f740 1 -- start start 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 Processor -- start 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b839557a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b83955ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83956220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8385bb90 con 0x563b8394f0b0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8358a250 con 0x563b8394f9d0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8383c200 con 0x563b839587a0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe08cb8f740 10 client.0 _ll_register_callbacks cb 0x563b838725d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b839557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b839557a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42890/0 (socket says 172.21.15.3:42890) 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 -- 172.21.15.3:0/1921143638 learned_addr learned my addr 172.21.15.3:0/1921143638 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b83955ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe083fff640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83956220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 -- 172.21.15.3:0/1921143638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 msgr2=0x563b83956220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83956220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 -- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 msgr2=0x563b83955ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b83955ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 -- 172.21.15.3:0/1921143638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0841a66e0 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b839557a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe074097150 tx=0x7fe07407e170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef0b0aa3bfa68518 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.086+0000 7fe062ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe062ffd640 1 -- 172.21.15.3:0/1921143638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe074005600 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe062ffd640 1 -- 172.21.15.3:0/1921143638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe07409a780 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b838dcee0 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe062ffd640 1 -- 172.21.15.3:0/1921143638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe07409e8d0 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 10 client.17334 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x563b838b4d90 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe062ffd640 1 -- 172.21.15.3:0/1921143638 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0740b7020 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe062ffd640 1 -- 172.21.15.3:0/1921143638 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe074016030 con 0x563b8394f0b0 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 10 client.17334 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 10 client.17334 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 20 client.17334 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b83968470 0x563b8396a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 4 client.17334 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b835b6600 con 0x563b83968470 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.087+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b83968470 0x563b8396a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.088+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b83968470 0x563b8396a860 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b4bfa08e50ec06 server_cookie=5a3d80b18ea36af8 in_seq=0 out_seq=0 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.088+0000 7fe062ffd640 10 client.17334 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe062ffd640 1 -- 172.21.15.3:0/1921143638 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563b835b6600 con 0x563b83968470 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe062ffd640 10 client.17334 handle_command_reply: tid=0 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 client.17334 shutdown 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 msgr2=0x563b839557a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b839557a0 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fe074097150 tx=0x7fe07407e170 comp rx=0 tx=0).stop 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 shutdown_connections 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b83968470 0x563b8396a860 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b83955ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83956220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe062ffd640 0 client.17334 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1921143638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b839557a0 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 >> 172.21.15.3:0/1921143638 conn(0x563b839499d0 msgr2=0x563b8394b720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 shutdown_connections 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1921143638 wait complete. 2024-02-09T22:56:06.099 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.089+0000 7fe08cb8f740 20 client.17334 trim_cache size 0 max 16384 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 Processor -- start 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 -- start start 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b83957ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 0x563b83956040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83954070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b83965470 con 0x563b8394f9d0 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b839513b0 con 0x563b8394f0b0 2024-02-09T22:56:06.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.102+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b83949500 con 0x563b839587a0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 0x563b83956040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 0x563b83956040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42904/0 (socket says 172.21.15.3:42904) 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 -- 172.21.15.3:0/2234019407 learned_addr learned my addr 172.21.15.3:0/2234019407 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe0837fe640 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b83957ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 -- 172.21.15.3:0/2234019407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 msgr2=0x563b83954070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83954070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 -- 172.21.15.3:0/2234019407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 msgr2=0x563b83957ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b83957ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 -- 172.21.15.3:0/2234019407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b83851820 con 0x563b8394f9d0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe082ffd640 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 0x563b83956040 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe07800be10 tx=0x7fe07800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=373efc1ab3376cc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe0697fa640 1 -- 172.21.15.3:0/2234019407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078016040 con 0x563b8394f9d0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe0697fa640 1 -- 172.21.15.3:0/2234019407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe078017040 con 0x563b8394f9d0 2024-02-09T22:56:06.108 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.103+0000 7fe0697fa640 1 -- 172.21.15.3:0/2234019407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe078003970 con 0x563b8394f9d0 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 -- 172.21.15.3:0/2234019407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 msgr2=0x563b83956040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 0x563b83956040 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fe07800be10 tx=0x7fe07800b410 comp rx=0 tx=0).stop 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 -- 172.21.15.3:0/2234019407 shutdown_connections 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b83954070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f9d0 0x563b83956040 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/2234019407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f0b0 0x563b83957ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 -- 172.21.15.3:0/2234019407 >> 172.21.15.3:0/2234019407 conn(0x563b839499d0 msgr2=0x563b8396f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 -- 172.21.15.3:0/2234019407 shutdown_connections 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 -- 172.21.15.3:0/2234019407 wait complete. 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 -- start start 2024-02-09T22:56:06.109 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.104+0000 7fe08cb8f740 1 Processor -- start 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b8395f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b8395dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b8395e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b8395eb90 con 0x563b8394f0b0 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b839be8f0 con 0x563b8394f9d0 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b839bea30 con 0x563b839587a0 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 10 client.0 _ll_register_callbacks cb 0x7fe07c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:06.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b8395f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b8395f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42908/0 (socket says 172.21.15.3:42908) 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 -- 172.21.15.3:0/1035328152 learned_addr learned my addr 172.21.15.3:0/1035328152 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe083fff640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b8395e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b8395dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 -- 172.21.15.3:0/1035328152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 msgr2=0x563b8395e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b8395e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 -- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 msgr2=0x563b8395dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b8395dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 -- 172.21.15.3:0/1035328152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07800bb40 con 0x563b8394f0b0 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b8395dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe083fff640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b8395e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe0837fe640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b8395f9e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe074007590 tx=0x7fe0740cba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8ac42b4281634 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe062ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:06.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe062ffd640 1 -- 172.21.15.3:0/1035328152 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0740792f0 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe062ffd640 1 -- 172.21.15.3:0/1035328152 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe0740b8080 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.105+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe07409ae80 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe062ffd640 1 -- 172.21.15.3:0/1035328152 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe07407cae0 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 10 client.17340 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x563b838776f0 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe062ffd640 1 -- 172.21.15.3:0/1035328152 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe07409ae80 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe062ffd640 1 -- 172.21.15.3:0/1035328152 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe074016040 con 0x563b8394f0b0 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 10 client.17340 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 10 client.17340 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 20 client.17340 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b839bfe70 0x563b839c2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 4 client.17340 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563b836071b0 con 0x563b839bfe70 2024-02-09T22:56:06.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.106+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b839bfe70 0x563b839c2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:06.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.107+0000 7fe062ffd640 10 client.17340 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:06.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.107+0000 7fe082ffd640 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b839bfe70 0x563b839c2290 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47890775cc6e9870 server_cookie=aaae4f1088dcd42 in_seq=0 out_seq=0 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe062ffd640 1 -- 172.21.15.3:0/1035328152 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563b836071b0 con 0x563b839bfe70 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe062ffd640 10 client.17340 handle_command_reply: tid=0 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 client.17340 shutdown 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 msgr2=0x563b8395f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b8395f9e0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7fe074007590 tx=0x7fe0740cba40 comp rx=0 tx=0).stop 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 shutdown_connections 2024-02-09T22:56:06.154 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b839bfe70 0x563b839c2290 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b8394f9d0 0x563b8395dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b839587a0 0x563b8395e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe062ffd640 0 client.17340 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 --2- 172.21.15.3:0/1035328152 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b8394f0b0 0x563b8395f9e0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 >> 172.21.15.3:0/1035328152 conn(0x563b839499d0 msgr2=0x563b83953fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 shutdown_connections 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 1 -- 172.21.15.3:0/1035328152 wait complete. 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.150+0000 7fe08cb8f740 20 client.17340 trim_cache size 0 max 16384 2024-02-09T22:56:06.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:56:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:56:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:56:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:56:06.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe06c0a2220 msgr2=0x7fe06c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe06c0a2220 0x7fe06c0a46e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe07c006010 tx=0x7fe07c032550 comp rx=0 tx=0).stop 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 msgr2=0x7fe084100cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 0x7fe084100cc0 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7fe07400c970 tx=0x7fe07400ce30 comp rx=0 tx=0).stop 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 shutdown_connections 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe06c0a2220 0x7fe06c0a46e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe0841004a0 0x7fe084101200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe084107420 0x7fe084101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 --2- 172.21.15.3:0/1395820693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe0840ffae0 0x7fe084100cc0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 >> 172.21.15.3:0/1395820693 conn(0x7fe0840753d0 msgr2=0x7fe0840fe690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 shutdown_connections 2024-02-09T22:56:06.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:06.151+0000 7fe089fbc640 1 -- 172.21.15.3:0/1395820693 wait complete. 2024-02-09T22:56:06.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:06 smithi089 ceph-mon[36665]: pgmap v148: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-02-09T22:56:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:06 smithi003 ceph-mon[32678]: pgmap v148: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-02-09T22:56:06.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142633 2.87 MB/sec execute 120 sec latency 3546.951 ms 2024-02-09T22:56:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:06 smithi064 ceph-mon[36638]: pgmap v148: 97 pgs: 97 active+clean; 59 MiB data, 794 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 194 op/s 2024-02-09T22:56:07.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:07.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:07.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:07.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:07.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142633 2.84 MB/sec execute 121 sec latency 4547.061 ms 2024-02-09T22:56:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:08 smithi089 ceph-mon[36665]: pgmap v149: 97 pgs: 97 active+clean; 62 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2024-02-09T22:56:08.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:08 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:08 smithi003 ceph-mon[32678]: pgmap v149: 97 pgs: 97 active+clean; 62 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2024-02-09T22:56:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:08 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:08.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142701 2.82 MB/sec execute 122 sec latency 4955.596 ms 2024-02-09T22:56:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:08 smithi064 ceph-mon[36638]: pgmap v149: 97 pgs: 97 active+clean; 62 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.2 MiB/s wr, 260 op/s 2024-02-09T22:56:08.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:08 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:09.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142701 2.80 MB/sec execute 123 sec latency 1545.508 ms 2024-02-09T22:56:10.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:10 smithi089 ceph-mon[36665]: pgmap v150: 97 pgs: 97 active+clean; 62 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-02-09T22:56:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:10 smithi003 ceph-mon[32678]: pgmap v150: 97 pgs: 97 active+clean; 62 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-02-09T22:56:10.637 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142701 2.78 MB/sec execute 124 sec latency 2545.559 ms 2024-02-09T22:56:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:10 smithi064 ceph-mon[36638]: pgmap v150: 97 pgs: 97 active+clean; 62 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 185 op/s 2024-02-09T22:56:11.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142701 2.75 MB/sec execute 125 sec latency 3545.660 ms 2024-02-09T22:56:12.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:12 smithi089 ceph-mon[36665]: pgmap v151: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-02-09T22:56:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:12 smithi003 ceph-mon[32678]: pgmap v151: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-02-09T22:56:12.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 142701 2.73 MB/sec execute 126 sec latency 4545.755 ms 2024-02-09T22:56:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:12 smithi064 ceph-mon[36638]: pgmap v151: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 218 op/s 2024-02-09T22:56:13.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 144352 2.76 MB/sec execute 127 sec latency 4954.504 ms 2024-02-09T22:56:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:14 smithi003 ceph-mon[32678]: pgmap v152: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 849 KiB/s wr, 98 op/s 2024-02-09T22:56:14.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 146056 2.84 MB/sec execute 128 sec latency 13.598 ms 2024-02-09T22:56:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:14 smithi064 ceph-mon[36638]: pgmap v152: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 849 KiB/s wr, 98 op/s 2024-02-09T22:56:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:14 smithi089 ceph-mon[36665]: pgmap v152: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 849 KiB/s wr, 98 op/s 2024-02-09T22:56:15.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 146250 2.82 MB/sec execute 129 sec latency 891.770 ms 2024-02-09T22:56:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:16 smithi003 ceph-mon[32678]: pgmap v153: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 849 KiB/s wr, 98 op/s 2024-02-09T22:56:16.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 146250 2.80 MB/sec execute 130 sec latency 1891.903 ms 2024-02-09T22:56:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:16 smithi064 ceph-mon[36638]: pgmap v153: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 849 KiB/s wr, 98 op/s 2024-02-09T22:56:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:16 smithi089 ceph-mon[36665]: pgmap v153: 97 pgs: 97 active+clean; 60 MiB data, 837 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 849 KiB/s wr, 98 op/s 2024-02-09T22:56:17.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 146250 2.77 MB/sec execute 131 sec latency 2892.001 ms 2024-02-09T22:56:18.529 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:18 smithi089 ceph-mon[36665]: pgmap v154: 97 pgs: 97 active+clean; 60 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-02-09T22:56:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:18 smithi003 ceph-mon[32678]: pgmap v154: 97 pgs: 97 active+clean; 60 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-02-09T22:56:18.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 147850 2.80 MB/sec execute 132 sec latency 3299.980 ms 2024-02-09T22:56:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:18 smithi064 ceph-mon[36638]: pgmap v154: 97 pgs: 97 active+clean; 60 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 -- 172.21.15.3:0/729865783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 msgr2=0x7f4f5010dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 --2- 172.21.15.3:0/729865783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f5010dc80 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f4f48009ca0 tx=0x7f4f4802c230 comp rx=0 tx=0).stop 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 -- 172.21.15.3:0/729865783 shutdown_connections 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 --2- 172.21.15.3:0/729865783 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f50102dc0 0x7f4f5010b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 --2- 172.21.15.3:0/729865783 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f5010dc80 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 --2- 172.21.15.3:0/729865783 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f501024a0 0x7f4f50102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 -- 172.21.15.3:0/729865783 >> 172.21.15.3:0/729865783 conn(0x7f4f50077250 msgr2=0x7f4f50077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.244+0000 7f4f55af4640 1 -- 172.21.15.3:0/729865783 shutdown_connections 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 -- 172.21.15.3:0/729865783 wait complete. 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 Processor -- start 2024-02-09T22:56:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 -- start start 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 0x7f4f5019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f50102dc0 0x7f4f5019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f501976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50197e30 con 0x7f4f5010b7e0 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50197fa0 con 0x7f4f501024a0 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.245+0000 7f4f55af4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f50198160 con 0x7f4f50102dc0 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f501976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 0x7f4f5019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f50102dc0 0x7f4f5019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 0x7f4f5019d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:59970/0 (socket says 172.21.15.3:59970) 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f501976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35518/0 (socket says 172.21.15.3:35518) 2024-02-09T22:56:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/692342951 learned_addr learned my addr 172.21.15.3:0/692342951 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 -- 172.21.15.3:0/692342951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 msgr2=0x7f4f5019d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 0x7f4f5019d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 -- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 msgr2=0x7f4f501976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f501976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 -- 172.21.15.3:0/692342951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f44009490 con 0x7f4f50102dc0 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f501976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 0x7f4f5019d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4effd640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f50102dc0 0x7f4f5019dad0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400e9f0 tx=0x7f4f4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4cff9640 1 -- 172.21.15.3:0/692342951 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f4400cd80 con 0x7f4f50102dc0 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4cff9640 1 -- 172.21.15.3:0/692342951 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f440035f0 con 0x7f4f50102dc0 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f4cff9640 1 -- 172.21.15.3:0/692342951 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f44010690 con 0x7f4f50102dc0 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f48009900 con 0x7f4f50102dc0 2024-02-09T22:56:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.246+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f501a4510 con 0x7f4f50102dc0 2024-02-09T22:56:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.248+0000 7f4f4cff9640 1 -- 172.21.15.3:0/692342951 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f4f44003af0 con 0x7f4f50102dc0 2024-02-09T22:56:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 Processor -- start 2024-02-09T22:56:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 -- start start 2024-02-09T22:56:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab5b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4cff9640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4f380a2220 0x7f4f380a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab64260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab66b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcaac9a00 con 0x555dcab647a0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcaac6f30 con 0x555dcab5b0b0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f587a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcaac2710 con 0x555dcab5b9d0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4f380a2220 0x7f4f380a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab5b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab64260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab5b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:59986/0 (socket says 172.21.15.3:59986) 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4ffff640 1 -- 172.21.15.3:0/385454704 learned_addr learned my addr 172.21.15.3:0/385454704 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab66b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.249+0000 7f4f4cff9640 1 -- 172.21.15.3:0/692342951 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f44014070 con 0x7f4f50102dc0 2024-02-09T22:56:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4ffff640 1 -- 172.21.15.3:0/385454704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 msgr2=0x555dcab64260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab64260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4ffff640 1 -- 172.21.15.3:0/385454704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 msgr2=0x555dcab66b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab66b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4ffff640 1 -- 172.21.15.3:0/385454704 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555dcaa873b0 con 0x555dcab5b0b0 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4effd640 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab64260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4f380a2220 0x7f4f380a46e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4f40011500 tx=0x7f4f40005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab5b490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x555dcab56b60 tx=0x7f4f4802e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fde8b85d7bbb224 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f35ffb640 1 -- 172.21.15.3:0/385454704 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48034800 con 0x555dcab5b0b0 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f35ffb640 1 -- 172.21.15.3:0/385454704 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f48003e50 con 0x555dcab5b0b0 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.250+0000 7f4f35ffb640 1 -- 172.21.15.3:0/385454704 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48004030 con 0x555dcab5b0b0 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- 172.21.15.3:0/385454704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 msgr2=0x555dcab5b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab5b490 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x555dcab56b60 tx=0x7f4f4802e040 comp rx=0 tx=0).stop 2024-02-09T22:56:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- 172.21.15.3:0/385454704 shutdown_connections 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab66b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab64260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- 172.21.15.3:0/385454704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab5b490 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- 172.21.15.3:0/385454704 >> 172.21.15.3:0/385454704 conn(0x555dcab559d0 msgr2=0x555dcab55db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- 172.21.15.3:0/385454704 shutdown_connections 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- 172.21.15.3:0/385454704 wait complete. 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- start start 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 Processor -- start 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab617c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab61d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab62240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcaa67b90 con 0x555dcab647a0 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dca796250 con 0x555dcab5b0b0 2024-02-09T22:56:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.251+0000 7f4f587a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcaa48200 con 0x555dcab5b9d0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f587a4740 10 client.0 _ll_register_callbacks cb 0x555dcaa7e5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab62240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab617c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab62240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35536/0 (socket says 172.21.15.3:35536) 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 -- 172.21.15.3:0/2687485465 learned_addr learned my addr 172.21.15.3:0/2687485465 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4effd640 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab61d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 -- 172.21.15.3:0/2687485465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 msgr2=0x555dcab61d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab61d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 -- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 msgr2=0x555dcab617c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab617c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 -- 172.21.15.3:0/2687485465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f5010e900 con 0x555dcab647a0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f4ffff640 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab62240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4f48037040 tx=0x7f4f48032b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37e1c725b02cbf9b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f2effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f2effd640 1 -- 172.21.15.3:0/2687485465 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f48034800 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f2effd640 1 -- 172.21.15.3:0/2687485465 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f48003970 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555dcaae8ee0 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f2effd640 1 -- 172.21.15.3:0/2687485465 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f48003b70 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f587a4740 10 client.17352 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.252+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x555dcaac0d90 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f2effd640 1 -- 172.21.15.3:0/2687485465 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555dcaae8ee0 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f2effd640 1 -- 172.21.15.3:0/2687485465 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f480424b0 con 0x555dcab647a0 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f587a4740 10 client.17352 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f587a4740 10 client.17352 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f587a4740 20 client.17352 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f587a4740 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcab74540 0x555dcab76930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f587a4740 4 client.17352 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555dca7c2600 con 0x555dcab74540 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcab74540 0x555dcab76930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcab74540 0x555dcab76930 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fb8f1e0ef3dc8e2 server_cookie=925953758389a220 in_seq=0 out_seq=0 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.253+0000 7f4f2effd640 10 client.17352 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f2effd640 1 -- 172.21.15.3:0/2687485465 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555dca7c2600 con 0x555dcab74540 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f2effd640 10 client.17352 handle_command_reply: tid=0 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 client.17352 shutdown 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 msgr2=0x555dcab62240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab62240 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f4f48037040 tx=0x7f4f48032b30 comp rx=0 tx=0).stop 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 shutdown_connections 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcab74540 0x555dcab76930 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab62240 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b9d0 0x555dcab61d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f2effd640 0 client.17352 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 --2- 172.21.15.3:0/2687485465 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b0b0 0x555dcab617c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 >> 172.21.15.3:0/2687485465 conn(0x555dcab559d0 msgr2=0x555dcab57720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 shutdown_connections 2024-02-09T22:56:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 1 -- 172.21.15.3:0/2687485465 wait complete. 2024-02-09T22:56:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.254+0000 7f4f587a4740 20 client.17352 trim_cache size 0 max 16384 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 Processor -- start 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 -- start start 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 0x555dcab63ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab62060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab647a0 0x555dcab5ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcaae8f10 con 0x555dcab5b0b0 2024-02-09T22:56:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcab5d460 con 0x555dcab5b9d0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f587a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcab71430 con 0x555dcab647a0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 0x555dcab63ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 0x555dcab63ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35538/0 (socket says 172.21.15.3:35538) 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/353439408 learned_addr learned my addr 172.21.15.3:0/353439408 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab647a0 0x555dcab5ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.269+0000 7f4f4effd640 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab62060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/353439408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab647a0 msgr2=0x555dcab5ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab647a0 0x555dcab5ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/353439408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 msgr2=0x555dcab62060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab62060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/353439408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555dcaa5d820 con 0x555dcab5b0b0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4effd640 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab62060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 0x555dcab63ef0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x555dcab771e0 tx=0x7f4f40014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec44253faac60757 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f357fa640 1 -- 172.21.15.3:0/353439408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40021040 con 0x555dcab5b0b0 2024-02-09T22:56:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f357fa640 1 -- 172.21.15.3:0/353439408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f40029030 con 0x555dcab5b0b0 2024-02-09T22:56:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.270+0000 7f4f357fa640 1 -- 172.21.15.3:0/353439408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f400286c0 con 0x555dcab5b0b0 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 -- 172.21.15.3:0/353439408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 msgr2=0x555dcab63ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 0x555dcab63ef0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x555dcab771e0 tx=0x7f4f40014000 comp rx=0 tx=0).stop 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 -- 172.21.15.3:0/353439408 shutdown_connections 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab62060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab647a0 0x555dcab5ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 --2- 172.21.15.3:0/353439408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab5b0b0 0x555dcab63ef0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 -- 172.21.15.3:0/353439408 >> 172.21.15.3:0/353439408 conn(0x555dcab559d0 msgr2=0x555dcab7b230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 -- 172.21.15.3:0/353439408 shutdown_connections 2024-02-09T22:56:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 -- 172.21.15.3:0/353439408 wait complete. 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.271+0000 7f4f587a4740 1 -- start start 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 Processor -- start 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 0x555dcab6b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab69d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab6a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcab6ab20 con 0x555dcab647a0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcabd31d0 con 0x555dcab5b9d0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dcabd3310 con 0x555dcab5b0b0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f587a4740 10 client.0 _ll_register_callbacks cb 0x7f4f40005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 0x555dcab6b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab6a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab69d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 0x555dcab6b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:48504/0 (socket says 172.21.15.3:48504) 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/530538336 learned_addr learned my addr 172.21.15.3:0/530538336 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 msgr2=0x555dcab69d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab69d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/530538336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 msgr2=0x555dcab6a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab6a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.272+0000 7f4f4f7fe640 1 -- 172.21.15.3:0/530538336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f40017050 con 0x555dcab5b0b0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f4f7fe640 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 0x555dcab6b970 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000f930 tx=0x7f4f400292e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3609f4dc372e57f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f2effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f2effd640 1 -- 172.21.15.3:0/530538336 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f40021050 con 0x555dcab5b0b0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f2effd640 1 -- 172.21.15.3:0/530538336 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4f4001b850 con 0x555dcab5b0b0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4f48003e50 con 0x555dcab5b0b0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f2effd640 1 -- 172.21.15.3:0/530538336 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f4001ba10 con 0x555dcab5b0b0 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 10 client.26680 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x555dcaa836f0 con 0x555dcab5b0b0 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f2effd640 1 -- 172.21.15.3:0/530538336 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f48003e50 con 0x555dcab5b0b0 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f2effd640 1 -- 172.21.15.3:0/530538336 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4f400047c0 con 0x555dcab5b0b0 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 10 client.26680 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 10 client.26680 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 20 client.26680 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcabd4a20 0x555dcabd6e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 4 client.26680 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555dca8131b0 con 0x555dcabd4a20 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.273+0000 7f4f4effd640 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcabd4a20 0x555dcabd6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.274+0000 7f4f4effd640 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcabd4a20 0x555dcabd6e40 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5bf9fd1782c7943c server_cookie=cbe10d7b98f779d1 in_seq=0 out_seq=0 2024-02-09T22:56:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.274+0000 7f4f2effd640 10 client.26680 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:19.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.304+0000 7f4f2effd640 1 -- 172.21.15.3:0/530538336 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555dca8131b0 con 0x555dcabd4a20 2024-02-09T22:56:19.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.304+0000 7f4f2effd640 10 client.26680 handle_command_reply: tid=0 2024-02-09T22:56:19.309 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 client.26680 shutdown 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 msgr2=0x555dcab6b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 0x555dcab6b970 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f4f4000f930 tx=0x7f4f400292e0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 shutdown_connections 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555dcabd4a20 0x555dcabd6e40 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555dcab5b9d0 0x555dcab69d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555dcab647a0 0x555dcab6a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 --2- 172.21.15.3:0/530538336 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555dcab5b0b0 0x555dcab6b970 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f2effd640 0 client.26680 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 >> 172.21.15.3:0/530538336 conn(0x555dcab559d0 msgr2=0x555dcab541a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 shutdown_connections 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 1 -- 172.21.15.3:0/530538336 wait complete. 2024-02-09T22:56:19.310 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.305+0000 7f4f587a4740 20 client.26680 trim_cache size 0 max 16384 2024-02-09T22:56:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:56:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:56:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:56:19.311 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4f380a2220 msgr2=0x7f4f380a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4f380a2220 0x7f4f380a46e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f4f40011500 tx=0x7f4f40005f90 comp rx=0 tx=0).stop 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f50102dc0 msgr2=0x7f4f5019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f50102dc0 0x7f4f5019dad0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4f4400e9f0 tx=0x7f4f4400eeb0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 shutdown_connections 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4f380a2220 0x7f4f380a46e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4f5010b7e0 0x7f4f501976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4f50102dc0 0x7f4f5019dad0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 --2- 172.21.15.3:0/692342951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4f501024a0 0x7f4f5019d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:19.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 >> 172.21.15.3:0/692342951 conn(0x7f4f50077250 msgr2=0x7f4f500fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:19.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 shutdown_connections 2024-02-09T22:56:19.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:19.306+0000 7f4f55af4640 1 -- 172.21.15.3:0/692342951 wait complete. 2024-02-09T22:56:19.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149451 2.87 MB/sec execute 133 sec latency 23.858 ms 2024-02-09T22:56:20.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:20 smithi003 ceph-mon[32678]: pgmap v155: 97 pgs: 97 active+clean; 60 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 573 KiB/s wr, 83 op/s 2024-02-09T22:56:20.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149667 2.85 MB/sec execute 134 sec latency 834.891 ms 2024-02-09T22:56:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:20 smithi064 ceph-mon[36638]: pgmap v155: 97 pgs: 97 active+clean; 60 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 573 KiB/s wr, 83 op/s 2024-02-09T22:56:20.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:20 smithi089 ceph-mon[36665]: pgmap v155: 97 pgs: 97 active+clean; 60 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 573 KiB/s wr, 83 op/s 2024-02-09T22:56:21.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149667 2.83 MB/sec execute 135 sec latency 1835.021 ms 2024-02-09T22:56:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:22 smithi003 ceph-mon[32678]: pgmap v156: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-02-09T22:56:22.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149667 2.81 MB/sec execute 136 sec latency 2835.071 ms 2024-02-09T22:56:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:22 smithi064 ceph-mon[36638]: pgmap v156: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-02-09T22:56:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:22 smithi089 ceph-mon[36665]: pgmap v156: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 149 op/s 2024-02-09T22:56:23.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149723 2.79 MB/sec execute 137 sec latency 3241.233 ms 2024-02-09T22:56:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:24 smithi003 ceph-mon[32678]: pgmap v157: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 787 KiB/s wr, 116 op/s 2024-02-09T22:56:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:56:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:56:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:56:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:24 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:56:24.638 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149723 2.77 MB/sec execute 138 sec latency 1556.786 ms 2024-02-09T22:56:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:24 smithi064 ceph-mon[36638]: pgmap v157: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 787 KiB/s wr, 116 op/s 2024-02-09T22:56:24.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:56:24.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:56:24.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:56:24.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:24 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:56:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:24 smithi089 ceph-mon[36665]: pgmap v157: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 787 KiB/s wr, 116 op/s 2024-02-09T22:56:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:56:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:56:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:56:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:24 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:56:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:25 smithi003 ceph-mon[32678]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 787 KiB/s wr, 116 op/s 2024-02-09T22:56:25.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149723 2.75 MB/sec execute 139 sec latency 2556.854 ms 2024-02-09T22:56:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:25 smithi064 ceph-mon[36638]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 787 KiB/s wr, 116 op/s 2024-02-09T22:56:25.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:25 smithi089 ceph-mon[36665]: pgmap v158: 97 pgs: 97 active+clean; 58 MiB data, 886 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 787 KiB/s wr, 116 op/s 2024-02-09T22:56:26.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149723 2.73 MB/sec execute 140 sec latency 3556.971 ms 2024-02-09T22:56:27.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149723 2.71 MB/sec execute 141 sec latency 4557.077 ms 2024-02-09T22:56:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:27 smithi064 ceph-mon[36638]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 KiB/s wr, 130 op/s 2024-02-09T22:56:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:27 smithi089 ceph-mon[36665]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 KiB/s wr, 130 op/s 2024-02-09T22:56:27.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:27 smithi003 ceph-mon[32678]: pgmap v159: 97 pgs: 97 active+clean; 58 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 837 KiB/s wr, 130 op/s 2024-02-09T22:56:28.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149791 2.69 MB/sec execute 142 sec latency 4963.130 ms 2024-02-09T22:56:29.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149791 2.68 MB/sec execute 143 sec latency 1548.153 ms 2024-02-09T22:56:30.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:56:30.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 -- 172.21.15.3:0/4251575103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0101de0 msgr2=0x7f5fa0105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 --2- 172.21.15.3:0/4251575103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0101de0 0x7f5fa0105e20 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f5f90009930 tx=0x7f5f9002bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 -- 172.21.15.3:0/4251575103 shutdown_connections 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 --2- 172.21.15.3:0/4251575103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fa0106500 0x7f5fa010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 --2- 172.21.15.3:0/4251575103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0101de0 0x7f5fa0105e20 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 --2- 172.21.15.3:0/4251575103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0101430 0x7f5fa0101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 -- 172.21.15.3:0/4251575103 >> 172.21.15.3:0/4251575103 conn(0x7f5fa00fccd0 msgr2=0x7f5fa00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.315 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 -- 172.21.15.3:0/4251575103 shutdown_connections 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.309+0000 7f5fa848f640 1 -- 172.21.15.3:0/4251575103 wait complete. 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 Processor -- start 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 -- start start 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fa0101430 0x7f5fa0073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0106500 0x7f5fa0073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 0x7f5fa01a5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa01a6040 con 0x7f5fa0073fb0 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa01a61b0 con 0x7f5fa0106500 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa848f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fa01a6370 con 0x7f5fa0101430 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa6a2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 0x7f5fa01a5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa6a2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 0x7f5fa01a5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33434/0 (socket says 172.21.15.3:33434) 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.310+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/1108692753 learned_addr learned my addr 172.21.15.3:0/1108692753 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa622b640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fa0101430 0x7f5fa0073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/1108692753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fa0101430 msgr2=0x7f5fa0073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fa0101430 0x7f5fa0073530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/1108692753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0106500 msgr2=0x7f5fa0073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0106500 0x7f5fa0073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0106500 0x7f5fa0073a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/1108692753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90009560 con 0x7f5fa0073fb0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 0x7f5fa01a5a70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800eb10 tx=0x7f5f9800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5f977fe640 1 -- 172.21.15.3:0/1108692753 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98004b20 con 0x7f5fa0073fb0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5f977fe640 1 -- 172.21.15.3:0/1108692753 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f98004c80 con 0x7f5fa0073fb0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa01a6650 con 0x7f5fa0073fb0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5f977fe640 1 -- 172.21.15.3:0/1108692753 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f98010650 con 0x7f5fa0073fb0 2024-02-09T22:56:30.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.311+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa01a6b60 con 0x7f5fa0073fb0 2024-02-09T22:56:30.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.313+0000 7f5f977fe640 1 -- 172.21.15.3:0/1108692753 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5f98005020 con 0x7f5fa0073fb0 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 Processor -- start 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 -- start start 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5f977fe640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f8c0a2260 0x7f5f8c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 0x5643fe990d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe9998e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe999eb0 0x5643fe99c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe94bd00 con 0x5643fe990980 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe9002f0 con 0x5643fe999eb0 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fab06e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe8fd820 con 0x5643fe991330 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fa622b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 0x5643fe990d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f8c0a2260 0x7f5f8c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5f977fe640 1 -- 172.21.15.3:0/1108692753 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5f98014070 con 0x7f5fa0073fb0 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fa622b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 0x5643fe990d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33442/0 (socket says 172.21.15.3:33442) 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.314+0000 7f5fa622b640 1 -- 172.21.15.3:0/611266239 learned_addr learned my addr 172.21.15.3:0/611266239 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe999eb0 0x5643fe99c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe9998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa622b640 1 -- 172.21.15.3:0/611266239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 msgr2=0x5643fe9998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa622b640 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe9998e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa622b640 1 -- 172.21.15.3:0/611266239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe999eb0 msgr2=0x5643fe99c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa622b640 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe999eb0 0x5643fe99c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa622b640 1 -- 172.21.15.3:0/611266239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643fe8baff0 con 0x5643fe990980 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f8c0a2260 0x7f5f8c0a4720 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5f90036040 tx=0x7f5f900047c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5fa622b640 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 0x5643fe990d60 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009c50 tx=0x7f5f9c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9ae1aee041f0292 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5f94bfe640 1 -- 172.21.15.3:0/611266239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c009720 con 0x5643fe990980 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5f94bfe640 1 -- 172.21.15.3:0/611266239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c0035f0 con 0x5643fe990980 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.315+0000 7f5f94bfe640 1 -- 172.21.15.3:0/611266239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c0126b0 con 0x5643fe990980 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 -- 172.21.15.3:0/611266239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 msgr2=0x5643fe990d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 0x5643fe990d60 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009c50 tx=0x7f5f9c010740 comp rx=0 tx=0).stop 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 -- 172.21.15.3:0/611266239 shutdown_connections 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe999eb0 0x5643fe99c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe9998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 --2- 172.21.15.3:0/611266239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe990980 0x5643fe990d60 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 -- 172.21.15.3:0/611266239 >> 172.21.15.3:0/611266239 conn(0x5643fe98b050 msgr2=0x5643fe98b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 -- 172.21.15.3:0/611266239 shutdown_connections 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 -- 172.21.15.3:0/611266239 wait complete. 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.316+0000 7f5fab06e740 1 -- start start 2024-02-09T22:56:30.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 Processor -- start 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe996e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe997360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 0x5643fe9978a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe8f9000 con 0x5643fe999eb0 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe5cc430 con 0x5643fe990980 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe588e00 con 0x5643fe991330 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fab06e740 10 client.0 _ll_register_callbacks cb 0x5643fe8ed7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 0x5643fe9978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 0x5643fe9978a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33444/0 (socket says 172.21.15.3:33444) 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/2051556716 learned_addr learned my addr 172.21.15.3:0/2051556716 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa622b640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe996e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe997360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/2051556716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 msgr2=0x5643fe997360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe997360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 msgr2=0x5643fe996e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe996e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 -- 172.21.15.3:0/2051556716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c009880 con 0x5643fe999eb0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 0x5643fe9978a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5f98097ab0 tx=0x7f5f98009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314ce9179b831d29 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.317+0000 7f5f89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5f89ffb640 1 -- 172.21.15.3:0/2051556716 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f980bc070 con 0x5643fe999eb0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5f89ffb640 1 -- 172.21.15.3:0/2051556716 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5f98097750 con 0x5643fe999eb0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5643fe5f87e0 con 0x5643fe999eb0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5f89ffb640 1 -- 172.21.15.3:0/2051556716 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f980bc1d0 con 0x5643fe999eb0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 10 client.17370 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5643fe8f7680 con 0x5643fe999eb0 2024-02-09T22:56:30.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5f89ffb640 1 -- 172.21.15.3:0/2051556716 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5643fe5f87e0 con 0x5643fe999eb0 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5f89ffb640 1 -- 172.21.15.3:0/2051556716 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5f9801f060 con 0x5643fe999eb0 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 10 client.17370 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 10 client.17370 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 20 client.17370 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe9aa410 0x5643fe9ac800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 4 client.17370 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5643fe635720 con 0x5643fe9aa410 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.318+0000 7f5fa622b640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe9aa410 0x5643fe9ac800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.319+0000 7f5fa622b640 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe9aa410 0x5643fe9ac800 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f853b9efa05d603c server_cookie=77bfa11bacd7a92c in_seq=0 out_seq=0 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.319+0000 7f5f89ffb640 10 client.17370 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.319+0000 7f5f89ffb640 1 -- 172.21.15.3:0/2051556716 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5643fe635720 con 0x5643fe9aa410 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.319+0000 7f5f89ffb640 10 client.17370 handle_command_reply: tid=0 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.319+0000 7f5fab06e740 1 client.17370 shutdown 2024-02-09T22:56:30.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 msgr2=0x5643fe9978a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 0x5643fe9978a0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f5f98097ab0 tx=0x7f5f98009a70 comp rx=0 tx=0).stop 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 shutdown_connections 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe9aa410 0x5643fe9ac800 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe999eb0 0x5643fe9978a0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe997360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5f89ffb640 0 client.17370 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 --2- 172.21.15.3:0/2051556716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe996e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 >> 172.21.15.3:0/2051556716 conn(0x5643fe98b050 msgr2=0x5643fe98cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 shutdown_connections 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 1 -- 172.21.15.3:0/2051556716 wait complete. 2024-02-09T22:56:30.331 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.320+0000 7f5fab06e740 20 client.17370 trim_cache size 0 max 16384 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 Processor -- start 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 -- start start 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9956f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe990980 0x5643fe995c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe991330 0x5643fe994080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe986780 con 0x5643fe98b050 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe87dfd0 con 0x5643fe991330 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fab06e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe997f60 con 0x5643fe990980 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9956f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9956f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33452/0 (socket says 172.21.15.3:33452) 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 -- 172.21.15.3:0/4044471143 learned_addr learned my addr 172.21.15.3:0/4044471143 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe991330 0x5643fe994080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe990980 0x5643fe995c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 -- 172.21.15.3:0/4044471143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe990980 msgr2=0x5643fe995c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe990980 0x5643fe995c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 -- 172.21.15.3:0/4044471143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe991330 msgr2=0x5643fe994080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe991330 0x5643fe994080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 -- 172.21.15.3:0/4044471143 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5643fe893b30 con 0x5643fe98b050 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.338+0000 7f5fa622b640 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9956f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009c50 tx=0x7f5f9c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39e4f3d7e4f6c31f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5f8bfff640 1 -- 172.21.15.3:0/4044471143 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c020040 con 0x5643fe98b050 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5f8bfff640 1 -- 172.21.15.3:0/4044471143 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c021040 con 0x5643fe98b050 2024-02-09T22:56:30.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5f8bfff640 1 -- 172.21.15.3:0/4044471143 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c0280d0 con 0x5643fe98b050 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 -- 172.21.15.3:0/4044471143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 msgr2=0x5643fe9956f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9956f0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009c50 tx=0x7f5f9c009fd0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 -- 172.21.15.3:0/4044471143 shutdown_connections 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe991330 0x5643fe994080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe990980 0x5643fe995c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 --2- 172.21.15.3:0/4044471143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9956f0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 -- 172.21.15.3:0/4044471143 >> 172.21.15.3:0/4044471143 conn(0x5643fe991a80 msgr2=0x5643fe9b1180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 -- 172.21.15.3:0/4044471143 shutdown_connections 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.339+0000 7f5fab06e740 1 -- 172.21.15.3:0/4044471143 wait complete. 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 -- start start 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 Processor -- start 2024-02-09T22:56:30.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe99f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe99fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe9a0360 con 0x5643fe98b050 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fe9a04d0 con 0x5643fe990980 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5643fea00910 con 0x5643fe991330 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fab06e740 10 client.0 _ll_register_callbacks cb 0x7f5f9c009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9a12f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33466/0 (socket says 172.21.15.3:33466) 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 -- 172.21.15.3:0/1725225583 learned_addr learned my addr 172.21.15.3:0/1725225583 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe99f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa6a2c640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe99fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 -- 172.21.15.3:0/1725225583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 msgr2=0x5643fe99fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe99fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 -- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 msgr2=0x5643fe99f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe99f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.340+0000 7f5fa622b640 1 -- 172.21.15.3:0/1725225583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5643fe8baff0 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fa622b640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9a12f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c004b50 tx=0x7f5f9c028300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f5a11da872e573f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5f89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5f89ffb640 1 -- 172.21.15.3:0/1725225583 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c01e030 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5f89ffb640 1 -- 172.21.15.3:0/1725225583 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5f9c016070 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5f9c0035f0 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5f89ffb640 1 -- 172.21.15.3:0/1725225583 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5f9c020050 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 10 client.17376 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5643fe8b9850 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5f89ffb640 1 -- 172.21.15.3:0/1725225583 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c0035f0 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5f89ffb640 1 -- 172.21.15.3:0/1725225583 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5f9c02f070 con 0x5643fe98b050 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 10 client.17376 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 10 client.17376 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 20 client.17376 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:30.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe999eb0 0x5643fea03ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 4 client.17376 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.341+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5643fe684890 con 0x5643fe999eb0 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe999eb0 0x5643fea03ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5f89ffb640 10 client.17376 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fa5a2a640 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe999eb0 0x5643fea03ce0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efc35b04460f64e0 server_cookie=78f91f5b7310a748 in_seq=0 out_seq=0 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5f89ffb640 1 -- 172.21.15.3:0/1725225583 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5643fe684890 con 0x5643fe999eb0 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5f89ffb640 10 client.17376 handle_command_reply: tid=0 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 client.17376 shutdown 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 msgr2=0x5643fe9a12f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9a12f0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c004b50 tx=0x7f5f9c028300 comp rx=0 tx=0).stop 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 shutdown_connections 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5643fe999eb0 0x5643fea03ce0 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5643fe990980 0x5643fe99f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5643fe991330 0x5643fe99fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5f89ffb640 0 client.17376 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 --2- 172.21.15.3:0/1725225583 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5643fe98b050 0x5643fe9a12f0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 >> 172.21.15.3:0/1725225583 conn(0x5643fe991a80 msgr2=0x5643fe9952c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 shutdown_connections 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.342+0000 7f5fab06e740 1 -- 172.21.15.3:0/1725225583 wait complete. 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fab06e740 20 client.17376 trim_cache size 0 max 16384 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:56:30.352 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:56:30.353 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:56:30.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f8c0a2260 msgr2=0x7f5f8c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f8c0a2260 0x7f5f8c0a4720 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5f90036040 tx=0x7f5f900047c0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 msgr2=0x7f5fa01a5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 0x7f5fa01a5a70 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f5f9800eb10 tx=0x7f5f9800c2d0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 shutdown_connections 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f8c0a2260 0x7f5f8c0a4720 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fa0106500 0x7f5fa0073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fa0073fb0 0x7f5fa01a5a70 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 --2- 172.21.15.3:0/1108692753 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fa0101430 0x7f5fa0073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 >> 172.21.15.3:0/1108692753 conn(0x7f5fa00fccd0 msgr2=0x7f5fa00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 shutdown_connections 2024-02-09T22:56:30.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.343+0000 7f5fa848f640 1 -- 172.21.15.3:0/1108692753 wait complete. 2024-02-09T22:56:30.357 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:56:30.357 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 -- 172.21.15.3:0/1561705524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7624102310 msgr2=0x7f76241026f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 --2- 172.21.15.3:0/1561705524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7624102310 0x7f76241026f0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f7614009900 tx=0x7f761402bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 -- 172.21.15.3:0/1561705524 shutdown_connections 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 --2- 172.21.15.3:0/1561705524 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7624100520 0x7f7624110230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 --2- 172.21.15.3:0/1561705524 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f76240ffaa0 0x7f76240fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 --2- 172.21.15.3:0/1561705524 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7624102310 0x7f76241026f0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 -- 172.21.15.3:0/1561705524 >> 172.21.15.3:0/1561705524 conn(0x7f76240765f0 msgr2=0x7f7624076a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 -- 172.21.15.3:0/1561705524 shutdown_connections 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.457+0000 7f762a943640 1 -- 172.21.15.3:0/1561705524 wait complete. 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 Processor -- start 2024-02-09T22:56:30.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 -- start start 2024-02-09T22:56:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 0x7f762419f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7624100520 0x7f762419fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7624102310 0x7f7624199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762419a0d0 con 0x7f76240ffaa0 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762419a240 con 0x7f7624100520 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f762a943640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762419a400 con 0x7f7624102310 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f7623fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 0x7f762419f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f7623fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 0x7f762419f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33482/0 (socket says 172.21.15.3:33482) 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f7623fff640 1 -- 172.21.15.3:0/2366851820 learned_addr learned my addr 172.21.15.3:0/2366851820 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.458+0000 7f76237fe640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7624100520 0x7f762419fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7628ee0640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7624102310 0x7f7624199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7623fff640 1 -- 172.21.15.3:0/2366851820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7624102310 msgr2=0x7f7624199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7623fff640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7624102310 0x7f7624199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7623fff640 1 -- 172.21.15.3:0/2366851820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7624100520 msgr2=0x7f762419fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7623fff640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7624100520 0x7f762419fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7623fff640 1 -- 172.21.15.3:0/2366851820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7614009560 con 0x7f76240ffaa0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f7623fff640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 0x7f762419f8d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7614005940 tx=0x7f7614004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614004440 con 0x7f76240ffaa0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76140052d0 con 0x7f76240ffaa0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762419a680 con 0x7f76240ffaa0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7614003da0 con 0x7f76240ffaa0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.459+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76241a6970 con 0x7f76240ffaa0 2024-02-09T22:56:30.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.460+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7624069600 con 0x7f76240ffaa0 2024-02-09T22:56:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.461+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7614005440 con 0x7f76240ffaa0 2024-02-09T22:56:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.462+0000 7f76217fa640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76100a2230 0x7f76100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.462+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f76140f01f0 con 0x7f76240ffaa0 2024-02-09T22:56:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.462+0000 7f76237fe640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76100a2230 0x7f76100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.462+0000 7f76237fe640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76100a2230 0x7f76100a46f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f76240ff5e0 tx=0x7f760c00b390 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.469 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.466+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f76140b8fc0 con 0x7f76240ffaa0 2024-02-09T22:56:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:30 smithi089 ceph-mon[36665]: pgmap v160: 97 pgs: 97 active+clean; 58 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 556 KiB/s wr, 79 op/s 2024-02-09T22:56:30.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:30 smithi003 ceph-mon[32678]: pgmap v160: 97 pgs: 97 active+clean; 58 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 556 KiB/s wr, 79 op/s 2024-02-09T22:56:30.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149791 2.66 MB/sec execute 144 sec latency 2548.191 ms 2024-02-09T22:56:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:30 smithi064 ceph-mon[36638]: pgmap v160: 97 pgs: 97 active+clean; 58 MiB data, 880 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 556 KiB/s wr, 79 op/s 2024-02-09T22:56:30.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.684+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f762419afb0 con 0x7f76240ffaa0 2024-02-09T22:56:30.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.685+0000 7f76217fa640 1 -- 172.21.15.3:0/2366851820 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f76140bc680 con 0x7f76240ffaa0 2024-02-09T22:56:30.688 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:56:30.688 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":36,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:55:02.762674+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:56:30.688 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 36 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76100a2230 msgr2=0x7f76100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76100a2230 0x7f76100a46f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f76240ff5e0 tx=0x7f760c00b390 comp rx=0 tx=0).stop 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 msgr2=0x7f762419f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 0x7f762419f8d0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f7614005940 tx=0x7f7614004040 comp rx=0 tx=0).stop 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 shutdown_connections 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76100a2230 0x7f76100a46f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7624100520 0x7f762419fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7624102310 0x7f7624199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 --2- 172.21.15.3:0/2366851820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f76240ffaa0 0x7f762419f8d0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 >> 172.21.15.3:0/2366851820 conn(0x7f76240765f0 msgr2=0x7f76240fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 shutdown_connections 2024-02-09T22:56:30.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.687+0000 7f762a943640 1 -- 172.21.15.3:0/2366851820 wait complete. 2024-02-09T22:56:30.701 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:56:30.701 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:56:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 -- 172.21.15.3:0/1700995597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd4100100 msgr2=0x7efdd410e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 --2- 172.21.15.3:0/1700995597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd4100100 0x7efdd410e1b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7efdcc009ca0 tx=0x7efdcc02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 -- 172.21.15.3:0/1700995597 shutdown_connections 2024-02-09T22:56:30.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 --2- 172.21.15.3:0/1700995597 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdd40ff740 0x7efdd40ffbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 --2- 172.21.15.3:0/1700995597 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd4100100 0x7efdd410e1b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 --2- 172.21.15.3:0/1700995597 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd41031b0 0x7efdd4103590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 -- 172.21.15.3:0/1700995597 >> 172.21.15.3:0/1700995597 conn(0x7efdd4076620 msgr2=0x7efdd40fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 -- 172.21.15.3:0/1700995597 shutdown_connections 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.803+0000 7efddb064640 1 -- 172.21.15.3:0/1700995597 wait complete. 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 Processor -- start 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 -- start start 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 0x7efdd419d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4100100 0x7efdd419dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdd41031b0 0x7efdd4197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4197df0 con 0x7efdd40ff740 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4197f60 con 0x7efdd41031b0 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efddb064640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdd4198120 con 0x7efdd4100100 2024-02-09T22:56:30.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 0x7efdd419d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 0x7efdd419d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33490/0 (socket says 172.21.15.3:33490) 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 -- 172.21.15.3:0/2841271185 learned_addr learned my addr 172.21.15.3:0/2841271185 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd9601640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdd41031b0 0x7efdd4197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4100100 0x7efdd419dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 -- 172.21.15.3:0/2841271185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4100100 msgr2=0x7efdd419dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4100100 0x7efdd419dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 -- 172.21.15.3:0/2841271185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdd41031b0 msgr2=0x7efdd4197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdd41031b0 0x7efdd4197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 -- 172.21.15.3:0/2841271185 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdcc009900 con 0x7efdd40ff740 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.804+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 0x7efdd419d5a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7efdc800d950 tx=0x7efdc800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.805+0000 7efdd1ffb640 1 -- 172.21.15.3:0/2841271185 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8009a30 con 0x7efdd40ff740 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.805+0000 7efdd1ffb640 1 -- 172.21.15.3:0/2841271185 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc8009b90 con 0x7efdd40ff740 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.805+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdd4198400 con 0x7efdd40ff740 2024-02-09T22:56:30.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.805+0000 7efdd1ffb640 1 -- 172.21.15.3:0/2841271185 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc8004a00 con 0x7efdd40ff740 2024-02-09T22:56:30.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.805+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdd41a4520 con 0x7efdd40ff740 2024-02-09T22:56:30.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.807+0000 7efdd1ffb640 1 -- 172.21.15.3:0/2841271185 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efdc8009d00 con 0x7efdd40ff740 2024-02-09T22:56:30.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.807+0000 7efdddc3f740 1 Processor -- start 2024-02-09T22:56:30.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.807+0000 7efdddc3f740 1 -- start start 2024-02-09T22:56:30.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.807+0000 7efdd1ffb640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdc40a2230 0x7efdc40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a80e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a89990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdddc3f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4a3c020 con 0x55e3f4a813e0 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdddc3f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f49ef320 con 0x55e3f4a89f60 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdddc3f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f49ec850 con 0x55e3f4a80a30 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd1ffb640 1 -- 172.21.15.3:0/2841271185 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efdc80cdad0 con 0x7efdd40ff740 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd9601640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdc40a2230 0x7efdc40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a89990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a89990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33506/0 (socket says 172.21.15.3:33506) 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 -- 172.21.15.3:0/18378474 learned_addr learned my addr 172.21.15.3:0/18378474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd8e00640 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a80e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 -- 172.21.15.3:0/18378474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 msgr2=0x55e3f4a80e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a80e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 -- 172.21.15.3:0/18378474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 msgr2=0x55e3f4a8c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 -- 172.21.15.3:0/18378474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3f49aa210 con 0x55e3f4a813e0 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd9601640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdc40a2230 0x7efdc40a46f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efdcc004af0 tx=0x7efdcc032ee0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdd3fff640 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a89990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7efdc000bc80 tx=0x7efdc000f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70966563015cb359 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdbeffd640 1 -- 172.21.15.3:0/18378474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc000fee0 con 0x55e3f4a813e0 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdbeffd640 1 -- 172.21.15.3:0/18378474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc001a940 con 0x55e3f4a813e0 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.808+0000 7efdbeffd640 1 -- 172.21.15.3:0/18378474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc00119b0 con 0x55e3f4a813e0 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 -- 172.21.15.3:0/18378474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 msgr2=0x55e3f4a89990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a89990 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7efdc000bc80 tx=0x7efdc000f730 comp rx=0 tx=0).stop 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 -- 172.21.15.3:0/18378474 shutdown_connections 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a89990 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 --2- 172.21.15.3:0/18378474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a80e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 -- 172.21.15.3:0/18378474 >> 172.21.15.3:0/18378474 conn(0x55e3f4a7b100 msgr2=0x55e3f4a7b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 -- 172.21.15.3:0/18378474 shutdown_connections 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 -- 172.21.15.3:0/18378474 wait complete. 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 -- start start 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.809+0000 7efdddc3f740 1 Processor -- start 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a80a30 0x55e3f4a86f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a87440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a89f60 0x55e3f4a87980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f49e8030 con 0x55e3f4a813e0 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f46bb430 con 0x55e3f4a80a30 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4677e00 con 0x55e3f4a89f60 2024-02-09T22:56:30.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdddc3f740 10 client.0 _ll_register_callbacks cb 0x55e3f49dd030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a87440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a87440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33512/0 (socket says 172.21.15.3:33512) 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 -- 172.21.15.3:0/2126430656 learned_addr learned my addr 172.21.15.3:0/2126430656 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a80a30 0x55e3f4a86f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd9601640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a89f60 0x55e3f4a87980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 -- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a89f60 msgr2=0x55e3f4a87980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a89f60 0x55e3f4a87980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 -- 172.21.15.3:0/2126430656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a80a30 msgr2=0x55e3f4a86f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a80a30 0x55e3f4a86f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 -- 172.21.15.3:0/2126430656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdc000b8e0 con 0x55e3f4a813e0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a87440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efdc0019000 tx=0x7efdc000bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18032e30918e85cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.810+0000 7efdb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:30.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdb3fff640 1 -- 172.21.15.3:0/2126430656 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0020070 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdb3fff640 1 -- 172.21.15.3:0/2126430656 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efdc001a6f0 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3f46e77e0 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdb3fff640 1 -- 172.21.15.3:0/2126430656 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdc001aab0 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 10 client.17394 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55e3f49e66b0 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdb3fff640 1 -- 172.21.15.3:0/2126430656 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc0032020 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdb3fff640 1 -- 172.21.15.3:0/2126430656 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdc0011a10 con 0x55e3f4a813e0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 10 client.17394 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 10 client.17394 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 20 client.17394 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4a9a3b0 0x55e3f4a9c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 4 client.17394 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3f4724720 con 0x55e3f4a9a3b0 2024-02-09T22:56:30.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.811+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4a9a3b0 0x55e3f4a9c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.812+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4a9a3b0 0x55e3f4a9c7d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5f505c71acbdc11 server_cookie=623c19c926954b0 in_seq=0 out_seq=0 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.812+0000 7efdb3fff640 10 client.17394 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.812+0000 7efdb3fff640 1 -- 172.21.15.3:0/2126430656 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e3f4724720 con 0x55e3f4a9a3b0 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.812+0000 7efdb3fff640 10 client.17394 handle_command_reply: tid=0 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 client.17394 shutdown 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 msgr2=0x55e3f4a87440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a87440 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7efdc0019000 tx=0x7efdc000bfa0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 shutdown_connections 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4a9a3b0 0x55e3f4a9c7d0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a89f60 0x55e3f4a87980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdb3fff640 0 client.17394 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a87440 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2126430656 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a80a30 0x55e3f4a86f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 >> 172.21.15.3:0/2126430656 conn(0x55e3f4a7b100 msgr2=0x55e3f4a7ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 shutdown_connections 2024-02-09T22:56:30.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 1 -- 172.21.15.3:0/2126430656 wait complete. 2024-02-09T22:56:30.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.813+0000 7efdddc3f740 20 client.17394 trim_cache size 0 max 16384 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.824+0000 7efdddc3f740 1 Processor -- start 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.824+0000 7efdddc3f740 1 -- start start 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a89630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a877a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a857d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdddc3f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f496d130 con 0x55e3f4a813e0 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdddc3f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4aa1a00 con 0x55e3f4a89f60 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdddc3f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4a7fd50 con 0x55e3f4a80a30 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a877a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a877a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33516/0 (socket says 172.21.15.3:33516) 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 -- 172.21.15.3:0/1327565879 learned_addr learned my addr 172.21.15.3:0/1327565879 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd9601640 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a857d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd8e00640 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a89630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 -- 172.21.15.3:0/1327565879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 msgr2=0x55e3f4a89630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a89630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 -- 172.21.15.3:0/1327565879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 msgr2=0x55e3f4a857d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a857d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 -- 172.21.15.3:0/1327565879 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3f4982c90 con 0x55e3f4a813e0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd8e00640 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a89630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdd3fff640 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a877a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efdc0031040 tx=0x7efdc00059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae39cb8e6fe18298 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdbe7fc640 1 -- 172.21.15.3:0/1327565879 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0020070 con 0x55e3f4a813e0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdbe7fc640 1 -- 172.21.15.3:0/1327565879 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc0032030 con 0x55e3f4a813e0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.825+0000 7efdbe7fc640 1 -- 172.21.15.3:0/1327565879 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0011a50 con 0x55e3f4a813e0 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 -- 172.21.15.3:0/1327565879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 msgr2=0x55e3f4a877a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a877a0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7efdc0031040 tx=0x7efdc00059b0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 -- 172.21.15.3:0/1327565879 shutdown_connections 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a857d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a877a0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 --2- 172.21.15.3:0/1327565879 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a89630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 -- 172.21.15.3:0/1327565879 >> 172.21.15.3:0/1327565879 conn(0x55e3f4a7b100 msgr2=0x55e3f4a82210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 -- 172.21.15.3:0/1327565879 shutdown_connections 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 -- 172.21.15.3:0/1327565879 wait complete. 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 -- start start 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.826+0000 7efdddc3f740 1 Processor -- start 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a90fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a8f380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4a918b0 con 0x55e3f4a813e0 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4a8fe00 con 0x55e3f4a89f60 2024-02-09T22:56:30.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3f4a8ff90 con 0x55e3f4a80a30 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 10 client.0 _ll_register_callbacks cb 0x7efdc000b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a8f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a8f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33526/0 (socket says 172.21.15.3:33526) 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 -- 172.21.15.3:0/2011074633 learned_addr learned my addr 172.21.15.3:0/2011074633 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a90fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd9601640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 -- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 msgr2=0x55e3f4a90fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a90fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 -- 172.21.15.3:0/2011074633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 msgr2=0x55e3f4a8f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 -- 172.21.15.3:0/2011074633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3f49e66b0 con 0x55e3f4a813e0 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a90fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:30.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd9601640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdd3fff640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a8f380 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efdc000fc30 tx=0x7efdc000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=616c47be92087be server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdb3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdb3fff640 1 -- 172.21.15.3:0/2011074633 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdc0020070 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdb3fff640 1 -- 172.21.15.3:0/2011074633 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efdc0033080 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdc001a940 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdb3fff640 1 -- 172.21.15.3:0/2011074633 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdc0011440 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 10 client.17400 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.827+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55e3f49a8a70 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdb3fff640 1 -- 172.21.15.3:0/2011074633 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdc001a940 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdb3fff640 1 -- 172.21.15.3:0/2011074633 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efdc002c040 con 0x55e3f4a813e0 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdddc3f740 10 client.17400 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdddc3f740 10 client.17400 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:56:30.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdddc3f740 20 client.17400 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4af1820 0x55e3f4af3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdddc3f740 4 client.17400 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e3f4773890 con 0x55e3f4af1820 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.828+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4af1820 0x55e3f4af3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdd8e00640 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4af1820 0x55e3f4af3c40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=66fecde22903db38 server_cookie=6433ab3b6791a76 in_seq=0 out_seq=0 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdb3fff640 10 client.17400 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdb3fff640 1 -- 172.21.15.3:0/2011074633 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e3f4773890 con 0x55e3f4af1820 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdb3fff640 10 client.17400 handle_command_reply: tid=0 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 client.17400 shutdown 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 msgr2=0x55e3f4a8f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a8f380 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7efdc000fc30 tx=0x7efdc000b840 comp rx=0 tx=0).stop 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 shutdown_connections 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55e3f4af1820 0x55e3f4af3c40 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3f4a89f60 0x55e3f4a8f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3f4a813e0 0x55e3f4a8f380 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdb3fff640 0 client.17400 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 --2- 172.21.15.3:0/2011074633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3f4a80a30 0x55e3f4a90fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 >> 172.21.15.3:0/2011074633 conn(0x55e3f4a7b100 msgr2=0x55e3f4a85790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 shutdown_connections 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 1 -- 172.21.15.3:0/2011074633 wait complete. 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.829+0000 7efdddc3f740 20 client.17400 trim_cache size 0 max 16384 2024-02-09T22:56:30.839 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:56:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdc40a2230 msgr2=0x7efdc40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdc40a2230 0x7efdc40a46f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7efdcc004af0 tx=0x7efdcc032ee0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 msgr2=0x7efdd419d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 0x7efdd419d5a0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7efdc800d950 tx=0x7efdc800de10 comp rx=0 tx=0).stop 2024-02-09T22:56:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 shutdown_connections 2024-02-09T22:56:30.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdc40a2230 0x7efdc40a46f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efdd41031b0 0x7efdd4197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efdd4100100 0x7efdd419dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 --2- 172.21.15.3:0/2841271185 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efdd40ff740 0x7efdd419d5a0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 >> 172.21.15.3:0/2841271185 conn(0x7efdd4076620 msgr2=0x7efdd40fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 shutdown_connections 2024-02-09T22:56:30.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.830+0000 7efddb064640 1 -- 172.21.15.3:0/2841271185 wait complete. 2024-02-09T22:56:30.844 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:56:30.844 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:56:30.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 -- 172.21.15.3:0/1924891244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 msgr2=0x7fae2010cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 --2- 172.21.15.3:0/1924891244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae2010cfd0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fae18009cd0 tx=0x7fae1802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 -- 172.21.15.3:0/1924891244 shutdown_connections 2024-02-09T22:56:30.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 --2- 172.21.15.3:0/1924891244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae20101470 0x7fae20104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 --2- 172.21.15.3:0/1924891244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae2010cfd0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 --2- 172.21.15.3:0/1924891244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae20100ac0 0x7fae20100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 -- 172.21.15.3:0/1924891244 >> 172.21.15.3:0/1924891244 conn(0x7fae20076630 msgr2=0x7fae200fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 -- 172.21.15.3:0/1924891244 shutdown_connections 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.949+0000 7fae25ed4640 1 -- 172.21.15.3:0/1924891244 wait complete. 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 Processor -- start 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 -- start start 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae20100ac0 0x7fae201986b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae20198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae20199130 0x7fae201a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae201a3da0 con 0x7fae20105400 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae201a3f10 con 0x7fae20100ac0 2024-02-09T22:56:30.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae25ed4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae201a40d0 con 0x7fae20199130 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae20198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae20198bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33542/0 (socket says 172.21.15.3:33542) 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae1effd640 1 -- 172.21.15.3:0/218630323 learned_addr learned my addr 172.21.15.3:0/218630323 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.950+0000 7fae1ffff640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae20199130 0x7fae201a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1effd640 1 -- 172.21.15.3:0/218630323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae20199130 msgr2=0x7fae201a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1effd640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae20199130 0x7fae201a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1effd640 1 -- 172.21.15.3:0/218630323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae20100ac0 msgr2=0x7fae201986b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1effd640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae20100ac0 0x7fae201986b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1effd640 1 -- 172.21.15.3:0/218630323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae18009900 con 0x7fae20105400 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1effd640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae20198bf0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fae1400d770 tx=0x7fae1400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1cff9640 1 -- 172.21.15.3:0/218630323 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14009c30 con 0x7fae20105400 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae201a43b0 con 0x7fae20105400 2024-02-09T22:56:30.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1cff9640 1 -- 172.21.15.3:0/218630323 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae1400de80 con 0x7fae20105400 2024-02-09T22:56:30.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae1cff9640 1 -- 172.21.15.3:0/218630323 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400f6b0 con 0x7fae20105400 2024-02-09T22:56:30.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.951+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae201a48c0 con 0x7fae20105400 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.953+0000 7fae1cff9640 1 -- 172.21.15.3:0/218630323 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fae14010460 con 0x7fae20105400 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.953+0000 7fae28b84740 1 Processor -- start 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 -- start start 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1cff9640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae080a2260 0x7fae080a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859ade10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d85969020 con 0x559d859ae3e0 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8591c320 con 0x559d859b6f60 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae28b84740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d85919850 con 0x559d859ada30 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1cff9640 1 -- 172.21.15.3:0/218630323 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fae140cdd70 con 0x7fae20105400 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1ffff640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae080a2260 0x7fae080a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b6990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33554/0 (socket says 172.21.15.3:33554) 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 -- 172.21.15.3:0/2852240385 learned_addr learned my addr 172.21.15.3:0/2852240385 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 -- 172.21.15.3:0/2852240385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 msgr2=0x559d859ade10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859ade10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1ffff640 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 -- 172.21.15.3:0/2852240385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 msgr2=0x559d859b93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 -- 172.21.15.3:0/2852240385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d858d7210 con 0x559d859ae3e0 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1ffff640 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1ffff640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae080a2260 0x7fae080a4720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fae18005cc0 tx=0x7fae18005f10 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.954+0000 7fae1effd640 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b6990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fae14097160 tx=0x7fae1409ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e66310634a1f738 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae05ffb640 1 -- 172.21.15.3:0/2852240385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1409e8e0 con 0x559d859ae3e0 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae05ffb640 1 -- 172.21.15.3:0/2852240385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae140a2f30 con 0x559d859ae3e0 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae05ffb640 1 -- 172.21.15.3:0/2852240385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae140ba0b0 con 0x559d859ae3e0 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 -- 172.21.15.3:0/2852240385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 msgr2=0x559d859b6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b6990 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fae14097160 tx=0x7fae1409ae90 comp rx=0 tx=0).stop 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 -- 172.21.15.3:0/2852240385 shutdown_connections 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b6990 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 --2- 172.21.15.3:0/2852240385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859ade10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 -- 172.21.15.3:0/2852240385 >> 172.21.15.3:0/2852240385 conn(0x559d859a8100 msgr2=0x559d859a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 -- 172.21.15.3:0/2852240385 shutdown_connections 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.955+0000 7fae28b84740 1 -- 172.21.15.3:0/2852240385 wait complete. 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 -- start start 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 Processor -- start 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859b4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b49a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d85915030 con 0x559d859ada30 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d855e8430 con 0x559d859b6f60 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d855a4e00 con 0x559d859ae3e0 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae28b84740 10 client.0 _ll_register_callbacks cb 0x559d8590a030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33562/0 (socket says 172.21.15.3:33562) 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 -- 172.21.15.3:0/1137215860 learned_addr learned my addr 172.21.15.3:0/1137215860 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1effd640 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859b4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 -- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 msgr2=0x559d859b4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859b4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 -- 172.21.15.3:0/1137215860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 msgr2=0x559d859b49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.956+0000 7fae1f7fe640 1 -- 172.21.15.3:0/1137215860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae201a48c0 con 0x559d859ada30 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859b3f20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fae10011720 tx=0x7fae10011be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce41ca4749ad11d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fadfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fadfeffd640 1 -- 172.21.15.3:0/1137215860 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000fd30 con 0x559d859ada30 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d856147e0 con 0x559d859ada30 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fadfeffd640 1 -- 172.21.15.3:0/1137215860 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fae10019430 con 0x559d859ada30 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fadfeffd640 1 -- 172.21.15.3:0/1137215860 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae1000fe90 con 0x559d859ada30 2024-02-09T22:56:30.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 10 client.17412 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x559d859136b0 con 0x559d859ada30 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fadfeffd640 1 -- 172.21.15.3:0/1137215860 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559d856147e0 con 0x559d859ada30 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fadfeffd640 1 -- 172.21.15.3:0/1137215860 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fae100215b0 con 0x559d859ada30 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 10 client.17412 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 10 client.17412 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 20 client.17412 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d859c74b0 0x559d859c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 4 client.17412 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.957+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d85651720 con 0x559d859c74b0 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.958+0000 7fae1effd640 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d859c74b0 0x559d859c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.958+0000 7fae1effd640 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d859c74b0 0x559d859c98d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ce0fd623ab2b749e server_cookie=c4c73c75558ab327 in_seq=0 out_seq=0 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.958+0000 7fadfeffd640 10 client.17412 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fadfeffd640 1 -- 172.21.15.3:0/1137215860 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d85651720 con 0x559d859c74b0 2024-02-09T22:56:30.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fadfeffd640 10 client.17412 handle_command_reply: tid=0 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 client.17412 shutdown 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 msgr2=0x559d859b3f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859b3f20 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7fae10011720 tx=0x7fae10011be0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 shutdown_connections 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d859c74b0 0x559d859c98d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859b49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859b4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fadfeffd640 0 client.17412 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 --2- 172.21.15.3:0/1137215860 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859b3f20 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 >> 172.21.15.3:0/1137215860 conn(0x559d859a8100 msgr2=0x559d859a9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 shutdown_connections 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 1 -- 172.21.15.3:0/1137215860 wait complete. 2024-02-09T22:56:30.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.959+0000 7fae28b84740 20 client.17412 trim_cache size 0 max 16384 2024-02-09T22:56:30.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 Processor -- start 2024-02-09T22:56:30.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 -- start start 2024-02-09T22:56:30.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859b6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859af1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d8589a130 con 0x559d859ae3e0 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d859a3a40 con 0x559d859b6f60 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae28b84740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d859b2470 con 0x559d859ada30 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33578/0 (socket says 172.21.15.3:33578) 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 -- 172.21.15.3:0/3698240197 learned_addr learned my addr 172.21.15.3:0/3698240197 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859b6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1ffff640 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859af1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 -- 172.21.15.3:0/3698240197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 msgr2=0x559d859b6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859b6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 -- 172.21.15.3:0/3698240197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 msgr2=0x559d859af1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859af1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 -- 172.21.15.3:0/3698240197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d858afc90 con 0x559d859ae3e0 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.971+0000 7fae1effd640 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b47c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fae140cb5a0 tx=0x7fae140ba040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b5bc4fa9e552856 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae057fa640 1 -- 172.21.15.3:0/3698240197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14009d90 con 0x559d859ae3e0 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae057fa640 1 -- 172.21.15.3:0/3698240197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae14098a90 con 0x559d859ae3e0 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae057fa640 1 -- 172.21.15.3:0/3698240197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae140ba0b0 con 0x559d859ae3e0 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 -- 172.21.15.3:0/3698240197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 msgr2=0x559d859b47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b47c0 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7fae140cb5a0 tx=0x7fae140ba040 comp rx=0 tx=0).stop 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 -- 172.21.15.3:0/3698240197 shutdown_connections 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859af1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ae3e0 0x559d859b47c0 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 --2- 172.21.15.3:0/3698240197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ada30 0x559d859b6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 -- 172.21.15.3:0/3698240197 >> 172.21.15.3:0/3698240197 conn(0x559d859a8100 msgr2=0x559d859a84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 -- 172.21.15.3:0/3698240197 shutdown_connections 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 -- 172.21.15.3:0/3698240197 wait complete. 2024-02-09T22:56:30.981 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.972+0000 7fae28b84740 1 -- start start 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 Processor -- start 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859bc8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859bcdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859bd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d859be7b0 con 0x559d859ada30 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d85a1da00 con 0x559d859b6f60 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d85a1db40 con 0x559d859ae3e0 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae28b84740 10 client.0 _ll_register_callbacks cb 0x7fae14096a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859bc8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859bc8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33594/0 (socket says 172.21.15.3:33594) 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 -- 172.21.15.3:0/2833756938 learned_addr learned my addr 172.21.15.3:0/2833756938 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1effd640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859bcdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1ffff640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 -- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 msgr2=0x559d859bcdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859bcdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 -- 172.21.15.3:0/2833756938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 msgr2=0x559d859bd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859bd330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1f7fe640 1 -- 172.21.15.3:0/2833756938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d858d7210 con 0x559d859ada30 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.973+0000 7fae1ffff640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859bd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae1f7fe640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859bc8b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fae10011720 tx=0x7fae100092f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423484d40329327 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fadfeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fadfeffd640 1 -- 172.21.15.3:0/2833756938 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10021380 con 0x559d859ada30 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fadfeffd640 1 -- 172.21.15.3:0/2833756938 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fae10000e40 con 0x559d859ada30 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fae140a2f30 con 0x559d859ada30 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fadfeffd640 1 -- 172.21.15.3:0/2833756938 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae100214e0 con 0x559d859ada30 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 10 client.17418 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x559d858d5a70 con 0x559d859ada30 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fadfeffd640 1 -- 172.21.15.3:0/2833756938 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fae140a2f30 con 0x559d859ada30 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fadfeffd640 1 -- 172.21.15.3:0/2833756938 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fae10019430 con 0x559d859ada30 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 10 client.17418 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 10 client.17418 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 20 client.17418 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d85a1f350 0x559d85a21770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 4 client.17418 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d856a0890 con 0x559d85a1f350 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.974+0000 7fae1effd640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d85a1f350 0x559d85a21770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae1effd640 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d85a1f350 0x559d85a21770 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=386573483752299b server_cookie=f96a0750da686263 in_seq=0 out_seq=0 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fadfeffd640 10 client.17418 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fadfeffd640 1 -- 172.21.15.3:0/2833756938 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d856a0890 con 0x559d85a1f350 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fadfeffd640 10 client.17418 handle_command_reply: tid=0 2024-02-09T22:56:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 client.17418 shutdown 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 msgr2=0x559d859bc8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859bc8b0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7fae10011720 tx=0x7fae100092f0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 shutdown_connections 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d85a1f350 0x559d85a21770 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d859b6f60 0x559d859bd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d859ae3e0 0x559d859bcdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 --2- 172.21.15.3:0/2833756938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d859ada30 0x559d859bc8b0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fadfeffd640 0 client.17418 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 >> 172.21.15.3:0/2833756938 conn(0x559d859a8100 msgr2=0x559d859af1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 shutdown_connections 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 1 -- 172.21.15.3:0/2833756938 wait complete. 2024-02-09T22:56:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.975+0000 7fae28b84740 20 client.17418 trim_cache size 0 max 16384 2024-02-09T22:56:30.986 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:56:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae080a2260 msgr2=0x7fae080a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae080a2260 0x7fae080a4720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fae18005cc0 tx=0x7fae18005f10 comp rx=0 tx=0).stop 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 msgr2=0x7fae20198bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae20198bf0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fae1400d770 tx=0x7fae1400dc30 comp rx=0 tx=0).stop 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 shutdown_connections 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fae080a2260 0x7fae080a4720 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fae20199130 0x7fae201a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fae20105400 0x7fae20198bf0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 --2- 172.21.15.3:0/218630323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fae20100ac0 0x7fae201986b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 >> 172.21.15.3:0/218630323 conn(0x7fae20076630 msgr2=0x7fae20103570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 shutdown_connections 2024-02-09T22:56:30.987 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:30.976+0000 7fae25ed4640 1 -- 172.21.15.3:0/218630323 wait complete. 2024-02-09T22:56:30.989 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:56:30.989 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:56:31.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 -- 172.21.15.3:0/2786936614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c101d20 msgr2=0x7fa46c102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 --2- 172.21.15.3:0/2786936614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c101d20 0x7fa46c102180 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7fa45c009930 tx=0x7fa45c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 -- 172.21.15.3:0/2786936614 shutdown_connections 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 --2- 172.21.15.3:0/2786936614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa46c1026c0 0x7fa46c10d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 --2- 172.21.15.3:0/2786936614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c101d20 0x7fa46c102180 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 --2- 172.21.15.3:0/2786936614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa46c100440 0x7fa46c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 -- 172.21.15.3:0/2786936614 >> 172.21.15.3:0/2786936614 conn(0x7fa46c077310 msgr2=0x7fa46c077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 -- 172.21.15.3:0/2786936614 shutdown_connections 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.092+0000 7fa473e23640 1 -- 172.21.15.3:0/2786936614 wait complete. 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 Processor -- start 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 -- start start 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 0x7fa46c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa46c101d20 0x7fa46c19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa46c1026c0 0x7fa46c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c197ea0 con 0x7fa46c100440 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c198010 con 0x7fa46c101d20 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa473e23640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c1981d0 con 0x7fa46c1026c0 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 0x7fa46c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 0x7fa46c19d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33604/0 (socket says 172.21.15.3:33604) 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 -- 172.21.15.3:0/1390268690 learned_addr learned my addr 172.21.15.3:0/1390268690 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa4723c0640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa46c1026c0 0x7fa46c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa4713be640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa46c101d20 0x7fa46c19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 -- 172.21.15.3:0/1390268690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa46c1026c0 msgr2=0x7fa46c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa46c1026c0 0x7fa46c197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 -- 172.21.15.3:0/1390268690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa46c101d20 msgr2=0x7fa46c19db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa46c101d20 0x7fa46c19db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 -- 172.21.15.3:0/1390268690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c009560 con 0x7fa46c100440 2024-02-09T22:56:31.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.093+0000 7fa471bbf640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 0x7fa46c19d650 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c890 tx=0x7fa46800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.094+0000 7fa462ffd640 1 -- 172.21.15.3:0/1390268690 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468004220 con 0x7fa46c100440 2024-02-09T22:56:31.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.094+0000 7fa462ffd640 1 -- 172.21.15.3:0/1390268690 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa468007e90 con 0x7fa46c100440 2024-02-09T22:56:31.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.094+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c1984b0 con 0x7fa46c100440 2024-02-09T22:56:31.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.094+0000 7fa462ffd640 1 -- 172.21.15.3:0/1390268690 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa468004a00 con 0x7fa46c100440 2024-02-09T22:56:31.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.094+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c1a45d0 con 0x7fa46c100440 2024-02-09T22:56:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.096+0000 7fa462ffd640 1 -- 172.21.15.3:0/1390268690 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa4680079b0 con 0x7fa46c100440 2024-02-09T22:56:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.096+0000 7fa4769fc740 1 Processor -- start 2024-02-09T22:56:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.096+0000 7fa4769fc740 1 -- start start 2024-02-09T22:56:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb25e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb2e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb313b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4769fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560deae1020 con 0x5560deb263e0 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4769fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560dea94320 con 0x5560deb25a30 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4769fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560dea91850 con 0x5560deb2ef60 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa462ffd640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4580a2230 0x7fa4580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb2e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4723c0640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4580a2230 0x7fa4580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb2e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33606/0 (socket says 172.21.15.3:33606) 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 -- 172.21.15.3:0/1566986629 learned_addr learned my addr 172.21.15.3:0/1566986629 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4713be640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb313b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa462ffd640 1 -- 172.21.15.3:0/1390268690 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa4680cdce0 con 0x7fa46c100440 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4723c0640 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb25e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 -- 172.21.15.3:0/1566986629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 msgr2=0x5560deb313b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb313b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 -- 172.21.15.3:0/1566986629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 msgr2=0x5560deb25e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb25e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 -- 172.21.15.3:0/1566986629 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560dea4f210 con 0x5560deb263e0 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4723c0640 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb25e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4713be640 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa471bbf640 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb2e990 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa46809a4b0 tx=0x7fa46806c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6960a032e7ea1c14 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa4723c0640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4580a2230 0x7fa4580a46f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa464005fd0 tx=0x7fa464009b40 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.097+0000 7fa457fff640 1 -- 172.21.15.3:0/1566986629 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa46809a7b0 con 0x5560deb263e0 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa457fff640 1 -- 172.21.15.3:0/1566986629 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa46809e850 con 0x5560deb263e0 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa457fff640 1 -- 172.21.15.3:0/1566986629 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4680cb6e0 con 0x5560deb263e0 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 -- 172.21.15.3:0/1566986629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 msgr2=0x5560deb2e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb2e990 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fa46809a4b0 tx=0x7fa46806c770 comp rx=0 tx=0).stop 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 -- 172.21.15.3:0/1566986629 shutdown_connections 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb313b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.108 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb2e990 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 --2- 172.21.15.3:0/1566986629 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb25e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 -- 172.21.15.3:0/1566986629 >> 172.21.15.3:0/1566986629 conn(0x5560deb20100 msgr2=0x5560deb204e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 -- 172.21.15.3:0/1566986629 shutdown_connections 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.098+0000 7fa4769fc740 1 -- 172.21.15.3:0/1566986629 wait complete. 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 -- start start 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 Processor -- start 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb2bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb2c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb2c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560dea8d030 con 0x5560deb2ef60 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de760430 con 0x5560deb263e0 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560de71ce00 con 0x5560deb25a30 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4769fc740 10 client.0 _ll_register_callbacks cb 0x5560dea82030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:31.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4723c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb2c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4723c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb2c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33618/0 (socket says 172.21.15.3:33618) 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4723c0640 1 -- 172.21.15.3:0/2511789988 learned_addr learned my addr 172.21.15.3:0/2511789988 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.099+0000 7fa4713be640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb2c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa471bbf640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb2bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4723c0640 1 -- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 msgr2=0x5560deb2bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4723c0640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb2bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4723c0640 1 -- 172.21.15.3:0/2511789988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 msgr2=0x5560deb2c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4723c0640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb2c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4723c0640 1 -- 172.21.15.3:0/2511789988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c1a45d0 con 0x5560deb2ef60 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa471bbf640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb2bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4723c0640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb2c940 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa464003c20 tx=0x7fa464004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58cd8ed65b846f31 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4557fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4557fa640 1 -- 172.21.15.3:0/2511789988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464003e50 con 0x5560deb2ef60 2024-02-09T22:56:31.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4557fa640 1 -- 172.21.15.3:0/2511789988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa4640045b0 con 0x5560deb2ef60 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5560de78c7e0 con 0x5560deb2ef60 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4557fa640 1 -- 172.21.15.3:0/2511789988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa464004770 con 0x5560deb2ef60 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4769fc740 10 client.17430 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5560dea8b6b0 con 0x5560deb2ef60 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4557fa640 1 -- 172.21.15.3:0/2511789988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5560de78c7e0 con 0x5560deb2ef60 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.100+0000 7fa4557fa640 1 -- 172.21.15.3:0/2511789988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa464029060 con 0x5560deb2ef60 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4769fc740 10 client.17430 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4769fc740 10 client.17430 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4769fc740 20 client.17430 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4769fc740 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb3f490 0x5560deb418b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4769fc740 4 client.17430 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5560de7c9720 con 0x5560deb3f490 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa471bbf640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb3f490 0x5560deb418b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa4557fa640 10 client.17430 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.101+0000 7fa471bbf640 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb3f490 0x5560deb418b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e469edecbb62884b server_cookie=bb63cdb83d21ae3d in_seq=0 out_seq=0 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4557fa640 1 -- 172.21.15.3:0/2511789988 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5560de7c9720 con 0x5560deb3f490 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4557fa640 10 client.17430 handle_command_reply: tid=0 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 client.17430 shutdown 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 msgr2=0x5560deb2c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb2c940 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7fa464003c20 tx=0x7fa464004ab0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 shutdown_connections 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb3f490 0x5560deb418b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb2c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb2c940 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4557fa640 0 client.17430 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 --2- 172.21.15.3:0/2511789988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb2bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 >> 172.21.15.3:0/2511789988 conn(0x5560deb20100 msgr2=0x5560deb21e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 shutdown_connections 2024-02-09T22:56:31.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 1 -- 172.21.15.3:0/2511789988 wait complete. 2024-02-09T22:56:31.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.102+0000 7fa4769fc740 20 client.17430 trim_cache size 0 max 16384 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 Processor -- start 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 -- start start 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb2c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb1f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb35ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560dea12130 con 0x5560deb263e0 2024-02-09T22:56:31.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560deb31ad0 con 0x5560deb25a30 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4769fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560deb31810 con 0x5560deb2ef60 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4713be640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb1f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4713be640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb1f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33628/0 (socket says 172.21.15.3:33628) 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa4713be640 1 -- 172.21.15.3:0/3940458147 learned_addr learned my addr 172.21.15.3:0/3940458147 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.114+0000 7fa471bbf640 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb2c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4713be640 1 -- 172.21.15.3:0/3940458147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 msgr2=0x5560deb35ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4713be640 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb35ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4713be640 1 -- 172.21.15.3:0/3940458147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 msgr2=0x5560deb2c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4713be640 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb2c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4713be640 1 -- 172.21.15.3:0/3940458147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5560dea27c90 con 0x5560deb263e0 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4713be640 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb1f0c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fa45c02c780 tx=0x7fa45c004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69f868ba3f6728b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4577fe640 1 -- 172.21.15.3:0/3940458147 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c044070 con 0x5560deb263e0 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4577fe640 1 -- 172.21.15.3:0/3940458147 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa45c004ae0 con 0x5560deb263e0 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4577fe640 1 -- 172.21.15.3:0/3940458147 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45c004fe0 con 0x5560deb263e0 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 -- 172.21.15.3:0/3940458147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 msgr2=0x5560deb1f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb1f0c0 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7fa45c02c780 tx=0x7fa45c004930 comp rx=0 tx=0).stop 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 -- 172.21.15.3:0/3940458147 shutdown_connections 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb2ef60 0x5560deb35ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb263e0 0x5560deb1f0c0 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3940458147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb25a30 0x5560deb2c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.115+0000 7fa4769fc740 1 -- 172.21.15.3:0/3940458147 >> 172.21.15.3:0/3940458147 conn(0x5560deb20100 msgr2=0x5560deb1e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 -- 172.21.15.3:0/3940458147 shutdown_connections 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 -- 172.21.15.3:0/3940458147 wait complete. 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 -- start start 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 Processor -- start 2024-02-09T22:56:31.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb34a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb34f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb35490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560deb959e0 con 0x5560deb2ef60 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560deb95b20 con 0x5560deb263e0 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560deb95ce0 con 0x5560deb25a30 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4769fc740 10 client.0 _ll_register_callbacks cb 0x7fa4680963d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4723c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb35490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4723c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb35490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33632/0 (socket says 172.21.15.3:33632) 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4723c0640 1 -- 172.21.15.3:0/3683226574 learned_addr learned my addr 172.21.15.3:0/3683226574 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.116+0000 7fa4713be640 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb34f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4723c0640 1 -- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 msgr2=0x5560deb34a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4723c0640 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb34a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4723c0640 1 -- 172.21.15.3:0/3683226574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 msgr2=0x5560deb34f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4723c0640 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb34f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4723c0640 1 -- 172.21.15.3:0/3683226574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c030070 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4723c0640 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb35490 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa4640127c0 tx=0x7fa4640127f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c0aecc8d267bc9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4557fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4557fa640 1 -- 172.21.15.3:0/3683226574 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa464004840 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4557fa640 1 -- 172.21.15.3:0/3683226574 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa464021470 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa46809e850 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4557fa640 1 -- 172.21.15.3:0/3683226574 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa464003e90 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 10 client.17436 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5560dea4da70 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4557fa640 1 -- 172.21.15.3:0/3683226574 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa464043080 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4557fa640 1 -- 172.21.15.3:0/3683226574 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa464034aa0 con 0x5560deb2ef60 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 10 client.17436 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:31.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 10 client.17436 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 20 client.17436 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb97460 0x5560deb99880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 4 client.17436 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.117+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5560de818890 con 0x5560deb97460 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa471bbf640 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb97460 0x5560deb99880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4557fa640 10 client.17436 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa471bbf640 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb97460 0x5560deb99880 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ce3e807619d0e827 server_cookie=b52b4acd85cd8eed in_seq=0 out_seq=0 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4557fa640 1 -- 172.21.15.3:0/3683226574 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5560de818890 con 0x5560deb97460 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4557fa640 10 client.17436 handle_command_reply: tid=0 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 client.17436 shutdown 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 msgr2=0x5560deb35490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb35490 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7fa4640127c0 tx=0x7fa4640127f0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 shutdown_connections 2024-02-09T22:56:31.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5560deb97460 0x5560deb99880 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5560deb263e0 0x5560deb34f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5560deb2ef60 0x5560deb35490 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 --2- 172.21.15.3:0/3683226574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5560deb25a30 0x5560deb34a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4557fa640 0 client.17436 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 >> 172.21.15.3:0/3683226574 conn(0x5560deb20100 msgr2=0x5560deb1f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 shutdown_connections 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.118+0000 7fa4769fc740 1 -- 172.21.15.3:0/3683226574 wait complete. 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa4769fc740 20 client.17436 trim_cache size 0 max 16384 2024-02-09T22:56:31.130 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:56:31.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4580a2230 msgr2=0x7fa4580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4580a2230 0x7fa4580a46f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa464005fd0 tx=0x7fa464009b40 comp rx=0 tx=0).stop 2024-02-09T22:56:31.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 msgr2=0x7fa46c19d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 0x7fa46c19d650 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fa46800c890 tx=0x7fa46800cd50 comp rx=0 tx=0).stop 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 shutdown_connections 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4580a2230 0x7fa4580a46f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa46c101d20 0x7fa46c19db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa46c1026c0 0x7fa46c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 --2- 172.21.15.3:0/1390268690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa46c100440 0x7fa46c19d650 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 >> 172.21.15.3:0/1390268690 conn(0x7fa46c077310 msgr2=0x7fa46c069c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 shutdown_connections 2024-02-09T22:56:31.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.119+0000 7fa473e23640 1 -- 172.21.15.3:0/1390268690 wait complete. 2024-02-09T22:56:31.133 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:56:31.133 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:56:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 -- 172.21.15.3:0/822661167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e8106550 msgr2=0x7ff1e810e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/822661167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e8106550 0x7ff1e810e160 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0009cd0 tx=0x7ff1e002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 -- 172.21.15.3:0/822661167 shutdown_connections 2024-02-09T22:56:31.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/822661167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1e8100250 0x7ff1e8105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/822661167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e8106550 0x7ff1e810e160 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/822661167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e80ff8a0 0x7ff1e80ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 -- 172.21.15.3:0/822661167 >> 172.21.15.3:0/822661167 conn(0x7ff1e80776a0 msgr2=0x7ff1e80fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 -- 172.21.15.3:0/822661167 shutdown_connections 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.235+0000 7ff1ede9d640 1 -- 172.21.15.3:0/822661167 wait complete. 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 Processor -- start 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 -- start start 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1e80ff8a0 0x7ff1e819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e8106550 0x7ff1e819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 0x7ff1e8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e81981c0 con 0x7ff1e81976f0 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e8198330 con 0x7ff1e80ff8a0 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1ede9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e81984f0 con 0x7ff1e8106550 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1e7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 0x7ff1e8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1e7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 0x7ff1e8197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33648/0 (socket says 172.21.15.3:33648) 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1e7fff640 1 -- 172.21.15.3:0/2942805145 learned_addr learned my addr 172.21.15.3:0/2942805145 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e8106550 0x7ff1e819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.236+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1e80ff8a0 0x7ff1e819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e7fff640 1 -- 172.21.15.3:0/2942805145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e8106550 msgr2=0x7ff1e819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e7fff640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e8106550 0x7ff1e819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e7fff640 1 -- 172.21.15.3:0/2942805145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1e80ff8a0 msgr2=0x7ff1e819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e7fff640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1e80ff8a0 0x7ff1e819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e7fff640 1 -- 172.21.15.3:0/2942805145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1e0009900 con 0x7ff1e81976f0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e7fff640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 0x7ff1e8197b70 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00348f0 tx=0x7ff1e0034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e4ff9640 1 -- 172.21.15.3:0/2942805145 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00328c0 con 0x7ff1e81976f0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e81a4430 con 0x7ff1e81976f0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e4ff9640 1 -- 172.21.15.3:0/2942805145 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1e0003e50 con 0x7ff1e81976f0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1e4ff9640 1 -- 172.21.15.3:0/2942805145 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0040450 con 0x7ff1e81976f0 2024-02-09T22:56:31.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.237+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e81a48b0 con 0x7ff1e81976f0 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.239+0000 7ff1e4ff9640 1 -- 172.21.15.3:0/2942805145 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff1e000a040 con 0x7ff1e81976f0 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.239+0000 7ff1f0a7c740 1 Processor -- start 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.239+0000 7ff1f0a7c740 1 -- start start 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dca30 0x561b404dce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dd3e0 0x561b404e5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b40498020 con 0x561b404e5f60 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b4044b320 con 0x561b404dca30 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b40448850 con 0x561b404dd3e0 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e4ff9640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1d00a2230 0x7ff1d00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33652/0 (socket says 172.21.15.3:33652) 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/1022053709 learned_addr learned my addr 172.21.15.3:0/1022053709 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dca30 0x561b404dce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e7fff640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1d00a2230 0x7ff1d00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e4ff9640 1 -- 172.21.15.3:0/2942805145 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff1e00f0d60 con 0x7ff1e81976f0 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dd3e0 0x561b404e5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/1022053709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dd3e0 msgr2=0x561b404e5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dd3e0 0x561b404e5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/1022053709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dca30 msgr2=0x561b404dce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dca30 0x561b404dce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/1022053709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b40406210 con 0x561b404e5f60 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e83b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00b6c0 tx=0x7ff1dc00b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d58151c2643b9537 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1e7fff640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1d00a2230 0x7ff1d00a46f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00bd530 tx=0x7ff1e00b2f00 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.240+0000 7ff1cdffb640 1 -- 172.21.15.3:0/1022053709 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc012c40 con 0x561b404e5f60 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1cdffb640 1 -- 172.21.15.3:0/1022053709 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc012da0 con 0x561b404e5f60 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1cdffb640 1 -- 172.21.15.3:0/1022053709 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0049a0 con 0x561b404e5f60 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/1022053709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 msgr2=0x561b404e83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e83b0 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc00b6c0 tx=0x7ff1dc00b650 comp rx=0 tx=0).stop 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/1022053709 shutdown_connections 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e83b0 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dd3e0 0x561b404e5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/1022053709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dca30 0x561b404dce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/1022053709 >> 172.21.15.3:0/1022053709 conn(0x561b404d7100 msgr2=0x561b404d74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/1022053709 shutdown_connections 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.241+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/1022053709 wait complete. 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 -- start start 2024-02-09T22:56:31.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 Processor -- start 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404e3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 0x561b404e39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b40444030 con 0x561b404dca30 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b40117430 con 0x561b404dd3e0 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b400d3e00 con 0x561b404e5f60 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1f0a7c740 10 client.0 _ll_register_callbacks cb 0x561b40439030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404e2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33656/0 (socket says 172.21.15.3:33656) 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/851279389 learned_addr learned my addr 172.21.15.3:0/851279389 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404e3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/851279389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 msgr2=0x561b404e39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 0x561b404e39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 msgr2=0x561b404e3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404e3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/851279389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1dc01a040 con 0x561b404dca30 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.242+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404e3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404e2f50 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc015040 tx=0x7ff1dc005230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=650a85e497905170 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/851279389 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc010950 con 0x561b404dca30 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/851279389 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff1dc000e40 con 0x561b404dca30 2024-02-09T22:56:31.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b401437e0 con 0x561b404dca30 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/851279389 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1dc010ab0 con 0x561b404dca30 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 10 client.17448 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x561b404426b0 con 0x561b404dca30 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/851279389 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561b401437e0 con 0x561b404dca30 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/851279389 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff1dc01f470 con 0x561b404dca30 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 10 client.17448 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 10 client.17448 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 20 client.17448 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b404f62e0 0x561b404f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 4 client.17448 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.243+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b40180720 con 0x561b404f62e0 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.244+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b404f62e0 0x561b404f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.244+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b404f62e0 0x561b404f8700 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c7ec19369b298ad0 server_cookie=63993c70b43ab86d in_seq=0 out_seq=0 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.244+0000 7ff1c6ffd640 10 client.17448 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/851279389 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b40180720 con 0x561b404f62e0 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1c6ffd640 10 client.17448 handle_command_reply: tid=0 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 client.17448 shutdown 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 msgr2=0x561b404e2f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404e2f50 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc015040 tx=0x7ff1dc005230 comp rx=0 tx=0).stop 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 shutdown_connections 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b404f62e0 0x561b404f8700 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404e3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 0x561b404e39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1c6ffd640 0 client.17448 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/851279389 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404e2f50 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 >> 172.21.15.3:0/851279389 conn(0x561b404d7100 msgr2=0x561b404d8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 shutdown_connections 2024-02-09T22:56:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/851279389 wait complete. 2024-02-09T22:56:31.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.245+0000 7ff1f0a7c740 20 client.17448 trim_cache size 0 max 16384 2024-02-09T22:56:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 Processor -- start 2024-02-09T22:56:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 -- start start 2024-02-09T22:56:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dca30 0x561b404e5680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404fd020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b403c9130 con 0x561b404e5f60 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b40143810 con 0x561b404dd3e0 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b404f3040 con 0x561b404dca30 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e01d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33668/0 (socket says 172.21.15.3:33668) 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.257+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/2583182591 learned_addr learned my addr 172.21.15.3:0/2583182591 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404fd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/2583182591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dca30 msgr2=0x561b404e5680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dca30 0x561b404e5680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/2583182591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 msgr2=0x561b404fd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404fd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/2583182591 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b403dec90 con 0x561b404e5f60 2024-02-09T22:56:31.267 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404fd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e01d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0126b0 tx=0x7ff1dc01fa00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c908f3b7b1d8e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1cd7fa640 1 -- 172.21.15.3:0/2583182591 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc012c40 con 0x561b404e5f60 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1cd7fa640 1 -- 172.21.15.3:0/2583182591 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc00bd00 con 0x561b404e5f60 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1cd7fa640 1 -- 172.21.15.3:0/2583182591 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc01fd60 con 0x561b404e5f60 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/2583182591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 msgr2=0x561b404e01d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e01d0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0126b0 tx=0x7ff1dc01fa00 comp rx=0 tx=0).stop 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/2583182591 shutdown_connections 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404fd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.258+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404e5f60 0x561b404e01d0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/2583182591 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404dca30 0x561b404e5680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/2583182591 >> 172.21.15.3:0/2583182591 conn(0x561b404d7100 msgr2=0x561b404dd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/2583182591 shutdown_connections 2024-02-09T22:56:31.268 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/2583182591 wait complete. 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- start start 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 Processor -- start 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404ed4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404eb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 0x561b404ebff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b404ec6f0 con 0x561b404dca30 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b4054c7b0 con 0x561b404dd3e0 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b4054c8f0 con 0x561b404e5f60 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1f0a7c740 10 client.0 _ll_register_callbacks cb 0x7ff1dc00ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404ed4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1e6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404ed4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33676/0 (socket says 172.21.15.3:33676) 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.259+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/821801690 learned_addr learned my addr 172.21.15.3:0/821801690 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/821801690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 msgr2=0x561b404ebff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 0x561b404ebff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 msgr2=0x561b404eb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.269 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404eb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404eb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e6ffd640 1 -- 172.21.15.3:0/821801690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b404426b0 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1e6ffd640 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404ed4c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc01f7e0 tx=0x7ff1dc012c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a240188b3318eccc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1c6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/821801690 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc00b710 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/821801690 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff1dc00bab0 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1dc012da0 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/821801690 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff1dc010950 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1f0a7c740 10 client.17454 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x561b40404a70 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/821801690 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1dc012da0 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.260+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/821801690 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff1dc028470 con 0x561b404dca30 2024-02-09T22:56:31.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 10 client.17454 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 10 client.17454 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 20 client.17454 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b4054db20 0x561b4054ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 4 client.17454 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561b401cf890 con 0x561b4054db20 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b4054db20 0x561b4054ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1c6ffd640 10 client.17454 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1e77fe640 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b4054db20 0x561b4054ff10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=88add1dcf4a518a0 server_cookie=2f85a64f20bd33f in_seq=0 out_seq=0 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1c6ffd640 1 -- 172.21.15.3:0/821801690 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561b401cf890 con 0x561b4054db20 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1c6ffd640 10 client.17454 handle_command_reply: tid=0 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.261+0000 7ff1f0a7c740 1 client.17454 shutdown 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 msgr2=0x561b404ed4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404ed4c0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc01f7e0 tx=0x7ff1dc012c40 comp rx=0 tx=0).stop 2024-02-09T22:56:31.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 shutdown_connections 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561b4054db20 0x561b4054ff10 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b404dd3e0 0x561b404eb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b404e5f60 0x561b404ebff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1c6ffd640 0 client.17454 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 --2- 172.21.15.3:0/821801690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b404dca30 0x561b404ed4c0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 >> 172.21.15.3:0/821801690 conn(0x561b404d7100 msgr2=0x561b404e13f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 shutdown_connections 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 1 -- 172.21.15.3:0/821801690 wait complete. 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1f0a7c740 20 client.17454 trim_cache size 0 max 16384 2024-02-09T22:56:31.272 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:56:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1d00a2230 msgr2=0x7ff1d00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1d00a2230 0x7ff1d00a46f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00bd530 tx=0x7ff1e00b2f00 comp rx=0 tx=0).stop 2024-02-09T22:56:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 msgr2=0x7ff1e8197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 0x7ff1e8197b70 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7ff1e00348f0 tx=0x7ff1e0034cc0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 shutdown_connections 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff1d00a2230 0x7ff1d00a46f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff1e81976f0 0x7ff1e8197b70 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff1e8106550 0x7ff1e819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 --2- 172.21.15.3:0/2942805145 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff1e80ff8a0 0x7ff1e819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 >> 172.21.15.3:0/2942805145 conn(0x7ff1e80776a0 msgr2=0x7ff1e81046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 shutdown_connections 2024-02-09T22:56:31.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.262+0000 7ff1ede9d640 1 -- 172.21.15.3:0/2942805145 wait complete. 2024-02-09T22:56:31.276 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:56:31.276 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:56:31.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:31 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2366851820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:56:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 -- 172.21.15.3:0/2095246804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 msgr2=0x7f3644110210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 --2- 172.21.15.3:0/2095246804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f3644110210 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f363c009ca0 tx=0x7f363c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 -- 172.21.15.3:0/2095246804 shutdown_connections 2024-02-09T22:56:31.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 --2- 172.21.15.3:0/2095246804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3644102530 0x7f36440ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 --2- 172.21.15.3:0/2095246804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f3644110210 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 --2- 172.21.15.3:0/2095246804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441020c0 0x7f36440ff6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.379+0000 7f364b41c640 1 -- 172.21.15.3:0/2095246804 >> 172.21.15.3:0/2095246804 conn(0x7f3644076630 msgr2=0x7f36440fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 -- 172.21.15.3:0/2095246804 shutdown_connections 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 -- 172.21.15.3:0/2095246804 wait complete. 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 Processor -- start 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 -- start start 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441020c0 0x7f364419f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3644102530 0x7f364419fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f36441997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644199e50 con 0x7f364410de20 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3644199fc0 con 0x7f3644102530 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f364b41c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f364419a180 con 0x7f36441020c0 2024-02-09T22:56:31.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f36499b9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f36441997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f36499b9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f36441997b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33692/0 (socket says 172.21.15.3:33692) 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f36499b9640 1 -- 172.21.15.3:0/2464704926 learned_addr learned my addr 172.21.15.3:0/2464704926 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.380+0000 7f36491b8640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441020c0 0x7f364419f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36489b7640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3644102530 0x7f364419fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36499b9640 1 -- 172.21.15.3:0/2464704926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441020c0 msgr2=0x7f364419f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36499b9640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441020c0 0x7f364419f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36499b9640 1 -- 172.21.15.3:0/2464704926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3644102530 msgr2=0x7f364419fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36499b9640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3644102530 0x7f364419fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36499b9640 1 -- 172.21.15.3:0/2464704926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c009900 con 0x7f364410de20 2024-02-09T22:56:31.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f36499b9640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f36441997b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f363c005e30 tx=0x7f363c035cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f363a7fc640 1 -- 172.21.15.3:0/2464704926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c032ab0 con 0x7f364410de20 2024-02-09T22:56:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f363a7fc640 1 -- 172.21.15.3:0/2464704926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f363c032ea0 con 0x7f364410de20 2024-02-09T22:56:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364419a400 con 0x7f364410de20 2024-02-09T22:56:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f363a7fc640 1 -- 172.21.15.3:0/2464704926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c0047a0 con 0x7f364410de20 2024-02-09T22:56:31.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.381+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36441a6670 con 0x7f364410de20 2024-02-09T22:56:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.383+0000 7f363a7fc640 1 -- 172.21.15.3:0/2464704926 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f363c03b040 con 0x7f364410de20 2024-02-09T22:56:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 Processor -- start 2024-02-09T22:56:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 -- start start 2024-02-09T22:56:31.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f363a7fc640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36300a2230 0x7f36300a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270e0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 0x5590270e9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270ec3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902709c020 con 0x5590270e0a30 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902704f320 con 0x5590270e9f60 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f364dff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55902704c850 con 0x5590270e13e0 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36489b7640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36300a2230 0x7f36300a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270e0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f363a7fc640 1 -- 172.21.15.3:0/2464704926 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f363c0f0ab0 con 0x7f364410de20 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270e0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33702/0 (socket says 172.21.15.3:33702) 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 -- 172.21.15.3:0/922376468 learned_addr learned my addr 172.21.15.3:0/922376468 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36499b9640 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270ec3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 -- 172.21.15.3:0/922376468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 msgr2=0x5590270e9990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 0x5590270e9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 -- 172.21.15.3:0/922376468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 msgr2=0x5590270ec3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270ec3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36491b8640 1 -- 172.21.15.3:0/922376468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55902700a210 con 0x5590270e0a30 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.384+0000 7f36489b7640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36300a2230 0x7f36300a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3634005d80 tx=0x7f36340094d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.385+0000 7f36491b8640 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270e0e10 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f364000ad00 tx=0x7f364000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be624586d1fefe80 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.385+0000 7f362f7fe640 1 -- 172.21.15.3:0/922376468 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364000a800 con 0x5590270e0a30 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.385+0000 7f362f7fe640 1 -- 172.21.15.3:0/922376468 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3640018b20 con 0x5590270e0a30 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.385+0000 7f362f7fe640 1 -- 172.21.15.3:0/922376468 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640010c90 con 0x5590270e0a30 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.385+0000 7f364dff2740 1 -- 172.21.15.3:0/922376468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 msgr2=0x5590270e0e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.385+0000 7f364dff2740 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270e0e10 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7f364000ad00 tx=0x7f364000f790 comp rx=0 tx=0).stop 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- 172.21.15.3:0/922376468 shutdown_connections 2024-02-09T22:56:31.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270ec3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 0x5590270e9990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 --2- 172.21.15.3:0/922376468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270e0e10 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- 172.21.15.3:0/922376468 >> 172.21.15.3:0/922376468 conn(0x5590270db100 msgr2=0x5590270db4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- 172.21.15.3:0/922376468 shutdown_connections 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- 172.21.15.3:0/922376468 wait complete. 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- start start 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 Processor -- start 2024-02-09T22:56:31.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559027048030 con 0x5590270e9f60 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559026d1b430 con 0x5590270e13e0 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559026cd7e00 con 0x5590270e0a30 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.386+0000 7f364dff2740 10 client.0 _ll_register_callbacks cb 0x55902703d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33704/0 (socket says 172.21.15.3:33704) 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 -- 172.21.15.3:0/2145378054 learned_addr learned my addr 172.21.15.3:0/2145378054 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36491b8640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36489b7640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 -- 172.21.15.3:0/2145378054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 msgr2=0x5590270e6ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e6ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 -- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 msgr2=0x5590270e7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 -- 172.21.15.3:0/2145378054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f364000a960 con 0x5590270e9f60 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f36499b9640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e7970 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f363c006010 tx=0x7f363c0bd000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3f40fb4a8995193 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f362cff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f362cff9640 1 -- 172.21.15.3:0/2145378054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c0c1730 con 0x5590270e9f60 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f362cff9640 1 -- 172.21.15.3:0/2145378054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f363c0c5d80 con 0x5590270e9f60 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559026d477e0 con 0x5590270e9f60 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f362cff9640 1 -- 172.21.15.3:0/2145378054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f363c04fe30 con 0x5590270e9f60 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f364dff2740 10 client.17466 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:31.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x5590270466b0 con 0x5590270e9f60 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.387+0000 7f362cff9640 1 -- 172.21.15.3:0/2145378054 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559026d477e0 con 0x5590270e9f60 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f362cff9640 1 -- 172.21.15.3:0/2145378054 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f363c0a14a0 con 0x5590270e9f60 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f364dff2740 10 client.17466 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f364dff2740 10 client.17466 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f364dff2740 20 client.17466 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f364dff2740 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5590270fa2b0 0x5590270fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f364dff2740 4 client.17466 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559026d84720 con 0x5590270fa2b0 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f36491b8640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5590270fa2b0 0x5590270fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f362cff9640 10 client.17466 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:31.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.388+0000 7f36491b8640 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5590270fa2b0 0x5590270fc6d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6af063cf6c91be6b server_cookie=181f89839c7c64e8 in_seq=0 out_seq=0 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f362cff9640 1 -- 172.21.15.3:0/2145378054 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559026d84720 con 0x5590270fa2b0 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f362cff9640 10 client.17466 handle_command_reply: tid=0 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 client.17466 shutdown 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 msgr2=0x5590270e7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e7970 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f363c006010 tx=0x7f363c0bd000 comp rx=0 tx=0).stop 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 shutdown_connections 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5590270fa2b0 0x5590270fc6d0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e7970 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 --2- 172.21.15.3:0/2145378054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e6ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f362cff9640 0 client.17466 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 >> 172.21.15.3:0/2145378054 conn(0x5590270db100 msgr2=0x5590270dce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 shutdown_connections 2024-02-09T22:56:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 1 -- 172.21.15.3:0/2145378054 wait complete. 2024-02-09T22:56:31.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.389+0000 7f364dff2740 20 client.17466 trim_cache size 0 max 16384 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 Processor -- start 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 -- start start 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e21b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559026fcd130 con 0x5590270e9f60 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590270d6ab0 con 0x5590270e13e0 2024-02-09T22:56:31.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f364dff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590271018e0 con 0x5590270e0a30 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f36499b9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f36499b9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33716/0 (socket says 172.21.15.3:33716) 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.401+0000 7f36499b9640 1 -- 172.21.15.3:0/2982304093 learned_addr learned my addr 172.21.15.3:0/2982304093 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36489b7640 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36491b8640 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36499b9640 1 -- 172.21.15.3:0/2982304093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 msgr2=0x5590270e9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36499b9640 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36499b9640 1 -- 172.21.15.3:0/2982304093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 msgr2=0x5590270e21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36499b9640 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36499b9640 1 -- 172.21.15.3:0/2982304093 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559026fe2c90 con 0x5590270e9f60 2024-02-09T22:56:31.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36489b7640 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f36499b9640 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e26f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f363c0bd2e0 tx=0x7f363c0bd120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42d1fd124534a857 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f362effd640 1 -- 172.21.15.3:0/2982304093 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c0c5d80 con 0x5590270e9f60 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f362effd640 1 -- 172.21.15.3:0/2982304093 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f363c0c1730 con 0x5590270e9f60 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.402+0000 7f362effd640 1 -- 172.21.15.3:0/2982304093 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f363c02cca0 con 0x5590270e9f60 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- 172.21.15.3:0/2982304093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 msgr2=0x5590270e26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e26f0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f363c0bd2e0 tx=0x7f363c0bd120 comp rx=0 tx=0).stop 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- 172.21.15.3:0/2982304093 shutdown_connections 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e13e0 0x5590270e21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e9f60 0x5590270e26f0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- 172.21.15.3:0/2982304093 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e0a30 0x5590270e9620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- 172.21.15.3:0/2982304093 >> 172.21.15.3:0/2982304093 conn(0x5590270db100 msgr2=0x5590270db4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- 172.21.15.3:0/2982304093 shutdown_connections 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- 172.21.15.3:0/2982304093 wait complete. 2024-02-09T22:56:31.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- start start 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 Processor -- start 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270ef940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 0x5590270efe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270f03c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590270f18e0 con 0x5590270e0a30 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590271507c0 con 0x5590270e9f60 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559027150930 con 0x5590270e13e0 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f364dff2740 10 client.0 _ll_register_callbacks cb 0x7f364000a730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f36491b8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f36491b8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270ef940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33732/0 (socket says 172.21.15.3:33732) 2024-02-09T22:56:31.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.403+0000 7f36491b8640 1 -- 172.21.15.3:0/1687050452 learned_addr learned my addr 172.21.15.3:0/1687050452 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36491b8640 1 -- 172.21.15.3:0/1687050452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 msgr2=0x5590270efe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36491b8640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 0x5590270efe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36499b9640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270f03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36491b8640 1 -- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 msgr2=0x5590270f03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36491b8640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270f03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36491b8640 1 -- 172.21.15.3:0/1687050452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590270466b0 con 0x5590270e0a30 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36499b9640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f36491b8640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270ef940 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3640010680 tx=0x7f3640010790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27251ff25014bd67 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f362cff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f362cff9640 1 -- 172.21.15.3:0/1687050452 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3640028070 con 0x5590270e0a30 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f362cff9640 1 -- 172.21.15.3:0/1687050452 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f364000fcc0 con 0x5590270e0a30 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3640018b20 con 0x5590270e0a30 2024-02-09T22:56:31.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f362cff9640 1 -- 172.21.15.3:0/1687050452 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3640029d10 con 0x5590270e0a30 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f364dff2740 10 client.17472 fetch_fsmap learned FSMap version 36 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x559027008a70 con 0x5590270e0a30 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.404+0000 7f362cff9640 1 -- 172.21.15.3:0/1687050452 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3640018b20 con 0x5590270e0a30 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f362cff9640 1 -- 172.21.15.3:0/1687050452 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 36) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3640029250 con 0x5590270e0a30 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 10 client.17472 fetch_fsmap finished waiting for FSMap version 36 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 10 client.17472 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 20 client.17472 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559027152150 0x559027154570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 4 client.17472 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559026dd3890 con 0x559027152150 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f36489b7640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559027152150 0x559027154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f36489b7640 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559027152150 0x559027154570 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f47c0e80a74bbf8f server_cookie=42b685aeaf2c3af8 in_seq=0 out_seq=0 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f362cff9640 10 client.17472 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:31.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f362cff9640 1 -- 172.21.15.3:0/1687050452 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559026dd3890 con 0x559027152150 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f362cff9640 10 client.17472 handle_command_reply: tid=0 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.405+0000 7f364dff2740 1 client.17472 shutdown 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 msgr2=0x5590270ef940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270ef940 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f3640010680 tx=0x7f3640010790 comp rx=0 tx=0).stop 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 shutdown_connections 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559027152150 0x559027154570 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5590270e9f60 0x5590270f03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f362cff9640 0 client.17472 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5590270e13e0 0x5590270efe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 --2- 172.21.15.3:0/1687050452 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5590270e0a30 0x5590270ef940 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 >> 172.21.15.3:0/1687050452 conn(0x5590270db100 msgr2=0x5590270e5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 shutdown_connections 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 1 -- 172.21.15.3:0/1687050452 wait complete. 2024-02-09T22:56:31.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364dff2740 20 client.17472 trim_cache size 0 max 16384 2024-02-09T22:56:31.418 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:56:31.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36300a2230 msgr2=0x7f36300a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36300a2230 0x7f36300a46f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f3634005d80 tx=0x7f36340094d0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 msgr2=0x7f36441997b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f36441997b0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f363c005e30 tx=0x7f363c035cd0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 shutdown_connections 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36300a2230 0x7f36300a46f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3644102530 0x7f364419fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f364410de20 0x7f36441997b0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 --2- 172.21.15.3:0/2464704926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f36441020c0 0x7f364419f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 >> 172.21.15.3:0/2464704926 conn(0x7f3644076630 msgr2=0x7f364410f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 shutdown_connections 2024-02-09T22:56:31.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:31.406+0000 7f364b41c640 1 -- 172.21.15.3:0/2464704926 wait complete. 2024-02-09T22:56:31.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:31 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2366851820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:56:31.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 149791 2.64 MB/sec execute 145 sec latency 3548.267 ms 2024-02-09T22:56:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:31 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2366851820' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:56:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/2576630702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38101e00 msgr2=0x7f4a38105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.411 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/2576630702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38101e00 0x7f4a38105e40 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f4a28009930 tx=0x7f4a2802bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/2576630702 shutdown_connections 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/2576630702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4a38106520 0x7f4a3810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/2576630702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38101e00 0x7f4a38105e40 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/2576630702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38101450 0x7f4a38101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/2576630702 >> 172.21.15.3:0/2576630702 conn(0x7f4a380fccb0 msgr2=0x7f4a380ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/2576630702 shutdown_connections 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.406+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/2576630702 wait complete. 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 Processor -- start 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 -- start start 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4a38101450 0x7f4a3819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38106520 0x7f4a3819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.412 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 0x7f4a38197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a381981f0 con 0x7f4a38197720 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38198360 con 0x7f4a38101450 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3f9e9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a38198520 con 0x7f4a38106520 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3df86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 0x7f4a38197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3df86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 0x7f4a38197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33756/0 (socket says 172.21.15.3:33756) 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3d785640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4a38101450 0x7f4a3819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3df86640 1 -- 172.21.15.3:0/3115281739 learned_addr learned my addr 172.21.15.3:0/3115281739 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.407+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38106520 0x7f4a3819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3115281739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4a38101450 msgr2=0x7f4a3819d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4a38101450 0x7f4a3819d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 msgr2=0x7f4a38197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 0x7f4a38197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3115281739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a28009560 con 0x7f4a38106520 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3df86640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 0x7f4a38197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38106520 0x7f4a3819db50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a2802c170 tx=0x7f4a2802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a2e7fc640 1 -- 172.21.15.3:0/3115281739 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280042d0 con 0x7f4a38106520 2024-02-09T22:56:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a381a4410 con 0x7f4a38106520 2024-02-09T22:56:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a2e7fc640 1 -- 172.21.15.3:0/3115281739 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28004430 con 0x7f4a38106520 2024-02-09T22:56:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a2e7fc640 1 -- 172.21.15.3:0/3115281739 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28003c90 con 0x7f4a38106520 2024-02-09T22:56:32.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.408+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a381a48f0 con 0x7f4a38106520 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.410+0000 7f4a2e7fc640 1 -- 172.21.15.3:0/3115281739 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f4a28004df0 con 0x7f4a38106520 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 Processor -- start 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 -- start start 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a2e7fc640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4a240a2220 0x7f4a240a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9b3470 0x55e88c9c0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c921a00 con 0x55e88c9b3470 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c91ef30 con 0x55e88c9c0bb0 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a425bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c91a710 con 0x55e88c9b2b50 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a3df86640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4a240a2220 0x7f4a240a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a3d785640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a2e7fc640 1 -- 172.21.15.3:0/3115281739 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a280efec0 con 0x7f4a38106520 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a3cf84640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9b3470 0x55e88c9c0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a3d785640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9b2f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59354/0 (socket says 172.21.15.3:59354) 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a3d785640 1 -- 172.21.15.3:0/543757608 learned_addr learned my addr 172.21.15.3:0/543757608 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.411+0000 7f4a3df86640 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3cf84640 1 -- 172.21.15.3:0/543757608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 msgr2=0x55e88c9b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3cf84640 1 -- 172.21.15.3:0/543757608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9c0bb0 msgr2=0x55e88c9c2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3cf84640 1 -- 172.21.15.3:0/543757608 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e88c8df3b0 con 0x55e88c9b3470 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3df86640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4a240a2220 0x7f4a240a46e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a300049d0 tx=0x7f4a300116d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9b3470 0x55e88c9c0670 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4a280ed750 tx=0x7f4a280bc880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc538f1872645912 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a237fe640 1 -- 172.21.15.3:0/543757608 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a2809a010 con 0x55e88c9b3470 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a237fe640 1 -- 172.21.15.3:0/543757608 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28099e30 con 0x55e88c9b3470 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.412+0000 7f4a237fe640 1 -- 172.21.15.3:0/543757608 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280c5190 con 0x55e88c9b3470 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 -- 172.21.15.3:0/543757608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9b3470 msgr2=0x55e88c9c0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9b3470 0x55e88c9c0670 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f4a280ed750 tx=0x7f4a280bc880 comp rx=0 tx=0).stop 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 -- 172.21.15.3:0/543757608 shutdown_connections 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9b3470 0x55e88c9c0670 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 --2- 172.21.15.3:0/543757608 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 -- 172.21.15.3:0/543757608 >> 172.21.15.3:0/543757608 conn(0x55e88c9ad9d0 msgr2=0x55e88c9addb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 -- 172.21.15.3:0/543757608 shutdown_connections 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 -- 172.21.15.3:0/543757608 wait complete. 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 -- start start 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.413+0000 7f4a425bf740 1 Processor -- start 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9bdb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9be0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9be610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c8bfb90 con 0x55e88c9c0bb0 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c5ee250 con 0x55e88c9b3470 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c8a0200 con 0x55e88c9b2b50 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a425bf740 10 client.0 _ll_register_callbacks cb 0x55e88c8d65d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9be610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9be610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33774/0 (socket says 172.21.15.3:33774) 2024-02-09T22:56:32.423 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 -- 172.21.15.3:0/118559666 learned_addr learned my addr 172.21.15.3:0/118559666 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3d785640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9bdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9be0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 -- 172.21.15.3:0/118559666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 msgr2=0x55e88c9bdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9bdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 -- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 msgr2=0x55e88c9be0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9be0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 -- 172.21.15.3:0/118559666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a280dc040 con 0x55e88c9c0bb0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a3df86640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9be610 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e640 tx=0x7f4a30009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a20d2a49948119f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.414+0000 7f4a20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a20ff9640 1 -- 172.21.15.3:0/118559666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a300217d0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a20ff9640 1 -- 172.21.15.3:0/118559666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a30021930 con 0x55e88c9c0bb0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e88c940ee0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a20ff9640 1 -- 172.21.15.3:0/118559666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a30021af0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.424 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 10 client.17484 fetch_fsmap learned FSMap version 37 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55e88c918d90 con 0x55e88c9c0bb0 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a20ff9640 1 -- 172.21.15.3:0/118559666 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e88c940ee0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a20ff9640 1 -- 172.21.15.3:0/118559666 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 37) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4a3002b410 con 0x55e88c9c0bb0 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 10 client.17484 fetch_fsmap finished waiting for FSMap version 37 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 10 client.17484 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 20 client.17484 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88c9d0b90 0x55e88c9d2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 4 client.17484 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e88c61a600 con 0x55e88c9d0b90 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.415+0000 7f4a3d785640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88c9d0b90 0x55e88c9d2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.416+0000 7f4a20ff9640 10 client.17484 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.416+0000 7f4a3d785640 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88c9d0b90 0x55e88c9d2fb0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d118e58a2eab5edc server_cookie=b6946fa53c0cbed0 in_seq=0 out_seq=0 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a20ff9640 1 -- 172.21.15.3:0/118559666 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e88c61a600 con 0x55e88c9d0b90 2024-02-09T22:56:32.425 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a20ff9640 10 client.17484 handle_command_reply: tid=0 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 client.17484 shutdown 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 msgr2=0x55e88c9be610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9be610 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000e640 tx=0x7f4a30009500 comp rx=0 tx=0).stop 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 shutdown_connections 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88c9d0b90 0x55e88c9d2fb0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9be0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9be610 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a20ff9640 0 client.17484 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 --2- 172.21.15.3:0/118559666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9bdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 >> 172.21.15.3:0/118559666 conn(0x55e88c9ad9d0 msgr2=0x55e88c9af720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 shutdown_connections 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 1 -- 172.21.15.3:0/118559666 wait complete. 2024-02-09T22:56:32.426 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.417+0000 7f4a425bf740 20 client.17484 trim_cache size 0 max 16384 2024-02-09T22:56:32.427 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:56:32.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:56:32.427 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 Processor -- start 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 -- start start 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b2b50 0x55e88c9be0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 0x55e88c9be620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9badf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c940ee0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c9beb60 con 0x55e88c9b2b50 2024-02-09T22:56:32.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a425bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c9cd620 con 0x55e88c9b3470 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a3cf84640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 0x55e88c9be620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a3df86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9badf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a3d785640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b2b50 0x55e88c9be0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a3cf84640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 0x55e88c9be620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59378/0 (socket says 172.21.15.3:59378) 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.431+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3278619265 learned_addr learned my addr 172.21.15.3:0/3278619265 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3278619265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b2b50 msgr2=0x55e88c9be0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b2b50 0x55e88c9be0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3278619265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 msgr2=0x55e88c9badf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9badf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3cf84640 1 -- 172.21.15.3:0/3278619265 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e88c8b5820 con 0x55e88c9b3470 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3df86640 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9badf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 0x55e88c9be620 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a280bc6f0 tx=0x7f4a2809d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58d01b5a2a28995c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a22ffd640 1 -- 172.21.15.3:0/3278619265 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280bd0f0 con 0x55e88c9b3470 2024-02-09T22:56:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a22ffd640 1 -- 172.21.15.3:0/3278619265 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a280c5190 con 0x55e88c9b3470 2024-02-09T22:56:32.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.432+0000 7f4a22ffd640 1 -- 172.21.15.3:0/3278619265 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a280ed8c0 con 0x55e88c9b3470 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 -- 172.21.15.3:0/3278619265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 msgr2=0x55e88c9be620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 0x55e88c9be620 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4a280bc6f0 tx=0x7f4a2809d040 comp rx=0 tx=0).stop 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 -- 172.21.15.3:0/3278619265 shutdown_connections 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9badf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b3470 0x55e88c9be620 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 --2- 172.21.15.3:0/3278619265 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b2b50 0x55e88c9be0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 -- 172.21.15.3:0/3278619265 >> 172.21.15.3:0/3278619265 conn(0x55e88c9ad9d0 msgr2=0x55e88c9bc460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 -- 172.21.15.3:0/3278619265 shutdown_connections 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 -- 172.21.15.3:0/3278619265 wait complete. 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 -- start start 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.433+0000 7f4a425bf740 1 Processor -- start 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9c5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88c9c6d60 con 0x55e88c9c0bb0 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88ca270a0 con 0x55e88c9b3470 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e88ca271e0 con 0x55e88c9b2b50 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a425bf740 10 client.0 _ll_register_callbacks cb 0x7f4a30009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a3cf84640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9c5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a3d785640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a3df86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a3cf84640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9c5f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:46406/0 (socket says 172.21.15.3:46406) 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a3cf84640 1 -- 172.21.15.3:0/38448903 learned_addr learned my addr 172.21.15.3:0/38448903 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.434+0000 7f4a3d785640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59394/0 (socket says 172.21.15.3:59394) 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3df86640 1 -- 172.21.15.3:0/38448903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 msgr2=0x55e88c9c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3df86640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3df86640 1 -- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 msgr2=0x55e88c9c5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3df86640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9c5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3df86640 1 -- 172.21.15.3:0/38448903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e88c918d90 con 0x55e88c9c0bb0 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3cf84640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3d785640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a3df86640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c6660 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000a010 tx=0x7f4a30011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5218b93265849b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a20ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a20ff9640 1 -- 172.21.15.3:0/38448903 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3003a050 con 0x55e88c9c0bb0 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a20ff9640 1 -- 172.21.15.3:0/38448903 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4a3002b410 con 0x55e88c9c0bb0 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4a28099e30 con 0x55e88c9c0bb0 2024-02-09T22:56:32.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a20ff9640 1 -- 172.21.15.3:0/38448903 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4a3002b5d0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a425bf740 10 client.17490 fetch_fsmap learned FSMap version 37 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55e88c8df3b0 con 0x55e88c9c0bb0 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.435+0000 7f4a20ff9640 1 -- 172.21.15.3:0/38448903 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a28099e30 con 0x55e88c9c0bb0 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a20ff9640 1 -- 172.21.15.3:0/38448903 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 37) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4a30034660 con 0x55e88c9c0bb0 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a425bf740 10 client.17490 fetch_fsmap finished waiting for FSMap version 37 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a425bf740 10 client.17490 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a425bf740 20 client.17490 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a425bf740 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88ca289e0 0x55e88ca2ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a425bf740 4 client.17490 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e88c66b1b0 con 0x55e88ca289e0 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a3d785640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88ca289e0 0x55e88ca2ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a20ff9640 10 client.17490 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.436+0000 7f4a3d785640 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88ca289e0 0x55e88ca2ae00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a705221fe7f7030 server_cookie=2af85b3102751e8a in_seq=0 out_seq=0 2024-02-09T22:56:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a20ff9640 1 -- 172.21.15.3:0/38448903 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e88c66b1b0 con 0x55e88ca289e0 2024-02-09T22:56:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a20ff9640 10 client.17490 handle_command_reply: tid=0 2024-02-09T22:56:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 client.17490 shutdown 2024-02-09T22:56:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 msgr2=0x55e88c9c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c6660 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f4a3000a010 tx=0x7f4a30011e80 comp rx=0 tx=0).stop 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 shutdown_connections 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e88ca289e0 0x55e88ca2ae00 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e88c9b3470 0x55e88c9c5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e88c9c0bb0 0x55e88c9c6660 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 --2- 172.21.15.3:0/38448903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e88c9b2b50 0x55e88c9c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 >> 172.21.15.3:0/38448903 conn(0x55e88c9ad9d0 msgr2=0x55e88c9aee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a20ff9640 0 client.17490 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.460+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 shutdown_connections 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a425bf740 1 -- 172.21.15.3:0/38448903 wait complete. 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a425bf740 20 client.17490 trim_cache size 0 max 16384 2024-02-09T22:56:32.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:56:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:56:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:56:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:56:32.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4a240a2220 msgr2=0x7f4a240a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4a240a2220 0x7f4a240a46e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a300049d0 tx=0x7f4a300116d0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38106520 msgr2=0x7f4a3819db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38106520 0x7f4a3819db50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f4a2802c170 tx=0x7f4a2802c5d0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 shutdown_connections 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f4a240a2220 0x7f4a240a46e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4a38197720 0x7f4a38197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4a38106520 0x7f4a3819db50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 --2- 172.21.15.3:0/3115281739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4a38101450 0x7f4a3819d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 >> 172.21.15.3:0/3115281739 conn(0x7f4a380fccb0 msgr2=0x7f4a380fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 shutdown_connections 2024-02-09T22:56:32.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:32.461+0000 7f4a3f9e9640 1 -- 172.21.15.3:0/3115281739 wait complete. 2024-02-09T22:56:32.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:32 smithi089 ceph-mon[36665]: pgmap v161: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-02-09T22:56:32.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/400184202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c80691a0 msgr2=0x7fe6c8105e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/400184202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c80691a0 0x7fe6c8105e00 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009930 tx=0x7fe6bc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:32.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/400184202 shutdown_connections 2024-02-09T22:56:32.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/400184202 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81064e0 0x7fe6c810e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/400184202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c80691a0 0x7fe6c8105e00 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/400184202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 0x7fe6c8068bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/400184202 >> 172.21.15.3:0/400184202 conn(0x7fe6c8077230 msgr2=0x7fe6c8077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/400184202 shutdown_connections 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.537+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/400184202 wait complete. 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 Processor -- start 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 -- start start 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 0x7fe6c819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 0x7fe6c819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81976c0 0x7fe6c8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8198190 con 0x7fe6c81064e0 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8198300 con 0x7fe6c81976c0 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6cdcdf640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c81984c0 con 0x7fe6c80687f0 2024-02-09T22:56:32.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 0x7fe6c819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 0x7fe6c819db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33810/0 (socket says 172.21.15.3:33810) 2024-02-09T22:56:32.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 0x7fe6c819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 -- 172.21.15.3:0/2911859694 learned_addr learned my addr 172.21.15.3:0/2911859694 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81976c0 0x7fe6c8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 -- 172.21.15.3:0/2911859694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 msgr2=0x7fe6c819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 0x7fe6c819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 -- 172.21.15.3:0/2911859694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81976c0 msgr2=0x7fe6c8197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81976c0 0x7fe6c8197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c6ffd640 1 -- 172.21.15.3:0/2911859694 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6bc009560 con 0x7fe6c81064e0 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 0x7fe6c819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.538+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81976c0 0x7fe6c8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.539+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 0x7fe6c819db40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009230 tx=0x7fe6bc02c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.539+0000 7fe6c4ff9640 1 -- 172.21.15.3:0/2911859694 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc004350 con 0x7fe6c81064e0 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.539+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c81a4400 con 0x7fe6c81064e0 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.539+0000 7fe6c4ff9640 1 -- 172.21.15.3:0/2911859694 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6bc0044b0 con 0x7fe6c81064e0 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.539+0000 7fe6c4ff9640 1 -- 172.21.15.3:0/2911859694 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc003da0 con 0x7fe6c81064e0 2024-02-09T22:56:32.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.539+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c81a48e0 con 0x7fe6c81064e0 2024-02-09T22:56:32.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.541+0000 7fe6c4ff9640 1 -- 172.21.15.3:0/2911859694 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe6bc004df0 con 0x7fe6c81064e0 2024-02-09T22:56:32.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.541+0000 7fe6d098e740 1 Processor -- start 2024-02-09T22:56:32.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.541+0000 7fe6d098e740 1 -- start start 2024-02-09T22:56:32.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c4ff9640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe6b00a2230 0x7fe6b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bba570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bbaab0 0x564d79bc3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bc5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6d098e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79b28000 con 0x564d79bbaab0 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6d098e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79b25530 con 0x564d79bba190 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6d098e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79b20d10 con 0x564d79bc3690 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe6b00a2230 0x7fe6b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bba570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bba570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:46426/0 (socket says 172.21.15.3:46426) 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 -- 172.21.15.3:0/3034552276 learned_addr learned my addr 172.21.15.3:0/3034552276 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c4ff9640 1 -- 172.21.15.3:0/2911859694 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe6bc0f00d0 con 0x7fe6c81064e0 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bc5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bbaab0 0x564d79bc3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 -- 172.21.15.3:0/3034552276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 msgr2=0x564d79bc5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bc5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 -- 172.21.15.3:0/3034552276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bbaab0 msgr2=0x564d79bc3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bbaab0 0x564d79bc3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 -- 172.21.15.3:0/3034552276 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d79ac7210 con 0x564d79bba190 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bbaab0 0x564d79bc3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe6b00a2230 0x7fe6b00a46f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0005fd0 tx=0x7fe6c0009b30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.542+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bba570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80115c0 tx=0x7fe6b8011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe37ed019aadc39 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6adffb640 1 -- 172.21.15.3:0/3034552276 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800f5c0 con 0x564d79bba190 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6adffb640 1 -- 172.21.15.3:0/3034552276 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6b800fc00 con 0x564d79bba190 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6adffb640 1 -- 172.21.15.3:0/3034552276 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8019c30 con 0x564d79bba190 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 -- 172.21.15.3:0/3034552276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 msgr2=0x564d79bba570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bba570 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80115c0 tx=0x7fe6b8011a80 comp rx=0 tx=0).stop 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 -- 172.21.15.3:0/3034552276 shutdown_connections 2024-02-09T22:56:32.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bc5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bbaab0 0x564d79bc3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 --2- 172.21.15.3:0/3034552276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bba570 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 -- 172.21.15.3:0/3034552276 >> 172.21.15.3:0/3034552276 conn(0x564d79bb48c0 msgr2=0x564d79bb4ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 -- 172.21.15.3:0/3034552276 shutdown_connections 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.543+0000 7fe6d098e740 1 -- 172.21.15.3:0/3034552276 wait complete. 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 -- start start 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 Processor -- start 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bc0610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 0x564d79bc1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79ac5e60 con 0x564d79bc3690 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d797f44b0 con 0x564d79bba190 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79aa6670 con 0x564d79bbaab0 2024-02-09T22:56:32.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6d098e740 10 client.0 _ll_register_callbacks cb 0x564d79abe190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6c7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 0x564d79bc1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6c7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 0x564d79bc1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33830/0 (socket says 172.21.15.3:33830) 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6c7fff640 1 -- 172.21.15.3:0/2008786571 learned_addr learned my addr 172.21.15.3:0/2008786571 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.544+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bc0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6c7fff640 1 -- 172.21.15.3:0/2008786571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 msgr2=0x564d79bc0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6c7fff640 1 -- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 msgr2=0x564d79bc0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bc0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6c7fff640 1 -- 172.21.15.3:0/2008786571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b8009d40 con 0x564d79bc3690 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 0x564d79bc1090 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000eed0 tx=0x7fe6c0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14b9db19191859 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/2008786571 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c001f810 con 0x564d79bc3690 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/2008786571 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe6c0010e20 con 0x564d79bc3690 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d79820860 con 0x564d79bc3690 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/2008786571 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6c001f970 con 0x564d79bc3690 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 10 client.17502 fetch_fsmap learned FSMap version 37 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x564d79b1f390 con 0x564d79bc3690 2024-02-09T22:56:32.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/2008786571 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564d79820860 con 0x564d79bc3690 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/2008786571 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 37) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe6c0027030 con 0x564d79bc3690 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 10 client.17502 fetch_fsmap finished waiting for FSMap version 37 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 10 client.17502 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 20 client.17502 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79bd3730 0x564d79bd5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 4 client.17502 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.545+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d79871410 con 0x564d79bd3730 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.546+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79bd3730 0x564d79bd5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.546+0000 7fe6a6ffd640 10 client.17502 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.546+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79bd3730 0x564d79bd5b50 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a644b60d39284e4 server_cookie=9b00d2dbba25f222 in_seq=0 out_seq=0 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/2008786571 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564d79871410 con 0x564d79bd3730 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6a6ffd640 10 client.17502 handle_command_reply: tid=0 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 client.17502 shutdown 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 msgr2=0x564d79bc1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 0x564d79bc1090 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000eed0 tx=0x7fe6c0004ab0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 shutdown_connections 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79bd3730 0x564d79bd5b50 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bc3690 0x564d79bc1090 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 --2- 172.21.15.3:0/2008786571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bba190 0x564d79bc0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6a6ffd640 0 client.17502 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.547+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 >> 172.21.15.3:0/2008786571 conn(0x564d79bb48c0 msgr2=0x564d79bb6610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.548+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 shutdown_connections 2024-02-09T22:56:32.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.548+0000 7fe6d098e740 1 -- 172.21.15.3:0/2008786571 wait complete. 2024-02-09T22:56:32.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.548+0000 7fe6d098e740 20 client.17502 trim_cache size 0 max 16384 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 Processor -- start 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 -- start start 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bbaab0 0x564d79bb37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79bb56f0 con 0x564d79bba190 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79bd0180 con 0x564d79bbaab0 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6d098e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79bb00a0 con 0x564d79bc3690 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33846/0 (socket says 172.21.15.3:33846) 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 -- 172.21.15.3:0/1482408710 learned_addr learned my addr 172.21.15.3:0/1482408710 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bbaab0 0x564d79bb37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 -- 172.21.15.3:0/1482408710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 msgr2=0x564d79bca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 -- 172.21.15.3:0/1482408710 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bbaab0 msgr2=0x564d79bb37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bbaab0 0x564d79bb37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.563+0000 7fe6c77fe640 1 -- 172.21.15.3:0/1482408710 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d79abbc70 con 0x564d79bba190 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.564+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc0eb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009d10 tx=0x7fe6b8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193367dc1701f518 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.564+0000 7fe6ad7fa640 1 -- 172.21.15.3:0/1482408710 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8019650 con 0x564d79bba190 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.564+0000 7fe6ad7fa640 1 -- 172.21.15.3:0/1482408710 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6b80050b0 con 0x564d79bba190 2024-02-09T22:56:32.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.564+0000 7fe6ad7fa640 1 -- 172.21.15.3:0/1482408710 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800faa0 con 0x564d79bba190 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.564+0000 7fe6d098e740 1 -- 172.21.15.3:0/1482408710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 msgr2=0x564d79bc0eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.564+0000 7fe6d098e740 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc0eb0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009d10 tx=0x7fe6b8009310 comp rx=0 tx=0).stop 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- 172.21.15.3:0/1482408710 shutdown_connections 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bbaab0 0x564d79bb37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bc3690 0x564d79bca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 --2- 172.21.15.3:0/1482408710 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc0eb0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- 172.21.15.3:0/1482408710 >> 172.21.15.3:0/1482408710 conn(0x564d79bb48c0 msgr2=0x564d79bb3090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- 172.21.15.3:0/1482408710 shutdown_connections 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- 172.21.15.3:0/1482408710 wait complete. 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- start start 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 Processor -- start 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bc3690 0x564d79c270a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79c275e0 con 0x564d79bba190 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79c27750 con 0x564d79bc3690 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d79c27910 con 0x564d79bbaab0 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6d098e740 10 client.0 _ll_register_callbacks cb 0x7fe6b8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.565+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc9a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33858/0 (socket says 172.21.15.3:33858) 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 -- 172.21.15.3:0/887054308 learned_addr learned my addr 172.21.15.3:0/887054308 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bc3690 0x564d79c270a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 -- 172.21.15.3:0/887054308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 msgr2=0x564d79bc9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 -- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bc3690 msgr2=0x564d79c270a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bc3690 0x564d79c270a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 -- 172.21.15.3:0/887054308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d79ac7210 con 0x564d79bba190 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c7fff640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bc3690 0x564d79c270a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6c77fe640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc9a50 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8011ac0 tx=0x7fe6b8019410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265c49745045cc2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/887054308 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800f5c0 con 0x564d79bba190 2024-02-09T22:56:32.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/887054308 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe6b800f720 con 0x564d79bba190 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6b800fc00 con 0x564d79bba190 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/887054308 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6b800f8e0 con 0x564d79bba190 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6d098e740 10 client.17508 fetch_fsmap learned FSMap version 37 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.566+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x564d79ae1ac0 con 0x564d79bba190 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/887054308 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe6b8042020 con 0x564d79bba190 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/887054308 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 37) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe6b80037f0 con 0x564d79bba190 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6d098e740 10 client.17508 fetch_fsmap finished waiting for FSMap version 37 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6d098e740 10 client.17508 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6d098e740 20 client.17508 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6d098e740 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79c301f0 0x564d79c32610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6d098e740 4 client.17508 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:56:32.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564d7985d7b0 con 0x564d79c301f0 2024-02-09T22:56:32.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79c301f0 0x564d79c32610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6a6ffd640 10 client.17508 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.567+0000 7fe6c6ffd640 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79c301f0 0x564d79c32610 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ac354cd10dcbd8b server_cookie=f444741223328fe5 in_seq=0 out_seq=0 2024-02-09T22:56:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.569+0000 7fe6a6ffd640 1 -- 172.21.15.3:0/887054308 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564d7985d7b0 con 0x564d79c301f0 2024-02-09T22:56:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.569+0000 7fe6a6ffd640 10 client.17508 handle_command_reply: tid=0 2024-02-09T22:56:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.569+0000 7fe6d098e740 1 client.17508 shutdown 2024-02-09T22:56:32.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 msgr2=0x564d79bc9a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc9a50 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8011ac0 tx=0x7fe6b8019410 comp rx=0 tx=0).stop 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 shutdown_connections 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564d79c301f0 0x564d79c32610 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564d79bc3690 0x564d79c270a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564d79bbaab0 0x564d79bc9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 --2- 172.21.15.3:0/887054308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564d79bba190 0x564d79bc9a50 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 >> 172.21.15.3:0/887054308 conn(0x564d79bb48c0 msgr2=0x564d79bb6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6a6ffd640 0 client.17508 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 shutdown_connections 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 1 -- 172.21.15.3:0/887054308 wait complete. 2024-02-09T22:56:32.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.570+0000 7fe6d098e740 20 client.17508 trim_cache size 0 max 16384 2024-02-09T22:56:32.580 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:56:32.580 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:56:32.580 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "76dd5488-d3a7-44fb-9801-0edb2d510f3d", 2024-02-09T22:56:32.580 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:56:32.580 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe6b00a2230 msgr2=0x7fe6b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe6b00a2230 0x7fe6b00a46f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0005fd0 tx=0x7fe6c0009b30 comp rx=0 tx=0).stop 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 msgr2=0x7fe6c819db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 0x7fe6c819db40 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc009230 tx=0x7fe6bc02c680 comp rx=0 tx=0).stop 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 shutdown_connections 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe6b00a2230 0x7fe6b00a46f0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe6c81976c0 0x7fe6c8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe6c81064e0 0x7fe6c819db40 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 --2- 172.21.15.3:0/2911859694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe6c80687f0 0x7fe6c819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 >> 172.21.15.3:0/2911859694 conn(0x7fe6c8077230 msgr2=0x7fe6c80fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 shutdown_connections 2024-02-09T22:56:32.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.571+0000 7fe6cdcdf640 1 -- 172.21.15.3:0/2911859694 wait complete. 2024-02-09T22:56:32.583 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:32 smithi064 ceph-mon[36638]: pgmap v161: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-02-09T22:56:32.584 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:32 smithi003 ceph-mon[32678]: pgmap v161: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 633 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-02-09T22:56:32.585 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:56:32.585 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:56:32.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 150530 2.65 MB/sec execute 146 sec latency 4128.534 ms 2024-02-09T22:56:32.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 -- 172.21.15.3:0/3441530922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c105420 msgr2=0x7f180c10cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 --2- 172.21.15.3:0/3441530922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c105420 0x7f180c10cff0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f1804009cd0 tx=0x7f180402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 -- 172.21.15.3:0/3441530922 shutdown_connections 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 --2- 172.21.15.3:0/3441530922 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f180c100d50 0x7f180c104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 --2- 172.21.15.3:0/3441530922 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c105420 0x7f180c10cff0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 --2- 172.21.15.3:0/3441530922 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c1003a0 0x7f180c100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 -- 172.21.15.3:0/3441530922 >> 172.21.15.3:0/3441530922 conn(0x7f180c077300 msgr2=0x7f180c077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 -- 172.21.15.3:0/3441530922 shutdown_connections 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.691+0000 7f18131f7640 1 -- 172.21.15.3:0/3441530922 wait complete. 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 Processor -- start 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 -- start start 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f180c1003a0 0x7f180c19d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c105420 0x7f180c19dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 0x7f180c197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c198130 con 0x7f180c197660 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c1982a0 con 0x7f180c1003a0 2024-02-09T22:56:32.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f18131f7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c198460 con 0x7f180c105420 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 0x7f180c197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f180bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c105420 0x7f180c19dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 0x7f180c197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33864/0 (socket says 172.21.15.3:33864) 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 -- 172.21.15.3:0/1777901573 learned_addr learned my addr 172.21.15.3:0/1777901573 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1810f93640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f180c1003a0 0x7f180c19d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 -- 172.21.15.3:0/1777901573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c105420 msgr2=0x7f180c19dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c105420 0x7f180c19dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 -- 172.21.15.3:0/1777901573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f180c1003a0 msgr2=0x7f180c19d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f180c1003a0 0x7f180c19d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 -- 172.21.15.3:0/1777901573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1804009900 con 0x7f180c197660 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f180bfff640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c105420 0x7f180c19dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1811794640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 0x7f180c197ae0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1804005e30 tx=0x7f18040325e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.692+0000 7f1809ffb640 1 -- 172.21.15.3:0/1777901573 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804034e20 con 0x7f180c197660 2024-02-09T22:56:32.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.693+0000 7f1809ffb640 1 -- 172.21.15.3:0/1777901573 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1804003e50 con 0x7f180c197660 2024-02-09T22:56:32.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.693+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c1a43a0 con 0x7f180c197660 2024-02-09T22:56:32.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.693+0000 7f1809ffb640 1 -- 172.21.15.3:0/1777901573 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1804004750 con 0x7f180c197660 2024-02-09T22:56:32.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.693+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180c1a4820 con 0x7f180c197660 2024-02-09T22:56:32.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.695+0000 7f1809ffb640 1 -- 172.21.15.3:0/1777901573 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f180400a040 con 0x7f180c197660 2024-02-09T22:56:32.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.695+0000 7f1815dcf740 1 Processor -- start 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.695+0000 7f1815dcf740 1 -- start start 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.695+0000 7f1809ffb640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17fc0a2230 0x7f17fc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b3cd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f180bfff640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17fc0a2230 0x7f17fc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b458e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b48300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1809ffb640 1 -- 172.21.15.3:0/1777901573 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f18040f0b00 con 0x7f180c197660 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1815dcf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11af7d00 con 0x55ff11b3d330 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1815dcf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11aac2f0 con 0x55ff11b45eb0 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1815dcf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11aa9820 con 0x55ff11b3c980 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b3cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f180bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b458e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b3cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59466/0 (socket says 172.21.15.3:59466) 2024-02-09T22:56:32.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1811794640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b48300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 -- 172.21.15.3:0/809446890 learned_addr learned my addr 172.21.15.3:0/809446890 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f180bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b458e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33874/0 (socket says 172.21.15.3:33874) 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 -- 172.21.15.3:0/809446890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 msgr2=0x55ff11b48300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b48300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 -- 172.21.15.3:0/809446890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 msgr2=0x55ff11b458e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b458e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 -- 172.21.15.3:0/809446890 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff11a66ff0 con 0x55ff11b3c980 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f180bfff640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17fc0a2230 0x7f17fc0a46f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f180c198eb0 tx=0x7f17f8013ba0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f1810f93640 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b3cd60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1800009d20 tx=0x7f1800011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1b1443bf32be556 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.696+0000 7f17f6ffd640 1 -- 172.21.15.3:0/809446890 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800009e10 con 0x55ff11b3c980 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f17f6ffd640 1 -- 172.21.15.3:0/809446890 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18000198f0 con 0x55ff11b3c980 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f17f6ffd640 1 -- 172.21.15.3:0/809446890 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18000189f0 con 0x55ff11b3c980 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 -- 172.21.15.3:0/809446890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 msgr2=0x55ff11b3cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b3cd60 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1800009d20 tx=0x7f1800011810 comp rx=0 tx=0).stop 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 -- 172.21.15.3:0/809446890 shutdown_connections 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b48300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b458e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 --2- 172.21.15.3:0/809446890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b3cd60 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 -- 172.21.15.3:0/809446890 >> 172.21.15.3:0/809446890 conn(0x55ff11b37050 msgr2=0x55ff11b37430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 -- 172.21.15.3:0/809446890 shutdown_connections 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.697+0000 7f1815dcf740 1 -- 172.21.15.3:0/809446890 wait complete. 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 -- start start 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 Processor -- start 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 0x55ff11b42e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b3d330 0x55ff11b433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b45eb0 0x55ff11b438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11aa5000 con 0x55ff11b3c980 2024-02-09T22:56:32.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11778430 con 0x55ff11b3d330 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11734e00 con 0x55ff11b45eb0 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1815dcf740 10 client.0 _ll_register_callbacks cb 0x55ff11a997f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1810f93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 0x55ff11b42e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1810f93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 0x55ff11b42e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33882/0 (socket says 172.21.15.3:33882) 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1810f93640 1 -- 172.21.15.3:0/3831406724 learned_addr learned my addr 172.21.15.3:0/3831406724 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f180bfff640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b3d330 0x55ff11b433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.698+0000 7f1811794640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b45eb0 0x55ff11b438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1810f93640 1 -- 172.21.15.3:0/3831406724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b45eb0 msgr2=0x55ff11b438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1810f93640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b45eb0 0x55ff11b438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1810f93640 1 -- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b3d330 msgr2=0x55ff11b433a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1810f93640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b3d330 0x55ff11b433a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1810f93640 1 -- 172.21.15.3:0/3831406724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1800009950 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1810f93640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 0x55ff11b42e60 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1800011d10 tx=0x7f1800019880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad8d89c3186b5a94 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f17ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f17ebfff640 1 -- 172.21.15.3:0/3831406724 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1800019c70 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f17ebfff640 1 -- 172.21.15.3:0/3831406724 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1800019dd0 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ff117a47e0 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f17ebfff640 1 -- 172.21.15.3:0/3831406724 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1800004050 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1815dcf740 10 client.17520 fetch_fsmap learned FSMap version 37 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55ff11aa3680 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f17ebfff640 1 -- 172.21.15.3:0/3831406724 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ff117a47e0 con 0x55ff11b3c980 2024-02-09T22:56:32.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.699+0000 7f17ebfff640 1 -- 172.21.15.3:0/3831406724 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 37) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1800003710 con 0x55ff11b3c980 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f1815dcf740 10 client.17520 fetch_fsmap finished waiting for FSMap version 37 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f1815dcf740 10 client.17520 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f1815dcf740 20 client.17520 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f1815dcf740 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11b56380 0x55ff11b587c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f1815dcf740 4 client.17520 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ff117e1720 con 0x55ff11b56380 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f180bfff640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11b56380 0x55ff11b587c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f17ebfff640 10 client.17520 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.700+0000 7f180bfff640 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11b56380 0x55ff11b587c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90425b06a7c502e5 server_cookie=91648cb1fb122222 in_seq=0 out_seq=0 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f17ebfff640 1 -- 172.21.15.3:0/3831406724 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ff117e1720 con 0x55ff11b56380 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f17ebfff640 10 client.17520 handle_command_reply: tid=0 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 client.17520 shutdown 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 msgr2=0x55ff11b42e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 0x55ff11b42e60 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1800011d10 tx=0x7f1800019880 comp rx=0 tx=0).stop 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 shutdown_connections 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11b56380 0x55ff11b587c0 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b3d330 0x55ff11b433a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b45eb0 0x55ff11b438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f17ebfff640 0 client.17520 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 --2- 172.21.15.3:0/3831406724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3c980 0x55ff11b42e60 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 >> 172.21.15.3:0/3831406724 conn(0x55ff11b37050 msgr2=0x55ff11b38da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 shutdown_connections 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 1 -- 172.21.15.3:0/3831406724 wait complete. 2024-02-09T22:56:32.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.701+0000 7f1815dcf740 20 client.17520 trim_cache size 0 max 16384 2024-02-09T22:56:32.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.716+0000 7f1815dcf740 1 Processor -- start 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.716+0000 7f1815dcf740 1 -- start start 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b45590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b433b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b438f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1815dcf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11a29fd0 con 0x55ff11b3d330 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1815dcf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11b32c90 con 0x55ff11b45eb0 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1815dcf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11b3c480 con 0x55ff11b3c980 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b433b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b433b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33890/0 (socket says 172.21.15.3:33890) 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 -- 172.21.15.3:0/2417670294 learned_addr learned my addr 172.21.15.3:0/2417670294 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1810f93640 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b45590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f1811794640 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b438f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 -- 172.21.15.3:0/2417670294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 msgr2=0x55ff11b45590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b45590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 -- 172.21.15.3:0/2417670294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 msgr2=0x55ff11b438f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b438f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 -- 172.21.15.3:0/2417670294 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ff11a3fb30 con 0x55ff11b3d330 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f180bfff640 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b433b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f180c0695d0 tx=0x7f17f8020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2fe166642ff2f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f17f67fc640 1 -- 172.21.15.3:0/2417670294 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f802d070 con 0x55ff11b3d330 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f17f67fc640 1 -- 172.21.15.3:0/2417670294 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17f800fe60 con 0x55ff11b3d330 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.717+0000 7f17f67fc640 1 -- 172.21.15.3:0/2417670294 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17f8018040 con 0x55ff11b3d330 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 -- 172.21.15.3:0/2417670294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 msgr2=0x55ff11b433b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b433b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f180c0695d0 tx=0x7f17f8020000 comp rx=0 tx=0).stop 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 -- 172.21.15.3:0/2417670294 shutdown_connections 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b438f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b433b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 --2- 172.21.15.3:0/2417670294 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b45590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 -- 172.21.15.3:0/2417670294 >> 172.21.15.3:0/2417670294 conn(0x55ff11b37050 msgr2=0x55ff11b5d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 -- 172.21.15.3:0/2417670294 shutdown_connections 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 -- 172.21.15.3:0/2417670294 wait complete. 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 -- start start 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.718+0000 7f1815dcf740 1 Processor -- start 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b4b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b4be70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b4c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11bac8f0 con 0x55ff11b3d330 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11baca30 con 0x55ff11b45eb0 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ff11bacb70 con 0x55ff11b3c980 2024-02-09T22:56:32.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1815dcf740 10 client.0 _ll_register_callbacks cb 0x7f1800009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f180bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b4be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1810f93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b4b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b4c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f180bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b4be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33900/0 (socket says 172.21.15.3:33900) 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1810f93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b4b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59484/0 (socket says 172.21.15.3:59484) 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f180bfff640 1 -- 172.21.15.3:0/1939558884 learned_addr learned my addr 172.21.15.3:0/1939558884 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 -- 172.21.15.3:0/1939558884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 msgr2=0x55ff11b4b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b4b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 -- 172.21.15.3:0/1939558884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 msgr2=0x55ff11b4be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b4be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 -- 172.21.15.3:0/1939558884 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17f8029a90 con 0x55ff11b45eb0 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f180bfff640 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b4be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:32.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.719+0000 7f1811794640 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b4c3b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18040ee390 tx=0x7f18040bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c38df3942ad5ee9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f17ebfff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f17ebfff640 1 -- 172.21.15.3:0/1939558884 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18040bd6e0 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f17ebfff640 1 -- 172.21.15.3:0/1939558884 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f18040c1780 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f18000198f0 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f17ebfff640 1 -- 172.21.15.3:0/1939558884 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18040c5dd0 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 10 client.26747 fetch_fsmap learned FSMap version 37 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) v3 -- 0x55ff11a65850 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f17ebfff640 1 -- 172.21.15.3:0/1939558884 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18000198f0 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f17ebfff640 1 -- 172.21.15.3:0/1939558884 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 37) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f18040c5940 con 0x55ff11b45eb0 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 10 client.26747 fetch_fsmap finished waiting for FSMap version 37 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 10 client.26747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 20 client.26747 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11bae160 0x55ff11bb0580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:32.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 4 client.26747 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ff11830890 con 0x55ff11bae160 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.720+0000 7f1810f93640 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11bae160 0x55ff11bb0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1810f93640 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11bae160 0x55ff11bb0580 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4680c81e47a28196 server_cookie=e05fc3725cb58d3d in_seq=0 out_seq=0 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f17ebfff640 10 client.26747 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f17ebfff640 1 -- 172.21.15.3:0/1939558884 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ff11830890 con 0x55ff11bae160 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f17ebfff640 10 client.26747 handle_command_reply: tid=0 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 client.26747 shutdown 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 msgr2=0x55ff11b4c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b4c3b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f18040ee390 tx=0x7f18040bd040 comp rx=0 tx=0).stop 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 shutdown_connections 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ff11bae160 0x55ff11bb0580 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ff11b45eb0 0x55ff11b4c3b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ff11b3d330 0x55ff11b4be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f17ebfff640 0 client.26747 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 --2- 172.21.15.3:0/1939558884 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ff11b3c980 0x55ff11b4b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 >> 172.21.15.3:0/1939558884 conn(0x55ff11b37050 msgr2=0x55ff11b35960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 shutdown_connections 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 1 -- 172.21.15.3:0/1939558884 wait complete. 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.721+0000 7f1815dcf740 20 client.26747 trim_cache size 0 max 16384 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:56:32.732 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: "76dd5488-d3a7-44fb-9801-0edb2d510f3d": { 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "76dd5488-d3a7-44fb-9801-0edb2d510f3d", 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:56:32.733 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17fc0a2230 msgr2=0x7f17fc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17fc0a2230 0x7f17fc0a46f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f180c198eb0 tx=0x7f17f8013ba0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 msgr2=0x7f180c197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 0x7f180c197ae0 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1804005e30 tx=0x7f18040325e0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 shutdown_connections 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17fc0a2230 0x7f17fc0a46f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f180c197660 0x7f180c197ae0 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f180c105420 0x7f180c19dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 --2- 172.21.15.3:0/1777901573 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f180c1003a0 0x7f180c19d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:32.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 >> 172.21.15.3:0/1777901573 conn(0x7f180c077300 msgr2=0x7f180c069c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:32.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 shutdown_connections 2024-02-09T22:56:32.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:56:32.722+0000 7f18131f7640 1 -- 172.21.15.3:0/1777901573 wait complete. 2024-02-09T22:56:32.736 INFO:tasks.cephfs.filesystem:scrub status for tag:76dd5488-d3a7-44fb-9801-0edb2d510f3d - {'path': '/', 'tag': '76dd5488-d3a7-44fb-9801-0edb2d510f3d', 'options': 'recursive,force'} 2024-02-09T22:56:32.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:56:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:56:32.580+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000020c [...7,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/EXCEL/ auth{3=1} v20576 ap=2 f(v737 m2024-02-09T22:56:19.717879+0000 2=2+0) n(v22 rc2024-02-09T22:56:19.717879+0000 b520192 3=2+1) old_inodes=4 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@2702} mcw={3=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d51f7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000020c:[<0x10000000009/EXCEL v1452>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x1000000020c:[<0x10000000009/EXCEL v20576>,<0x10000000008/~dmtmp v9599>,<0x10000000007/client0 v405>,<0x10000000006/clients v395>,<0x10000000005/tmp v397>,<0x100000001fb/client.0 v381>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v396>,<0x10000000001/sv_1 v411>,<0x10000000000/qa v399>,<0x1/volumes v381>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:56:18.807890+0000 8=8+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:56:19.717879+0000 b520192 3=2+1)","memory_value.dirstat":"f(v737 m2024-02-09T22:56:19.717879+0000 2=2+0)","memory_value.rstat":"n(v22 rc2024-02-09T22:56:19.717879+0000 b520192 3=2+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:56:32.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:56:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:56:32.581+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000215 [...7,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/COREL/ auth{1=1,2=1,3=1,4=1} v20578 ap=2 f(v513 m2024-02-09T22:56:19.742879+0000 11=11+0) n(v22 rc2024-02-09T22:56:19.742879+0000 b2861056 12=11+1) old_inodes=4 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@1834} mcw={3=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d5150680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000215:[<0x10000000009/COREL v1464>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x10000000215:[<0x10000000009/COREL v20578>,<0x10000000008/~dmtmp v9599>,<0x10000000007/client0 v405>,<0x10000000006/clients v395>,<0x10000000005/tmp v397>,<0x100000001fb/client.0 v381>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v396>,<0x10000000001/sv_1 v411>,<0x10000000000/qa v399>,<0x1/volumes v381>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:56:19.245885+0000 15=15+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:56:19.742879+0000 b2861056 12=11+1)","memory_value.dirstat":"f(v513 m2024-02-09T22:56:19.742879+0000 11=11+0)","memory_value.rstat":"n(v22 rc2024-02-09T22:56:19.742879+0000 b2861056 12=11+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:56:32.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:56:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:56:32.592+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000225 [...7,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM/ auth{1=1} v20600 ap=2 f(v1255 m2024-02-09T22:56:19.755879+0000 3=3+0) n(v25 rc2024-02-09T22:56:19.755879+0000 b780288 4=3+1) old_inodes=4 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@4261} mcw={1=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyrstat=0 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d5398580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x10000000225:[<0x10000000009/PM v1478>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x10000000225:[<0x10000000009/PM v20600>,<0x10000000008/~dmtmp v9599>,<0x10000000007/client0 v405>,<0x10000000006/clients v395>,<0x10000000005/tmp v397>,<0x100000001fb/client.0 v381>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v396>,<0x10000000001/sv_1 v411>,<0x10000000000/qa v399>,<0x1/volumes v381>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:56:19.677880+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:56:19.755879+0000 b780288 4=3+1)","memory_value.dirstat":"f(v1255 m2024-02-09T22:56:19.755879+0000 3=3+0)","memory_value.rstat":"n(v25 rc2024-02-09T22:56:19.755879+0000 b780288 4=3+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:56:32.936 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 22:56:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T22:56:32.594+0000 7efe631e3700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023e [...7,head] /volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS/ auth{4=1} v20571 ap=2 f(v256 m2024-02-09T22:56:19.784878+0000 1=1+0) n(v22 rc2024-02-09T22:56:19.784878+0000 b260096 2=1+1) old_inodes=4 (idft sync r=1) (inest sync r=1) (ifile sync r=1) (iversion lock) caps={24979=pAsLsXsFs/pAsLsXsFsx@907} mcw={4=AsLsXsFs} | dirtyscattered=0 request=0 lock=3 dirfrag=1 caps=1 dirtyparent=0 replicated=1 dirty=0 waiter=0 authpin=1 scrubqueue=0 0x5651d53d5b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":true,"read_ret_val":0,"ondisk_value":"(2)0x1000000023e:[<0x10000000009/ACCESS v1492>,<0x10000000008/~dmtmp v34>,<0x10000000007/client0 v18>,<0x10000000006/clients v8>,<0x10000000005/tmp v8>,<0x100000001fb/client.0 v10>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v26>,<0x10000000001/sv_1 v49>,<0x10000000000/qa v37>,<0x1/volumes v34>]//[]","memoryvalue":"(2)0x1000000023e:[<0x10000000009/ACCESS v20571>,<0x10000000008/~dmtmp v9603>,<0x10000000007/client0 v405>,<0x10000000006/clients v395>,<0x10000000005/tmp v397>,<0x100000001fb/client.0 v381>,<0x100000001fa/6e490f76-85e9-49a3-85a1-cc03f2ced6d0 v396>,<0x10000000001/sv_1 v411>,<0x10000000000/qa v399>,<0x1/volumes v381>]//[]","error_str":""},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f(v0 m2024-02-09T22:56:19.092887+0000 4=4+0)","ondisk_value.rstat":"n(v0 rc2024-02-09T22:56:19.784878+0000 b260096 2=1+1)","memory_value.dirstat":"f(v256 m2024-02-09T22:56:19.784878+0000 1=1+0)","memory_value.rstat":"n(v22 rc2024-02-09T22:56:19.784878+0000 b260096 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":0} 2024-02-09T22:56:33.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:56:33.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.479 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:33 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:56:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:56:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:33.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:33 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 150692 2.64 MB/sec execute 147 sec latency 917.383 ms 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: Scrub error on inode 0x1000000020c (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/EXCEL) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: Scrub error on inode 0x10000000215 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/COREL) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: Scrub error on inode 0x10000000225 (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/PM) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: Scrub error on inode 0x1000000023e (/volumes/qa/sv_1/6e490f76-85e9-49a3-85a1-cc03f2ced6d0/client.0/tmp/clients/client0/~dmtmp/ACCESS) see mds.b log and `damage ls` output for details 2024-02-09T22:56:33.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:33 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:34 smithi089 ceph-mon[36665]: pgmap v162: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 641 KiB/s wr, 37 op/s 2024-02-09T22:56:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:34 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:34 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:34.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:34 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:56:34.478 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:34 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:34 smithi003 ceph-mon[32678]: pgmap v162: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 641 KiB/s wr, 37 op/s 2024-02-09T22:56:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:34 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:34 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:34 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:56:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:34 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:34.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 150692 2.62 MB/sec execute 148 sec latency 1917.451 ms 2024-02-09T22:56:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:34 smithi064 ceph-mon[36638]: pgmap v162: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 641 KiB/s wr, 37 op/s 2024-02-09T22:56:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:34 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:34 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:34 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:56:34.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:34 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:56:35.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 151293 2.61 MB/sec execute 149 sec latency 2740.384 ms 2024-02-09T22:56:36.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:36 smithi089 ceph-mon[36665]: pgmap v163: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 641 KiB/s wr, 37 op/s 2024-02-09T22:56:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:36 smithi003 ceph-mon[32678]: pgmap v163: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 641 KiB/s wr, 37 op/s 2024-02-09T22:56:36.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 152989 2.68 MB/sec execute 150 sec latency 14.898 ms 2024-02-09T22:56:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:36 smithi064 ceph-mon[36638]: pgmap v163: 97 pgs: 97 active+clean; 64 MiB data, 903 MiB used, 1.0 TiB / 1.0 TiB avail; 89 KiB/s rd, 641 KiB/s wr, 37 op/s 2024-02-09T22:56:37.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 153217 2.66 MB/sec execute 151 sec latency 819.215 ms 2024-02-09T22:56:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:38 smithi089 ceph-mon[36665]: pgmap v164: 97 pgs: 97 active+clean; 65 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.2 MiB/s wr, 91 op/s 2024-02-09T22:56:38.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:38 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:56:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:38 smithi003 ceph-mon[32678]: pgmap v164: 97 pgs: 97 active+clean; 65 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.2 MiB/s wr, 91 op/s 2024-02-09T22:56:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:38 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:56:38.639 INFO:tasks.workunit.client.0.smithi003.stdout: 1 154242 2.67 MB/sec execute 152 sec latency 1229.649 ms 2024-02-09T22:56:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:38 smithi064 ceph-mon[36638]: pgmap v164: 97 pgs: 97 active+clean; 65 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 1.2 MiB/s wr, 91 op/s 2024-02-09T22:56:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:38 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:56:39.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 154242 2.66 MB/sec execute 153 sec latency 1034.661 ms 2024-02-09T22:56:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:40 smithi003 ceph-mon[32678]: pgmap v165: 97 pgs: 97 active+clean; 65 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-02-09T22:56:40.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 154810 2.64 MB/sec execute 154 sec latency 1855.934 ms 2024-02-09T22:56:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:40 smithi064 ceph-mon[36638]: pgmap v165: 97 pgs: 97 active+clean; 65 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-02-09T22:56:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:40 smithi089 ceph-mon[36665]: pgmap v165: 97 pgs: 97 active+clean; 65 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 1.2 MiB/s wr, 77 op/s 2024-02-09T22:56:41.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156547 2.71 MB/sec execute 155 sec latency 12.538 ms 2024-02-09T22:56:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:42 smithi003 ceph-mon[32678]: pgmap v166: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-02-09T22:56:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:42.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:42.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156759 2.70 MB/sec execute 156 sec latency 824.374 ms 2024-02-09T22:56:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:42 smithi064 ceph-mon[36638]: pgmap v166: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-02-09T22:56:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:42 smithi089 ceph-mon[36665]: pgmap v166: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.5 MiB/s wr, 133 op/s 2024-02-09T22:56:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:43 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:56:43.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156815 2.68 MB/sec execute 157 sec latency 1232.561 ms 2024-02-09T22:56:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:43 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:56:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:43 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:56:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:44 smithi003 ceph-mon[32678]: pgmap v167: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 902 KiB/s wr, 109 op/s 2024-02-09T22:56:44.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156815 2.66 MB/sec execute 158 sec latency 1552.777 ms 2024-02-09T22:56:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:44 smithi064 ceph-mon[36638]: pgmap v167: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 902 KiB/s wr, 109 op/s 2024-02-09T22:56:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:44 smithi089 ceph-mon[36665]: pgmap v167: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 902 KiB/s wr, 109 op/s 2024-02-09T22:56:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 -- 172.21.15.3:0/3433659371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 msgr2=0x7fa87810e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 --2- 172.21.15.3:0/3433659371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87810e0e0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7fa870009cd0 tx=0x7fa87002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 -- 172.21.15.3:0/3433659371 shutdown_connections 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 --2- 172.21.15.3:0/3433659371 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8780696f0 0x7fa878105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 --2- 172.21.15.3:0/3433659371 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87810e0e0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 --2- 172.21.15.3:0/3433659371 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa878068d40 0x7fa878069120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 -- 172.21.15.3:0/3433659371 >> 172.21.15.3:0/3433659371 conn(0x7fa878077220 msgr2=0x7fa878077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 -- 172.21.15.3:0/3433659371 shutdown_connections 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.559+0000 7fa87dda1640 1 -- 172.21.15.3:0/3433659371 wait complete. 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 Processor -- start 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 -- start start 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa878068d40 0x7fa87819d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87819db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8781976a0 0x7fa878197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa878198170 con 0x7fa878106510 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781982e0 con 0x7fa8781976a0 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa87dda1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8781984a0 con 0x7fa878068d40 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa876ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87819db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa876ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87819db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55948/0 (socket says 172.21.15.3:55948) 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa876ffd640 1 -- 172.21.15.3:0/1763598369 learned_addr learned my addr 172.21.15.3:0/1763598369 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa8777fe640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa878068d40 0x7fa87819d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.560+0000 7fa877fff640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8781976a0 0x7fa878197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa876ffd640 1 -- 172.21.15.3:0/1763598369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa878068d40 msgr2=0x7fa87819d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa878068d40 0x7fa87819d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa876ffd640 1 -- 172.21.15.3:0/1763598369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8781976a0 msgr2=0x7fa878197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8781976a0 0x7fa878197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa876ffd640 1 -- 172.21.15.3:0/1763598369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa870009900 con 0x7fa878106510 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87819db20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00d770 tx=0x7fa86c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:45.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa874ff9640 1 -- 172.21.15.3:0/1763598369 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c009c30 con 0x7fa878106510 2024-02-09T22:56:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa874ff9640 1 -- 172.21.15.3:0/1763598369 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa86c00de80 con 0x7fa878106510 2024-02-09T22:56:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8781a4440 con 0x7fa878106510 2024-02-09T22:56:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa874ff9640 1 -- 172.21.15.3:0/1763598369 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa86c00f6b0 con 0x7fa878106510 2024-02-09T22:56:45.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.561+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8781a4920 con 0x7fa878106510 2024-02-09T22:56:45.577 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.563+0000 7fa874ff9640 1 -- 172.21.15.3:0/1763598369 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa86c010460 con 0x7fa878106510 2024-02-09T22:56:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.563+0000 7fa880978740 1 Processor -- start 2024-02-09T22:56:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 -- start start 2024-02-09T22:56:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f6d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f76260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa874ff9640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8600a2220 0x7fa8600a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3f78b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3edba00 con 0x5559b3f6d0b0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3ed8f30 con 0x5559b3f767a0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa880978740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3ed4710 con 0x5559b3f6d9d0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f6d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa8777fe640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8600a2220 0x7fa8600a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f6d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55958/0 (socket says 172.21.15.3:55958) 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 -- 172.21.15.3:0/2035535931 learned_addr learned my addr 172.21.15.3:0/2035535931 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa874ff9640 1 -- 172.21.15.3:0/1763598369 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa86c0cddf0 con 0x7fa878106510 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa876ffd640 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f76260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3f78b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 -- 172.21.15.3:0/2035535931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 msgr2=0x5559b3f76260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f76260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 -- 172.21.15.3:0/2035535931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 msgr2=0x5559b3f78b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3f78b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.564+0000 7fa877fff640 1 -- 172.21.15.3:0/2035535931 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559b3e993b0 con 0x5559b3f6d0b0 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa8777fe640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8600a2220 0x7fa8600a46e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa868005fd0 tx=0x7fa86800b390 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa877fff640 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f6d490 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa870004af0 tx=0x7fa870004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b971952d3a2e3310 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa85dffb640 1 -- 172.21.15.3:0/2035535931 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870034640 con 0x5559b3f6d0b0 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa85dffb640 1 -- 172.21.15.3:0/2035535931 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8700347a0 con 0x5559b3f6d0b0 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa85dffb640 1 -- 172.21.15.3:0/2035535931 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870003c90 con 0x5559b3f6d0b0 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 -- 172.21.15.3:0/2035535931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 msgr2=0x5559b3f6d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f6d490 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fa870004af0 tx=0x7fa870004a40 comp rx=0 tx=0).stop 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 -- 172.21.15.3:0/2035535931 shutdown_connections 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3f78b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f76260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 --2- 172.21.15.3:0/2035535931 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f6d490 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.565+0000 7fa880978740 1 -- 172.21.15.3:0/2035535931 >> 172.21.15.3:0/2035535931 conn(0x5559b3f679d0 msgr2=0x5559b3f67db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 -- 172.21.15.3:0/2035535931 shutdown_connections 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 -- 172.21.15.3:0/2035535931 wait complete. 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 -- start start 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 Processor -- start 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f737c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f73d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f74240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3e79b90 con 0x5559b3f767a0 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3ba8250 con 0x5559b3f6d9d0 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3e5a200 con 0x5559b3f6d0b0 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.566+0000 7fa880978740 10 client.0 _ll_register_callbacks cb 0x5559b3e905d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f74240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f74240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55964/0 (socket says 172.21.15.3:55964) 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa8777fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 -- 172.21.15.3:0/3112111817 learned_addr learned my addr 172.21.15.3:0/3112111817 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa876ffd640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f73d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 -- 172.21.15.3:0/3112111817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 msgr2=0x5559b3f737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f737c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 -- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 msgr2=0x5559b3f73d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f73d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 -- 172.21.15.3:0/3112111817 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa87810e8f0 con 0x5559b3f767a0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa876ffd640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f73d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa8777fe640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa877fff640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f74240 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa870045040 tx=0x7fa870032960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=511c68877bb4e777 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa856ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa856ffd640 1 -- 172.21.15.3:0/3112111817 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870034ea0 con 0x5559b3f767a0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa856ffd640 1 -- 172.21.15.3:0/3112111817 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa870041440 con 0x5559b3f767a0 2024-02-09T22:56:45.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559b3bd4600 con 0x5559b3f767a0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.567+0000 7fa856ffd640 1 -- 172.21.15.3:0/3112111817 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa870005cf0 con 0x5559b3f767a0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 10 client.17538 fetch_fsmap learned FSMap version 38 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5559b3ed2d90 con 0x5559b3f767a0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa856ffd640 1 -- 172.21.15.3:0/3112111817 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5559b3bd4600 con 0x5559b3f767a0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa856ffd640 1 -- 172.21.15.3:0/3112111817 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa87003e040 con 0x5559b3f767a0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 10 client.17538 fetch_fsmap finished waiting for FSMap version 38 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 10 client.17538 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 20 client.17538 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3f88b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 4 client.17538 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559b3c251b0 con 0x5559b3f866e0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.568+0000 7fa8777fe640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3f88b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.569+0000 7fa856ffd640 10 client.17538 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.569+0000 7fa8777fe640 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3f88b00 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5be3cd11a1cae20f server_cookie=fc7744c582aec6b1 in_seq=0 out_seq=0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.569+0000 7fa856ffd640 1 -- 172.21.15.3:0/3112111817 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5559b3c251b0 con 0x5559b3f866e0 2024-02-09T22:56:45.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.569+0000 7fa856ffd640 10 client.17538 handle_command_reply: tid=0 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.569+0000 7fa880978740 1 client.17538 shutdown 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 msgr2=0x5559b3f74240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f74240 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa870045040 tx=0x7fa870032960 comp rx=0 tx=0).stop 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 shutdown_connections 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3f88b00 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f73d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa856ffd640 0 client.17538 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f74240 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 --2- 172.21.15.3:0/3112111817 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 >> 172.21.15.3:0/3112111817 conn(0x5559b3f679d0 msgr2=0x5559b3f69720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 shutdown_connections 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 1 -- 172.21.15.3:0/3112111817 wait complete. 2024-02-09T22:56:45.584 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.570+0000 7fa880978740 20 client.17538 trim_cache size 0 max 16384 2024-02-09T22:56:45.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 Processor -- start 2024-02-09T22:56:45.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 -- start start 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f75ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f74060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f6ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3f8dcb0 con 0x5559b3f767a0 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3f71d10 con 0x5559b3f6d9d0 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa880978740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3f82340 con 0x5559b3f6d0b0 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa877fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f6ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa877fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f6ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55980/0 (socket says 172.21.15.3:55980) 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.583+0000 7fa877fff640 1 -- 172.21.15.3:0/1888919620 learned_addr learned my addr 172.21.15.3:0/1888919620 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f74060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa877fff640 1 -- 172.21.15.3:0/1888919620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 msgr2=0x5559b3f75ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa877fff640 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f75ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa877fff640 1 -- 172.21.15.3:0/1888919620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 msgr2=0x5559b3f74060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa877fff640 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f74060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa877fff640 1 -- 172.21.15.3:0/1888919620 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559b3e6f820 con 0x5559b3f767a0 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa877fff640 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f6ea80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa87002c130 tx=0x7fa870009880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de32513b7a20621b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa85d7fa640 1 -- 172.21.15.3:0/1888919620 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa870005bf0 con 0x5559b3f767a0 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa85d7fa640 1 -- 172.21.15.3:0/1888919620 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa870043e50 con 0x5559b3f767a0 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.584+0000 7fa85d7fa640 1 -- 172.21.15.3:0/1888919620 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87003d7c0 con 0x5559b3f767a0 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- 172.21.15.3:0/1888919620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 msgr2=0x5559b3f6ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f6ea80 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa87002c130 tx=0x7fa870009880 comp rx=0 tx=0).stop 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- 172.21.15.3:0/1888919620 shutdown_connections 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f6d9d0 0x5559b3f74060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f767a0 0x5559b3f6ea80 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- 172.21.15.3:0/1888919620 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d0b0 0x5559b3f75ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- 172.21.15.3:0/1888919620 >> 172.21.15.3:0/1888919620 conn(0x5559b3f679d0 msgr2=0x5559b3f6df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- 172.21.15.3:0/1888919620 shutdown_connections 2024-02-09T22:56:45.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- 172.21.15.3:0/1888919620 wait complete. 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- start start 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 Processor -- start 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f7b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f7ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3fdbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3fdc0e0 con 0x5559b3f6d0b0 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3fdc220 con 0x5559b3f767a0 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.585+0000 7fa880978740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559b3fdc360 con 0x5559b3f6d9d0 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa880978740 10 client.0 _ll_register_callbacks cb 0x7fa86c096330 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f7b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f7b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55990/0 (socket says 172.21.15.3:55990) 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 -- 172.21.15.3:0/1726155443 learned_addr learned my addr 172.21.15.3:0/1726155443 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f7ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.593 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa877fff640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3fdbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 -- 172.21.15.3:0/1726155443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 msgr2=0x5559b3f7ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f7ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 -- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 msgr2=0x5559b3fdbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3fdbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 -- 172.21.15.3:0/1726155443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559b3ed2d90 con 0x5559b3f6d0b0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f7ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa8777fe640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f7b550 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa868004430 tx=0x7fa8680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19320038de9196f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa856ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa856ffd640 1 -- 172.21.15.3:0/1726155443 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa868025070 con 0x5559b3f6d0b0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa856ffd640 1 -- 172.21.15.3:0/1726155443 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa86802b030 con 0x5559b3f6d0b0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5559b3efaee0 con 0x5559b3f6d0b0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa856ffd640 1 -- 172.21.15.3:0/1726155443 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa868020110 con 0x5559b3f6d0b0 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa880978740 10 client.17544 fetch_fsmap learned FSMap version 38 2024-02-09T22:56:45.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.586+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x5559b3e956f0 con 0x5559b3f6d0b0 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa856ffd640 1 -- 172.21.15.3:0/1726155443 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5559b3efaee0 con 0x5559b3f6d0b0 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa856ffd640 1 -- 172.21.15.3:0/1726155443 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7898+0+0 (secure 0 0 0) 0x7fa86801a040 con 0x5559b3f6d0b0 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa880978740 10 client.17544 fetch_fsmap finished waiting for FSMap version 38 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa880978740 10 client.17544 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa880978740 20 client.17544 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa880978740 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3fdfca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa880978740 4 client.17544 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5559b3c11550 con 0x5559b3f866e0 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3fdfca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa876ffd640 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3fdfca0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24e9a51a3b06b0ad server_cookie=2f326f1bf37025ce in_seq=0 out_seq=0 2024-02-09T22:56:45.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.587+0000 7fa856ffd640 10 client.17544 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.614+0000 7fa856ffd640 1 -- 172.21.15.3:0/1726155443 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5559b3c11550 con 0x5559b3f866e0 2024-02-09T22:56:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.614+0000 7fa856ffd640 10 client.17544 handle_command_reply: tid=0 2024-02-09T22:56:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.614+0000 7fa880978740 1 client.17544 shutdown 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 msgr2=0x5559b3f7b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f7b550 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7fa868004430 tx=0x7fa8680046c0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 shutdown_connections 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559b3f866e0 0x5559b3fdfca0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559b3f767a0 0x5559b3fdbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559b3f6d9d0 0x5559b3f7ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa856ffd640 0 client.17544 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 --2- 172.21.15.3:0/1726155443 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559b3f6d0b0 0x5559b3f7b550 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 >> 172.21.15.3:0/1726155443 conn(0x5559b3f679d0 msgr2=0x5559b3f6ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 shutdown_connections 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 1 -- 172.21.15.3:0/1726155443 wait complete. 2024-02-09T22:56:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa880978740 20 client.17544 trim_cache size 0 max 16384 2024-02-09T22:56:45.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:56:45.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:56:45.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:56:45.620 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8600a2220 msgr2=0x7fa8600a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa87dda1640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8600a2220 0x7fa8600a46e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa868005fd0 tx=0x7fa86800b390 comp rx=0 tx=0).stop 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 msgr2=0x7fa87819db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.615+0000 7fa87dda1640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87819db20 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fa86c00d770 tx=0x7fa86c00dc30 comp rx=0 tx=0).stop 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 shutdown_connections 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa8600a2220 0x7fa8600a46e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa8781976a0 0x7fa878197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa878106510 0x7fa87819db20 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 --2- 172.21.15.3:0/1763598369 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa878068d40 0x7fa87819d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 >> 172.21.15.3:0/1763598369 conn(0x7fa878077220 msgr2=0x7fa8780fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:45.621 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 shutdown_connections 2024-02-09T22:56:45.622 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:45.616+0000 7fa87dda1640 1 -- 172.21.15.3:0/1763598369 wait complete. 2024-02-09T22:56:45.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156883 2.65 MB/sec execute 159 sec latency 2008.717 ms 2024-02-09T22:56:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:46 smithi003 ceph-mon[32678]: pgmap v168: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 902 KiB/s wr, 109 op/s 2024-02-09T22:56:46.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156883 2.63 MB/sec execute 160 sec latency 1494.551 ms 2024-02-09T22:56:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:46 smithi064 ceph-mon[36638]: pgmap v168: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 902 KiB/s wr, 109 op/s 2024-02-09T22:56:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:46 smithi089 ceph-mon[36665]: pgmap v168: 97 pgs: 97 active+clean; 62 MiB data, 907 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 902 KiB/s wr, 109 op/s 2024-02-09T22:56:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:47 smithi003 ceph-mon[32678]: pgmap v169: 97 pgs: 97 active+clean; 66 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-02-09T22:56:47.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 156883 2.61 MB/sec execute 161 sec latency 2494.585 ms 2024-02-09T22:56:47.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:47 smithi064 ceph-mon[36638]: pgmap v169: 97 pgs: 97 active+clean; 66 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-02-09T22:56:47.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:47 smithi089 ceph-mon[36665]: pgmap v169: 97 pgs: 97 active+clean; 66 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 156 op/s 2024-02-09T22:56:48.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 157785 2.63 MB/sec execute 162 sec latency 2899.168 ms 2024-02-09T22:56:49.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 157785 2.61 MB/sec execute 163 sec latency 1125.958 ms 2024-02-09T22:56:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:50 smithi089 ceph-mon[36665]: pgmap v170: 97 pgs: 97 active+clean; 66 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 736 KiB/s wr, 102 op/s 2024-02-09T22:56:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:50 smithi003 ceph-mon[32678]: pgmap v170: 97 pgs: 97 active+clean; 66 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 736 KiB/s wr, 102 op/s 2024-02-09T22:56:50.640 INFO:tasks.workunit.client.0.smithi003.stdout: 1 158429 2.61 MB/sec execute 164 sec latency 1943.980 ms 2024-02-09T22:56:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:50 smithi064 ceph-mon[36638]: pgmap v170: 97 pgs: 97 active+clean; 66 MiB data, 923 MiB used, 1.0 TiB / 1.0 TiB avail; 907 KiB/s rd, 736 KiB/s wr, 102 op/s 2024-02-09T22:56:51.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 160087 2.66 MB/sec execute 165 sec latency 13.841 ms 2024-02-09T22:56:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:52 smithi089 ceph-mon[36665]: pgmap v171: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:56:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:52 smithi003 ceph-mon[32678]: pgmap v171: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:56:52.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 160306 2.65 MB/sec execute 166 sec latency 817.572 ms 2024-02-09T22:56:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:52 smithi064 ceph-mon[36638]: pgmap v171: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:56:53.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 161331 2.66 MB/sec execute 167 sec latency 1223.691 ms 2024-02-09T22:56:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:54 smithi089 ceph-mon[36665]: pgmap v172: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 785 KiB/s wr, 84 op/s 2024-02-09T22:56:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:54 smithi003 ceph-mon[32678]: pgmap v172: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 785 KiB/s wr, 84 op/s 2024-02-09T22:56:54.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 161331 2.65 MB/sec execute 168 sec latency 1031.524 ms 2024-02-09T22:56:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:54 smithi064 ceph-mon[36638]: pgmap v172: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 785 KiB/s wr, 84 op/s 2024-02-09T22:56:55.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 162808 2.68 MB/sec execute 169 sec latency 1482.222 ms 2024-02-09T22:56:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:56 smithi089 ceph-mon[36665]: pgmap v173: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 785 KiB/s wr, 84 op/s 2024-02-09T22:56:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:56 smithi003 ceph-mon[32678]: pgmap v173: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 785 KiB/s wr, 84 op/s 2024-02-09T22:56:56.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 163974 2.70 MB/sec execute 170 sec latency 141.022 ms 2024-02-09T22:56:56.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:56 smithi064 ceph-mon[36638]: pgmap v173: 97 pgs: 97 active+clean; 62 MiB data, 920 MiB used, 1.0 TiB / 1.0 TiB avail; 623 KiB/s rd, 785 KiB/s wr, 84 op/s 2024-02-09T22:56:57.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 163974 2.68 MB/sec execute 171 sec latency 1141.111 ms 2024-02-09T22:56:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:56:58 smithi089 ceph-mon[36665]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-02-09T22:56:58.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:56:58 smithi064 ceph-mon[36638]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-02-09T22:56:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:56:58 smithi003 ceph-mon[32678]: pgmap v174: 97 pgs: 97 active+clean; 63 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 144 op/s 2024-02-09T22:56:58.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 164875 2.70 MB/sec execute 172 sec latency 1546.039 ms 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 -- 172.21.15.3:0/1361100648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c100d70 msgr2=0x7f2f3c104d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 --2- 172.21.15.3:0/1361100648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c100d70 0x7f2f3c104d60 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c009930 tx=0x7f2f2c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 -- 172.21.15.3:0/1361100648 shutdown_connections 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 --2- 172.21.15.3:0/1361100648 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 0x7f2f3c10d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 --2- 172.21.15.3:0/1361100648 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c100d70 0x7f2f3c104d60 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 --2- 172.21.15.3:0/1361100648 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 0x7f2f3c1007a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 -- 172.21.15.3:0/1361100648 >> 172.21.15.3:0/1361100648 conn(0x7f2f3c077340 msgr2=0x7f2f3c077750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:58.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 -- 172.21.15.3:0/1361100648 shutdown_connections 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.703+0000 7f2f4336c640 1 -- 172.21.15.3:0/1361100648 wait complete. 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 Processor -- start 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 -- start start 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 0x7f2f3c19d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 0x7f2f3c19db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 0x7f2f3c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c1981a0 con 0x7f2f3c1976d0 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c198310 con 0x7f2f3c1003c0 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f4336c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f3c1984d0 con 0x7f2f3c105440 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f41909640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 0x7f2f3c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f41909640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 0x7f2f3c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36194/0 (socket says 172.21.15.3:36194) 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f41909640 1 -- 172.21.15.3:0/2348131639 learned_addr learned my addr 172.21.15.3:0/2348131639 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.704+0000 7f2f40907640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 0x7f2f3c19db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41108640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 0x7f2f3c19d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41909640 1 -- 172.21.15.3:0/2348131639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 msgr2=0x7f2f3c19db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41909640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 0x7f2f3c19db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41909640 1 -- 172.21.15.3:0/2348131639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 msgr2=0x7f2f3c19d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41909640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 0x7f2f3c19d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41909640 1 -- 172.21.15.3:0/2348131639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f2c009560 con 0x7f2f3c1976d0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41108640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 0x7f2f3c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f40907640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 0x7f2f3c19db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f41909640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 0x7f2f3c197b50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400cce0 tx=0x7f2f3400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f327fc640 1 -- 172.21.15.3:0/2348131639 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3400f450 con 0x7f2f3c1976d0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f327fc640 1 -- 172.21.15.3:0/2348131639 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f3400fa90 con 0x7f2f3c1976d0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f3c1a4470 con 0x7f2f3c1976d0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f327fc640 1 -- 172.21.15.3:0/2348131639 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f34018b90 con 0x7f2f3c1976d0 2024-02-09T22:56:58.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.705+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c1a4950 con 0x7f2f3c1976d0 2024-02-09T22:56:58.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.707+0000 7f2f327fc640 1 -- 172.21.15.3:0/2348131639 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f2f3400fc00 con 0x7f2f3c1976d0 2024-02-09T22:56:58.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 Processor -- start 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 -- start start 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515bfd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f327fc640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2f280a22a0 0x7f2f280a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c06260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c08b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515b6ba00 con 0x55b515bfd0b0 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515b68f30 con 0x55b515c067a0 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f4601e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515b64710 con 0x55b515bfd9d0 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515bfd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f41108640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2f280a22a0 0x7f2f280a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515bfd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36204/0 (socket says 172.21.15.3:36204) 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 -- 172.21.15.3:0/3713266512 learned_addr learned my addr 172.21.15.3:0/3713266512 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:58.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f327fc640 1 -- 172.21.15.3:0/2348131639 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2f340ce830 con 0x7f2f3c1976d0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f41909640 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c08b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c06260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 -- 172.21.15.3:0/3713266512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 msgr2=0x55b515c06260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c06260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 -- 172.21.15.3:0/3713266512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 msgr2=0x55b515c08b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c08b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f40907640 1 -- 172.21.15.3:0/3713266512 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b515b293b0 con 0x55b515bfd0b0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f41909640 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c08b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.708+0000 7f2f41108640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2f280a22a0 0x7f2f280a4760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f380058c0 tx=0x7f2f38005d30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f40907640 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515bfd490 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c005940 tx=0x7f2f2c0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94082a1d77eba86a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f277fe640 1 -- 172.21.15.3:0/3713266512 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c02c9e0 con 0x55b515bfd0b0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f277fe640 1 -- 172.21.15.3:0/3713266512 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f2c004500 con 0x55b515bfd0b0 2024-02-09T22:56:58.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f277fe640 1 -- 172.21.15.3:0/3713266512 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c005600 con 0x55b515bfd0b0 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 -- 172.21.15.3:0/3713266512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 msgr2=0x55b515bfd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515bfd490 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c005940 tx=0x7f2f2c0058b0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 -- 172.21.15.3:0/3713266512 shutdown_connections 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c08b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c06260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3713266512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515bfd490 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 -- 172.21.15.3:0/3713266512 >> 172.21.15.3:0/3713266512 conn(0x55b515bf79d0 msgr2=0x55b515bf7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 -- 172.21.15.3:0/3713266512 shutdown_connections 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.709+0000 7f2f4601e740 1 -- 172.21.15.3:0/3713266512 wait complete. 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 -- start start 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 Processor -- start 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515c037f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c03d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c04270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515b09b90 con 0x55b515bfd0b0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515838250 con 0x55b515c067a0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515aea200 con 0x55b515bfd9d0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f4601e740 10 client.0 _ll_register_callbacks cb 0x55b515b205d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f41108640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515c037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f41108640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515c037f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36214/0 (socket says 172.21.15.3:36214) 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.710+0000 7f2f41108640 1 -- 172.21.15.3:0/3811911899 learned_addr learned my addr 172.21.15.3:0/3811911899 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f40907640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c03d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41909640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c04270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41108640 1 -- 172.21.15.3:0/3811911899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 msgr2=0x55b515c03d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41108640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c03d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41108640 1 -- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 msgr2=0x55b515c04270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41108640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c04270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41108640 1 -- 172.21.15.3:0/3811911899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f3c10d820 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f40907640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c03d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41909640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c04270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f41108640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515c037f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800fa00 tx=0x7f2f380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d02c10a0b94acc5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f24ff9640 1 -- 172.21.15.3:0/3811911899 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f3800ec40 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f24ff9640 1 -- 172.21.15.3:0/3811911899 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2f3800eda0 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b515864600 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f24ff9640 1 -- 172.21.15.3:0/3811911899 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f3801c450 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f4601e740 10 client.17556 fetch_fsmap learned FSMap version 38 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55b515b62d90 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f24ff9640 1 -- 172.21.15.3:0/3811911899 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b515864600 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f24ff9640 1 -- 172.21.15.3:0/3811911899 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f2f3800b050 con 0x55b515bfd0b0 2024-02-09T22:56:58.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f4601e740 10 client.17556 fetch_fsmap finished waiting for FSMap version 38 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.711+0000 7f2f4601e740 10 client.17556 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f4601e740 20 client.17556 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c16af0 0x55b515c18f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f4601e740 4 client.17556 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b5158b51b0 con 0x55b515c16af0 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f40907640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c16af0 0x55b515c18f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f24ff9640 10 client.17556 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.712+0000 7f2f40907640 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c16af0 0x55b515c18f10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64ab6d21d2675fb0 server_cookie=55cc38bece1382d1 in_seq=0 out_seq=0 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f24ff9640 1 -- 172.21.15.3:0/3811911899 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b5158b51b0 con 0x55b515c16af0 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f24ff9640 10 client.17556 handle_command_reply: tid=0 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 client.17556 shutdown 2024-02-09T22:56:58.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 msgr2=0x55b515c037f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515c037f0 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f2f3800fa00 tx=0x7f2f380046c0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 shutdown_connections 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c16af0 0x55b515c18f10 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c04270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd9d0 0x55b515c03d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f24ff9640 0 client.17556 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 --2- 172.21.15.3:0/3811911899 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd0b0 0x55b515c037f0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 >> 172.21.15.3:0/3811911899 conn(0x55b515bf79d0 msgr2=0x55b515bf9720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 shutdown_connections 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 1 -- 172.21.15.3:0/3811911899 wait complete. 2024-02-09T22:56:58.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.713+0000 7f2f4601e740 20 client.17556 trim_cache size 0 max 16384 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 Processor -- start 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 -- start start 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c05f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c04090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c020c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515c08f00 con 0x55b515bfd9d0 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515c1e100 con 0x55b515c067a0 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.726+0000 7f2f4601e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515c01d40 con 0x55b515bfd0b0 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c04090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c04090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36226/0 (socket says 172.21.15.3:36226) 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 -- 172.21.15.3:0/2767883356 learned_addr learned my addr 172.21.15.3:0/2767883356 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:58.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f41108640 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c05f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f41909640 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c020c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 -- 172.21.15.3:0/2767883356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 msgr2=0x55b515c05f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c05f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 -- 172.21.15.3:0/2767883356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 msgr2=0x55b515c020c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c020c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 -- 172.21.15.3:0/2767883356 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b515aff820 con 0x55b515bfd9d0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f41909640 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f40907640 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c04090 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c005210 tx=0x7f2f2c0052e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75c0d016b439acae server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f26ffd640 1 -- 172.21.15.3:0/2767883356 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c005550 con 0x55b515bfd9d0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f26ffd640 1 -- 172.21.15.3:0/2767883356 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f2c02c7e0 con 0x55b515bfd9d0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.727+0000 7f2f26ffd640 1 -- 172.21.15.3:0/2767883356 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c051710 con 0x55b515bfd9d0 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- 172.21.15.3:0/2767883356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 msgr2=0x55b515c04090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c04090 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c005210 tx=0x7f2f2c0052e0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- 172.21.15.3:0/2767883356 shutdown_connections 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c020c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c04090 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- 172.21.15.3:0/2767883356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c05f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- 172.21.15.3:0/2767883356 >> 172.21.15.3:0/2767883356 conn(0x55b515bf79d0 msgr2=0x55b515bfeb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- 172.21.15.3:0/2767883356 shutdown_connections 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- 172.21.15.3:0/2767883356 wait complete. 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- start start 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 Processor -- start 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c0b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c0b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c0bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515c6c110 con 0x55b515bfd9d0 2024-02-09T22:56:58.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515c6c280 con 0x55b515c067a0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b515c6c440 con 0x55b515bfd0b0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.728+0000 7f2f4601e740 10 client.0 _ll_register_callbacks cb 0x7f2f38013070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c0b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c0b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36230/0 (socket says 172.21.15.3:36230) 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 -- 172.21.15.3:0/1020093223 learned_addr learned my addr 172.21.15.3:0/1020093223 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f41108640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c0b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f41909640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c0bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 -- 172.21.15.3:0/1020093223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 msgr2=0x55b515c0b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c0b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 -- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 msgr2=0x55b515c0bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c0bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 -- 172.21.15.3:0/1020093223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b515b293b0 con 0x55b515bfd9d0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f40907640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c0b5c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c02c170 tx=0x7f2f2c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8725221f11341021 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:56:58.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f24ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f24ff9640 1 -- 172.21.15.3:0/1020093223 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c02cb50 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b515b8aee0 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f24ff9640 1 -- 172.21.15.3:0/1020093223 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2f2c005560 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f24ff9640 1 -- 172.21.15.3:0/1020093223 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f2c02ccb0 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f4601e740 10 client.17562 fetch_fsmap learned FSMap version 38 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.729+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x55b515b256f0 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f24ff9640 1 -- 172.21.15.3:0/1020093223 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b515b8aee0 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f24ff9640 1 -- 172.21.15.3:0/1020093223 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 38) v1 ==== 7898+0+0 (secure 0 0 0) 0x7f2f2c047020 con 0x55b515bfd9d0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f4601e740 10 client.17562 fetch_fsmap finished waiting for FSMap version 38 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f4601e740 10 client.17562 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f4601e740 20 client.17562 populate_metadata read hostname 'smithi003' 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f4601e740 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c6d7c0 0x55b515c6fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f4601e740 4 client.17562 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b5158a1550 con 0x55b515c6d7c0 2024-02-09T22:56:58.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f41108640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c6d7c0 0x55b515c6fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:56:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f24ff9640 10 client.17562 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:58.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.730+0000 7f2f41108640 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c6d7c0 0x55b515c6fbe0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca280d34ee59502c server_cookie=63090ffc43ce40ac in_seq=0 out_seq=0 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f24ff9640 1 -- 172.21.15.3:0/1020093223 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b5158a1550 con 0x55b515c6d7c0 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f24ff9640 10 client.17562 handle_command_reply: tid=0 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 client.17562 shutdown 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 msgr2=0x55b515c0b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c0b5c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c02c170 tx=0x7f2f2c005850 comp rx=0 tx=0).stop 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 shutdown_connections 2024-02-09T22:56:58.799 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b515c6d7c0 0x55b515c6fbe0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b515c067a0 0x55b515c0bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b515bfd9d0 0x55b515c0b5c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f24ff9640 0 client.17562 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 --2- 172.21.15.3:0/1020093223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b515bfd0b0 0x55b515c0b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 >> 172.21.15.3:0/1020093223 conn(0x55b515bf79d0 msgr2=0x55b515c02080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 shutdown_connections 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 1 -- 172.21.15.3:0/1020093223 wait complete. 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.795+0000 7f2f4601e740 20 client.17562 trim_cache size 0 max 16384 2024-02-09T22:56:58.800 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:56:58.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:56:58.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:56:58.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:56:58.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2f280a22a0 msgr2=0x7f2f280a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.801 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2f280a22a0 0x7f2f280a4760 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2f380058c0 tx=0x7f2f38005d30 comp rx=0 tx=0).stop 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 msgr2=0x7f2f3c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 0x7f2f3c197b50 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2f3400cce0 tx=0x7f2f3400a620 comp rx=0 tx=0).stop 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 shutdown_connections 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f2f280a22a0 0x7f2f280a4760 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2f3c1976d0 0x7f2f3c197b50 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2f3c105440 0x7f2f3c19db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 --2- 172.21.15.3:0/2348131639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2f3c1003c0 0x7f2f3c19d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 >> 172.21.15.3:0/2348131639 conn(0x7f2f3c077340 msgr2=0x7f2f3c104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 shutdown_connections 2024-02-09T22:56:58.802 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:56:58.796+0000 7f2f4336c640 1 -- 172.21.15.3:0/2348131639 wait complete. 2024-02-09T22:56:59.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 164875 2.68 MB/sec execute 173 sec latency 1094.134 ms 2024-02-09T22:57:00.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:00 smithi089 ceph-mon[36665]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 700 KiB/s wr, 97 op/s 2024-02-09T22:57:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:00 smithi003 ceph-mon[32678]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 700 KiB/s wr, 97 op/s 2024-02-09T22:57:00.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 165447 2.67 MB/sec execute 174 sec latency 1911.019 ms 2024-02-09T22:57:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:00 smithi064 ceph-mon[36638]: pgmap v175: 97 pgs: 97 active+clean; 63 MiB data, 899 MiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 700 KiB/s wr, 97 op/s 2024-02-09T22:57:01.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 166963 2.73 MB/sec execute 175 sec latency 23.073 ms 2024-02-09T22:57:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:02 smithi089 ceph-mon[36665]: pgmap v176: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 970 KiB/s wr, 154 op/s 2024-02-09T22:57:02.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:02 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:57:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:02 smithi003 ceph-mon[32678]: pgmap v176: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 970 KiB/s wr, 154 op/s 2024-02-09T22:57:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:02 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:57:02.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 167455 2.72 MB/sec execute 176 sec latency 548.931 ms 2024-02-09T22:57:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:02 smithi064 ceph-mon[36638]: pgmap v176: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 970 KiB/s wr, 154 op/s 2024-02-09T22:57:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:02 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:57:02.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:57:02.737 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 -- 172.21.15.3:0/710086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c101d20 msgr2=0x7f146c102180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 --2- 172.21.15.3:0/710086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c101d20 0x7f146c102180 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f1454009930 tx=0x7f145402bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 -- 172.21.15.3:0/710086349 shutdown_connections 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 --2- 172.21.15.3:0/710086349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c1026c0 0x7f146c10d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 --2- 172.21.15.3:0/710086349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c101d20 0x7f146c102180 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 --2- 172.21.15.3:0/710086349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c100440 0x7f146c100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 -- 172.21.15.3:0/710086349 >> 172.21.15.3:0/710086349 conn(0x7f146c077310 msgr2=0x7f146c077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 -- 172.21.15.3:0/710086349 shutdown_connections 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.849+0000 7f1471840640 1 -- 172.21.15.3:0/710086349 wait complete. 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 Processor -- start 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 -- start start 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 0x7f146c19d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c101d20 0x7f146c19db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c1026c0 0x7f146c197710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c197ea0 con 0x7f146c100440 2024-02-09T22:57:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c198010 con 0x7f146c1026c0 2024-02-09T22:57:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f1471840640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146c1981d0 con 0x7f146c101d20 2024-02-09T22:57:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 0x7f146c19d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 0x7f146c19d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36252/0 (socket says 172.21.15.3:36252) 2024-02-09T22:57:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 -- 172.21.15.3:0/1646764526 learned_addr learned my addr 172.21.15.3:0/1646764526 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:02.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c1026c0 0x7f146c197710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f14627fc640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c101d20 0x7f146c19db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 -- 172.21.15.3:0/1646764526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c101d20 msgr2=0x7f146c19db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c101d20 0x7f146c19db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 -- 172.21.15.3:0/1646764526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c1026c0 msgr2=0x7f146c197710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c1026c0 0x7f146c197710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146affd640 1 -- 172.21.15.3:0/1646764526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1454009560 con 0x7f146c100440 2024-02-09T22:57:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.850+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c1026c0 0x7f146c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.851+0000 7f146affd640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 0x7f146c19d650 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c890 tx=0x7f145c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.851+0000 7f147083e640 1 -- 172.21.15.3:0/1646764526 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c004220 con 0x7f146c100440 2024-02-09T22:57:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.851+0000 7f147083e640 1 -- 172.21.15.3:0/1646764526 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f145c007e90 con 0x7f146c100440 2024-02-09T22:57:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.851+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f146c1984b0 con 0x7f146c100440 2024-02-09T22:57:02.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.851+0000 7f147083e640 1 -- 172.21.15.3:0/1646764526 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c004a00 con 0x7f146c100440 2024-02-09T22:57:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.851+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c1a45d0 con 0x7f146c100440 2024-02-09T22:57:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.853+0000 7f147083e640 1 -- 172.21.15.3:0/1646764526 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f145c0079b0 con 0x7f146c100440 2024-02-09T22:57:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.853+0000 7f147440e740 1 Processor -- start 2024-02-09T22:57:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 -- start start 2024-02-09T22:57:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a7220d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7221330 0x55e2a72298e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7229eb0 0x55e2a722c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147083e640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14580a2230 0x7f14580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a71dbd00 con 0x55e2a7220980 2024-02-09T22:57:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a71902f0 con 0x55e2a7229eb0 2024-02-09T22:57:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147440e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a718d820 con 0x55e2a7221330 2024-02-09T22:57:02.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a7220d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a7220d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36260/0 (socket says 172.21.15.3:36260) 2024-02-09T22:57:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 -- 172.21.15.3:0/1712727426 learned_addr learned my addr 172.21.15.3:0/1712727426 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14580a2230 0x7f14580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f147083e640 1 -- 172.21.15.3:0/1646764526 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f145c0cde40 con 0x7f146c100440 2024-02-09T22:57:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146affd640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7221330 0x55e2a72298e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f14627fc640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7229eb0 0x55e2a722c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 -- 172.21.15.3:0/1712727426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7221330 msgr2=0x55e2a72298e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7221330 0x55e2a72298e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 -- 172.21.15.3:0/1712727426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7229eb0 msgr2=0x55e2a722c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7229eb0 0x55e2a722c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146b7fe640 1 -- 172.21.15.3:0/1712727426 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2a714aff0 con 0x55e2a7220980 2024-02-09T22:57:02.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f146affd640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7221330 0x55e2a72298e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.854+0000 7f14627fc640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7229eb0 0x55e2a722c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a7220d60 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1464013e90 tx=0x7f14640107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33be9ac47e4c02ad server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14580a2230 0x7f14580a46f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f146400afd0 tx=0x7f146401b000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f1461ffb640 1 -- 172.21.15.3:0/1712727426 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146400bea0 con 0x55e2a7220980 2024-02-09T22:57:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f1461ffb640 1 -- 172.21.15.3:0/1712727426 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1464019040 con 0x55e2a7220980 2024-02-09T22:57:02.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f1461ffb640 1 -- 172.21.15.3:0/1712727426 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464003bf0 con 0x55e2a7220980 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 -- 172.21.15.3:0/1712727426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 msgr2=0x55e2a7220d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a7220d60 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f1464013e90 tx=0x7f14640107a0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 -- 172.21.15.3:0/1712727426 shutdown_connections 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7229eb0 0x55e2a722c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7221330 0x55e2a72298e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 --2- 172.21.15.3:0/1712727426 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a7220d60 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.855+0000 7f147440e740 1 -- 172.21.15.3:0/1712727426 >> 172.21.15.3:0/1712727426 conn(0x55e2a721b050 msgr2=0x55e2a721b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 -- 172.21.15.3:0/1712727426 shutdown_connections 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 -- 172.21.15.3:0/1712727426 wait complete. 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 -- start start 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 Processor -- start 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7220980 0x55e2a7226e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 0x55e2a72273c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7227900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a7189000 con 0x55e2a7221330 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a6e5c430 con 0x55e2a7220980 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a6e18e00 con 0x55e2a7229eb0 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f147440e740 10 client.0 _ll_register_callbacks cb 0x55e2a717d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f14627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 0x55e2a72273c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f14627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 0x55e2a72273c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36264/0 (socket says 172.21.15.3:36264) 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.856+0000 7f14627fc640 1 -- 172.21.15.3:0/2193371084 learned_addr learned my addr 172.21.15.3:0/2193371084 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f146b7fe640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7227900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f146affd640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7220980 0x55e2a7226e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f14627fc640 1 -- 172.21.15.3:0/2193371084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 msgr2=0x55e2a7227900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f14627fc640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7227900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f14627fc640 1 -- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7220980 msgr2=0x55e2a7226e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f14627fc640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7220980 0x55e2a7226e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f14627fc640 1 -- 172.21.15.3:0/2193371084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1464013a40 con 0x55e2a7221330 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f146affd640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7220980 0x55e2a7226e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f14627fc640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 0x55e2a72273c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1454006010 tx=0x7f145402ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f872745a662239 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f144affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:02.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f144affd640 1 -- 172.21.15.3:0/2193371084 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145402cea0 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f144affd640 1 -- 172.21.15.3:0/2193371084 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1454004250 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e2a6e887e0 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f144affd640 1 -- 172.21.15.3:0/2193371084 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f145402cc80 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f147440e740 10 client.17574 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.857+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e2a7187680 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f144affd640 1 -- 172.21.15.3:0/2193371084 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e2a6e887e0 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f144affd640 1 -- 172.21.15.3:0/2193371084 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f145403c050 con 0x55e2a7221330 2024-02-09T22:57:02.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f147440e740 10 client.17574 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f147440e740 10 client.17574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f147440e740 20 client.17574 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f147440e740 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a723a3c0 0x55e2a723c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f147440e740 4 client.17574 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2a6ec5720 con 0x55e2a723a3c0 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f146affd640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a723a3c0 0x55e2a723c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f146affd640 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a723a3c0 0x55e2a723c7e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b1b7ec6e7d319b server_cookie=b32ff5655672a4d5 in_seq=0 out_seq=0 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.858+0000 7f144affd640 10 client.17574 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:02.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f144affd640 1 -- 172.21.15.3:0/2193371084 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e2a6ec5720 con 0x55e2a723a3c0 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f144affd640 10 client.17574 handle_command_reply: tid=0 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 client.17574 shutdown 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 msgr2=0x55e2a72273c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 0x55e2a72273c0 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f1454006010 tx=0x7f145402ca90 comp rx=0 tx=0).stop 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 shutdown_connections 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a723a3c0 0x55e2a723c7e0 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7227900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7221330 0x55e2a72273c0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f144affd640 0 client.17574 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 --2- 172.21.15.3:0/2193371084 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7220980 0x55e2a7226e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 >> 172.21.15.3:0/2193371084 conn(0x55e2a721b050 msgr2=0x55e2a721cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 shutdown_connections 2024-02-09T22:57:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 1 -- 172.21.15.3:0/2193371084 wait complete. 2024-02-09T22:57:02.884 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.859+0000 7f147440e740 20 client.17574 trim_cache size 0 max 16384 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.874+0000 7f147440e740 1 Processor -- start 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.874+0000 7f147440e740 1 -- start start 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f147440e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a72295b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f147440e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a72240e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f147440e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7224620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f147440e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a710dfd0 con 0x55e2a7220980 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f147440e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a7222ab0 con 0x55e2a7221330 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f147440e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a71f0ab0 con 0x55e2a7229eb0 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a72295b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a72295b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36270/0 (socket says 172.21.15.3:36270) 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 -- 172.21.15.3:0/2759536795 learned_addr learned my addr 172.21.15.3:0/2759536795 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:02.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f14627fc640 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a72240e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 -- 172.21.15.3:0/2759536795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 msgr2=0x55e2a7224620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7224620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 -- 172.21.15.3:0/2759536795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 msgr2=0x55e2a72240e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a72240e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 -- 172.21.15.3:0/2759536795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2a7123b30 con 0x55e2a7220980 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f14627fc640 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a72240e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f146affd640 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a72295b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f145c0b8fd0 tx=0x7f145c0ba040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7610d32ebd25728 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f14617fa640 1 -- 172.21.15.3:0/2759536795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c09a860 con 0x55e2a7220980 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f14617fa640 1 -- 172.21.15.3:0/2759536795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f145c09e9b0 con 0x55e2a7220980 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.875+0000 7f14617fa640 1 -- 172.21.15.3:0/2759536795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145c098c80 con 0x55e2a7220980 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 -- 172.21.15.3:0/2759536795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 msgr2=0x55e2a72295b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a72295b0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f145c0b8fd0 tx=0x7f145c0ba040 comp rx=0 tx=0).stop 2024-02-09T22:57:02.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 -- 172.21.15.3:0/2759536795 shutdown_connections 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a72240e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7229eb0 0x55e2a7224620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 --2- 172.21.15.3:0/2759536795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7220980 0x55e2a72295b0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 -- 172.21.15.3:0/2759536795 >> 172.21.15.3:0/2759536795 conn(0x55e2a721b050 msgr2=0x55e2a7241140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 -- 172.21.15.3:0/2759536795 shutdown_connections 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 -- 172.21.15.3:0/2759536795 wait complete. 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.876+0000 7f147440e740 1 -- start start 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 Processor -- start 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7220980 0x55e2a728b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a728b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 0x55e2a7230880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a7230e40 con 0x55e2a7229eb0 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a7230fb0 con 0x55e2a7221330 2024-02-09T22:57:02.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2a7231170 con 0x55e2a7220980 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f147440e740 10 client.0 _ll_register_callbacks cb 0x7f146c069730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 0x55e2a7230880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 0x55e2a7230880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36274/0 (socket says 172.21.15.3:36274) 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 -- 172.21.15.3:0/1253730414 learned_addr learned my addr 172.21.15.3:0/1253730414 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f14627fc640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a728b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146affd640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7220980 0x55e2a728b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 -- 172.21.15.3:0/1253730414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7220980 msgr2=0x55e2a728b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7220980 0x55e2a728b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 -- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 msgr2=0x55e2a728b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a728b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 -- 172.21.15.3:0/1253730414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f146c1a45d0 con 0x55e2a7229eb0 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f146b7fe640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 0x55e2a7230880 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1464010d20 tx=0x7f1464013d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395aa87b786ca454 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:02.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f144affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.877+0000 7f144affd640 1 -- 172.21.15.3:0/1253730414 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1464013250 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1464019040 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f144affd640 1 -- 172.21.15.3:0/1253730414 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14640133b0 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f144affd640 1 -- 172.21.15.3:0/1253730414 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1464004a00 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 10 client.17580 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e2a7149850 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f144affd640 1 -- 172.21.15.3:0/1253730414 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1464019040 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f144affd640 1 -- 172.21.15.3:0/1253730414 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f1464023070 con 0x55e2a7229eb0 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 10 client.17580 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 10 client.17580 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 20 client.17580 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a7293360 0x55e2a7295780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 4 client.17580 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:57:02.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e2a6f14890 con 0x55e2a7293360 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.878+0000 7f146affd640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a7293360 0x55e2a7295780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f146affd640 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a7293360 0x55e2a7295780 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=878cc74eb8b11ec6 server_cookie=a2598101bae5c2a7 in_seq=0 out_seq=0 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f144affd640 10 client.17580 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f144affd640 1 -- 172.21.15.3:0/1253730414 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e2a6f14890 con 0x55e2a7293360 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f144affd640 10 client.17580 handle_command_reply: tid=0 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 client.17580 shutdown 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 msgr2=0x55e2a7230880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 0x55e2a7230880 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f1464010d20 tx=0x7f1464013d80 comp rx=0 tx=0).stop 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 shutdown_connections 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2a7293360 0x55e2a7295780 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2a7221330 0x55e2a728b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2a7229eb0 0x55e2a7230880 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f144affd640 0 client.17580 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:02.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 --2- 172.21.15.3:0/1253730414 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2a7220980 0x55e2a728b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 >> 172.21.15.3:0/1253730414 conn(0x55e2a721b050 msgr2=0x55e2a7226e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 shutdown_connections 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 1 -- 172.21.15.3:0/1253730414 wait complete. 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.879+0000 7f147440e740 20 client.17580 trim_cache size 0 max 16384 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:57:02.897 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14580a2230 msgr2=0x7f14580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14580a2230 0x7f14580a46f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f146400afd0 tx=0x7f146401b000 comp rx=0 tx=0).stop 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 msgr2=0x7f146c19d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 0x7f146c19d650 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f145c00c890 tx=0x7f145c00cd50 comp rx=0 tx=0).stop 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 shutdown_connections 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f14580a2230 0x7f14580a46f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f146c1026c0 0x7f146c197710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f146c101d20 0x7f146c19db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 --2- 172.21.15.3:0/1646764526 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f146c100440 0x7f146c19d650 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:02.899 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 >> 172.21.15.3:0/1646764526 conn(0x7f146c077310 msgr2=0x7f146c103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 shutdown_connections 2024-02-09T22:57:02.900 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:02.880+0000 7f1471840640 1 -- 172.21.15.3:0/1646764526 wait complete. 2024-02-09T22:57:02.900 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:57:02.900 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:57:03.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 -- 172.21.15.3:0/823794121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100d00 msgr2=0x7f1b20104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 --2- 172.21.15.3:0/823794121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100d00 0x7f1b20104d40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f1b08009930 tx=0x7f1b0802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:03.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 -- 172.21.15.3:0/823794121 shutdown_connections 2024-02-09T22:57:03.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 --2- 172.21.15.3:0/823794121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20105420 0x7f1b2010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 --2- 172.21.15.3:0/823794121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100d00 0x7f1b20104d40 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 --2- 172.21.15.3:0/823794121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b20100350 0x7f1b20100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 -- 172.21.15.3:0/823794121 >> 172.21.15.3:0/823794121 conn(0x7f1b200772b0 msgr2=0x7f1b200776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 -- 172.21.15.3:0/823794121 shutdown_connections 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 -- 172.21.15.3:0/823794121 wait complete. 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.000+0000 7f1b2551e640 1 Processor -- start 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 -- start start 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 0x7f1b20101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b20105420 0x7f1b20102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20102860 0x7f1b201a5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b201a6010 con 0x7f1b20100350 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b201a6180 con 0x7f1b20105420 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b2551e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b201a6340 con 0x7f1b20102860 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 0x7f1b20101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 0x7f1b20101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36280/0 (socket says 172.21.15.3:36280) 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 -- 172.21.15.3:0/854075721 learned_addr learned my addr 172.21.15.3:0/854075721 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1e7fc640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b20105420 0x7f1b20102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1f7fe640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20102860 0x7f1b201a5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 -- 172.21.15.3:0/854075721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20102860 msgr2=0x7f1b201a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20102860 0x7f1b201a5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 -- 172.21.15.3:0/854075721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b20105420 msgr2=0x7f1b20102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b20105420 0x7f1b20102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 -- 172.21.15.3:0/854075721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b08009560 con 0x7f1b20100350 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1f7fe640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20102860 0x7f1b201a5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b1effd640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 0x7f1b20101de0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000ebc0 tx=0x7f1b1000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.001+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b100099b0 con 0x7f1b20100350 2024-02-09T22:57:03.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.002+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b201a6620 con 0x7f1b20100350 2024-02-09T22:57:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.002+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b1000eec0 con 0x7f1b20100350 2024-02-09T22:57:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.002+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b10018b60 con 0x7f1b20100350 2024-02-09T22:57:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.002+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b201a6b00 con 0x7f1b20100350 2024-02-09T22:57:03.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.002+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b200680b0 con 0x7f1b20100350 2024-02-09T22:57:03.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.004+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1b10018cc0 con 0x7f1b20100350 2024-02-09T22:57:03.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.004+0000 7f1b17fff640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b0c0a2260 0x7f1b0c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.004+0000 7f1b1e7fc640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b0c0a2260 0x7f1b0c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.004+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1b10014070 con 0x7f1b20100350 2024-02-09T22:57:03.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.005+0000 7f1b1e7fc640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b0c0a2260 0x7f1b0c0a4720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b08000c00 tx=0x7f1b08005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.008+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1b100978a0 con 0x7f1b20100350 2024-02-09T22:57:03.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.239+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b20109660 con 0x7f1b20100350 2024-02-09T22:57:03.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.239+0000 7f1b17fff640 1 -- 172.21.15.3:0/854075721 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 39 v39) v1 ==== 94+0+11369 (secure 0 0 0) 0x7f1b10097070 con 0x7f1b20100350 2024-02-09T22:57:03.243 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:57:03.243 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":39,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":39,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:57:00.445994+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:57:03.243 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 39 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b0c0a2260 msgr2=0x7f1b0c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b0c0a2260 0x7f1b0c0a4720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1b08000c00 tx=0x7f1b08005fd0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 msgr2=0x7f1b20101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 0x7f1b20101de0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f1b1000ebc0 tx=0x7f1b1000c6a0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 shutdown_connections 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b0c0a2260 0x7f1b0c0a4720 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b20105420 0x7f1b20102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.241+0000 7f1b2551e640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b20102860 0x7f1b201a5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.242+0000 7f1b2551e640 1 --2- 172.21.15.3:0/854075721 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b20100350 0x7f1b20101de0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.242+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 >> 172.21.15.3:0/854075721 conn(0x7f1b200772b0 msgr2=0x7f1b20104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.242+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 shutdown_connections 2024-02-09T22:57:03.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.242+0000 7f1b2551e640 1 -- 172.21.15.3:0/854075721 wait complete. 2024-02-09T22:57:03.256 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:03.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:57:03.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 -- 172.21.15.3:0/3537885770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 msgr2=0x7fc1ac0ffb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 --2- 172.21.15.3:0/3537885770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac0ffb80 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7fc194009930 tx=0x7fc19402bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:03.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 -- 172.21.15.3:0/3537885770 shutdown_connections 2024-02-09T22:57:03.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 --2- 172.21.15.3:0/3537885770 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 0x7fc1ac10e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 --2- 172.21.15.3:0/3537885770 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac0ffb80 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 --2- 172.21.15.3:0/3537885770 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 0x7fc1ac1065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 -- 172.21.15.3:0/3537885770 >> 172.21.15.3:0/3537885770 conn(0x7fc1ac076630 msgr2=0x7fc1ac0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 -- 172.21.15.3:0/3537885770 shutdown_connections 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.361+0000 7fc1b1878640 1 -- 172.21.15.3:0/3537885770 wait complete. 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 Processor -- start 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 -- start start 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac19d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 0x7fc1ac19dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 0x7fc1ac1977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac197f50 con 0x7fc1ac0ff700 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac1980c0 con 0x7fc1ac106200 2024-02-09T22:57:03.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1b1878640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1ac198280 con 0x7fc1ac1000c0 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac19d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac19d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36300/0 (socket says 172.21.15.3:36300) 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 -- 172.21.15.3:0/2930076941 learned_addr learned my addr 172.21.15.3:0/2930076941 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 0x7fc1ac19dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 0x7fc1ac1977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 -- 172.21.15.3:0/2930076941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 msgr2=0x7fc1ac19dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 0x7fc1ac19dbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 -- 172.21.15.3:0/2930076941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 msgr2=0x7fc1ac1977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 0x7fc1ac1977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1aaffd640 1 -- 172.21.15.3:0/2930076941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc194009560 con 0x7fc1ac0ff700 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 0x7fc1ac19dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.362+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 0x7fc1ac1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.363+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac19d6b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00e9f0 tx=0x7fc19c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.363+0000 7fc1b0876640 1 -- 172.21.15.3:0/2930076941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c009700 con 0x7fc1ac0ff700 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.363+0000 7fc1b0876640 1 -- 172.21.15.3:0/2930076941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc19c00cd80 con 0x7fc1ac0ff700 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.363+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1ac198560 con 0x7fc1ac0ff700 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.363+0000 7fc1b0876640 1 -- 172.21.15.3:0/2930076941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c018920 con 0x7fc1ac0ff700 2024-02-09T22:57:03.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.363+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1a4630 con 0x7fc1ac0ff700 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.365+0000 7fc1b0876640 1 -- 172.21.15.3:0/2930076941 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc19c018a80 con 0x7fc1ac0ff700 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.365+0000 7fc1b4445740 1 Processor -- start 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.365+0000 7fc1b4445740 1 -- start start 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc13e0 0x5639c3fc9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc9f60 0x5639c3fcc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b4445740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3f7c020 con 0x5639c3fc0a30 2024-02-09T22:57:03.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b4445740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3f2f320 con 0x5639c3fc9f60 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b4445740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3f2c850 con 0x5639c3fc13e0 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b0876640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1980a2230 0x7fc1980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc13e0 0x5639c3fc9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc0e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36314/0 (socket says 172.21.15.3:36314) 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/3904751732 learned_addr learned my addr 172.21.15.3:0/3904751732 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1980a2230 0x7fc1980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1b0876640 1 -- 172.21.15.3:0/2930076941 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc19c014070 con 0x7fc1ac0ff700 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc9f60 0x5639c3fcc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3904751732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc9f60 msgr2=0x5639c3fcc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc9f60 0x5639c3fcc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3904751732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 msgr2=0x5639c3fc0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3904751732 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639c3eea210 con 0x5639c3fc13e0 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc13e0 0x5639c3fc9990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0cbae0 tx=0x7fc19c06df60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a65c1df339f6b913 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1980a2230 0x7fc1980a46f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40099e0 tx=0x7fc1a4009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.366+0000 7fc1a1ffb640 1 -- 172.21.15.3:0/3904751732 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c09eed0 con 0x5639c3fc13e0 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1a1ffb640 1 -- 172.21.15.3:0/3904751732 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc19c0a3520 con 0x5639c3fc13e0 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1a1ffb640 1 -- 172.21.15.3:0/3904751732 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0a3090 con 0x5639c3fc13e0 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 -- 172.21.15.3:0/3904751732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc13e0 msgr2=0x5639c3fc9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc13e0 0x5639c3fc9990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0cbae0 tx=0x7fc19c06df60 comp rx=0 tx=0).stop 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 -- 172.21.15.3:0/3904751732 shutdown_connections 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc9f60 0x5639c3fcc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc13e0 0x5639c3fc9990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3904751732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 -- 172.21.15.3:0/3904751732 >> 172.21.15.3:0/3904751732 conn(0x5639c3fbb100 msgr2=0x5639c3fbb4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 -- 172.21.15.3:0/3904751732 shutdown_connections 2024-02-09T22:57:03.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.367+0000 7fc1b4445740 1 -- 172.21.15.3:0/3904751732 wait complete. 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 -- start start 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 Processor -- start 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3f28030 con 0x5639c3fc0a30 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3bfb430 con 0x5639c3fc13e0 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3bb7e00 con 0x5639c3fc9f60 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1b4445740 10 client.0 _ll_register_callbacks cb 0x5639c3f1d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36320/0 (socket says 172.21.15.3:36320) 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1aaffd640 1 -- 172.21.15.3:0/462555542 learned_addr learned my addr 172.21.15.3:0/462555542 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.368+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/462555542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 msgr2=0x5639c3fc74d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/462555542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 msgr2=0x5639c3fc6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/462555542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1ac1a4630 con 0x5639c3fc9f60 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4003e90 tx=0x7fc1a4004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43e9bcfbbd469f75 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc18affd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc18affd640 1 -- 172.21.15.3:0/462555542 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4004d20 con 0x5639c3fc9f60 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc18affd640 1 -- 172.21.15.3:0/462555542 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc1a4020670 con 0x5639c3fc9f60 2024-02-09T22:57:03.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639c3c277e0 con 0x5639c3fc9f60 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc18affd640 1 -- 172.21.15.3:0/462555542 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1a4004e80 con 0x5639c3fc9f60 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1b4445740 10 client.26854 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5639c3f266b0 con 0x5639c3fc9f60 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.369+0000 7fc18affd640 1 -- 172.21.15.3:0/462555542 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5639c3c277e0 con 0x5639c3fc9f60 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc18affd640 1 -- 172.21.15.3:0/462555542 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc1a4005330 con 0x5639c3fc9f60 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1b4445740 10 client.26854 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1b4445740 10 client.26854 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1b4445740 20 client.26854 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1b4445740 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c3fda620 0x5639c3fdca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1b4445740 4 client.26854 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:03.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639c3c64720 con 0x5639c3fda620 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c3fda620 0x5639c3fdca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc18affd640 10 client.26854 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.370+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c3fda620 0x5639c3fdca10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5b2ce302532715f server_cookie=707c7a7b36aa142e in_seq=0 out_seq=0 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc18affd640 1 -- 172.21.15.3:0/462555542 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5639c3c64720 con 0x5639c3fda620 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc18affd640 10 client.26854 handle_command_reply: tid=0 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 client.26854 shutdown 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc9f60 msgr2=0x5639c3fc7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a10 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4003e90 tx=0x7fc1a4004a40 comp rx=0 tx=0).stop 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 shutdown_connections 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c3fda620 0x5639c3fdca10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a10 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 --2- 172.21.15.3:0/462555542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc0a30 0x5639c3fc6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc18affd640 0 client.26854 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:03.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 >> 172.21.15.3:0/462555542 conn(0x5639c3fbb100 msgr2=0x5639c3fbce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 shutdown_connections 2024-02-09T22:57:03.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 1 -- 172.21.15.3:0/462555542 wait complete. 2024-02-09T22:57:03.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.371+0000 7fc1b4445740 20 client.26854 trim_cache size 0 max 16384 2024-02-09T22:57:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 Processor -- start 2024-02-09T22:57:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 -- start start 2024-02-09T22:57:03.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fc96c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3ead130 con 0x5639c3fc9f60 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3fd7080 con 0x5639c3fc13e0 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1b4445740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c3fe1c20 con 0x5639c3fc0a30 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fc96c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fc96c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42188/0 (socket says 172.21.15.3:42188) 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3795749748 learned_addr learned my addr 172.21.15.3:0/3795749748 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3795749748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 msgr2=0x5639c3fc74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3795749748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 msgr2=0x5639c3fc7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 -- 172.21.15.3:0/3795749748 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639c3ec2c90 con 0x5639c3fc0a30 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.384+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fc96c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0a2ff0 tx=0x7fc19c0c5680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ba8ce1ae044c96c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1a17fa640 1 -- 172.21.15.3:0/3795749748 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c0974a0 con 0x5639c3fc0a30 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1a17fa640 1 -- 172.21.15.3:0/3795749748 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc19c097600 con 0x5639c3fc0a30 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1a17fa640 1 -- 172.21.15.3:0/3795749748 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc19c09a280 con 0x5639c3fc0a30 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 -- 172.21.15.3:0/3795749748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 msgr2=0x5639c3fc96c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fc96c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fc19c0a2ff0 tx=0x7fc19c0c5680 comp rx=0 tx=0).stop 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 -- 172.21.15.3:0/3795749748 shutdown_connections 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fc74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fc7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 --2- 172.21.15.3:0/3795749748 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fc96c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 -- 172.21.15.3:0/3795749748 >> 172.21.15.3:0/3795749748 conn(0x5639c3fbb100 msgr2=0x5639c3fc2250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 -- 172.21.15.3:0/3795749748 shutdown_connections 2024-02-09T22:57:03.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.385+0000 7fc1b4445740 1 -- 172.21.15.3:0/3795749748 wait complete. 2024-02-09T22:57:03.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 -- start start 2024-02-09T22:57:03.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 Processor -- start 2024-02-09T22:57:03.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fcfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fcffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fd0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c4030b00 con 0x5639c3fc9f60 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c4030c40 con 0x5639c3fc13e0 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639c4030db0 con 0x5639c3fc0a30 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1b4445740 10 client.0 _ll_register_callbacks cb 0x7fc19c00cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fd0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fd0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36334/0 (socket says 172.21.15.3:36334) 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/91139667 learned_addr learned my addr 172.21.15.3:0/91139667 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fcffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fcfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 msgr2=0x5639c3fcfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fcfa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/91139667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 msgr2=0x5639c3fcffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fcffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1ab7fe640 1 -- 172.21.15.3:0/91139667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639c3eea210 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.386+0000 7fc1a27fc640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fcffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1ab7fe640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fd0500 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4023040 tx=0x7fc1a4003f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa39fc98e1308f8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc18affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc18affd640 1 -- 172.21.15.3:0/91139667 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1a4020870 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc18affd640 1 -- 172.21.15.3:0/91139667 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc1a40209d0 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc19c0a3520 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc18affd640 1 -- 172.21.15.3:0/91139667 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1a4020b90 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 10 client.17604 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x5639c3ee8a70 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc18affd640 1 -- 172.21.15.3:0/91139667 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc19c0a3520 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc18affd640 1 -- 172.21.15.3:0/91139667 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc1a403a430 con 0x5639c3fc9f60 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 10 client.17604 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 10 client.17604 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 20 client.17604 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c4032570 0x5639c4034990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 4 client.17604 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.387+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5639c3cb3890 con 0x5639c4032570 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c4032570 0x5639c4034990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc18affd640 10 client.17604 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1aaffd640 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c4032570 0x5639c4034990 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=70258e4fa2af8cc2 server_cookie=b0856c17cce44932 in_seq=0 out_seq=0 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc18affd640 1 -- 172.21.15.3:0/91139667 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5639c3cb3890 con 0x5639c4032570 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc18affd640 10 client.17604 handle_command_reply: tid=0 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 client.17604 shutdown 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 msgr2=0x5639c3fd0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fd0500 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fc1a4023040 tx=0x7fc1a4003f40 comp rx=0 tx=0).stop 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 shutdown_connections 2024-02-09T22:57:03.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639c4032570 0x5639c4034990 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639c3fc13e0 0x5639c3fcffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639c3fc9f60 0x5639c3fd0500 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc18affd640 0 client.17604 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 --2- 172.21.15.3:0/91139667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639c3fc0a30 0x5639c3fcfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 >> 172.21.15.3:0/91139667 conn(0x5639c3fbb100 msgr2=0x5639c3fb99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 shutdown_connections 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 1 -- 172.21.15.3:0/91139667 wait complete. 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.388+0000 7fc1b4445740 20 client.17604 trim_cache size 0 max 16384 2024-02-09T22:57:03.398 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1980a2230 msgr2=0x7fc1980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1980a2230 0x7fc1980a46f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc1a40099e0 tx=0x7fc1a4009500 comp rx=0 tx=0).stop 2024-02-09T22:57:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 msgr2=0x7fc1ac19d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac19d6b0 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7fc19c00e9f0 tx=0x7fc19c00eeb0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 shutdown_connections 2024-02-09T22:57:03.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1980a2230 0x7fc1980a46f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1ac106200 0x7fc1ac1977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1ac1000c0 0x7fc1ac19dbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 --2- 172.21.15.3:0/2930076941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1ac0ff700 0x7fc1ac19d6b0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 >> 172.21.15.3:0/2930076941 conn(0x7fc1ac076630 msgr2=0x7fc1ac104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 shutdown_connections 2024-02-09T22:57:03.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.389+0000 7fc1b1878640 1 -- 172.21.15.3:0/2930076941 wait complete. 2024-02-09T22:57:03.403 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:03.403 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 -- 172.21.15.3:0/3583751581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c105410 msgr2=0x7f389c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 --2- 172.21.15.3:0/3583751581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c105410 0x7f389c10cfe0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f3894009cd0 tx=0x7f389402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 -- 172.21.15.3:0/3583751581 shutdown_connections 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 --2- 172.21.15.3:0/3583751581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c105410 0x7f389c10cfe0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 --2- 172.21.15.3:0/3583751581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c100cf0 0x7f389c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 --2- 172.21.15.3:0/3583751581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f389c100340 0x7f389c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 -- 172.21.15.3:0/3583751581 >> 172.21.15.3:0/3583751581 conn(0x7f389c0772e0 msgr2=0x7f389c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 -- 172.21.15.3:0/3583751581 shutdown_connections 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.508+0000 7f38a11b5640 1 -- 172.21.15.3:0/3583751581 wait complete. 2024-02-09T22:57:03.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 Processor -- start 2024-02-09T22:57:03.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 -- start start 2024-02-09T22:57:03.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 0x7f389c19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c105410 0x7f389c19db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f389c197690 0x7f389c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c198160 con 0x7f389c100340 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c1982d0 con 0x7f389c197690 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.509+0000 7f38a11b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389c198490 con 0x7f389c105410 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 0x7f389c19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c105410 0x7f389c19db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 0x7f389c19d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36340/0 (socket says 172.21.15.3:36340) 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 -- 172.21.15.3:0/2566523051 learned_addr learned my addr 172.21.15.3:0/2566523051 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389b59e640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f389c197690 0x7f389c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 -- 172.21.15.3:0/2566523051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c105410 msgr2=0x7f389c19db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c105410 0x7f389c19db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 -- 172.21.15.3:0/2566523051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f389c197690 msgr2=0x7f389c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f389c197690 0x7f389c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 -- 172.21.15.3:0/2566523051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c009490 con 0x7f389c100340 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389259c640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c105410 0x7f389c19db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f389ad9d640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 0x7f389c19d5d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f388c00ea60 tx=0x7f388c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f3898d99640 1 -- 172.21.15.3:0/2566523051 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c00cde0 con 0x7f389c100340 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f3898d99640 1 -- 172.21.15.3:0/2566523051 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c0035f0 con 0x7f389c100340 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3894009900 con 0x7f389c100340 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f3898d99640 1 -- 172.21.15.3:0/2566523051 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c010690 con 0x7f389c100340 2024-02-09T22:57:03.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.510+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c1a4780 con 0x7f389c100340 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f3898d99640 1 -- 172.21.15.3:0/2566523051 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f388c003af0 con 0x7f389c100340 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 Processor -- start 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 -- start start 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe430e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe4313e0 0x564dbe439990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe43c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe3ec020 con 0x564dbe430a30 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe39f320 con 0x564dbe439f60 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f38a3d89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe39c850 con 0x564dbe4313e0 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f389b59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe430e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f3898d99640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38880a2230 0x7f38880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe4313e0 0x564dbe439990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.513+0000 7f389b59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe430e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36352/0 (socket says 172.21.15.3:36352) 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389b59e640 1 -- 172.21.15.3:0/925041674 learned_addr learned my addr 172.21.15.3:0/925041674 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389259c640 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe43c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389b59e640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38880a2230 0x7f38880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f3898d99640 1 -- 172.21.15.3:0/2566523051 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f388c014070 con 0x7f389c100340 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389ad9d640 1 -- 172.21.15.3:0/925041674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 msgr2=0x564dbe43c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389ad9d640 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe43c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389ad9d640 1 -- 172.21.15.3:0/925041674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 msgr2=0x564dbe430e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389ad9d640 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe430e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389ad9d640 1 -- 172.21.15.3:0/925041674 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564dbe35a210 con 0x564dbe4313e0 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389b59e640 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe430e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389ad9d640 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe4313e0 0x564dbe439990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f388c0cbc50 tx=0x7f388c005790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd0bf04f416ef496 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389b59e640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38880a2230 0x7f38880a46f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3894033000 tx=0x7f389402c0f0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389159a640 1 -- 172.21.15.3:0/925041674 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c09b880 con 0x564dbe4313e0 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389159a640 1 -- 172.21.15.3:0/925041674 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c09aef0 con 0x564dbe4313e0 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.514+0000 7f389159a640 1 -- 172.21.15.3:0/925041674 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c0cbed0 con 0x564dbe4313e0 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 -- 172.21.15.3:0/925041674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe4313e0 msgr2=0x564dbe439990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe4313e0 0x564dbe439990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f388c0cbc50 tx=0x7f388c005790 comp rx=0 tx=0).stop 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 -- 172.21.15.3:0/925041674 shutdown_connections 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe43c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe4313e0 0x564dbe439990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 --2- 172.21.15.3:0/925041674 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe430e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 -- 172.21.15.3:0/925041674 >> 172.21.15.3:0/925041674 conn(0x564dbe42b100 msgr2=0x564dbe42b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 -- 172.21.15.3:0/925041674 shutdown_connections 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.515+0000 7f38a3d89740 1 -- 172.21.15.3:0/925041674 wait complete. 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 -- start start 2024-02-09T22:57:03.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 Processor -- start 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe436eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe4373f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe437930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe398030 con 0x564dbe430a30 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe06b430 con 0x564dbe4313e0 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe027e00 con 0x564dbe439f60 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe436eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f38a3d89740 10 client.0 _ll_register_callbacks cb 0x564dbe38d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe436eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36358/0 (socket says 172.21.15.3:36358) 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 -- 172.21.15.3:0/652263170 learned_addr learned my addr 172.21.15.3:0/652263170 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389259c640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe4373f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389b59e640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe437930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 -- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 msgr2=0x564dbe437930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe437930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 -- 172.21.15.3:0/652263170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 msgr2=0x564dbe4373f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe4373f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389ad9d640 1 -- 172.21.15.3:0/652263170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389c1a4780 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.516+0000 7f389b59e640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe437930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f389ad9d640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe436eb0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f388c0cc250 tx=0x7f388c097190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89549878fc7c5663 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38827fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38827fc640 1 -- 172.21.15.3:0/652263170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c0991f0 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38827fc640 1 -- 172.21.15.3:0/652263170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f388c09b5f0 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564dbe0977e0 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38827fc640 1 -- 172.21.15.3:0/652263170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f388c09f040 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 10 client.17616 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564dbe3966b0 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38827fc640 1 -- 172.21.15.3:0/652263170 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564dbe0977e0 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38827fc640 1 -- 172.21.15.3:0/652263170 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f388c0037d0 con 0x564dbe430a30 2024-02-09T22:57:03.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 10 client.17616 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 10 client.17616 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 20 client.17616 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe44c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 4 client.17616 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564dbe0d4720 con 0x564dbe44a0a0 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.517+0000 7f389259c640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe44c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38827fc640 10 client.17616 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f389259c640 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe44c4c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=eae3ee1788bc9c5e server_cookie=998643791558d4ff in_seq=0 out_seq=0 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38827fc640 1 -- 172.21.15.3:0/652263170 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564dbe0d4720 con 0x564dbe44a0a0 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38827fc640 10 client.17616 handle_command_reply: tid=0 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38a3d89740 1 client.17616 shutdown 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 msgr2=0x564dbe436eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38a3d89740 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe436eb0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f388c0cc250 tx=0x7f388c097190 comp rx=0 tx=0).stop 2024-02-09T22:57:03.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 shutdown_connections 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38a3d89740 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe44c4c0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38827fc640 0 client.17616 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.518+0000 7f38a3d89740 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe4373f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38a3d89740 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe437930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38a3d89740 1 --2- 172.21.15.3:0/652263170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe436eb0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 >> 172.21.15.3:0/652263170 conn(0x564dbe42b100 msgr2=0x564dbe42ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 shutdown_connections 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38a3d89740 1 -- 172.21.15.3:0/652263170 wait complete. 2024-02-09T22:57:03.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.519+0000 7f38a3d89740 20 client.17616 trim_cache size 0 max 16384 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.530+0000 7f38a3d89740 1 Processor -- start 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.530+0000 7f38a3d89740 1 -- start start 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe430a30 0x564dbe4395e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 0x564dbe426210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe432130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f38a3d89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe31d130 con 0x564dbe4313e0 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f38a3d89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe426ab0 con 0x564dbe439f60 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f38a3d89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe437e80 con 0x564dbe430a30 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 0x564dbe426210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 0x564dbe426210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36368/0 (socket says 172.21.15.3:36368) 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 -- 172.21.15.3:0/885075055 learned_addr learned my addr 172.21.15.3:0/885075055 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389b59e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe432130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389ad9d640 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe430a30 0x564dbe4395e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 -- 172.21.15.3:0/885075055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe430a30 msgr2=0x564dbe4395e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe430a30 0x564dbe4395e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 -- 172.21.15.3:0/885075055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 msgr2=0x564dbe432130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe432130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 -- 172.21.15.3:0/885075055 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564dbe332c90 con 0x564dbe4313e0 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389b59e640 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe432130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.531+0000 7f389259c640 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 0x564dbe426210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3884009fd0 tx=0x7f3884012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f38a5de6b41025b9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f3890d99640 1 -- 172.21.15.3:0/885075055 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3884017070 con 0x564dbe4313e0 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f3890d99640 1 -- 172.21.15.3:0/885075055 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3884018040 con 0x564dbe4313e0 2024-02-09T22:57:03.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f3890d99640 1 -- 172.21.15.3:0/885075055 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38840053a0 con 0x564dbe4313e0 2024-02-09T22:57:03.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 -- 172.21.15.3:0/885075055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 msgr2=0x564dbe426210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 0x564dbe426210 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f3884009fd0 tx=0x7f3884012b00 comp rx=0 tx=0).stop 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 -- 172.21.15.3:0/885075055 shutdown_connections 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe439f60 0x564dbe432130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe4313e0 0x564dbe426210 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 --2- 172.21.15.3:0/885075055 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe430a30 0x564dbe4395e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 -- 172.21.15.3:0/885075055 >> 172.21.15.3:0/885075055 conn(0x564dbe42b100 msgr2=0x564dbe42b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 -- 172.21.15.3:0/885075055 shutdown_connections 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.532+0000 7f38a3d89740 1 -- 172.21.15.3:0/885075055 wait complete. 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 -- start start 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 Processor -- start 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe440210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe440750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe49a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe49ab30 con 0x564dbe430a30 2024-02-09T22:57:03.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe49aca0 con 0x564dbe4313e0 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564dbe49ae60 con 0x564dbe439f60 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f38a3d89740 10 client.0 _ll_register_callbacks cb 0x7f388c09acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe440210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe440210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36372/0 (socket says 172.21.15.3:36372) 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 -- 172.21.15.3:0/1216398792 learned_addr learned my addr 172.21.15.3:0/1216398792 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389259c640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe440750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389b59e640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe49a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 -- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 msgr2=0x564dbe49a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe49a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 -- 172.21.15.3:0/1216398792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 msgr2=0x564dbe440750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe440750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.533+0000 7f389ad9d640 1 -- 172.21.15.3:0/1216398792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389404a040 con 0x564dbe430a30 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f389ad9d640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe440210 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c760 tx=0x7f388c09ade0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679a03d5427ccd15 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38827fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38827fc640 1 -- 172.21.15.3:0/1216398792 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388c09c220 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38827fc640 1 -- 172.21.15.3:0/1216398792 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f388c07be40 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f388c09aef0 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38827fc640 1 -- 172.21.15.3:0/1216398792 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f388c09b880 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 10 client.17622 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x7f3884012040 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38827fc640 1 -- 172.21.15.3:0/1216398792 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f388c0ae020 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38827fc640 1 -- 172.21.15.3:0/1216398792 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f388c003760 con 0x564dbe430a30 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 10 client.17622 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 10 client.17622 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 20 client.17622 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe4a4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 4 client.17622 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x564dbe123890 con 0x564dbe44a0a0 2024-02-09T22:57:03.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.534+0000 7f389259c640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe4a4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f389259c640 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe4a4180 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9ef3150bd60ba80e server_cookie=ff0bba0bb99bc338 in_seq=0 out_seq=0 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38827fc640 10 client.17622 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38827fc640 1 -- 172.21.15.3:0/1216398792 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x564dbe123890 con 0x564dbe44a0a0 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38827fc640 10 client.17622 handle_command_reply: tid=0 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 client.17622 shutdown 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 msgr2=0x564dbe440210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe440210 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f388c00c760 tx=0x7f388c09ade0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 shutdown_connections 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x564dbe44a0a0 0x564dbe4a4180 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564dbe4313e0 0x564dbe440750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564dbe439f60 0x564dbe49a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38827fc640 0 client.17622 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 --2- 172.21.15.3:0/1216398792 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564dbe430a30 0x564dbe440210 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 >> 172.21.15.3:0/1216398792 conn(0x564dbe42b100 msgr2=0x564dbe432130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 shutdown_connections 2024-02-09T22:57:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 1 -- 172.21.15.3:0/1216398792 wait complete. 2024-02-09T22:57:03.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.535+0000 7f38a3d89740 20 client.17622 trim_cache size 0 max 16384 2024-02-09T22:57:03.545 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38880a2230 msgr2=0x7f38880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38880a2230 0x7f38880a46f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3894033000 tx=0x7f389402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 msgr2=0x7f389c19d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 0x7f389c19d5d0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f388c00ea60 tx=0x7f388c00ef20 comp rx=0 tx=0).stop 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 shutdown_connections 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38880a2230 0x7f38880a46f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f389c197690 0x7f389c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f389c105410 0x7f389c19db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 --2- 172.21.15.3:0/2566523051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f389c100340 0x7f389c19d5d0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 >> 172.21.15.3:0/2566523051 conn(0x7f389c0772e0 msgr2=0x7f389c0688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 shutdown_connections 2024-02-09T22:57:03.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.536+0000 7f38a11b5640 1 -- 172.21.15.3:0/2566523051 wait complete. 2024-02-09T22:57:03.549 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:03.549 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:57:03.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 168354 2.73 MB/sec execute 177 sec latency 953.255 ms 2024-02-09T22:57:03.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 -- 172.21.15.3:0/919962823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 msgr2=0x7ff47c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 --2- 172.21.15.3:0/919962823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c100720 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff478009930 tx=0x7ff47802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 -- 172.21.15.3:0/919962823 shutdown_connections 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 --2- 172.21.15.3:0/919962823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c100cf0 0x7ff47c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 --2- 172.21.15.3:0/919962823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c105410 0x7ff47c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 --2- 172.21.15.3:0/919962823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c100720 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 -- 172.21.15.3:0/919962823 >> 172.21.15.3:0/919962823 conn(0x7ff47c0772e0 msgr2=0x7ff47c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 -- 172.21.15.3:0/919962823 shutdown_connections 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 -- 172.21.15.3:0/919962823 wait complete. 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 Processor -- start 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.655+0000 7ff4841e2640 1 -- start start 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c19d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c105410 0x7ff47c19dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197780 0x7ff47c197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c198250 con 0x7ff47c100340 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c1983c0 con 0x7ff47c197780 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c198580 con 0x7ff47c105410 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c19d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c19d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36390/0 (socket says 172.21.15.3:36390) 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 -- 172.21.15.3:0/1078768472 learned_addr learned my addr 172.21.15.3:0/1078768472 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff48177d640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c105410 0x7ff47c19dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff48277f640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197780 0x7ff47c197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 -- 172.21.15.3:0/1078768472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c105410 msgr2=0x7ff47c19dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c105410 0x7ff47c19dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 -- 172.21.15.3:0/1078768472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197780 msgr2=0x7ff47c197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197780 0x7ff47c197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 -- 172.21.15.3:0/1078768472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff478009560 con 0x7ff47c100340 2024-02-09T22:57:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff48277f640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197780 0x7ff47c197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c19d670 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff478000c00 tx=0x7ff47802c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff472ffd640 1 -- 172.21.15.3:0/1078768472 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4780042d0 con 0x7ff47c100340 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff472ffd640 1 -- 172.21.15.3:0/1078768472 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff478004430 con 0x7ff47c100340 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c1a4470 con 0x7ff47c100340 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff472ffd640 1 -- 172.21.15.3:0/1078768472 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff478003c50 con 0x7ff47c100340 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.656+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c1a49a0 con 0x7ff47c100340 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.658+0000 7ff472ffd640 1 -- 172.21.15.3:0/1078768472 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff478004df0 con 0x7ff47c100340 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 Processor -- start 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 -- start start 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff472ffd640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4680a22b0 0x7ff4680a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d40e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d413e0 0x558ea0d49990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d49f60 0x558ea0d4c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0cfc020 con 0x558ea0d413e0 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0caf320 con 0x558ea0d40a30 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff486db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0cac850 con 0x558ea0d49f60 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff48177d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d413e0 0x558ea0d49990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff48277f640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4680a22b0 0x7ff4680a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff481f7e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d40e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff48177d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d413e0 0x558ea0d49990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36392/0 (socket says 172.21.15.3:36392) 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff48177d640 1 -- 172.21.15.3:0/1931642889 learned_addr learned my addr 172.21.15.3:0/1931642889 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff472ffd640 1 -- 172.21.15.3:0/1078768472 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff4780f06b0 con 0x7ff47c100340 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.659+0000 7ff48277f640 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d49f60 0x558ea0d4c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff481f7e640 1 -- 172.21.15.3:0/1931642889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d49f60 msgr2=0x558ea0d4c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d49f60 0x558ea0d4c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff481f7e640 1 -- 172.21.15.3:0/1931642889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d413e0 msgr2=0x558ea0d49990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d413e0 0x558ea0d49990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff481f7e640 1 -- 172.21.15.3:0/1931642889 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ea0c6a210 con 0x558ea0d40a30 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff48277f640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4680a22b0 0x7ff4680a4770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff474009810 tx=0x7ff474009e80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d40e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff4780edee0 tx=0x7ff4780bccc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd23df2f4797458e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff467fff640 1 -- 172.21.15.3:0/1931642889 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4780bd180 con 0x558ea0d40a30 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff467fff640 1 -- 172.21.15.3:0/1931642889 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4780c12d0 con 0x558ea0d40a30 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.660+0000 7ff467fff640 1 -- 172.21.15.3:0/1931642889 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4780de0e0 con 0x558ea0d40a30 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 -- 172.21.15.3:0/1931642889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 msgr2=0x558ea0d40e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d40e10 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff4780edee0 tx=0x7ff4780bccc0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 -- 172.21.15.3:0/1931642889 shutdown_connections 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d49f60 0x558ea0d4c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d413e0 0x558ea0d49990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 --2- 172.21.15.3:0/1931642889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d40e10 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 -- 172.21.15.3:0/1931642889 >> 172.21.15.3:0/1931642889 conn(0x558ea0d3b100 msgr2=0x558ea0d3b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 -- 172.21.15.3:0/1931642889 shutdown_connections 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 -- 172.21.15.3:0/1931642889 wait complete. 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 -- start start 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.661+0000 7ff486db2740 1 Processor -- start 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d497f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d42b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d430c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0ca8030 con 0x558ea0d49f60 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea097b430 con 0x558ea0d40a30 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0937e00 con 0x558ea0d413e0 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff486db2740 10 client.0 _ll_register_callbacks cb 0x558ea0c9d030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d42b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff481f7e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d497f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d42b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42312/0 (socket says 172.21.15.3:42312) 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 -- 172.21.15.3:0/2141855442 learned_addr learned my addr 172.21.15.3:0/2141855442 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48277f640 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d430c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 -- 172.21.15.3:0/2141855442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 msgr2=0x558ea0d497f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d497f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 -- 172.21.15.3:0/2141855442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 msgr2=0x558ea0d430c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d430c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 -- 172.21.15.3:0/2141855442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c1a49a0 con 0x558ea0d413e0 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff48177d640 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d42b40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009fc0 tx=0x7ff46c00e780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c1f486b031c0b34 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.662+0000 7ff4657fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:57:03.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff4657fa640 1 -- 172.21.15.3:0/2141855442 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c004220 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff4657fa640 1 -- 172.21.15.3:0/2141855442 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff46c004380 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558ea09a77e0 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff4657fa640 1 -- 172.21.15.3:0/2141855442 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff46c0047a0 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 10 client.26890 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x558ea0ca66b0 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff4657fa640 1 -- 172.21.15.3:0/2141855442 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff46c02f080 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff4657fa640 1 -- 172.21.15.3:0/2141855442 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff46c0035f0 con 0x558ea0d413e0 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 10 client.26890 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 10 client.26890 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 20 client.26890 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0d5a3a0 0x558ea0d5c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 4 client.26890 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558ea09e4720 con 0x558ea0d5a3a0 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.663+0000 7ff481f7e640 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0d5a3a0 0x558ea0d5c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff4657fa640 10 client.26890 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff481f7e640 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0d5a3a0 0x558ea0d5c7c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79bfc67d54035de8 server_cookie=7c9e1e3a6f2e747b in_seq=0 out_seq=0 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff4657fa640 1 -- 172.21.15.3:0/2141855442 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558ea09e4720 con 0x558ea0d5a3a0 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff4657fa640 10 client.26890 handle_command_reply: tid=0 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff486db2740 1 client.26890 shutdown 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 msgr2=0x558ea0d42b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.664+0000 7ff486db2740 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d42b40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff46c009fc0 tx=0x7ff46c00e780 comp rx=0 tx=0).stop 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 shutdown_connections 2024-02-09T22:57:03.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0d5a3a0 0x558ea0d5c7c0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d430c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d42b40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 --2- 172.21.15.3:0/2141855442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d497f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff4657fa640 0 client.26890 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 >> 172.21.15.3:0/2141855442 conn(0x558ea0d3b100 msgr2=0x558ea0d3ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 shutdown_connections 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 1 -- 172.21.15.3:0/2141855442 wait complete. 2024-02-09T22:57:03.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.665+0000 7ff486db2740 20 client.26890 trim_cache size 0 max 16384 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 Processor -- start 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 -- start start 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d4cb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d49300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d61120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0c2d130 con 0x558ea0d49f60 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0d48130 con 0x558ea0d40a30 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff486db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0d3bcd0 con 0x558ea0d413e0 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d61120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d61120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36406/0 (socket says 172.21.15.3:36406) 2024-02-09T22:57:03.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 -- 172.21.15.3:0/1232563754 learned_addr learned my addr 172.21.15.3:0/1232563754 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48177d640 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d49300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff481f7e640 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d4cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 -- 172.21.15.3:0/1232563754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 msgr2=0x558ea0d49300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d49300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 -- 172.21.15.3:0/1232563754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 msgr2=0x558ea0d4cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d4cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 -- 172.21.15.3:0/1232563754 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558ea0c42c90 con 0x558ea0d49f60 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff48277f640 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d61120 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7ff474014110 tx=0x7ff474004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=547fffc36ce25649 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.677+0000 7ff4677fe640 1 -- 172.21.15.3:0/1232563754 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474004030 con 0x558ea0d49f60 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff4677fe640 1 -- 172.21.15.3:0/1232563754 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff47401a040 con 0x558ea0d49f60 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff4677fe640 1 -- 172.21.15.3:0/1232563754 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474012b30 con 0x558ea0d49f60 2024-02-09T22:57:03.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 -- 172.21.15.3:0/1232563754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 msgr2=0x558ea0d61120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d61120 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7ff474014110 tx=0x7ff474004ab0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 -- 172.21.15.3:0/1232563754 shutdown_connections 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0d61120 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d413e0 0x558ea0d49300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 --2- 172.21.15.3:0/1232563754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d40a30 0x558ea0d4cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 -- 172.21.15.3:0/1232563754 >> 172.21.15.3:0/1232563754 conn(0x558ea0d3b100 msgr2=0x558ea0d4d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 -- 172.21.15.3:0/1232563754 shutdown_connections 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 -- 172.21.15.3:0/1232563754 wait complete. 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.678+0000 7ff486db2740 1 -- start start 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 Processor -- start 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d40a30 0x558ea0d53460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d413e0 0x558ea0d539a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0db08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0d542a0 con 0x558ea0d49f60 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0db0de0 con 0x558ea0d413e0 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558ea0db0f20 con 0x558ea0d40a30 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff486db2740 10 client.0 _ll_register_callbacks cb 0x7ff46c017d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0db08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48177d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d413e0 0x558ea0d539a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0db08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36422/0 (socket says 172.21.15.3:36422) 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 -- 172.21.15.3:0/2494181967 learned_addr learned my addr 172.21.15.3:0/2494181967 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff481f7e640 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d40a30 0x558ea0d53460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 -- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d40a30 msgr2=0x558ea0d53460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d40a30 0x558ea0d53460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 -- 172.21.15.3:0/2494181967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d413e0 msgr2=0x558ea0d539a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d413e0 0x558ea0d539a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 -- 172.21.15.3:0/2494181967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff474009900 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.679+0000 7ff48277f640 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0db08a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff474000f80 tx=0x7ff474012e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5066adf1c8eceee server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff4657fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff4657fa640 1 -- 172.21.15.3:0/2494181967 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff474004230 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff4657fa640 1 -- 172.21.15.3:0/2494181967 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff474004390 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff4780c12d0 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff4657fa640 1 -- 172.21.15.3:0/2494181967 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff474012450 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 10 client.17640 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x558ea0c68a70 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff4657fa640 1 -- 172.21.15.3:0/2494181967 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4780c12d0 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff4657fa640 1 -- 172.21.15.3:0/2494181967 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff474029a80 con 0x558ea0d49f60 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 10 client.17640 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 10 client.17640 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:57:03.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 20 client.17640 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0db2660 0x558ea0db4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 4 client.17640 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558ea0a33890 con 0x558ea0db2660 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.680+0000 7ff481f7e640 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0db2660 0x558ea0db4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff481f7e640 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0db2660 0x558ea0db4a80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d0d69bdf2fcf922a server_cookie=3a1d06b8a6d55b63 in_seq=0 out_seq=0 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff4657fa640 10 client.17640 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff4657fa640 1 -- 172.21.15.3:0/2494181967 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558ea0a33890 con 0x558ea0db2660 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff4657fa640 10 client.17640 handle_command_reply: tid=0 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 client.17640 shutdown 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 msgr2=0x558ea0db08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0db08a0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7ff474000f80 tx=0x7ff474012e50 comp rx=0 tx=0).stop 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 shutdown_connections 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558ea0db2660 0x558ea0db4a80 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558ea0d413e0 0x558ea0d539a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558ea0d49f60 0x558ea0db08a0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 --2- 172.21.15.3:0/2494181967 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558ea0d40a30 0x558ea0d53460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff4657fa640 0 client.17640 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 >> 172.21.15.3:0/2494181967 conn(0x558ea0d3b100 msgr2=0x558ea0d61120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 shutdown_connections 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 1 -- 172.21.15.3:0/2494181967 wait complete. 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.681+0000 7ff486db2740 20 client.17640 trim_cache size 0 max 16384 2024-02-09T22:57:03.692 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4680a22b0 msgr2=0x7ff4680a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4680a22b0 0x7ff4680a4770 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff474009810 tx=0x7ff474009e80 comp rx=0 tx=0).stop 2024-02-09T22:57:03.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 msgr2=0x7ff47c19d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c19d670 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7ff478000c00 tx=0x7ff47802c5f0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 shutdown_connections 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff4680a22b0 0x7ff4680a4770 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff47c197780 0x7ff47c197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff47c105410 0x7ff47c19dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 --2- 172.21.15.3:0/1078768472 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff47c100340 0x7ff47c19d670 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 >> 172.21.15.3:0/1078768472 conn(0x7ff47c0772e0 msgr2=0x7ff47c1040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 shutdown_connections 2024-02-09T22:57:03.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.682+0000 7ff4841e2640 1 -- 172.21.15.3:0/1078768472 wait complete. 2024-02-09T22:57:03.695 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:03.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:57:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/538037775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c105400 msgr2=0x7f0f9c10cfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/538037775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c105400 0x7f0f9c10cfd0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7f0f94009cd0 tx=0x7f0f9402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/538037775 shutdown_connections 2024-02-09T22:57:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/538037775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f9c100ce0 0x7f0f9c104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/538037775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c105400 0x7f0f9c10cfd0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/538037775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c100330 0x7f0f9c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/538037775 >> 172.21.15.3:0/538037775 conn(0x7f0f9c077250 msgr2=0x7f0f9c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/538037775 shutdown_connections 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.797+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/538037775 wait complete. 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 Processor -- start 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 -- start start 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c100330 0x7f0f9c196770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f9c105400 0x7f0f9c196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 0x7f0f9c19b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19b9d0 con 0x7f0f9c19afe0 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19bb40 con 0x7f0f9c105400 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa3cb5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9c19bcb0 con 0x7f0f9c100330 2024-02-09T22:57:03.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 0x7f0f9c19b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 0x7f0f9c19b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36448/0 (socket says 172.21.15.3:36448) 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 -- 172.21.15.3:0/1237741872 learned_addr learned my addr 172.21.15.3:0/1237741872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c100330 0x7f0f9c196770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa1250640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f9c105400 0x7f0f9c196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 -- 172.21.15.3:0/1237741872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c100330 msgr2=0x7f0f9c196770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c100330 0x7f0f9c196770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 -- 172.21.15.3:0/1237741872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f9c105400 msgr2=0x7f0f9c196cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f9c105400 0x7f0f9c196cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 -- 172.21.15.3:0/1237741872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94009900 con 0x7f0f9c19afe0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.798+0000 7f0fa2252640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 0x7f0f9c19b490 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f0f94004af0 tx=0x7f0f94034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.799+0000 7f0f92ffd640 1 -- 172.21.15.3:0/1237741872 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f940328c0 con 0x7f0f9c19afe0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.799+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f9c1a3290 con 0x7f0f9c19afe0 2024-02-09T22:57:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.799+0000 7f0f92ffd640 1 -- 172.21.15.3:0/1237741872 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f94003e50 con 0x7f0f9c19afe0 2024-02-09T22:57:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.799+0000 7f0f92ffd640 1 -- 172.21.15.3:0/1237741872 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f94004860 con 0x7f0f9c19afe0 2024-02-09T22:57:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.799+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9c1a3710 con 0x7f0f9c19afe0 2024-02-09T22:57:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.801+0000 7f0f92ffd640 1 -- 172.21.15.3:0/1237741872 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0f9400a040 con 0x7f0f9c19afe0 2024-02-09T22:57:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.801+0000 7f0fa6967740 1 Processor -- start 2024-02-09T22:57:03.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 -- start start 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e68a30 0x55f7d3e68e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e71990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e71f60 0x55f7d3e743b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3e24020 con 0x55f7d3e693e0 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3dd7320 con 0x55f7d3e71f60 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa6967740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3dd4850 con 0x55f7d3e68a30 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0f92ffd640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f880a2230 0x7f0f880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e71990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e71990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36450/0 (socket says 172.21.15.3:36450) 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 -- 172.21.15.3:0/3344714031 learned_addr learned my addr 172.21.15.3:0/3344714031 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1250640 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e71f60 0x55f7d3e743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa2252640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f880a2230 0x7f0f880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1250640 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e68a30 0x55f7d3e68e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0f92ffd640 1 -- 172.21.15.3:0/1237741872 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0f940f08f0 con 0x7f0f9c19afe0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 -- 172.21.15.3:0/3344714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e68a30 msgr2=0x55f7d3e68e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e68a30 0x55f7d3e68e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 -- 172.21.15.3:0/3344714031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e71f60 msgr2=0x55f7d3e743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e71f60 0x55f7d3e743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1a51640 1 -- 172.21.15.3:0/3344714031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7d3d92210 con 0x55f7d3e693e0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.802+0000 7f0fa1250640 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e71f60 0x55f7d3e743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e71990 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009ca0 tx=0x7f0f980119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf562bff0772adb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0fa2252640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f880a2230 0x7f0f880a46f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f940b9d90 tx=0x7f0f940095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0f87fff640 1 -- 172.21.15.3:0/3344714031 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98016070 con 0x55f7d3e693e0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0f87fff640 1 -- 172.21.15.3:0/3344714031 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f9801a8f0 con 0x55f7d3e693e0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0f87fff640 1 -- 172.21.15.3:0/3344714031 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9800e960 con 0x55f7d3e693e0 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0fa6967740 1 -- 172.21.15.3:0/3344714031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 msgr2=0x55f7d3e71990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.803+0000 7f0fa6967740 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e71990 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009ca0 tx=0x7f0f980119a0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- 172.21.15.3:0/3344714031 shutdown_connections 2024-02-09T22:57:03.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e71f60 0x55f7d3e743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e71990 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 --2- 172.21.15.3:0/3344714031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e68a30 0x55f7d3e68e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- 172.21.15.3:0/3344714031 >> 172.21.15.3:0/3344714031 conn(0x55f7d3e63100 msgr2=0x55f7d3e634e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- 172.21.15.3:0/3344714031 shutdown_connections 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- 172.21.15.3:0/3344714031 wait complete. 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- start start 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 Processor -- start 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e71f60 0x55f7d3e6fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3dd0030 con 0x55f7d3e693e0 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3aa3430 con 0x55f7d3e68a30 2024-02-09T22:57:03.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3a5fe00 con 0x55f7d3e71f60 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.804+0000 7f0fa6967740 10 client.0 _ll_register_callbacks cb 0x55f7d3dc5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6f4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36456/0 (socket says 172.21.15.3:36456) 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2049382727 learned_addr learned my addr 172.21.15.3:0/2049382727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e71f60 0x55f7d3e6fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2049382727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e71f60 msgr2=0x55f7d3e6fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e71f60 0x55f7d3e6fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 msgr2=0x55f7d3e6ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2049382727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f98009950 con 0x55f7d3e693e0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6f4c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009250 tx=0x7f0f9801ac90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25e78a3270a67447 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0f857fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0f857fa640 1 -- 172.21.15.3:0/2049382727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98027070 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0f857fa640 1 -- 172.21.15.3:0/2049382727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0f9801ae00 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f7d3acf7e0 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0f857fa640 1 -- 172.21.15.3:0/2049382727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f98016070 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa6967740 10 client.17652 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55f7d3dce6b0 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.805+0000 7f0f857fa640 1 -- 172.21.15.3:0/2049382727 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f7d3acf7e0 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0f857fa640 1 -- 172.21.15.3:0/2049382727 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0f9802a030 con 0x55f7d3e693e0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa6967740 10 client.17652 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa6967740 10 client.17652 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa6967740 20 client.17652 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e823c0 0x55f7d3e847e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa6967740 4 client.17652 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f7d3b0c720 con 0x55f7d3e823c0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e823c0 0x55f7d3e847e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0f857fa640 10 client.17652 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.806+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e823c0 0x55f7d3e847e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a09c767afddde27e server_cookie=e1b20275d4ba901b in_seq=0 out_seq=0 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0f857fa640 1 -- 172.21.15.3:0/2049382727 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f7d3b0c720 con 0x55f7d3e823c0 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0f857fa640 10 client.17652 handle_command_reply: tid=0 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 client.17652 shutdown 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 msgr2=0x55f7d3e6f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6f4c0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f0f98009250 tx=0x7f0f9801ac90 comp rx=0 tx=0).stop 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 shutdown_connections 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e823c0 0x55f7d3e847e0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e71f60 0x55f7d3e6fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6f4c0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2049382727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0f857fa640 0 client.17652 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:03.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 >> 172.21.15.3:0/2049382727 conn(0x55f7d3e63100 msgr2=0x55f7d3e64e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 shutdown_connections 2024-02-09T22:57:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 1 -- 172.21.15.3:0/2049382727 wait complete. 2024-02-09T22:57:03.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.807+0000 7f0fa6967740 20 client.17652 trim_cache size 0 max 16384 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 Processor -- start 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 -- start start 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3e79370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3d55130 con 0x55f7d3e68a30 2024-02-09T22:57:03.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3e798b0 con 0x55f7d3e693e0 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa6967740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3e71a30 con 0x55f7d3e63100 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36464/0 (socket says 172.21.15.3:36464) 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2629722956 learned_addr learned my addr 172.21.15.3:0/2629722956 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2629722956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 msgr2=0x55f7d3e79370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3e79370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2629722956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e693e0 msgr2=0x55f7d3e6a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.819+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa1a51640 1 -- 172.21.15.3:0/2629722956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f7d3d6ac90 con 0x55f7d3e68a30 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6a210 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0f98011ea0 tx=0x7f0f98005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4663d2a3eda61e35 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0f877fe640 1 -- 172.21.15.3:0/2629722956 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98016070 con 0x55f7d3e68a30 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0f877fe640 1 -- 172.21.15.3:0/2629722956 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f98017040 con 0x55f7d3e68a30 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0f877fe640 1 -- 172.21.15.3:0/2629722956 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f98027070 con 0x55f7d3e68a30 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 -- 172.21.15.3:0/2629722956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 msgr2=0x55f7d3e6a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6a210 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f0f98011ea0 tx=0x7f0f98005b90 comp rx=0 tx=0).stop 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 -- 172.21.15.3:0/2629722956 shutdown_connections 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e693e0 0x55f7d3e6a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e68a30 0x55f7d3e6a210 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2629722956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3e79370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 -- 172.21.15.3:0/2629722956 >> 172.21.15.3:0/2629722956 conn(0x55f7d3e69b30 msgr2=0x55f7d3e634e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 -- 172.21.15.3:0/2629722956 shutdown_connections 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.820+0000 7f0fa6967740 1 -- 172.21.15.3:0/2629722956 wait complete. 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 -- start start 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 Processor -- start 2024-02-09T22:57:03.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3ed6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3ed6880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3ed0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3ed0d10 con 0x55f7d3e693e0 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3ed0e80 con 0x55f7d3e68a30 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f7d3ed1040 con 0x55f7d3e63100 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa6967740 10 client.0 _ll_register_callbacks cb 0x7f0f98009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa1250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3ed0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa1250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3ed0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36476/0 (socket says 172.21.15.3:36476) 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa1250640 1 -- 172.21.15.3:0/2020448283 learned_addr learned my addr 172.21.15.3:0/2020448283 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3ed6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3ed6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.821+0000 7f0fa1250640 1 -- 172.21.15.3:0/2020448283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 msgr2=0x55f7d3ed6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3ed6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa1250640 1 -- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 msgr2=0x55f7d3ed6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3ed6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa1250640 1 -- 172.21.15.3:0/2020448283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f7d3dce6b0 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3ed6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa1250640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3ed0750 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x55f7d3e85110 tx=0x7f0f8c01d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20b8d106088fc17 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0f857fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0f857fa640 1 -- 172.21.15.3:0/2020448283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f8c02c070 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0f857fa640 1 -- 172.21.15.3:0/2020448283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0f8c005020 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f9801a8f0 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0f857fa640 1 -- 172.21.15.3:0/2020448283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f8c0051e0 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 10 client.17658 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55f7d3d90a70 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0f857fa640 1 -- 172.21.15.3:0/2020448283 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f9801a8f0 con 0x55f7d3e693e0 2024-02-09T22:57:03.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0f857fa640 1 -- 172.21.15.3:0/2020448283 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f0f8c02f030 con 0x55f7d3e693e0 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 10 client.17658 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 10 client.17658 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 20 client.17658 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e71f60 0x55f7d3ee0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 4 client.17658 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.822+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f7d3b5b890 con 0x55f7d3e71f60 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e71f60 0x55f7d3ee0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0fa1a51640 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e71f60 0x55f7d3ee0290 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=85fef67333d26b3b server_cookie=fa051d5e6385ab69 in_seq=0 out_seq=0 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0f857fa640 10 client.17658 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0f857fa640 1 -- 172.21.15.3:0/2020448283 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f7d3b5b890 con 0x55f7d3e71f60 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0f857fa640 10 client.17658 handle_command_reply: tid=0 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0fa6967740 1 client.17658 shutdown 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 msgr2=0x55f7d3ed0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.823+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3ed0750 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x55f7d3e85110 tx=0x7f0f8c01d040 comp rx=0 tx=0).stop 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 shutdown_connections 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f7d3e71f60 0x55f7d3ee0290 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f7d3e68a30 0x55f7d3ed6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f7d3e693e0 0x55f7d3ed0750 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0f857fa640 0 client.17658 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 --2- 172.21.15.3:0/2020448283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f7d3e63100 0x55f7d3ed6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 >> 172.21.15.3:0/2020448283 conn(0x55f7d3e69b30 msgr2=0x55f7d3e6a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 shutdown_connections 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 1 -- 172.21.15.3:0/2020448283 wait complete. 2024-02-09T22:57:03.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa6967740 20 client.17658 trim_cache size 0 max 16384 2024-02-09T22:57:03.834 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:03.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f880a2230 msgr2=0x7f0f880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f880a2230 0x7f0f880a46f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0f940b9d90 tx=0x7f0f940095c0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 msgr2=0x7f0f9c19b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 0x7f0f9c19b490 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7f0f94004af0 tx=0x7f0f94034cc0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 shutdown_connections 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f880a2230 0x7f0f880a46f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f9c105400 0x7f0f9c196cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f9c19afe0 0x7f0f9c19b490 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 --2- 172.21.15.3:0/1237741872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f9c100330 0x7f0f9c196770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 >> 172.21.15.3:0/1237741872 conn(0x7f0f9c077250 msgr2=0x7f0f9c0fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 shutdown_connections 2024-02-09T22:57:03.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.824+0000 7f0fa3cb5640 1 -- 172.21.15.3:0/1237741872 wait complete. 2024-02-09T22:57:03.838 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:03.838 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:57:03.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 -- 172.21.15.3:0/1758240572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 msgr2=0x7fc800100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 --2- 172.21.15.3:0/1758240572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800100720 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009930 tx=0x7fc7f802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:03.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 -- 172.21.15.3:0/1758240572 shutdown_connections 2024-02-09T22:57:03.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 --2- 172.21.15.3:0/1758240572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800100cf0 0x7fc800104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 --2- 172.21.15.3:0/1758240572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc800105410 0x7fc80010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 --2- 172.21.15.3:0/1758240572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800100720 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 -- 172.21.15.3:0/1758240572 >> 172.21.15.3:0/1758240572 conn(0x7fc8000772e0 msgr2=0x7fc8000776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.940+0000 7fc806bd0640 1 -- 172.21.15.3:0/1758240572 shutdown_connections 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 -- 172.21.15.3:0/1758240572 wait complete. 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 Processor -- start 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 -- start start 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc800105410 0x7fc800102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800102850 0x7fc8001a5b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001a60e0 con 0x7fc800100340 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001a6250 con 0x7fc800102850 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc806bd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8001a6410 con 0x7fc800105410 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc80496c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc80496c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800101dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36488/0 (socket says 172.21.15.3:36488) 2024-02-09T22:57:03.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc80496c640 1 -- 172.21.15.3:0/2843193212 learned_addr learned my addr 172.21.15.3:0/2843193212 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.941+0000 7fc80516d640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800102850 0x7fc8001a5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc7fffff640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc800105410 0x7fc800102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80496c640 1 -- 172.21.15.3:0/2843193212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc800105410 msgr2=0x7fc800102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80496c640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc800105410 0x7fc800102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80496c640 1 -- 172.21.15.3:0/2843193212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800102850 msgr2=0x7fc8001a5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80496c640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800102850 0x7fc8001a5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80496c640 1 -- 172.21.15.3:0/2843193212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7f8009560 con 0x7fc800100340 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80516d640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800102850 0x7fc8001a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc80496c640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800101dd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009230 tx=0x7fc7f8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc7fdffb640 1 -- 172.21.15.3:0/2843193212 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8004440 con 0x7fc800100340 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8001a6690 con 0x7fc800100340 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc7fdffb640 1 -- 172.21.15.3:0/2843193212 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f80052d0 con 0x7fc800100340 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc7fdffb640 1 -- 172.21.15.3:0/2843193212 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8003d70 con 0x7fc800100340 2024-02-09T22:57:03.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.942+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8001a6bf0 con 0x7fc800100340 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.944+0000 7fc7fdffb640 1 -- 172.21.15.3:0/2843193212 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc7f8005440 con 0x7fc800100340 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 Processor -- start 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 -- start start 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc7fdffb640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc7ec0a2230 0x7fc7ec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 0x55e017a58e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a593e0 0x55e017a61990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc7fdffb640 1 -- 172.21.15.3:0/2843193212 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc7f80f02a0 con 0x7fc800100340 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017a14020 con 0x55e017a58a30 2024-02-09T22:57:03.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80516d640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc7ec0a2230 0x7fc7ec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0179c7320 con 0x55e017a593e0 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc8097a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0179c4850 con 0x55e017a61f60 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 0x55e017a58e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 0x55e017a58e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36494/0 (socket says 172.21.15.3:36494) 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 -- 172.21.15.3:0/1028390445 learned_addr learned my addr 172.21.15.3:0/1028390445 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc7fffff640 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a593e0 0x55e017a61990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 -- 172.21.15.3:0/1028390445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 msgr2=0x55e017a643b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a643b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 -- 172.21.15.3:0/1028390445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a593e0 msgr2=0x55e017a61990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a593e0 0x55e017a61990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc80496c640 1 -- 172.21.15.3:0/1028390445 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e017982210 con 0x55e017a58a30 2024-02-09T22:57:03.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.945+0000 7fc7fffff640 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a593e0 0x55e017a61990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc80516d640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc7ec0a2230 0x7fc7ec0a46f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8005d80 tx=0x7fc7e8005cf0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc80496c640 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 0x55e017a58e10 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80bcd60 tx=0x7fc7f8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=360c9c301392709d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc7f6ffd640 1 -- 172.21.15.3:0/1028390445 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80c0f20 con 0x55e017a58a30 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc7f6ffd640 1 -- 172.21.15.3:0/1028390445 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f80c5570 con 0x55e017a58a30 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc7f6ffd640 1 -- 172.21.15.3:0/1028390445 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80c50e0 con 0x55e017a58a30 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc8097a6740 1 -- 172.21.15.3:0/1028390445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 msgr2=0x55e017a58e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.946+0000 7fc8097a6740 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 0x55e017a58e10 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80bcd60 tx=0x7fc7f8005fd0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- 172.21.15.3:0/1028390445 shutdown_connections 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a593e0 0x55e017a61990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a643b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 --2- 172.21.15.3:0/1028390445 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a58a30 0x55e017a58e10 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- 172.21.15.3:0/1028390445 >> 172.21.15.3:0/1028390445 conn(0x55e017a53100 msgr2=0x55e017a534e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- 172.21.15.3:0/1028390445 shutdown_connections 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- 172.21.15.3:0/1028390445 wait complete. 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- start start 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 Processor -- start 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a5f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e0179c0030 con 0x55e017a593e0 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017693430 con 0x55e017a58a30 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e01764fe00 con 0x55e017a61f60 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc8097a6740 10 client.0 _ll_register_callbacks cb 0x55e0179b5030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc7fffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc7fffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5f420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36500/0 (socket says 172.21.15.3:36500) 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc7fffff640 1 -- 172.21.15.3:0/434733941 learned_addr learned my addr 172.21.15.3:0/434733941 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.947+0000 7fc80516d640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a5f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7fffff640 1 -- 172.21.15.3:0/434733941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 msgr2=0x55e017a5f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7fffff640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a5f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc80496c640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7fffff640 1 -- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 msgr2=0x55e017a5eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7fffff640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7fffff640 1 -- 172.21.15.3:0/434733941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8001a6bf0 con 0x55e017a593e0 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7fffff640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5f420 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0005fd0 tx=0x7fc7f0004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a88498e890ae436 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7dffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7dffff640 1 -- 172.21.15.3:0/434733941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f0011780 con 0x55e017a593e0 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7dffff640 1 -- 172.21.15.3:0/434733941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc7f00118e0 con 0x55e017a593e0 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e0176bf7e0 con 0x55e017a593e0 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7dffff640 1 -- 172.21.15.3:0/434733941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7f0011aa0 con 0x55e017a593e0 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc8097a6740 10 client.17670 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e0179be6b0 con 0x55e017a593e0 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.948+0000 7fc7dffff640 1 -- 172.21.15.3:0/434733941 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e0176bf7e0 con 0x55e017a593e0 2024-02-09T22:57:03.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc7dffff640 1 -- 172.21.15.3:0/434733941 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc7f00244e0 con 0x55e017a593e0 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc8097a6740 10 client.17670 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc8097a6740 10 client.17670 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc8097a6740 20 client.17670 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc8097a6740 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a722d0 0x55e017a746f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc8097a6740 4 client.17670 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e0176fc720 con 0x55e017a722d0 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc80496c640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a722d0 0x55e017a746f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc7dffff640 10 client.17670 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:03.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.949+0000 7fc80496c640 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a722d0 0x55e017a746f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b97f55ac47a8d4d4 server_cookie=8628a79c15879c40 in_seq=0 out_seq=0 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc7dffff640 1 -- 172.21.15.3:0/434733941 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e0176fc720 con 0x55e017a722d0 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc7dffff640 10 client.17670 handle_command_reply: tid=0 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 client.17670 shutdown 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 msgr2=0x55e017a5f420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5f420 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0005fd0 tx=0x7fc7f0004410 comp rx=0 tx=0).stop 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 shutdown_connections 2024-02-09T22:57:03.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a722d0 0x55e017a746f0 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a61f60 0x55e017a5f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5f420 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 --2- 172.21.15.3:0/434733941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 >> 172.21.15.3:0/434733941 conn(0x55e017a53100 msgr2=0x55e017a54e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc7dffff640 0 client.17670 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 shutdown_connections 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 1 -- 172.21.15.3:0/434733941 wait complete. 2024-02-09T22:57:03.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.950+0000 7fc8097a6740 20 client.17670 trim_cache size 0 max 16384 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 Processor -- start 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 -- start start 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a53100 0x55e017a61610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017945130 con 0x55e017a593e0 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017a529c0 con 0x55e017a58a30 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc8097a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017a6ed90 con 0x55e017a53100 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5a210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36506/0 (socket says 172.21.15.3:36506) 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 -- 172.21.15.3:0/3823369061 learned_addr learned my addr 172.21.15.3:0/3823369061 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80496c640 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a53100 0x55e017a61610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 -- 172.21.15.3:0/3823369061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a53100 msgr2=0x55e017a61610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a53100 0x55e017a61610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 -- 172.21.15.3:0/3823369061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 msgr2=0x55e017a5f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc7fffff640 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.962+0000 7fc80516d640 1 -- 172.21.15.3:0/3823369061 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e01795ac90 con 0x55e017a593e0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc80516d640 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5a210 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800bfb0 tx=0x7fc7e8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c5b22d1f52a3780 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc7f67fc640 1 -- 172.21.15.3:0/3823369061 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e8005220 con 0x55e017a593e0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc7f67fc640 1 -- 172.21.15.3:0/3823369061 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7e800b3e0 con 0x55e017a593e0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc7f67fc640 1 -- 172.21.15.3:0/3823369061 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7e801f940 con 0x55e017a593e0 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 -- 172.21.15.3:0/3823369061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 msgr2=0x55e017a5a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5a210 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fc7e800bfb0 tx=0x7fc7e8012000 comp rx=0 tx=0).stop 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 -- 172.21.15.3:0/3823369061 shutdown_connections 2024-02-09T22:57:03.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a5f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a593e0 0x55e017a5a210 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 --2- 172.21.15.3:0/3823369061 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a53100 0x55e017a61610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 -- 172.21.15.3:0/3823369061 >> 172.21.15.3:0/3823369061 conn(0x55e017a59b30 msgr2=0x55e017a79050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 -- 172.21.15.3:0/3823369061 shutdown_connections 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.963+0000 7fc8097a6740 1 -- 172.21.15.3:0/3823369061 wait complete. 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 -- start start 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 Processor -- start 2024-02-09T22:57:03.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 0x55e017a67750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a67c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a593e0 0x55e017a681d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017a696a0 con 0x55e017a53100 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017ac87e0 con 0x55e017a58a30 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e017ac8950 con 0x55e017a593e0 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc8097a6740 10 client.0 _ll_register_callbacks cb 0x7fc7f8005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc80496c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 0x55e017a67750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc80496c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 0x55e017a67750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36516/0 (socket says 172.21.15.3:36516) 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc80496c640 1 -- 172.21.15.3:0/4071131014 learned_addr learned my addr 172.21.15.3:0/4071131014 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc7fffff640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a67c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.964+0000 7fc80516d640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a593e0 0x55e017a681d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc80496c640 1 -- 172.21.15.3:0/4071131014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a593e0 msgr2=0x55e017a681d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc80496c640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a593e0 0x55e017a681d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc80496c640 1 -- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 msgr2=0x55e017a67c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc80496c640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a67c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc80496c640 1 -- 172.21.15.3:0/4071131014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e800bcb0 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc80496c640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 0x55e017a67750 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b9230 tx=0x7fc7f802bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26979b223c35e8b6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc7dffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc7dffff640 1 -- 172.21.15.3:0/4071131014 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7f80c0f20 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc7dffff640 1 -- 172.21.15.3:0/4071131014 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc7f80da070 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc7f80c5570 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc7dffff640 1 -- 172.21.15.3:0/4071131014 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc7f80bb0d0 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 10 client.17676 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x55e017980a70 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc7dffff640 1 -- 172.21.15.3:0/4071131014 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7f80c5570 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc7dffff640 1 -- 172.21.15.3:0/4071131014 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fc7f803c070 con 0x55e017a53100 2024-02-09T22:57:03.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 10 client.17676 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 10 client.17676 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 20 client.17676 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a61f60 0x55e017acbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 4 client.17676 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.965+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e01774b890 con 0x55e017a61f60 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc7fffff640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a61f60 0x55e017acbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc7dffff640 10 client.17676 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc7fffff640 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a61f60 0x55e017acbdb0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=75d877d655e728a1 server_cookie=d178c12ed7c64bf1 in_seq=0 out_seq=0 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc7dffff640 1 -- 172.21.15.3:0/4071131014 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e01774b890 con 0x55e017a61f60 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc7dffff640 10 client.17676 handle_command_reply: tid=0 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 client.17676 shutdown 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 msgr2=0x55e017a67750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 0x55e017a67750 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80b9230 tx=0x7fc7f802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 shutdown_connections 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e017a61f60 0x55e017acbdb0 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e017a58a30 0x55e017a67c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc7dffff640 0 client.17676 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e017a593e0 0x55e017a681d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 --2- 172.21.15.3:0/4071131014 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e017a53100 0x55e017a67750 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 >> 172.21.15.3:0/4071131014 conn(0x55e017a59b30 msgr2=0x55e017a5a210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 shutdown_connections 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 1 -- 172.21.15.3:0/4071131014 wait complete. 2024-02-09T22:57:03.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.966+0000 7fc8097a6740 20 client.17676 trim_cache size 0 max 16384 2024-02-09T22:57:03.978 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:03.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc7ec0a2230 msgr2=0x7fc7ec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc7ec0a2230 0x7fc7ec0a46f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc7e8005d80 tx=0x7fc7e8005cf0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 msgr2=0x7fc800101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800101dd0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7fc7f8009230 tx=0x7fc7f8004040 comp rx=0 tx=0).stop 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 shutdown_connections 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc7ec0a2230 0x7fc7ec0a46f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc800102850 0x7fc8001a5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc800105410 0x7fc800102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 --2- 172.21.15.3:0/2843193212 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc800100340 0x7fc800101dd0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 >> 172.21.15.3:0/2843193212 conn(0x7fc8000772e0 msgr2=0x7fc8000688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 shutdown_connections 2024-02-09T22:57:03.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:03.967+0000 7fc806bd0640 1 -- 172.21.15.3:0/2843193212 wait complete. 2024-02-09T22:57:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:04 smithi089 ceph-mon[36665]: pgmap v177: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 616 KiB/s wr, 116 op/s 2024-02-09T22:57:04.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:04 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/854075721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:57:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:04 smithi003 ceph-mon[32678]: pgmap v177: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 616 KiB/s wr, 116 op/s 2024-02-09T22:57:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:04 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/854075721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:57:04.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 168424 2.71 MB/sec execute 178 sec latency 954.026 ms 2024-02-09T22:57:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:04 smithi064 ceph-mon[36638]: pgmap v177: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 616 KiB/s wr, 116 op/s 2024-02-09T22:57:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:04 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/854075721' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:57:04.981 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:57:04.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:57:04.982 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:57:05.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 -- 172.21.15.3:0/822824002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc100cc0 msgr2=0x7f79fc104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 --2- 172.21.15.3:0/822824002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc100cc0 0x7f79fc104d00 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009930 tx=0x7f79ec02bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 -- 172.21.15.3:0/822824002 shutdown_connections 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 --2- 172.21.15.3:0/822824002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79fc1053e0 0x7f79fc10cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 --2- 172.21.15.3:0/822824002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc100cc0 0x7f79fc104d00 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 --2- 172.21.15.3:0/822824002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79fc100310 0x7f79fc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 -- 172.21.15.3:0/822824002 >> 172.21.15.3:0/822824002 conn(0x7f79fc077230 msgr2=0x7f79fc077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 -- 172.21.15.3:0/822824002 shutdown_connections 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.099+0000 7f7a03be2640 1 -- 172.21.15.3:0/822824002 wait complete. 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 Processor -- start 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 -- start start 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79fc100310 0x7f79fc19d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 0x7f79fc19db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79fc197690 0x7f79fc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc198160 con 0x7f79fc1053e0 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc1982d0 con 0x7f79fc100310 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a03be2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79fc198490 con 0x7f79fc197690 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 0x7f79fc19db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 0x7f79fc19db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36546/0 (socket says 172.21.15.3:36546) 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 -- 172.21.15.3:0/683216507 learned_addr learned my addr 172.21.15.3:0/683216507 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0197e640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79fc100310 0x7f79fc19d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0217f640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79fc197690 0x7f79fc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 -- 172.21.15.3:0/683216507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79fc197690 msgr2=0x7f79fc197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79fc197690 0x7f79fc197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 -- 172.21.15.3:0/683216507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79fc100310 msgr2=0x7f79fc19d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.100+0000 7f7a0117d640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79fc100310 0x7f79fc19d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f7a0117d640 1 -- 172.21.15.3:0/683216507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79ec009560 con 0x7f79fc1053e0 2024-02-09T22:57:05.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f7a0117d640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 0x7f79fc19db10 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009230 tx=0x7f79ec004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f79f2ffd640 1 -- 172.21.15.3:0/683216507 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec0043d0 con 0x7f79fc1053e0 2024-02-09T22:57:05.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f79f2ffd640 1 -- 172.21.15.3:0/683216507 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79ec0052d0 con 0x7f79fc1053e0 2024-02-09T22:57:05.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79fc1a43d0 con 0x7f79fc1053e0 2024-02-09T22:57:05.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f79f2ffd640 1 -- 172.21.15.3:0/683216507 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec003d80 con 0x7f79fc1053e0 2024-02-09T22:57:05.107 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.101+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79fc1a4900 con 0x7f79fc1053e0 2024-02-09T22:57:05.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.103+0000 7f79f2ffd640 1 -- 172.21.15.3:0/683216507 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f79ec005440 con 0x7f79fc1053e0 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 Processor -- start 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 -- start start 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d52260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 0x562273d5f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f79f2ffd640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79e80a22b0 0x7f79e80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d5faa0 0x562273d61e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273cc0000 con 0x562273d527a0 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273cbd530 con 0x562273d5faa0 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a067b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273cb8d10 con 0x562273d51e80 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a0197e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 0x562273d5f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a0117d640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79e80a22b0 0x7f79e80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a0197e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 0x562273d5f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36554/0 (socket says 172.21.15.3:36554) 2024-02-09T22:57:05.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a0197e640 1 -- 172.21.15.3:0/3205762639 learned_addr learned my addr 172.21.15.3:0/3205762639 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a0217f640 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d5faa0 0x562273d61e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f79f2ffd640 1 -- 172.21.15.3:0/683216507 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f79ec0f00b0 con 0x7f79fc1053e0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.104+0000 7f7a0217f640 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d52260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0197e640 1 -- 172.21.15.3:0/3205762639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 msgr2=0x562273d52260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0197e640 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d52260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0197e640 1 -- 172.21.15.3:0/3205762639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d5faa0 msgr2=0x562273d61e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0197e640 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d5faa0 0x562273d61e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0197e640 1 -- 172.21.15.3:0/3205762639 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562273c5f210 con 0x562273d527a0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0197e640 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 0x562273d5f560 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f79f800bcd0 tx=0x7f79f80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d01ae97b6d451627 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f7a0117d640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79e80a22b0 0x7f79e80a4770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0ede30 tx=0x7f79ec03af10 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f79e7fff640 1 -- 172.21.15.3:0/3205762639 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8005020 con 0x562273d527a0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f79e7fff640 1 -- 172.21.15.3:0/3205762639 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79f8015070 con 0x562273d527a0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.105+0000 7f79e7fff640 1 -- 172.21.15.3:0/3205762639 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8011640 con 0x562273d527a0 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 -- 172.21.15.3:0/3205762639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 msgr2=0x562273d5f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 0x562273d5f560 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7f79f800bcd0 tx=0x7f79f80107a0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 -- 172.21.15.3:0/3205762639 shutdown_connections 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d5faa0 0x562273d61e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d527a0 0x562273d5f560 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3205762639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d52260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 -- 172.21.15.3:0/3205762639 >> 172.21.15.3:0/3205762639 conn(0x562273d4c8c0 msgr2=0x562273d4cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 -- 172.21.15.3:0/3205762639 shutdown_connections 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 -- 172.21.15.3:0/3205762639 wait complete. 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 -- start start 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.106+0000 7f7a067b2740 1 Processor -- start 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d51e80 0x562273d5ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d527a0 0x562273d5cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273c5de60 con 0x562273d5faa0 2024-02-09T22:57:05.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56227398c4b0 con 0x562273d51e80 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273c3e670 con 0x562273d527a0 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a067b2740 10 client.0 _ll_register_callbacks cb 0x562273c56190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d51e80 0x562273d5ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0197e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d527a0 0x562273d5cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36558/0 (socket says 172.21.15.3:36558) 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0197e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d527a0 0x562273d5cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42428/0 (socket says 172.21.15.3:42428) 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 -- 172.21.15.3:0/930199163 learned_addr learned my addr 172.21.15.3:0/930199163 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 -- 172.21.15.3:0/930199163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d527a0 msgr2=0x562273d5cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d527a0 0x562273d5cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 -- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d51e80 msgr2=0x562273d5ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d51e80 0x562273d5ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.107+0000 7f7a0217f640 1 -- 172.21.15.3:0/930199163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f800b900 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a0217f640 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5d490 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f79f4012e60 tx=0x7f79f40099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c729b31fea5869 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f79e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f79e57fa640 1 -- 172.21.15.3:0/930199163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4010ed0 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f79e57fa640 1 -- 172.21.15.3:0/930199163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f79f4004b20 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5622739b8860 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f79e57fa640 1 -- 172.21.15.3:0/930199163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79f4004ce0 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 10 client.17688 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x562273cb7390 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f79e57fa640 1 -- 172.21.15.3:0/930199163 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5622739b8860 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f79e57fa640 1 -- 172.21.15.3:0/930199163 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f79f40046a0 con 0x562273d5faa0 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 10 client.17688 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 10 client.17688 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 20 client.17688 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:05.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273d6fb80 0x562273d71fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 4 client.17688 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.108+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562273a09410 con 0x562273d6fb80 2024-02-09T22:57:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.109+0000 7f7a0197e640 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273d6fb80 0x562273d71fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.109+0000 7f79e57fa640 10 client.17688 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.109+0000 7f7a0197e640 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273d6fb80 0x562273d71fa0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f471e166ef46a9a6 server_cookie=eb9ddb7662f2d793 in_seq=0 out_seq=0 2024-02-09T22:57:05.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.110+0000 7f79e57fa640 1 -- 172.21.15.3:0/930199163 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562273a09410 con 0x562273d6fb80 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.110+0000 7f79e57fa640 10 client.17688 handle_command_reply: tid=0 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.110+0000 7f7a067b2740 1 client.17688 shutdown 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.110+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 msgr2=0x562273d5d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.110+0000 7f7a067b2740 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5d490 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f79f4012e60 tx=0x7f79f40099c0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 shutdown_connections 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273d6fb80 0x562273d71fa0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5d490 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d527a0 0x562273d5cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f79e57fa640 0 client.17688 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 --2- 172.21.15.3:0/930199163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d51e80 0x562273d5ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 >> 172.21.15.3:0/930199163 conn(0x562273d4c8c0 msgr2=0x562273d4e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 shutdown_connections 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 1 -- 172.21.15.3:0/930199163 wait complete. 2024-02-09T22:57:05.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.111+0000 7f7a067b2740 20 client.17688 trim_cache size 0 max 16384 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 Processor -- start 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 -- start start 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d5f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273d59c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273d480a0 con 0x562273d5faa0 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273d62770 con 0x562273d527a0 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a067b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273d6c5c0 con 0x562273d51e80 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d5f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d5f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42440/0 (socket says 172.21.15.3:42440) 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0197e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273d59c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 -- 172.21.15.3:0/3143309042 learned_addr learned my addr 172.21.15.3:0/3143309042 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 -- 172.21.15.3:0/3143309042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 msgr2=0x562273d59c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273d59c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 -- 172.21.15.3:0/3143309042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 msgr2=0x562273d5a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.127+0000 7f7a0217f640 1 -- 172.21.15.3:0/3143309042 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562273c53c70 con 0x562273d51e80 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.128+0000 7f7a0217f640 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d5f140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f79f4005fd0 tx=0x7f79f4000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1930bc64b87e7fb0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.128+0000 7f79e77fe640 1 -- 172.21.15.3:0/3143309042 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4012540 con 0x562273d51e80 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.128+0000 7f79e77fe640 1 -- 172.21.15.3:0/3143309042 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79f40215e0 con 0x562273d51e80 2024-02-09T22:57:05.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.128+0000 7f79e77fe640 1 -- 172.21.15.3:0/3143309042 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4012330 con 0x562273d51e80 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- 172.21.15.3:0/3143309042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 msgr2=0x562273d5f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d5f140 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f79f4005fd0 tx=0x7f79f4000f40 comp rx=0 tx=0).stop 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- 172.21.15.3:0/3143309042 shutdown_connections 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273d59c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d5faa0 0x562273d5a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- 172.21.15.3:0/3143309042 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d51e80 0x562273d5f140 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- 172.21.15.3:0/3143309042 >> 172.21.15.3:0/3143309042 conn(0x562273d4c8c0 msgr2=0x562273d57d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- 172.21.15.3:0/3143309042 shutdown_connections 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- 172.21.15.3:0/3143309042 wait complete. 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- start start 2024-02-09T22:57:05.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 Processor -- start 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 0x562273dc0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273dc0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d5faa0 0x562273d65da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273d66360 con 0x562273d51e80 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273d664d0 con 0x562273d527a0 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562273d66690 con 0x562273d5faa0 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.129+0000 7f7a067b2740 10 client.0 _ll_register_callbacks cb 0x7f79f800b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 0x562273dc0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 0x562273dc0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36578/0 (socket says 172.21.15.3:36578) 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 -- 172.21.15.3:0/2816123779 learned_addr learned my addr 172.21.15.3:0/2816123779 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0197e640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273dc0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d5faa0 0x562273d65da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 -- 172.21.15.3:0/2816123779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d5faa0 msgr2=0x562273d65da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d5faa0 0x562273d65da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 -- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 msgr2=0x562273dc0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273dc0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 -- 172.21.15.3:0/2816123779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562273cb7390 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0197e640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273dc0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a0217f640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 0x562273dc0540 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009a40 tx=0x7f79f4021470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5dfcfec7a1ad733 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f79e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f79e57fa640 1 -- 172.21.15.3:0/2816123779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f4021750 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f79e57fa640 1 -- 172.21.15.3:0/2816123779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f79f402a430 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79f8015070 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f79e57fa640 1 -- 172.21.15.3:0/2816123779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79f402a5f0 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a067b2740 10 client.17694 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.130+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x562273c79ac0 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f79e57fa640 1 -- 172.21.15.3:0/2816123779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79f8015070 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f79e57fa640 1 -- 172.21.15.3:0/2816123779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f79f4034040 con 0x562273d51e80 2024-02-09T22:57:05.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a067b2740 10 client.17694 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a067b2740 10 client.17694 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a067b2740 20 client.17694 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a067b2740 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273dc7f80 0x562273dca3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a067b2740 4 client.17694 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5622739f57b0 con 0x562273dc7f80 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.131+0000 7f7a0197e640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273dc7f80 0x562273dca3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f79e57fa640 10 client.17694 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f7a0197e640 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273dc7f80 0x562273dca3a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eff823a157fa3cd server_cookie=dff2da8e820dd08b in_seq=0 out_seq=0 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f79e57fa640 1 -- 172.21.15.3:0/2816123779 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5622739f57b0 con 0x562273dc7f80 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f79e57fa640 10 client.17694 handle_command_reply: tid=0 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f7a067b2740 1 client.17694 shutdown 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 msgr2=0x562273dc0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.132+0000 7f7a067b2740 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 0x562273dc0540 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7f79f4009a40 tx=0x7f79f4021470 comp rx=0 tx=0).stop 2024-02-09T22:57:05.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 shutdown_connections 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562273dc7f80 0x562273dca3a0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562273d527a0 0x562273dc0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562273d5faa0 0x562273d65da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f79e57fa640 0 client.17694 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 --2- 172.21.15.3:0/2816123779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562273d51e80 0x562273dc0540 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 >> 172.21.15.3:0/2816123779 conn(0x562273d4c8c0 msgr2=0x562273d4b090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 shutdown_connections 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 1 -- 172.21.15.3:0/2816123779 wait complete. 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a067b2740 20 client.17694 trim_cache size 0 max 16384 2024-02-09T22:57:05.141 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:57:05.142 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:57:05.142 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "2cb94a79-ffaf-4599-9f08-bfd7b54b5c42", 2024-02-09T22:57:05.142 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:57:05.142 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:57:05.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79e80a22b0 msgr2=0x7f79e80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79e80a22b0 0x7f79e80a4770 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f79ec0ede30 tx=0x7f79ec03af10 comp rx=0 tx=0).stop 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 msgr2=0x7f79fc19db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 0x7f79fc19db10 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7f79ec009230 tx=0x7f79ec004c00 comp rx=0 tx=0).stop 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 shutdown_connections 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f79e80a22b0 0x7f79e80a4770 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79fc197690 0x7f79fc197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f79fc1053e0 0x7f79fc19db10 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 --2- 172.21.15.3:0/683216507 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f79fc100310 0x7f79fc19d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 >> 172.21.15.3:0/683216507 conn(0x7f79fc077230 msgr2=0x7f79fc0ff860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 shutdown_connections 2024-02-09T22:57:05.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.133+0000 7f7a03be2640 1 -- 172.21.15.3:0/683216507 wait complete. 2024-02-09T22:57:05.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:57:05.148 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:57:05.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.256+0000 7fee27264640 1 -- 172.21.15.3:0/3468610462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20100330 msgr2=0x7fee20100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.256+0000 7fee27264640 1 --2- 172.21.15.3:0/3468610462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20100330 0x7fee20100710 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fee14009930 tx=0x7fee1402bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- 172.21.15.3:0/3468610462 shutdown_connections 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 --2- 172.21.15.3:0/3468610462 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20105400 0x7fee2010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 --2- 172.21.15.3:0/3468610462 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee20100ce0 0x7fee20104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 --2- 172.21.15.3:0/3468610462 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20100330 0x7fee20100710 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- 172.21.15.3:0/3468610462 >> 172.21.15.3:0/3468610462 conn(0x7fee20077250 msgr2=0x7fee20077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- 172.21.15.3:0/3468610462 shutdown_connections 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- 172.21.15.3:0/3468610462 wait complete. 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 Processor -- start 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- start start 2024-02-09T22:57:05.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20100330 0x7fee2019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 0x7fee2019db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee201976b0 0x7fee20197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee20198180 con 0x7fee20105400 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201982f0 con 0x7fee20100330 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.257+0000 7fee27264640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201984b0 con 0x7fee201976b0 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 0x7fee2019db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee25801640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee201976b0 0x7fee20197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee25000640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20100330 0x7fee2019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 0x7fee2019db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36592/0 (socket says 172.21.15.3:36592) 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 -- 172.21.15.3:0/1473499703 learned_addr learned my addr 172.21.15.3:0/1473499703 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 -- 172.21.15.3:0/1473499703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee201976b0 msgr2=0x7fee20197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee201976b0 0x7fee20197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 -- 172.21.15.3:0/1473499703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20100330 msgr2=0x7fee2019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.263 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20100330 0x7fee2019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 -- 172.21.15.3:0/1473499703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee14009560 con 0x7fee20105400 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee25000640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20100330 0x7fee2019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1ffff640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 0x7fee2019db30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e9c0 tx=0x7fee0c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1dffb640 1 -- 172.21.15.3:0/1473499703 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c009700 con 0x7fee20105400 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee201a4450 con 0x7fee20105400 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1dffb640 1 -- 172.21.15.3:0/1473499703 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee0c00cd80 con 0x7fee20105400 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee1dffb640 1 -- 172.21.15.3:0/1473499703 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c010430 con 0x7fee20105400 2024-02-09T22:57:05.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.258+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee201a4980 con 0x7fee20105400 2024-02-09T22:57:05.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.260+0000 7fee1dffb640 1 -- 172.21.15.3:0/1473499703 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fee0c005000 con 0x7fee20105400 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee1dffb640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee100a2230 0x7fee100a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 Processor -- start 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 -- start start 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee1dffb640 1 -- 172.21.15.3:0/1473499703 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fee0c014070 con 0x7fee20105400 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee25000640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee100a2230 0x7fee100a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x564698120d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981298e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x56469812c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646980dbd00 con 0x564698120980 2024-02-09T22:57:05.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5646980902f0 con 0x564698129eb0 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee29e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56469808d820 con 0x564698121330 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981298e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981298e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42474/0 (socket says 172.21.15.3:42474) 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee1ffff640 1 -- 172.21.15.3:0/3719827900 learned_addr learned my addr 172.21.15.3:0/3719827900 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.261+0000 7fee25801640 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x56469812c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee25000640 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x564698120d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee25000640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee100a2230 0x7fee100a46f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fee14009900 tx=0x7fee14004b10 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee1ffff640 1 -- 172.21.15.3:0/3719827900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 msgr2=0x56469812c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee1ffff640 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x56469812c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee1ffff640 1 -- 172.21.15.3:0/3719827900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 msgr2=0x564698120d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee1ffff640 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x564698120d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee1ffff640 1 -- 172.21.15.3:0/3719827900 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56469804aff0 con 0x564698121330 2024-02-09T22:57:05.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee25801640 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x56469812c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee1ffff640 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981298e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00f010 tx=0x7fee0c09b5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=acd261329c8fae8f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee0affd640 1 -- 172.21.15.3:0/3719827900 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0a3640 con 0x564698121330 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee0affd640 1 -- 172.21.15.3:0/3719827900 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee0c0046c0 con 0x564698121330 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.262+0000 7fee0affd640 1 -- 172.21.15.3:0/3719827900 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0c0052b0 con 0x564698121330 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 -- 172.21.15.3:0/3719827900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 msgr2=0x5646981298e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981298e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00f010 tx=0x7fee0c09b5a0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 -- 172.21.15.3:0/3719827900 shutdown_connections 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x56469812c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981298e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 --2- 172.21.15.3:0/3719827900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x564698120d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 -- 172.21.15.3:0/3719827900 >> 172.21.15.3:0/3719827900 conn(0x56469811b050 msgr2=0x56469811b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 -- 172.21.15.3:0/3719827900 shutdown_connections 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 -- 172.21.15.3:0/3719827900 wait complete. 2024-02-09T22:57:05.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 -- start start 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.263+0000 7fee29e38740 1 Processor -- start 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698120980 0x564698126da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981272e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 0x564698127820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564698089000 con 0x564698129eb0 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564697d5c430 con 0x564698120980 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564697d18e00 con 0x564698121330 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 10 client.0 _ll_register_callbacks cb 0x56469807d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 0x564698127820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 0x564698127820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36616/0 (socket says 172.21.15.3:36616) 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 -- 172.21.15.3:0/2952580606 learned_addr learned my addr 172.21.15.3:0/2952580606 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25000640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698120980 0x564698126da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee1ffff640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981272e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 -- 172.21.15.3:0/2952580606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 msgr2=0x5646981272e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981272e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 -- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698120980 msgr2=0x564698126da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698120980 0x564698126da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 -- 172.21.15.3:0/2952580606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee201a4980 con 0x564698129eb0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee1ffff640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981272e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee25801640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 0x564698127820 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fee18013d00 tx=0x7fee180117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648e17b347352221 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fedfffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fedfffff640 1 -- 172.21.15.3:0/2952580606 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee18003970 con 0x564698129eb0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fedfffff640 1 -- 172.21.15.3:0/2952580606 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fee18003ad0 con 0x564698129eb0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.264+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564697d887e0 con 0x564698129eb0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fedfffff640 1 -- 172.21.15.3:0/2952580606 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee18003cd0 con 0x564698129eb0 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 10 client.17706 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:05.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564698087680 con 0x564698129eb0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fedfffff640 1 -- 172.21.15.3:0/2952580606 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564697d887e0 con 0x564698129eb0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fedfffff640 1 -- 172.21.15.3:0/2952580606 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fee1801b580 con 0x564698129eb0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 10 client.17706 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 10 client.17706 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 20 client.17706 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56469813a4c0 0x56469813c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 4 client.17706 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564697dc5720 con 0x56469813a4c0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.265+0000 7fee25000640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56469813a4c0 0x56469813c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fedfffff640 10 client.17706 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee25000640 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56469813a4c0 0x56469813c8e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4633c556ca5f9 server_cookie=e8ba36fb8ffa580e in_seq=0 out_seq=0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fedfffff640 1 -- 172.21.15.3:0/2952580606 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564697dc5720 con 0x56469813a4c0 2024-02-09T22:57:05.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fedfffff640 10 client.17706 handle_command_reply: tid=0 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 client.17706 shutdown 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 msgr2=0x564698127820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 0x564698127820 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fee18013d00 tx=0x7fee180117b0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 shutdown_connections 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56469813a4c0 0x56469813c8e0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698129eb0 0x564698127820 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698121330 0x5646981272e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 --2- 172.21.15.3:0/2952580606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698120980 0x564698126da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fedfffff640 0 client.17706 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.266+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 >> 172.21.15.3:0/2952580606 conn(0x56469811b050 msgr2=0x56469811cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.267+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 shutdown_connections 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.267+0000 7fee29e38740 1 -- 172.21.15.3:0/2952580606 wait complete. 2024-02-09T22:57:05.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.267+0000 7fee29e38740 20 client.17706 trim_cache size 0 max 16384 2024-02-09T22:57:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 Processor -- start 2024-02-09T22:57:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 -- start start 2024-02-09T22:57:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x5646981294d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698121330 0x564698124000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698129eb0 0x564698124540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56469800dfd0 con 0x564698120980 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564697d88810 con 0x564698121330 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.282+0000 7fee29e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56469811fca0 con 0x564698129eb0 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x5646981294d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x5646981294d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36618/0 (socket says 172.21.15.3:36618) 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 -- 172.21.15.3:0/77680047 learned_addr learned my addr 172.21.15.3:0/77680047 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25801640 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698129eb0 0x564698124540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee1ffff640 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698121330 0x564698124000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 -- 172.21.15.3:0/77680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698129eb0 msgr2=0x564698124540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698129eb0 0x564698124540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 -- 172.21.15.3:0/77680047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698121330 msgr2=0x564698124000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698121330 0x564698124000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 -- 172.21.15.3:0/77680047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564698023b30 con 0x564698120980 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee25000640 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x5646981294d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fee14006010 tx=0x7fee14043040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0f8f0f3cb45662 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee0a7fc640 1 -- 172.21.15.3:0/77680047 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14052ea0 con 0x564698120980 2024-02-09T22:57:05.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee0a7fc640 1 -- 172.21.15.3:0/77680047 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee140526b0 con 0x564698120980 2024-02-09T22:57:05.288 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.283+0000 7fee0a7fc640 1 -- 172.21.15.3:0/77680047 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee140051c0 con 0x564698120980 2024-02-09T22:57:05.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 -- 172.21.15.3:0/77680047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 msgr2=0x5646981294d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x5646981294d0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fee14006010 tx=0x7fee14043040 comp rx=0 tx=0).stop 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 -- 172.21.15.3:0/77680047 shutdown_connections 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698121330 0x564698124000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698129eb0 0x564698124540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 --2- 172.21.15.3:0/77680047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698120980 0x5646981294d0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 -- 172.21.15.3:0/77680047 >> 172.21.15.3:0/77680047 conn(0x56469811b050 msgr2=0x564698125670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 -- 172.21.15.3:0/77680047 shutdown_connections 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 -- 172.21.15.3:0/77680047 wait complete. 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 -- start start 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.284+0000 7fee29e38740 1 Processor -- start 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 0x56469812fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698121330 0x5646981301f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x5646981909d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564698130ad0 con 0x564698121330 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564698190f10 con 0x564698129eb0 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564698191050 con 0x564698120980 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee29e38740 10 client.0 _ll_register_callbacks cb 0x7fee0c097140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 0x56469812fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee1ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698121330 0x5646981301f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25801640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x5646981909d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 0x56469812fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:42498/0 (socket says 172.21.15.3:42498) 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 -- 172.21.15.3:0/3071652562 learned_addr learned my addr 172.21.15.3:0/3071652562 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 -- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 msgr2=0x5646981909d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x5646981909d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 -- 172.21.15.3:0/3071652562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698121330 msgr2=0x5646981301f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698121330 0x5646981301f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.285+0000 7fee25000640 1 -- 172.21.15.3:0/3071652562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2010d7e0 con 0x564698120980 2024-02-09T22:57:05.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee25000640 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 0x56469812fcb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fee14005890 tx=0x7fee14052820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=334117491fbee1d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fedfffff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fedfffff640 1 -- 172.21.15.3:0/3071652562 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee14052ea0 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fedfffff640 1 -- 172.21.15.3:0/3071652562 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fee140521d0 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fee0c0046c0 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fedfffff640 1 -- 172.21.15.3:0/3071652562 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) v2 ==== 24+0+0 (secure 0 0 0) 0x7fee14052390 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 10 client.26962 fetch_fsmap learned FSMap version 39 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) v3 -- 0x564698049850 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fedfffff640 1 -- 172.21.15.3:0/3071652562 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee0c0046c0 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fedfffff640 1 -- 172.21.15.3:0/3071652562 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 39) v1 ==== 7902+0+0 (secure 0 0 0) 0x7fee14003950 con 0x564698120980 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 10 client.26962 fetch_fsmap finished waiting for FSMap version 39 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 10 client.26962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 20 client.26962 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564698192490 0x5646981948b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:05.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 4 client.26962 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.286+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564697e14890 con 0x564698192490 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee1ffff640 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564698192490 0x5646981948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fedfffff640 10 client.26962 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee1ffff640 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564698192490 0x5646981948b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6609a345ed57b525 server_cookie=c45de99451af9ab1 in_seq=0 out_seq=0 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fedfffff640 1 -- 172.21.15.3:0/3071652562 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564697e14890 con 0x564698192490 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fedfffff640 10 client.26962 handle_command_reply: tid=0 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 client.26962 shutdown 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 msgr2=0x56469812fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 0x56469812fcb0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fee14005890 tx=0x7fee14052820 comp rx=0 tx=0).stop 2024-02-09T22:57:05.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 shutdown_connections 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564698192490 0x5646981948b0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564698129eb0 0x5646981909d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564698121330 0x5646981301f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fedfffff640 0 client.26962 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 --2- 172.21.15.3:0/3071652562 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564698120980 0x56469812fcb0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 >> 172.21.15.3:0/3071652562 conn(0x56469811b050 msgr2=0x564698119960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 shutdown_connections 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.287+0000 7fee29e38740 1 -- 172.21.15.3:0/3071652562 wait complete. 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee29e38740 20 client.26962 trim_cache size 0 max 16384 2024-02-09T22:57:05.297 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: "2cb94a79-ffaf-4599-9f08-bfd7b54b5c42": { 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "2cb94a79-ffaf-4599-9f08-bfd7b54b5c42", 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:57:05.298 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee100a2230 msgr2=0x7fee100a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee100a2230 0x7fee100a46f0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fee14009900 tx=0x7fee14004b10 comp rx=0 tx=0).stop 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 msgr2=0x7fee2019db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 0x7fee2019db30 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fee0c00e9c0 tx=0x7fee0c00ee80 comp rx=0 tx=0).stop 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 shutdown_connections 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee100a2230 0x7fee100a46f0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fee201976b0 0x7fee20197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fee20105400 0x7fee2019db30 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 --2- 172.21.15.3:0/1473499703 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fee20100330 0x7fee2019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 >> 172.21.15.3:0/1473499703 conn(0x7fee20077250 msgr2=0x7fee200fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 shutdown_connections 2024-02-09T22:57:05.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:05.288+0000 7fee27264640 1 -- 172.21.15.3:0/1473499703 wait complete. 2024-02-09T22:57:05.302 INFO:tasks.cephfs.filesystem:scrub status for tag:2cb94a79-ffaf-4599-9f08-bfd7b54b5c42 - {'path': '/', 'tag': '2cb94a79-ffaf-4599-9f08-bfd7b54b5c42', 'options': 'recursive,force'} 2024-02-09T22:57:05.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 169898 2.74 MB/sec execute 179 sec latency 1412.292 ms 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: pgmap v178: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 616 KiB/s wr, 116 op/s 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:06.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:06 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 170939 2.76 MB/sec execute 180 sec latency 252.660 ms 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: pgmap v178: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 616 KiB/s wr, 116 op/s 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:06 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: pgmap v178: 97 pgs: 97 active+clean; 65 MiB data, 927 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 616 KiB/s wr, 116 op/s 2024-02-09T22:57:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:06.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:06 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:07 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:57:07.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 170939 2.75 MB/sec execute 181 sec latency 1252.706 ms 2024-02-09T22:57:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:07 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:57:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:07 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:57:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:08 smithi003 ceph-mon[32678]: pgmap v179: 97 pgs: 97 active+clean; 70 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-02-09T22:57:08.641 INFO:tasks.workunit.client.0.smithi003.stdout: 1 171807 2.76 MB/sec execute 182 sec latency 1660.845 ms 2024-02-09T22:57:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:08 smithi064 ceph-mon[36638]: pgmap v179: 97 pgs: 97 active+clean; 70 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-02-09T22:57:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:08 smithi089 ceph-mon[36665]: pgmap v179: 97 pgs: 97 active+clean; 70 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 215 op/s 2024-02-09T22:57:09.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 171965 2.74 MB/sec execute 183 sec latency 860.924 ms 2024-02-09T22:57:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:10 smithi003 ceph-mon[32678]: pgmap v180: 97 pgs: 97 active+clean; 70 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 812 KiB/s wr, 155 op/s 2024-02-09T22:57:10.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 171965 2.73 MB/sec execute 184 sec latency 1861.000 ms 2024-02-09T22:57:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:10 smithi064 ceph-mon[36638]: pgmap v180: 97 pgs: 97 active+clean; 70 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 812 KiB/s wr, 155 op/s 2024-02-09T22:57:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:10 smithi089 ceph-mon[36665]: pgmap v180: 97 pgs: 97 active+clean; 70 MiB data, 936 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 812 KiB/s wr, 155 op/s 2024-02-09T22:57:11.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 171965 2.71 MB/sec execute 185 sec latency 2861.089 ms 2024-02-09T22:57:11.899 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.894+0000 7fc04bc9d640 1 -- 172.21.15.3:0/4108547561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc0441024a0 msgr2=0x7fc044102880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.894+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/4108547561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc0441024a0 0x7fc044102880 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc040009860 tx=0x7fc04002bc00 comp rx=0 tx=0).stop 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.894+0000 7fc04bc9d640 1 -- 172.21.15.3:0/4108547561 shutdown_connections 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.894+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/4108547561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc04410b870 0x7fc04410dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.894+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/4108547561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc044102dc0 0x7fc04410b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.894+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/4108547561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc0441024a0 0x7fc044102880 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 -- 172.21.15.3:0/4108547561 >> 172.21.15.3:0/4108547561 conn(0x7fc044077250 msgr2=0x7fc044077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 -- 172.21.15.3:0/4108547561 shutdown_connections 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 -- 172.21.15.3:0/4108547561 wait complete. 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 Processor -- start 2024-02-09T22:57:11.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 -- start start 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 0x7fc0441000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc044102dc0 0x7fc044100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc04410b870 0x7fc044100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.895+0000 7fc04bc9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0441a5c00 con 0x7fc0441024a0 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc04bc9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0441a5d70 con 0x7fc04410b870 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc04bc9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0441a5f30 con 0x7fc044102dc0 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 0x7fc0441000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 0x7fc0441000f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36622/0 (socket says 172.21.15.3:36622) 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 -- 172.21.15.3:0/1203882868 learned_addr learned my addr 172.21.15.3:0/1203882868 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049238640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc044102dc0 0x7fc044100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc04a23a640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc04410b870 0x7fc044100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 -- 172.21.15.3:0/1203882868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc044102dc0 msgr2=0x7fc044100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc044102dc0 0x7fc044100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 -- 172.21.15.3:0/1203882868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc04410b870 msgr2=0x7fc044100b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc04410b870 0x7fc044100b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 -- 172.21.15.3:0/1203882868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc040009490 con 0x7fc0441024a0 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc049a39640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 0x7fc0441000f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc040037040 tx=0x7fc0400346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc03affd640 1 -- 172.21.15.3:0/1203882868 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04002c7a0 con 0x7fc0441024a0 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0441a61b0 con 0x7fc0441024a0 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc03affd640 1 -- 172.21.15.3:0/1203882868 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0400035f0 con 0x7fc0441024a0 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc03affd640 1 -- 172.21.15.3:0/1203882868 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04003d650 con 0x7fc0441024a0 2024-02-09T22:57:11.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.896+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0441a66b0 con 0x7fc0441024a0 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.899+0000 7fc03affd640 1 -- 172.21.15.3:0/1203882868 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc04003b040 con 0x7fc0441024a0 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.899+0000 7fc04e873740 1 Processor -- start 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.899+0000 7fc04e873740 1 -- start start 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.899+0000 7fc03affd640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc0300a2250 0x7fc0300a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04e873740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f46490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04e873740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04e873740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f51b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04e873740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2eb4a00 con 0x55e2d2f4f7a0 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04e873740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2eb1f30 con 0x55e2d2f469d0 2024-02-09T22:57:11.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04e873740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2ead710 con 0x55e2d2f460b0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04a23a640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc0300a2250 0x7fc0300a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f46490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f46490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:52580/0 (socket says 172.21.15.3:52580) 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 -- 172.21.15.3:0/1115073280 learned_addr learned my addr 172.21.15.3:0/1115073280 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04a23a640 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f51b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049238640 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc03affd640 1 -- 172.21.15.3:0/1203882868 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc0400ef870 con 0x7fc0441024a0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 -- 172.21.15.3:0/1115073280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 msgr2=0x55e2d2f4f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 -- 172.21.15.3:0/1115073280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 msgr2=0x55e2d2f51b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f51b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 -- 172.21.15.3:0/1115073280 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2d2e723b0 con 0x55e2d2f460b0 2024-02-09T22:57:11.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049238640 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc04a23a640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc0300a2250 0x7fc0300a4710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc04406a4f0 tx=0x7fc03c009e90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc049a39640 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f46490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc0400bc150 tx=0x7fc0400b8090 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7ed74afad5c94bb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.900+0000 7fc02ffff640 1 -- 172.21.15.3:0/1115073280 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400bbdb0 con 0x55e2d2f460b0 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc02ffff640 1 -- 172.21.15.3:0/1115073280 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc040081ed0 con 0x55e2d2f460b0 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc02ffff640 1 -- 172.21.15.3:0/1115073280 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400c04f0 con 0x55e2d2f460b0 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 -- 172.21.15.3:0/1115073280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 msgr2=0x55e2d2f46490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f46490 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc0400bc150 tx=0x7fc0400b8090 comp rx=0 tx=0).stop 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 -- 172.21.15.3:0/1115073280 shutdown_connections 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f51b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 --2- 172.21.15.3:0/1115073280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f46490 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.901+0000 7fc04e873740 1 -- 172.21.15.3:0/1115073280 >> 172.21.15.3:0/1115073280 conn(0x55e2d2f409d0 msgr2=0x55e2d2f40db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.902+0000 7fc04e873740 1 -- 172.21.15.3:0/1115073280 shutdown_connections 2024-02-09T22:57:11.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.902+0000 7fc04e873740 1 -- 172.21.15.3:0/1115073280 wait complete. 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.902+0000 7fc04e873740 1 -- start start 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.902+0000 7fc04e873740 1 Processor -- start 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2e52b90 con 0x55e2d2f4f7a0 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2b81250 con 0x55e2d2f469d0 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2e33200 con 0x55e2d2f460b0 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 10 client.0 _ll_register_callbacks cb 0x55e2d2e695d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04a23a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4c760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:52590/0 (socket says 172.21.15.3:52590) 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 -- 172.21.15.3:0/916971201 learned_addr learned my addr 172.21.15.3:0/916971201 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:11.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049238640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 -- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 msgr2=0x55e2d2f4cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 -- 172.21.15.3:0/916971201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 msgr2=0x55e2d2f4d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 -- 172.21.15.3:0/916971201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0441a66b0 con 0x55e2d2f460b0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04a23a640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc049a39640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4c760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc0400ed1b0 tx=0x7fc040005bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb623fa170f9dd2d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc02d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc02d7fa640 1 -- 172.21.15.3:0/916971201 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400c4b40 con 0x55e2d2f460b0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc02d7fa640 1 -- 172.21.15.3:0/916971201 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc0400c04f0 con 0x55e2d2f460b0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.903+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e2d2ed3ee0 con 0x55e2d2f460b0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc02d7fa640 1 -- 172.21.15.3:0/916971201 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0400d7070 con 0x55e2d2f460b0 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 10 client.26974 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:11.913 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55e2d2eabd90 con 0x55e2d2f460b0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc02d7fa640 1 -- 172.21.15.3:0/916971201 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e2d2ed3ee0 con 0x55e2d2f460b0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc02d7fa640 1 -- 172.21.15.3:0/916971201 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc040003a80 con 0x55e2d2f460b0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 10 client.26974 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 10 client.26974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 20 client.26974 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2f5f2e0 0x55e2d2f61700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 4 client.26974 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2d2bad600 con 0x55e2d2f5f2e0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.904+0000 7fc049238640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2f5f2e0 0x55e2d2f61700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.905+0000 7fc049238640 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2f5f2e0 0x55e2d2f61700 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd064119774a003 server_cookie=7d4d055021fd838b in_seq=0 out_seq=0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.905+0000 7fc02d7fa640 10 client.26974 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.905+0000 7fc02d7fa640 1 -- 172.21.15.3:0/916971201 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e2d2bad600 con 0x55e2d2f5f2e0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.905+0000 7fc02d7fa640 10 client.26974 handle_command_reply: tid=0 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.905+0000 7fc04e873740 1 client.26974 shutdown 2024-02-09T22:57:11.914 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 msgr2=0x55e2d2f4c760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4c760 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc0400ed1b0 tx=0x7fc040005bd0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 shutdown_connections 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2f5f2e0 0x55e2d2f61700 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 --2- 172.21.15.3:0/916971201 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4c760 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc02d7fa640 0 client.26974 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 >> 172.21.15.3:0/916971201 conn(0x55e2d2f409d0 msgr2=0x55e2d2f42720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 shutdown_connections 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 1 -- 172.21.15.3:0/916971201 wait complete. 2024-02-09T22:57:11.915 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.906+0000 7fc04e873740 20 client.26974 trim_cache size 0 max 16384 2024-02-09T22:57:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 Processor -- start 2024-02-09T22:57:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 -- start start 2024-02-09T22:57:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2ed3ee0 con 0x55e2d2f460b0 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2f46f30 con 0x55e2d2f469d0 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04e873740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2f51fb0 con 0x55e2d2f4f7a0 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4ee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36654/0 (socket says 172.21.15.3:36654) 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 -- 172.21.15.3:0/2542055423 learned_addr learned my addr 172.21.15.3:0/2542055423 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc04a23a640 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049238640 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 -- 172.21.15.3:0/2542055423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f4f7a0 msgr2=0x55e2d2f4b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 -- 172.21.15.3:0/2542055423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 msgr2=0x55e2d2f4d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.921+0000 7fc049a39640 1 -- 172.21.15.3:0/2542055423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2d2e48820 con 0x55e2d2f460b0 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.922+0000 7fc049a39640 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4ee90 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc0400ed5f0 tx=0x7fc0400ba960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c2dada36527034 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.922+0000 7fc02f7fe640 1 -- 172.21.15.3:0/2542055423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400d7050 con 0x55e2d2f460b0 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.922+0000 7fc02f7fe640 1 -- 172.21.15.3:0/2542055423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0400bc4c0 con 0x55e2d2f460b0 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.922+0000 7fc02f7fe640 1 -- 172.21.15.3:0/2542055423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400c4b40 con 0x55e2d2f460b0 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- 172.21.15.3:0/2542055423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 msgr2=0x55e2d2f4ee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4ee90 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7fc0400ed5f0 tx=0x7fc0400ba960 comp rx=0 tx=0).stop 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- 172.21.15.3:0/2542055423 shutdown_connections 2024-02-09T22:57:11.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f469d0 0x55e2d2f4d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f4b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- 172.21.15.3:0/2542055423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f460b0 0x55e2d2f4ee90 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- 172.21.15.3:0/2542055423 >> 172.21.15.3:0/2542055423 conn(0x55e2d2f409d0 msgr2=0x55e2d2f47a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- 172.21.15.3:0/2542055423 shutdown_connections 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- 172.21.15.3:0/2542055423 wait complete. 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- start start 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 Processor -- start 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f460b0 0x55e2d2f53b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f469d0 0x55e2d2f54080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f545c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2f56af0 con 0x55e2d2f4f7a0 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.923+0000 7fc04e873740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2fb47a0 con 0x55e2d2f460b0 2024-02-09T22:57:11.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc04e873740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2d2fb4930 con 0x55e2d2f469d0 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc04e873740 10 client.0 _ll_register_callbacks cb 0x7fc03c011720 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049a39640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f460b0 0x55e2d2f53b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc04a23a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f545c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f469d0 0x55e2d2f54080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049a39640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f460b0 0x55e2d2f53b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:41582/0 (socket says 172.21.15.3:41582) 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc04a23a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f545c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36668/0 (socket says 172.21.15.3:36668) 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049a39640 1 -- 172.21.15.3:0/906959300 learned_addr learned my addr 172.21.15.3:0/906959300 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 -- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f460b0 msgr2=0x55e2d2f53b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f460b0 0x55e2d2f53b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 -- 172.21.15.3:0/906959300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 msgr2=0x55e2d2f545c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f545c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 -- 172.21.15.3:0/906959300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2d2eabd90 con 0x55e2d2f469d0 2024-02-09T22:57:11.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc049238640 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f469d0 0x55e2d2f54080 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x55e2d2f61fd0 tx=0x7fc034004560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b69fa14679c8e43 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc02d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc02d7fa640 1 -- 172.21.15.3:0/906959300 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc034017040 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc02d7fa640 1 -- 172.21.15.3:0/906959300 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc034003a00 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.924+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e2d2bfe1b0 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc02d7fa640 1 -- 172.21.15.3:0/906959300 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc034003bc0 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 10 client.26980 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55e2d2e6e6f0 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc02d7fa640 1 -- 172.21.15.3:0/906959300 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e2d2bfe1b0 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc02d7fa640 1 -- 172.21.15.3:0/906959300 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fc0340104a0 con 0x55e2d2f469d0 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 10 client.26980 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 10 client.26980 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:11.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 20 client.26980 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2fb61f0 0x55e2d2fb8610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 4 client.26980 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:57:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e2d2bea550 con 0x55e2d2fb61f0 2024-02-09T22:57:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.925+0000 7fc049a39640 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2fb61f0 0x55e2d2fb8610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.926+0000 7fc049a39640 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2fb61f0 0x55e2d2fb8610 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46cffd8a66de78c5 server_cookie=82dfd8a76094670f in_seq=0 out_seq=0 2024-02-09T22:57:11.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.926+0000 7fc02d7fa640 10 client.26980 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.950+0000 7fc02d7fa640 1 -- 172.21.15.3:0/906959300 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e2d2bea550 con 0x55e2d2fb61f0 2024-02-09T22:57:11.954 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.950+0000 7fc02d7fa640 10 client.26980 handle_command_reply: tid=0 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 client.26980 shutdown 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f469d0 msgr2=0x55e2d2f54080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f469d0 0x55e2d2f54080 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x55e2d2f61fd0 tx=0x7fc034004560 comp rx=0 tx=0).stop 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 shutdown_connections 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e2d2fb61f0 0x55e2d2fb8610 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e2d2f4f7a0 0x55e2d2f545c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e2d2f469d0 0x55e2d2f54080 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 --2- 172.21.15.3:0/906959300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e2d2f460b0 0x55e2d2f53b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc02d7fa640 0 client.26980 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 >> 172.21.15.3:0/906959300 conn(0x55e2d2f409d0 msgr2=0x55e2d2f4ac30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 shutdown_connections 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 1 -- 172.21.15.3:0/906959300 wait complete. 2024-02-09T22:57:11.955 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04e873740 20 client.26980 trim_cache size 0 max 16384 2024-02-09T22:57:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:57:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:57:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:57:11.956 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:57:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc0300a2250 msgr2=0x7fc0300a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.951+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc0300a2250 0x7fc0300a4710 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc04406a4f0 tx=0x7fc03c009e90 comp rx=0 tx=0).stop 2024-02-09T22:57:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 msgr2=0x7fc0441000f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 0x7fc0441000f0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc040037040 tx=0x7fc0400346f0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 shutdown_connections 2024-02-09T22:57:11.957 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc0300a2250 0x7fc0300a4710 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc04410b870 0x7fc044100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc044102dc0 0x7fc044100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 --2- 172.21.15.3:0/1203882868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc0441024a0 0x7fc0441000f0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 >> 172.21.15.3:0/1203882868 conn(0x7fc044077250 msgr2=0x7fc0440fd280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 shutdown_connections 2024-02-09T22:57:11.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:11.952+0000 7fc04bc9d640 1 -- 172.21.15.3:0/1203882868 wait complete. 2024-02-09T22:57:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:12 smithi003 ceph-mon[32678]: pgmap v181: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 205 op/s 2024-02-09T22:57:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:12.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174131 2.77 MB/sec execute 186 sec latency 2901.119 ms 2024-02-09T22:57:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:12 smithi064 ceph-mon[36638]: pgmap v181: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 205 op/s 2024-02-09T22:57:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:12 smithi089 ceph-mon[36665]: pgmap v181: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 205 op/s 2024-02-09T22:57:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:13 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:13.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174613 2.76 MB/sec execute 187 sec latency 523.576 ms 2024-02-09T22:57:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:13 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:13 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:14 smithi003 ceph-mon[32678]: pgmap v182: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-02-09T22:57:14.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174613 2.75 MB/sec execute 188 sec latency 1523.692 ms 2024-02-09T22:57:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:14 smithi064 ceph-mon[36638]: pgmap v182: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-02-09T22:57:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:14 smithi089 ceph-mon[36665]: pgmap v182: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-02-09T22:57:15.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:15 smithi003 ceph-mon[32678]: pgmap v183: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-02-09T22:57:15.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174613 2.73 MB/sec execute 189 sec latency 2523.754 ms 2024-02-09T22:57:15.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:15 smithi064 ceph-mon[36638]: pgmap v183: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-02-09T22:57:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:15 smithi089 ceph-mon[36665]: pgmap v183: 97 pgs: 97 active+clean; 75 MiB data, 955 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.0 MiB/s wr, 149 op/s 2024-02-09T22:57:16.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174613 2.72 MB/sec execute 190 sec latency 3523.888 ms 2024-02-09T22:57:17.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 174613 2.70 MB/sec execute 191 sec latency 4524.032 ms 2024-02-09T22:57:17.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:17 smithi064 ceph-mon[36638]: pgmap v184: 97 pgs: 97 active+clean; 73 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-02-09T22:57:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:17 smithi089 ceph-mon[36665]: pgmap v184: 97 pgs: 97 active+clean; 73 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-02-09T22:57:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:17 smithi003 ceph-mon[32678]: pgmap v184: 97 pgs: 97 active+clean; 73 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.5 MiB/s wr, 205 op/s 2024-02-09T22:57:18.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 175514 2.71 MB/sec execute 192 sec latency 4936.050 ms 2024-02-09T22:57:19.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 175514 2.70 MB/sec execute 193 sec latency 1097.496 ms 2024-02-09T22:57:20.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:20 smithi089 ceph-mon[36665]: pgmap v185: 97 pgs: 97 active+clean; 73 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 971 KiB/s wr, 106 op/s 2024-02-09T22:57:20.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:20 smithi003 ceph-mon[32678]: pgmap v185: 97 pgs: 97 active+clean; 73 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 971 KiB/s wr, 106 op/s 2024-02-09T22:57:20.642 INFO:tasks.workunit.client.0.smithi003.stdout: 1 176844 2.72 MB/sec execute 194 sec latency 1554.806 ms 2024-02-09T22:57:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:20 smithi064 ceph-mon[36638]: pgmap v185: 97 pgs: 97 active+clean; 73 MiB data, 970 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 971 KiB/s wr, 106 op/s 2024-02-09T22:57:21.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 178156 2.75 MB/sec execute 195 sec latency 157.539 ms 2024-02-09T22:57:22.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:22 smithi089 ceph-mon[36665]: pgmap v186: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-02-09T22:57:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:22 smithi003 ceph-mon[32678]: pgmap v186: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-02-09T22:57:22.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 178156 2.73 MB/sec execute 196 sec latency 1157.630 ms 2024-02-09T22:57:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:22 smithi064 ceph-mon[36638]: pgmap v186: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.1 MiB/s wr, 137 op/s 2024-02-09T22:57:23.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 179015 2.74 MB/sec execute 197 sec latency 1561.141 ms 2024-02-09T22:57:24.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:24 smithi089 ceph-mon[36665]: pgmap v187: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 637 KiB/s wr, 87 op/s 2024-02-09T22:57:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:24 smithi003 ceph-mon[32678]: pgmap v187: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 637 KiB/s wr, 87 op/s 2024-02-09T22:57:24.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 179058 2.73 MB/sec execute 198 sec latency 971.939 ms 2024-02-09T22:57:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:24 smithi064 ceph-mon[36638]: pgmap v187: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 637 KiB/s wr, 87 op/s 2024-02-09T22:57:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 -- 172.21.15.3:0/242956051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88105410 msgr2=0x7fbb8810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/242956051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88105410 0x7fbb8810cfe0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009cd0 tx=0x7fbb8402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 -- 172.21.15.3:0/242956051 shutdown_connections 2024-02-09T22:57:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/242956051 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88105410 0x7fbb8810cfe0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/242956051 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb88100cf0 0x7fbb88104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/242956051 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb88100340 0x7fbb88100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 -- 172.21.15.3:0/242956051 >> 172.21.15.3:0/242956051 conn(0x7fbb88077220 msgr2=0x7fbb88077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 -- 172.21.15.3:0/242956051 shutdown_connections 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 -- 172.21.15.3:0/242956051 wait complete. 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 Processor -- start 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.040+0000 7fbb8ff25640 1 -- start start 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 0x7fbb88101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb88105410 0x7fbb88102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb88102850 0x7fbb881a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881a6030 con 0x7fbb88100340 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881a61a0 con 0x7fbb88105410 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb881a6360 con 0x7fbb88102850 2024-02-09T22:57:25.046 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 0x7fbb88101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 0x7fbb88101dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34294/0 (socket says 172.21.15.3:34294) 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 -- 172.21.15.3:0/895632352 learned_addr learned my addr 172.21.15.3:0/895632352 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb88102850 0x7fbb881a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb88105410 0x7fbb88102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 -- 172.21.15.3:0/895632352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb88102850 msgr2=0x7fbb881a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb88102850 0x7fbb881a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 -- 172.21.15.3:0/895632352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb88105410 msgr2=0x7fbb88102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb88105410 0x7fbb88102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 -- 172.21.15.3:0/895632352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb84009900 con 0x7fbb88100340 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8dcc1640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 0x7fbb88101dd0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000d950 tx=0x7fbb8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb7effd640 1 -- 172.21.15.3:0/895632352 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80009a30 con 0x7fbb88100340 2024-02-09T22:57:25.047 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb881a6640 con 0x7fbb88100340 2024-02-09T22:57:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb7effd640 1 -- 172.21.15.3:0/895632352 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb80009b90 con 0x7fbb88100340 2024-02-09T22:57:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb7effd640 1 -- 172.21.15.3:0/895632352 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb80004a00 con 0x7fbb88100340 2024-02-09T22:57:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.041+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb881a6ba0 con 0x7fbb88100340 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb7effd640 1 -- 172.21.15.3:0/895632352 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fbb80009d00 con 0x7fbb88100340 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 Processor -- start 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 -- start start 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d60b0 0x5563765d6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb7effd640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb740a2250 0x7fbb740a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d69d0 0x5563765df260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765e1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556376544a00 con 0x5563765df7a0 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556376541f30 con 0x5563765d69d0 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.044+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637653d710 con 0x5563765d60b0 2024-02-09T22:57:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765e1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8dcc1640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb740a2250 0x7fbb740a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765e1b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34306/0 (socket says 172.21.15.3:34306) 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3824256252 learned_addr learned my addr 172.21.15.3:0/3824256252 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d69d0 0x5563765df260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d60b0 0x5563765d6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb7effd640 1 -- 172.21.15.3:0/895632352 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbb800cda40 con 0x7fbb88100340 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3824256252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d60b0 msgr2=0x5563765d6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d60b0 0x5563765d6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3824256252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d69d0 msgr2=0x5563765df260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d69d0 0x5563765df260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3824256252 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563765023b0 con 0x5563765df7a0 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8dcc1640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb740a2250 0x7fbb740a4710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb800cb7c0 tx=0x7fbb8008fb30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765e1b90 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009430 tx=0x7fbb84009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=597adfd33e0f5005 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb73fff640 1 -- 172.21.15.3:0/3824256252 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84034a90 con 0x5563765df7a0 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb73fff640 1 -- 172.21.15.3:0/3824256252 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb84003970 con 0x5563765df7a0 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.045+0000 7fbb73fff640 1 -- 172.21.15.3:0/3824256252 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84004290 con 0x5563765df7a0 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3824256252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 msgr2=0x5563765e1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765e1b90 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009430 tx=0x7fbb84009360 comp rx=0 tx=0).stop 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3824256252 shutdown_connections 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d69d0 0x5563765df260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765e1b90 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3824256252 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d60b0 0x5563765d6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3824256252 >> 172.21.15.3:0/3824256252 conn(0x5563765d09d0 msgr2=0x5563765d0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3824256252 shutdown_connections 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3824256252 wait complete. 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 -- start start 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.046+0000 7fbb92bd4740 1 Processor -- start 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765dc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765dcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765dd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563764e2b90 con 0x5563765df7a0 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556376211250 con 0x5563765d60b0 2024-02-09T22:57:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563764c3200 con 0x5563765d69d0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb92bd4740 10 client.0 _ll_register_callbacks cb 0x5563764f95d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765dd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765dc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765dc7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:60380/0 (socket says 172.21.15.3:60380) 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/819407425 learned_addr learned my addr 172.21.15.3:0/819407425 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765dcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/819407425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 msgr2=0x5563765dcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765dcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/819407425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 msgr2=0x5563765dd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765dd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/819407425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8810d7f0 con 0x5563765d60b0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765dc7d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbb840348f0 tx=0x7fbb84005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27b1d611f9f3c338 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.047+0000 7fbb717fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb717fa640 1 -- 172.21.15.3:0/819407425 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84003de0 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb717fa640 1 -- 172.21.15.3:0/819407425 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbb84034a90 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556376563ee0 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb717fa640 1 -- 172.21.15.3:0/819407425 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb84034c50 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 10 client.26921 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55637653bd90 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb717fa640 1 -- 172.21.15.3:0/819407425 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556376563ee0 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb717fa640 1 -- 172.21.15.3:0/819407425 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbb8404b030 con 0x5563765d60b0 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 10 client.26921 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 10 client.26921 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 20 client.26921 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563765ef510 0x5563765f1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 4 client.26921 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55637623d600 con 0x5563765ef510 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.048+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563765ef510 0x5563765f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.049+0000 7fbb717fa640 10 client.26921 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.049+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563765ef510 0x5563765f1900 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae65a5a46d27cf21 server_cookie=a1665c8e08f931a in_seq=0 out_seq=0 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.049+0000 7fbb717fa640 1 -- 172.21.15.3:0/819407425 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55637623d600 con 0x5563765ef510 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.049+0000 7fbb717fa640 10 client.26921 handle_command_reply: tid=0 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.049+0000 7fbb92bd4740 1 client.26921 shutdown 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 msgr2=0x5563765dc7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765dc7d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbb840348f0 tx=0x7fbb84005fd0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.061 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 shutdown_connections 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563765ef510 0x5563765f1900 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765df7a0 0x5563765dd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765dcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb717fa640 0 client.26921 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/819407425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765dc7d0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 >> 172.21.15.3:0/819407425 conn(0x5563765d09d0 msgr2=0x5563765d2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 shutdown_connections 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 1 -- 172.21.15.3:0/819407425 wait complete. 2024-02-09T22:57:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.050+0000 7fbb92bd4740 20 client.26921 trim_cache size 0 max 16384 2024-02-09T22:57:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 Processor -- start 2024-02-09T22:57:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 -- start start 2024-02-09T22:57:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 0x5563765def00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765d9a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.071 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765df7a0 0x5563765d9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563765cc310 con 0x5563765d60b0 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563765ec450 con 0x5563765df7a0 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563765caec0 con 0x5563765d69d0 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 0x5563765def00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 0x5563765def00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34328/0 (socket says 172.21.15.3:34328) 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3480044944 learned_addr learned my addr 172.21.15.3:0/3480044944 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765d9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3480044944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 msgr2=0x5563765d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765d9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3480044944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765df7a0 msgr2=0x5563765d9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.072 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765df7a0 0x5563765d9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.063+0000 7fbb8e4c2640 1 -- 172.21.15.3:0/3480044944 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563764d8820 con 0x5563765d60b0 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 0x5563765def00 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009890 tx=0x7fbb84003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=945e73d63896a33d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb737fe640 1 -- 172.21.15.3:0/3480044944 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84032d70 con 0x5563765d60b0 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb737fe640 1 -- 172.21.15.3:0/3480044944 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb84003dc0 con 0x5563765d60b0 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb737fe640 1 -- 172.21.15.3:0/3480044944 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb84032b20 con 0x5563765d60b0 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3480044944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 msgr2=0x5563765def00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 0x5563765def00 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7fbb84009890 tx=0x7fbb84003b00 comp rx=0 tx=0).stop 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3480044944 shutdown_connections 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765df7a0 0x5563765d9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.073 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765d69d0 0x5563765d9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/3480044944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d60b0 0x5563765def00 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3480044944 >> 172.21.15.3:0/3480044944 conn(0x5563765d09d0 msgr2=0x5563765f6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.064+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3480044944 shutdown_connections 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 -- 172.21.15.3:0/3480044944 wait complete. 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 -- start start 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 Processor -- start 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765e46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 0x5563765e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765df7a0 0x55637663e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637663ef10 con 0x5563765d69d0 2024-02-09T22:57:25.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637663f080 con 0x5563765d60b0 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55637663f240 con 0x5563765df7a0 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb92bd4740 10 client.0 _ll_register_callbacks cb 0x7fbb7800b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb8d4c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 0x5563765e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb8d4c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 0x5563765e4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34330/0 (socket says 172.21.15.3:34330) 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb8d4c0640 1 -- 172.21.15.3:0/1918902439 learned_addr learned my addr 172.21.15.3:0/1918902439 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.065+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765df7a0 0x55637663e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765e46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8d4c0640 1 -- 172.21.15.3:0/1918902439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765df7a0 msgr2=0x55637663e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765df7a0 0x55637663e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8d4c0640 1 -- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 msgr2=0x5563765e46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:25.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765e46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8d4c0640 1 -- 172.21.15.3:0/1918902439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55637653bd90 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb8d4c0640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 0x5563765e4c00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x5563765d5b60 tx=0x7fbb780118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449d7bb86bd6382a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb717fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb717fa640 1 -- 172.21.15.3:0/1918902439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb78017040 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb717fa640 1 -- 172.21.15.3:0/1918902439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbb7801c8a0 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb84003970 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb717fa640 1 -- 172.21.15.3:0/1918902439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb78005020 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 10 client.17748 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5563764fe6f0 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb717fa640 1 -- 172.21.15.3:0/1918902439 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb84003970 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb717fa640 1 -- 172.21.15.3:0/1918902439 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbb78024420 con 0x5563765d69d0 2024-02-09T22:57:25.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 10 client.17748 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 10 client.17748 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 20 client.17748 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556376646bc0 0x556376648fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 4 client.17748 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.066+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55637628e1b0 con 0x556376646bc0 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.067+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556376646bc0 0x556376648fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.067+0000 7fbb717fa640 10 client.17748 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:25.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.067+0000 7fbb8e4c2640 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556376646bc0 0x556376648fe0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=80ac1add4380e8d server_cookie=34096fc7deee607d in_seq=0 out_seq=0 2024-02-09T22:57:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.099+0000 7fbb717fa640 1 -- 172.21.15.3:0/1918902439 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55637628e1b0 con 0x556376646bc0 2024-02-09T22:57:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.099+0000 7fbb717fa640 10 client.17748 handle_command_reply: tid=0 2024-02-09T22:57:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.099+0000 7fbb92bd4740 1 client.17748 shutdown 2024-02-09T22:57:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 msgr2=0x5563765e4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 0x5563765e4c00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x5563765d5b60 tx=0x7fbb780118d0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 shutdown_connections 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556376646bc0 0x556376648fe0 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563765df7a0 0x55637663e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563765d69d0 0x5563765e4c00 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb717fa640 0 client.17748 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 --2- 172.21.15.3:0/1918902439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563765d60b0 0x5563765e46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 >> 172.21.15.3:0/1918902439 conn(0x5563765d09d0 msgr2=0x5563765db0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 shutdown_connections 2024-02-09T22:57:25.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 1 -- 172.21.15.3:0/1918902439 wait complete. 2024-02-09T22:57:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb92bd4740 20 client.17748 trim_cache size 0 max 16384 2024-02-09T22:57:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:57:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:57:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:57:25.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:57:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb740a2250 msgr2=0x7fbb740a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb740a2250 0x7fbb740a4710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fbb800cb7c0 tx=0x7fbb8008fb30 comp rx=0 tx=0).stop 2024-02-09T22:57:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 msgr2=0x7fbb88101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 0x7fbb88101dd0 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000d950 tx=0x7fbb8000de10 comp rx=0 tx=0).stop 2024-02-09T22:57:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 shutdown_connections 2024-02-09T22:57:25.106 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb740a2250 0x7fbb740a4710 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb88105410 0x7fbb88102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb88102850 0x7fbb881a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 --2- 172.21.15.3:0/895632352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb88100340 0x7fbb88101dd0 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.100+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 >> 172.21.15.3:0/895632352 conn(0x7fbb88077220 msgr2=0x7fbb88068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.101+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 shutdown_connections 2024-02-09T22:57:25.107 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:25.101+0000 7fbb8ff25640 1 -- 172.21.15.3:0/895632352 wait complete. 2024-02-09T22:57:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:25 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:57:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:25 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:57:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:25 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:57:25.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:25 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:57:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:25 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:57:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:25 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:57:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:25 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:57:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:25 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:57:25.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 180571 2.76 MB/sec execute 199 sec latency 1416.816 ms 2024-02-09T22:57:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:25 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:57:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:25 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:57:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:25 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:57:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:25 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:57:26.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:26 smithi089 ceph-mon[36665]: pgmap v188: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 637 KiB/s wr, 87 op/s 2024-02-09T22:57:26.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:26 smithi003 ceph-mon[32678]: pgmap v188: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 637 KiB/s wr, 87 op/s 2024-02-09T22:57:26.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181579 2.77 MB/sec execute 200 sec latency 264.772 ms 2024-02-09T22:57:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:26 smithi064 ceph-mon[36638]: pgmap v188: 97 pgs: 97 active+clean; 71 MiB data, 975 MiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 637 KiB/s wr, 87 op/s 2024-02-09T22:57:27.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181579 2.76 MB/sec execute 201 sec latency 1264.852 ms 2024-02-09T22:57:28.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:28 smithi089 ceph-mon[36665]: pgmap v189: 97 pgs: 97 active+clean; 73 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-02-09T22:57:28.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:28 smithi064 ceph-mon[36638]: pgmap v189: 97 pgs: 97 active+clean; 73 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-02-09T22:57:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:28 smithi003 ceph-mon[32678]: pgmap v189: 97 pgs: 97 active+clean; 73 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.2 MiB/s wr, 149 op/s 2024-02-09T22:57:28.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181635 2.74 MB/sec execute 202 sec latency 1671.105 ms 2024-02-09T22:57:29.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181635 2.73 MB/sec execute 203 sec latency 1553.282 ms 2024-02-09T22:57:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:30 smithi089 ceph-mon[36665]: pgmap v190: 97 pgs: 97 active+clean; 73 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 712 KiB/s wr, 93 op/s 2024-02-09T22:57:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:30 smithi003 ceph-mon[32678]: pgmap v190: 97 pgs: 97 active+clean; 73 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 712 KiB/s wr, 93 op/s 2024-02-09T22:57:30.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181635 2.72 MB/sec execute 204 sec latency 2553.335 ms 2024-02-09T22:57:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:30 smithi064 ceph-mon[36638]: pgmap v190: 97 pgs: 97 active+clean; 73 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 712 KiB/s wr, 93 op/s 2024-02-09T22:57:31.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181635 2.70 MB/sec execute 205 sec latency 3553.391 ms 2024-02-09T22:57:32.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:32 smithi089 ceph-mon[36665]: pgmap v191: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-02-09T22:57:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:32 smithi003 ceph-mon[32678]: pgmap v191: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-02-09T22:57:32.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181635 2.69 MB/sec execute 206 sec latency 4553.491 ms 2024-02-09T22:57:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:32 smithi064 ceph-mon[36638]: pgmap v191: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 134 op/s 2024-02-09T22:57:33.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181703 2.68 MB/sec execute 207 sec latency 4959.857 ms 2024-02-09T22:57:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:34 smithi003 ceph-mon[32678]: pgmap v192: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 960 KiB/s wr, 103 op/s 2024-02-09T22:57:34.643 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181703 2.67 MB/sec execute 208 sec latency 1550.489 ms 2024-02-09T22:57:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:34 smithi064 ceph-mon[36638]: pgmap v192: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 960 KiB/s wr, 103 op/s 2024-02-09T22:57:34.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:34 smithi089 ceph-mon[36665]: pgmap v192: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 960 KiB/s wr, 103 op/s 2024-02-09T22:57:35.303 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:57:35.303 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:57:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.413+0000 7ff3d2f19640 1 -- 172.21.15.3:0/2305454789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc100330 msgr2=0x7ff3cc100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.413+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/2305454789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc100330 0x7ff3cc100710 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009930 tx=0x7ff3bc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- 172.21.15.3:0/2305454789 shutdown_connections 2024-02-09T22:57:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/2305454789 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3cc100ce0 0x7ff3cc104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/2305454789 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3cc105400 0x7ff3cc10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/2305454789 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc100330 0x7ff3cc100710 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- 172.21.15.3:0/2305454789 >> 172.21.15.3:0/2305454789 conn(0x7ff3cc077250 msgr2=0x7ff3cc077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- 172.21.15.3:0/2305454789 shutdown_connections 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- 172.21.15.3:0/2305454789 wait complete. 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 Processor -- start 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- start start 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3cc100330 0x7ff3cc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3cc105400 0x7ff3cc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 0x7ff3cc199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc19a2d0 con 0x7ff3cc1998e0 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc19a440 con 0x7ff3cc105400 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.414+0000 7ff3d2f19640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3cc19a600 con 0x7ff3cc100330 2024-02-09T22:57:35.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 0x7ff3cc199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 0x7ff3cc199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53260/0 (socket says 172.21.15.3:53260) 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 -- 172.21.15.3:0/246521274 learned_addr learned my addr 172.21.15.3:0/246521274 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3cc105400 0x7ff3cc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3cc100330 0x7ff3cc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 -- 172.21.15.3:0/246521274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3cc100330 msgr2=0x7ff3cc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3cc100330 0x7ff3cc19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 -- 172.21.15.3:0/246521274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3cc105400 msgr2=0x7ff3cc19fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3cc105400 0x7ff3cc19fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 -- 172.21.15.3:0/246521274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc009560 con 0x7ff3cc1998e0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 0x7ff3cc199d90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400eaa0 tx=0x7ff3c400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3c9ffb640 1 -- 172.21.15.3:0/246521274 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c400cdb0 con 0x7ff3cc1998e0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3c9ffb640 1 -- 172.21.15.3:0/246521274 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c4005020 con 0x7ff3cc1998e0 2024-02-09T22:57:35.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3cc1a65f0 con 0x7ff3cc1998e0 2024-02-09T22:57:35.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.415+0000 7ff3c9ffb640 1 -- 172.21.15.3:0/246521274 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c40116b0 con 0x7ff3cc1998e0 2024-02-09T22:57:35.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.416+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3cc1a6ad0 con 0x7ff3cc1998e0 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.417+0000 7ff3c9ffb640 1 -- 172.21.15.3:0/246521274 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff3c4005520 con 0x7ff3cc1998e0 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3c9ffb640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3b40a2230 0x7ff3b40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 Processor -- start 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 -- start start 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3b40a2230 0x7ff3b40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3c9ffb640 1 -- 172.21.15.3:0/246521274 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff3c40ce1f0 con 0x7ff3cc1998e0 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f3980 0x55cfda6f3d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 0x55cfda6fc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda6ff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda6aed00 con 0x55cfda6f4330 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda6632f0 con 0x55cfda6f3980 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda660820 con 0x55cfda6fceb0 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 0x55cfda6fc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3cbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 0x55cfda6fc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53264/0 (socket says 172.21.15.3:53264) 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.418+0000 7ff3cbfff640 1 -- 172.21.15.3:0/511138735 learned_addr learned my addr 172.21.15.3:0/511138735 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f3980 0x55cfda6f3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda6ff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3cbfff640 1 -- 172.21.15.3:0/511138735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 msgr2=0x55cfda6ff300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda6ff300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3cbfff640 1 -- 172.21.15.3:0/511138735 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f3980 msgr2=0x55cfda6f3d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f3980 0x55cfda6f3d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3b40a2230 0x7ff3b40a46f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005ab0 tx=0x7ff3bc004740 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3cbfff640 1 -- 172.21.15.3:0/511138735 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfda61dff0 con 0x55cfda6f4330 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda6ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 0x55cfda6fc8e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000bc30 tx=0x7ff3c00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=231a2b7589ce57c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3b2ffd640 1 -- 172.21.15.3:0/511138735 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c001c480 con 0x55cfda6f4330 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3b2ffd640 1 -- 172.21.15.3:0/511138735 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c0015070 con 0x55cfda6f4330 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.419+0000 7ff3b2ffd640 1 -- 172.21.15.3:0/511138735 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0011910 con 0x55cfda6f4330 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/511138735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 msgr2=0x55cfda6fc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 0x55cfda6fc8e0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000bc30 tx=0x7ff3c00107a0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/511138735 shutdown_connections 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda6ff300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f4330 0x55cfda6fc8e0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/511138735 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f3980 0x55cfda6f3d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/511138735 >> 172.21.15.3:0/511138735 conn(0x55cfda6ee050 msgr2=0x55cfda6ee430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/511138735 shutdown_connections 2024-02-09T22:57:35.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/511138735 wait complete. 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 -- start start 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.420+0000 7ff3d5bc5740 1 Processor -- start 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6f9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda6fa330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda6fa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda65c000 con 0x55cfda6f3980 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda32f430 con 0x55cfda6fceb0 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda2ebe00 con 0x55cfda6f4330 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 10 client.0 _ll_register_callbacks cb 0x55cfda6507f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6f9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6f9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53280/0 (socket says 172.21.15.3:53280) 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/2077824000 learned_addr learned my addr 172.21.15.3:0/2077824000 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda6fa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda6fa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/2077824000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 msgr2=0x55cfda6fa330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda6fa330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 msgr2=0x55cfda6fa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda6fa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/2077824000 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c000b860 con 0x55cfda6f3980 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda6fa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6f9df0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc004270 tx=0x7ff3bc02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e2a837fd99ed38 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3abfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3abfff640 1 -- 172.21.15.3:0/2077824000 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc005200 con 0x55cfda6f3980 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3abfff640 1 -- 172.21.15.3:0/2077824000 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff3bc005360 con 0x55cfda6f3980 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.421+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cfda35b7e0 con 0x55cfda6f3980 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3abfff640 1 -- 172.21.15.3:0/2077824000 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3bc005520 con 0x55cfda6f3980 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 10 client.17760 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55cfda65a680 con 0x55cfda6f3980 2024-02-09T22:57:35.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3abfff640 1 -- 172.21.15.3:0/2077824000 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cfda35b7e0 con 0x55cfda6f3980 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3abfff640 1 -- 172.21.15.3:0/2077824000 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7ff3bc0416c0 con 0x55cfda6f3980 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 10 client.17760 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 10 client.17760 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 20 client.17760 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda70d3b0 0x55cfda70f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 4 client.17760 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cfda398720 con 0x55cfda70d3b0 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.422+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda70d3b0 0x55cfda70f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3abfff640 10 client.17760 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda70d3b0 0x55cfda70f7d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6bdd116d4dfac3a server_cookie=e31b8cf3bf1b13fa in_seq=0 out_seq=0 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3abfff640 1 -- 172.21.15.3:0/2077824000 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cfda398720 con 0x55cfda70d3b0 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3abfff640 10 client.17760 handle_command_reply: tid=0 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 client.17760 shutdown 2024-02-09T22:57:35.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 msgr2=0x55cfda6f9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6f9df0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc004270 tx=0x7ff3bc02bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 shutdown_connections 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda70d3b0 0x55cfda70f7d0 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda6fa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda6fa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3abfff640 0 client.17760 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/2077824000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6f9df0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.423+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 >> 172.21.15.3:0/2077824000 conn(0x55cfda6ee050 msgr2=0x55cfda6efda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.424+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 shutdown_connections 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.424+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/2077824000 wait complete. 2024-02-09T22:57:35.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.424+0000 7ff3d5bc5740 20 client.17760 trim_cache size 0 max 16384 2024-02-09T22:57:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 Processor -- start 2024-02-09T22:57:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 -- start start 2024-02-09T22:57:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6fc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f4330 0x55cfda6fa690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda714130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda6e9780 con 0x55cfda6f3980 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda5e0fd0 con 0x55cfda6f4330 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda6faf60 con 0x55cfda6fceb0 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6fc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6fc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53296/0 (socket says 172.21.15.3:53296) 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/968387981 learned_addr learned my addr 172.21.15.3:0/968387981 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/968387981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 msgr2=0x55cfda714130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda714130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/968387981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f4330 msgr2=0x55cfda6fa690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f4330 0x55cfda6fa690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/968387981 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfda5f6b30 con 0x55cfda6f3980 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.439+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6fc520 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc000c00 tx=0x7ff3bc009890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8dd4e248ef7fe74 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3b27fc640 1 -- 172.21.15.3:0/968387981 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc02cc40 con 0x55cfda6f3980 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3b27fc640 1 -- 172.21.15.3:0/968387981 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3bc041c10 con 0x55cfda6f3980 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3b27fc640 1 -- 172.21.15.3:0/968387981 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc003850 con 0x55cfda6f3980 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/968387981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 msgr2=0x55cfda6fc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6fc520 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc000c00 tx=0x7ff3bc009890 comp rx=0 tx=0).stop 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/968387981 shutdown_connections 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6f4330 0x55cfda6fa690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6fceb0 0x55cfda714130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/968387981 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda6fc520 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/968387981 >> 172.21.15.3:0/968387981 conn(0x55cfda6ee050 msgr2=0x55cfda6ee430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/968387981 shutdown_connections 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/968387981 wait complete. 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 -- start start 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.440+0000 7ff3d5bc5740 1 Processor -- start 2024-02-09T22:57:35.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda703fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda7024c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda702a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda702f40 con 0x55cfda6f3980 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda7030b0 con 0x55cfda6fceb0 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfda703270 con 0x55cfda6f4330 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d5bc5740 10 client.0 _ll_register_callbacks cb 0x7ff3c000b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda703fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda703fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53300/0 (socket says 172.21.15.3:53300) 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/127180172 learned_addr learned my addr 172.21.15.3:0/127180172 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d14b6640 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda702a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/127180172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 msgr2=0x55cfda7024c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda7024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 msgr2=0x55cfda702a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda702a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 -- 172.21.15.3:0/127180172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cfda65a680 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3d0cb5640 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda703fe0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009930 tx=0x7ff3bc02ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96000bd562138e6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.441+0000 7ff3abfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3abfff640 1 -- 172.21.15.3:0/127180172 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc041660 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3abfff640 1 -- 172.21.15.3:0/127180172 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff3bc0417c0 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3c0015070 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3abfff640 1 -- 172.21.15.3:0/127180172 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3bc0035f0 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 10 client.17766 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55cfda61c850 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3abfff640 1 -- 172.21.15.3:0/127180172 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3c0015070 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3abfff640 1 -- 172.21.15.3:0/127180172 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7ff3bc061a10 con 0x55cfda6f3980 2024-02-09T22:57:35.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 10 client.17766 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 10 client.17766 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 20 client.17766 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda764960 0x55cfda766d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 4 client.17766 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cfda3e7890 con 0x55cfda764960 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.442+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda764960 0x55cfda766d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3cbfff640 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda764960 0x55cfda766d80 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7532ed5af031361 server_cookie=56ab58dbbc01a318 in_seq=0 out_seq=0 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3abfff640 10 client.17766 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3abfff640 1 -- 172.21.15.3:0/127180172 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cfda3e7890 con 0x55cfda764960 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3abfff640 10 client.17766 handle_command_reply: tid=0 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 client.17766 shutdown 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 msgr2=0x55cfda703fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda703fe0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009930 tx=0x7ff3bc02ccf0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 shutdown_connections 2024-02-09T22:57:35.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cfda764960 0x55cfda766d80 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cfda6fceb0 0x55cfda702a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cfda6f4330 0x55cfda7024c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3abfff640 0 client.17766 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 --2- 172.21.15.3:0/127180172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cfda6f3980 0x55cfda703fe0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 >> 172.21.15.3:0/127180172 conn(0x55cfda6ee050 msgr2=0x55cfda714130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 shutdown_connections 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 1 -- 172.21.15.3:0/127180172 wait complete. 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.443+0000 7ff3d5bc5740 20 client.17766 trim_cache size 0 max 16384 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:57:35.454 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:57:35.455 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:57:35.455 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:57:35.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3b40a2230 msgr2=0x7ff3b40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3b40a2230 0x7ff3b40a46f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc005ab0 tx=0x7ff3bc004740 comp rx=0 tx=0).stop 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 msgr2=0x7ff3cc199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 0x7ff3cc199d90 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7ff3c400eaa0 tx=0x7ff3c400ef60 comp rx=0 tx=0).stop 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 shutdown_connections 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3b40a2230 0x7ff3b40a46f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3cc105400 0x7ff3cc19fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3cc1998e0 0x7ff3cc199d90 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 --2- 172.21.15.3:0/246521274 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3cc100330 0x7ff3cc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 >> 172.21.15.3:0/246521274 conn(0x7ff3cc077250 msgr2=0x7ff3cc0fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 shutdown_connections 2024-02-09T22:57:35.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.444+0000 7ff3d2f19640 1 -- 172.21.15.3:0/246521274 wait complete. 2024-02-09T22:57:35.459 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:57:35.459 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 -- 172.21.15.3:0/698741483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac101440 msgr2=0x7fa4ac101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/698741483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac101440 0x7fa4ac101820 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009930 tx=0x7fa49c02bc20 comp rx=0 tx=0).stop 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 -- 172.21.15.3:0/698741483 shutdown_connections 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/698741483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 0x7fa4ac10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/698741483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101df0 0x7fa4ac105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/698741483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac101440 0x7fa4ac101820 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 -- 172.21.15.3:0/698741483 >> 172.21.15.3:0/698741483 conn(0x7fa4ac0fcce0 msgr2=0x7fa4ac0ff100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 -- 172.21.15.3:0/698741483 shutdown_connections 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.558+0000 7fa4b0a81640 1 -- 172.21.15.3:0/698741483 wait complete. 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 Processor -- start 2024-02-09T22:57:35.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 -- start start 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101440 0x7fa4ac19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 0x7fa4ac19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 0x7fa4ac197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac198000 con 0x7fa4ac197610 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac198170 con 0x7fa4ac106510 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4b0a81640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4ac198330 con 0x7fa4ac101440 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4aad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 0x7fa4ac197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4aad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 0x7fa4ac197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53316/0 (socket says 172.21.15.3:53316) 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.559+0000 7fa4aad9d640 1 -- 172.21.15.3:0/1797867115 learned_addr learned my addr 172.21.15.3:0/1797867115 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aa59c640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101440 0x7fa4ac19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4a3fff640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 0x7fa4ac19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aad9d640 1 -- 172.21.15.3:0/1797867115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101440 msgr2=0x7fa4ac19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aad9d640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101440 0x7fa4ac19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aad9d640 1 -- 172.21.15.3:0/1797867115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 msgr2=0x7fa4ac19d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aad9d640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 0x7fa4ac19d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aad9d640 1 -- 172.21.15.3:0/1797867115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa49c009560 con 0x7fa4ac197610 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aa59c640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101440 0x7fa4ac19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4a3fff640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 0x7fa4ac19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4aad9d640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 0x7fa4ac197ac0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400cce0 tx=0x7fa4a400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a400f450 con 0x7fa4ac197610 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4a400fa90 con 0x7fa4ac197610 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4ac1a42d0 con 0x7fa4ac197610 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4018b90 con 0x7fa4ac197610 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.560+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4ac1a4790 con 0x7fa4ac197610 2024-02-09T22:57:35.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.561+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4ac068500 con 0x7fa4ac197610 2024-02-09T22:57:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.563+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa4a400fc00 con 0x7fa4ac197610 2024-02-09T22:57:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.563+0000 7fa4a37fe640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4980a2230 0x7fa4980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.563+0000 7fa4aa59c640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4980a2230 0x7fa4980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.563+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa4a40ce960 con 0x7fa4ac197610 2024-02-09T22:57:35.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.564+0000 7fa4aa59c640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4980a2230 0x7fa4980a46f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa49c006010 tx=0x7fa49c004960 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.566+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa4a40976b0 con 0x7fa4ac197610 2024-02-09T22:57:35.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181703 2.65 MB/sec execute 209 sec latency 2550.550 ms 2024-02-09T22:57:35.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.788+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa4ac1097a0 con 0x7fa4ac197610 2024-02-09T22:57:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.789+0000 7fa4a37fe640 1 -- 172.21.15.3:0/1797867115 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+11373 (secure 0 0 0) 0x7fa4a409ad70 con 0x7fa4ac197610 2024-02-09T22:57:35.793 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:57:35.793 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:57:05.445840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:57:35.793 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 40 2024-02-09T22:57:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4980a2230 msgr2=0x7fa4980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4980a2230 0x7fa4980a46f0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa49c006010 tx=0x7fa49c004960 comp rx=0 tx=0).stop 2024-02-09T22:57:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 msgr2=0x7fa4ac197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 0x7fa4ac197ac0 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400cce0 tx=0x7fa4a400a620 comp rx=0 tx=0).stop 2024-02-09T22:57:35.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 shutdown_connections 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4980a2230 0x7fa4980a46f0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4ac106510 0x7fa4ac19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4ac197610 0x7fa4ac197ac0 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 --2- 172.21.15.3:0/1797867115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4ac101440 0x7fa4ac19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 >> 172.21.15.3:0/1797867115 conn(0x7fa4ac0fcce0 msgr2=0x7fa4ac0fd850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 shutdown_connections 2024-02-09T22:57:35.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.791+0000 7fa4b0a81640 1 -- 172.21.15.3:0/1797867115 wait complete. 2024-02-09T22:57:35.806 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:35.806 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:57:35.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/585276579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd08105420 msgr2=0x7fdd0810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/585276579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd08105420 0x7fdd0810cff0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009cd0 tx=0x7fdd0002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/585276579 shutdown_connections 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/585276579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd08105420 0x7fdd0810cff0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/585276579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08100d50 0x7fdd08104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/585276579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd081003a0 0x7fdd08100780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/585276579 >> 172.21.15.3:0/585276579 conn(0x7fdd08077300 msgr2=0x7fdd08077710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/585276579 shutdown_connections 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.908+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/585276579 wait complete. 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 Processor -- start 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 -- start start 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 0x7fdd0819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd08105420 0x7fdd0819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08197720 0x7fdd08197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd081981f0 con 0x7fdd081003a0 2024-02-09T22:57:35.915 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd08198360 con 0x7fdd08105420 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0ea4d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd08198520 con 0x7fdd08197720 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 0x7fdd0819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 0x7fdd0819d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53328/0 (socket says 172.21.15.3:53328) 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 -- 172.21.15.3:0/3253158312 learned_addr learned my addr 172.21.15.3:0/3253158312 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd08105420 0x7fdd0819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08197720 0x7fdd08197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 -- 172.21.15.3:0/3253158312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08197720 msgr2=0x7fdd08197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08197720 0x7fdd08197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.916 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 -- 172.21.15.3:0/3253158312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd08105420 msgr2=0x7fdd0819db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd08105420 0x7fdd0819db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd07fff640 1 -- 172.21.15.3:0/3253158312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd00009900 con 0x7fdd081003a0 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd08105420 0x7fdd0819db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.909+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08197720 0x7fdd08197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.910+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 0x7fdd0819d610 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800d950 tx=0x7fdcf800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.910+0000 7fdd057fa640 1 -- 172.21.15.3:0/3253158312 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8009a30 con 0x7fdd081003a0 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.910+0000 7fdd057fa640 1 -- 172.21.15.3:0/3253158312 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcf8009b90 con 0x7fdd081003a0 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.910+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd081a4470 con 0x7fdd081003a0 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.910+0000 7fdd057fa640 1 -- 172.21.15.3:0/3253158312 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf8004a00 con 0x7fdd081003a0 2024-02-09T22:57:35.917 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.910+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd081a4950 con 0x7fdd081003a0 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.912+0000 7fdd057fa640 1 -- 172.21.15.3:0/3253158312 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fdcf8009d00 con 0x7fdd081003a0 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.912+0000 7fdd11619740 1 Processor -- start 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.912+0000 7fdd11619740 1 -- start start 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.912+0000 7fdd057fa640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fdcf00a2230 0x7fdcf00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.912+0000 7fdd11619740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd54ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd11619740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd553d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd11619740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd554820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd11619740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd506020 con 0x5639fd54b240 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd11619740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd4b9320 con 0x5639fd54a890 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd11619740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd4b6850 con 0x5639fd5543a0 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fdcf00a2230 0x7fdcf00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd057fa640 1 -- 172.21.15.3:0/3253158312 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fdcf80cdba0 con 0x7fdd081003a0 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd553d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd553d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53330/0 (socket says 172.21.15.3:53330) 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 -- 172.21.15.3:0/3774904022 learned_addr learned my addr 172.21.15.3:0/3774904022 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd554820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 -- 172.21.15.3:0/3774904022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 msgr2=0x5639fd554820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd554820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 -- 172.21.15.3:0/3774904022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 msgr2=0x5639fd54ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd54ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 -- 172.21.15.3:0/3774904022 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639fd474210 con 0x5639fd54b240 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fdcf00a2230 0x7fdcf00a46f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdd00033000 tx=0x7fdd000325d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd553d80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00bc80 tx=0x7fdcfc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f96ab4b7b35ead9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdcee7fc640 1 -- 172.21.15.3:0/3774904022 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc005020 con 0x5639fd54b240 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdcee7fc640 1 -- 172.21.15.3:0/3774904022 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcfc005660 con 0x5639fd54b240 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.913+0000 7fdcee7fc640 1 -- 172.21.15.3:0/3774904022 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc0116a0 con 0x5639fd54b240 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 -- 172.21.15.3:0/3774904022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 msgr2=0x5639fd553d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd553d80 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00bc80 tx=0x7fdcfc00f730 comp rx=0 tx=0).stop 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 -- 172.21.15.3:0/3774904022 shutdown_connections 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd554820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd553d80 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 --2- 172.21.15.3:0/3774904022 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd54ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 -- 172.21.15.3:0/3774904022 >> 172.21.15.3:0/3774904022 conn(0x5639fd545100 msgr2=0x5639fd5454e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 -- 172.21.15.3:0/3774904022 shutdown_connections 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 -- 172.21.15.3:0/3774904022 wait complete. 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.914+0000 7fdd11619740 1 -- start start 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 Processor -- start 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 0x5639fd557ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54b240 0x5639fd5583e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd558920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd4b2030 con 0x5639fd54a890 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd185430 con 0x5639fd54b240 2024-02-09T22:57:35.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd141e00 con 0x5639fd5543a0 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 10 client.0 _ll_register_callbacks cb 0x5639fd4a7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 0x5639fd557ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 0x5639fd557ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53338/0 (socket says 172.21.15.3:53338) 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 -- 172.21.15.3:0/3766244419 learned_addr learned my addr 172.21.15.3:0/3766244419 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54b240 0x5639fd5583e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 -- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 msgr2=0x5639fd558920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd558920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 -- 172.21.15.3:0/3766244419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54b240 msgr2=0x5639fd5583e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54b240 0x5639fd5583e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 -- 172.21.15.3:0/3766244419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc00b8e0 con 0x5639fd54a890 2024-02-09T22:57:35.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 0x5639fd557ea0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdcf809a260 tx=0x7fdcf8085220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d989376e190abf6c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdce77fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdce77fe640 1 -- 172.21.15.3:0/3766244419 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcf809e6b0 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdce77fe640 1 -- 172.21.15.3:0/3766244419 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fdcf80a2d00 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.915+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5639fd1b17e0 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdce77fe640 1 -- 172.21.15.3:0/3766244419 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcf8005680 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 10 client.17784 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5639fd4b06b0 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdce77fe640 1 -- 172.21.15.3:0/3766244419 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5639fd1b17e0 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdce77fe640 1 -- 172.21.15.3:0/3766244419 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fdcf80cb540 con 0x5639fd54a890 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 10 client.17784 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:35.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 10 client.17784 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 20 client.17784 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5689d0 0x5639fd56adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 4 client.17784 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5639fd1ee720 con 0x5639fd5689d0 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.916+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5689d0 0x5639fd56adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.917+0000 7fdce77fe640 10 client.17784 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.917+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5689d0 0x5639fd56adc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=bd1d29fe16ee2e3e server_cookie=9320d1ec52eb1653 in_seq=0 out_seq=0 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.917+0000 7fdce77fe640 1 -- 172.21.15.3:0/3766244419 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5639fd1ee720 con 0x5639fd5689d0 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.917+0000 7fdce77fe640 10 client.17784 handle_command_reply: tid=0 2024-02-09T22:57:35.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 client.17784 shutdown 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 msgr2=0x5639fd557ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 0x5639fd557ea0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fdcf809a260 tx=0x7fdcf8085220 comp rx=0 tx=0).stop 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 shutdown_connections 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5689d0 0x5639fd56adc0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54b240 0x5639fd5583e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd558920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 --2- 172.21.15.3:0/3766244419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54a890 0x5639fd557ea0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdce77fe640 0 client.17784 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:35.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 >> 172.21.15.3:0/3766244419 conn(0x5639fd545100 msgr2=0x5639fd546e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 shutdown_connections 2024-02-09T22:57:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 1 -- 172.21.15.3:0/3766244419 wait complete. 2024-02-09T22:57:35.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.918+0000 7fdd11619740 20 client.17784 trim_cache size 0 max 16384 2024-02-09T22:57:35.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.929+0000 7fdd11619740 1 Processor -- start 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.929+0000 7fdd11619740 1 -- start start 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd11619740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd11619740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd540420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd11619740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd555790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd11619740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd437130 con 0x5639fd54b240 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd11619740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd540ab0 con 0x5639fd54a890 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd11619740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd556ea0 con 0x5639fd5543a0 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd540420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd540420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53342/0 (socket says 172.21.15.3:53342) 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 -- 172.21.15.3:0/1437076129 learned_addr learned my addr 172.21.15.3:0/1437076129 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd555790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 -- 172.21.15.3:0/1437076129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 msgr2=0x5639fd555790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd07fff640 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd555790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 -- 172.21.15.3:0/1437076129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 msgr2=0x5639fd55a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 -- 172.21.15.3:0/1437076129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5639fd554130 con 0x5639fd54b240 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd07fff640 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:35.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdd077fe640 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd540420 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00bc80 tx=0x7fdcfc01d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d508421f4e3ca69d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdcedffb640 1 -- 172.21.15.3:0/1437076129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc020030 con 0x5639fd54b240 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdcedffb640 1 -- 172.21.15.3:0/1437076129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcfc021040 con 0x5639fd54b240 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.930+0000 7fdcedffb640 1 -- 172.21.15.3:0/1437076129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc0282b0 con 0x5639fd54b240 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 -- 172.21.15.3:0/1437076129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 msgr2=0x5639fd540420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd540420 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00bc80 tx=0x7fdcfc01d040 comp rx=0 tx=0).stop 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 -- 172.21.15.3:0/1437076129 shutdown_connections 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd555790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd540420 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 --2- 172.21.15.3:0/1437076129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 -- 172.21.15.3:0/1437076129 >> 172.21.15.3:0/1437076129 conn(0x5639fd545100 msgr2=0x5639fd556790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 -- 172.21.15.3:0/1437076129 shutdown_connections 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 -- 172.21.15.3:0/1437076129 wait complete. 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 -- start start 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.931+0000 7fdd11619740 1 Processor -- start 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd55e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd5beed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd55f0a0 con 0x5639fd54b240 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd5bf410 con 0x5639fd54a890 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5639fd5bf550 con 0x5639fd5543a0 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd11619740 10 client.0 _ll_register_callbacks cb 0x7fdcfc00b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd55e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd55e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53354/0 (socket says 172.21.15.3:53354) 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 -- 172.21.15.3:0/3295244810 learned_addr learned my addr 172.21.15.3:0/3295244810 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd0cfea640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd5beed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 -- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 msgr2=0x5639fd5beed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd5beed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 -- 172.21.15.3:0/3295244810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 msgr2=0x5639fd55e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 -- 172.21.15.3:0/3295244810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5639fd474210 con 0x5639fd54b240 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdd077fe640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd55e9a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f770 tx=0x7fdcfc00ff00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b046f878c5ac7ffc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:35.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdce77fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdce77fe640 1 -- 172.21.15.3:0/3295244810 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcfc020040 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.932+0000 7fdce77fe640 1 -- 172.21.15.3:0/3295244810 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fdcfc005280 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdcfc005660 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdce77fe640 1 -- 172.21.15.3:0/3295244810 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdcfc005440 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 10 client.17790 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5639fd44cc90 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdce77fe640 1 -- 172.21.15.3:0/3295244810 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdcfc005660 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdce77fe640 1 -- 172.21.15.3:0/3295244810 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fdcfc0116e0 con 0x5639fd54b240 2024-02-09T22:57:35.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 10 client.17790 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 10 client.17790 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:57:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 20 client.17790 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5c0990 0x5639fd5c2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 4 client.17790 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:35.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5639fd23d890 con 0x5639fd5c0990 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.933+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5c0990 0x5639fd5c2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd07fff640 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5c0990 0x5639fd5c2db0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3e2555ff11cd5d67 server_cookie=59af17e685d8fd2 in_seq=0 out_seq=0 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdce77fe640 10 client.17790 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdce77fe640 1 -- 172.21.15.3:0/3295244810 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5639fd23d890 con 0x5639fd5c0990 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdce77fe640 10 client.17790 handle_command_reply: tid=0 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 client.17790 shutdown 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 msgr2=0x5639fd55e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd55e9a0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc00f770 tx=0x7fdcfc00ff00 comp rx=0 tx=0).stop 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 shutdown_connections 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5639fd5c0990 0x5639fd5c2db0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5639fd5543a0 0x5639fd5beed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdce77fe640 0 client.17790 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5639fd54b240 0x5639fd55e9a0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 --2- 172.21.15.3:0/3295244810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5639fd54a890 0x5639fd55e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 >> 172.21.15.3:0/3295244810 conn(0x5639fd545100 msgr2=0x5639fd556360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 shutdown_connections 2024-02-09T22:57:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 1 -- 172.21.15.3:0/3295244810 wait complete. 2024-02-09T22:57:35.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.934+0000 7fdd11619740 20 client.17790 trim_cache size 0 max 16384 2024-02-09T22:57:35.950 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:35.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fdcf00a2230 msgr2=0x7fdcf00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fdcf00a2230 0x7fdcf00a46f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdd00033000 tx=0x7fdd000325d0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 msgr2=0x7fdd0819d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 0x7fdd0819d610 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7fdcf800d950 tx=0x7fdcf800de10 comp rx=0 tx=0).stop 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 shutdown_connections 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fdcf00a2230 0x7fdcf00a46f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fdd08105420 0x7fdd0819db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fdd08197720 0x7fdd08197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 --2- 172.21.15.3:0/3253158312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fdd081003a0 0x7fdd0819d610 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 >> 172.21.15.3:0/3253158312 conn(0x7fdd08077300 msgr2=0x7fdd080fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 shutdown_connections 2024-02-09T22:57:35.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:35.935+0000 7fdd0ea4d640 1 -- 172.21.15.3:0/3253158312 wait complete. 2024-02-09T22:57:35.956 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:35.956 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:57:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.056+0000 7f3957655640 1 -- 172.21.15.3:0/3955092063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 msgr2=0x7f3950100ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.056+0000 7f3957655640 1 --2- 172.21.15.3:0/3955092063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f3950100ea0 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7f3948009930 tx=0x7f394802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- 172.21.15.3:0/3955092063 shutdown_connections 2024-02-09T22:57:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 --2- 172.21.15.3:0/3955092063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3950101470 0x7f3950104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 --2- 172.21.15.3:0/3955092063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3950105400 0x7f395010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 --2- 172.21.15.3:0/3955092063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f3950100ea0 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- 172.21.15.3:0/3955092063 >> 172.21.15.3:0/3955092063 conn(0x7f3950076630 msgr2=0x7f39500fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- 172.21.15.3:0/3955092063 shutdown_connections 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- 172.21.15.3:0/3955092063 wait complete. 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 Processor -- start 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- start start 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f395019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3950105400 0x7f395019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f39501976c0 0x7f3950197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3950198190 con 0x7f3950100ac0 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3950198300 con 0x7f39501976c0 2024-02-09T22:57:36.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.057+0000 7f3957655640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39501984c0 con 0x7f3950105400 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f395019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f395019d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53384/0 (socket says 172.21.15.3:53384) 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 -- 172.21.15.3:0/1136643936 learned_addr learned my addr 172.21.15.3:0/1136643936 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f3954bf0640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3950105400 0x7f395019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 -- 172.21.15.3:0/1136643936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3950105400 msgr2=0x7f395019db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3950105400 0x7f395019db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f3955bf2640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f39501976c0 0x7f3950197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 -- 172.21.15.3:0/1136643936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f39501976c0 msgr2=0x7f3950197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f39501976c0 0x7f3950197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 -- 172.21.15.3:0/1136643936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3948009560 con 0x7f3950100ac0 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f39553f1640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f395019d600 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f3948009230 tx=0x7f394802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f394e7fc640 1 -- 172.21.15.3:0/1136643936 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948004350 con 0x7f3950100ac0 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39501a4400 con 0x7f3950100ac0 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f394e7fc640 1 -- 172.21.15.3:0/1136643936 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39480044b0 con 0x7f3950100ac0 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f394e7fc640 1 -- 172.21.15.3:0/1136643936 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948003d00 con 0x7f3950100ac0 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.058+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39501a48e0 con 0x7f3950100ac0 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.060+0000 7f394e7fc640 1 -- 172.21.15.3:0/1136643936 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3948004df0 con 0x7f3950100ac0 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 Processor -- start 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 -- start start 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f394e7fc640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39380a2230 0x7f39380a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac17bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac184d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac1853a0 0x563cac185820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac137020 con 0x563cac17b890 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac0ea320 con 0x563cac1853a0 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f395a22c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac0e7850 con 0x563cac17c240 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f3955bf2640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39380a2230 0x7f39380a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f394e7fc640 1 -- 172.21.15.3:0/1136643936 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f39480effe0 con 0x7f3950100ac0 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f39553f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac17bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f39553f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac17bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53392/0 (socket says 172.21.15.3:53392) 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.061+0000 7f39553f1640 1 -- 172.21.15.3:0/1416823820 learned_addr learned my addr 172.21.15.3:0/1416823820 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39553f1640 1 -- 172.21.15.3:0/1416823820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 msgr2=0x563cac184d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f3954bf0640 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac184d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39553f1640 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac184d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39553f1640 1 -- 172.21.15.3:0/1416823820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac1853a0 msgr2=0x563cac185820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39553f1640 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac1853a0 0x563cac185820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39553f1640 1 -- 172.21.15.3:0/1416823820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cac0a5210 con 0x563cac17b890 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f3954bf0640 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac184d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f3955bf2640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39380a2230 0x7f39380a46f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3940005fd0 tx=0x7f39400099c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39553f1640 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac17bc70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f39480bcaa0 tx=0x7f39480037d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec4297a75ede5166 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39377fe640 1 -- 172.21.15.3:0/1416823820 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394808f7f0 con 0x563cac17b890 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39377fe640 1 -- 172.21.15.3:0/1416823820 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f394808ed80 con 0x563cac17b890 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.062+0000 7f39377fe640 1 -- 172.21.15.3:0/1416823820 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394808ef70 con 0x563cac17b890 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 -- 172.21.15.3:0/1416823820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 msgr2=0x563cac17bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac17bc70 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f39480bcaa0 tx=0x7f39480037d0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 -- 172.21.15.3:0/1416823820 shutdown_connections 2024-02-09T22:57:36.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac1853a0 0x563cac185820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac184d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 --2- 172.21.15.3:0/1416823820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac17bc70 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 -- 172.21.15.3:0/1416823820 >> 172.21.15.3:0/1416823820 conn(0x563cac176100 msgr2=0x563cac1764e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 -- 172.21.15.3:0/1416823820 shutdown_connections 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 -- 172.21.15.3:0/1416823820 wait complete. 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 -- start start 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.063+0000 7f395a22c740 1 Processor -- start 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac188d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac1892b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac1897f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac0e3030 con 0x563cac1853a0 2024-02-09T22:57:36.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cabdb6430 con 0x563cac17b890 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cabd72e00 con 0x563cac17c240 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 10 client.0 _ll_register_callbacks cb 0x563cac0d8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac1897f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac1897f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53398/0 (socket says 172.21.15.3:53398) 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 -- 172.21.15.3:0/3159089815 learned_addr learned my addr 172.21.15.3:0/3159089815 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3954bf0640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac1892b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f39553f1640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac188d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 -- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 msgr2=0x563cac1892b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac1892b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 -- 172.21.15.3:0/3159089815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 msgr2=0x563cac188d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac188d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 -- 172.21.15.3:0/3159089815 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39501a48e0 con 0x563cac1853a0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f39553f1640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac188d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac1897f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3940012770 tx=0x7f3940004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdb4056694d2b58 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3934ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3934ff9640 1 -- 172.21.15.3:0/3159089815 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f394000ea90 con 0x563cac1853a0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f3934ff9640 1 -- 172.21.15.3:0/3159089815 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3940018040 con 0x563cac1853a0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.064+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563cabde27e0 con 0x563cac1853a0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f3934ff9640 1 -- 172.21.15.3:0/3159089815 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f394000ebf0 con 0x563cac1853a0 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 10 client.17802 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x563cac0e16b0 con 0x563cac1853a0 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f3934ff9640 1 -- 172.21.15.3:0/3159089815 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563cabde27e0 con 0x563cac1853a0 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f3934ff9640 1 -- 172.21.15.3:0/3159089815 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f39400273b0 con 0x563cac1853a0 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 10 client.17802 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 10 client.17802 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 20 client.17802 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac199590 0x563cac19b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 4 client.17802 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563cabe1f720 con 0x563cac199590 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.065+0000 7f39553f1640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac199590 0x563cac19b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f3934ff9640 10 client.17802 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:36.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f39553f1640 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac199590 0x563cac19b9b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c6fb10d1fe31af4c server_cookie=252dc18f2946c9c5 in_seq=0 out_seq=0 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f3934ff9640 1 -- 172.21.15.3:0/3159089815 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563cabe1f720 con 0x563cac199590 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f3934ff9640 10 client.17802 handle_command_reply: tid=0 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f395a22c740 1 client.17802 shutdown 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 msgr2=0x563cac1897f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.066+0000 7f395a22c740 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac1897f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f3940012770 tx=0x7f3940004ab0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 shutdown_connections 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac199590 0x563cac19b9b0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac1897f0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f3934ff9640 0 client.17802 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac1892b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 --2- 172.21.15.3:0/3159089815 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac188d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 >> 172.21.15.3:0/3159089815 conn(0x563cac176100 msgr2=0x563cac177e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 shutdown_connections 2024-02-09T22:57:36.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 1 -- 172.21.15.3:0/3159089815 wait complete. 2024-02-09T22:57:36.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.067+0000 7f395a22c740 20 client.17802 trim_cache size 0 max 16384 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.078+0000 7f395a22c740 1 Processor -- start 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.078+0000 7f395a22c740 1 -- start start 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f395a22c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac18b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f395a22c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17c240 0x563cac186790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f395a22c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac1853a0 0x563cac189610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f395a22c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac068130 con 0x563cac17b890 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f395a22c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac1759c0 con 0x563cac17c240 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f395a22c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac175fc0 con 0x563cac1853a0 2024-02-09T22:57:36.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac18b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac18b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53400/0 (socket says 172.21.15.3:53400) 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 -- 172.21.15.3:0/3186224975 learned_addr learned my addr 172.21.15.3:0/3186224975 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac1853a0 0x563cac189610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f3954bf0640 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17c240 0x563cac186790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 -- 172.21.15.3:0/3186224975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac1853a0 msgr2=0x563cac189610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac1853a0 0x563cac189610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 -- 172.21.15.3:0/3186224975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17c240 msgr2=0x563cac186790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17c240 0x563cac186790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 -- 172.21.15.3:0/3186224975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563cac07dc90 con 0x563cac17b890 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f3954bf0640 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17c240 0x563cac186790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:36.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f39553f1640 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac18b450 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3948005ec0 tx=0x7f39480b8ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1517d185c91a7a06 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f3936ffd640 1 -- 172.21.15.3:0/3186224975 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3948003e60 con 0x563cac17b890 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f3936ffd640 1 -- 172.21.15.3:0/3186224975 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39480bd210 con 0x563cac17b890 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.079+0000 7f3936ffd640 1 -- 172.21.15.3:0/3186224975 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39480bae10 con 0x563cac17b890 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 -- 172.21.15.3:0/3186224975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 msgr2=0x563cac18b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac18b450 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7f3948005ec0 tx=0x7f39480b8ee0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 -- 172.21.15.3:0/3186224975 shutdown_connections 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17c240 0x563cac186790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac1853a0 0x563cac189610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 --2- 172.21.15.3:0/3186224975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac17b890 0x563cac18b450 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 -- 172.21.15.3:0/3186224975 >> 172.21.15.3:0/3186224975 conn(0x563cac176100 msgr2=0x563cac187640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 -- 172.21.15.3:0/3186224975 shutdown_connections 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 -- 172.21.15.3:0/3186224975 wait complete. 2024-02-09T22:57:36.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 -- start start 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.080+0000 7f395a22c740 1 Processor -- start 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac190310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac18e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac18ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac18f240 con 0x563cac1853a0 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac18f3b0 con 0x563cac17b890 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563cac18f570 con 0x563cac17c240 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f395a22c740 10 client.0 _ll_register_callbacks cb 0x7f3940011f90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac18ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac18ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53412/0 (socket says 172.21.15.3:53412) 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 -- 172.21.15.3:0/3337019344 learned_addr learned my addr 172.21.15.3:0/3337019344 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f39553f1640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac190310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3954bf0640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac18e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 -- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 msgr2=0x563cac18e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac18e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 -- 172.21.15.3:0/3337019344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 msgr2=0x563cac190310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac190310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 -- 172.21.15.3:0/3337019344 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563cac0a5210 con 0x563cac1853a0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3955bf2640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac18ed00 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3940015040 tx=0x7f3940027f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a35ce499c40bd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3934ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.081+0000 7f3934ff9640 1 -- 172.21.15.3:0/3337019344 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3940027930 con 0x563cac1853a0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f3934ff9640 1 -- 172.21.15.3:0/3337019344 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3940018040 con 0x563cac1853a0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f394808ed80 con 0x563cac1853a0 2024-02-09T22:57:36.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f3934ff9640 1 -- 172.21.15.3:0/3337019344 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3940027a90 con 0x563cac1853a0 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 10 client.17808 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x563cac0a3a70 con 0x563cac1853a0 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f3934ff9640 1 -- 172.21.15.3:0/3337019344 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f394808ed80 con 0x563cac1853a0 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f3934ff9640 1 -- 172.21.15.3:0/3337019344 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3940028710 con 0x563cac1853a0 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 10 client.17808 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 10 client.17808 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 20 client.17808 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac1f0f20 0x563cac1f3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 4 client.17808 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563cabe6e890 con 0x563cac1f0f20 2024-02-09T22:57:36.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.082+0000 7f39553f1640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac1f0f20 0x563cac1f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f3934ff9640 10 client.17808 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f39553f1640 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac1f0f20 0x563cac1f3340 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a6cd910f68a99545 server_cookie=fbdd7dd3a56dafc5 in_seq=0 out_seq=0 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f3934ff9640 1 -- 172.21.15.3:0/3337019344 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563cabe6e890 con 0x563cac1f0f20 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f3934ff9640 10 client.17808 handle_command_reply: tid=0 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 client.17808 shutdown 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 msgr2=0x563cac18ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac18ed00 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7f3940015040 tx=0x7f3940027f90 comp rx=0 tx=0).stop 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 shutdown_connections 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x563cac1f0f20 0x563cac1f3340 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563cac1853a0 0x563cac18ed00 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563cac17c240 0x563cac18e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f3934ff9640 0 client.17808 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:57:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 --2- 172.21.15.3:0/3337019344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563cac17b890 0x563cac190310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 >> 172.21.15.3:0/3337019344 conn(0x563cac176100 msgr2=0x563cac1749b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 shutdown_connections 2024-02-09T22:57:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 1 -- 172.21.15.3:0/3337019344 wait complete. 2024-02-09T22:57:36.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.083+0000 7f395a22c740 20 client.17808 trim_cache size 0 max 16384 2024-02-09T22:57:36.099 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39380a2230 msgr2=0x7f39380a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39380a2230 0x7f39380a46f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3940005fd0 tx=0x7f39400099c0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 msgr2=0x7f395019d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f395019d600 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f3948009230 tx=0x7f394802c650 comp rx=0 tx=0).stop 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 shutdown_connections 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f39380a2230 0x7f39380a46f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f39501976c0 0x7f3950197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3950105400 0x7f395019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 --2- 172.21.15.3:0/1136643936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3950100ac0 0x7f395019d600 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 >> 172.21.15.3:0/1136643936 conn(0x7f3950076630 msgr2=0x7f39500ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 shutdown_connections 2024-02-09T22:57:36.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.084+0000 7f3957655640 1 -- 172.21.15.3:0/1136643936 wait complete. 2024-02-09T22:57:36.102 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:36.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 -- 172.21.15.3:0/1390574353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8450101de0 msgr2=0x7f8450105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 --2- 172.21.15.3:0/1390574353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8450101de0 0x7f8450105e20 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f8440009930 tx=0x7f844002bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 -- 172.21.15.3:0/1390574353 shutdown_connections 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 --2- 172.21.15.3:0/1390574353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8450106500 0x7f845010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 --2- 172.21.15.3:0/1390574353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8450101de0 0x7f8450105e20 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 --2- 172.21.15.3:0/1390574353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8450101430 0x7f8450101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 -- 172.21.15.3:0/1390574353 >> 172.21.15.3:0/1390574353 conn(0x7f84500fccd0 msgr2=0x7f84500ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 -- 172.21.15.3:0/1390574353 shutdown_connections 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 -- 172.21.15.3:0/1390574353 wait complete. 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.203+0000 7f8457fb5640 1 Processor -- start 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 -- start start 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8450101430 0x7f845019d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8450106500 0x7f845019db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 0x7f8450197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450198190 con 0x7f84501976c0 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8450198300 con 0x7f8450101430 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84501984c0 con 0x7f8450106500 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 0x7f8450197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 0x7f8450197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53420/0 (socket says 172.21.15.3:53420) 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 -- 172.21.15.3:0/3748384645 learned_addr learned my addr 172.21.15.3:0/3748384645 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8455d51640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8450101430 0x7f845019d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8455550640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8450106500 0x7f845019db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 -- 172.21.15.3:0/3748384645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8450106500 msgr2=0x7f845019db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8450106500 0x7f845019db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 -- 172.21.15.3:0/3748384645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8450101430 msgr2=0x7f845019d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8450101430 0x7f845019d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 -- 172.21.15.3:0/3748384645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8440009560 con 0x7f84501976c0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8456552640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 0x7f8450197b40 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f844800eb10 tx=0x7f844800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8446ffd640 1 -- 172.21.15.3:0/3748384645 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448004b20 con 0x7f84501976c0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8446ffd640 1 -- 172.21.15.3:0/3748384645 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8448004c80 con 0x7f84501976c0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84501a4460 con 0x7f84501976c0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.204+0000 7f8446ffd640 1 -- 172.21.15.3:0/3748384645 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8448010650 con 0x7f84501976c0 2024-02-09T22:57:36.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.205+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84501a4990 con 0x7f84501976c0 2024-02-09T22:57:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f8446ffd640 1 -- 172.21.15.3:0/3748384645 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8448005020 con 0x7f84501976c0 2024-02-09T22:57:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f845ab89740 1 Processor -- start 2024-02-09T22:57:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f845ab89740 1 -- start start 2024-02-09T22:57:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f8446ffd640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f843c0a22b0 0x7f843c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f845ab89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045ece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f845ab89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045f5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.207+0000 7f845ab89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f845ab89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597045a8020 con 0x5597045eca30 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f845ab89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55970455b320 con 0x5597045ed3e0 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f845ab89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559704558850 con 0x5597045f5f60 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045ece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045ece10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53432/0 (socket says 172.21.15.3:53432) 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 -- 172.21.15.3:0/283447635 learned_addr learned my addr 172.21.15.3:0/283447635 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8446ffd640 1 -- 172.21.15.3:0/3748384645 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f8448014070 con 0x7f84501976c0 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455550640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f843c0a22b0 0x7f843c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8456552640 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455550640 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045f5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 -- 172.21.15.3:0/283447635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 msgr2=0x5597045f83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 -- 172.21.15.3:0/283447635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 msgr2=0x5597045f5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045f5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 -- 172.21.15.3:0/283447635 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559704516210 con 0x5597045eca30 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455d51640 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045ece10 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f844c00bcd0 tx=0x7f844c0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa5e4a3d9f4302e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f8455550640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f843c0a22b0 0x7f843c0a4770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8440000c00 tx=0x7f844002c630 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f843bfff640 1 -- 172.21.15.3:0/283447635 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c005020 con 0x5597045eca30 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f843bfff640 1 -- 172.21.15.3:0/283447635 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f844c015070 con 0x5597045eca30 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.208+0000 7f843bfff640 1 -- 172.21.15.3:0/283447635 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c0116a0 con 0x5597045eca30 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 -- 172.21.15.3:0/283447635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 msgr2=0x5597045ece10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045ece10 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7f844c00bcd0 tx=0x7f844c0107a0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 -- 172.21.15.3:0/283447635 shutdown_connections 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045f5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 --2- 172.21.15.3:0/283447635 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045ece10 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 -- 172.21.15.3:0/283447635 >> 172.21.15.3:0/283447635 conn(0x5597045e7100 msgr2=0x5597045e74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 -- 172.21.15.3:0/283447635 shutdown_connections 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 -- 172.21.15.3:0/283447635 wait complete. 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.209+0000 7f845ab89740 1 -- start start 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 Processor -- start 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045ed3e0 0x5597045f3430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045f5f60 0x5597045f3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559704554030 con 0x5597045eca30 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559704227430 con 0x5597045f5f60 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597041e3e00 con 0x5597045ed3e0 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f845ab89740 10 client.0 _ll_register_callbacks cb 0x559704549030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045f2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53438/0 (socket says 172.21.15.3:53438) 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 -- 172.21.15.3:0/4275899078 learned_addr learned my addr 172.21.15.3:0/4275899078 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455550640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045ed3e0 0x5597045f3430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8456552640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045f5f60 0x5597045f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 -- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045ed3e0 msgr2=0x5597045f3430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045ed3e0 0x5597045f3430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 -- 172.21.15.3:0/4275899078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045f5f60 msgr2=0x5597045f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045f5f60 0x5597045f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.210+0000 7f8455d51640 1 -- 172.21.15.3:0/4275899078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f844c00b900 con 0x5597045eca30 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f8455d51640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045f2ef0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f844c004db0 tx=0x7f844c004960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be72e0e5d6519f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f84397fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f84397fa640 1 -- 172.21.15.3:0/4275899078 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844c021070 con 0x5597045eca30 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f84397fa640 1 -- 172.21.15.3:0/4275899078 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f844c017070 con 0x5597045eca30 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5597042537e0 con 0x5597045eca30 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f84397fa640 1 -- 172.21.15.3:0/4275899078 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f844c005060 con 0x5597045eca30 2024-02-09T22:57:36.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 10 client.17820 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5597045526b0 con 0x5597045eca30 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f84397fa640 1 -- 172.21.15.3:0/4275899078 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5597042537e0 con 0x5597045eca30 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f84397fa640 1 -- 172.21.15.3:0/4275899078 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f844c011c10 con 0x5597045eca30 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 10 client.17820 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 10 client.17820 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 20 client.17820 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x559704606410 0x559704608800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 4 client.17820 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559704290720 con 0x559704606410 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.211+0000 7f8455550640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x559704606410 0x559704608800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.212+0000 7f8455550640 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x559704606410 0x559704608800 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=be910c1dcf23ab56 server_cookie=7dfed4ab527f058f in_seq=0 out_seq=0 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.212+0000 7f84397fa640 10 client.17820 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:36.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f84397fa640 1 -- 172.21.15.3:0/4275899078 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559704290720 con 0x559704606410 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f84397fa640 10 client.17820 handle_command_reply: tid=0 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 client.17820 shutdown 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 msgr2=0x5597045f2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045f2ef0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f844c004db0 tx=0x7f844c004960 comp rx=0 tx=0).stop 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 shutdown_connections 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x559704606410 0x559704608800 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045f5f60 0x5597045f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045ed3e0 0x5597045f3430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 --2- 172.21.15.3:0/4275899078 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045eca30 0x5597045f2ef0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f84397fa640 0 client.17820 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 >> 172.21.15.3:0/4275899078 conn(0x5597045e7100 msgr2=0x5597045e8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 shutdown_connections 2024-02-09T22:57:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 1 -- 172.21.15.3:0/4275899078 wait complete. 2024-02-09T22:57:36.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.213+0000 7f845ab89740 20 client.17820 trim_cache size 0 max 16384 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 Processor -- start 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 -- start start 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045eca30 0x5597045f5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 0x5597045f3440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597045e28b0 con 0x5597045ed3e0 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597044d9130 con 0x5597045eca30 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f845ab89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5597045ebbc0 con 0x5597045f5f60 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 0x5597045f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 0x5597045f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53454/0 (socket says 172.21.15.3:53454) 2024-02-09T22:57:36.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 -- 172.21.15.3:0/3137028941 learned_addr learned my addr 172.21.15.3:0/3137028941 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8456552640 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 -- 172.21.15.3:0/3137028941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 msgr2=0x5597045f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455d51640 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045eca30 0x5597045f5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 -- 172.21.15.3:0/3137028941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045eca30 msgr2=0x5597045f5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045eca30 0x5597045f5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 -- 172.21.15.3:0/3137028941 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5597044eec90 con 0x5597045ed3e0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8456552640 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.225+0000 7f8455550640 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 0x5597045f3440 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f845006a990 tx=0x7f844002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751fae4091552087 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f843b7fe640 1 -- 172.21.15.3:0/3137028941 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844002cbb0 con 0x5597045ed3e0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f843b7fe640 1 -- 172.21.15.3:0/3137028941 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84400052d0 con 0x5597045ed3e0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f843b7fe640 1 -- 172.21.15.3:0/3137028941 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8440003c50 con 0x5597045ed3e0 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 -- 172.21.15.3:0/3137028941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 msgr2=0x5597045f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 0x5597045f3440 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f845006a990 tx=0x7f844002bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 -- 172.21.15.3:0/3137028941 shutdown_connections 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045f5f60 0x5597045f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045ed3e0 0x5597045f3440 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 --2- 172.21.15.3:0/3137028941 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045eca30 0x5597045f5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 -- 172.21.15.3:0/3137028941 >> 172.21.15.3:0/3137028941 conn(0x5597045e7100 msgr2=0x55970460d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 -- 172.21.15.3:0/3137028941 shutdown_connections 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.226+0000 7f845ab89740 1 -- 172.21.15.3:0/3137028941 wait complete. 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 -- start start 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 Processor -- start 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045eca30 0x5597045fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045fc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 0x5597045fc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55970465ca70 con 0x5597045f5f60 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55970465cbe0 con 0x5597045ed3e0 2024-02-09T22:57:36.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55970465cda0 con 0x5597045eca30 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f845ab89740 10 client.0 _ll_register_callbacks cb 0x7f844c00b810 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8456552640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 0x5597045fc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8456552640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 0x5597045fc640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53466/0 (socket says 172.21.15.3:53466) 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8456552640 1 -- 172.21.15.3:0/404238544 learned_addr learned my addr 172.21.15.3:0/404238544 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8455d51640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045eca30 0x5597045fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8455550640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8456552640 1 -- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045eca30 msgr2=0x5597045fbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8456552640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045eca30 0x5597045fbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.227+0000 7f8456552640 1 -- 172.21.15.3:0/404238544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 msgr2=0x5597045fc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.228+0000 7f8456552640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045fc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.228+0000 7f8456552640 1 -- 172.21.15.3:0/404238544 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f845010e8e0 con 0x5597045f5f60 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.228+0000 7f8455550640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.228+0000 7f8456552640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 0x5597045fc640 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8448097350 tx=0x7f844800cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883807e20c3d4499 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.228+0000 7f84397fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.231+0000 7f84397fa640 1 -- 172.21.15.3:0/404238544 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84480a3690 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.231+0000 7f84397fa640 1 -- 172.21.15.3:0/404238544 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f84480991f0 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.231+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f844c015070 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.231+0000 7f84397fa640 1 -- 172.21.15.3:0/404238544 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f84480840b0 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 10 client.17826 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x559704514a70 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f84397fa640 1 -- 172.21.15.3:0/404238544 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84480b2030 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f84397fa640 1 -- 172.21.15.3:0/404238544 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f844801f060 con 0x5597045f5f60 2024-02-09T22:57:36.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 10 client.17826 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 10 client.17826 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 20 client.17826 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55970465e540 0x559704660960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 4 client.17826 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5597042df890 con 0x55970465e540 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.232+0000 7f8455d51640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55970465e540 0x559704660960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f84397fa640 10 client.17826 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f8455d51640 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55970465e540 0x559704660960 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=284ad2516f412cb0 server_cookie=6b7d1b761739089c in_seq=0 out_seq=0 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f84397fa640 1 -- 172.21.15.3:0/404238544 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5597042df890 con 0x55970465e540 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f84397fa640 10 client.17826 handle_command_reply: tid=0 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 client.17826 shutdown 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 msgr2=0x5597045fc640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 0x5597045fc640 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f8448097350 tx=0x7f844800cf70 comp rx=0 tx=0).stop 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 shutdown_connections 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55970465e540 0x559704660960 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5597045ed3e0 0x5597045fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5597045f5f60 0x5597045fc640 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 --2- 172.21.15.3:0/404238544 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5597045eca30 0x5597045fbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f84397fa640 0 client.17826 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 >> 172.21.15.3:0/404238544 conn(0x5597045e7100 msgr2=0x5597045e59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 shutdown_connections 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 1 -- 172.21.15.3:0/404238544 wait complete. 2024-02-09T22:57:36.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.233+0000 7f845ab89740 20 client.17826 trim_cache size 0 max 16384 2024-02-09T22:57:36.241 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f843c0a22b0 msgr2=0x7f843c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f843c0a22b0 0x7f843c0a4770 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8440000c00 tx=0x7f844002c630 comp rx=0 tx=0).stop 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 msgr2=0x7f8450197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 0x7f8450197b40 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f844800eb10 tx=0x7f844800c2d0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 shutdown_connections 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f843c0a22b0 0x7f843c0a4770 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f84501976c0 0x7f8450197b40 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8450106500 0x7f845019db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 --2- 172.21.15.3:0/3748384645 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8450101430 0x7f845019d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 >> 172.21.15.3:0/3748384645 conn(0x7f84500fccd0 msgr2=0x7f84500fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 shutdown_connections 2024-02-09T22:57:36.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.234+0000 7f8457fb5640 1 -- 172.21.15.3:0/3748384645 wait complete. 2024-02-09T22:57:36.248 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:36.248 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:57:36.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 -- 172.21.15.3:0/391243818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84100ce0 msgr2=0x7f7c84104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/391243818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84100ce0 0x7f7c84104d20 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009930 tx=0x7f7c7802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 -- 172.21.15.3:0/391243818 shutdown_connections 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/391243818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c84105400 0x7f7c8410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/391243818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84100ce0 0x7f7c84104d20 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/391243818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c84100330 0x7f7c84100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 -- 172.21.15.3:0/391243818 >> 172.21.15.3:0/391243818 conn(0x7f7c84077250 msgr2=0x7f7c84077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 -- 172.21.15.3:0/391243818 shutdown_connections 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.349+0000 7f7c89c9d640 1 -- 172.21.15.3:0/391243818 wait complete. 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 Processor -- start 2024-02-09T22:57:36.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 -- start start 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c84100330 0x7f7c8419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 0x7f7c8419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c841976f0 0x7f7c84197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c841981c0 con 0x7f7c84105400 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c84198330 con 0x7f7c84100330 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c89c9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c841984f0 con 0x7f7c841976f0 2024-02-09T22:57:36.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 0x7f7c8419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 0x7f7c8419db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53494/0 (socket says 172.21.15.3:53494) 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 -- 172.21.15.3:0/310258644 learned_addr learned my addr 172.21.15.3:0/310258644 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 -- 172.21.15.3:0/310258644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c841976f0 msgr2=0x7f7c84197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c841976f0 0x7f7c84197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 -- 172.21.15.3:0/310258644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c84100330 msgr2=0x7f7c8419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c837fe640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c84100330 0x7f7c8419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c84100330 0x7f7c8419d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.350+0000 7f7c82ffd640 1 -- 172.21.15.3:0/310258644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c78009560 con 0x7f7c84105400 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.351+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 0x7f7c8419db20 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009230 tx=0x7f7c78004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.351+0000 7f7c80ff9640 1 -- 172.21.15.3:0/310258644 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7802cbc0 con 0x7f7c84105400 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.351+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c841a43e0 con 0x7f7c84105400 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.351+0000 7f7c80ff9640 1 -- 172.21.15.3:0/310258644 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c780052d0 con 0x7f7c84105400 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.351+0000 7f7c80ff9640 1 -- 172.21.15.3:0/310258644 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c78003e30 con 0x7f7c84105400 2024-02-09T22:57:36.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.351+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c841a48c0 con 0x7f7c84105400 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.353+0000 7f7c80ff9640 1 -- 172.21.15.3:0/310258644 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7c78005440 con 0x7f7c84105400 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.353+0000 7f7c8c87a740 1 Processor -- start 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.353+0000 7f7c8c87a740 1 -- start start 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.353+0000 7f7c80ff9640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c6c0a2230 0x7f7c6c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a06de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c80ff9640 1 -- 172.21.15.3:0/310258644 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7c780f02e0 con 0x7f7c84105400 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a076990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c83fff640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c6c0a2230 0x7f7c6c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a0793b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900a029020 con 0x55900a076f60 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559009fdc320 con 0x55900a06e3e0 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559009fd9850 con 0x55900a06da30 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a0793b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a06de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a0793b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53500/0 (socket says 172.21.15.3:53500) 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c83fff640 1 -- 172.21.15.3:0/775628039 learned_addr learned my addr 172.21.15.3:0/775628039 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a06de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:32910/0 (socket says 172.21.15.3:32910) 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a076990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c83fff640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c6c0a2230 0x7f7c6c0a46f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00be80 tx=0x7f7c7c0147e0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 -- 172.21.15.3:0/775628039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 msgr2=0x55900a076990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a076990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 -- 172.21.15.3:0/775628039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 msgr2=0x55900a0793b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a0793b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c837fe640 1 -- 172.21.15.3:0/775628039 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559009f97210 con 0x55900a06da30 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.354+0000 7f7c83fff640 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a0793b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c837fe640 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a06de10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009da0 tx=0x7f7c7400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbdb6d4c565f2f9f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c69ffb640 1 -- 172.21.15.3:0/775628039 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74011440 con 0x55900a06da30 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c69ffb640 1 -- 172.21.15.3:0/775628039 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c74011a80 con 0x55900a06da30 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c69ffb640 1 -- 172.21.15.3:0/775628039 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c74019c20 con 0x55900a06da30 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 -- 172.21.15.3:0/775628039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 msgr2=0x55900a06de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a06de10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7c74009da0 tx=0x7f7c7400f730 comp rx=0 tx=0).stop 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 -- 172.21.15.3:0/775628039 shutdown_connections 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a076990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a0793b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/775628039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a06de10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.355+0000 7f7c8c87a740 1 -- 172.21.15.3:0/775628039 >> 172.21.15.3:0/775628039 conn(0x55900a068100 msgr2=0x55900a0684e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 -- 172.21.15.3:0/775628039 shutdown_connections 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 -- 172.21.15.3:0/775628039 wait complete. 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 -- start start 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 Processor -- start 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06da30 0x55900a073e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06e3e0 0x55900a0743c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a074900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559009fd5030 con 0x55900a076f60 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559009ca8430 con 0x55900a06da30 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559009c64e00 con 0x55900a06e3e0 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c8c87a740 10 client.0 _ll_register_callbacks cb 0x559009fca030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a074900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a074900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53512/0 (socket says 172.21.15.3:53512) 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.356+0000 7f7c83fff640 1 -- 172.21.15.3:0/1575441867 learned_addr learned my addr 172.21.15.3:0/1575441867 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06e3e0 0x55900a0743c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c837fe640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06da30 0x55900a073e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c83fff640 1 -- 172.21.15.3:0/1575441867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06e3e0 msgr2=0x55900a0743c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c83fff640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06e3e0 0x55900a0743c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c83fff640 1 -- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06da30 msgr2=0x55900a073e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c83fff640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06da30 0x55900a073e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c83fff640 1 -- 172.21.15.3:0/1575441867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c74009a00 con 0x55900a076f60 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c837fe640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06da30 0x55900a073e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c83fff640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a074900 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00b800 tx=0x7f7c7c005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4231d35b218ff3cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c62ffd640 1 -- 172.21.15.3:0/1575441867 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c012cd0 con 0x55900a076f60 2024-02-09T22:57:36.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c62ffd640 1 -- 172.21.15.3:0/1575441867 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7c7c012e30 con 0x55900a076f60 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559009cd47e0 con 0x55900a076f60 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c62ffd640 1 -- 172.21.15.3:0/1575441867 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c7c028460 con 0x55900a076f60 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c8c87a740 10 client.17838 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x559009fd36b0 con 0x55900a076f60 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.357+0000 7f7c62ffd640 1 -- 172.21.15.3:0/1575441867 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559009cd47e0 con 0x55900a076f60 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c62ffd640 1 -- 172.21.15.3:0/1575441867 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7c7c031490 con 0x55900a076f60 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c8c87a740 10 client.17838 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c8c87a740 10 client.17838 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c8c87a740 20 client.17838 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0874b0 0x55900a0898d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c8c87a740 4 client.17838 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559009d11720 con 0x55900a0874b0 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c837fe640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0874b0 0x55900a0898d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c62ffd640 10 client.17838 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.358+0000 7f7c837fe640 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0874b0 0x55900a0898d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=25d4faac6bd76578 server_cookie=93f33cc7bc6c4d1e in_seq=0 out_seq=0 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c62ffd640 1 -- 172.21.15.3:0/1575441867 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559009d11720 con 0x55900a0874b0 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c62ffd640 10 client.17838 handle_command_reply: tid=0 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 client.17838 shutdown 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 msgr2=0x55900a074900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a074900 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00b800 tx=0x7f7c7c005850 comp rx=0 tx=0).stop 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 shutdown_connections 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0874b0 0x55900a0898d0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a074900 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06e3e0 0x55900a0743c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c62ffd640 0 client.17838 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/1575441867 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06da30 0x55900a073e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 >> 172.21.15.3:0/1575441867 conn(0x55900a068100 msgr2=0x55900a069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.359+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 shutdown_connections 2024-02-09T22:57:36.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.360+0000 7f7c8c87a740 1 -- 172.21.15.3:0/1575441867 wait complete. 2024-02-09T22:57:36.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.360+0000 7f7c8c87a740 20 client.17838 trim_cache size 0 max 16384 2024-02-09T22:57:36.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 Processor -- start 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 -- start start 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a0765b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a074720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a071100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559009f5a130 con 0x55900a076f60 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900a068f00 con 0x55900a06e3e0 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900a0679c0 con 0x55900a06da30 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.371+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a071100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a071100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53528/0 (socket says 172.21.15.3:53528) 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 -- 172.21.15.3:0/530994795 learned_addr learned my addr 172.21.15.3:0/530994795 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a074720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c837fe640 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a0765b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 -- 172.21.15.3:0/530994795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 msgr2=0x55900a0765b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a0765b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 -- 172.21.15.3:0/530994795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 msgr2=0x55900a074720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a074720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 -- 172.21.15.3:0/530994795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559009f6fc90 con 0x55900a076f60 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c83fff640 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a071100 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c014ce0 tx=0x7f7c7c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1396c45dd01132eb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c697fa640 1 -- 172.21.15.3:0/530994795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c011440 con 0x55900a076f60 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c697fa640 1 -- 172.21.15.3:0/530994795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c7c031a10 con 0x55900a076f60 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.372+0000 7f7c697fa640 1 -- 172.21.15.3:0/530994795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c7c0287d0 con 0x55900a076f60 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 -- 172.21.15.3:0/530994795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 msgr2=0x55900a071100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a071100 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c014ce0 tx=0x7f7c7c00b910 comp rx=0 tx=0).stop 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 -- 172.21.15.3:0/530994795 shutdown_connections 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a06e3e0 0x55900a074720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a076f60 0x55900a071100 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/530994795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a0765b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 -- 172.21.15.3:0/530994795 >> 172.21.15.3:0/530994795 conn(0x55900a068100 msgr2=0x55900a06f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 -- 172.21.15.3:0/530994795 shutdown_connections 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 -- 172.21.15.3:0/530994795 wait complete. 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 -- start start 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 Processor -- start 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a07cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 0x55900a07d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.373+0000 7f7c8c87a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a076f60 0x55900a07d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900a0ddb00 con 0x55900a06e3e0 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900a0ddc70 con 0x55900a076f60 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55900a0dde30 con 0x55900a06da30 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 10 client.0 _ll_register_callbacks cb 0x7f7c74009540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 0x55900a07d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 0x55900a07d0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53544/0 (socket says 172.21.15.3:53544) 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 -- 172.21.15.3:0/4180842836 learned_addr learned my addr 172.21.15.3:0/4180842836 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c837fe640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a07cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c83fff640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a076f60 0x55900a07d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 -- 172.21.15.3:0/4180842836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 msgr2=0x55900a07cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a07cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 -- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a076f60 msgr2=0x55900a07d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a076f60 0x55900a07d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 -- 172.21.15.3:0/4180842836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559009fd36b0 con 0x55900a06e3e0 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c83fff640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a076f60 0x55900a07d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c82ffd640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 0x55900a07d0a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7c780b9780 tx=0x7f7c780bd5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65105b7195873338 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c62ffd640 1 -- 172.21.15.3:0/4180842836 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c780ede90 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c62ffd640 1 -- 172.21.15.3:0/4180842836 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7c780c5550 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c74011a80 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c62ffd640 1 -- 172.21.15.3:0/4180842836 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c780035f0 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 10 client.17844 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x559009f95a70 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.374+0000 7f7c62ffd640 1 -- 172.21.15.3:0/4180842836 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c78088020 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c62ffd640 1 -- 172.21.15.3:0/4180842836 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7c7803c070 con 0x55900a06e3e0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 10 client.17844 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 10 client.17844 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 20 client.17844 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0df4c0 0x55900a0e18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 4 client.17844 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559009d60890 con 0x55900a0df4c0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c837fe640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0df4c0 0x55900a0e18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c62ffd640 10 client.17844 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c837fe640 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0df4c0 0x55900a0e18e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8e353402d500ef48 server_cookie=b4a7da2e1d3075a in_seq=0 out_seq=0 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c62ffd640 1 -- 172.21.15.3:0/4180842836 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559009d60890 con 0x55900a0df4c0 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c62ffd640 10 client.17844 handle_command_reply: tid=0 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.375+0000 7f7c8c87a740 1 client.17844 shutdown 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 msgr2=0x55900a07d0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 0x55900a07d0a0 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7c780b9780 tx=0x7f7c780bd5f0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 shutdown_connections 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55900a0df4c0 0x55900a0e18e0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55900a076f60 0x55900a07d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55900a06e3e0 0x55900a07d0a0 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 --2- 172.21.15.3:0/4180842836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55900a06da30 0x55900a07cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c62ffd640 0 client.17844 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 >> 172.21.15.3:0/4180842836 conn(0x55900a068100 msgr2=0x55900a0669b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 shutdown_connections 2024-02-09T22:57:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 1 -- 172.21.15.3:0/4180842836 wait complete. 2024-02-09T22:57:36.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c8c87a740 20 client.17844 trim_cache size 0 max 16384 2024-02-09T22:57:36.387 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c6c0a2230 msgr2=0x7f7c6c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c6c0a2230 0x7f7c6c0a46f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f7c7c00be80 tx=0x7f7c7c0147e0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 msgr2=0x7f7c8419db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 0x7f7c8419db20 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f7c78009230 tx=0x7f7c78004270 comp rx=0 tx=0).stop 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 shutdown_connections 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c6c0a2230 0x7f7c6c0a46f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c841976f0 0x7f7c84197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c84105400 0x7f7c8419db20 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 --2- 172.21.15.3:0/310258644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c84100330 0x7f7c8419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 >> 172.21.15.3:0/310258644 conn(0x7f7c84077250 msgr2=0x7f7c84069c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.376+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 shutdown_connections 2024-02-09T22:57:36.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.377+0000 7f7c89c9d640 1 -- 172.21.15.3:0/310258644 wait complete. 2024-02-09T22:57:36.389 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:36 smithi003 ceph-mon[32678]: pgmap v193: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 960 KiB/s wr, 103 op/s 2024-02-09T22:57:36.389 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:36 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1797867115' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:57:36.391 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:57:36.391 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:57:36.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.492+0000 7f0cc280e640 1 -- 172.21.15.3:0/272436186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc100cf0 msgr2=0x7f0cbc104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.492+0000 7f0cc280e640 1 --2- 172.21.15.3:0/272436186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc100cf0 0x7f0cbc104d30 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009930 tx=0x7f0cb002bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 -- 172.21.15.3:0/272436186 shutdown_connections 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 --2- 172.21.15.3:0/272436186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cbc105410 0x7f0cbc10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 --2- 172.21.15.3:0/272436186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc100cf0 0x7f0cbc104d30 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 --2- 172.21.15.3:0/272436186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cbc100340 0x7f0cbc100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 -- 172.21.15.3:0/272436186 >> 172.21.15.3:0/272436186 conn(0x7f0cbc0772e0 msgr2=0x7f0cbc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 -- 172.21.15.3:0/272436186 shutdown_connections 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 -- 172.21.15.3:0/272436186 wait complete. 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 Processor -- start 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 -- start start 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cbc100340 0x7f0cbc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.493+0000 7f0cc280e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cbc105410 0x7f0cbc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc280e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 0x7f0cbc1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc280e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1a3e20 con 0x7f0cbc1991a0 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc280e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1a3f90 con 0x7f0cbc100340 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc280e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc1a4150 con 0x7f0cbc105410 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 0x7f0cbc1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 0x7f0cbc1a3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42214/0 (socket says 172.21.15.3:42214) 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 -- 172.21.15.3:0/1168365926 learned_addr learned my addr 172.21.15.3:0/1168365926 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cbc105410 0x7f0cbc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 -- 172.21.15.3:0/1168365926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cbc105410 msgr2=0x7f0cbc198c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cbc105410 0x7f0cbc198c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cbc100340 0x7f0cbc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 -- 172.21.15.3:0/1168365926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cbc100340 msgr2=0x7f0cbc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cbc100340 0x7f0cbc1986d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 -- 172.21.15.3:0/1168365926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0009560 con 0x7f0cbc1991a0 2024-02-09T22:57:36.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc0dab640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 0x7f0cbc1a3750 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400ea60 tx=0x7f0cb400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cb97fa640 1 -- 172.21.15.3:0/1168365926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400ce40 con 0x7f0cbc1991a0 2024-02-09T22:57:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cb97fa640 1 -- 172.21.15.3:0/1168365926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb4005020 con 0x7f0cbc1991a0 2024-02-09T22:57:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc1a4430 con 0x7f0cbc1991a0 2024-02-09T22:57:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cb97fa640 1 -- 172.21.15.3:0/1168365926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4010690 con 0x7f0cbc1991a0 2024-02-09T22:57:36.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.494+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc1a4910 con 0x7f0cbc1991a0 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cb97fa640 1 -- 172.21.15.3:0/1168365926 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0cb4005500 con 0x7f0cbc1991a0 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 Processor -- start 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 -- start start 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b45e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b4e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b513b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918b01020 con 0x559918b45a30 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918ab4320 con 0x559918b4ef60 2024-02-09T22:57:36.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918ab1850 con 0x559918b463e0 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b45e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b45e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42222/0 (socket says 172.21.15.3:42222) 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/1497459822 learned_addr learned my addr 172.21.15.3:0/1497459822 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b4e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cb97fa640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ca40a2230 0x7f0ca40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/1497459822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 msgr2=0x559918b4e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b4e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/1497459822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 msgr2=0x559918b513b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b513b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/1497459822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559918a6f210 con 0x559918b45a30 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.497+0000 7f0cc0dab640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ca40a2230 0x7f0ca40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0cb97fa640 1 -- 172.21.15.3:0/1168365926 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0cb4014070 con 0x7f0cbc1991a0 2024-02-09T22:57:36.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b45e10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009900 tx=0x7f0cb0005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d1334653b85768 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0ca27fc640 1 -- 172.21.15.3:0/1497459822 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb002c7e0 con 0x559918b45a30 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0ca27fc640 1 -- 172.21.15.3:0/1497459822 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb0004510 con 0x559918b45a30 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0ca27fc640 1 -- 172.21.15.3:0/1497459822 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0005220 con 0x559918b45a30 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0cc53e7740 1 -- 172.21.15.3:0/1497459822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 msgr2=0x559918b45e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b45e10 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009900 tx=0x7f0cb0005d90 comp rx=0 tx=0).stop 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.498+0000 7f0cc0dab640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ca40a2230 0x7f0ca40a46f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40cc450 tx=0x7f0cb407b040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- 172.21.15.3:0/1497459822 shutdown_connections 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b513b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b4e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/1497459822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b45e10 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- 172.21.15.3:0/1497459822 >> 172.21.15.3:0/1497459822 conn(0x559918b40100 msgr2=0x559918b404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- 172.21.15.3:0/1497459822 shutdown_connections 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- 172.21.15.3:0/1497459822 wait complete. 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- start start 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 Processor -- start 2024-02-09T22:57:36.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b4bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 0x559918b4c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b4ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918aad030 con 0x559918b463e0 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918780430 con 0x559918b4ef60 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55991873ce00 con 0x559918b45a30 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cc53e7740 10 client.0 _ll_register_callbacks cb 0x559918aa2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cbbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 0x559918b4c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.499+0000 7f0cbbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 0x559918b4c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42234/0 (socket says 172.21.15.3:42234) 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 -- 172.21.15.3:0/375780193 learned_addr learned my addr 172.21.15.3:0/375780193 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b4bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 -- 172.21.15.3:0/375780193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 msgr2=0x559918b4bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b4bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 -- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 msgr2=0x559918b4ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b4ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 -- 172.21.15.3:0/375780193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc10d7f0 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 0x559918b4c4c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009d80 tx=0x7f0cac0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1c3eb34a401a4cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0c9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/375780193 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cac012860 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/375780193 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0cac0129c0 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5599187ac7e0 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/375780193 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cac009e70 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cc53e7740 10 client.17856 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x559918aab6b0 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.500+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/375780193 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5599187ac7e0 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/375780193 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0cac00c070 con 0x559918b463e0 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cc53e7740 10 client.17856 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cc53e7740 10 client.17856 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cc53e7740 20 client.17856 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918b63ad0 0x559918b65ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cc53e7740 4 client.17856 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5599187e9720 con 0x559918b63ad0 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918b63ad0 0x559918b65ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0c9b7fe640 10 client.17856 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.501+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918b63ad0 0x559918b65ef0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47836bacec4183 server_cookie=9f225cd21570fb20 in_seq=0 out_seq=0 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/375780193 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5599187e9720 con 0x559918b63ad0 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0c9b7fe640 10 client.17856 handle_command_reply: tid=0 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 client.17856 shutdown 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 msgr2=0x559918b4c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 0x559918b4c4c0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f0cac009d80 tx=0x7f0cac0117b0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 shutdown_connections 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918b63ad0 0x559918b65ef0 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b4ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b463e0 0x559918b4c4c0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/375780193 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b4bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0c9b7fe640 0 client.17856 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 >> 172.21.15.3:0/375780193 conn(0x559918b40100 msgr2=0x559918b41e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 shutdown_connections 2024-02-09T22:57:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 1 -- 172.21.15.3:0/375780193 wait complete. 2024-02-09T22:57:36.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.502+0000 7f0cc53e7740 20 client.17856 trim_cache size 0 max 16384 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 Processor -- start 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 -- start start 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b4e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b467c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b4c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918a32130 con 0x559918b45a30 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918b5f250 con 0x559918b4ef60 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918b6b100 con 0x559918b463e0 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b4e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b4e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42246/0 (socket says 172.21.15.3:42246) 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.514+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/281096253 learned_addr learned my addr 172.21.15.3:0/281096253 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/281096253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 msgr2=0x559918b467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/281096253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 msgr2=0x559918b4c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b4c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/281096253 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559918a47c90 con 0x559918b45a30 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b4e6b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009930 tx=0x7f0cb0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82604cef77371d02 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0ca1ffb640 1 -- 172.21.15.3:0/281096253 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0052040 con 0x559918b45a30 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0ca1ffb640 1 -- 172.21.15.3:0/281096253 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb0054b70 con 0x559918b45a30 2024-02-09T22:57:36.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.515+0000 7f0ca1ffb640 1 -- 172.21.15.3:0/281096253 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb00055d0 con 0x559918b45a30 2024-02-09T22:57:36.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- 172.21.15.3:0/281096253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 msgr2=0x559918b4e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b4e6b0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0009930 tx=0x7f0cb0005fd0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- 172.21.15.3:0/281096253 shutdown_connections 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b4ef60 0x559918b4c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b463e0 0x559918b467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/281096253 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b45a30 0x559918b4e6b0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- 172.21.15.3:0/281096253 >> 172.21.15.3:0/281096253 conn(0x559918b40100 msgr2=0x559918b404e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- 172.21.15.3:0/281096253 shutdown_connections 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- 172.21.15.3:0/281096253 wait complete. 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- start start 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 Processor -- start 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b59fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b463e0 0x559918b5a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 0x559918b58460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918b58aa0 con 0x559918b4ef60 2024-02-09T22:57:36.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918b58c10 con 0x559918b463e0 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559918b58dd0 con 0x559918b45a30 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cc53e7740 10 client.0 _ll_register_callbacks cb 0x7f0cb0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 0x559918b58460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.516+0000 7f0cbb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 0x559918b58460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42254/0 (socket says 172.21.15.3:42254) 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/3230294363 learned_addr learned my addr 172.21.15.3:0/3230294363 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b59fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b463e0 0x559918b5a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/3230294363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 msgr2=0x559918b59fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b59fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b463e0 msgr2=0x559918b5a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b463e0 0x559918b5a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 -- 172.21.15.3:0/3230294363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559918a6f210 con 0x559918b4ef60 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b463e0 0x559918b5a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cbb7fe640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 0x559918b58460 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0004fd0 tx=0x7f0cb0005290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b011805dd9d67ae server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:36.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0c9b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/3230294363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb0054de0 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/3230294363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0cb0005580 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0cb0004510 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/3230294363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cb002c860 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cc53e7740 10 client.17862 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x559918a6da70 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.517+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/3230294363 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cb0004510 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/3230294363 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0cb003c070 con 0x559918b4ef60 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 10 client.17862 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 10 client.17862 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 20 client.17862 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918bbad00 0x559918bbd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 4 client.17862 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559918838890 con 0x559918bbad00 2024-02-09T22:57:36.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918bbad00 0x559918bbd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0c9b7fe640 10 client.17862 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cbbfff640 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918bbad00 0x559918bbd120 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae4139253f1b33a0 server_cookie=cfb8990581ad71a6 in_seq=0 out_seq=0 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0c9b7fe640 1 -- 172.21.15.3:0/3230294363 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559918838890 con 0x559918bbad00 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0c9b7fe640 10 client.17862 handle_command_reply: tid=0 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.518+0000 7f0cc53e7740 1 client.17862 shutdown 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 msgr2=0x559918b58460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 0x559918b58460 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f0cb0004fd0 tx=0x7f0cb0005290 comp rx=0 tx=0).stop 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 shutdown_connections 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559918bbad00 0x559918bbd120 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559918b463e0 0x559918b5a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0c9b7fe640 0 client.17862 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559918b4ef60 0x559918b58460 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 --2- 172.21.15.3:0/3230294363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559918b45a30 0x559918b59fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 >> 172.21.15.3:0/3230294363 conn(0x559918b40100 msgr2=0x559918b467c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 shutdown_connections 2024-02-09T22:57:36.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 1 -- 172.21.15.3:0/3230294363 wait complete. 2024-02-09T22:57:36.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc53e7740 20 client.17862 trim_cache size 0 max 16384 2024-02-09T22:57:36.529 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ca40a2230 msgr2=0x7f0ca40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ca40a2230 0x7f0ca40a46f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40cc450 tx=0x7f0cb407b040 comp rx=0 tx=0).stop 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 msgr2=0x7f0cbc1a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 0x7f0cbc1a3750 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400ea60 tx=0x7f0cb400ef20 comp rx=0 tx=0).stop 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 shutdown_connections 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ca40a2230 0x7f0ca40a46f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cbc1991a0 0x7f0cbc1a3750 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cbc105410 0x7f0cbc198c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 --2- 172.21.15.3:0/1168365926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cbc100340 0x7f0cbc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 >> 172.21.15.3:0/1168365926 conn(0x7f0cbc0772e0 msgr2=0x7f0cbc0fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 shutdown_connections 2024-02-09T22:57:36.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:36.519+0000 7f0cc280e640 1 -- 172.21.15.3:0/1168365926 wait complete. 2024-02-09T22:57:36.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181703 2.64 MB/sec execute 210 sec latency 3550.576 ms 2024-02-09T22:57:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:36 smithi064 ceph-mon[36638]: pgmap v193: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 960 KiB/s wr, 103 op/s 2024-02-09T22:57:36.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:36 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1797867115' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:57:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:36 smithi089 ceph-mon[36665]: pgmap v193: 97 pgs: 97 active+clean; 74 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 960 KiB/s wr, 103 op/s 2024-02-09T22:57:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:36 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1797867115' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:57:37.534 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:57:37.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:57:37.534 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:57:37.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 181703 2.63 MB/sec execute 211 sec latency 4550.607 ms 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 -- 172.21.15.3:0/2034047156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570100480 msgr2=0x7f357010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2034047156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570100480 0x7f357010e080 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f3568009930 tx=0x7f356802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 -- 172.21.15.3:0/2034047156 shutdown_connections 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2034047156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35700ffac0 0x7f35700fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2034047156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570100480 0x7f357010e080 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2034047156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3570107400 0x7f35701077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 -- 172.21.15.3:0/2034047156 >> 172.21.15.3:0/2034047156 conn(0x7f3570075350 msgr2=0x7f3570075760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.649+0000 7f3575c9d640 1 -- 172.21.15.3:0/2034047156 shutdown_connections 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 -- 172.21.15.3:0/2034047156 wait complete. 2024-02-09T22:57:37.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 Processor -- start 2024-02-09T22:57:37.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 -- start start 2024-02-09T22:57:37.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35700ffac0 0x7f357019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3570100480 0x7f357019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 0x7f3570197780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570197f10 con 0x7f3570107400 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570198080 con 0x7f3570100480 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.650+0000 7f3575c9d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3570198240 con 0x7f35700ffac0 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 0x7f3570197780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 0x7f3570197780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42274/0 (socket says 172.21.15.3:42274) 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 -- 172.21.15.3:0/2867301229 learned_addr learned my addr 172.21.15.3:0/2867301229 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356f7fe640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35700ffac0 0x7f357019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356effd640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3570100480 0x7f357019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 -- 172.21.15.3:0/2867301229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35700ffac0 msgr2=0x7f357019d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35700ffac0 0x7f357019d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 -- 172.21.15.3:0/2867301229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3570100480 msgr2=0x7f357019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3570100480 0x7f357019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 -- 172.21.15.3:0/2867301229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3568009560 con 0x7f3570107400 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356effd640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3570100480 0x7f357019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356ffff640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 0x7f3570197780 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3568005940 tx=0x7f356802c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356cff9640 1 -- 172.21.15.3:0/2867301229 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680043d0 con 0x7f3570107400 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356cff9640 1 -- 172.21.15.3:0/2867301229 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35680052d0 con 0x7f3570107400 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35701984c0 con 0x7f3570107400 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f356cff9640 1 -- 172.21.15.3:0/2867301229 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568003d30 con 0x7f3570107400 2024-02-09T22:57:37.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.651+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35701a45d0 con 0x7f3570107400 2024-02-09T22:57:37.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356cff9640 1 -- 172.21.15.3:0/2867301229 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3568005440 con 0x7f3570107400 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 Processor -- start 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 -- start start 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356cff9640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35580a2230 0x7f35580a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf66220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf66760 0x56178cf73560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf75e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178ced4000 con 0x56178cf73aa0 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178ced1530 con 0x56178cf65e40 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f357887a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178ceccd10 con 0x56178cf66760 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf75e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356effd640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35580a2230 0x7f35580a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf75e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42278/0 (socket says 172.21.15.3:42278) 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356ffff640 1 -- 172.21.15.3:0/1214748975 learned_addr learned my addr 172.21.15.3:0/1214748975 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356cff9640 1 -- 172.21.15.3:0/2867301229 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f35680f0880 con 0x7f3570107400 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.654+0000 7f356f7fe640 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf66220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356effd640 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf66760 0x56178cf73560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356ffff640 1 -- 172.21.15.3:0/1214748975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf66760 msgr2=0x56178cf73560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356ffff640 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf66760 0x56178cf73560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356ffff640 1 -- 172.21.15.3:0/1214748975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 msgr2=0x56178cf66220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356ffff640 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf66220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356ffff640 1 -- 172.21.15.3:0/1214748975 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56178ce73210 con 0x56178cf73aa0 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356ffff640 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf75e90 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f35680b9810 tx=0x7f356803bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529716c3b053fa53 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f356effd640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35580a2230 0x7f35580a46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3564005d00 tx=0x7f3564009610 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f3555ffb640 1 -- 172.21.15.3:0/1214748975 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568082ea0 con 0x56178cf73aa0 2024-02-09T22:57:37.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f3555ffb640 1 -- 172.21.15.3:0/1214748975 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35680bd180 con 0x56178cf73aa0 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.655+0000 7f3555ffb640 1 -- 172.21.15.3:0/1214748975 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680bd3b0 con 0x56178cf73aa0 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 -- 172.21.15.3:0/1214748975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 msgr2=0x56178cf75e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf75e90 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f35680b9810 tx=0x7f356803bbb0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 -- 172.21.15.3:0/1214748975 shutdown_connections 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf75e90 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf66760 0x56178cf73560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 --2- 172.21.15.3:0/1214748975 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf66220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 -- 172.21.15.3:0/1214748975 >> 172.21.15.3:0/1214748975 conn(0x56178cf608c0 msgr2=0x56178cf60ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 -- 172.21.15.3:0/1214748975 shutdown_connections 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 -- 172.21.15.3:0/1214748975 wait complete. 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.656+0000 7f357887a740 1 -- start start 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 Processor -- start 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf70970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf70eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178ce71e60 con 0x56178cf73aa0 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cba04b0 con 0x56178cf66760 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178ce52670 con 0x56178cf65e40 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f357887a740 10 client.0 _ll_register_callbacks cb 0x56178ce6a190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf713f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42286/0 (socket says 172.21.15.3:42286) 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 -- 172.21.15.3:0/134189495 learned_addr learned my addr 172.21.15.3:0/134189495 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356f7fe640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf70970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356effd640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf70eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 -- 172.21.15.3:0/134189495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 msgr2=0x56178cf70970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf70970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 -- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 msgr2=0x56178cf70eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf70eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 -- 172.21.15.3:0/134189495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35701a45d0 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f356ffff640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf713f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f35680bd340 tx=0x7f35680b9c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e424727f99241382 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f354effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f354effd640 1 -- 172.21.15.3:0/134189495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568082ea0 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.657+0000 7f354effd640 1 -- 172.21.15.3:0/134189495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f35680035f0 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56178cbcc860 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f354effd640 1 -- 172.21.15.3:0/134189495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35680bcef0 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 10 client.17874 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56178cecb390 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f354effd640 1 -- 172.21.15.3:0/134189495 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56178cbcc860 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f354effd640 1 -- 172.21.15.3:0/134189495 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f356803c070 con 0x56178cf73aa0 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 10 client.17874 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 10 client.17874 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:37.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 20 client.17874 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cf83550 0x56178cf85970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 4 client.17874 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56178cc1d410 con 0x56178cf83550 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.658+0000 7f356f7fe640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cf83550 0x56178cf85970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.659+0000 7f354effd640 10 client.17874 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.659+0000 7f356f7fe640 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cf83550 0x56178cf85970 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc7aded7db8fc0f6 server_cookie=4dc2cd182943e74c in_seq=0 out_seq=0 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.659+0000 7f354effd640 1 -- 172.21.15.3:0/134189495 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56178cc1d410 con 0x56178cf83550 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.659+0000 7f354effd640 10 client.17874 handle_command_reply: tid=0 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.659+0000 7f357887a740 1 client.17874 shutdown 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 msgr2=0x56178cf713f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf713f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f35680bd340 tx=0x7f35680b9c50 comp rx=0 tx=0).stop 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 shutdown_connections 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cf83550 0x56178cf85970 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf70eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf713f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f354effd640 0 client.17874 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 --2- 172.21.15.3:0/134189495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf70970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 >> 172.21.15.3:0/134189495 conn(0x56178cf608c0 msgr2=0x56178cf62610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 shutdown_connections 2024-02-09T22:57:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 1 -- 172.21.15.3:0/134189495 wait complete. 2024-02-09T22:57:37.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.660+0000 7f357887a740 20 client.17874 trim_cache size 0 max 16384 2024-02-09T22:57:37.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 Processor -- start 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 -- start start 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf730a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf71210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf5f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cbcc860 con 0x56178cf73aa0 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cf7f6b0 con 0x56178cf66760 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f357887a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cf5c0a0 con 0x56178cf65e40 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf5f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f356ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf5f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42292/0 (socket says 172.21.15.3:42292) 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.675+0000 7f356ffff640 1 -- 172.21.15.3:0/2348172786 learned_addr learned my addr 172.21.15.3:0/2348172786 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356f7fe640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf730a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356effd640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf71210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356ffff640 1 -- 172.21.15.3:0/2348172786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 msgr2=0x56178cf730a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356ffff640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf730a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356ffff640 1 -- 172.21.15.3:0/2348172786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 msgr2=0x56178cf71210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356ffff640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf71210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356ffff640 1 -- 172.21.15.3:0/2348172786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56178ce67c70 con 0x56178cf73aa0 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356f7fe640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356effd640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf71210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f356ffff640 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf5f7a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f35680b9810 tx=0x7f356802bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b310ea97eaf25cce server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f35557fa640 1 -- 172.21.15.3:0/2348172786 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3568082ea0 con 0x56178cf73aa0 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f35557fa640 1 -- 172.21.15.3:0/2348172786 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35680bcef0 con 0x56178cf73aa0 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.676+0000 7f35557fa640 1 -- 172.21.15.3:0/2348172786 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35680c1240 con 0x56178cf73aa0 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- 172.21.15.3:0/2348172786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 msgr2=0x56178cf5f7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf5f7a0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f35680b9810 tx=0x7f356802bcd0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- 172.21.15.3:0/2348172786 shutdown_connections 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf66760 0x56178cf71210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf73aa0 0x56178cf5f7a0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- 172.21.15.3:0/2348172786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf65e40 0x56178cf730a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- 172.21.15.3:0/2348172786 >> 172.21.15.3:0/2348172786 conn(0x56178cf608c0 msgr2=0x56178cf5f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- 172.21.15.3:0/2348172786 shutdown_connections 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- 172.21.15.3:0/2348172786 wait complete. 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- start start 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 Processor -- start 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf792a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 0x56178cf797e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf73aa0 0x56178cf79d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.677+0000 7f357887a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cfd9a40 con 0x56178cf66760 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f357887a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cfd9b80 con 0x56178cf65e40 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f357887a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56178cfd9d40 con 0x56178cf73aa0 2024-02-09T22:57:37.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f357887a740 10 client.0 _ll_register_callbacks cb 0x7f3560007560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 0x56178cf797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 0x56178cf797e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42294/0 (socket says 172.21.15.3:42294) 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 -- 172.21.15.3:0/3486865302 learned_addr learned my addr 172.21.15.3:0/3486865302 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356ffff640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf73aa0 0x56178cf79d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356f7fe640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf792a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 -- 172.21.15.3:0/3486865302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf73aa0 msgr2=0x56178cf79d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf73aa0 0x56178cf79d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 -- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 msgr2=0x56178cf792a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf792a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 -- 172.21.15.3:0/3486865302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35680d8020 con 0x56178cf66760 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f356effd640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 0x56178cf797e0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x56178cf701d0 tx=0x7f35640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=649642fc77b841e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f354effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f354effd640 1 -- 172.21.15.3:0/3486865302 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3564005220 con 0x56178cf66760 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f354effd640 1 -- 172.21.15.3:0/3486865302 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3564005380 con 0x56178cf66760 2024-02-09T22:57:37.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35680bd180 con 0x56178cf66760 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f354effd640 1 -- 172.21.15.3:0/3486865302 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3564005540 con 0x56178cf66760 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f357887a740 10 client.17880 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.678+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56178ce8dac0 con 0x56178cf66760 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f354effd640 1 -- 172.21.15.3:0/3486865302 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35680bd180 con 0x56178cf66760 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f354effd640 1 -- 172.21.15.3:0/3486865302 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3564027070 con 0x56178cf66760 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f357887a740 10 client.17880 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f357887a740 10 client.17880 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f357887a740 20 client.17880 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f357887a740 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cfdb5f0 0x56178cfdda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f357887a740 4 client.17880 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56178cc097b0 con 0x56178cfdb5f0 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f356f7fe640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cfdb5f0 0x56178cfdda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f356f7fe640 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cfdb5f0 0x56178cfdda10 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef9155ca1eecf243 server_cookie=d0f16e317130959f in_seq=0 out_seq=0 2024-02-09T22:57:37.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.679+0000 7f354effd640 10 client.17880 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f354effd640 1 -- 172.21.15.3:0/3486865302 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56178cc097b0 con 0x56178cfdb5f0 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f354effd640 10 client.17880 handle_command_reply: tid=0 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 client.17880 shutdown 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 msgr2=0x56178cf797e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 0x56178cf797e0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x56178cf701d0 tx=0x7f35640046c0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 shutdown_connections 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56178cfdb5f0 0x56178cfdda10 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56178cf73aa0 0x56178cf79d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56178cf66760 0x56178cf797e0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 --2- 172.21.15.3:0/3486865302 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56178cf65e40 0x56178cf792a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 >> 172.21.15.3:0/3486865302 conn(0x56178cf608c0 msgr2=0x56178cf5f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f354effd640 0 client.17880 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 shutdown_connections 2024-02-09T22:57:37.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.681+0000 7f357887a740 1 -- 172.21.15.3:0/3486865302 wait complete. 2024-02-09T22:57:37.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f357887a740 20 client.17880 trim_cache size 0 max 16384 2024-02-09T22:57:37.691 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:57:37.691 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:57:37.691 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "255132d2-b1a4-4fc3-b346-329cfc3ca039", 2024-02-09T22:57:37.691 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:57:37.691 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35580a2230 msgr2=0x7f35580a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35580a2230 0x7f35580a46f0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3564005d00 tx=0x7f3564009610 comp rx=0 tx=0).stop 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 msgr2=0x7f3570197780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 0x7f3570197780 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f3568005940 tx=0x7f356802c7c0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 shutdown_connections 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35580a2230 0x7f35580a46f0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3570100480 0x7f357019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3570107400 0x7f3570197780 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 --2- 172.21.15.3:0/2867301229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35700ffac0 0x7f357019d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 >> 172.21.15.3:0/2867301229 conn(0x7f3570075350 msgr2=0x7f35700fe510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 shutdown_connections 2024-02-09T22:57:37.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.682+0000 7f3575c9d640 1 -- 172.21.15.3:0/2867301229 wait complete. 2024-02-09T22:57:37.698 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:57:37.699 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:57:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 -- 172.21.15.3:0/3065780344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 msgr2=0x7f1150069d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 --2- 172.21.15.3:0/3065780344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 0x7f1150069d90 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f114c009860 tx=0x7f114c02bc00 comp rx=0 tx=0).stop 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 -- 172.21.15.3:0/3065780344 shutdown_connections 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 --2- 172.21.15.3:0/3065780344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115006a2d0 0x7f115010d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 --2- 172.21.15.3:0/3065780344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115010dbd0 0x7f115010ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 --2- 172.21.15.3:0/3065780344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 0x7f1150069d90 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 -- 172.21.15.3:0/3065780344 >> 172.21.15.3:0/3065780344 conn(0x7f11500771d0 msgr2=0x7f11500775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 -- 172.21.15.3:0/3065780344 shutdown_connections 2024-02-09T22:57:37.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.806+0000 7f1158725640 1 -- 172.21.15.3:0/3065780344 wait complete. 2024-02-09T22:57:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 Processor -- start 2024-02-09T22:57:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 -- start start 2024-02-09T22:57:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 0x7f115019f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 0x7f115019fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115010dbd0 0x7f11501997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150199e80 con 0x7f115006a2d0 2024-02-09T22:57:37.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1150199ff0 con 0x7f115010dbd0 2024-02-09T22:57:37.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1158725640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115019a1b0 con 0x7f11500699b0 2024-02-09T22:57:37.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 0x7f115019fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 0x7f115019fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42318/0 (socket says 172.21.15.3:42318) 2024-02-09T22:57:37.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 -- 172.21.15.3:0/2878710068 learned_addr learned my addr 172.21.15.3:0/2878710068 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f11564c1640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 0x7f115019f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1156cc2640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115010dbd0 0x7f11501997e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 -- 172.21.15.3:0/2878710068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 msgr2=0x7f115019f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 0x7f115019f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 -- 172.21.15.3:0/2878710068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115010dbd0 msgr2=0x7f11501997e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115010dbd0 0x7f11501997e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.807+0000 7f1155cc0640 1 -- 172.21.15.3:0/2878710068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1148009900 con 0x7f115006a2d0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.808+0000 7f1155cc0640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 0x7f115019fbb0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f114000e9c0 tx=0x7f114000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.808+0000 7f11477fe640 1 -- 172.21.15.3:0/2878710068 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11400096e0 con 0x7f115006a2d0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.808+0000 7f11477fe640 1 -- 172.21.15.3:0/2878710068 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f114000cd80 con 0x7f115006a2d0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.808+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c009490 con 0x7f115006a2d0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.808+0000 7f11477fe640 1 -- 172.21.15.3:0/2878710068 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140010430 con 0x7f115006a2d0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.808+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11501a64c0 con 0x7f115006a2d0 2024-02-09T22:57:37.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.810+0000 7f11477fe640 1 -- 172.21.15.3:0/2878710068 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1140003680 con 0x7f115006a2d0 2024-02-09T22:57:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.810+0000 7f115b3dc740 1 Processor -- start 2024-02-09T22:57:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 -- start start 2024-02-09T22:57:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x558932110d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x5589321198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11477fe640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f113c0a22b0 0x7f113c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932119eb0 0x55893211c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589320cbd00 con 0x558932119eb0 2024-02-09T22:57:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5589320802f0 con 0x558932110980 2024-02-09T22:57:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f115b3dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893207d820 con 0x558932111330 2024-02-09T22:57:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f1156cc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932119eb0 0x55893211c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x5589321198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x5589321198e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54964/0 (socket says 172.21.15.3:54964) 2024-02-09T22:57:37.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 -- 172.21.15.3:0/3809771183 learned_addr learned my addr 172.21.15.3:0/3809771183 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f1156cc2640 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x558932110d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f1155cc0640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f113c0a22b0 0x7f113c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11477fe640 1 -- 172.21.15.3:0/2878710068 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1140014070 con 0x7f115006a2d0 2024-02-09T22:57:37.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 -- 172.21.15.3:0/3809771183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 msgr2=0x558932110d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x558932110d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 -- 172.21.15.3:0/3809771183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932119eb0 msgr2=0x55893211c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932119eb0 0x55893211c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f11564c1640 1 -- 172.21.15.3:0/3809771183 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55893203aff0 con 0x558932111330 2024-02-09T22:57:37.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.811+0000 7f1156cc2640 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x558932110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:37.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f11564c1640 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x5589321198e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f114c005e00 tx=0x7f114c0040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e10d3e20d4a24fe0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f1155cc0640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f113c0a22b0 0x7f113c0a4770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1140097240 tx=0x7f114006de50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f1144ff9640 1 -- 172.21.15.3:0/3809771183 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c004370 con 0x558932111330 2024-02-09T22:57:37.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f1144ff9640 1 -- 172.21.15.3:0/3809771183 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f114c0044d0 con 0x558932111330 2024-02-09T22:57:37.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f1144ff9640 1 -- 172.21.15.3:0/3809771183 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c003850 con 0x558932111330 2024-02-09T22:57:37.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f115b3dc740 1 -- 172.21.15.3:0/3809771183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 msgr2=0x5589321198e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.812+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x5589321198e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f114c005e00 tx=0x7f114c0040c0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- 172.21.15.3:0/3809771183 shutdown_connections 2024-02-09T22:57:37.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932119eb0 0x55893211c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x5589321198e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3809771183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x558932110d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- 172.21.15.3:0/3809771183 >> 172.21.15.3:0/3809771183 conn(0x55893210b050 msgr2=0x55893210b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- 172.21.15.3:0/3809771183 shutdown_connections 2024-02-09T22:57:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- 172.21.15.3:0/3809771183 wait complete. 2024-02-09T22:57:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- start start 2024-02-09T22:57:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 Processor -- start 2024-02-09T22:57:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x558932116cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x558932117210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.838 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932117750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558932079000 con 0x558932110980 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558931d4c430 con 0x558932119eb0 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558931d08e00 con 0x558932111330 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f115b3dc740 10 client.0 _ll_register_callbacks cb 0x55893206d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f1156cc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x558932116cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f1156cc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x558932116cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42334/0 (socket says 172.21.15.3:42334) 2024-02-09T22:57:37.839 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f11564c1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x558932117210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f1156cc2640 1 -- 172.21.15.3:0/3457515530 learned_addr learned my addr 172.21.15.3:0/3457515530 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.813+0000 7f1156cc2640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932117750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1156cc2640 1 -- 172.21.15.3:0/3457515530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 msgr2=0x558932117210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1156cc2640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x558932117210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1156cc2640 1 -- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 msgr2=0x558932117750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.840 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1156cc2640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932117750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1156cc2640 1 -- 172.21.15.3:0/3457515530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1150110b20 con 0x558932110980 2024-02-09T22:57:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f11564c1640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x558932117210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1156cc2640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x558932116cd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f114800ef50 tx=0x7f1148004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3930ad8170dc29e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1139ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1139ffb640 1 -- 172.21.15.3:0/3457515530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1148013040 con 0x558932110980 2024-02-09T22:57:37.841 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1139ffb640 1 -- 172.21.15.3:0/3457515530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f114801e670 con 0x558932110980 2024-02-09T22:57:37.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558931d787e0 con 0x558932110980 2024-02-09T22:57:37.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1139ffb640 1 -- 172.21.15.3:0/3457515530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1148004b20 con 0x558932110980 2024-02-09T22:57:37.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f115b3dc740 10 client.17892 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:37.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x558932077680 con 0x558932110980 2024-02-09T22:57:37.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.814+0000 7f1139ffb640 1 -- 172.21.15.3:0/3457515530 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558931d787e0 con 0x558932110980 2024-02-09T22:57:37.842 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f1139ffb640 1 -- 172.21.15.3:0/3457515530 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f11480051b0 con 0x558932110980 2024-02-09T22:57:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f115b3dc740 10 client.17892 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f115b3dc740 10 client.17892 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f115b3dc740 20 client.17892 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893212a390 0x55893212c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f115b3dc740 4 client.17892 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:37.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558931db5720 con 0x55893212a390 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f11564c1640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893212a390 0x55893212c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f11564c1640 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893212a390 0x55893212c7b0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b71898b403430908 server_cookie=2989702dd622638e in_seq=0 out_seq=0 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.815+0000 7f1139ffb640 10 client.17892 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f1139ffb640 1 -- 172.21.15.3:0/3457515530 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558931db5720 con 0x55893212a390 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f1139ffb640 10 client.17892 handle_command_reply: tid=0 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 client.17892 shutdown 2024-02-09T22:57:37.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 msgr2=0x558932116cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x558932116cd0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f114800ef50 tx=0x7f1148004ab0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 shutdown_connections 2024-02-09T22:57:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55893212a390 0x55893212c7b0 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932117750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x558932117210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 --2- 172.21.15.3:0/3457515530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x558932116cd0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f1139ffb640 0 client.17892 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 >> 172.21.15.3:0/3457515530 conn(0x55893210b050 msgr2=0x55893210cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 shutdown_connections 2024-02-09T22:57:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 1 -- 172.21.15.3:0/3457515530 wait complete. 2024-02-09T22:57:37.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.816+0000 7f115b3dc740 20 client.17892 trim_cache size 0 max 16384 2024-02-09T22:57:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 Processor -- start 2024-02-09T22:57:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 -- start start 2024-02-09T22:57:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x55893210a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 0x558932120eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932119eb0 0x5589321213f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558931ffdfd0 con 0x558932111330 2024-02-09T22:57:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558932118130 con 0x558932110980 2024-02-09T22:57:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f115b3dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55893211c2a0 con 0x558932119eb0 2024-02-09T22:57:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 0x558932120eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 0x558932120eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42346/0 (socket says 172.21.15.3:42346) 2024-02-09T22:57:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 -- 172.21.15.3:0/2936324147 learned_addr learned my addr 172.21.15.3:0/2936324147 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f1156cc2640 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x55893210a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f1156cc2640 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932119eb0 0x5589321213f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 -- 172.21.15.3:0/2936324147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932119eb0 msgr2=0x5589321213f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932119eb0 0x5589321213f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 -- 172.21.15.3:0/2936324147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 msgr2=0x55893210a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x55893210a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.832+0000 7f11564c1640 1 -- 172.21.15.3:0/2936324147 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558932013b30 con 0x558932111330 2024-02-09T22:57:37.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.833+0000 7f11564c1640 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 0x558932120eb0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f114c009860 tx=0x7f114c041040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ab2f4304ab4e59 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.833+0000 7f113bfff640 1 -- 172.21.15.3:0/2936324147 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c051420 con 0x558932111330 2024-02-09T22:57:37.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.833+0000 7f113bfff640 1 -- 172.21.15.3:0/2936324147 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f114c005050 con 0x558932111330 2024-02-09T22:57:37.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.833+0000 7f113bfff640 1 -- 172.21.15.3:0/2936324147 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114c0346a0 con 0x558932111330 2024-02-09T22:57:37.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.833+0000 7f115b3dc740 1 -- 172.21.15.3:0/2936324147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 msgr2=0x558932120eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.833+0000 7f115b3dc740 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 0x558932120eb0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f114c009860 tx=0x7f114c041040 comp rx=0 tx=0).stop 2024-02-09T22:57:37.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- 172.21.15.3:0/2936324147 shutdown_connections 2024-02-09T22:57:37.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932119eb0 0x5589321213f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932111330 0x558932120eb0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 --2- 172.21.15.3:0/2936324147 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932110980 0x55893210a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- 172.21.15.3:0/2936324147 >> 172.21.15.3:0/2936324147 conn(0x55893210b050 msgr2=0x558932109960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- 172.21.15.3:0/2936324147 shutdown_connections 2024-02-09T22:57:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- 172.21.15.3:0/2936324147 wait complete. 2024-02-09T22:57:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- start start 2024-02-09T22:57:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 Processor -- start 2024-02-09T22:57:37.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x55893217a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x55893217a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932174800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558932174ed0 con 0x558932110980 2024-02-09T22:57:37.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558932175040 con 0x558932119eb0 2024-02-09T22:57:37.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558932175200 con 0x558932111330 2024-02-09T22:57:37.866 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f115b3dc740 10 client.0 _ll_register_callbacks cb 0x7f114c037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f1156cc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x55893217a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f1156cc2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x55893217a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42348/0 (socket says 172.21.15.3:42348) 2024-02-09T22:57:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.834+0000 7f1156cc2640 1 -- 172.21.15.3:0/460557102 learned_addr learned my addr 172.21.15.3:0/460557102 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f11564c1640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x55893217a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932174800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.867 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 -- 172.21.15.3:0/460557102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 msgr2=0x55893217a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x55893217a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 -- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 msgr2=0x558932174800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:37.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932174800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 -- 172.21.15.3:0/460557102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55893203aff0 con 0x558932110980 2024-02-09T22:57:37.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1156cc2640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x55893217a100 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f114800ef20 tx=0x7f1148009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73becde59885b3cf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:37.868 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1139ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1139ffb640 1 -- 172.21.15.3:0/460557102 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f114801ecb0 con 0x558932110980 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1139ffb640 1 -- 172.21.15.3:0/460557102 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1148013040 con 0x558932110980 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f114c0044d0 con 0x558932110980 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1139ffb640 1 -- 172.21.15.3:0/460557102 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f114801ee10 con 0x558932110980 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f115b3dc740 10 client.17898 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x558932039850 con 0x558932110980 2024-02-09T22:57:37.869 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.835+0000 7f1139ffb640 1 -- 172.21.15.3:0/460557102 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f114c0044d0 con 0x558932110980 2024-02-09T22:57:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f1139ffb640 1 -- 172.21.15.3:0/460557102 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f11480206b0 con 0x558932110980 2024-02-09T22:57:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f115b3dc740 10 client.17898 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f115b3dc740 10 client.17898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f115b3dc740 20 client.17898 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f115b3dc740 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558932181f50 0x558932184370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:37.870 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f115b3dc740 4 client.17898 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558931e04890 con 0x558932181f50 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f11564c1640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558932181f50 0x558932184370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f11564c1640 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558932181f50 0x558932184370 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d73039f1e19b032c server_cookie=cd7f841ca77a2904 in_seq=0 out_seq=0 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f1139ffb640 10 client.17898 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f1139ffb640 1 -- 172.21.15.3:0/460557102 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x558931e04890 con 0x558932181f50 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.836+0000 7f1139ffb640 10 client.17898 handle_command_reply: tid=0 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 client.17898 shutdown 2024-02-09T22:57:37.871 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 msgr2=0x55893217a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x55893217a100 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f114800ef20 tx=0x7f1148009ea0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 shutdown_connections 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558932181f50 0x558932184370 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558932119eb0 0x558932174800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558932111330 0x55893217a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1139ffb640 0 client.17898 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 --2- 172.21.15.3:0/460557102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558932110980 0x55893217a100 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.872 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 >> 172.21.15.3:0/460557102 conn(0x55893210b050 msgr2=0x55893210a1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 shutdown_connections 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 1 -- 172.21.15.3:0/460557102 wait complete. 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f115b3dc740 20 client.17898 trim_cache size 0 max 16384 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stdout: "255132d2-b1a4-4fc3-b346-329cfc3ca039": { 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:57:37.873 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "255132d2-b1a4-4fc3-b346-329cfc3ca039", 2024-02-09T22:57:37.874 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:57:37.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:57:37.874 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:57:37.874 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f113c0a22b0 msgr2=0x7f113c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f113c0a22b0 0x7f113c0a4770 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1140097240 tx=0x7f114006de50 comp rx=0 tx=0).stop 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 msgr2=0x7f115019fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 0x7f115019fbb0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f114000e9c0 tx=0x7f114000ee80 comp rx=0 tx=0).stop 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 shutdown_connections 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f113c0a22b0 0x7f113c0a4770 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115010dbd0 0x7f11501997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f115006a2d0 0x7f115019fbb0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 --2- 172.21.15.3:0/2878710068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11500699b0 0x7f115019f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.837+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 >> 172.21.15.3:0/2878710068 conn(0x7f11500771d0 msgr2=0x7f115010f120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.838+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 shutdown_connections 2024-02-09T22:57:37.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:57:37.838+0000 7f1158725640 1 -- 172.21.15.3:0/2878710068 wait complete. 2024-02-09T22:57:37.876 INFO:tasks.cephfs.filesystem:scrub status for tag:255132d2-b1a4-4fc3-b346-329cfc3ca039 - {'path': '/', 'tag': '255132d2-b1a4-4fc3-b346-329cfc3ca039', 'options': 'recursive,force'} 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/2754487013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101e10 msgr2=0x7fbbc4105e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/2754487013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101e10 0x7fbbc4105e50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009930 tx=0x7fbbb802bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/2754487013 shutdown_connections 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/2754487013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc4106530 0x7fbbc410e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/2754487013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101e10 0x7fbbc4105e50 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/2754487013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbbc4101460 0x7fbbc4101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/2754487013 >> 172.21.15.3:0/2754487013 conn(0x7fbbc40fccc0 msgr2=0x7fbbc40ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/2754487013 shutdown_connections 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/2754487013 wait complete. 2024-02-09T22:57:38.199 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 Processor -- start 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.193+0000 7fbbc8dfd640 1 -- start start 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 0x7fbbc419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc4106530 0x7fbbc419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbbc4197710 0x7fbbc4197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc41981e0 con 0x7fbbc4101460 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4198350 con 0x7fbbc4197710 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc4198510 con 0x7fbbc4106530 2024-02-09T22:57:38.200 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 0x7fbbc419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 0x7fbbc419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42368/0 (socket says 172.21.15.3:42368) 2024-02-09T22:57:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 -- 172.21.15.3:0/4069604135 learned_addr learned my addr 172.21.15.3:0/4069604135 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc4106530 0x7fbbc419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.204 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbbc4197710 0x7fbbc4197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 -- 172.21.15.3:0/4069604135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc4106530 msgr2=0x7fbbc419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc4106530 0x7fbbc419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 -- 172.21.15.3:0/4069604135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbbc4197710 msgr2=0x7fbbc4197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbbc4197710 0x7fbbc4197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 -- 172.21.15.3:0/4069604135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb8009560 con 0x7fbbc4101460 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc259c640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 0x7fbbc419d600 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400e9f0 tx=0x7fbbb400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbb37fe640 1 -- 172.21.15.3:0/4069604135 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4009700 con 0x7fbbc4101460 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbb37fe640 1 -- 172.21.15.3:0/4069604135 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb400cd80 con 0x7fbbc4101460 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc41a4460 con 0x7fbbc4101460 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbb37fe640 1 -- 172.21.15.3:0/4069604135 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4018920 con 0x7fbbc4101460 2024-02-09T22:57:38.205 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.194+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41a4990 con 0x7fbbc4101460 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.196+0000 7fbbb37fe640 1 -- 172.21.15.3:0/4069604135 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fbbb4018a80 con 0x7fbbc4101460 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 Processor -- start 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbb37fe640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbbac0a22a0 0x7fbbac0a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbbac0a22a0 0x7fbbac0a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbb37fe640 1 -- 172.21.15.3:0/4069604135 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbbb4014070 con 0x7fbbc4101460 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 -- start start 2024-02-09T22:57:38.212 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba2b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba34260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba36b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053b999a00 con 0x56053ba2b0b0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053b996f30 con 0x56053ba2b9d0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.197+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053b992710 con 0x56053ba347a0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba2b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba2b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42372/0 (socket says 172.21.15.3:42372) 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 -- 172.21.15.3:0/2416444168 learned_addr learned my addr 172.21.15.3:0/2416444168 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbbac0a22a0 0x7fbbac0a4760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc005fd0 tx=0x7fbbbc009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba36b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba34260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 -- 172.21.15.3:0/2416444168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 msgr2=0x56053ba36b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.213 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba36b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 -- 172.21.15.3:0/2416444168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 msgr2=0x56053ba34260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba34260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 -- 172.21.15.3:0/2416444168 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56053b9573b0 con 0x56053ba2b0b0 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbc259c640 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba2b490 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009380 tx=0x7fbbb409b530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6af873d80f54f0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbb0ff9640 1 -- 172.21.15.3:0/2416444168 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb409ef80 con 0x56053ba2b0b0 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbb0ff9640 1 -- 172.21.15.3:0/2416444168 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb40a35d0 con 0x56053ba2b0b0 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.198+0000 7fbbb0ff9640 1 -- 172.21.15.3:0/2416444168 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb4097680 con 0x56053ba2b0b0 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/2416444168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 msgr2=0x56053ba2b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba2b490 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009380 tx=0x7fbbb409b530 comp rx=0 tx=0).stop 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/2416444168 shutdown_connections 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba34260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba36b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/2416444168 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba2b490 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/2416444168 >> 172.21.15.3:0/2416444168 conn(0x56053ba259d0 msgr2=0x56053ba25db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/2416444168 shutdown_connections 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/2416444168 wait complete. 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 -- start start 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.199+0000 7fbbcb9cb740 1 Processor -- start 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba31790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba31cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba347a0 0x56053ba32210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053b937b90 con 0x56053ba2b0b0 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053b666250 con 0x56053ba347a0 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053b918200 con 0x56053ba2b9d0 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 10 client.0 _ll_register_callbacks cb 0x56053b94e5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba31790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba31790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42384/0 (socket says 172.21.15.3:42384) 2024-02-09T22:57:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 -- 172.21.15.3:0/3471919188 learned_addr learned my addr 172.21.15.3:0/3471919188 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba31cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba347a0 0x56053ba32210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 -- 172.21.15.3:0/3471919188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 msgr2=0x56053ba31cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba31cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 -- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba347a0 msgr2=0x56053ba32210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba347a0 0x56053ba32210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 -- 172.21.15.3:0/3471919188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc41a4990 con 0x56053ba2b0b0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbc259c640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba31790 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fbbb40ce3e0 tx=0x7fbbb4010db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24657a7372ff2885 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbba9ffb640 1 -- 172.21.15.3:0/3471919188 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40b9170 con 0x56053ba2b0b0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbba9ffb640 1 -- 172.21.15.3:0/3471919188 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbbb40b92d0 con 0x56053ba2b0b0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56053b9b8ee0 con 0x56053ba2b0b0 2024-02-09T22:57:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbba9ffb640 1 -- 172.21.15.3:0/3471919188 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbb40b3b60 con 0x56053ba2b0b0 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 10 client.17910 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.200+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56053b990d90 con 0x56053ba2b0b0 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbba9ffb640 1 -- 172.21.15.3:0/3471919188 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb4089080 con 0x56053ba2b0b0 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbba9ffb640 1 -- 172.21.15.3:0/3471919188 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbbb4085af0 con 0x56053ba2b0b0 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbcb9cb740 10 client.17910 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbcb9cb740 10 client.17910 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbcb9cb740 20 client.17910 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba44600 0x56053ba469f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbcb9cb740 4 client.17910 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56053b692600 con 0x56053ba44600 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba44600 0x56053ba469f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbba9ffb640 10 client.17910 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.201+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba44600 0x56053ba469f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aef0972baee9b5b5 server_cookie=3d16874cafc04538 in_seq=0 out_seq=0 2024-02-09T22:57:38.217 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbba9ffb640 1 -- 172.21.15.3:0/3471919188 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56053b692600 con 0x56053ba44600 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbba9ffb640 10 client.17910 handle_command_reply: tid=0 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 client.17910 shutdown 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 msgr2=0x56053ba31790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba31790 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7fbbb40ce3e0 tx=0x7fbbb4010db0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 shutdown_connections 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba44600 0x56053ba469f0 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba347a0 0x56053ba32210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba31cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbba9ffb640 0 client.17910 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/3471919188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba31790 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 >> 172.21.15.3:0/3471919188 conn(0x56053ba259d0 msgr2=0x56053ba27720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 shutdown_connections 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/3471919188 wait complete. 2024-02-09T22:57:38.218 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.202+0000 7fbbcb9cb740 20 client.17910 trim_cache size 0 max 16384 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 Processor -- start 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 -- start start 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba33ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba31d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba32240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053ba25500 con 0x56053ba2b0b0 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053ba4bc20 con 0x56053ba2b9d0 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.215+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053ba2fd00 con 0x56053ba347a0 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba33ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba33ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42388/0 (socket says 172.21.15.3:42388) 2024-02-09T22:57:38.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 -- 172.21.15.3:0/847441898 learned_addr learned my addr 172.21.15.3:0/847441898 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba32240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba31d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 -- 172.21.15.3:0/847441898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 msgr2=0x56053ba32240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba32240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 -- 172.21.15.3:0/847441898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 msgr2=0x56053ba31d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba31d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 -- 172.21.15.3:0/847441898 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56053b92d820 con 0x56053ba2b0b0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba31d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbc259c640 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba33ec0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fbbb409ab30 tx=0x7fbbb400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e966bc710cedb9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbabfff640 1 -- 172.21.15.3:0/847441898 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb40cc020 con 0x56053ba2b0b0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbabfff640 1 -- 172.21.15.3:0/847441898 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56053b9b8ee0 con 0x56053ba2b0b0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.216+0000 7fbbabfff640 1 -- 172.21.15.3:0/847441898 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb409a280 con 0x56053ba2b0b0 2024-02-09T22:57:38.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/847441898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 msgr2=0x56053ba33ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba33ec0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7fbbb409ab30 tx=0x7fbbb400c840 comp rx=0 tx=0).stop 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/847441898 shutdown_connections 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b9d0 0x56053ba31d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba347a0 0x56053ba32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/847441898 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba2b0b0 0x56053ba33ec0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/847441898 >> 172.21.15.3:0/847441898 conn(0x56053ba259d0 msgr2=0x56053ba2cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/847441898 shutdown_connections 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/847441898 wait complete. 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- start start 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 Processor -- start 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b0b0 0x56053ba3b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba3b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 0x56053ba39820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053ba39ef0 con 0x56053ba347a0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053ba3a060 con 0x56053ba2b0b0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.217+0000 7fbbcb9cb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56053ba3a220 con 0x56053ba2b9d0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbcb9cb740 10 client.0 _ll_register_callbacks cb 0x7fbbbc009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 0x56053ba39820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 0x56053ba39820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42400/0 (socket says 172.21.15.3:42400) 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 -- 172.21.15.3:0/4012894977 learned_addr learned my addr 172.21.15.3:0/4012894977 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba3b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc259c640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b0b0 0x56053ba3b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 -- 172.21.15.3:0/4012894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 msgr2=0x56053ba3b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba3b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 -- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b0b0 msgr2=0x56053ba3b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b0b0 0x56053ba3b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 -- 172.21.15.3:0/4012894977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbbc0114f0 con 0x56053ba347a0 2024-02-09T22:57:38.227 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc1d9b640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba3b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbc2d9d640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 0x56053ba39820 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc011ef0 tx=0x7fbbbc013070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1866dc07b0540cff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbba9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbba9ffb640 1 -- 172.21.15.3:0/4012894977 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc024040 con 0x56053ba347a0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbba9ffb640 1 -- 172.21.15.3:0/4012894977 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fbbbc017070 con 0x56053ba347a0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbb40a35d0 con 0x56053ba347a0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbba9ffb640 1 -- 172.21.15.3:0/4012894977 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbbc01fe80 con 0x56053ba347a0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbcb9cb740 10 client.17916 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56053b990d90 con 0x56053ba347a0 2024-02-09T22:57:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.218+0000 7fbba9ffb640 1 -- 172.21.15.3:0/4012894977 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb40a35d0 con 0x56053ba347a0 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbba9ffb640 1 -- 172.21.15.3:0/4012894977 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fbbbc01a040 con 0x56053ba347a0 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbcb9cb740 10 client.17916 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbcb9cb740 10 client.17916 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbcb9cb740 20 client.17916 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba9be60 0x56053ba9e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbcb9cb740 4 client.17916 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56053b6e31b0 con 0x56053ba9be60 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbc259c640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba9be60 0x56053ba9e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbbc259c640 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba9be60 0x56053ba9e280 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adbfe44c6b4de307 server_cookie=91eba74744ba5264 in_seq=0 out_seq=0 2024-02-09T22:57:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.219+0000 7fbba9ffb640 10 client.17916 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:38.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbba9ffb640 1 -- 172.21.15.3:0/4012894977 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56053b6e31b0 con 0x56053ba9be60 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbba9ffb640 10 client.17916 handle_command_reply: tid=0 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 client.17916 shutdown 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 msgr2=0x56053ba39820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 0x56053ba39820 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc011ef0 tx=0x7fbbbc013070 comp rx=0 tx=0).stop 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 shutdown_connections 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56053ba9be60 0x56053ba9e280 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56053ba347a0 0x56053ba39820 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56053ba2b9d0 0x56053ba3b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbba9ffb640 0 client.17916 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 --2- 172.21.15.3:0/4012894977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56053ba2b0b0 0x56053ba3b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 >> 172.21.15.3:0/4012894977 conn(0x56053ba259d0 msgr2=0x56053ba2ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:38.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 shutdown_connections 2024-02-09T22:57:38.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 1 -- 172.21.15.3:0/4012894977 wait complete. 2024-02-09T22:57:38.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.247+0000 7fbbcb9cb740 20 client.17916 trim_cache size 0 max 16384 2024-02-09T22:57:38.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:57:38.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:57:38.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:57:38.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbbac0a22a0 msgr2=0x7fbbac0a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbbac0a22a0 0x7fbbac0a4760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc005fd0 tx=0x7fbbbc009500 comp rx=0 tx=0).stop 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 msgr2=0x7fbbc419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 0x7fbbc419d600 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fbbb400e9f0 tx=0x7fbbb400eeb0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 shutdown_connections 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbbac0a22a0 0x7fbbac0a4760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbbc4197710 0x7fbbc4197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbbc4106530 0x7fbbc419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 --2- 172.21.15.3:0/4069604135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbbc4101460 0x7fbbc419d600 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 >> 172.21.15.3:0/4069604135 conn(0x7fbbc40fccc0 msgr2=0x7fbbc40fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 shutdown_connections 2024-02-09T22:57:38.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:38.248+0000 7fbbc8dfd640 1 -- 172.21.15.3:0/4069604135 wait complete. 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: pgmap v194: 97 pgs: 97 active+clean; 74 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:38 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 183792 2.67 MB/sec execute 212 sec latency 4620.407 ms 2024-02-09T22:57:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: pgmap v194: 97 pgs: 97 active+clean; 74 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-02-09T22:57:38.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:57:38.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:38 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: pgmap v194: 97 pgs: 97 active+clean; 74 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 1.2 MiB/s wr, 125 op/s 2024-02-09T22:57:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:57:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:38.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:38 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:39.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:39 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:57:39.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:39 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:39.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185122 2.69 MB/sec execute 213 sec latency 140.277 ms 2024-02-09T22:57:39.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:39 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:57:39.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:39 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:39.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:39 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:57:39.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:39 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:57:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:40 smithi003 ceph-mon[32678]: pgmap v195: 97 pgs: 97 active+clean; 74 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 708 KiB/s wr, 63 op/s 2024-02-09T22:57:40.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185122 2.68 MB/sec execute 214 sec latency 1140.372 ms 2024-02-09T22:57:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:40 smithi064 ceph-mon[36638]: pgmap v195: 97 pgs: 97 active+clean; 74 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 708 KiB/s wr, 63 op/s 2024-02-09T22:57:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:40 smithi089 ceph-mon[36665]: pgmap v195: 97 pgs: 97 active+clean; 74 MiB data, 1015 MiB used, 1.0 TiB / 1.0 TiB avail; 275 KiB/s rd, 708 KiB/s wr, 63 op/s 2024-02-09T22:57:41.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185122 2.67 MB/sec execute 215 sec latency 2140.493 ms 2024-02-09T22:57:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:42 smithi003 ceph-mon[32678]: pgmap v196: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-02-09T22:57:42.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185122 2.65 MB/sec execute 216 sec latency 3140.622 ms 2024-02-09T22:57:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:42 smithi064 ceph-mon[36638]: pgmap v196: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-02-09T22:57:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:42 smithi089 ceph-mon[36665]: pgmap v196: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 132 op/s 2024-02-09T22:57:43.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185246 2.64 MB/sec execute 217 sec latency 3547.928 ms 2024-02-09T22:57:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:44 smithi003 ceph-mon[32678]: pgmap v197: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 KiB/s wr, 91 op/s 2024-02-09T22:57:44.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185246 2.63 MB/sec execute 218 sec latency 1506.946 ms 2024-02-09T22:57:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:44 smithi064 ceph-mon[36638]: pgmap v197: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 KiB/s wr, 91 op/s 2024-02-09T22:57:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:44 smithi089 ceph-mon[36665]: pgmap v197: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 KiB/s wr, 91 op/s 2024-02-09T22:57:45.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185246 2.62 MB/sec execute 219 sec latency 2506.988 ms 2024-02-09T22:57:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:46 smithi003 ceph-mon[32678]: pgmap v198: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 KiB/s wr, 91 op/s 2024-02-09T22:57:46.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185246 2.61 MB/sec execute 220 sec latency 3507.049 ms 2024-02-09T22:57:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:46 smithi064 ceph-mon[36638]: pgmap v198: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 KiB/s wr, 91 op/s 2024-02-09T22:57:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:46 smithi089 ceph-mon[36665]: pgmap v198: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 890 KiB/s wr, 91 op/s 2024-02-09T22:57:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:47 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:47 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:47 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:47.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:47 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:47.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 185246 2.60 MB/sec execute 221 sec latency 4507.122 ms 2024-02-09T22:57:47.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:47 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:47.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:47 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:47.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:47 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:47.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:47 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:47.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:47 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:47.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:47 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:47.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:47 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:47.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:47 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:48 smithi003 ceph-mon[32678]: pgmap v199: 97 pgs: 97 active+clean; 74 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 948 KiB/s wr, 109 op/s 2024-02-09T22:57:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:48 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:57:48.644 INFO:tasks.workunit.client.0.smithi003.stdout: 1 186148 2.61 MB/sec execute 222 sec latency 4908.206 ms 2024-02-09T22:57:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:48 smithi064 ceph-mon[36638]: pgmap v199: 97 pgs: 97 active+clean; 74 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 948 KiB/s wr, 109 op/s 2024-02-09T22:57:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:48 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:57:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:48 smithi089 ceph-mon[36665]: pgmap v199: 97 pgs: 97 active+clean; 74 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 948 KiB/s wr, 109 op/s 2024-02-09T22:57:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:48 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:57:49.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 186148 2.59 MB/sec execute 223 sec latency 1138.923 ms 2024-02-09T22:57:50.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:50 smithi089 ceph-mon[36665]: pgmap v200: 97 pgs: 97 active+clean; 74 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 642 KiB/s wr, 87 op/s 2024-02-09T22:57:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:50 smithi003 ceph-mon[32678]: pgmap v200: 97 pgs: 97 active+clean; 74 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 642 KiB/s wr, 87 op/s 2024-02-09T22:57:50.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 186148 2.58 MB/sec execute 224 sec latency 2139.023 ms 2024-02-09T22:57:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:50 smithi064 ceph-mon[36638]: pgmap v200: 97 pgs: 97 active+clean; 74 MiB data, 1018 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 642 KiB/s wr, 87 op/s 2024-02-09T22:57:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 -- 172.21.15.3:0/1522936637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238101e20 msgr2=0x7f5238105e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 --2- 172.21.15.3:0/1522936637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238101e20 0x7f5238105e60 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f5220009930 tx=0x7f522002bc30 comp rx=0 tx=0).stop 2024-02-09T22:57:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 -- 172.21.15.3:0/1522936637 shutdown_connections 2024-02-09T22:57:51.349 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 --2- 172.21.15.3:0/1522936637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238106540 0x7f523810e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 --2- 172.21.15.3:0/1522936637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238101e20 0x7f5238105e60 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 --2- 172.21.15.3:0/1522936637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5238101470 0x7f5238101850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 -- 172.21.15.3:0/1522936637 >> 172.21.15.3:0/1522936637 conn(0x7f52380fcd50 msgr2=0x7f52380ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 -- 172.21.15.3:0/1522936637 shutdown_connections 2024-02-09T22:57:51.350 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.346+0000 7f523c82a640 1 -- 172.21.15.3:0/1522936637 wait complete. 2024-02-09T22:57:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 Processor -- start 2024-02-09T22:57:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 -- start start 2024-02-09T22:57:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5238101470 0x7f5238073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 0x7f5238073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238073fb0 0x7f52381a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52381a6040 con 0x7f5238106540 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52381a61b0 con 0x7f5238101470 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523c82a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52381a6370 con 0x7f5238073fb0 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 0x7f5238073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 0x7f5238073a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54034/0 (socket says 172.21.15.3:54034) 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 -- 172.21.15.3:0/3756750836 learned_addr learned my addr 172.21.15.3:0/3756750836 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f523659c640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5238101470 0x7f5238073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f5236d9d640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238073fb0 0x7f52381a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 -- 172.21.15.3:0/3756750836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238073fb0 msgr2=0x7f52381a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238073fb0 0x7f52381a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 -- 172.21.15.3:0/3756750836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5238101470 msgr2=0x7f5238073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5238101470 0x7f5238073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f522ffff640 1 -- 172.21.15.3:0/3756750836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5220009560 con 0x7f5238106540 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.347+0000 7f5236d9d640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238073fb0 0x7f52381a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.348+0000 7f522ffff640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 0x7f5238073a70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5220009230 tx=0x7f5220004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.348+0000 7f522f7fe640 1 -- 172.21.15.3:0/3756750836 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f522002cbc0 con 0x7f5238106540 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.348+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52381a65f0 con 0x7f5238106540 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.348+0000 7f522f7fe640 1 -- 172.21.15.3:0/3756750836 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f52200052d0 con 0x7f5238106540 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.348+0000 7f522f7fe640 1 -- 172.21.15.3:0/3756750836 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5220003e30 con 0x7f5238106540 2024-02-09T22:57:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.348+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52381a6b00 con 0x7f5238106540 2024-02-09T22:57:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.350+0000 7f522f7fe640 1 -- 172.21.15.3:0/3756750836 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5220005440 con 0x7f5238106540 2024-02-09T22:57:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.350+0000 7f523f401740 1 Processor -- start 2024-02-09T22:57:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 -- start start 2024-02-09T22:57:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa58490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.361 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa61260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa617a0 0x55684aa63b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684a9c6a00 con 0x55684aa580b0 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684a9c3f30 con 0x55684aa589d0 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523f401740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684a9bf710 con 0x55684aa617a0 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f522f7fe640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f52240a2220 0x7f52240a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f5236d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa58490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f5236d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa58490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54042/0 (socket says 172.21.15.3:54042) 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f5236d9d640 1 -- 172.21.15.3:0/1315517033 learned_addr learned my addr 172.21.15.3:0/1315517033 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f5236d9d640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f52240a2220 0x7f52240a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f522f7fe640 1 -- 172.21.15.3:0/3756750836 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f52200f04a0 con 0x7f5238106540 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f523659c640 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa61260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.351+0000 7f522ffff640 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa617a0 0x55684aa63b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 -- 172.21.15.3:0/1315517033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa617a0 msgr2=0x55684aa63b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa617a0 0x55684aa63b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 -- 172.21.15.3:0/1315517033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 msgr2=0x55684aa61260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa61260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 -- 172.21.15.3:0/1315517033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55684a9843b0 con 0x55684aa580b0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa58490 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5230012cf0 tx=0x7f5230010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3ad152581dd12d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f5236d9d640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f52240a2220 0x7f52240a46e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5230010c40 tx=0x7f5230017000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f522cff9640 1 -- 172.21.15.3:0/1315517033 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5230012de0 con 0x55684aa580b0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f522cff9640 1 -- 172.21.15.3:0/1315517033 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5230009cb0 con 0x55684aa580b0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.352+0000 7f522cff9640 1 -- 172.21.15.3:0/1315517033 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523000cd90 con 0x55684aa580b0 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 -- 172.21.15.3:0/1315517033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 msgr2=0x55684aa58490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa58490 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f5230012cf0 tx=0x7f5230010740 comp rx=0 tx=0).stop 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 -- 172.21.15.3:0/1315517033 shutdown_connections 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa61260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa617a0 0x55684aa63b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 --2- 172.21.15.3:0/1315517033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa58490 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 -- 172.21.15.3:0/1315517033 >> 172.21.15.3:0/1315517033 conn(0x55684aa529d0 msgr2=0x55684aa52db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 -- 172.21.15.3:0/1315517033 shutdown_connections 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 -- 172.21.15.3:0/1315517033 wait complete. 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 -- start start 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.353+0000 7f523f401740 1 Processor -- start 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa5e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa5edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 0x55684aa5f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684a964b90 con 0x55684aa617a0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684a693250 con 0x55684aa589d0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684a945200 con 0x55684aa580b0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 10 client.0 _ll_register_callbacks cb 0x55684a97b5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 0x55684aa5f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 0x55684aa5f300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54044/0 (socket says 172.21.15.3:54044) 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 -- 172.21.15.3:0/132732786 learned_addr learned my addr 172.21.15.3:0/132732786 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f522ffff640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa5edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523659c640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa5e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 -- 172.21.15.3:0/132732786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 msgr2=0x55684aa5e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa5e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 -- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 msgr2=0x55684aa5edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa5edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 -- 172.21.15.3:0/132732786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523001a040 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523659c640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa5e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f5236d9d640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 0x55684aa5f300 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5230009ed0 tx=0x7f5230012620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3492c092caf79cb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f521dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f521dffb640 1 -- 172.21.15.3:0/132732786 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523000cd50 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f521dffb640 1 -- 172.21.15.3:0/132732786 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5230004a00 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.354+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55684a9e5ee0 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f521dffb640 1 -- 172.21.15.3:0/132732786 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5230004bc0 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 10 client.17928 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55684a9bdd90 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f521dffb640 1 -- 172.21.15.3:0/132732786 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55684a9e5ee0 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f521dffb640 1 -- 172.21.15.3:0/132732786 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f5230039450 con 0x55684aa617a0 2024-02-09T22:57:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 10 client.17928 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 10 client.17928 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 20 client.17928 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aa71a80 0x55684aa73ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 4 client.17928 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55684a6bf600 con 0x55684aa71a80 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.355+0000 7f523659c640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aa71a80 0x55684aa73ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.356+0000 7f523659c640 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aa71a80 0x55684aa73ea0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc7f22538b53b6e server_cookie=aa0653fbe2d0f670 in_seq=0 out_seq=0 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.356+0000 7f521dffb640 10 client.17928 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.356+0000 7f521dffb640 1 -- 172.21.15.3:0/132732786 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55684a6bf600 con 0x55684aa71a80 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.356+0000 7f521dffb640 10 client.17928 handle_command_reply: tid=0 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.356+0000 7f523f401740 1 client.17928 shutdown 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 msgr2=0x55684aa5f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 0x55684aa5f300 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f5230009ed0 tx=0x7f5230012620 comp rx=0 tx=0).stop 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 shutdown_connections 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aa71a80 0x55684aa73ea0 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa589d0 0x55684aa5edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa617a0 0x55684aa5f300 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f521dffb640 0 client.17928 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 --2- 172.21.15.3:0/132732786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa5e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 >> 172.21.15.3:0/132732786 conn(0x55684aa529d0 msgr2=0x55684aa54720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 shutdown_connections 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 1 -- 172.21.15.3:0/132732786 wait complete. 2024-02-09T22:57:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.357+0000 7f523f401740 20 client.17928 trim_cache size 0 max 16384 2024-02-09T22:57:51.374 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.370+0000 7f523f401740 1 Processor -- start 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.370+0000 7f523f401740 1 -- start start 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523f401740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa60fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523f401740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa589d0 0x55684aa5edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523f401740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa5f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523f401740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684aa521d0 con 0x55684aa589d0 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523f401740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684aa57890 con 0x55684aa617a0 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523f401740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684aa79070 con 0x55684aa580b0 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f522ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa589d0 0x55684aa5edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f522ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa589d0 0x55684aa5edd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54054/0 (socket says 172.21.15.3:54054) 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f522ffff640 1 -- 172.21.15.3:0/3668987004 learned_addr learned my addr 172.21.15.3:0/3668987004 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:51.375 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523659c640 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa60fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f5236d9d640 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa5f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523659c640 1 -- 172.21.15.3:0/3668987004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 msgr2=0x55684aa5f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523659c640 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa5f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523659c640 1 -- 172.21.15.3:0/3668987004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa589d0 msgr2=0x55684aa5edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523659c640 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa589d0 0x55684aa5edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.371+0000 7f523659c640 1 -- 172.21.15.3:0/3668987004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55684a95a820 con 0x55684aa580b0 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523659c640 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa60fb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55684aa71e60 tx=0x7f522800de70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47566f9dda1b6d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f521ffff640 1 -- 172.21.15.3:0/3668987004 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228017040 con 0x55684aa580b0 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f521ffff640 1 -- 172.21.15.3:0/3668987004 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f522801c3b0 con 0x55684aa580b0 2024-02-09T22:57:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f521ffff640 1 -- 172.21.15.3:0/3668987004 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228025e30 con 0x55684aa580b0 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 -- 172.21.15.3:0/3668987004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 msgr2=0x55684aa60fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa60fb0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x55684aa71e60 tx=0x7f522800de70 comp rx=0 tx=0).stop 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 -- 172.21.15.3:0/3668987004 shutdown_connections 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa5f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa589d0 0x55684aa5edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 --2- 172.21.15.3:0/3668987004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa580b0 0x55684aa60fb0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.372+0000 7f523f401740 1 -- 172.21.15.3:0/3668987004 >> 172.21.15.3:0/3668987004 conn(0x55684aa529d0 msgr2=0x55684aa59b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 -- 172.21.15.3:0/3668987004 shutdown_connections 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 -- 172.21.15.3:0/3668987004 wait complete. 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 -- start start 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 Processor -- start 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa66eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa589d0 0x55684aa673f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa67930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684aac7f50 con 0x55684aa580b0 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684aac8090 con 0x55684aa617a0 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55684aac81d0 con 0x55684aa589d0 2024-02-09T22:57:51.381 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523f401740 10 client.0 _ll_register_callbacks cb 0x7f522800d3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa66eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa66eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54058/0 (socket says 172.21.15.3:54058) 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f523659c640 1 -- 172.21.15.3:0/776942087 learned_addr learned my addr 172.21.15.3:0/776942087 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f5236d9d640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa67930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.373+0000 7f522ffff640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa589d0 0x55684aa673f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523659c640 1 -- 172.21.15.3:0/776942087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa589d0 msgr2=0x55684aa673f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523659c640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa589d0 0x55684aa673f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523659c640 1 -- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 msgr2=0x55684aa67930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523659c640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa67930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523659c640 1 -- 172.21.15.3:0/776942087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5228004300 con 0x55684aa580b0 2024-02-09T22:57:51.382 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523659c640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa66eb0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5228025990 tx=0x7f52280259c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=783f42f2a18d76d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f521dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f521dffb640 1 -- 172.21.15.3:0/776942087 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228017040 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5230009cb0 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f521dffb640 1 -- 172.21.15.3:0/776942087 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5228026e00 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f521dffb640 1 -- 172.21.15.3:0/776942087 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f522801c730 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523f401740 10 client.17934 fetch_fsmap learned FSMap version 40 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.374+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55684a9806f0 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f521dffb640 1 -- 172.21.15.3:0/776942087 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5230009cb0 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f521dffb640 1 -- 172.21.15.3:0/776942087 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f52280046c0 con 0x55684aa580b0 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f523f401740 10 client.17934 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f523f401740 10 client.17934 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:57:51.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f523f401740 20 client.17934 populate_metadata read hostname 'smithi003' 2024-02-09T22:57:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f523f401740 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aac9a60 0x55684aacbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:57:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f523f401740 4 client.17934 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:57:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55684a7101b0 con 0x55684aac9a60 2024-02-09T22:57:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f522ffff640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aac9a60 0x55684aacbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:57:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f521dffb640 10 client.17934 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:51.384 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.375+0000 7f522ffff640 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aac9a60 0x55684aacbe80 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c67130d348d3241 server_cookie=6c45eda294d28fc7 in_seq=0 out_seq=0 2024-02-09T22:57:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f521dffb640 1 -- 172.21.15.3:0/776942087 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55684a7101b0 con 0x55684aac9a60 2024-02-09T22:57:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f521dffb640 10 client.17934 handle_command_reply: tid=0 2024-02-09T22:57:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 client.17934 shutdown 2024-02-09T22:57:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 msgr2=0x55684aa66eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa66eb0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f5228025990 tx=0x7f52280259c0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 shutdown_connections 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55684aac9a60 0x55684aacbe80 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55684aa617a0 0x55684aa67930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55684aa589d0 0x55684aa673f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 --2- 172.21.15.3:0/776942087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55684aa580b0 0x55684aa66eb0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 >> 172.21.15.3:0/776942087 conn(0x55684aa529d0 msgr2=0x55684aa511a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f521dffb640 0 client.17934 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 shutdown_connections 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.410+0000 7f523f401740 1 -- 172.21.15.3:0/776942087 wait complete. 2024-02-09T22:57:51.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523f401740 20 client.17934 trim_cache size 0 max 16384 2024-02-09T22:57:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:57:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:57:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:57:51.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f52240a2220 msgr2=0x7f52240a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f52240a2220 0x7f52240a46e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5230010c40 tx=0x7f5230017000 comp rx=0 tx=0).stop 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 msgr2=0x7f5238073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 0x7f5238073a70 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f5220009230 tx=0x7f5220004270 comp rx=0 tx=0).stop 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 shutdown_connections 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f52240a2220 0x7f52240a46e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5238073fb0 0x7f52381a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5238106540 0x7f5238073a70 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 --2- 172.21.15.3:0/3756750836 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5238101470 0x7f5238073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:57:51.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 >> 172.21.15.3:0/3756750836 conn(0x7f52380fcd50 msgr2=0x7f52380fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:57:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 shutdown_connections 2024-02-09T22:57:51.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:57:51.411+0000 7f523c82a640 1 -- 172.21.15.3:0/3756750836 wait complete. 2024-02-09T22:57:51.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 186885 2.58 MB/sec execute 225 sec latency 2890.377 ms 2024-02-09T22:57:52.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:52 smithi089 ceph-mon[36665]: pgmap v201: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 966 KiB/s wr, 116 op/s 2024-02-09T22:57:52.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:52 smithi003 ceph-mon[32678]: pgmap v201: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 966 KiB/s wr, 116 op/s 2024-02-09T22:57:52.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 188607 2.62 MB/sec execute 226 sec latency 13.827 ms 2024-02-09T22:57:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:52 smithi064 ceph-mon[36638]: pgmap v201: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 966 KiB/s wr, 116 op/s 2024-02-09T22:57:53.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 189665 2.63 MB/sec execute 227 sec latency 297.828 ms 2024-02-09T22:57:54.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:54 smithi089 ceph-mon[36665]: pgmap v202: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 381 KiB/s wr, 47 op/s 2024-02-09T22:57:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:54 smithi003 ceph-mon[32678]: pgmap v202: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 381 KiB/s wr, 47 op/s 2024-02-09T22:57:54.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 189698 2.62 MB/sec execute 228 sec latency 976.988 ms 2024-02-09T22:57:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:54 smithi064 ceph-mon[36638]: pgmap v202: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 381 KiB/s wr, 47 op/s 2024-02-09T22:57:55.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 191015 2.64 MB/sec execute 229 sec latency 1429.755 ms 2024-02-09T22:57:56.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:56 smithi089 ceph-mon[36665]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 381 KiB/s wr, 47 op/s 2024-02-09T22:57:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:56 smithi003 ceph-mon[32678]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 381 KiB/s wr, 47 op/s 2024-02-09T22:57:56.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 192213 2.66 MB/sec execute 230 sec latency 201.136 ms 2024-02-09T22:57:56.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:56 smithi064 ceph-mon[36638]: pgmap v203: 97 pgs: 97 active+clean; 74 MiB data, 1020 MiB used, 1.0 TiB / 1.0 TiB avail; 407 KiB/s rd, 381 KiB/s wr, 47 op/s 2024-02-09T22:57:57.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 192213 2.65 MB/sec execute 231 sec latency 1201.260 ms 2024-02-09T22:57:58.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:57:58 smithi089 ceph-mon[36665]: pgmap v204: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 809 KiB/s wr, 114 op/s 2024-02-09T22:57:58.487 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:57:58 smithi064 ceph-mon[36638]: pgmap v204: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 809 KiB/s wr, 114 op/s 2024-02-09T22:57:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:57:58 smithi003 ceph-mon[32678]: pgmap v204: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 981 KiB/s rd, 809 KiB/s wr, 114 op/s 2024-02-09T22:57:58.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 193113 2.66 MB/sec execute 232 sec latency 1610.266 ms 2024-02-09T22:57:59.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 193239 2.65 MB/sec execute 233 sec latency 915.028 ms 2024-02-09T22:58:00.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:00 smithi003 ceph-mon[32678]: pgmap v205: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 752 KiB/s wr, 95 op/s 2024-02-09T22:58:00.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 194568 2.67 MB/sec execute 234 sec latency 1362.991 ms 2024-02-09T22:58:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:00 smithi064 ceph-mon[36638]: pgmap v205: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 752 KiB/s wr, 95 op/s 2024-02-09T22:58:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:00 smithi089 ceph-mon[36665]: pgmap v205: 97 pgs: 97 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 945 KiB/s rd, 752 KiB/s wr, 95 op/s 2024-02-09T22:58:01.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195565 2.68 MB/sec execute 235 sec latency 25.635 ms 2024-02-09T22:58:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:02 smithi003 ceph-mon[32678]: pgmap v206: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-02-09T22:58:02.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195763 2.67 MB/sec execute 236 sec latency 853.415 ms 2024-02-09T22:58:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:02 smithi064 ceph-mon[36638]: pgmap v206: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-02-09T22:58:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:02 smithi089 ceph-mon[36665]: pgmap v206: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.4 MiB/s wr, 157 op/s 2024-02-09T22:58:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:03 smithi003 ceph-mon[32678]: osdmap e90: 12 total, 12 up, 12 in 2024-02-09T22:58:03.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195819 2.66 MB/sec execute 237 sec latency 1255.108 ms 2024-02-09T22:58:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:03 smithi064 ceph-mon[36638]: osdmap e90: 12 total, 12 up, 12 in 2024-02-09T22:58:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:03 smithi089 ceph-mon[36665]: osdmap e90: 12 total, 12 up, 12 in 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 -- 172.21.15.3:0/3273622156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 msgr2=0x7f0cb0100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3273622156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb0100160 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009930 tx=0x7f0ca002bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 -- 172.21.15.3:0/3273622156 shutdown_connections 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3273622156 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cb0106600 0x7f0cb010e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3273622156 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb0100730 0x7f0cb0105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3273622156 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb0100160 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 -- 172.21.15.3:0/3273622156 >> 172.21.15.3:0/3273622156 conn(0x7f0cb0075410 msgr2=0x7f0cb0075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 -- 172.21.15.3:0/3273622156 shutdown_connections 2024-02-09T22:58:04.512 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.508+0000 7f0cb66df640 1 -- 172.21.15.3:0/3273622156 wait complete. 2024-02-09T22:58:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 Processor -- start 2024-02-09T22:58:04.513 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 -- start start 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb019d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb0106600 0x7f0cb019db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cb0197710 0x7f0cb0197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb01981e0 con 0x7f0cb00ffd80 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0198350 con 0x7f0cb0197710 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0cb66df640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cb0198510 con 0x7f0cb0106600 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb019d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.509+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb019d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54926/0 (socket says 172.21.15.3:54926) 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 -- 172.21.15.3:0/3045064558 learned_addr learned my addr 172.21.15.3:0/3045064558 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb0106600 0x7f0cb019db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cb0197710 0x7f0cb0197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 -- 172.21.15.3:0/3045064558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb0106600 msgr2=0x7f0cb019db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.515 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb0106600 0x7f0cb019db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 -- 172.21.15.3:0/3045064558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cb0197710 msgr2=0x7f0cb0197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cb0197710 0x7f0cb0197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 -- 172.21.15.3:0/3045064558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0009560 con 0x7f0cb00ffd80 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0caffff640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb019d650 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009230 tx=0x7f0ca0004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0cad7fa640 1 -- 172.21.15.3:0/3045064558 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0004440 con 0x7f0cb00ffd80 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cb01a4450 con 0x7f0cb00ffd80 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0cad7fa640 1 -- 172.21.15.3:0/3045064558 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca00052d0 con 0x7f0cb00ffd80 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0cad7fa640 1 -- 172.21.15.3:0/3045064558 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0003d70 con 0x7f0cb00ffd80 2024-02-09T22:58:04.516 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.510+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01a4930 con 0x7f0cb00ffd80 2024-02-09T22:58:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cad7fa640 1 -- 172.21.15.3:0/3045064558 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0ca0005440 con 0x7f0cb00ffd80 2024-02-09T22:58:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 Processor -- start 2024-02-09T22:58:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 -- start start 2024-02-09T22:58:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ee5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 0x5641d9ef3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9e54a00 con 0x5641d9ee6390 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9e51f30 con 0x5641d9ee5a70 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb938c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9e4d710 con 0x5641d9ef3bb0 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cad7fa640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0c980a2220 0x7f0c980a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 0x5641d9ef3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0cb4c7c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ee5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 0x5641d9ef3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54938/0 (socket says 172.21.15.3:54938) 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0caffff640 1 -- 172.21.15.3:0/1042588425 learned_addr learned my addr 172.21.15.3:0/1042588425 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.513+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0c980a2220 0x7f0c980a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0cad7fa640 1 -- 172.21.15.3:0/3045064558 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f0ca00f0220 con 0x7f0cb00ffd80 2024-02-09T22:58:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0caffff640 1 -- 172.21.15.3:0/1042588425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 msgr2=0x5641d9ef5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0caffff640 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0caffff640 1 -- 172.21.15.3:0/1042588425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 msgr2=0x5641d9ee5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0caffff640 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ee5e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0caffff640 1 -- 172.21.15.3:0/1042588425 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641d9e123b0 con 0x5641d9ee6390 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ee5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0caffff640 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 0x5641d9ef3670 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009430 tx=0x7f0ca002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1a0ed0531d0d5d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0c980a2220 0x7f0c980a46e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8005c40 tx=0x7f0ca8005bd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0c967fc640 1 -- 172.21.15.3:0/1042588425 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00bcd00 con 0x5641d9ee6390 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0c967fc640 1 -- 172.21.15.3:0/1042588425 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca00f0020 con 0x5641d9ee6390 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.514+0000 7f0c967fc640 1 -- 172.21.15.3:0/1042588425 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00ecf20 con 0x5641d9ee6390 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 -- 172.21.15.3:0/1042588425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 msgr2=0x5641d9ef3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 0x5641d9ef3670 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009430 tx=0x7f0ca002bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 -- 172.21.15.3:0/1042588425 shutdown_connections 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee6390 0x5641d9ef3670 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1042588425 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ee5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 -- 172.21.15.3:0/1042588425 >> 172.21.15.3:0/1042588425 conn(0x5641d9ee09d0 msgr2=0x5641d9ee0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 -- 172.21.15.3:0/1042588425 shutdown_connections 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 -- 172.21.15.3:0/1042588425 wait complete. 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.515+0000 7f0cb938c740 1 -- start start 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 Processor -- start 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ef0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ee6390 0x5641d9ef1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef1680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9df2b90 con 0x5641d9ef3bb0 2024-02-09T22:58:04.526 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9b21250 con 0x5641d9ee5a70 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9dd3200 con 0x5641d9ee6390 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb938c740 10 client.0 _ll_register_callbacks cb 0x5641d9e095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54952/0 (socket says 172.21.15.3:54952) 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 -- 172.21.15.3:0/548263964 learned_addr learned my addr 172.21.15.3:0/548263964 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0caffff640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ef0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ee6390 0x5641d9ef1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 -- 172.21.15.3:0/548263964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ee6390 msgr2=0x5641d9ef1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ee6390 0x5641d9ef1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 -- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 msgr2=0x5641d9ef0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.516+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ef0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb4c7c640 1 -- 172.21.15.3:0/548263964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cb01a4930 con 0x5641d9ef3bb0 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0caffff640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ef0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:04.527 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef1680 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80123e0 tx=0x7f0ca8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ce3251735b9b98 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0c8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/548263964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca800ddd0 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/548263964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f0ca800f8f0 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5641d9e73ee0 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/548263964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ca800fab0 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 10 client.17946 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5641d9e4bd90 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/548263964 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5641d9e73ee0 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/548263964 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0ca8021cc0 con 0x5641d9ef3bb0 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 10 client.17946 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 10 client.17946 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 20 client.17946 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:04.528 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f03c70 0x5641d9f06090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 4 client.17946 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.517+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641d9b4d600 con 0x5641d9f03c70 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.518+0000 7f0caffff640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f03c70 0x5641d9f06090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.518+0000 7f0caffff640 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f03c70 0x5641d9f06090 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2da376f55720b594 server_cookie=15b41d47f10ca6ba in_seq=0 out_seq=0 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.518+0000 7f0c8f7fe640 10 client.17946 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/548263964 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5641d9b4d600 con 0x5641d9f03c70 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0c8f7fe640 10 client.17946 handle_command_reply: tid=0 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 client.17946 shutdown 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 msgr2=0x5641d9ef1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef1680 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80123e0 tx=0x7f0ca8004a40 comp rx=0 tx=0).stop 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 shutdown_connections 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f03c70 0x5641d9f06090 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.529 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef1680 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ee6390 0x5641d9ef1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0c8f7fe640 0 client.17946 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 --2- 172.21.15.3:0/548263964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee5a70 0x5641d9ef0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 >> 172.21.15.3:0/548263964 conn(0x5641d9ee09d0 msgr2=0x5641d9ee2720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 shutdown_connections 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 1 -- 172.21.15.3:0/548263964 wait complete. 2024-02-09T22:58:04.530 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.519+0000 7f0cb938c740 20 client.17946 trim_cache size 0 max 16384 2024-02-09T22:58:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 Processor -- start 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 -- start start 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9ef11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ee1b90 con 0x5641d9ee5a70 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ee5390 con 0x5641d9ee6390 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb938c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9eec860 con 0x5641d9ef3bb0 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54966/0 (socket says 172.21.15.3:54966) 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 -- 172.21.15.3:0/1931296757 learned_addr learned my addr 172.21.15.3:0/1931296757 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9ef11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 -- 172.21.15.3:0/1931296757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 msgr2=0x5641d9ef16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 -- 172.21.15.3:0/1931296757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 msgr2=0x5641d9ef11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9ef11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 -- 172.21.15.3:0/1931296757 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641d9de8820 con 0x5641d9ee5a70 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9ef11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0cb4c7c640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.533+0000 7f0caffff640 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef3330 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00bcc90 tx=0x7f0ca0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c80d40009e4ab7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0c95ffb640 1 -- 172.21.15.3:0/1931296757 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00b8fa0 con 0x5641d9ee5a70 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0c95ffb640 1 -- 172.21.15.3:0/1931296757 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca00bd400 con 0x5641d9ee5a70 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0c95ffb640 1 -- 172.21.15.3:0/1931296757 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00ecba0 con 0x5641d9ee5a70 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 -- 172.21.15.3:0/1931296757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 msgr2=0x5641d9ef3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef3330 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00bcc90 tx=0x7f0ca0009260 comp rx=0 tx=0).stop 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 -- 172.21.15.3:0/1931296757 shutdown_connections 2024-02-09T22:58:04.542 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9ef11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1931296757 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef3330 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 -- 172.21.15.3:0/1931296757 >> 172.21.15.3:0/1931296757 conn(0x5641d9ee09d0 msgr2=0x5641d9f0a9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 -- 172.21.15.3:0/1931296757 shutdown_connections 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.534+0000 7f0cb938c740 1 -- 172.21.15.3:0/1931296757 wait complete. 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 -- start start 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 Processor -- start 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef9fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9efa520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ef8ce0 con 0x5641d9ee5a70 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ef8e50 con 0x5641d9ee6390 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641d9ef9010 con 0x5641d9ef3bb0 2024-02-09T22:58:04.543 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0cb938c740 10 client.0 _ll_register_callbacks cb 0x7f0ca8012860 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef9fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef9fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54982/0 (socket says 172.21.15.3:54982) 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 -- 172.21.15.3:0/1704191184 learned_addr learned my addr 172.21.15.3:0/1704191184 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9efa520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 -- 172.21.15.3:0/1704191184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 msgr2=0x5641d9ef86d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 -- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 msgr2=0x5641d9efa520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9efa520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caffff640 1 -- 172.21.15.3:0/1704191184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641d9e123b0 con 0x5641d9ee5a70 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.535+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9efa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0caffff640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef9fe0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00edb10 tx=0x7f0ca0003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb42db08576b773b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0c8f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:04.544 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/1704191184 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca00d9070 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/1704191184 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f0ca00bcd00 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ca00f0020 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/1704191184 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ca00edbd0 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 10 client.17952 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5641d9e0e6f0 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/1704191184 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ca00f0020 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/1704191184 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0ca003c070 con 0x5641d9ee5a70 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 10 client.17952 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 10 client.17952 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 20 client.17952 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f5ad60 0x5641d9f5d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 4 client.17952 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.536+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5641d9b9e1b0 con 0x5641d9f5ad60 2024-02-09T22:58:04.545 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.537+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f5ad60 0x5641d9f5d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.537+0000 7f0c8f7fe640 10 client.17952 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:04.546 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.537+0000 7f0caf7fe640 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f5ad60 0x5641d9f5d150 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fda6f62dc240899e server_cookie=1893433af5f8faea in_seq=0 out_seq=0 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.576+0000 7f0c8f7fe640 1 -- 172.21.15.3:0/1704191184 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5641d9b9e1b0 con 0x5641d9f5ad60 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.576+0000 7f0c8f7fe640 10 client.17952 handle_command_reply: tid=0 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.576+0000 7f0cb938c740 1 client.17952 shutdown 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 msgr2=0x5641d9ef9fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef9fe0 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0ca00edb10 tx=0x7f0ca0003840 comp rx=0 tx=0).stop 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 shutdown_connections 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5641d9f5ad60 0x5641d9f5d150 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5641d9ee6390 0x5641d9efa520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5641d9ef3bb0 0x5641d9ef86d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0c8f7fe640 0 client.17952 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 --2- 172.21.15.3:0/1704191184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5641d9ee5a70 0x5641d9ef9fe0 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 >> 172.21.15.3:0/1704191184 conn(0x5641d9ee09d0 msgr2=0x5641d9eef4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 shutdown_connections 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 1 -- 172.21.15.3:0/1704191184 wait complete. 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb938c740 20 client.17952 trim_cache size 0 max 16384 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:58:04.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:58:04.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0c980a2220 msgr2=0x7f0c980a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0c980a2220 0x7f0c980a46e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0ca8005c40 tx=0x7f0ca8005bd0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 msgr2=0x7f0cb019d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:04.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.577+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb019d650 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0009230 tx=0x7f0ca0004040 comp rx=0 tx=0).stop 2024-02-09T22:58:04.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 shutdown_connections 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0c980a2220 0x7f0c980a46e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0cb0197710 0x7f0cb0197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0cb0106600 0x7f0cb019db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 --2- 172.21.15.3:0/3045064558 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0cb00ffd80 0x7f0cb019d650 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 >> 172.21.15.3:0/3045064558 conn(0x7f0cb0075410 msgr2=0x7f0cb00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 shutdown_connections 2024-02-09T22:58:04.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:04.578+0000 7f0cb66df640 1 -- 172.21.15.3:0/3045064558 wait complete. 2024-02-09T22:58:04.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:04 smithi003 ceph-mon[32678]: pgmap v208: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-02-09T22:58:04.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195819 2.65 MB/sec execute 238 sec latency 1562.239 ms 2024-02-09T22:58:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:04 smithi064 ceph-mon[36638]: pgmap v208: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-02-09T22:58:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:04 smithi089 ceph-mon[36665]: pgmap v208: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-02-09T22:58:05.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195819 2.64 MB/sec execute 239 sec latency 2562.304 ms 2024-02-09T22:58:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:06 smithi003 ceph-mon[32678]: pgmap v209: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-02-09T22:58:06.645 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195819 2.63 MB/sec execute 240 sec latency 3562.353 ms 2024-02-09T22:58:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:06 smithi064 ceph-mon[36638]: pgmap v209: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-02-09T22:58:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:06 smithi089 ceph-mon[36665]: pgmap v209: 97 pgs: 97 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-02-09T22:58:07.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195819 2.62 MB/sec execute 241 sec latency 4562.479 ms 2024-02-09T22:58:07.877 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:58:07.877 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:58:07.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 -- 172.21.15.3:0/4289776019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c102e00 msgr2=0x7f106c103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:07.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 --2- 172.21.15.3:0/4289776019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c102e00 0x7f106c103260 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f1060009930 tx=0x7f106002bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:07.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 -- 172.21.15.3:0/4289776019 shutdown_connections 2024-02-09T22:58:07.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 --2- 172.21.15.3:0/4289776019 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 0x7f106c10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:07.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 --2- 172.21.15.3:0/4289776019 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c102e00 0x7f106c103260 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:07.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 --2- 172.21.15.3:0/4289776019 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c101520 0x7f106c101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 -- 172.21.15.3:0/4289776019 >> 172.21.15.3:0/4289776019 conn(0x7f106c0fccc0 msgr2=0x7f106c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 -- 172.21.15.3:0/4289776019 shutdown_connections 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 -- 172.21.15.3:0/4289776019 wait complete. 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.992+0000 7f107198a640 1 Processor -- start 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 -- start start 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 0x7f106c19d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c102e00 0x7f106c19dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 0x7f106c197770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c197f00 con 0x7f106c101520 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c198070 con 0x7f106c102e00 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f107198a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c198230 con 0x7f106c1037a0 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 0x7f106c19d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 0x7f106c19d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41068/0 (socket says 172.21.15.3:41068) 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 -- 172.21.15.3:0/3584930679 learned_addr learned my addr 172.21.15.3:0/3584930679 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:07.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c102e00 0x7f106c19dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106b7fe640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 0x7f106c197770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 -- 172.21.15.3:0/3584930679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 msgr2=0x7f106c197770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 0x7f106c197770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 -- 172.21.15.3:0/3584930679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c102e00 msgr2=0x7f106c19dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c102e00 0x7f106c19dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 -- 172.21.15.3:0/3584930679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060009560 con 0x7f106c101520 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c102e00 0x7f106c19dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106b7fe640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 0x7f106c197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.993+0000 7f106affd640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 0x7f106c19d660 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f105c00d950 tx=0x7f105c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.994+0000 7f1070988640 1 -- 172.21.15.3:0/3584930679 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c009c30 con 0x7f106c101520 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.994+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c198510 con 0x7f106c101520 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.994+0000 7f1070988640 1 -- 172.21.15.3:0/3584930679 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c009d90 con 0x7f106c101520 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.994+0000 7f1070988640 1 -- 172.21.15.3:0/3584930679 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c00f7e0 con 0x7f106c101520 2024-02-09T22:58:07.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.994+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c1a45e0 con 0x7f106c101520 2024-02-09T22:58:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.996+0000 7f1070988640 1 -- 172.21.15.3:0/3584930679 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f105c010ad0 con 0x7f106c101520 2024-02-09T22:58:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.996+0000 7f1070988640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f10540a2230 0x7f10540a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.996+0000 7f1074639740 1 Processor -- start 2024-02-09T22:58:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.996+0000 7f1074639740 1 -- start start 2024-02-09T22:58:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.996+0000 7f1070988640 1 -- 172.21.15.3:0/3584930679 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f105c0cdde0 con 0x7f106c101520 2024-02-09T22:58:08.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f10540a2230 0x7f10540a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1074639740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd57d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1074639740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1074639740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd63300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1074639740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dd12d00 con 0x56099dd57980 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1074639740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dcc72f0 con 0x56099dd60eb0 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1074639740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dcc4820 con 0x56099dd58330 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd57d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd57d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41072/0 (socket says 172.21.15.3:41072) 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 -- 172.21.15.3:0/799924596 learned_addr learned my addr 172.21.15.3:0/799924596 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106b7fe640 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd63300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f10540a2230 0x7f10540a46f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1060006010 tx=0x7f10600047a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106a7fc640 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.007 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 -- 172.21.15.3:0/799924596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 msgr2=0x56099dd608e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd608e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 -- 172.21.15.3:0/799924596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 msgr2=0x56099dd63300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd63300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 -- 172.21.15.3:0/799924596 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56099dc81ff0 con 0x56099dd57980 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f106affd640 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd57d60 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f105c0a2a20 tx=0x7f105c005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84be416aba7abbfd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.997+0000 7f1051ffb640 1 -- 172.21.15.3:0/799924596 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c09a7b0 con 0x56099dd57980 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1051ffb640 1 -- 172.21.15.3:0/799924596 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c09e900 con 0x56099dd57980 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1051ffb640 1 -- 172.21.15.3:0/799924596 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c099c00 con 0x56099dd57980 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 -- 172.21.15.3:0/799924596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 msgr2=0x56099dd57d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd57d60 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f105c0a2a20 tx=0x7f105c005880 comp rx=0 tx=0).stop 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 -- 172.21.15.3:0/799924596 shutdown_connections 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd63300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.008 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd608e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 --2- 172.21.15.3:0/799924596 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd57d60 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 -- 172.21.15.3:0/799924596 >> 172.21.15.3:0/799924596 conn(0x56099dd52050 msgr2=0x56099dd52430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 -- 172.21.15.3:0/799924596 shutdown_connections 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.998+0000 7f1074639740 1 -- 172.21.15.3:0/799924596 wait complete. 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 -- start start 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 Processor -- start 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd5de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd58330 0x56099dd5e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 0x56099dd5e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dcc0000 con 0x56099dd60eb0 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099d993430 con 0x56099dd58330 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099d94fe00 con 0x56099dd57980 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f1074639740 10 client.0 _ll_register_callbacks cb 0x56099dcb47f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 0x56099dd5e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.009 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f106b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 0x56099dd5e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41078/0 (socket says 172.21.15.3:41078) 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f106b7fe640 1 -- 172.21.15.3:0/1676440077 learned_addr learned my addr 172.21.15.3:0/1676440077 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f106a7fc640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd58330 0x56099dd5e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:07.999+0000 7f106affd640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd5de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f106b7fe640 1 -- 172.21.15.3:0/1676440077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 msgr2=0x56099dd5de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f106b7fe640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd5de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f106b7fe640 1 -- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd58330 msgr2=0x56099dd5e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f106b7fe640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd58330 0x56099dd5e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f106b7fe640 1 -- 172.21.15.3:0/1676440077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c1a45e0 con 0x56099dd60eb0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f106b7fe640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 0x56099dd5e8a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f1064011d40 tx=0x7f10640107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=121f141b2445f50b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f104affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f104affd640 1 -- 172.21.15.3:0/1676440077 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064016070 con 0x56099dd60eb0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f104affd640 1 -- 172.21.15.3:0/1676440077 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f106401a410 con 0x56099dd60eb0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56099d9bf7e0 con 0x56099dd60eb0 2024-02-09T22:58:08.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f104affd640 1 -- 172.21.15.3:0/1676440077 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1064009e70 con 0x56099dd60eb0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f1074639740 10 client.17964 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56099dcbe680 con 0x56099dd60eb0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f104affd640 1 -- 172.21.15.3:0/1676440077 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1064027020 con 0x56099dd60eb0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f104affd640 1 -- 172.21.15.3:0/1676440077 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f106400e4e0 con 0x56099dd60eb0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f1074639740 10 client.17964 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f1074639740 10 client.17964 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.000+0000 7f1074639740 20 client.17964 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.001+0000 7f1074639740 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099dd714c0 0x56099dd738e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.001+0000 7f1074639740 4 client.17964 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.001+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56099d9fc720 con 0x56099dd714c0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.001+0000 7f106affd640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099dd714c0 0x56099dd738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.001+0000 7f106affd640 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099dd714c0 0x56099dd738e0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae297dbd9359111d server_cookie=f81a2d8e9ce4e751 in_seq=0 out_seq=0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.001+0000 7f104affd640 10 client.17964 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f104affd640 1 -- 172.21.15.3:0/1676440077 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56099d9fc720 con 0x56099dd714c0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f104affd640 10 client.17964 handle_command_reply: tid=0 2024-02-09T22:58:08.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 client.17964 shutdown 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 msgr2=0x56099dd5e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 0x56099dd5e8a0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f1064011d40 tx=0x7f10640107a0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 shutdown_connections 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099dd714c0 0x56099dd738e0 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd58330 0x56099dd5e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd60eb0 0x56099dd5e8a0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 --2- 172.21.15.3:0/1676440077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd5de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f104affd640 0 client.17964 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 >> 172.21.15.3:0/1676440077 conn(0x56099dd52050 msgr2=0x56099dd53da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 shutdown_connections 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 1 -- 172.21.15.3:0/1676440077 wait complete. 2024-02-09T22:58:08.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.002+0000 7f1074639740 20 client.17964 trim_cache size 0 max 16384 2024-02-09T22:58:08.025 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 Processor -- start 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 -- start start 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd5e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd5e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd5b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dc44fd0 con 0x56099dd57980 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dd52c20 con 0x56099dd60eb0 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f1074639740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099dd4d6f0 con 0x56099dd58330 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f106affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd5e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f106affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd5e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41092/0 (socket says 172.21.15.3:41092) 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.017+0000 7f106affd640 1 -- 172.21.15.3:0/548543242 learned_addr learned my addr 172.21.15.3:0/548543242 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106a7fc640 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd5e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106b7fe640 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd5b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106affd640 1 -- 172.21.15.3:0/548543242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 msgr2=0x56099dd5e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.026 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106affd640 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd5e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106affd640 1 -- 172.21.15.3:0/548543242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 msgr2=0x56099dd5b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106affd640 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd5b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106affd640 1 -- 172.21.15.3:0/548543242 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56099dc5ab30 con 0x56099dd57980 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f106affd640 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd5e3b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f105c0ce0c0 tx=0x7f105c010e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82c6d4858c72f19 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f10517fa640 1 -- 172.21.15.3:0/548543242 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0a2f50 con 0x56099dd57980 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f10517fa640 1 -- 172.21.15.3:0/548543242 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c065c40 con 0x56099dd57980 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.018+0000 7f10517fa640 1 -- 172.21.15.3:0/548543242 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c0ba0c0 con 0x56099dd57980 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 -- 172.21.15.3:0/548543242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 msgr2=0x56099dd5e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd5e3b0 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f105c0ce0c0 tx=0x7f105c010e70 comp rx=0 tx=0).stop 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 -- 172.21.15.3:0/548543242 shutdown_connections 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd5b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd58330 0x56099dd5e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 --2- 172.21.15.3:0/548543242 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd57980 0x56099dd5e3b0 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 -- 172.21.15.3:0/548543242 >> 172.21.15.3:0/548543242 conn(0x56099dd52050 msgr2=0x56099dd78240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 -- 172.21.15.3:0/548543242 shutdown_connections 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 -- 172.21.15.3:0/548543242 wait complete. 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 -- start start 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 Processor -- start 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.019+0000 7f1074639740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd668c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 0x56099dd66e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd67340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099ddc79d0 con 0x56099dd58330 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099ddc7b10 con 0x56099dd60eb0 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56099ddc7cd0 con 0x56099dd57980 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 10 client.0 _ll_register_callbacks cb 0x7f105c009f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 0x56099dd66e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 0x56099dd66e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41102/0 (socket says 172.21.15.3:41102) 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 -- 172.21.15.3:0/3287461548 learned_addr learned my addr 172.21.15.3:0/3287461548 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106affd640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 -- 172.21.15.3:0/3287461548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 msgr2=0x56099dd668c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd668c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 -- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 msgr2=0x56099dd67340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106b7fe640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd67340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd67340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 -- 172.21.15.3:0/3287461548 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56099dc81ff0 con 0x56099dd58330 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106b7fe640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd67340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f106a7fc640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 0x56099dd66e00 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1060009930 tx=0x7f1060009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a852ea871616bb7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f104affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f104affd640 1 -- 172.21.15.3:0/3287461548 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1060005340 con 0x56099dd58330 2024-02-09T22:58:08.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f104affd640 1 -- 172.21.15.3:0/3287461548 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f1060046040 con 0x56099dd58330 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f105c09e900 con 0x56099dd58330 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.020+0000 7f104affd640 1 -- 172.21.15.3:0/3287461548 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10600054a0 con 0x56099dd58330 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 10 client.17970 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56099dc80850 con 0x56099dd58330 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f104affd640 1 -- 172.21.15.3:0/3287461548 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f105c09e900 con 0x56099dd58330 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f104affd640 1 -- 172.21.15.3:0/3287461548 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f106003c040 con 0x56099dd58330 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 10 client.17970 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 10 client.17970 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 20 client.17970 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099ddc96b0 0x56099ddcbad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 4 client.17970 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56099da4b890 con 0x56099ddc96b0 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.021+0000 7f106affd640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099ddc96b0 0x56099ddcbad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f104affd640 10 client.17970 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f106affd640 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099ddc96b0 0x56099ddcbad0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18dcf770c27eb4d2 server_cookie=8162371228947176 in_seq=0 out_seq=0 2024-02-09T22:58:08.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f104affd640 1 -- 172.21.15.3:0/3287461548 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56099da4b890 con 0x56099ddc96b0 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f104affd640 10 client.17970 handle_command_reply: tid=0 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 client.17970 shutdown 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 msgr2=0x56099dd66e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 0x56099dd66e00 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f1060009930 tx=0x7f1060009260 comp rx=0 tx=0).stop 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 shutdown_connections 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56099ddc96b0 0x56099ddcbad0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56099dd60eb0 0x56099dd67340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56099dd58330 0x56099dd66e00 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f104affd640 0 client.17970 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 --2- 172.21.15.3:0/3287461548 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56099dd57980 0x56099dd668c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 >> 172.21.15.3:0/3287461548 conn(0x56099dd52050 msgr2=0x56099dd5f460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 shutdown_connections 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 1 -- 172.21.15.3:0/3287461548 wait complete. 2024-02-09T22:58:08.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.022+0000 7f1074639740 20 client.17970 trim_cache size 0 max 16384 2024-02-09T22:58:08.032 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:58:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:58:08.032 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:58:08.032 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f10540a2230 msgr2=0x7f10540a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f10540a2230 0x7f10540a46f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1060006010 tx=0x7f10600047a0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 msgr2=0x7f106c19d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 0x7f106c19d660 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f105c00d950 tx=0x7f105c00de10 comp rx=0 tx=0).stop 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 shutdown_connections 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f10540a2230 0x7f10540a46f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f106c102e00 0x7f106c19dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f106c1037a0 0x7f106c197770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 --2- 172.21.15.3:0/3584930679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f106c101520 0x7f106c19d660 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 >> 172.21.15.3:0/3584930679 conn(0x7f106c0fccc0 msgr2=0x7f106c0fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 shutdown_connections 2024-02-09T22:58:08.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.023+0000 7f107198a640 1 -- 172.21.15.3:0/3584930679 wait complete. 2024-02-09T22:58:08.037 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:58:08.037 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:58:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 -- 172.21.15.3:0/3064866582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c104830 msgr2=0x7efe5c104c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 --2- 172.21.15.3:0/3064866582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c104830 0x7efe5c104c10 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7efe50009900 tx=0x7efe5002bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 -- 172.21.15.3:0/3064866582 shutdown_connections 2024-02-09T22:58:08.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 --2- 172.21.15.3:0/3064866582 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1003e0 0x7efe5c10c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 --2- 172.21.15.3:0/3064866582 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c0ffa20 0x7efe5c0ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 --2- 172.21.15.3:0/3064866582 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c104830 0x7efe5c104c10 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 -- 172.21.15.3:0/3064866582 >> 172.21.15.3:0/3064866582 conn(0x7efe5c076580 msgr2=0x7efe5c076990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 -- 172.21.15.3:0/3064866582 shutdown_connections 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 -- 172.21.15.3:0/3064866582 wait complete. 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 Processor -- start 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.142+0000 7efe6316c640 1 -- start start 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 0x7efe5c19d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1003e0 0x7efe5c19d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c104830 0x7efe5c1975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c197c40 con 0x7efe5c0ffa20 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c197db0 con 0x7efe5c1003e0 2024-02-09T22:58:08.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c197f70 con 0x7efe5c104830 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 0x7efe5c19d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 0x7efe5c19d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41126/0 (socket says 172.21.15.3:41126) 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 -- 172.21.15.3:0/1858092370 learned_addr learned my addr 172.21.15.3:0/1858092370 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe61709640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c104830 0x7efe5c1975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe5bfff640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1003e0 0x7efe5c19d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 -- 172.21.15.3:0/1858092370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c104830 msgr2=0x7efe5c1975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c104830 0x7efe5c1975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 -- 172.21.15.3:0/1858092370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1003e0 msgr2=0x7efe5c19d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1003e0 0x7efe5c19d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 -- 172.21.15.3:0/1858092370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe50009560 con 0x7efe5c0ffa20 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe61709640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c104830 0x7efe5c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe60f08640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 0x7efe5c19d440 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7efe50005940 tx=0x7efe50004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe50004440 con 0x7efe5c0ffa20 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe500052d0 con 0x7efe5c0ffa20 2024-02-09T22:58:08.149 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe50003da0 con 0x7efe5c0ffa20 2024-02-09T22:58:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.143+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c1981f0 con 0x7efe5c0ffa20 2024-02-09T22:58:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.144+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c1a44e0 con 0x7efe5c0ffa20 2024-02-09T22:58:08.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.145+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c068500 con 0x7efe5c0ffa20 2024-02-09T22:58:08.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.146+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efe50005440 con 0x7efe5c0ffa20 2024-02-09T22:58:08.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.146+0000 7efe59ffb640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe4c0a2230 0x7efe4c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.146+0000 7efe5bfff640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe4c0a2230 0x7efe4c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.146+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7efe500f0850 con 0x7efe5c0ffa20 2024-02-09T22:58:08.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.147+0000 7efe5bfff640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe4c0a2230 0x7efe4c0a46f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe48005fd0 tx=0x7efe48005850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.150+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efe500b95d0 con 0x7efe5c0ffa20 2024-02-09T22:58:08.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.371+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe5c198b20 con 0x7efe5c0ffa20 2024-02-09T22:58:08.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.372+0000 7efe59ffb640 1 -- 172.21.15.3:0/1858092370 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+11373 (secure 0 0 0) 0x7efe500bcc90 con 0x7efe5c0ffa20 2024-02-09T22:58:08.375 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:58:08.375 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":40,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:57:05.445840+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24470":{"gid":24470,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3082808593","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3082808593},{"type":"v1","addr":"172.21.15.64:6835","nonce":3082808593}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:58:08.375 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 40 2024-02-09T22:58:08.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe4c0a2230 msgr2=0x7efe4c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe4c0a2230 0x7efe4c0a46f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efe48005fd0 tx=0x7efe48005850 comp rx=0 tx=0).stop 2024-02-09T22:58:08.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 msgr2=0x7efe5c19d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 0x7efe5c19d440 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7efe50005940 tx=0x7efe50004040 comp rx=0 tx=0).stop 2024-02-09T22:58:08.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 shutdown_connections 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe4c0a2230 0x7efe4c0a46f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe5c1003e0 0x7efe5c19d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe5c104830 0x7efe5c1975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 --2- 172.21.15.3:0/1858092370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe5c0ffa20 0x7efe5c19d440 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 >> 172.21.15.3:0/1858092370 conn(0x7efe5c076580 msgr2=0x7efe5c10d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 shutdown_connections 2024-02-09T22:58:08.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.374+0000 7efe6316c640 1 -- 172.21.15.3:0/1858092370 wait complete. 2024-02-09T22:58:08.388 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:08.388 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 -- 172.21.15.3:0/997790855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4614100440 msgr2=0x7f4614100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 --2- 172.21.15.3:0/997790855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4614100440 0x7f4614100820 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f4610009930 tx=0x7f461002bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 -- 172.21.15.3:0/997790855 shutdown_connections 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 --2- 172.21.15.3:0/997790855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 0x7f4614102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 --2- 172.21.15.3:0/997790855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f46141026c0 0x7f461410d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 --2- 172.21.15.3:0/997790855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4614100440 0x7f4614100820 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 -- 172.21.15.3:0/997790855 >> 172.21.15.3:0/997790855 conn(0x7f4614077310 msgr2=0x7f4614077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 -- 172.21.15.3:0/997790855 shutdown_connections 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.490+0000 7f461b2a4640 1 -- 172.21.15.3:0/997790855 wait complete. 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 Processor -- start 2024-02-09T22:58:08.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 -- start start 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4614100440 0x7f461419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 0x7f461419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 0x7f46141976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4614197e50 con 0x7f46141026c0 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4614197fc0 con 0x7f4614101d20 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461b2a4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4614198180 con 0x7f4614100440 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 0x7f46141976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 0x7f46141976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41154/0 (socket says 172.21.15.3:41154) 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 -- 172.21.15.3:0/351917033 learned_addr learned my addr 172.21.15.3:0/351917033 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461883f640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 0x7f461419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619040640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4614100440 0x7f461419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 -- 172.21.15.3:0/351917033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4614100440 msgr2=0x7f461419d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4614100440 0x7f461419d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 -- 172.21.15.3:0/351917033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 msgr2=0x7f461419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 0x7f461419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f4619841640 1 -- 172.21.15.3:0/351917033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4610009560 con 0x7f46141026c0 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.491+0000 7f461883f640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 0x7f461419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.492+0000 7f4619841640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 0x7f46141976c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f460c00e960 tx=0x7f460c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.492+0000 7f460a7fc640 1 -- 172.21.15.3:0/351917033 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c004d20 con 0x7f46141026c0 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.492+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4614198460 con 0x7f46141026c0 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.492+0000 7f460a7fc640 1 -- 172.21.15.3:0/351917033 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f460c004e80 con 0x7f46141026c0 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.492+0000 7f460a7fc640 1 -- 172.21.15.3:0/351917033 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f460c0116d0 con 0x7f46141026c0 2024-02-09T22:58:08.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.492+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46141a4580 con 0x7f46141026c0 2024-02-09T22:58:08.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.494+0000 7f460a7fc640 1 -- 172.21.15.3:0/351917033 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f460c0052b0 con 0x7f46141026c0 2024-02-09T22:58:08.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 Processor -- start 2024-02-09T22:58:08.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 -- start start 2024-02-09T22:58:08.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563401105a30 0x563401105e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 0x56340110e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x5634011113b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634010c1020 con 0x5634011063e0 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563401074320 con 0x56340110ef60 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461de7f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563401071850 con 0x563401105a30 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f460a7fc640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f46000a2230 0x7f46000a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 0x56340110e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 0x56340110e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41168/0 (socket says 172.21.15.3:41168) 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 -- 172.21.15.3:0/864141722 learned_addr learned my addr 172.21.15.3:0/864141722 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461883f640 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563401105a30 0x563401105e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f461883f640 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x5634011113b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619841640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f46000a2230 0x7f46000a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f460a7fc640 1 -- 172.21.15.3:0/351917033 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f460c0ce390 con 0x7f46141026c0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 -- 172.21.15.3:0/864141722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563401105a30 msgr2=0x563401105e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563401105a30 0x563401105e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 -- 172.21.15.3:0/864141722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 msgr2=0x5634011113b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x5634011113b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.495+0000 7f4619040640 1 -- 172.21.15.3:0/864141722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56340102f210 con 0x5634011063e0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f4619841640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f46000a2230 0x7f46000a46f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f460c0976d0 tx=0x7f460c0903d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f4619040640 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 0x56340110e990 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f4610036040 tx=0x7f4610005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c374fa1f5bd182f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f45ff7fe640 1 -- 172.21.15.3:0/864141722 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4610004a80 con 0x5634011063e0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f45ff7fe640 1 -- 172.21.15.3:0/864141722 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4610004030 con 0x5634011063e0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f45ff7fe640 1 -- 172.21.15.3:0/864141722 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46100052a0 con 0x5634011063e0 2024-02-09T22:58:08.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f461de7f740 1 -- 172.21.15.3:0/864141722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 msgr2=0x56340110e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.496+0000 7f461de7f740 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 0x56340110e990 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f4610036040 tx=0x7f4610005e80 comp rx=0 tx=0).stop 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- 172.21.15.3:0/864141722 shutdown_connections 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x5634011113b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5634011063e0 0x56340110e990 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 --2- 172.21.15.3:0/864141722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563401105a30 0x563401105e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- 172.21.15.3:0/864141722 >> 172.21.15.3:0/864141722 conn(0x563401100100 msgr2=0x5634011004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- 172.21.15.3:0/864141722 shutdown_connections 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- 172.21.15.3:0/864141722 wait complete. 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- start start 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 Processor -- start 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x56340110c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x56340110c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56340106d030 con 0x563401105a30 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563400d40430 con 0x56340110ef60 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563400cfce00 con 0x5634011063e0 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461de7f740 10 client.0 _ll_register_callbacks cb 0x563401062030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461883f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.497+0000 7f461883f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41170/0 (socket says 172.21.15.3:41170) 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 -- 172.21.15.3:0/1638036508 learned_addr learned my addr 172.21.15.3:0/1638036508 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f4619040640 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x56340110c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 -- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 msgr2=0x56340110c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x56340110c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 -- 172.21.15.3:0/1638036508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 msgr2=0x56340110c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x56340110c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 -- 172.21.15.3:0/1638036508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f461410d9c0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461883f640 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110bec0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f4604013fa0 tx=0x7f4604010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be82c153fabcaef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f45fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f45fcff9640 1 -- 172.21.15.3:0/1638036508 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46040132d0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f45fcff9640 1 -- 172.21.15.3:0/1638036508 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f4604005020 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563400d6c7e0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f45fcff9640 1 -- 172.21.15.3:0/1638036508 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46040051e0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461de7f740 10 client.17988 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56340106b6b0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.498+0000 7f45fcff9640 1 -- 172.21.15.3:0/1638036508 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563400d6c7e0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f45fcff9640 1 -- 172.21.15.3:0/1638036508 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f460400f5d0 con 0x563401105a30 2024-02-09T22:58:08.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f461de7f740 10 client.17988 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f461de7f740 10 client.17988 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f461de7f740 20 client.17988 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f461de7f740 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56340111f440 0x563401121860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f461de7f740 4 client.17988 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563400da9720 con 0x56340111f440 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f4619040640 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56340111f440 0x563401121860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f45fcff9640 10 client.17988 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.499+0000 7f4619040640 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56340111f440 0x563401121860 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=69bfca4fb5f8de32 server_cookie=f0417268f07a95ee in_seq=0 out_seq=0 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f45fcff9640 1 -- 172.21.15.3:0/1638036508 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563400da9720 con 0x56340111f440 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f45fcff9640 10 client.17988 handle_command_reply: tid=0 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 client.17988 shutdown 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 msgr2=0x56340110bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110bec0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f4604013fa0 tx=0x7f4604010800 comp rx=0 tx=0).stop 2024-02-09T22:58:08.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 shutdown_connections 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56340111f440 0x563401121860 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x56340110c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x56340110c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f45fcff9640 0 client.17988 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 --2- 172.21.15.3:0/1638036508 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110bec0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 >> 172.21.15.3:0/1638036508 conn(0x563401100100 msgr2=0x563401101e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 shutdown_connections 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 1 -- 172.21.15.3:0/1638036508 wait complete. 2024-02-09T22:58:08.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.500+0000 7f461de7f740 20 client.17988 trim_cache size 0 max 16384 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 Processor -- start 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 -- start start 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401109120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x563401109660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563400ff2130 con 0x563401105a30 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56340110d320 con 0x56340110ef60 2024-02-09T22:58:08.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461de7f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563401105530 con 0x5634011063e0 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41186/0 (socket says 172.21.15.3:41186) 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 -- 172.21.15.3:0/2480262482 learned_addr learned my addr 172.21.15.3:0/2480262482 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f4619040640 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401109120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 -- 172.21.15.3:0/2480262482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 msgr2=0x563401109120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401109120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 -- 172.21.15.3:0/2480262482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 msgr2=0x563401109660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x563401109660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f461883f640 1 -- 172.21.15.3:0/2480262482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563401007c90 con 0x563401105a30 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.512+0000 7f4619040640 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461883f640 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110e5f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f4604001eb0 tx=0x7f460400bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8092ca5c483587c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f45feffd640 1 -- 172.21.15.3:0/2480262482 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604005ec0 con 0x563401105a30 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f45feffd640 1 -- 172.21.15.3:0/2480262482 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4604019070 con 0x563401105a30 2024-02-09T22:58:08.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f45feffd640 1 -- 172.21.15.3:0/2480262482 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4604013330 con 0x563401105a30 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 -- 172.21.15.3:0/2480262482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 msgr2=0x56340110e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110e5f0 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f4604001eb0 tx=0x7f460400bfc0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 -- 172.21.15.3:0/2480262482 shutdown_connections 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56340110ef60 0x563401109660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401109120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 --2- 172.21.15.3:0/2480262482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563401105a30 0x56340110e5f0 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 -- 172.21.15.3:0/2480262482 >> 172.21.15.3:0/2480262482 conn(0x563401100100 msgr2=0x56340110a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 -- 172.21.15.3:0/2480262482 shutdown_connections 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.513+0000 7f461de7f740 1 -- 172.21.15.3:0/2480262482 wait complete. 2024-02-09T22:58:08.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 -- start start 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 Processor -- start 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563401105a30 0x563401115010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401115550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 0x563401175970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563401115c90 con 0x56340110ef60 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563401175eb0 con 0x563401105a30 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563401176040 con 0x5634011063e0 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461de7f740 10 client.0 _ll_register_callbacks cb 0x7f4610009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461883f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 0x563401175970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461883f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 0x563401175970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41202/0 (socket says 172.21.15.3:41202) 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461883f640 1 -- 172.21.15.3:0/2708810731 learned_addr learned my addr 172.21.15.3:0/2708810731 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f461883f640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563401105a30 0x563401115010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.514+0000 7f4619040640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401115550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461883f640 1 -- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 msgr2=0x563401115550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461883f640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401115550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461883f640 1 -- 172.21.15.3:0/2708810731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563401105a30 msgr2=0x563401115010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461883f640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563401105a30 0x563401115010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461883f640 1 -- 172.21.15.3:0/2708810731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56340106b6b0 con 0x56340110ef60 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461883f640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 0x563401175970 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f460400f9e0 tx=0x7f4604021040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f8a16d9c79d919d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f45fcff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f45fcff9640 1 -- 172.21.15.3:0/2708810731 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46040130d0 con 0x56340110ef60 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f45fcff9640 1 -- 172.21.15.3:0/2708810731 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f4604000e40 con 0x56340110ef60 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4610004030 con 0x56340110ef60 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f45fcff9640 1 -- 172.21.15.3:0/2708810731 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4604013230 con 0x56340110ef60 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 10 client.17994 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56340102da70 con 0x56340110ef60 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f45fcff9640 1 -- 172.21.15.3:0/2708810731 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4610004030 con 0x56340110ef60 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f45fcff9640 1 -- 172.21.15.3:0/2708810731 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f460402f030 con 0x56340110ef60 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 10 client.17994 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 10 client.17994 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 20 client.17994 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5634011774a0 0x563401179890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 4 client.17994 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.515+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563400df8890 con 0x5634011774a0 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f4619040640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5634011774a0 0x563401179890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f4619040640 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5634011774a0 0x563401179890 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3d33820d8c718420 server_cookie=d2e39578eb4988c9 in_seq=0 out_seq=0 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f45fcff9640 10 client.17994 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f45fcff9640 1 -- 172.21.15.3:0/2708810731 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563400df8890 con 0x5634011774a0 2024-02-09T22:58:08.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f45fcff9640 10 client.17994 handle_command_reply: tid=0 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 client.17994 shutdown 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 msgr2=0x563401175970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 0x563401175970 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f460400f9e0 tx=0x7f4604021040 comp rx=0 tx=0).stop 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 shutdown_connections 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5634011774a0 0x563401179890 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56340110ef60 0x563401175970 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5634011063e0 0x563401115550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 --2- 172.21.15.3:0/2708810731 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563401105a30 0x563401115010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f45fcff9640 0 client.17994 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 >> 172.21.15.3:0/2708810731 conn(0x563401100100 msgr2=0x5634010fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 shutdown_connections 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.516+0000 7f461de7f740 1 -- 172.21.15.3:0/2708810731 wait complete. 2024-02-09T22:58:08.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461de7f740 20 client.17994 trim_cache size 0 max 16384 2024-02-09T22:58:08.527 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:08.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f46000a2230 msgr2=0x7f46000a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f46000a2230 0x7f46000a46f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f460c0976d0 tx=0x7f460c0903d0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 msgr2=0x7f46141976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 0x7f46141976c0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7f460c00e960 tx=0x7f460c00ee20 comp rx=0 tx=0).stop 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 shutdown_connections 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f46000a2230 0x7f46000a46f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4614101d20 0x7f461419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f46141026c0 0x7f46141976c0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 --2- 172.21.15.3:0/351917033 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4614100440 0x7f461419d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 >> 172.21.15.3:0/351917033 conn(0x7f4614077310 msgr2=0x7f4614069bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 shutdown_connections 2024-02-09T22:58:08.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.517+0000 7f461b2a4640 1 -- 172.21.15.3:0/351917033 wait complete. 2024-02-09T22:58:08.529 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:08 smithi003 ceph-mon[32678]: pgmap v210: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:58:08.530 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:08.531 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- 172.21.15.3:0/364542545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f010b820 msgr2=0x7f53f010dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- 172.21.15.3:0/364542545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f010b820 0x7f53f010dc50 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009ca0 tx=0x7f53e002c180 comp rx=0 tx=0).stop 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- 172.21.15.3:0/364542545 shutdown_connections 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- 172.21.15.3:0/364542545 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f010b820 0x7f53f010dc50 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- 172.21.15.3:0/364542545 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f0102d70 0x7f53f010b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- 172.21.15.3:0/364542545 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102450 0x7f53f0102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- 172.21.15.3:0/364542545 >> 172.21.15.3:0/364542545 conn(0x7f53f00772e0 msgr2=0x7f53f00776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- 172.21.15.3:0/364542545 shutdown_connections 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- 172.21.15.3:0/364542545 wait complete. 2024-02-09T22:58:08.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 Processor -- start 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- start start 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 0x7f53f019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102d70 0x7f53f019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f010b820 0x7f53f0197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0197dc0 con 0x7f53f0102450 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0197f30 con 0x7f53f0102d70 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f80eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f01980f0 con 0x7f53f010b820 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.633+0000 7f53f5e87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 0x7f53f019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 0x7f53f019d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41214/0 (socket says 172.21.15.3:41214) 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 -- 172.21.15.3:0/886732676 learned_addr learned my addr 172.21.15.3:0/886732676 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f6688640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f010b820 0x7f53f0197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5686640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102d70 0x7f53f019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 -- 172.21.15.3:0/886732676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f010b820 msgr2=0x7f53f0197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f010b820 0x7f53f0197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 -- 172.21.15.3:0/886732676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102d70 msgr2=0x7f53f019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102d70 0x7f53f019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 -- 172.21.15.3:0/886732676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0009900 con 0x7f53f0102450 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5686640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102d70 0x7f53f019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f6688640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f010b820 0x7f53f0197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f5e87640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 0x7f53f019d570 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f53e800d950 tx=0x7f53e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53eeffd640 1 -- 172.21.15.3:0/886732676 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e8009c30 con 0x7f53f0102450 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53eeffd640 1 -- 172.21.15.3:0/886732676 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53e8009d90 con 0x7f53f0102450 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f01983d0 con 0x7f53f0102450 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53eeffd640 1 -- 172.21.15.3:0/886732676 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e800f7e0 con 0x7f53f0102450 2024-02-09T22:58:08.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.634+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f01a44f0 con 0x7f53f0102450 2024-02-09T22:58:08.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.636+0000 7f53eeffd640 1 -- 172.21.15.3:0/886732676 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f53e8010ad0 con 0x7f53f0102450 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 Processor -- start 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 -- start start 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193ce5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193cee990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ceef60 0x558193cf13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193ca1020 con 0x558193ce5a30 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193c54320 con 0x558193ceef60 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53fad96740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193c51850 con 0x558193ce63e0 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53eeffd640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53d80a2230 0x7f53d80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53f6688640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193ce5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53f6688640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193ce5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41220/0 (socket says 172.21.15.3:41220) 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53f6688640 1 -- 172.21.15.3:0/637228094 learned_addr learned my addr 172.21.15.3:0/637228094 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53f6688640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53d80a2230 0x7f53d80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53f5e87640 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193cee990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53f5686640 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ceef60 0x558193cf13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.637+0000 7f53eeffd640 1 -- 172.21.15.3:0/886732676 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f53e80cde40 con 0x7f53f0102450 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 -- 172.21.15.3:0/637228094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 msgr2=0x558193cee990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193cee990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 -- 172.21.15.3:0/637228094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ceef60 msgr2=0x558193cf13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ceef60 0x558193cf13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 -- 172.21.15.3:0/637228094 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558193c0f210 con 0x558193ce5a30 2024-02-09T22:58:08.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193ce5e10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f53e0005fe0 tx=0x7f53e0003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cecfd720232e7dd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53f6688640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53d80a2230 0x7f53d80a46f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009ca0 tx=0x7f53e0048000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53d7fff640 1 -- 172.21.15.3:0/637228094 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0032550 con 0x558193ce5a30 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53d7fff640 1 -- 172.21.15.3:0/637228094 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53e0003e90 con 0x558193ce5a30 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.638+0000 7f53d7fff640 1 -- 172.21.15.3:0/637228094 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0004520 con 0x558193ce5a30 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 -- 172.21.15.3:0/637228094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 msgr2=0x558193ce5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193ce5e10 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f53e0005fe0 tx=0x7f53e0003b20 comp rx=0 tx=0).stop 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 -- 172.21.15.3:0/637228094 shutdown_connections 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ceef60 0x558193cf13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193cee990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 --2- 172.21.15.3:0/637228094 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193ce5e10 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 -- 172.21.15.3:0/637228094 >> 172.21.15.3:0/637228094 conn(0x558193ce0100 msgr2=0x558193ce04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 -- 172.21.15.3:0/637228094 shutdown_connections 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 -- 172.21.15.3:0/637228094 wait complete. 2024-02-09T22:58:08.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 -- start start 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.639+0000 7f53fad96740 1 Processor -- start 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce5a30 0x558193cebfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193ceca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193c4d030 con 0x558193ceef60 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193920430 con 0x558193ce63e0 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581938dce00 con 0x558193ce5a30 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53fad96740 10 client.0 _ll_register_callbacks cb 0x558193c42030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193ceca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193ceca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41222/0 (socket says 172.21.15.3:41222) 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 -- 172.21.15.3:0/1647949483 learned_addr learned my addr 172.21.15.3:0/1647949483 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f5e87640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce5a30 0x558193cebfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f5686640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 -- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce5a30 msgr2=0x558193cebfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce5a30 0x558193cebfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 -- 172.21.15.3:0/1647949483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 msgr2=0x558193cec500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 -- 172.21.15.3:0/1647949483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f010e8d0 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f5686640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53f6688640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193ceca40 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f53e002c790 tx=0x7f53e0009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27d0b11f4d2e203 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.640+0000 7f53d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53d57fa640 1 -- 172.21.15.3:0/1647949483 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0034600 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53d57fa640 1 -- 172.21.15.3:0/1647949483 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f53e0004040 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55819394c7e0 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53d57fa640 1 -- 172.21.15.3:0/1647949483 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53e0004200 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 10 client.18006 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x558193c4b6b0 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53d57fa640 1 -- 172.21.15.3:0/1647949483 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55819394c7e0 con 0x558193ceef60 2024-02-09T22:58:08.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53d57fa640 1 -- 172.21.15.3:0/1647949483 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f53e003e040 con 0x558193ceef60 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 10 client.18006 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 10 client.18006 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 20 client.18006 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d01750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 4 client.18006 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.641+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558193989720 con 0x558193cff330 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.642+0000 7f53f5e87640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d01750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.642+0000 7f53f5e87640 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d01750 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=48356c3635d23a46 server_cookie=de4161a933e36bf7 in_seq=0 out_seq=0 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.642+0000 7f53d57fa640 10 client.18006 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53d57fa640 1 -- 172.21.15.3:0/1647949483 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558193989720 con 0x558193cff330 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53d57fa640 10 client.18006 handle_command_reply: tid=0 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 client.18006 shutdown 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 msgr2=0x558193ceca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193ceca40 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f53e002c790 tx=0x7f53e0009540 comp rx=0 tx=0).stop 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 shutdown_connections 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d01750 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193ceca40 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53d57fa640 0 client.18006 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 --2- 172.21.15.3:0/1647949483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce5a30 0x558193cebfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 >> 172.21.15.3:0/1647949483 conn(0x558193ce0100 msgr2=0x558193ce1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195887 2.61 MB/sec execute 242 sec latency 4966.830 ms 2024-02-09T22:58:08.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 shutdown_connections 2024-02-09T22:58:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 1 -- 172.21.15.3:0/1647949483 wait complete. 2024-02-09T22:58:08.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.643+0000 7f53fad96740 20 client.18006 trim_cache size 0 max 16384 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 Processor -- start 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 -- start start 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193cee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ceef60 0x558193ce9260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193bd2130 con 0x558193ce5a30 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193cf1870 con 0x558193ce63e0 2024-02-09T22:58:08.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53fad96740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193ce4ac0 con 0x558193ceef60 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193cee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193cee6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41230/0 (socket says 172.21.15.3:41230) 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 -- 172.21.15.3:0/550011581 learned_addr learned my addr 172.21.15.3:0/550011581 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5686640 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f6688640 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ceef60 0x558193ce9260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 -- 172.21.15.3:0/550011581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ceef60 msgr2=0x558193ce9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ceef60 0x558193ce9260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 -- 172.21.15.3:0/550011581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 msgr2=0x558193cec860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 -- 172.21.15.3:0/550011581 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558193be7c90 con 0x558193ce5a30 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5686640 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.655+0000 7f53f5e87640 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193cee6f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f53e8096c10 tx=0x7f53e8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6103c9567b20e09d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53d77fe640 1 -- 172.21.15.3:0/550011581 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e80a2f50 con 0x558193ce5a30 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53d77fe640 1 -- 172.21.15.3:0/550011581 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53e809a7b0 con 0x558193ce5a30 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53d77fe640 1 -- 172.21.15.3:0/550011581 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e80851d0 con 0x558193ce5a30 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 -- 172.21.15.3:0/550011581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 msgr2=0x558193cee6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193cee6f0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f53e8096c10 tx=0x7f53e8007f90 comp rx=0 tx=0).stop 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 -- 172.21.15.3:0/550011581 shutdown_connections 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce63e0 0x558193cec860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ceef60 0x558193ce9260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 --2- 172.21.15.3:0/550011581 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ce5a30 0x558193cee6f0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 -- 172.21.15.3:0/550011581 >> 172.21.15.3:0/550011581 conn(0x558193ce0100 msgr2=0x558193d060b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 -- 172.21.15.3:0/550011581 shutdown_connections 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 -- 172.21.15.3:0/550011581 wait complete. 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 -- start start 2024-02-09T22:58:08.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.656+0000 7f53fad96740 1 Processor -- start 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce5a30 0x558193d500a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193d505e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193cf55c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193cf5b80 con 0x558193ceef60 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193cf5cf0 con 0x558193ce5a30 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558193cf5eb0 con 0x558193ce63e0 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53fad96740 10 client.0 _ll_register_callbacks cb 0x7f53f006aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193cf55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193cf55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41236/0 (socket says 172.21.15.3:41236) 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 -- 172.21.15.3:0/251039326 learned_addr learned my addr 172.21.15.3:0/251039326 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f5e87640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce5a30 0x558193d500a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 -- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 msgr2=0x558193d505e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193d505e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 -- 172.21.15.3:0/251039326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce5a30 msgr2=0x558193d500a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce5a30 0x558193d500a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 -- 172.21.15.3:0/251039326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f01a44f0 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f5e87640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce5a30 0x558193d500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53f6688640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193cf55c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f53e0044de0 tx=0x7f53e002c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a543bb5b6e76b1a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53d57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.657+0000 7f53d57fa640 1 -- 172.21.15.3:0/251039326 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53e0004240 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53d57fa640 1 -- 172.21.15.3:0/251039326 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f53e00043a0 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f53e0003e90 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53d57fa640 1 -- 172.21.15.3:0/251039326 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53e0005cf0 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 10 client.18012 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x558193c0da70 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53d57fa640 1 -- 172.21.15.3:0/251039326 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53e0003e90 con 0x558193ceef60 2024-02-09T22:58:08.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53d57fa640 1 -- 172.21.15.3:0/251039326 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f53e004c640 con 0x558193ceef60 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 10 client.18012 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 10 client.18012 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 20 client.18012 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d59b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 4 client.18012 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5581939d8890 con 0x558193cff330 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.658+0000 7f53f5e87640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d59b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53d57fa640 10 client.18012 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53f5e87640 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d59b30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f0eaf6f9e067ee4 server_cookie=1cd6e7ac4eac9c99 in_seq=0 out_seq=0 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53d57fa640 1 -- 172.21.15.3:0/251039326 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5581939d8890 con 0x558193cff330 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53d57fa640 10 client.18012 handle_command_reply: tid=0 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 client.18012 shutdown 2024-02-09T22:58:08.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 msgr2=0x558193cf55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193cf55c0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 crypto rx=0x7f53e0044de0 tx=0x7f53e002c880 comp rx=0 tx=0).stop 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 shutdown_connections 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x558193cff330 0x558193d59b30 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558193ceef60 0x558193cf55c0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558193ce63e0 0x558193d505e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 --2- 172.21.15.3:0/251039326 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558193ce5a30 0x558193d500a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53d57fa640 0 client.18012 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 >> 172.21.15.3:0/251039326 conn(0x558193ce0100 msgr2=0x558193ce1530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 shutdown_connections 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 1 -- 172.21.15.3:0/251039326 wait complete. 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.659+0000 7f53fad96740 20 client.18012 trim_cache size 0 max 16384 2024-02-09T22:58:08.669 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53d80a2230 msgr2=0x7f53d80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53d80a2230 0x7f53d80a46f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f53e0009ca0 tx=0x7f53e0048000 comp rx=0 tx=0).stop 2024-02-09T22:58:08.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 msgr2=0x7f53f019d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 0x7f53f019d570 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7f53e800d950 tx=0x7f53e800de10 comp rx=0 tx=0).stop 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 shutdown_connections 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53d80a2230 0x7f53d80a46f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53f0102d70 0x7f53f019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53f010b820 0x7f53f0197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 --2- 172.21.15.3:0/886732676 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53f0102450 0x7f53f019d570 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 >> 172.21.15.3:0/886732676 conn(0x7f53f00772e0 msgr2=0x7f53f00fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 shutdown_connections 2024-02-09T22:58:08.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.660+0000 7f53f80eb640 1 -- 172.21.15.3:0/886732676 wait complete. 2024-02-09T22:58:08.673 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:08.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:58:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:08 smithi064 ceph-mon[36638]: pgmap v210: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:58:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:08 smithi089 ceph-mon[36665]: pgmap v210: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 -- 172.21.15.3:0/1888511295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c4106560 msgr2=0x7fd6c410e150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/1888511295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c4106560 0x7fd6c410e150 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc009cd0 tx=0x7fd6bc02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 -- 172.21.15.3:0/1888511295 shutdown_connections 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/1888511295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c4106560 0x7fd6c410e150 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/1888511295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c4100750 0x7fd6c4105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/1888511295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6c40ffda0 0x7fd6c4100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 -- 172.21.15.3:0/1888511295 >> 172.21.15.3:0/1888511295 conn(0x7fd6c4075380 msgr2=0x7fd6c4075790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 -- 172.21.15.3:0/1888511295 shutdown_connections 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.776+0000 7fd6cacc0640 1 -- 172.21.15.3:0/1888511295 wait complete. 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 Processor -- start 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 -- start start 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6c40ffda0 0x7fd6c419d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c4106560 0x7fd6c419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 0x7fd6c4197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41981c0 con 0x7fd6c41976f0 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c4198330 con 0x7fd6c40ffda0 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6cacc0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6c41984f0 con 0x7fd6c4106560 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 0x7fd6c4197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 0x7fd6c4197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41248/0 (socket says 172.21.15.3:41248) 2024-02-09T22:58:08.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 -- 172.21.15.3:0/463375227 learned_addr learned my addr 172.21.15.3:0/463375227 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6c40ffda0 0x7fd6c419d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c4106560 0x7fd6c419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 -- 172.21.15.3:0/463375227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c4106560 msgr2=0x7fd6c419db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c4106560 0x7fd6c419db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 -- 172.21.15.3:0/463375227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6c40ffda0 msgr2=0x7fd6c419d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6c40ffda0 0x7fd6c419d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 -- 172.21.15.3:0/463375227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6bc009900 con 0x7fd6c41976f0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.777+0000 7fd6c925d640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 0x7fd6c4197b70 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0348f0 tx=0x7fd6bc034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.778+0000 7fd6c1ffb640 1 -- 172.21.15.3:0/463375227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc0328c0 con 0x7fd6c41976f0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.778+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6c41a4430 con 0x7fd6c41976f0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.778+0000 7fd6c1ffb640 1 -- 172.21.15.3:0/463375227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6bc003e50 con 0x7fd6c41976f0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.778+0000 7fd6c1ffb640 1 -- 172.21.15.3:0/463375227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6bc040450 con 0x7fd6c41976f0 2024-02-09T22:58:08.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.778+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6c41a4900 con 0x7fd6c41976f0 2024-02-09T22:58:08.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.780+0000 7fd6c1ffb640 1 -- 172.21.15.3:0/463375227 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd6bc00a040 con 0x7fd6c41976f0 2024-02-09T22:58:08.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.780+0000 7fd6cd96a740 1 Processor -- start 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.780+0000 7fd6cd96a740 1 -- start start 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa8605c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa860ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa860f820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa85c1020 con 0x55faa8606240 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa8574320 con 0x55faa8605890 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa8571850 con 0x55faa860f3a0 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c1ffb640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6b40a2230 0x7fd6b40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa860ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa860ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41262/0 (socket says 172.21.15.3:41262) 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/3221839633 learned_addr learned my addr 172.21.15.3:0/3221839633 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c925d640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6b40a2230 0x7fd6b40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa860f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c1ffb640 1 -- 172.21.15.3:0/463375227 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fd6bc0f0a60 con 0x7fd6c41976f0 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa8605c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/3221839633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 msgr2=0x55faa860f820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa860f820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/3221839633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 msgr2=0x55faa8605c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa8605c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/3221839633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faa852f210 con 0x55faa8606240 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa8605c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa860ed80 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bd10 tx=0x7fd6b80107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491d226a0b09823c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.781+0000 7fd6c925d640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6b40a2230 0x7fd6b40a46f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0ee790 tx=0x7fd6bc0095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6aeffd640 1 -- 172.21.15.3:0/3221839633 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8004030 con 0x55faa8606240 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6aeffd640 1 -- 172.21.15.3:0/3221839633 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6b8015070 con 0x55faa8606240 2024-02-09T22:58:08.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6aeffd640 1 -- 172.21.15.3:0/3221839633 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8011640 con 0x55faa8606240 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 -- 172.21.15.3:0/3221839633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 msgr2=0x55faa860ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa860ed80 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bd10 tx=0x7fd6b80107a0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 -- 172.21.15.3:0/3221839633 shutdown_connections 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa860f820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa860ed80 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/3221839633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa8605c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 -- 172.21.15.3:0/3221839633 >> 172.21.15.3:0/3221839633 conn(0x55faa8600100 msgr2=0x55faa86004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 -- 172.21.15.3:0/3221839633 shutdown_connections 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.782+0000 7fd6cd96a740 1 -- 172.21.15.3:0/3221839633 wait complete. 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 -- start start 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 Processor -- start 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 0x55faa8612e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8606240 0x55faa86133b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa86138f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa856d030 con 0x55faa8605890 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa8240430 con 0x55faa8606240 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa81fce00 con 0x55faa860f3a0 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6cd96a740 10 client.0 _ll_register_callbacks cb 0x55faa8562030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 0x55faa8612e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 0x55faa8612e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41278/0 (socket says 172.21.15.3:41278) 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c3fff640 1 -- 172.21.15.3:0/1432239722 learned_addr learned my addr 172.21.15.3:0/1432239722 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa86138f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8606240 0x55faa86133b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c3fff640 1 -- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 msgr2=0x55faa86138f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.783+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa86138f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6c3fff640 1 -- 172.21.15.3:0/1432239722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8606240 msgr2=0x55faa86133b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8606240 0x55faa86133b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6c3fff640 1 -- 172.21.15.3:0/1432239722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b800b940 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8606240 0x55faa86133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 0x55faa8612e70 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000bf80 tx=0x7fd6b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ef096a76ebf45f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6a3fff640 1 -- 172.21.15.3:0/1432239722 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b0017040 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6a3fff640 1 -- 172.21.15.3:0/1432239722 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fd6b00140d0 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55faa826c7e0 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6a3fff640 1 -- 172.21.15.3:0/1432239722 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6b0014290 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 10 client.18024 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55faa856b6b0 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6a3fff640 1 -- 172.21.15.3:0/1432239722 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55faa826c7e0 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6a3fff640 1 -- 172.21.15.3:0/1432239722 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd6b0012890 con 0x55faa8605890 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 10 client.18024 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 10 client.18024 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 20 client.18024 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa8623840 0x55faa8625c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 4 client.18024 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.784+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55faa82a9720 con 0x55faa8623840 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.785+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa8623840 0x55faa8625c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.785+0000 7fd6a3fff640 10 client.18024 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.785+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa8623840 0x55faa8625c80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=22068f19b1201012 server_cookie=4fdcd1c0952b7ba9 in_seq=0 out_seq=0 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6a3fff640 1 -- 172.21.15.3:0/1432239722 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55faa82a9720 con 0x55faa8623840 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6a3fff640 10 client.18024 handle_command_reply: tid=0 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 client.18024 shutdown 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 msgr2=0x55faa8612e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 0x55faa8612e70 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fd6b000bf80 tx=0x7fd6b00046c0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 shutdown_connections 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa8623840 0x55faa8625c80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8606240 0x55faa86133b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa86138f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6a3fff640 0 client.18024 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1432239722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8605890 0x55faa8612e70 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 >> 172.21.15.3:0/1432239722 conn(0x55faa8600100 msgr2=0x55faa8601e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 shutdown_connections 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1432239722 wait complete. 2024-02-09T22:58:08.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.786+0000 7fd6cd96a740 20 client.18024 trim_cache size 0 max 16384 2024-02-09T22:58:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 Processor -- start 2024-02-09T22:58:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 -- start start 2024-02-09T22:58:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa8605890 0x55faa8615550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8610790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa860f3a0 0x55faa8611760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa84f2130 con 0x55faa8606240 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa826c7e0 con 0x55faa860f3a0 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa8604d20 con 0x55faa8605890 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8610790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8610790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41288/0 (socket says 172.21.15.3:41288) 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/1729389679 learned_addr learned my addr 172.21.15.3:0/1729389679 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa860f3a0 0x55faa8611760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa8605890 0x55faa8615550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/1729389679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa8605890 msgr2=0x55faa8615550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa8605890 0x55faa8615550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/1729389679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa860f3a0 msgr2=0x55faa8611760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa860f3a0 0x55faa8611760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/1729389679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55faa8507c90 con 0x55faa8606240 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.798+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa860f3a0 0x55faa8611760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8610790 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bd10 tx=0x7fd6b8003d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e57b5536724e7e2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6ae7fc640 1 -- 172.21.15.3:0/1729389679 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8021040 con 0x55faa8606240 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6ae7fc640 1 -- 172.21.15.3:0/1729389679 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6b8005040 con 0x55faa8606240 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6ae7fc640 1 -- 172.21.15.3:0/1729389679 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b8027340 con 0x55faa8606240 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1729389679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 msgr2=0x55faa8610790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8610790 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bd10 tx=0x7fd6b8003d10 comp rx=0 tx=0).stop 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1729389679 shutdown_connections 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa860f3a0 0x55faa8611760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8610790 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/1729389679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa8605890 0x55faa8615550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1729389679 >> 172.21.15.3:0/1729389679 conn(0x55faa8600100 msgr2=0x55faa860cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1729389679 shutdown_connections 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.799+0000 7fd6cd96a740 1 -- 172.21.15.3:0/1729389679 wait complete. 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 -- start start 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 Processor -- start 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa86190f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8619630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa8619b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa8679ee0 con 0x55faa8606240 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa867a050 con 0x55faa8605890 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55faa867a210 con 0x55faa860f3a0 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6cd96a740 10 client.0 _ll_register_callbacks cb 0x7fd6b800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8619630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8619630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41302/0 (socket says 172.21.15.3:41302) 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/622558828 learned_addr learned my addr 172.21.15.3:0/622558828 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa8619b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 msgr2=0x55faa8619b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa8619b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/622558828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 msgr2=0x55faa86190f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa86190f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.800+0000 7fd6c8a5c640 1 -- 172.21.15.3:0/622558828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55faa852f210 con 0x55faa8606240 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6c8a5c640 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8619630 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bce0 tx=0x7fd6b8005390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=324eddf2e6603825 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6a3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6a3fff640 1 -- 172.21.15.3:0/622558828 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6b801f070 con 0x55faa8606240 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6a3fff640 1 -- 172.21.15.3:0/622558828 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fd6b8017070 con 0x55faa8606240 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6b8015070 con 0x55faa8606240 2024-02-09T22:58:08.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6a3fff640 1 -- 172.21.15.3:0/622558828 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6b8021050 con 0x55faa8606240 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 10 client.18030 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55faa852da70 con 0x55faa8606240 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6a3fff640 1 -- 172.21.15.3:0/622558828 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6b8042020 con 0x55faa8606240 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6a3fff640 1 -- 172.21.15.3:0/622558828 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd6b803e480 con 0x55faa8606240 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 10 client.18030 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 10 client.18030 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 20 client.18030 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa867b720 0x55faa867db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 4 client.18030 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.801+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55faa82f8890 con 0x55faa867b720 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa867b720 0x55faa867db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6c3fff640 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa867b720 0x55faa867db40 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=37278ec4af7ecfd2 server_cookie=2d7e32bbba714ea9 in_seq=0 out_seq=0 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6a3fff640 10 client.18030 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:08.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6a3fff640 1 -- 172.21.15.3:0/622558828 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55faa82f8890 con 0x55faa867b720 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6a3fff640 10 client.18030 handle_command_reply: tid=0 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 client.18030 shutdown 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 msgr2=0x55faa8619630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8619630 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fd6b800bce0 tx=0x7fd6b8005390 comp rx=0 tx=0).stop 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 shutdown_connections 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55faa867b720 0x55faa867db40 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55faa860f3a0 0x55faa8619b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55faa8606240 0x55faa8619630 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6a3fff640 0 client.18030 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 --2- 172.21.15.3:0/622558828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55faa8605890 0x55faa86190f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 >> 172.21.15.3:0/622558828 conn(0x55faa8600100 msgr2=0x55faa8611700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 shutdown_connections 2024-02-09T22:58:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.802+0000 7fd6cd96a740 1 -- 172.21.15.3:0/622558828 wait complete. 2024-02-09T22:58:08.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cd96a740 20 client.18030 trim_cache size 0 max 16384 2024-02-09T22:58:08.814 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6b40a2230 msgr2=0x7fd6b40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6b40a2230 0x7fd6b40a46f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0ee790 tx=0x7fd6bc0095c0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 msgr2=0x7fd6c4197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 0x7fd6c4197b70 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fd6bc0348f0 tx=0x7fd6bc034cc0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 shutdown_connections 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd6b40a2230 0x7fd6b40a46f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd6c41976f0 0x7fd6c4197b70 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd6c4106560 0x7fd6c419db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 --2- 172.21.15.3:0/463375227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd6c40ffda0 0x7fd6c419d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 >> 172.21.15.3:0/463375227 conn(0x7fd6c4075380 msgr2=0x7fd6c40fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 shutdown_connections 2024-02-09T22:58:08.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.803+0000 7fd6cacc0640 1 -- 172.21.15.3:0/463375227 wait complete. 2024-02-09T22:58:08.816 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:08.817 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- 172.21.15.3:0/327461511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c100cf0 msgr2=0x7f679c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- 172.21.15.3:0/327461511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c100cf0 0x7f679c104d30 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7f678c009cd0 tx=0x7f678c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- 172.21.15.3:0/327461511 shutdown_connections 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- 172.21.15.3:0/327461511 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c105410 0x7f679c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- 172.21.15.3:0/327461511 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c100cf0 0x7f679c104d30 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- 172.21.15.3:0/327461511 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f679c100340 0x7f679c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- 172.21.15.3:0/327461511 >> 172.21.15.3:0/327461511 conn(0x7f679c0772e0 msgr2=0x7f679c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- 172.21.15.3:0/327461511 shutdown_connections 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- 172.21.15.3:0/327461511 wait complete. 2024-02-09T22:58:08.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 Processor -- start 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- start start 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f679c100340 0x7f679c19c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c105410 0x7f679c19c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 0x7f679c196a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c196fa0 con 0x7f679c1965b0 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c197110 con 0x7f679c100340 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.919+0000 7f67a31e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c1972d0 con 0x7f679c105410 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 0x7f679c196a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 0x7f679c196a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41320/0 (socket says 172.21.15.3:41320) 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 -- 172.21.15.3:0/1509161818 learned_addr learned my addr 172.21.15.3:0/1509161818 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f679c100340 0x7f679c19c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f679bfff640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c105410 0x7f679c19c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 -- 172.21.15.3:0/1509161818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c105410 msgr2=0x7f679c19c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c105410 0x7f679c19c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 -- 172.21.15.3:0/1509161818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f679c100340 msgr2=0x7f679c19c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f679c100340 0x7f679c19c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 -- 172.21.15.3:0/1509161818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678c009900 con 0x7f679c1965b0 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a177f640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 0x7f679c196a60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f678400d680 tx=0x7f678400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f6799ffb640 1 -- 172.21.15.3:0/1509161818 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784005220 con 0x7f679c1965b0 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f6799ffb640 1 -- 172.21.15.3:0/1509161818 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6784004c80 con 0x7f679c1965b0 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c1a32c0 con 0x7f679c1965b0 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f6799ffb640 1 -- 172.21.15.3:0/1509161818 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6784009db0 con 0x7f679c1965b0 2024-02-09T22:58:08.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.920+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c1a37a0 con 0x7f679c1965b0 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f6799ffb640 1 -- 172.21.15.3:0/1509161818 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f67840054a0 con 0x7f679c1965b0 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 Processor -- start 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 -- start start 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f6799ffb640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67880a2230 0x7f67880a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442eba30 0x561c442ebe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442f73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c442a7020 con 0x561c442ec3e0 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c4425a320 con 0x561c442eba30 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.923+0000 7f67a5db4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c44257850 con 0x561c442f4f60 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67880a2230 0x7f67880a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f6799ffb640 1 -- 172.21.15.3:0/1509161818 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f67840cd8c0 con 0x7f679c1965b0 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41332/0 (socket says 172.21.15.3:41332) 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 -- 172.21.15.3:0/1277803694 learned_addr learned my addr 172.21.15.3:0/1277803694 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f67a177f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442f73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442eba30 0x561c442ebe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 -- 172.21.15.3:0/1277803694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 msgr2=0x561c442f73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442f73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 -- 172.21.15.3:0/1277803694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442eba30 msgr2=0x561c442ebe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442eba30 0x561c442ebe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 -- 172.21.15.3:0/1277803694 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c44215210 con 0x561c442ec3e0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442eba30 0x561c442ebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f67a177f640 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67880a2230 0x7f67880a46f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f678c004af0 tx=0x7f678c032f60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f679bfff640 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f4990 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f679c06a920 tx=0x7f678c048000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=580c24992a3e81c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f6792ffd640 1 -- 172.21.15.3:0/1277803694 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c048070 con 0x561c442ec3e0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f6792ffd640 1 -- 172.21.15.3:0/1277803694 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f678c003e50 con 0x561c442ec3e0 2024-02-09T22:58:08.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.924+0000 7f6792ffd640 1 -- 172.21.15.3:0/1277803694 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c004040 con 0x561c442ec3e0 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 -- 172.21.15.3:0/1277803694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 msgr2=0x561c442f4990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f4990 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7f679c06a920 tx=0x7f678c048000 comp rx=0 tx=0).stop 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 -- 172.21.15.3:0/1277803694 shutdown_connections 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442f73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f4990 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 --2- 172.21.15.3:0/1277803694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442eba30 0x561c442ebe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 -- 172.21.15.3:0/1277803694 >> 172.21.15.3:0/1277803694 conn(0x561c442e6100 msgr2=0x561c442e64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 -- 172.21.15.3:0/1277803694 shutdown_connections 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 -- 172.21.15.3:0/1277803694 wait complete. 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 -- start start 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.925+0000 7f67a5db4740 1 Processor -- start 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442eba30 0x561c442f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442f2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c44253030 con 0x561c442ec3e0 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c43f26430 con 0x561c442f4f60 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c43ee2e00 con 0x561c442eba30 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a5db4740 10 client.0 _ll_register_callbacks cb 0x561c44248030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41346/0 (socket says 172.21.15.3:41346) 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 -- 172.21.15.3:0/3607283889 learned_addr learned my addr 172.21.15.3:0/3607283889 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a177f640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442f2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442eba30 0x561c442f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 -- 172.21.15.3:0/3607283889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442eba30 msgr2=0x561c442f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442eba30 0x561c442f1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 -- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 msgr2=0x561c442f2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442f2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 -- 172.21.15.3:0/3607283889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c10d7f0 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f67a177f640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f679bfff640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f24c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f678c037040 tx=0x7f678c02c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=919563898720831f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f677bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.926+0000 7f677bfff640 1 -- 172.21.15.3:0/3607283889 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c004200 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f677bfff640 1 -- 172.21.15.3:0/3607283889 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f678c004360 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561c43f527e0 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f677bfff640 1 -- 172.21.15.3:0/3607283889 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f678c032b40 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 10 client.18042 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x561c442516b0 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f677bfff640 1 -- 172.21.15.3:0/3607283889 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561c43f527e0 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f677bfff640 1 -- 172.21.15.3:0/3607283889 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f678c043430 con 0x561c442ec3e0 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 10 client.18042 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 10 client.18042 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 20 client.18042 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c44307520 0x561c44309940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 4 client.18042 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561c43f8f720 con 0x561c44307520 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.927+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c44307520 0x561c44309940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.928+0000 7f677bfff640 10 client.18042 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.928+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c44307520 0x561c44309940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=87b3bf0cb4baf5da server_cookie=6f6eb90ddeb5210 in_seq=0 out_seq=0 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.928+0000 7f677bfff640 1 -- 172.21.15.3:0/3607283889 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561c43f8f720 con 0x561c44307520 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.928+0000 7f677bfff640 10 client.18042 handle_command_reply: tid=0 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.928+0000 7f67a5db4740 1 client.18042 shutdown 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 msgr2=0x561c442f24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f24c0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f678c037040 tx=0x7f678c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 shutdown_connections 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c44307520 0x561c44309940 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442f2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442ec3e0 0x561c442f24c0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3607283889 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442eba30 0x561c442f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f677bfff640 0 client.18042 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 >> 172.21.15.3:0/3607283889 conn(0x561c442e6100 msgr2=0x561c442e7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 shutdown_connections 2024-02-09T22:58:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 1 -- 172.21.15.3:0/3607283889 wait complete. 2024-02-09T22:58:08.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.929+0000 7f67a5db4740 20 client.18042 trim_cache size 0 max 16384 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.940+0000 7f67a5db4740 1 Processor -- start 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.940+0000 7f67a5db4740 1 -- start start 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442f2510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442ec3e0 0x561c442f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442ef240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a5db4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c441d8130 con 0x561c442eba30 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a5db4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c442f2f90 con 0x561c442ec3e0 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a5db4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c44304270 con 0x561c442f4f60 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41348/0 (socket says 172.21.15.3:41348) 2024-02-09T22:58:08.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 -- 172.21.15.3:0/2473804313 learned_addr learned my addr 172.21.15.3:0/2473804313 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a177f640 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442ef240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 -- 172.21.15.3:0/2473804313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 msgr2=0x561c442ef240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442ef240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 -- 172.21.15.3:0/2473804313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442ec3e0 msgr2=0x561c442f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f679bfff640 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442ec3e0 0x561c442f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442ec3e0 0x561c442f2a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 -- 172.21.15.3:0/2473804313 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561c441edc90 con 0x561c442eba30 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442f2510 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x561c4430a1e0 tx=0x7f6794013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d377aa975cee32d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67927fc640 1 -- 172.21.15.3:0/2473804313 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794017040 con 0x561c442eba30 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67927fc640 1 -- 172.21.15.3:0/2473804313 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6794020030 con 0x561c442eba30 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.941+0000 7f67927fc640 1 -- 172.21.15.3:0/2473804313 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794004a00 con 0x561c442eba30 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 -- 172.21.15.3:0/2473804313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 msgr2=0x561c442f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442f2510 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x561c4430a1e0 tx=0x7f6794013830 comp rx=0 tx=0).stop 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 -- 172.21.15.3:0/2473804313 shutdown_connections 2024-02-09T22:58:08.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442ec3e0 0x561c442f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442f4f60 0x561c442ef240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 --2- 172.21.15.3:0/2473804313 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442f2510 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 -- 172.21.15.3:0/2473804313 >> 172.21.15.3:0/2473804313 conn(0x561c442e6100 msgr2=0x561c442ed2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 -- 172.21.15.3:0/2473804313 shutdown_connections 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 -- 172.21.15.3:0/2473804313 wait complete. 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 -- start start 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.942+0000 7f67a5db4740 1 Processor -- start 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442ec3e0 0x561c442fd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442fd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c4435da20 con 0x561c442eba30 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c4435db90 con 0x561c442f4f60 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561c4435dd50 con 0x561c442ec3e0 2024-02-09T22:58:08.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a5db4740 10 client.0 _ll_register_callbacks cb 0x7f678c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442fcce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41358/0 (socket says 172.21.15.3:41358) 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 -- 172.21.15.3:0/3088502170 learned_addr learned my addr 172.21.15.3:0/3088502170 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f679bfff640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442ec3e0 0x561c442fd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a177f640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442fd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 -- 172.21.15.3:0/3088502170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442ec3e0 msgr2=0x561c442fd220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442ec3e0 0x561c442fd220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 -- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 msgr2=0x561c442fd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442fd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 -- 172.21.15.3:0/3088502170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679400a8b0 con 0x561c442eba30 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f67a0f7e640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442fcce0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f67940133f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709262db72e8a1aa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.943+0000 7f677bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:08.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f677bfff640 1 -- 172.21.15.3:0/3088502170 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794017070 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f677bfff640 1 -- 172.21.15.3:0/3088502170 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f679401c3d0 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f678c003e50 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f677bfff640 1 -- 172.21.15.3:0/3088502170 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f679401c590 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 10 client.18048 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x561c44213a70 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f677bfff640 1 -- 172.21.15.3:0/3088502170 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f678c003e50 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f677bfff640 1 -- 172.21.15.3:0/3088502170 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f6794011670 con 0x561c442eba30 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 10 client.18048 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 10 client.18048 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 20 client.18048 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c4435f520 0x561c44361940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 4 client.18048 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561c43fde890 con 0x561c4435f520 2024-02-09T22:58:08.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.944+0000 7f679bfff640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c4435f520 0x561c44361940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:08.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f679bfff640 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c4435f520 0x561c44361940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3e501159fc06d53b server_cookie=aa3f284407df2b7a in_seq=0 out_seq=0 2024-02-09T22:58:08.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f677bfff640 10 client.18048 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f677bfff640 1 -- 172.21.15.3:0/3088502170 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561c43fde890 con 0x561c4435f520 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f677bfff640 10 client.18048 handle_command_reply: tid=0 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 client.18048 shutdown 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 msgr2=0x561c442fcce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442fcce0 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f6794000c00 tx=0x7f67940133f0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 shutdown_connections 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x561c4435f520 0x561c44361940 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561c442f4f60 0x561c442fd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561c442ec3e0 0x561c442fd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 --2- 172.21.15.3:0/3088502170 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561c442eba30 0x561c442fcce0 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f677bfff640 0 client.18048 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:08.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 >> 172.21.15.3:0/3088502170 conn(0x561c442e6100 msgr2=0x561c442e49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 shutdown_connections 2024-02-09T22:58:08.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 1 -- 172.21.15.3:0/3088502170 wait complete. 2024-02-09T22:58:08.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.945+0000 7f67a5db4740 20 client.18048 trim_cache size 0 max 16384 2024-02-09T22:58:08.956 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67880a2230 msgr2=0x7f67880a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67880a2230 0x7f67880a46f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f678c004af0 tx=0x7f678c032f60 comp rx=0 tx=0).stop 2024-02-09T22:58:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 msgr2=0x7f679c196a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 0x7f679c196a60 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7f678400d680 tx=0x7f678400da80 comp rx=0 tx=0).stop 2024-02-09T22:58:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 shutdown_connections 2024-02-09T22:58:08.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67880a2230 0x7f67880a46f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f679c1965b0 0x7f679c196a60 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f679c105410 0x7f679c19c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 --2- 172.21.15.3:0/1509161818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f679c100340 0x7f679c19c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 >> 172.21.15.3:0/1509161818 conn(0x7f679c0772e0 msgr2=0x7f679c0fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 shutdown_connections 2024-02-09T22:58:08.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:08.946+0000 7f67a31e2640 1 -- 172.21.15.3:0/1509161818 wait complete. 2024-02-09T22:58:08.959 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:08.960 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:58:09.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 -- 172.21.15.3:0/707007143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 msgr2=0x7f24140ffb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 --2- 172.21.15.3:0/707007143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f24140ffb40 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f2408009900 tx=0x7f240802bc80 comp rx=0 tx=0).stop 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 -- 172.21.15.3:0/707007143 shutdown_connections 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 --2- 172.21.15.3:0/707007143 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2414100080 0x7f241410e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 --2- 172.21.15.3:0/707007143 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f24140ffb40 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 --2- 172.21.15.3:0/707007143 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f24141061c0 0x7f24141065a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 -- 172.21.15.3:0/707007143 >> 172.21.15.3:0/707007143 conn(0x7f2414076630 msgr2=0x7f24140fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 -- 172.21.15.3:0/707007143 shutdown_connections 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.061+0000 7f241b8eb640 1 -- 172.21.15.3:0/707007143 wait complete. 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 Processor -- start 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 -- start start 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f241419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2414100080 0x7f241419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24141061c0 0x7f24141976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414197e50 con 0x7f24140ff6e0 2024-02-09T22:58:09.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414197fc0 con 0x7f2414100080 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f241b8eb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2414198180 con 0x7f24141061c0 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f2419687640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f241419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f2419687640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f241419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41376/0 (socket says 172.21.15.3:41376) 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.062+0000 7f2419687640 1 -- 172.21.15.3:0/2598835435 learned_addr learned my addr 172.21.15.3:0/2598835435 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419e88640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24141061c0 0x7f24141976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2418e86640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2414100080 0x7f241419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419687640 1 -- 172.21.15.3:0/2598835435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24141061c0 msgr2=0x7f24141976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419687640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24141061c0 0x7f24141976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419687640 1 -- 172.21.15.3:0/2598835435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2414100080 msgr2=0x7f241419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419687640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2414100080 0x7f241419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419687640 1 -- 172.21.15.3:0/2598835435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2408009560 con 0x7f24140ff6e0 2024-02-09T22:58:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f2419687640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f241419d600 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f240c00d950 tx=0x7f240c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:09.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f24127fc640 1 -- 172.21.15.3:0/2598835435 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c009c30 con 0x7f24140ff6e0 2024-02-09T22:58:09.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2414198460 con 0x7f24140ff6e0 2024-02-09T22:58:09.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f24127fc640 1 -- 172.21.15.3:0/2598835435 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f240c009d90 con 0x7f24140ff6e0 2024-02-09T22:58:09.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f24127fc640 1 -- 172.21.15.3:0/2598835435 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240c00f7e0 con 0x7f24140ff6e0 2024-02-09T22:58:09.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.063+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24141a4580 con 0x7f24140ff6e0 2024-02-09T22:58:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.065+0000 7f24127fc640 1 -- 172.21.15.3:0/2598835435 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f240c010ad0 con 0x7f24140ff6e0 2024-02-09T22:58:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 Processor -- start 2024-02-09T22:58:09.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 -- start start 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e840fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e8418990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e83cb020 con 0x55c5e8418f60 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e837e320 con 0x55c5e84103e0 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f241e599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e837b850 con 0x55c5e840fa30 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f24127fc640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f23fc0a2230 0x7f23fc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41380/0 (socket says 172.21.15.3:41380) 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 -- 172.21.15.3:0/3193388709 learned_addr learned my addr 172.21.15.3:0/3193388709 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2419687640 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e8418990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2419e88640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f23fc0a2230 0x7f23fc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f24127fc640 1 -- 172.21.15.3:0/2598835435 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f240c0cdce0 con 0x7f24140ff6e0 2024-02-09T22:58:09.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2419e88640 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e840fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 -- 172.21.15.3:0/3193388709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 msgr2=0x55c5e840fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e840fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 -- 172.21.15.3:0/3193388709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 msgr2=0x55c5e8418990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e8418990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2418e86640 1 -- 172.21.15.3:0/3193388709 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e8339210 con 0x55c5e8418f60 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.066+0000 7f2419e88640 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e840fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f2419e88640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f23fc0a2230 0x7f23fc0a46f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2404005fd0 tx=0x7f2404009be0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f2418e86640 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841b3b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2408009900 tx=0x7f240802c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb76b0ebecdbbc4b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f23fb7fe640 1 -- 172.21.15.3:0/3193388709 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240802c980 con 0x55c5e8418f60 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f23fb7fe640 1 -- 172.21.15.3:0/3193388709 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f240802cae0 con 0x55c5e8418f60 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f23fb7fe640 1 -- 172.21.15.3:0/3193388709 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408005160 con 0x55c5e8418f60 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 -- 172.21.15.3:0/3193388709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 msgr2=0x55c5e841b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841b3b0 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f2408009900 tx=0x7f240802c740 comp rx=0 tx=0).stop 2024-02-09T22:58:09.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 -- 172.21.15.3:0/3193388709 shutdown_connections 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e8418990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841b3b0 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 --2- 172.21.15.3:0/3193388709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e840fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.067+0000 7f241e599740 1 -- 172.21.15.3:0/3193388709 >> 172.21.15.3:0/3193388709 conn(0x55c5e840a100 msgr2=0x55c5e840a4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 -- 172.21.15.3:0/3193388709 shutdown_connections 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 -- 172.21.15.3:0/3193388709 wait complete. 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 -- start start 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 Processor -- start 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e8415e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e84163d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e8416910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e8377030 con 0x55c5e8418f60 2024-02-09T22:58:09.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e804a430 con 0x55c5e84103e0 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e8006e00 con 0x55c5e840fa30 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f241e599740 10 client.0 _ll_register_callbacks cb 0x55c5e836c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f2419e88640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e8416910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f2419e88640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e8416910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41396/0 (socket says 172.21.15.3:41396) 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.068+0000 7f2419e88640 1 -- 172.21.15.3:0/51209704 learned_addr learned my addr 172.21.15.3:0/51209704 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419687640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e8415e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2418e86640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e84163d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419e88640 1 -- 172.21.15.3:0/51209704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 msgr2=0x55c5e8415e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419e88640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e8415e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419e88640 1 -- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 msgr2=0x55c5e84163d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419e88640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e84163d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419e88640 1 -- 172.21.15.3:0/51209704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241410e9e0 con 0x55c5e8418f60 2024-02-09T22:58:09.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f2419e88640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e8416910 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f240400fbc0 tx=0x7f2404004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f89660fad077c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f23f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f23f8ff9640 1 -- 172.21.15.3:0/51209704 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404004820 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f23f8ff9640 1 -- 172.21.15.3:0/51209704 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f2404020450 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c5e80767e0 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f23f8ff9640 1 -- 172.21.15.3:0/51209704 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2404020610 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f241e599740 10 client.18060 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55c5e83756b0 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.069+0000 7f23f8ff9640 1 -- 172.21.15.3:0/51209704 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c5e80767e0 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f23f8ff9640 1 -- 172.21.15.3:0/51209704 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f240400e7f0 con 0x55c5e8418f60 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f241e599740 10 client.18060 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f241e599740 10 client.18060 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f241e599740 20 client.18060 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:09.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f241e599740 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8429610 0x55c5e842ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f241e599740 4 client.18060 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5e80b3720 con 0x55c5e8429610 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f2419687640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8429610 0x55c5e842ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f2419687640 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8429610 0x55c5e842ba30 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7277b01255e1bff server_cookie=3b64f5aad4e644a7 in_seq=0 out_seq=0 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.070+0000 7f23f8ff9640 10 client.18060 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f23f8ff9640 1 -- 172.21.15.3:0/51209704 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c5e80b3720 con 0x55c5e8429610 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f23f8ff9640 10 client.18060 handle_command_reply: tid=0 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 client.18060 shutdown 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 msgr2=0x55c5e8416910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e8416910 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f240400fbc0 tx=0x7f2404004ab0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 shutdown_connections 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8429610 0x55c5e842ba30 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e84103e0 0x55c5e84163d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e8416910 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f23f8ff9640 0 client.18060 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:09.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 --2- 172.21.15.3:0/51209704 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e8415e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 >> 172.21.15.3:0/51209704 conn(0x55c5e840a100 msgr2=0x55c5e840be50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:09.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 shutdown_connections 2024-02-09T22:58:09.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 1 -- 172.21.15.3:0/51209704 wait complete. 2024-02-09T22:58:09.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.071+0000 7f241e599740 20 client.18060 trim_cache size 0 max 16384 2024-02-09T22:58:09.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 Processor -- start 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 -- start start 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e84185c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 0x55c5e8416730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e8418f60 0x55c5e84130f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e84058b0 con 0x55c5e84103e0 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e82fc130 con 0x55c5e8418f60 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f241e599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e8425f30 con 0x55c5e840fa30 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f2418e86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 0x55c5e8416730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f2418e86640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 0x55c5e8416730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41400/0 (socket says 172.21.15.3:41400) 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.083+0000 7f2418e86640 1 -- 172.21.15.3:0/3441988565 learned_addr learned my addr 172.21.15.3:0/3441988565 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:09.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2419e88640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e8418f60 0x55c5e84130f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2419687640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e84185c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2418e86640 1 -- 172.21.15.3:0/3441988565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 msgr2=0x55c5e84185c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2418e86640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e84185c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2418e86640 1 -- 172.21.15.3:0/3441988565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e8418f60 msgr2=0x55c5e84130f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2418e86640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e8418f60 0x55c5e84130f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2418e86640 1 -- 172.21.15.3:0/3441988565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e8311c90 con 0x55c5e84103e0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2419e88640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e8418f60 0x55c5e84130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2419687640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e84185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f2418e86640 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 0x55c5e8416730 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f24080092e0 tx=0x7f240802bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e2051be1f2d31a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f23faffd640 1 -- 172.21.15.3:0/3441988565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240802c980 con 0x55c5e84103e0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f23faffd640 1 -- 172.21.15.3:0/3441988565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24080048c0 con 0x55c5e84103e0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f23faffd640 1 -- 172.21.15.3:0/3441988565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408005400 con 0x55c5e84103e0 2024-02-09T22:58:09.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f241e599740 1 -- 172.21.15.3:0/3441988565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 msgr2=0x55c5e8416730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.084+0000 7f241e599740 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 0x55c5e8416730 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f24080092e0 tx=0x7f240802bc80 comp rx=0 tx=0).stop 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- 172.21.15.3:0/3441988565 shutdown_connections 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e8418f60 0x55c5e84130f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e84103e0 0x55c5e8416730 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 --2- 172.21.15.3:0/3441988565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e840fa30 0x55c5e84185c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- 172.21.15.3:0/3441988565 >> 172.21.15.3:0/3441988565 conn(0x55c5e840a100 msgr2=0x55c5e8411170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- 172.21.15.3:0/3441988565 shutdown_connections 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- 172.21.15.3:0/3441988565 wait complete. 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- start start 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 Processor -- start 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e840fa30 0x55c5e84202d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e84103e0 0x55c5e841e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e841f2b0 con 0x55c5e8418f60 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e841f420 con 0x55c5e840fa30 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e847fb60 con 0x55c5e84103e0 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f241e599740 10 client.0 _ll_register_callbacks cb 0x7f2408009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f2419e88640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f2419e88640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41414/0 (socket says 172.21.15.3:41414) 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.085+0000 7f2419e88640 1 -- 172.21.15.3:0/818244827 learned_addr learned my addr 172.21.15.3:0/818244827 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2418e86640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e84103e0 0x55c5e841e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419687640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e840fa30 0x55c5e84202d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419e88640 1 -- 172.21.15.3:0/818244827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e84103e0 msgr2=0x55c5e841e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419e88640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e84103e0 0x55c5e841e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419e88640 1 -- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e840fa30 msgr2=0x55c5e84202d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419e88640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e840fa30 0x55c5e84202d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419e88640 1 -- 172.21.15.3:0/818244827 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5e8339210 con 0x55c5e8418f60 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419687640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e840fa30 0x55c5e84202d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f2419e88640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841ebb0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f240400fb90 tx=0x7f240400e8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76faab67aa2d865a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f23f8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f23f8ff9640 1 -- 172.21.15.3:0/818244827 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2404003c80 con 0x55c5e8418f60 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f23f8ff9640 1 -- 172.21.15.3:0/818244827 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f2404011760 con 0x55c5e8418f60 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f240802cae0 con 0x55c5e8418f60 2024-02-09T22:58:09.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f23f8ff9640 1 -- 172.21.15.3:0/818244827 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f240400ede0 con 0x55c5e8418f60 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f241e599740 10 client.18066 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55c5e8337a70 con 0x55c5e8418f60 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f23f8ff9640 1 -- 172.21.15.3:0/818244827 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2404046020 con 0x55c5e8418f60 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.086+0000 7f23f8ff9640 1 -- 172.21.15.3:0/818244827 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f2404039430 con 0x55c5e8418f60 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 10 client.18066 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 10 client.18066 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 20 client.18066 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8481220 0x55c5e8483640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 4 client.18066 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c5e8102890 con 0x55c5e8481220 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f2419687640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8481220 0x55c5e8483640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f2419687640 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8481220 0x55c5e8483640 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9271e7af9e7e28fd server_cookie=f878c1919df93990 in_seq=0 out_seq=0 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f23f8ff9640 10 client.18066 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:09.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f23f8ff9640 1 -- 172.21.15.3:0/818244827 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c5e8102890 con 0x55c5e8481220 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f23f8ff9640 10 client.18066 handle_command_reply: tid=0 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 client.18066 shutdown 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 msgr2=0x55c5e841ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841ebb0 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 crypto rx=0x7f240400fb90 tx=0x7f240400e8d0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 shutdown_connections 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c5e8481220 0x55c5e8483640 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c5e8418f60 0x55c5e841ebb0 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c5e84103e0 0x55c5e841e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f23f8ff9640 0 client.18066 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 --2- 172.21.15.3:0/818244827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c5e840fa30 0x55c5e84202d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.087+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 >> 172.21.15.3:0/818244827 conn(0x55c5e840a100 msgr2=0x55c5e840b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 shutdown_connections 2024-02-09T22:58:09.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241e599740 1 -- 172.21.15.3:0/818244827 wait complete. 2024-02-09T22:58:09.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241e599740 20 client.18066 trim_cache size 0 max 16384 2024-02-09T22:58:09.099 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f23fc0a2230 msgr2=0x7f23fc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f23fc0a2230 0x7f23fc0a46f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2404005fd0 tx=0x7f2404009be0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 msgr2=0x7f241419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f241419d600 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f240c00d950 tx=0x7f240c00de10 comp rx=0 tx=0).stop 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 shutdown_connections 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f23fc0a2230 0x7f23fc0a46f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2414100080 0x7f241419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f24141061c0 0x7f24141976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 --2- 172.21.15.3:0/2598835435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f24140ff6e0 0x7f241419d600 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 >> 172.21.15.3:0/2598835435 conn(0x7f2414076630 msgr2=0x7f2414104190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 shutdown_connections 2024-02-09T22:58:09.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:09.088+0000 7f241b8eb640 1 -- 172.21.15.3:0/2598835435 wait complete. 2024-02-09T22:58:09.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1858092370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:58:09.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:09 smithi003 ceph-mon[32678]: osdmap e91: 12 total, 12 up, 12 in 2024-02-09T22:58:09.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 195887 2.60 MB/sec execute 243 sec latency 1549.671 ms 2024-02-09T22:58:09.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1858092370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:58:09.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:09 smithi064 ceph-mon[36638]: osdmap e91: 12 total, 12 up, 12 in 2024-02-09T22:58:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1858092370' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:58:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:09 smithi089 ceph-mon[36665]: osdmap e91: 12 total, 12 up, 12 in 2024-02-09T22:58:10.102 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:58:10.102 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:58:10.102 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:58:10.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.218+0000 7f6188502640 1 -- 172.21.15.3:0/1411304126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 msgr2=0x7f618010dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.218+0000 7f6188502640 1 --2- 172.21.15.3:0/1411304126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618010dca0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f6178009900 tx=0x7f617802bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- 172.21.15.3:0/1411304126 shutdown_connections 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 --2- 172.21.15.3:0/1411304126 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 0x7f618010b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 --2- 172.21.15.3:0/1411304126 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618010dca0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 --2- 172.21.15.3:0/1411304126 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6180102300 0x7f61801026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- 172.21.15.3:0/1411304126 >> 172.21.15.3:0/1411304126 conn(0x7f61800fbb70 msgr2=0x7f61800fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- 172.21.15.3:0/1411304126 shutdown_connections 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- 172.21.15.3:0/1411304126 wait complete. 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 Processor -- start 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- start start 2024-02-09T22:58:10.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6180102300 0x7f61801977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 0x7f6180197ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618019c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618019c610 con 0x7f618010b800 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618019c780 con 0x7f6180102cb0 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.219+0000 7f6188502640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f618019c940 con 0x7f6180102300 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618019c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618019c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41428/0 (socket says 172.21.15.3:41428) 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 -- 172.21.15.3:0/1137632100 learned_addr learned my addr 172.21.15.3:0/1137632100 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f618629e640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6180102300 0x7f61801977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6185a9d640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 0x7f6180197ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 -- 172.21.15.3:0/1137632100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6180102300 msgr2=0x7f61801977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6180102300 0x7f61801977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 -- 172.21.15.3:0/1137632100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 msgr2=0x7f6180197ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 0x7f6180197ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 -- 172.21.15.3:0/1137632100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178009560 con 0x7f618010b800 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6185a9d640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 0x7f6180197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6186a9f640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618019c010 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6178009230 tx=0x7f6178004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f61777fe640 1 -- 172.21.15.3:0/1137632100 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178004440 con 0x7f618010b800 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f618019cbc0 con 0x7f618010b800 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f61777fe640 1 -- 172.21.15.3:0/1137632100 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61780052d0 con 0x7f618010b800 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f61777fe640 1 -- 172.21.15.3:0/1137632100 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6178003cc0 con 0x7f618010b800 2024-02-09T22:58:10.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.220+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61801a44a0 con 0x7f618010b800 2024-02-09T22:58:10.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.223+0000 7f61777fe640 1 -- 172.21.15.3:0/1137632100 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6178005440 con 0x7f618010b800 2024-02-09T22:58:10.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.223+0000 7f618b1b2740 1 Processor -- start 2024-02-09T22:58:10.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.223+0000 7f618b1b2740 1 -- start start 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.223+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 0x55c9450a7980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f61777fe640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f616c0a2230 0x7f616c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450b0560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450b0aa0 0x55c9450b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618b1b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c945011000 con 0x55c9450a75a0 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618b1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c94500e530 con 0x55c9450b0aa0 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618b1b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c945009d10 con 0x55c9450a7ec0 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 0x55c9450a7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618629e640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f616c0a2230 0x7f616c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 0x55c9450a7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41432/0 (socket says 172.21.15.3:41432) 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 -- 172.21.15.3:0/655945210 learned_addr learned my addr 172.21.15.3:0/655945210 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f61777fe640 1 -- 172.21.15.3:0/1137632100 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f61780f02e0 con 0x7f618010b800 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6186a9f640 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450b0aa0 0x55c9450b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 -- 172.21.15.3:0/655945210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 msgr2=0x55c9450b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450b0560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 -- 172.21.15.3:0/655945210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450b0aa0 msgr2=0x55c9450b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450b0aa0 0x55c9450b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 -- 172.21.15.3:0/655945210 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c944fb0210 con 0x55c9450a75a0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f618629e640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f616c0a2230 0x7f616c0a46f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f617c005fd0 tx=0x7f617c009680 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6185a9d640 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 0x55c9450a7980 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f61700058c0 tx=0x7f617000b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc6b640d6b41a975 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6174bfe640 1 -- 172.21.15.3:0/655945210 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170011540 con 0x55c9450a75a0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.224+0000 7f6174bfe640 1 -- 172.21.15.3:0/655945210 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6170011b80 con 0x55c9450a75a0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f6174bfe640 1 -- 172.21.15.3:0/655945210 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f617001db90 con 0x55c9450a75a0 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 -- 172.21.15.3:0/655945210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 msgr2=0x55c9450a7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 0x55c9450a7980 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7f61700058c0 tx=0x7f617000b730 comp rx=0 tx=0).stop 2024-02-09T22:58:10.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 -- 172.21.15.3:0/655945210 shutdown_connections 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450b0aa0 0x55c9450b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 --2- 172.21.15.3:0/655945210 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a75a0 0x55c9450a7980 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 -- 172.21.15.3:0/655945210 >> 172.21.15.3:0/655945210 conn(0x55c9450a1cd0 msgr2=0x55c9450a20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 -- 172.21.15.3:0/655945210 shutdown_connections 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.225+0000 7f618b1b2740 1 -- 172.21.15.3:0/655945210 wait complete. 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 -- start start 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 Processor -- start 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450adad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450ae010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450ae550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c944faee60 con 0x55c9450b0aa0 2024-02-09T22:58:10.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c944cdd4b0 con 0x55c9450a75a0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c944f8f670 con 0x55c9450a7ec0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618b1b2740 10 client.0 _ll_register_callbacks cb 0x55c944fa7190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450ae550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41444/0 (socket says 172.21.15.3:41444) 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 -- 172.21.15.3:0/4222623127 learned_addr learned my addr 172.21.15.3:0/4222623127 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f618629e640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6185a9d640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450ae010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 -- 172.21.15.3:0/4222623127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 msgr2=0x55c9450ae010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450ae010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 -- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 msgr2=0x55c9450adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6186a9f640 1 -- 172.21.15.3:0/4222623127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6170013400 con 0x55c9450b0aa0 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.226+0000 7f6185a9d640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:10.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6186a9f640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450ae550 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6178005940 tx=0x7f61780bc980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=729711ad6db8a270 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6169ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6169ffb640 1 -- 172.21.15.3:0/4222623127 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61780c5290 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6169ffb640 1 -- 172.21.15.3:0/4222623127 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f61780bd480 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c944d09860 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6169ffb640 1 -- 172.21.15.3:0/4222623127 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61780ede60 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 10 client.18078 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55c945008390 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6169ffb640 1 -- 172.21.15.3:0/4222623127 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61780d4080 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f6169ffb640 1 -- 172.21.15.3:0/4222623127 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f617803c070 con 0x55c9450b0aa0 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 10 client.18078 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 10 client.18078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 20 client.18078 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c9450c0a20 0x55c9450c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.238 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 4 client.18078 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.227+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c944d5a410 con 0x55c9450c0a20 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.228+0000 7f618629e640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c9450c0a20 0x55c9450c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.228+0000 7f618629e640 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c9450c0a20 0x55c9450c2e40 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbdcab57d97af10a server_cookie=ff1daa566cc15853 in_seq=0 out_seq=0 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.228+0000 7f6169ffb640 10 client.18078 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f6169ffb640 1 -- 172.21.15.3:0/4222623127 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c944d5a410 con 0x55c9450c0a20 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f6169ffb640 10 client.18078 handle_command_reply: tid=0 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 client.18078 shutdown 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 msgr2=0x55c9450ae550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450ae550 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f6178005940 tx=0x7f61780bc980 comp rx=0 tx=0).stop 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 shutdown_connections 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c9450c0a20 0x55c9450c2e40 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450ae550 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a7ec0 0x55c9450ae010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 --2- 172.21.15.3:0/4222623127 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f6169ffb640 0 client.18078 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 >> 172.21.15.3:0/4222623127 conn(0x55c9450a1cd0 msgr2=0x55c9450a3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 shutdown_connections 2024-02-09T22:58:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 1 -- 172.21.15.3:0/4222623127 wait complete. 2024-02-09T22:58:10.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.229+0000 7f618b1b2740 20 client.18078 trim_cache size 0 max 16384 2024-02-09T22:58:10.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 Processor -- start 2024-02-09T22:58:10.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 -- start start 2024-02-09T22:58:10.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450ae370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 0x55c94509c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450b0aa0 0x55c9450b7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c94509bec0 con 0x55c9450a7ec0 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9450990a0 con 0x55c9450a75a0 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f618b1b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9450c8050 con 0x55c9450b0aa0 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f6185a9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 0x55c94509c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.244+0000 7f6185a9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 0x55c94509c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41458/0 (socket says 172.21.15.3:41458) 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 -- 172.21.15.3:0/1875733416 learned_addr learned my addr 172.21.15.3:0/1875733416 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 -- 172.21.15.3:0/1875733416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450b0aa0 msgr2=0x55c9450b7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450b0aa0 0x55c9450b7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 -- 172.21.15.3:0/1875733416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 msgr2=0x55c9450ae370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450ae370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 -- 172.21.15.3:0/1875733416 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c944fa4c70 con 0x55c9450a7ec0 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f6185a9d640 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 0x55c94509c7a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6170013ee0 tx=0x7f617000bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cad55f305166746b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f616bfff640 1 -- 172.21.15.3:0/1875733416 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170023070 con 0x55c9450a7ec0 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f616bfff640 1 -- 172.21.15.3:0/1875733416 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f617000b3a0 con 0x55c9450a7ec0 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f616bfff640 1 -- 172.21.15.3:0/1875733416 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6170005040 con 0x55c9450a7ec0 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f618b1b2740 1 -- 172.21.15.3:0/1875733416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 msgr2=0x55c94509c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.245+0000 7f618b1b2740 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 0x55c94509c7a0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f6170013ee0 tx=0x7f617000bd20 comp rx=0 tx=0).stop 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- 172.21.15.3:0/1875733416 shutdown_connections 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450b0aa0 0x55c9450b7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450a7ec0 0x55c94509c7a0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 --2- 172.21.15.3:0/1875733416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a75a0 0x55c9450ae370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- 172.21.15.3:0/1875733416 >> 172.21.15.3:0/1875733416 conn(0x55c9450a1cd0 msgr2=0x55c94509c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- 172.21.15.3:0/1875733416 shutdown_connections 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- 172.21.15.3:0/1875733416 wait complete. 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- start start 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 Processor -- start 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a75a0 0x55c945110350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a7ec0 0x55c945110890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450b6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9450b7440 con 0x55c9450b0aa0 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9450b75b0 con 0x55c9450a7ec0 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9450b7770 con 0x55c9450a75a0 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f618b1b2740 10 client.0 _ll_register_callbacks cb 0x7f617000b590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:10.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f6186a9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450b6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f6186a9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450b6d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41464/0 (socket says 172.21.15.3:41464) 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f6186a9f640 1 -- 172.21.15.3:0/3944952699 learned_addr learned my addr 172.21.15.3:0/3944952699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.246+0000 7f6185a9d640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a7ec0 0x55c945110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6186a9f640 1 -- 172.21.15.3:0/3944952699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a75a0 msgr2=0x55c945110350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6186a9f640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a75a0 0x55c945110350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6186a9f640 1 -- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a7ec0 msgr2=0x55c945110890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6186a9f640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a7ec0 0x55c945110890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6186a9f640 1 -- 172.21.15.3:0/3944952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c944fb0210 con 0x55c9450b0aa0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6185a9d640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a7ec0 0x55c945110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6186a9f640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450b6d40 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f61780bcd10 tx=0x7f6178005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e4ce04cb901b4ff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6169ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6169ffb640 1 -- 172.21.15.3:0/3944952699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61780bc470 con 0x55c9450b0aa0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6169ffb640 1 -- 172.21.15.3:0/3944952699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f61780bd480 con 0x55c9450b0aa0 2024-02-09T22:58:10.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6170011b80 con 0x55c9450b0aa0 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6169ffb640 1 -- 172.21.15.3:0/3944952699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61780ede60 con 0x55c9450b0aa0 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 10 client.18084 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x55c944fcaac0 con 0x55c9450b0aa0 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6169ffb640 1 -- 172.21.15.3:0/3944952699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6170011b80 con 0x55c9450b0aa0 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f6169ffb640 1 -- 172.21.15.3:0/3944952699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f617803c050 con 0x55c9450b0aa0 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 10 client.18084 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 10 client.18084 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 20 client.18084 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c945118e30 0x55c94511b220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 4 client.18084 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.247+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c944d467b0 con 0x55c945118e30 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.248+0000 7f618629e640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c945118e30 0x55c94511b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.248+0000 7f618629e640 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c945118e30 0x55c94511b220 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ab4c7209a28841f server_cookie=f7238ca52142e655 in_seq=0 out_seq=0 2024-02-09T22:58:10.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.248+0000 7f6169ffb640 10 client.18084 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f6169ffb640 1 -- 172.21.15.3:0/3944952699 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c944d467b0 con 0x55c945118e30 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f6169ffb640 10 client.18084 handle_command_reply: tid=0 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 client.18084 shutdown 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 msgr2=0x55c9450b6d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450b6d40 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f61780bcd10 tx=0x7f6178005b50 comp rx=0 tx=0).stop 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 shutdown_connections 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c945118e30 0x55c94511b220 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f6169ffb640 0 client.18084 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c9450a7ec0 0x55c945110890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c9450b0aa0 0x55c9450b6d40 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 --2- 172.21.15.3:0/3944952699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c9450a75a0 0x55c945110350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.249+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 >> 172.21.15.3:0/3944952699 conn(0x55c9450a1cd0 msgr2=0x55c94509c920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 shutdown_connections 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f618b1b2740 1 -- 172.21.15.3:0/3944952699 wait complete. 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f618b1b2740 20 client.18084 trim_cache size 0 max 16384 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "dd352801-a9ad-4d84-9f08-5676edd2a185", 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:58:10.259 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:58:10.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f616c0a2230 msgr2=0x7f616c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f616c0a2230 0x7f616c0a46f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f617c005fd0 tx=0x7f617c009680 comp rx=0 tx=0).stop 2024-02-09T22:58:10.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 msgr2=0x7f618019c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618019c010 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f6178009230 tx=0x7f6178004040 comp rx=0 tx=0).stop 2024-02-09T22:58:10.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 shutdown_connections 2024-02-09T22:58:10.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f616c0a2230 0x7f616c0a46f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6180102cb0 0x7f6180197ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f618010b800 0x7f618019c010 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 --2- 172.21.15.3:0/1137632100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6180102300 0x7f61801977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 >> 172.21.15.3:0/1137632100 conn(0x7f61800fbb70 msgr2=0x7f61800fc090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 shutdown_connections 2024-02-09T22:58:10.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.250+0000 7f6188502640 1 -- 172.21.15.3:0/1137632100 wait complete. 2024-02-09T22:58:10.265 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:58:10.265 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:58:10.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 -- 172.21.15.3:0/1569437464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c100340 msgr2=0x7f985c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 --2- 172.21.15.3:0/1569437464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c100340 0x7f985c100720 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7f9850009930 tx=0x7f985002bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:10.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 -- 172.21.15.3:0/1569437464 shutdown_connections 2024-02-09T22:58:10.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 --2- 172.21.15.3:0/1569437464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f985c100cf0 0x7f985c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 --2- 172.21.15.3:0/1569437464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985c105410 0x7f985c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 --2- 172.21.15.3:0/1569437464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c100340 0x7f985c100720 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 -- 172.21.15.3:0/1569437464 >> 172.21.15.3:0/1569437464 conn(0x7f985c0772e0 msgr2=0x7f985c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 -- 172.21.15.3:0/1569437464 shutdown_connections 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.375+0000 7f98631c9640 1 -- 172.21.15.3:0/1569437464 wait complete. 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 Processor -- start 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 -- start start 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985c100340 0x7f985c198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f985c105410 0x7f985c198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 0x7f985c1a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1a3e40 con 0x7f985c1991f0 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1a3fb0 con 0x7f985c105410 2024-02-09T22:58:10.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f98631c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f985c1a4170 con 0x7f985c100340 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 0x7f985c1a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9860f65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985c100340 0x7f985c198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 0x7f985c1a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41494/0 (socket says 172.21.15.3:41494) 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 -- 172.21.15.3:0/974256536 learned_addr learned my addr 172.21.15.3:0/974256536 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f985bfff640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f985c105410 0x7f985c198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 -- 172.21.15.3:0/974256536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985c100340 msgr2=0x7f985c198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985c100340 0x7f985c198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 -- 172.21.15.3:0/974256536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f985c105410 msgr2=0x7f985c198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f985c105410 0x7f985c198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.376+0000 7f9861766640 1 -- 172.21.15.3:0/974256536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9850009560 con 0x7f985c1991f0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.377+0000 7f9861766640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 0x7f985c1a37a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f985400ea60 tx=0x7f985400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.377+0000 7f9859ffb640 1 -- 172.21.15.3:0/974256536 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f985400ce40 con 0x7f985c1991f0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.377+0000 7f9859ffb640 1 -- 172.21.15.3:0/974256536 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9854005020 con 0x7f985c1991f0 2024-02-09T22:58:10.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.377+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f985c1a4450 con 0x7f985c1991f0 2024-02-09T22:58:10.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.377+0000 7f9859ffb640 1 -- 172.21.15.3:0/974256536 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9854010690 con 0x7f985c1991f0 2024-02-09T22:58:10.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.377+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c1a49b0 con 0x7f985c1991f0 2024-02-09T22:58:10.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:10 smithi003 ceph-mon[32678]: pgmap v211: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:58:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9859ffb640 1 -- 172.21.15.3:0/974256536 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9854005500 con 0x7f985c1991f0 2024-02-09T22:58:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 Processor -- start 2024-02-09T22:58:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 -- start start 2024-02-09T22:58:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f610d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f6198e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f61c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f5cbd00 con 0x563b7f610980 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f5802f0 con 0x563b7f619eb0 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9865d9e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f57d820 con 0x563b7f611330 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.380+0000 7f9859ffb640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f984c0a22b0 0x7f984c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f6198e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f985bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f610d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f985bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f610d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41496/0 (socket says 172.21.15.3:41496) 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f6198e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:47002/0 (socket says 172.21.15.3:47002) 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f985bfff640 1 -- 172.21.15.3:0/3140922540 learned_addr learned my addr 172.21.15.3:0/3140922540 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f985bfff640 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f61c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9859ffb640 1 -- 172.21.15.3:0/974256536 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9854014070 con 0x7f985c1991f0 2024-02-09T22:58:10.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9861766640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f984c0a22b0 0x7f984c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 -- 172.21.15.3:0/3140922540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 msgr2=0x563b7f61c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f61c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 -- 172.21.15.3:0/3140922540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 msgr2=0x563b7f610d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f610d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 -- 172.21.15.3:0/3140922540 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b7f53aff0 con 0x563b7f611330 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9861766640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f984c0a22b0 0x7f984c0a4770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f98540049d0 tx=0x7f985400ef90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9860f65640 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f6198e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9850009230 tx=0x7f985002c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5150e28be5507a7a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.381+0000 7f9846ffd640 1 -- 172.21.15.3:0/3140922540 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850004800 con 0x563b7f611330 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9846ffd640 1 -- 172.21.15.3:0/3140922540 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9850004960 con 0x563b7f611330 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9846ffd640 1 -- 172.21.15.3:0/3140922540 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9850005030 con 0x563b7f611330 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 -- 172.21.15.3:0/3140922540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 msgr2=0x563b7f6198e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f6198e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9850009230 tx=0x7f985002c680 comp rx=0 tx=0).stop 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 -- 172.21.15.3:0/3140922540 shutdown_connections 2024-02-09T22:58:10.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f61c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f6198e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 --2- 172.21.15.3:0/3140922540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f610d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 -- 172.21.15.3:0/3140922540 >> 172.21.15.3:0/3140922540 conn(0x563b7f60b050 msgr2=0x563b7f60b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 -- 172.21.15.3:0/3140922540 shutdown_connections 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.382+0000 7f9865d9e740 1 -- 172.21.15.3:0/3140922540 wait complete. 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 -- start start 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 Processor -- start 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f616d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f611330 0x563b7f617260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f6177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f579000 con 0x563b7f611330 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f24c430 con 0x563b7f619eb0 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f208e00 con 0x563b7f610980 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9865d9e740 10 client.0 _ll_register_callbacks cb 0x563b7f56d7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:10.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f9860f65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f611330 0x563b7f617260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f985bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f616d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f985bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f616d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:47004/0 (socket says 172.21.15.3:47004) 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.383+0000 7f985bfff640 1 -- 172.21.15.3:0/1057125258 learned_addr learned my addr 172.21.15.3:0/1057125258 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f6177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 -- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 msgr2=0x563b7f6177a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f6177a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 -- 172.21.15.3:0/1057125258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f611330 msgr2=0x563b7f617260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f611330 0x563b7f617260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 -- 172.21.15.3:0/1057125258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f985c10d7f0 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f9860f65640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f611330 0x563b7f617260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f985bfff640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f616d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9848012ee0 tx=0x7f984800bc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d4dfc4680a84f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f983bfff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f983bfff640 1 -- 172.21.15.3:0/1057125258 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f984800ee80 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f983bfff640 1 -- 172.21.15.3:0/1057125258 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9848004020 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563b7f2787e0 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f983bfff640 1 -- 172.21.15.3:0/1057125258 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f98480041e0 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f9865d9e740 10 client.27265 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x563b7f577680 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.384+0000 7f983bfff640 1 -- 172.21.15.3:0/1057125258 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563b7f2787e0 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f983bfff640 1 -- 172.21.15.3:0/1057125258 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9848023460 con 0x563b7f610980 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9865d9e740 10 client.27265 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9865d9e740 10 client.27265 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:10.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9865d9e740 20 client.27265 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9865d9e740 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f62a400 0x563b7f62c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9865d9e740 4 client.27265 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b7f2b5720 con 0x563b7f62a400 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9860f65640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f62a400 0x563b7f62c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f983bfff640 10 client.27265 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.385+0000 7f9860f65640 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f62a400 0x563b7f62c820 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=913ff74238069f9c server_cookie=58439693199820a0 in_seq=0 out_seq=0 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f983bfff640 1 -- 172.21.15.3:0/1057125258 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563b7f2b5720 con 0x563b7f62a400 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f983bfff640 10 client.27265 handle_command_reply: tid=0 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 client.27265 shutdown 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 msgr2=0x563b7f616d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f616d20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f9848012ee0 tx=0x7f984800bc00 comp rx=0 tx=0).stop 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 shutdown_connections 2024-02-09T22:58:10.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f62a400 0x563b7f62c820 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f6177a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f611330 0x563b7f617260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 --2- 172.21.15.3:0/1057125258 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f616d20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f983bfff640 0 client.27265 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 >> 172.21.15.3:0/1057125258 conn(0x563b7f60b050 msgr2=0x563b7f60cda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 shutdown_connections 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 1 -- 172.21.15.3:0/1057125258 wait complete. 2024-02-09T22:58:10.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.386+0000 7f9865d9e740 20 client.27265 trim_cache size 0 max 16384 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 Processor -- start 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 -- start start 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f60cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f611330 0x563b7f620e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 0x563b7f6213c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f4fdfd0 con 0x563b7f619eb0 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f6102d0 con 0x563b7f611330 2024-02-09T22:58:10.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9865d9e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f612910 con 0x563b7f610980 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f985bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 0x563b7f6213c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f985bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 0x563b7f6213c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41508/0 (socket says 172.21.15.3:41508) 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f985bfff640 1 -- 172.21.15.3:0/845187322 learned_addr learned my addr 172.21.15.3:0/845187322 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f985bfff640 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f60cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.403+0000 7f9860f65640 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f611330 0x563b7f620e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f985bfff640 1 -- 172.21.15.3:0/845187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 msgr2=0x563b7f60cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f985bfff640 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f60cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f985bfff640 1 -- 172.21.15.3:0/845187322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f611330 msgr2=0x563b7f620e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f985bfff640 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f611330 0x563b7f620e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f985bfff640 1 -- 172.21.15.3:0/845187322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b7f513b30 con 0x563b7f619eb0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f985bfff640 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 0x563b7f6213c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f9848012a30 tx=0x7f9848012560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96ebc88447f245b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f98467fc640 1 -- 172.21.15.3:0/845187322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480127c0 con 0x563b7f619eb0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f98467fc640 1 -- 172.21.15.3:0/845187322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f984800ee80 con 0x563b7f619eb0 2024-02-09T22:58:10.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.404+0000 7f98467fc640 1 -- 172.21.15.3:0/845187322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98480046c0 con 0x563b7f619eb0 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- 172.21.15.3:0/845187322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 msgr2=0x563b7f6213c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 0x563b7f6213c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f9848012a30 tx=0x7f9848012560 comp rx=0 tx=0).stop 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- 172.21.15.3:0/845187322 shutdown_connections 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f611330 0x563b7f620e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f619eb0 0x563b7f6213c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- 172.21.15.3:0/845187322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f610980 0x563b7f60cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- 172.21.15.3:0/845187322 >> 172.21.15.3:0/845187322 conn(0x563b7f60b050 msgr2=0x563b7f60c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- 172.21.15.3:0/845187322 shutdown_connections 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- 172.21.15.3:0/845187322 wait complete. 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- start start 2024-02-09T22:58:10.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 Processor -- start 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f67a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f67acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f674b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f675140 con 0x563b7f610980 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f6752b0 con 0x563b7f619eb0 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b7f675470 con 0x563b7f611330 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.405+0000 7f9865d9e740 10 client.0 _ll_register_callbacks cb 0x7f9850009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:10.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f67a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f67a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:41514/0 (socket says 172.21.15.3:41514) 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 -- 172.21.15.3:0/2670947083 learned_addr learned my addr 172.21.15.3:0/2670947083 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f9860f65640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f67acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f674b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 -- 172.21.15.3:0/2670947083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 msgr2=0x563b7f67acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f67acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 -- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 msgr2=0x563b7f674b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f674b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 -- 172.21.15.3:0/2670947083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b7f577680 con 0x563b7f610980 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f985bfff640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f67a770 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x563b7f62a360 tx=0x7f9848023670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf81a2155191ebd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f983bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f983bfff640 1 -- 172.21.15.3:0/2670947083 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9848023bf0 con 0x563b7f610980 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f983bfff640 1 -- 172.21.15.3:0/2670947083 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9848023460 con 0x563b7f610980 2024-02-09T22:58:10.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9850004960 con 0x563b7f610980 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f983bfff640 1 -- 172.21.15.3:0/2670947083 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9848023e10 con 0x563b7f610980 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f9865d9e740 10 client.18102 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x563b7f539850 con 0x563b7f610980 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.406+0000 7f983bfff640 1 -- 172.21.15.3:0/2670947083 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9848049020 con 0x563b7f610980 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f983bfff640 1 -- 172.21.15.3:0/2670947083 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f9848004b30 con 0x563b7f610980 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 10 client.18102 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 10 client.18102 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 20 client.18102 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f682270 0x563b7f684690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 4 client.18102 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563b7f304890 con 0x563b7f682270 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9860f65640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f682270 0x563b7f684690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f983bfff640 10 client.18102 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9860f65640 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f682270 0x563b7f684690 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34c6e8e38aac75b2 server_cookie=81ab190a8947ed16 in_seq=0 out_seq=0 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f983bfff640 1 -- 172.21.15.3:0/2670947083 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x563b7f304890 con 0x563b7f682270 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f983bfff640 10 client.18102 handle_command_reply: tid=0 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.407+0000 7f9865d9e740 1 client.18102 shutdown 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 msgr2=0x563b7f67a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f67a770 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x563b7f62a360 tx=0x7f9848023670 comp rx=0 tx=0).stop 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 shutdown_connections 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563b7f682270 0x563b7f684690 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563b7f619eb0 0x563b7f674b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563b7f611330 0x563b7f67acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 --2- 172.21.15.3:0/2670947083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563b7f610980 0x563b7f67a770 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 >> 172.21.15.3:0/2670947083 conn(0x563b7f60b050 msgr2=0x563b7f60ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f983bfff640 0 client.18102 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 shutdown_connections 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 1 -- 172.21.15.3:0/2670947083 wait complete. 2024-02-09T22:58:10.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f9865d9e740 20 client.18102 trim_cache size 0 max 16384 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "dd352801-a9ad-4d84-9f08-5676edd2a185": { 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "dd352801-a9ad-4d84-9f08-5676edd2a185", 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:58:10.418 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:58:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f984c0a22b0 msgr2=0x7f984c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f984c0a22b0 0x7f984c0a4770 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f98540049d0 tx=0x7f985400ef90 comp rx=0 tx=0).stop 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 msgr2=0x7f985c1a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 0x7f985c1a37a0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f985400ea60 tx=0x7f985400ef20 comp rx=0 tx=0).stop 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 shutdown_connections 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f984c0a22b0 0x7f984c0a4770 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f985c105410 0x7f985c198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f985c1991f0 0x7f985c1a37a0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 --2- 172.21.15.3:0/974256536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f985c100340 0x7f985c198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.408+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 >> 172.21.15.3:0/974256536 conn(0x7f985c0772e0 msgr2=0x7f985c0fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.409+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 shutdown_connections 2024-02-09T22:58:10.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:10.409+0000 7f98631c9640 1 -- 172.21.15.3:0/974256536 wait complete. 2024-02-09T22:58:10.422 INFO:tasks.cephfs.filesystem:scrub status for tag:dd352801-a9ad-4d84-9f08-5676edd2a185 - {'path': '/', 'tag': 'dd352801-a9ad-4d84-9f08-5676edd2a185', 'options': 'recursive,force'} 2024-02-09T22:58:10.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 196615 2.61 MB/sec execute 244 sec latency 2183.646 ms 2024-02-09T22:58:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:10 smithi064 ceph-mon[36638]: pgmap v211: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:58:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:10 smithi089 ceph-mon[36665]: pgmap v211: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.2 MiB/s wr, 174 op/s 2024-02-09T22:58:11.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:58:11.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:11.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:11 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 196788 2.60 MB/sec execute 245 sec latency 881.773 ms 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:11 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:58:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:11.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:11.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:11 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:12 smithi003 ceph-mon[32678]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 658 KiB/s wr, 150 op/s 2024-02-09T22:58:12.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 198959 2.64 MB/sec execute 246 sec latency 928.722 ms 2024-02-09T22:58:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:12 smithi064 ceph-mon[36638]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 658 KiB/s wr, 150 op/s 2024-02-09T22:58:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:12 smithi089 ceph-mon[36665]: pgmap v213: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 658 KiB/s wr, 150 op/s 2024-02-09T22:58:13.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:13 smithi003 ceph-mon[32678]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 536 KiB/s wr, 122 op/s 2024-02-09T22:58:13.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 200198 2.65 MB/sec execute 247 sec latency 32.848 ms 2024-02-09T22:58:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:13 smithi064 ceph-mon[36638]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 536 KiB/s wr, 122 op/s 2024-02-09T22:58:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:13 smithi089 ceph-mon[36665]: pgmap v214: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 536 KiB/s wr, 122 op/s 2024-02-09T22:58:14.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 200331 2.64 MB/sec execute 248 sec latency 901.523 ms 2024-02-09T22:58:15.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 200331 2.63 MB/sec execute 249 sec latency 1901.602 ms 2024-02-09T22:58:16.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:16 smithi089 ceph-mon[36665]: pgmap v215: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 536 KiB/s wr, 122 op/s 2024-02-09T22:58:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:16 smithi003 ceph-mon[32678]: pgmap v215: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 536 KiB/s wr, 122 op/s 2024-02-09T22:58:16.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 200331 2.62 MB/sec execute 250 sec latency 2901.711 ms 2024-02-09T22:58:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:16 smithi064 ceph-mon[36638]: pgmap v215: 97 pgs: 97 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 344 KiB/s rd, 536 KiB/s wr, 122 op/s 2024-02-09T22:58:17.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:17 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:17 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:17 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:17.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:17 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:17.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202475 2.66 MB/sec execute 251 sec latency 2954.839 ms 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 -- 172.21.15.3:0/2060812890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018100340 msgr2=0x7fe018100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/2060812890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018100340 0x7fe018100720 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7fe014009930 tx=0x7fe01402bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 -- 172.21.15.3:0/2060812890 shutdown_connections 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/2060812890 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe018105410 0x7fe01810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/2060812890 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe018100cf0 0x7fe018104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/2060812890 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018100340 0x7fe018100720 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.669+0000 7fe01fe6f640 1 -- 172.21.15.3:0/2060812890 >> 172.21.15.3:0/2060812890 conn(0x7fe018077220 msgr2=0x7fe018077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 -- 172.21.15.3:0/2060812890 shutdown_connections 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 -- 172.21.15.3:0/2060812890 wait complete. 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 Processor -- start 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 -- start start 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe018100340 0x7fe01819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 0x7fe01819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe018197700 0x7fe018197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0181981d0 con 0x7fe018105410 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018198340 con 0x7fe018100340 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01fe6f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe018198500 con 0x7fe018197700 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.670+0000 7fe01d40a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 0x7fe01819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 0x7fe01819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55118/0 (socket says 172.21.15.3:55118) 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 -- 172.21.15.3:0/4244549217 learned_addr learned my addr 172.21.15.3:0/4244549217 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe018100340 0x7fe01819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01e40c640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe018197700 0x7fe018197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 -- 172.21.15.3:0/4244549217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe018197700 msgr2=0x7fe018197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe018197700 0x7fe018197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 -- 172.21.15.3:0/4244549217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe018100340 msgr2=0x7fe01819d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe018100340 0x7fe01819d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 -- 172.21.15.3:0/4244549217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe014009560 con 0x7fe018105410 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01d40a640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 0x7fe01819db30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e9c0 tx=0x7fe00800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe00effd640 1 -- 172.21.15.3:0/4244549217 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008009700 con 0x7fe018105410 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe00effd640 1 -- 172.21.15.3:0/4244549217 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe00800cd80 con 0x7fe018105410 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0181a4450 con 0x7fe018105410 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe00effd640 1 -- 172.21.15.3:0/4244549217 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe008010430 con 0x7fe018105410 2024-02-09T22:58:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.671+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0181a4930 con 0x7fe018105410 2024-02-09T22:58:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.673+0000 7fe00effd640 1 -- 172.21.15.3:0/4244549217 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe008005000 con 0x7fe018105410 2024-02-09T22:58:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 Processor -- start 2024-02-09T22:58:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 -- start start 2024-02-09T22:58:17.684 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 0x561204263490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042639d0 0x56120426c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe00effd640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0040a2220 0x7fe0040a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120426c7a0 0x56120426eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612041d1a00 con 0x5612042630b0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612041cef30 con 0x56120426c7a0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe022a43740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612041ca710 con 0x5612042639d0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe01e40c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 0x561204263490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe01d40a640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0040a2220 0x7fe0040a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe01e40c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 0x561204263490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55134/0 (socket says 172.21.15.3:55134) 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe01e40c640 1 -- 172.21.15.3:0/2433447196 learned_addr learned my addr 172.21.15.3:0/2433447196 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.674+0000 7fe01e40c640 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120426c7a0 0x56120426eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe00effd640 1 -- 172.21.15.3:0/4244549217 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe008014070 con 0x7fe018105410 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042639d0 0x56120426c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.685 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01e40c640 1 -- 172.21.15.3:0/2433447196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042639d0 msgr2=0x56120426c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01e40c640 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042639d0 0x56120426c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01e40c640 1 -- 172.21.15.3:0/2433447196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120426c7a0 msgr2=0x56120426eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01e40c640 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120426c7a0 0x56120426eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01e40c640 1 -- 172.21.15.3:0/2433447196 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56120418f3b0 con 0x5612042630b0 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01d40a640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0040a2220 0x7fe0040a46e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe008000c00 tx=0x7fe008005880 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe01e40c640 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 0x561204263490 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x56120425eb60 tx=0x7fe010005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb0d3ba077729e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe003fff640 1 -- 172.21.15.3:0/2433447196 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe010013130 con 0x5612042630b0 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe003fff640 1 -- 172.21.15.3:0/2433447196 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe010009d30 con 0x5612042630b0 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.675+0000 7fe003fff640 1 -- 172.21.15.3:0/2433447196 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01000fb40 con 0x5612042630b0 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 -- 172.21.15.3:0/2433447196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 msgr2=0x561204263490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 0x561204263490 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x56120425eb60 tx=0x7fe010005bd0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 -- 172.21.15.3:0/2433447196 shutdown_connections 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56120426c7a0 0x56120426eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.686 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042639d0 0x56120426c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 --2- 172.21.15.3:0/2433447196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042630b0 0x561204263490 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 -- 172.21.15.3:0/2433447196 >> 172.21.15.3:0/2433447196 conn(0x56120425d9d0 msgr2=0x56120425ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 -- 172.21.15.3:0/2433447196 shutdown_connections 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 -- 172.21.15.3:0/2433447196 wait complete. 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 -- start start 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.676+0000 7fe022a43740 1 Processor -- start 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042630b0 0x561204269720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042639d0 0x561204269c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 0x56120426a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120416fb90 con 0x56120426c7a0 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561203e9e250 con 0x5612042639d0 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561204150200 con 0x5612042630b0 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe022a43740 10 client.0 _ll_register_callbacks cb 0x5612041865d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01e40c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 0x56120426a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.687 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01e40c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 0x56120426a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55140/0 (socket says 172.21.15.3:55140) 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01e40c640 1 -- 172.21.15.3:0/3196518713 learned_addr learned my addr 172.21.15.3:0/3196518713 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042639d0 0x561204269c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042630b0 0x561204269720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3196518713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042630b0 msgr2=0x561204269720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042630b0 0x561204269720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3196518713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 msgr2=0x56120426a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 0x56120426a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3196518713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01001b040 con 0x5612042639d0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.677+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 0x56120426a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042639d0 0x561204269c60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe014009230 tx=0x7fe01402ca10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7698da6e80326c8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe0017fa640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe0017fa640 1 -- 172.21.15.3:0/3196518713 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014004040 con 0x5612042639d0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe0017fa640 1 -- 172.21.15.3:0/3196518713 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe0140041a0 con 0x5612042639d0 2024-02-09T22:58:17.688 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561203eca600 con 0x5612042639d0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe0017fa640 1 -- 172.21.15.3:0/3196518713 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe014004360 con 0x5612042639d0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 10 client.27164 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x5612041c8d90 con 0x5612042639d0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe0017fa640 1 -- 172.21.15.3:0/3196518713 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x561203eca600 con 0x5612042639d0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe0017fa640 1 -- 172.21.15.3:0/3196518713 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe01403c040 con 0x5612042639d0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 10 client.27164 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 10 client.27164 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 20 client.27164 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56120427c8d0 0x56120427ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 4 client.27164 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.678+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561203f1b1b0 con 0x56120427c8d0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.679+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56120427c8d0 0x56120427ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.679+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56120427c8d0 0x56120427ecf0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3faea5f8476ae7f4 server_cookie=f08efdcab537c4ec in_seq=0 out_seq=0 2024-02-09T22:58:17.689 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.679+0000 7fe0017fa640 10 client.27164 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe0017fa640 1 -- 172.21.15.3:0/3196518713 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561203f1b1b0 con 0x56120427c8d0 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe0017fa640 10 client.27164 handle_command_reply: tid=0 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 client.27164 shutdown 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042639d0 msgr2=0x561204269c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042639d0 0x561204269c60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe014009230 tx=0x7fe01402ca10 comp rx=0 tx=0).stop 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 shutdown_connections 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56120427c8d0 0x56120427ecf0 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042639d0 0x561204269c60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56120426c7a0 0x56120426a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 --2- 172.21.15.3:0/3196518713 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5612042630b0 0x561204269720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe0017fa640 0 client.27164 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 >> 172.21.15.3:0/3196518713 conn(0x56120425d9d0 msgr2=0x56120425f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 shutdown_connections 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 1 -- 172.21.15.3:0/3196518713 wait complete. 2024-02-09T22:58:17.690 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.680+0000 7fe022a43740 20 client.27164 trim_cache size 0 max 16384 2024-02-09T22:58:17.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:17 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:17.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:17 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:17.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:17 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:17.691 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:17 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 Processor -- start 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 -- start start 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x56120426be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204269fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x561204267ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120427c6c0 con 0x5612042639d0 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120425d500 con 0x5612042630b0 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe022a43740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56120425e9d0 con 0x56120426c7a0 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204269fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204269fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55146/0 (socket says 172.21.15.3:55146) 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3232362365 learned_addr learned my addr 172.21.15.3:0/3232362365 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:17.702 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x56120426be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x561204267ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3232362365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 msgr2=0x561204267ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x561204267ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3232362365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 msgr2=0x56120426be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x56120426be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01dc0b640 1 -- 172.21.15.3:0/3232362365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561204165820 con 0x5612042639d0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.694+0000 7fe01e40c640 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x561204267ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.695+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204269fc0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fe01402c170 tx=0x7fe01402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd20f3990853230 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.695+0000 7fe0037fe640 1 -- 172.21.15.3:0/3232362365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01402c5d0 con 0x5612042639d0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.695+0000 7fe0037fe640 1 -- 172.21.15.3:0/3232362365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0140523e0 con 0x5612042639d0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.695+0000 7fe0037fe640 1 -- 172.21.15.3:0/3232362365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe014052900 con 0x5612042639d0 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.695+0000 7fe022a43740 1 -- 172.21.15.3:0/3232362365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 msgr2=0x561204269fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.703 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.695+0000 7fe022a43740 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204269fc0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7fe01402c170 tx=0x7fe01402bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- 172.21.15.3:0/3232362365 shutdown_connections 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x561204267ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204269fc0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 --2- 172.21.15.3:0/3232362365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x56120426be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- 172.21.15.3:0/3232362365 >> 172.21.15.3:0/3232362365 conn(0x56120425d9d0 msgr2=0x561204283650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- 172.21.15.3:0/3232362365 shutdown_connections 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- 172.21.15.3:0/3232362365 wait complete. 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- start start 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 Processor -- start 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x561204271660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204271ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x5612042cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612042cc2e0 con 0x5612042639d0 2024-02-09T22:58:17.704 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612042cc450 con 0x5612042630b0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5612042cc610 con 0x56120426c7a0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.696+0000 7fe022a43740 10 client.0 _ll_register_callbacks cb 0x7fe014009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204271ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01e40c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x561204271660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01e40c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x561204271660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:45856/0 (socket says 172.21.15.3:45856) 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204271ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55154/0 (socket says 172.21.15.3:55154) 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01e40c640 1 -- 172.21.15.3:0/359143964 learned_addr learned my addr 172.21.15.3:0/359143964 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01e40c640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x5612042cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 -- 172.21.15.3:0/359143964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 msgr2=0x5612042cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x5612042cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 -- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 msgr2=0x561204271660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x561204271660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 -- 172.21.15.3:0/359143964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5612041c8d90 con 0x5612042639d0 2024-02-09T22:58:17.705 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01e40c640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x561204271660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe01dc0b640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204271ba0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x56120426c4c0 tx=0x7fe014004230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a40180a3ad458f2e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe0017fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe0017fa640 1 -- 172.21.15.3:0/359143964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe01402c9f0 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe0017fa640 1 -- 172.21.15.3:0/359143964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe014000e40 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5612041f0ee0 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe0017fa640 1 -- 172.21.15.3:0/359143964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0140038b0 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe022a43740 10 client.18120 fetch_fsmap learned FSMap version 40 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.697+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=4+}) v3 -- 0x56120418b6f0 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe0017fa640 1 -- 172.21.15.3:0/359143964 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5612041f0ee0 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe0017fa640 1 -- 172.21.15.3:0/359143964 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 40) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fe01403c040 con 0x5612042639d0 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe022a43740 10 client.18120 fetch_fsmap finished waiting for FSMap version 40 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe022a43740 10 client.18120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe022a43740 20 client.18120 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:17.706 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe022a43740 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5612042d3a60 0x5612042d5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe022a43740 4 client.18120 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:58:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x561203f07550 con 0x5612042d3a60 2024-02-09T22:58:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe01e40c640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5612042d3a60 0x5612042d5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe0017fa640 10 client.18120 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:17.707 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.698+0000 7fe01e40c640 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5612042d3a60 0x5612042d5e80 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acd1729c2233b38d server_cookie=e13154ed56bb8ca2 in_seq=0 out_seq=0 2024-02-09T22:58:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:17 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:17 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:17 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:17.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:17 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.727+0000 7fe0017fa640 1 -- 172.21.15.3:0/359143964 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x561203f07550 con 0x5612042d3a60 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.727+0000 7fe0017fa640 10 client.18120 handle_command_reply: tid=0 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.727+0000 7fe022a43740 1 client.18120 shutdown 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 msgr2=0x561204271ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204271ba0 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x56120426c4c0 tx=0x7fe014004230 comp rx=0 tx=0).stop 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 shutdown_connections 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5612042d3a60 0x5612042d5e80 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56120426c7a0 0x5612042cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe0017fa640 0 client.18120 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5612042639d0 0x561204271ba0 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 --2- 172.21.15.3:0/359143964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5612042630b0 0x561204271660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 >> 172.21.15.3:0/359143964 conn(0x56120425d9d0 msgr2=0x561204267f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 shutdown_connections 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 1 -- 172.21.15.3:0/359143964 wait complete. 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.728+0000 7fe022a43740 20 client.18120 trim_cache size 0 max 16384 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:58:17.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:58:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:58:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:58:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0040a2220 msgr2=0x7fe0040a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0040a2220 0x7fe0040a46e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe008000c00 tx=0x7fe008005880 comp rx=0 tx=0).stop 2024-02-09T22:58:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 msgr2=0x7fe01819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:17.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 0x7fe01819db30 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 crypto rx=0x7fe00800e9c0 tx=0x7fe00800ee80 comp rx=0 tx=0).stop 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 shutdown_connections 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe0040a2220 0x7fe0040a46e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe018197700 0x7fe018197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe018105410 0x7fe01819db30 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 --2- 172.21.15.3:0/4244549217 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe018100340 0x7fe01819d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 >> 172.21.15.3:0/4244549217 conn(0x7fe018077220 msgr2=0x7fe0181040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 shutdown_connections 2024-02-09T22:58:17.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:17.729+0000 7fe01fe6f640 1 -- 172.21.15.3:0/4244549217 wait complete. 2024-02-09T22:58:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-mon[36638]: pgmap v216: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 937 KiB/s wr, 108 op/s 2024-02-09T22:58:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:18.435 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-mon[36638]: daemon mds.e restarted 2024-02-09T22:58:18.436 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setuser ceph since I am not root 2024-02-09T22:58:18.436 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setgroup ceph since I am not root 2024-02-09T22:58:18.436 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-mds[66607]: -- 172.21.15.64:0/1973620834 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55722652e1a0 con 0x557226505000 2024-02-09T22:58:18.436 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:58:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: starting mds.e at 2024-02-09T22:58:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:18 smithi003 ceph-mon[32678]: pgmap v216: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 937 KiB/s wr, 108 op/s 2024-02-09T22:58:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:18 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:18 smithi003 ceph-mon[32678]: daemon mds.e restarted 2024-02-09T22:58:18.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202852 2.66 MB/sec execute 252 sec latency 592.319 ms 2024-02-09T22:58:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:18 smithi089 ceph-mon[36665]: pgmap v216: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 937 KiB/s wr, 108 op/s 2024-02-09T22:58:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:18 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:18 smithi089 ceph-mon[36665]: daemon mds.e restarted 2024-02-09T22:58:19.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:19 smithi003 ceph-mon[32678]: osdmap e92: 12 total, 12 up, 12 in 2024-02-09T22:58:19.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:19 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6834/3552345002,v1:172.21.15.64:6835/3552345002] up:boot 2024-02-09T22:58:19.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:19 smithi003 ceph-mon[32678]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T22:58:19.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:19 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:58:19.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:19 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:19.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202852 2.64 MB/sec execute 253 sec latency 1592.365 ms 2024-02-09T22:58:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:19 smithi064 ceph-mon[36638]: osdmap e92: 12 total, 12 up, 12 in 2024-02-09T22:58:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:19 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6834/3552345002,v1:172.21.15.64:6835/3552345002] up:boot 2024-02-09T22:58:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:19 smithi064 ceph-mon[36638]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T22:58:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:19 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:58:19.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:19 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:19 smithi089 ceph-mon[36665]: osdmap e92: 12 total, 12 up, 12 in 2024-02-09T22:58:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:19 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6834/3552345002,v1:172.21.15.64:6835/3552345002] up:boot 2024-02-09T22:58:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:19 smithi089 ceph-mon[36665]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T22:58:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:19 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:58:19.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:19 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:20.628 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:20 smithi089 ceph-mon[36665]: pgmap v218: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1024 KiB/s wr, 118 op/s 2024-02-09T22:58:20.638 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:20 smithi003 ceph-mon[32678]: pgmap v218: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1024 KiB/s wr, 118 op/s 2024-02-09T22:58:20.646 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202852 2.63 MB/sec execute 254 sec latency 2592.431 ms 2024-02-09T22:58:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:20 smithi064 ceph-mon[36638]: pgmap v218: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1024 KiB/s wr, 118 op/s 2024-02-09T22:58:21.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202852 2.62 MB/sec execute 255 sec latency 3592.528 ms 2024-02-09T22:58:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:22 smithi003 ceph-mon[32678]: pgmap v219: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:22.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202852 2.61 MB/sec execute 256 sec latency 4592.632 ms 2024-02-09T22:58:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:22 smithi064 ceph-mon[36638]: pgmap v219: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:22 smithi089 ceph-mon[36665]: pgmap v219: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:23.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202908 2.60 MB/sec execute 257 sec latency 4997.070 ms 2024-02-09T22:58:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:24 smithi003 ceph-mon[32678]: pgmap v220: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:24.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202908 2.59 MB/sec execute 258 sec latency 1554.920 ms 2024-02-09T22:58:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:24 smithi064 ceph-mon[36638]: pgmap v220: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:24 smithi089 ceph-mon[36665]: pgmap v220: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:25 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:58:25.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202908 2.58 MB/sec execute 259 sec latency 2554.970 ms 2024-02-09T22:58:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:25 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:58:25.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:25 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:58:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:26 smithi003 ceph-mon[32678]: pgmap v221: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:26 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:58:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:26 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:58:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:26 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:58:26.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202908 2.57 MB/sec execute 260 sec latency 3555.071 ms 2024-02-09T22:58:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:26 smithi064 ceph-mon[36638]: pgmap v221: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:26 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:58:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:26 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:58:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:26 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:58:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:26 smithi089 ceph-mon[36665]: pgmap v221: 97 pgs: 97 active+clean; 77 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.1 MiB/s wr, 138 op/s 2024-02-09T22:58:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:26 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:58:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:26 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:58:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:26 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:58:27.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202908 2.56 MB/sec execute 261 sec latency 4555.166 ms 2024-02-09T22:58:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:28 smithi003 ceph-mon[32678]: pgmap v222: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 446 KiB/s wr, 73 op/s 2024-02-09T22:58:28.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202976 2.55 MB/sec execute 262 sec latency 4956.232 ms 2024-02-09T22:58:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:28 smithi064 ceph-mon[36638]: pgmap v222: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 446 KiB/s wr, 73 op/s 2024-02-09T22:58:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:28 smithi089 ceph-mon[36665]: pgmap v222: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 446 KiB/s wr, 73 op/s 2024-02-09T22:58:29.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202976 2.55 MB/sec execute 263 sec latency 1554.195 ms 2024-02-09T22:58:30.477 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:30 smithi089 ceph-mon[36665]: pgmap v223: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 439 KiB/s wr, 72 op/s 2024-02-09T22:58:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:30 smithi003 ceph-mon[32678]: pgmap v223: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 439 KiB/s wr, 72 op/s 2024-02-09T22:58:30.647 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202976 2.54 MB/sec execute 264 sec latency 2554.296 ms 2024-02-09T22:58:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:30 smithi064 ceph-mon[36638]: pgmap v223: 97 pgs: 97 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 346 KiB/s rd, 439 KiB/s wr, 72 op/s 2024-02-09T22:58:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 -- 172.21.15.3:0/3692659632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40101470 msgr2=0x7f7a40101850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.840 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 --2- 172.21.15.3:0/3692659632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40101470 0x7f7a40101850 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009930 tx=0x7f7a3c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 -- 172.21.15.3:0/3692659632 shutdown_connections 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 --2- 172.21.15.3:0/3692659632 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40106540 0x7f7a4010e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 --2- 172.21.15.3:0/3692659632 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40101e20 0x7f7a40105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 --2- 172.21.15.3:0/3692659632 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40101470 0x7f7a40101850 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 -- 172.21.15.3:0/3692659632 >> 172.21.15.3:0/3692659632 conn(0x7f7a400fccf0 msgr2=0x7f7a400ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 -- 172.21.15.3:0/3692659632 shutdown_connections 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.826+0000 7f7a4772c640 1 -- 172.21.15.3:0/3692659632 wait complete. 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 Processor -- start 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 -- start start 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101470 0x7f7a40073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 0x7f7a40073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40073fb0 0x7f7a401a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.841 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401a6060 con 0x7f7a40106540 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401a61d0 con 0x7f7a40101470 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a4772c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a401a6390 con 0x7f7a40073fb0 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 0x7f7a40073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 0x7f7a40073a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47786/0 (socket says 172.21.15.3:47786) 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 -- 172.21.15.3:0/988531609 learned_addr learned my addr 172.21.15.3:0/988531609 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40073fb0 0x7f7a401a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a454c8640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101470 0x7f7a40073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 -- 172.21.15.3:0/988531609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40073fb0 msgr2=0x7f7a401a5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40073fb0 0x7f7a401a5a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 -- 172.21.15.3:0/988531609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101470 msgr2=0x7f7a40073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101470 0x7f7a40073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a44cc7640 1 -- 172.21.15.3:0/988531609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a3c009560 con 0x7f7a40106540 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.827+0000 7f7a454c8640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101470 0x7f7a40073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:30.842 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.828+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 0x7f7a40073a70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f7a30005fd0 tx=0x7f7a3000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.828+0000 7f7a367fc640 1 -- 172.21.15.3:0/988531609 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30009c30 con 0x7f7a40106540 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.828+0000 7f7a367fc640 1 -- 172.21.15.3:0/988531609 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a30009d90 con 0x7f7a40106540 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.828+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a401a6670 con 0x7f7a40106540 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.828+0000 7f7a367fc640 1 -- 172.21.15.3:0/988531609 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30010460 con 0x7f7a40106540 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.828+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a401a6b80 con 0x7f7a40106540 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.830+0000 7f7a367fc640 1 -- 172.21.15.3:0/988531609 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7a30003f20 con 0x7f7a40106540 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.830+0000 7f7a4a3de740 1 Processor -- start 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 -- start start 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9ef490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a367fc640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7a2c0a2250 0x7f7a2c0a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.843 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9fab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d95da00 con 0x55e19d9ef0b0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d95af30 con 0x55e19d9ef9d0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d956710 con 0x55e19d9f87a0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9ef490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a454c8640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7a2c0a2250 0x7f7a2c0a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9ef490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47794/0 (socket says 172.21.15.3:47794) 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 -- 172.21.15.3:0/2175327972 learned_addr learned my addr 172.21.15.3:0/2175327972 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a367fc640 1 -- 172.21.15.3:0/988531609 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7a300cdda0 con 0x7f7a40106540 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9fab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 -- 172.21.15.3:0/2175327972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 msgr2=0x55e19d9fab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.844 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9fab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 -- 172.21.15.3:0/2175327972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 msgr2=0x55e19d9f8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a45cc9640 1 -- 172.21.15.3:0/2175327972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e19d91b3b0 con 0x55e19d9ef0b0 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.831+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9ef490 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009fa0 tx=0x7f7a38009870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db879097d433962 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a454c8640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7a2c0a2250 0x7f7a2c0a4710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c005ec0 tx=0x7f7a3c000bc0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a2b7fe640 1 -- 172.21.15.3:0/2175327972 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a38014d60 con 0x55e19d9ef0b0 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a2b7fe640 1 -- 172.21.15.3:0/2175327972 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a380128e0 con 0x55e19d9ef0b0 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a2b7fe640 1 -- 172.21.15.3:0/2175327972 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a380053e0 con 0x55e19d9ef0b0 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2175327972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 msgr2=0x55e19d9ef490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.832+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9ef490 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7a38009fa0 tx=0x7f7a38009870 comp rx=0 tx=0).stop 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2175327972 shutdown_connections 2024-02-09T22:58:30.845 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9fab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2175327972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9ef490 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2175327972 >> 172.21.15.3:0/2175327972 conn(0x55e19d9e99d0 msgr2=0x55e19d9e9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2175327972 shutdown_connections 2024-02-09T22:58:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2175327972 wait complete. 2024-02-09T22:58:30.846 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 -- start start 2024-02-09T22:58:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.833+0000 7f7a4a3de740 1 Processor -- start 2024-02-09T22:58:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19d9f6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d8fbb90 con 0x55e19d9ef9d0 2024-02-09T22:58:30.847 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d62a250 con 0x55e19d9f87a0 2024-02-09T22:58:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d8dc200 con 0x55e19d9ef0b0 2024-02-09T22:58:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 10 client.0 _ll_register_callbacks cb 0x55e19d9125d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.848 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47796/0 (socket says 172.21.15.3:47796) 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 -- 172.21.15.3:0/531323936 learned_addr learned my addr 172.21.15.3:0/531323936 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a454c8640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19d9f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 -- 172.21.15.3:0/531323936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 msgr2=0x55e19d9f56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 -- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 msgr2=0x55e19d9f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19d9f6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 -- 172.21.15.3:0/531323936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a38014410 con 0x55e19d9ef9d0 2024-02-09T22:58:30.849 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a454c8640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5c00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7a30097190 tx=0x7f7a3009b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56ade36289a0817b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a28ff9640 1 -- 172.21.15.3:0/531323936 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3009a7c0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a28ff9640 1 -- 172.21.15.3:0/531323936 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7a3009b150 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.834+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e19d97cee0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a28ff9640 1 -- 172.21.15.3:0/531323936 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a3009baf0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 10 client.18138 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55e19d954d90 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a28ff9640 1 -- 172.21.15.3:0/531323936 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e19d97cee0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a28ff9640 1 -- 172.21.15.3:0/531323936 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7a3009bd20 con 0x55e19d9ef9d0 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 10 client.18138 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 10 client.18138 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 20 client.18138 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da088d0 0x55e19da0acf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.850 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 4 client.18138 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e19d656600 con 0x55e19da088d0 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.835+0000 7f7a454c8640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da088d0 0x55e19da0acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.836+0000 7f7a454c8640 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da088d0 0x55e19da0acf0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8666ec3f84dbf1e server_cookie=c32aa97c3f12ca86 in_seq=0 out_seq=0 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.836+0000 7f7a28ff9640 10 client.18138 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.836+0000 7f7a28ff9640 1 -- 172.21.15.3:0/531323936 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e19d656600 con 0x55e19da088d0 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.836+0000 7f7a28ff9640 10 client.18138 handle_command_reply: tid=0 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.836+0000 7f7a4a3de740 1 client.18138 shutdown 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 msgr2=0x55e19d9f5c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5c00 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7a30097190 tx=0x7f7a3009b8f0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 shutdown_connections 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da088d0 0x55e19da0acf0 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19d9f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.851 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5c00 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/531323936 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a28ff9640 0 client.18138 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 >> 172.21.15.3:0/531323936 conn(0x55e19d9e99d0 msgr2=0x55e19d9eb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 shutdown_connections 2024-02-09T22:58:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 1 -- 172.21.15.3:0/531323936 wait complete. 2024-02-09T22:58:30.852 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.837+0000 7f7a4a3de740 20 client.18138 trim_cache size 0 max 16384 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.850+0000 7f7a4a3de740 1 Processor -- start 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.850+0000 7f7a4a3de740 1 -- start start 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19da086f0 con 0x55e19d9ef0b0 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d9fabc0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19d9f12d0 con 0x55e19d9f87a0 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47800/0 (socket says 172.21.15.3:47800) 2024-02-09T22:58:30.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 -- 172.21.15.3:0/2877633615 learned_addr learned my addr 172.21.15.3:0/2877633615 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 -- 172.21.15.3:0/2877633615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 msgr2=0x55e19d9f2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9f2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 -- 172.21.15.3:0/2877633615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 msgr2=0x55e19d9f5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 -- 172.21.15.3:0/2877633615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e19d8f1820 con 0x55e19d9ef0b0 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f7df0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009900 tx=0x7f7a3c03d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=423c0702dcf198c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a2affd640 1 -- 172.21.15.3:0/2877633615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c056070 con 0x55e19d9ef0b0 2024-02-09T22:58:30.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a2affd640 1 -- 172.21.15.3:0/2877633615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a3c051070 con 0x55e19d9ef0b0 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.851+0000 7f7a2affd640 1 -- 172.21.15.3:0/2877633615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a3c003a00 con 0x55e19d9ef0b0 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2877633615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 msgr2=0x55e19d9f7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f7df0 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c009900 tx=0x7f7a3c03d000 comp rx=0 tx=0).stop 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2877633615 shutdown_connections 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9ef9d0 0x55e19d9f5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9f87a0 0x55e19d9f2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2877633615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef0b0 0x55e19d9f7df0 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2877633615 >> 172.21.15.3:0/2877633615 conn(0x55e19d9e99d0 msgr2=0x55e19da0f670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2877633615 shutdown_connections 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2877633615 wait complete. 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 -- start start 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.852+0000 7f7a4a3de740 1 Processor -- start 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9fd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9fdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19da5ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19da5e330 con 0x55e19d9ef9d0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19da5e470 con 0x55e19d9f87a0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e19da5e5e0 con 0x55e19d9ef0b0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a4a3de740 10 client.0 _ll_register_callbacks cb 0x7f7a3000df10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9fdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9fdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:47814/0 (socket says 172.21.15.3:47814) 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 -- 172.21.15.3:0/2717168077 learned_addr learned my addr 172.21.15.3:0/2717168077 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a45cc9640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19da5ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9fd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 -- 172.21.15.3:0/2717168077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 msgr2=0x55e19d9fd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9fd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 -- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 msgr2=0x55e19da5ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:30.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19da5ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a44cc7640 1 -- 172.21.15.3:0/2717168077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4010e920 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.853+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9fd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a44cc7640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9fdbf0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009610 tx=0x7f7a30005b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c44494d5d3ffa006 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a28ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a28ff9640 1 -- 172.21.15.3:0/2717168077 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a300b32f0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a28ff9640 1 -- 172.21.15.3:0/2717168077 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f7a3009aec0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a380128e0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a28ff9640 1 -- 172.21.15.3:0/2717168077 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a3009a7c0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 10 client.18144 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55e19d9176f0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a28ff9640 1 -- 172.21.15.3:0/2717168077 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a380128e0 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a28ff9640 1 -- 172.21.15.3:0/2717168077 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f7a30022020 con 0x55e19d9ef9d0 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 10 client.18144 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 10 client.18144 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:30.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 20 client.18144 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da5fb70 0x55e19da61f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 4 client.18144 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:58:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.854+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e19d6a71b0 con 0x55e19da5fb70 2024-02-09T22:58:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.855+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da5fb70 0x55e19da61f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.855+0000 7f7a454c8640 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da5fb70 0x55e19da61f90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fdc8952100758df server_cookie=37bb68040266d99f in_seq=0 out_seq=0 2024-02-09T22:58:30.864 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.855+0000 7f7a28ff9640 10 client.18144 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:30.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a28ff9640 1 -- 172.21.15.3:0/2717168077 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e19d6a71b0 con 0x55e19da5fb70 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a28ff9640 10 client.18144 handle_command_reply: tid=0 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 client.18144 shutdown 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 msgr2=0x55e19d9fdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9fdbf0 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009610 tx=0x7f7a30005b20 comp rx=0 tx=0).stop 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 shutdown_connections 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e19da5fb70 0x55e19da61f90 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e19d9f87a0 0x55e19da5ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a28ff9640 0 client.18144 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e19d9ef9d0 0x55e19d9fdbf0 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 --2- 172.21.15.3:0/2717168077 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e19d9ef0b0 0x55e19d9fd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 >> 172.21.15.3:0/2717168077 conn(0x55e19d9e99d0 msgr2=0x55e19d9f3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 shutdown_connections 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 1 -- 172.21.15.3:0/2717168077 wait complete. 2024-02-09T22:58:30.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.881+0000 7f7a4a3de740 20 client.18144 trim_cache size 0 max 16384 2024-02-09T22:58:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:58:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:58:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:58:30.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7a2c0a2250 msgr2=0x7f7a2c0a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7a2c0a2250 0x7f7a2c0a4710 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c005ec0 tx=0x7f7a3c000bc0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 msgr2=0x7f7a40073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 0x7f7a40073a70 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7f7a30005fd0 tx=0x7f7a3000dde0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 shutdown_connections 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7a2c0a2250 0x7f7a2c0a4710 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7a40073fb0 0x7f7a401a5a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7a40106540 0x7f7a40073a70 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 --2- 172.21.15.3:0/988531609 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7a40101470 0x7f7a40073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 >> 172.21.15.3:0/988531609 conn(0x7f7a400fccf0 msgr2=0x7f7a400fe0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 shutdown_connections 2024-02-09T22:58:30.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:30.882+0000 7f7a4772c640 1 -- 172.21.15.3:0/988531609 wait complete. 2024-02-09T22:58:31.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202976 2.53 MB/sec execute 265 sec latency 3554.412 ms 2024-02-09T22:58:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:32 smithi003 ceph-mon[32678]: pgmap v224: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 881 KiB/s wr, 83 op/s 2024-02-09T22:58:32.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 202976 2.52 MB/sec execute 266 sec latency 4554.484 ms 2024-02-09T22:58:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:32 smithi064 ceph-mon[36638]: pgmap v224: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 881 KiB/s wr, 83 op/s 2024-02-09T22:58:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:32 smithi089 ceph-mon[36665]: pgmap v224: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 330 KiB/s rd, 881 KiB/s wr, 83 op/s 2024-02-09T22:58:33.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 203778 2.52 MB/sec execute 267 sec latency 4959.048 ms 2024-02-09T22:58:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:34 smithi003 ceph-mon[32678]: pgmap v225: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 647 KiB/s wr, 40 op/s 2024-02-09T22:58:34.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 203878 2.52 MB/sec execute 268 sec latency 846.895 ms 2024-02-09T22:58:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:34 smithi064 ceph-mon[36638]: pgmap v225: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 647 KiB/s wr, 40 op/s 2024-02-09T22:58:34.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:34 smithi089 ceph-mon[36665]: pgmap v225: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 647 KiB/s wr, 40 op/s 2024-02-09T22:58:35.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 203878 2.51 MB/sec execute 269 sec latency 1846.987 ms 2024-02-09T22:58:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:36 smithi003 ceph-mon[32678]: pgmap v226: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 647 KiB/s wr, 40 op/s 2024-02-09T22:58:36.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 203878 2.50 MB/sec execute 270 sec latency 2847.103 ms 2024-02-09T22:58:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:36 smithi064 ceph-mon[36638]: pgmap v226: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 647 KiB/s wr, 40 op/s 2024-02-09T22:58:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:36 smithi089 ceph-mon[36665]: pgmap v226: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 647 KiB/s wr, 40 op/s 2024-02-09T22:58:37.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 205876 2.53 MB/sec execute 271 sec latency 2885.969 ms 2024-02-09T22:58:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:38 smithi003 ceph-mon[32678]: pgmap v227: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 901 KiB/s wr, 80 op/s 2024-02-09T22:58:38.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 207288 2.55 MB/sec execute 272 sec latency 12.212 ms 2024-02-09T22:58:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:38 smithi064 ceph-mon[36638]: pgmap v227: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 901 KiB/s wr, 80 op/s 2024-02-09T22:58:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:38 smithi089 ceph-mon[36665]: pgmap v227: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 901 KiB/s wr, 80 op/s 2024-02-09T22:58:39.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 207421 2.54 MB/sec execute 273 sec latency 922.174 ms 2024-02-09T22:58:40.423 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:58:40.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 -- 172.21.15.3:0/966814856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c106580 msgr2=0x7fb67c10e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 --2- 172.21.15.3:0/966814856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c106580 0x7fb67c10e130 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb674009cd0 tx=0x7fb67402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 -- 172.21.15.3:0/966814856 shutdown_connections 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 --2- 172.21.15.3:0/966814856 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb67c06a5f0 0x7fb67c105ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 --2- 172.21.15.3:0/966814856 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c106580 0x7fb67c10e130 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 --2- 172.21.15.3:0/966814856 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c069c40 0x7fb67c06a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 -- 172.21.15.3:0/966814856 >> 172.21.15.3:0/966814856 conn(0x7fb67c077210 msgr2=0x7fb67c077620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 -- 172.21.15.3:0/966814856 shutdown_connections 2024-02-09T22:58:40.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.536+0000 7fb680b93640 1 -- 172.21.15.3:0/966814856 wait complete. 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 Processor -- start 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 -- start start 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 0x7fb67c19d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c106580 0x7fb67c19dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb67c197630 0x7fb67c197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c198100 con 0x7fb67c069c40 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c198270 con 0x7fb67c197630 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb680b93640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c198430 con 0x7fb67c106580 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.537+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 0x7fb67c19d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 0x7fb67c19d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60688/0 (socket says 172.21.15.3:60688) 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 -- 172.21.15.3:0/2656851139 learned_addr learned my addr 172.21.15.3:0/2656851139 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb673fff640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c106580 0x7fb67c19dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb67c197630 0x7fb67c197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 -- 172.21.15.3:0/2656851139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c106580 msgr2=0x7fb67c19dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c106580 0x7fb67c19dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 -- 172.21.15.3:0/2656851139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb67c197630 msgr2=0x7fb67c197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb67c197630 0x7fb67c197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 -- 172.21.15.3:0/2656851139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb66c009490 con 0x7fb67c069c40 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb673fff640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c106580 0x7fb67c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb67a59c640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 0x7fb67c19d570 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00ea60 tx=0x7fb66c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb6737fe640 1 -- 172.21.15.3:0/2656851139 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c00cde0 con 0x7fb67c069c40 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb6737fe640 1 -- 172.21.15.3:0/2656851139 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c0035f0 con 0x7fb67c069c40 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb674009900 con 0x7fb67c069c40 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb6737fe640 1 -- 172.21.15.3:0/2656851139 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c010690 con 0x7fb67c069c40 2024-02-09T22:58:40.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.538+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c1a4720 con 0x7fb67c069c40 2024-02-09T22:58:40.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.540+0000 7fb6737fe640 1 -- 172.21.15.3:0/2656851139 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb66c003af0 con 0x7fb67c069c40 2024-02-09T22:58:40.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 Processor -- start 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 -- start start 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb6737fe640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb6680a2230 0x7fb6680a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff31d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff32330 0x55861ff3a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff3aeb0 0x55861ff3d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb6737fe640 1 -- 172.21.15.3:0/2656851139 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb66c014070 con 0x7fb67c069c40 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861feecd00 con 0x55861ff32330 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fea12f0 con 0x55861ff3aeb0 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb68376d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fe9e820 con 0x55861ff31980 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb6680a2230 0x7fb6680a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb673fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff32330 0x55861ff3a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff31d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb673fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff32330 0x55861ff3a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60700/0 (socket says 172.21.15.3:60700) 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.541+0000 7fb673fff640 1 -- 172.21.15.3:0/2102744550 learned_addr learned my addr 172.21.15.3:0/2102744550 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff3aeb0 0x55861ff3d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67a59c640 1 -- 172.21.15.3:0/2102744550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff3aeb0 msgr2=0x55861ff3d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67a59c640 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff3aeb0 0x55861ff3d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67a59c640 1 -- 172.21.15.3:0/2102744550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff32330 msgr2=0x55861ff3a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67a59c640 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff32330 0x55861ff3a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67a59c640 1 -- 172.21.15.3:0/2102744550 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55861fe5bff0 con 0x55861ff31980 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67a59c640 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff31d60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0cbc50 tx=0x7fb66c09a910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83467c3603f1dbcf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb6680a2230 0x7fb6680a46f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb674005cc0 tx=0x7fb6740325d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb670ff9640 1 -- 172.21.15.3:0/2102744550 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c09f040 con 0x55861ff31980 2024-02-09T22:58:40.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb670ff9640 1 -- 172.21.15.3:0/2102744550 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c0a3690 con 0x55861ff31980 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.542+0000 7fb670ff9640 1 -- 172.21.15.3:0/2102744550 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0ce070 con 0x55861ff31980 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 -- 172.21.15.3:0/2102744550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 msgr2=0x55861ff31d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff31d60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fb66c0cbc50 tx=0x7fb66c09a910 comp rx=0 tx=0).stop 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 -- 172.21.15.3:0/2102744550 shutdown_connections 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff3aeb0 0x55861ff3d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff32330 0x55861ff3a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- 172.21.15.3:0/2102744550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff31d60 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 -- 172.21.15.3:0/2102744550 >> 172.21.15.3:0/2102744550 conn(0x55861ff2c050 msgr2=0x55861ff2c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 -- 172.21.15.3:0/2102744550 shutdown_connections 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 -- 172.21.15.3:0/2102744550 wait complete. 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 -- start start 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 Processor -- start 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff37dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff38300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.543+0000 7fb68376d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff38840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb68376d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fe9a000 con 0x55861ff31980 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb68376d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fb6d430 con 0x55861ff32330 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb68376d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fb29e00 con 0x55861ff3aeb0 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb68376d740 10 client.0 _ll_register_callbacks cb 0x55861fe8e7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff37dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff38840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff38840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35670/0 (socket says 172.21.15.3:35670) 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 -- 172.21.15.3:0/3619346421 learned_addr learned my addr 172.21.15.3:0/3619346421 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb673fff640 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff38300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 -- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 msgr2=0x55861ff38300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff38300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 -- 172.21.15.3:0/3619346421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 msgr2=0x55861ff37dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff37dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 -- 172.21.15.3:0/3619346421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c1a4720 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff38840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb674036000 tx=0x7fb67402c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e24b019f4b868f37 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb661ffb640 1 -- 172.21.15.3:0/3619346421 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674003b70 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb661ffb640 1 -- 172.21.15.3:0/3619346421 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb674003cd0 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.544+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55861fb997e0 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb661ffb640 1 -- 172.21.15.3:0/3619346421 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb674003e90 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 10 client.27325 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55861fe98680 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb661ffb640 1 -- 172.21.15.3:0/3619346421 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55861fb997e0 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb661ffb640 1 -- 172.21.15.3:0/3619346421 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fb674044600 con 0x55861ff3aeb0 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 10 client.27325 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 10 client.27325 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 20 client.27325 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:40.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ff4b4e0 0x55861ff4d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 4 client.27325 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55861fbd6720 con 0x55861ff4b4e0 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb67a59c640 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ff4b4e0 0x55861ff4d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb661ffb640 10 client.27325 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.545+0000 7fb67a59c640 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ff4b4e0 0x55861ff4d900 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4f176c2bd94921a server_cookie=e28abe94fba51095 in_seq=0 out_seq=0 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb661ffb640 1 -- 172.21.15.3:0/3619346421 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55861fbd6720 con 0x55861ff4b4e0 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb661ffb640 10 client.27325 handle_command_reply: tid=0 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 client.27325 shutdown 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 msgr2=0x55861ff38840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff38840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb674036000 tx=0x7fb67402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 shutdown_connections 2024-02-09T22:58:40.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ff4b4e0 0x55861ff4d900 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff38300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb661ffb640 0 client.27325 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff38840 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 --2- 172.21.15.3:0/3619346421 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff37dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 >> 172.21.15.3:0/3619346421 conn(0x55861ff2c050 msgr2=0x55861ff2dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 shutdown_connections 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.546+0000 7fb68376d740 1 -- 172.21.15.3:0/3619346421 wait complete. 2024-02-09T22:58:40.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.547+0000 7fb68376d740 20 client.27325 trim_cache size 0 max 16384 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 Processor -- start 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 -- start start 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff3a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff35020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff35560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fe1efd0 con 0x55861ff31980 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861fb997e0 con 0x55861ff32330 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb68376d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ff38d90 con 0x55861ff3aeb0 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff3a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff3a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60716/0 (socket says 172.21.15.3:60716) 2024-02-09T22:58:40.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 -- 172.21.15.3:0/613983024 learned_addr learned my addr 172.21.15.3:0/613983024 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb673fff640 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff35020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff35560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 -- 172.21.15.3:0/613983024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 msgr2=0x55861ff35560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff35560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 -- 172.21.15.3:0/613983024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 msgr2=0x55861ff35020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff35020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 -- 172.21.15.3:0/613983024 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55861fe34b30 con 0x55861ff31980 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.563+0000 7fb67a59c640 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff3a4f0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb66c01f000 tx=0x7fb66c09b030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de8e7b473e48adf2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb663fff640 1 -- 172.21.15.3:0/613983024 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c0840c0 con 0x55861ff31980 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb663fff640 1 -- 172.21.15.3:0/613983024 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c084220 con 0x55861ff31980 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb663fff640 1 -- 172.21.15.3:0/613983024 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c003760 con 0x55861ff31980 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 -- 172.21.15.3:0/613983024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 msgr2=0x55861ff3a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff3a4f0 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fb66c01f000 tx=0x7fb66c09b030 comp rx=0 tx=0).stop 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 -- 172.21.15.3:0/613983024 shutdown_connections 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff35020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff3aeb0 0x55861ff35560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 --2- 172.21.15.3:0/613983024 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff31980 0x55861ff3a4f0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 -- 172.21.15.3:0/613983024 >> 172.21.15.3:0/613983024 conn(0x55861ff2c050 msgr2=0x55861ff33080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 -- 172.21.15.3:0/613983024 shutdown_connections 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.564+0000 7fb68376d740 1 -- 172.21.15.3:0/613983024 wait complete. 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 -- start start 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 Processor -- start 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff41100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff41640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 0x55861ff9b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ff9bf00 con 0x55861ff3aeb0 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ff9c070 con 0x55861ff32330 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55861ff9c230 con 0x55861ff31980 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb68376d740 10 client.0 _ll_register_callbacks cb 0x7fb66c096a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 0x55861ff9b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff41100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 0x55861ff9b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60724/0 (socket says 172.21.15.3:60724) 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 -- 172.21.15.3:0/1823066795 learned_addr learned my addr 172.21.15.3:0/1823066795 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb673fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff41640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 -- 172.21.15.3:0/1823066795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 msgr2=0x55861ff41100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff41100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 -- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 msgr2=0x55861ff41640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff41640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67ad9d640 1 -- 172.21.15.3:0/1823066795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb66c0b7040 con 0x55861ff3aeb0 2024-02-09T22:58:40.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb67a59c640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff41100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.565+0000 7fb673fff640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff41640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb67ad9d640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 0x55861ff9b9c0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb674009ba0 tx=0x7fb674032e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cee4ef4cf4b97be server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb661ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb661ffb640 1 -- 172.21.15.3:0/1823066795 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb674065070 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb661ffb640 1 -- 172.21.15.3:0/1823066795 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb674032400 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66c0a3690 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb661ffb640 1 -- 172.21.15.3:0/1823066795 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb674003b80 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 10 client.18162 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55861fe5a850 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb661ffb640 1 -- 172.21.15.3:0/1823066795 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c0a3690 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb661ffb640 1 -- 172.21.15.3:0/1823066795 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fb674068030 con 0x55861ff3aeb0 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 10 client.18162 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:40.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 10 client.18162 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 20 client.18162 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ffa3780 0x55861ffa5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 4 client.18162 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.566+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55861fc25890 con 0x55861ffa3780 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.567+0000 7fb67a59c640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ffa3780 0x55861ffa5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.567+0000 7fb661ffb640 10 client.18162 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.567+0000 7fb67a59c640 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ffa3780 0x55861ffa5ba0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d58ed001d06342 server_cookie=5b4a022fc284961d in_seq=0 out_seq=0 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.567+0000 7fb661ffb640 1 -- 172.21.15.3:0/1823066795 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55861fc25890 con 0x55861ffa3780 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.567+0000 7fb661ffb640 10 client.18162 handle_command_reply: tid=0 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.567+0000 7fb68376d740 1 client.18162 shutdown 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 msgr2=0x55861ff9b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 0x55861ff9b9c0 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fb674009ba0 tx=0x7fb674032e90 comp rx=0 tx=0).stop 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 shutdown_connections 2024-02-09T22:58:40.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55861ffa3780 0x55861ffa5ba0 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55861ff32330 0x55861ff41640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55861ff3aeb0 0x55861ff9b9c0 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb661ffb640 0 client.18162 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 --2- 172.21.15.3:0/1823066795 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55861ff31980 0x55861ff41100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 >> 172.21.15.3:0/1823066795 conn(0x55861ff2c050 msgr2=0x55861ff2d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 shutdown_connections 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 1 -- 172.21.15.3:0/1823066795 wait complete. 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb68376d740 20 client.18162 trim_cache size 0 max 16384 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:58:40.577 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:58:40.578 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:58:40.578 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb6680a2230 msgr2=0x7fb6680a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb6680a2230 0x7fb6680a46f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb674005cc0 tx=0x7fb6740325d0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 msgr2=0x7fb67c19d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 0x7fb67c19d570 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7fb66c00ea60 tx=0x7fb66c00ef20 comp rx=0 tx=0).stop 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 shutdown_connections 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb6680a2230 0x7fb6680a46f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb67c197630 0x7fb67c197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb67c106580 0x7fb67c19dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 --2- 172.21.15.3:0/2656851139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb67c069c40 0x7fb67c19d570 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 >> 172.21.15.3:0/2656851139 conn(0x7fb67c077210 msgr2=0x7fb67c0fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 shutdown_connections 2024-02-09T22:58:40.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.568+0000 7fb680b93640 1 -- 172.21.15.3:0/2656851139 wait complete. 2024-02-09T22:58:40.581 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:40 smithi003 ceph-mon[32678]: pgmap v228: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 765 KiB/s wr, 62 op/s 2024-02-09T22:58:40.582 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:58:40.582 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:58:40.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 208894 2.56 MB/sec execute 274 sec latency 1371.439 ms 2024-02-09T22:58:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:40 smithi064 ceph-mon[36638]: pgmap v228: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 765 KiB/s wr, 62 op/s 2024-02-09T22:58:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3839371101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 msgr2=0x7f73e010e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3839371101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e010e0c0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f73d8009cd0 tx=0x7f73d802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3839371101 shutdown_connections 2024-02-09T22:58:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3839371101 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e010e0c0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3839371101 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73e00ffa90 0x7f73e00fff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3839371101 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73e0106230 0x7f73e0106610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3839371101 >> 172.21.15.3:0/3839371101 conn(0x7f73e0075410 msgr2=0x7f73e0075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3839371101 shutdown_connections 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.686+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3839371101 wait complete. 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 Processor -- start 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 -- start start 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73e00ffa90 0x7f73e019d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e019dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73e0106230 0x7f73e0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0197e20 con 0x7f73e0100450 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0197f90 con 0x7f73e00ffa90 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e7d0f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73e0198150 con 0x7f73e0106230 2024-02-09T22:58:40.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e52aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e019dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e52aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e019dac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60738/0 (socket says 172.21.15.3:60738) 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e52aa640 1 -- 172.21.15.3:0/3190615057 learned_addr learned my addr 172.21.15.3:0/3190615057 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e62ac640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73e0106230 0x7f73e0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.687+0000 7f73e5aab640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73e00ffa90 0x7f73e019d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e52aa640 1 -- 172.21.15.3:0/3190615057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73e0106230 msgr2=0x7f73e0197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e52aa640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73e0106230 0x7f73e0197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e52aa640 1 -- 172.21.15.3:0/3190615057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73e00ffa90 msgr2=0x7f73e019d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e52aa640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73e00ffa90 0x7f73e019d580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e52aa640 1 -- 172.21.15.3:0/3190615057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d8009900 con 0x7f73e0100450 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e52aa640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e019dac0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f73d000e9f0 tx=0x7f73d000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0009720 con 0x7f73e0100450 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73e0198430 con 0x7f73e0100450 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73d000cd80 con 0x7f73e0100450 2024-02-09T22:58:40.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73d0010690 con 0x7f73e0100450 2024-02-09T22:58:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.688+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73e01a4500 con 0x7f73e0100450 2024-02-09T22:58:40.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.689+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73e0069450 con 0x7f73e0100450 2024-02-09T22:58:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.691+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f73d00046c0 con 0x7f73e0100450 2024-02-09T22:58:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.691+0000 7f73d6ffd640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f73cc0a2230 0x7f73cc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.691+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f73d0014070 con 0x7f73e0100450 2024-02-09T22:58:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.691+0000 7f73e5aab640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f73cc0a2230 0x7f73cc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:40.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.692+0000 7f73e5aab640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f73cc0a2230 0x7f73cc0a46f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009bd0 tx=0x7f73dc005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:40.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.695+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f73d0097290 con 0x7f73e0100450 2024-02-09T22:58:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:40 smithi089 ceph-mon[36665]: pgmap v228: 97 pgs: 97 active+clean; 79 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 910 KiB/s rd, 765 KiB/s wr, 62 op/s 2024-02-09T22:58:40.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.918+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73e0068500 con 0x7f73e0100450 2024-02-09T22:58:40.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.919+0000 7f73d6ffd640 1 -- 172.21.15.3:0/3190615057 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+11373 (secure 0 0 0) 0x7f73d009a950 con 0x7f73e0100450 2024-02-09T22:58:40.922 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:58:40.922 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":42,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":42,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:58:18.236747+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[1,2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27176":{"gid":27176,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3552345002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3552345002},{"type":"v1","addr":"172.21.15.64:6835","nonce":3552345002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:58:40.922 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 42 2024-02-09T22:58:40.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f73cc0a2230 msgr2=0x7f73cc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f73cc0a2230 0x7f73cc0a46f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f73dc009bd0 tx=0x7f73dc005f90 comp rx=0 tx=0).stop 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 msgr2=0x7f73e019dac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e019dac0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7f73d000e9f0 tx=0x7f73d000edf0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 shutdown_connections 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f73cc0a2230 0x7f73cc0a46f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73e0106230 0x7f73e0197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f73e0100450 0x7f73e019dac0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 --2- 172.21.15.3:0/3190615057 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f73e00ffa90 0x7f73e019d580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 >> 172.21.15.3:0/3190615057 conn(0x7f73e0075410 msgr2=0x7f73e00fdae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 shutdown_connections 2024-02-09T22:58:40.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:40.921+0000 7f73e7d0f640 1 -- 172.21.15.3:0/3190615057 wait complete. 2024-02-09T22:58:40.935 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:40.935 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 -- 172.21.15.3:0/1412790702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c1053f0 msgr2=0x7fd78c10cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 --2- 172.21.15.3:0/1412790702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c1053f0 0x7fd78c10cfc0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7fd784009cd0 tx=0x7fd78402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 -- 172.21.15.3:0/1412790702 shutdown_connections 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 --2- 172.21.15.3:0/1412790702 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c1053f0 0x7fd78c10cfc0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 --2- 172.21.15.3:0/1412790702 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd78c100ce0 0x7fd78c104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 --2- 172.21.15.3:0/1412790702 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c100330 0x7fd78c100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 -- 172.21.15.3:0/1412790702 >> 172.21.15.3:0/1412790702 conn(0x7fd78c0771d0 msgr2=0x7fd78c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 -- 172.21.15.3:0/1412790702 shutdown_connections 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 -- 172.21.15.3:0/1412790702 wait complete. 2024-02-09T22:58:41.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 Processor -- start 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.041+0000 7fd792a23640 1 -- start start 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd792a23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd78c100330 0x7fd78c101db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd792a23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 0x7fd78c1022f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd792a23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 0x7fd78c1a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd792a23640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c1a5fb0 con 0x7fd78c102830 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd792a23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c1a6120 con 0x7fd78c1053f0 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd792a23640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd78c1a62e0 con 0x7fd78c100330 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 0x7fd78c1a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 0x7fd78c1022f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 0x7fd78c1022f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52024/0 (socket says 172.21.15.3:52024) 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 0x7fd78c1a59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60768/0 (socket says 172.21.15.3:60768) 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd78b7fe640 1 -- 172.21.15.3:0/670103395 learned_addr learned my addr 172.21.15.3:0/670103395 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd78bfff640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd78c100330 0x7fd78c101db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 -- 172.21.15.3:0/670103395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd78c100330 msgr2=0x7fd78c101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd78c100330 0x7fd78c101db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 -- 172.21.15.3:0/670103395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 msgr2=0x7fd78c1022f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 0x7fd78c1022f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 -- 172.21.15.3:0/670103395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd784009900 con 0x7fd78c102830 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 0x7fd78c1022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.042+0000 7fd790fc0640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 0x7fd78c1a59e0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fd78402c630 tx=0x7fd784034b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.043+0000 7fd7897fa640 1 -- 172.21.15.3:0/670103395 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840328b0 con 0x7fd78c102830 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.043+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c1a6560 con 0x7fd78c102830 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.043+0000 7fd7897fa640 1 -- 172.21.15.3:0/670103395 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd784003e50 con 0x7fd78c102830 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.043+0000 7fd7897fa640 1 -- 172.21.15.3:0/670103395 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd784040450 con 0x7fd78c102830 2024-02-09T22:58:41.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.043+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c1a6a30 con 0x7fd78c102830 2024-02-09T22:58:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7897fa640 1 -- 172.21.15.3:0/670103395 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd78400a040 con 0x7fd78c102830 2024-02-09T22:58:41.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 Processor -- start 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 -- start start 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeaae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7897fa640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7740a2260 0x7fd7740a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbe66020 con 0x5606bbeab3e0 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbe19320 con 0x5606bbeb3f60 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.045+0000 7fd7955fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbe16850 con 0x5606bbeaaa30 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeaae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78bfff640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7740a2260 0x7fd7740a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd7897fa640 1 -- 172.21.15.3:0/670103395 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd7840f0c40 con 0x7fd78c102830 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeaae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35746/0 (socket says 172.21.15.3:35746) 2024-02-09T22:58:41.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd790fc0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 -- 172.21.15.3:0/2149913728 learned_addr learned my addr 172.21.15.3:0/2149913728 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 -- 172.21.15.3:0/2149913728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 msgr2=0x5606bbeb63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 -- 172.21.15.3:0/2149913728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 msgr2=0x5606bbeb3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 -- 172.21.15.3:0/2149913728 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606bbdd4210 con 0x5606bbeaaa30 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78bfff640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7740a2260 0x7fd7740a4720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009bd0 tx=0x7fd77c005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeaae10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd78000bfd0 tx=0x7fd780005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a46a05dfc7b343f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.046+0000 7fd7727fc640 1 -- 172.21.15.3:0/2149913728 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780012c70 con 0x5606bbeaaa30 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7727fc640 1 -- 172.21.15.3:0/2149913728 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd780004510 con 0x5606bbeaaa30 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7727fc640 1 -- 172.21.15.3:0/2149913728 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7800053d0 con 0x5606bbeaaa30 2024-02-09T22:58:41.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 -- 172.21.15.3:0/2149913728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 msgr2=0x5606bbeaae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeaae10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd78000bfd0 tx=0x7fd780005d90 comp rx=0 tx=0).stop 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 -- 172.21.15.3:0/2149913728 shutdown_connections 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 --2- 172.21.15.3:0/2149913728 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeaae10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 -- 172.21.15.3:0/2149913728 >> 172.21.15.3:0/2149913728 conn(0x5606bbea5100 msgr2=0x5606bbea54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 -- 172.21.15.3:0/2149913728 shutdown_connections 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.047+0000 7fd7955fa740 1 -- 172.21.15.3:0/2149913728 wait complete. 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 -- start start 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 Processor -- start 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbe12030 con 0x5606bbeab3e0 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbae5430 con 0x5606bbeb3f60 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbaa1e00 con 0x5606bbeaaa30 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd7955fa740 10 client.0 _ll_register_callbacks cb 0x5606bbe07030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60800/0 (socket says 172.21.15.3:60800) 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 -- 172.21.15.3:0/3346832912 learned_addr learned my addr 172.21.15.3:0/3346832912 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd790fc0640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 -- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 msgr2=0x5606bbeb0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 -- 172.21.15.3:0/3346832912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 msgr2=0x5606bbeb1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.048+0000 7fd78b7fe640 1 -- 172.21.15.3:0/3346832912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78001a040 con 0x5606bbeab3e0 2024-02-09T22:58:41.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb14d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd7800129f0 tx=0x7fd780014fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ebac232f35eaa1a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd76b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3346832912 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd780005170 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3346832912 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd7800052d0 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5606bbb117e0 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3346832912 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd780005490 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 10 client.18180 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x5606bbe106b0 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3346832912 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5606bbb117e0 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3346832912 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd7800143e0 con 0x5606bbeab3e0 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 10 client.18180 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 10 client.18180 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 20 client.18180 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbec4400 0x5606bbec67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 4 client.18180 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.049+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606bbb4e720 con 0x5606bbec4400 2024-02-09T22:58:41.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.050+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbec4400 0x5606bbec67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.050+0000 7fd76b7fe640 10 client.18180 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.050+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbec4400 0x5606bbec67f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=768e55bb1b053ae8 server_cookie=27c5ae13a738f83b in_seq=0 out_seq=0 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.050+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3346832912 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5606bbb4e720 con 0x5606bbec4400 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.050+0000 7fd76b7fe640 10 client.18180 handle_command_reply: tid=0 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.050+0000 7fd7955fa740 1 client.18180 shutdown 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 msgr2=0x5606bbeb14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb14d0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7fd7800129f0 tx=0x7fd780014fc0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 shutdown_connections 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbec4400 0x5606bbec67f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeab3e0 0x5606bbeb14d0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd76b7fe640 0 client.18180 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:41.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3346832912 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 >> 172.21.15.3:0/3346832912 conn(0x5606bbea5100 msgr2=0x5606bbea6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 shutdown_connections 2024-02-09T22:58:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 1 -- 172.21.15.3:0/3346832912 wait complete. 2024-02-09T22:58:41.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.051+0000 7fd7955fa740 20 client.18180 trim_cache size 0 max 16384 2024-02-09T22:58:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 Processor -- start 2024-02-09T22:58:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 -- start start 2024-02-09T22:58:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 0x5606bbecb150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeab3e0 0x5606bbeb1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbd97130 con 0x5606bbeaaa30 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbeacc00 con 0x5606bbeb3f60 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd7955fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbeaff90 con 0x5606bbeab3e0 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 0x5606bbecb150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 0x5606bbecb150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60804/0 (socket says 172.21.15.3:60804) 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 -- 172.21.15.3:0/3206284891 learned_addr learned my addr 172.21.15.3:0/3206284891 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd790fc0640 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 -- 172.21.15.3:0/3206284891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeab3e0 msgr2=0x5606bbeb1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeab3e0 0x5606bbeb1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 -- 172.21.15.3:0/3206284891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 msgr2=0x5606bbeb1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.063+0000 7fd78bfff640 1 -- 172.21.15.3:0/3206284891 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606bbdacc90 con 0x5606bbeaaa30 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 0x5606bbecb150 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x5606bbec70d0 tx=0x7fd77c014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9cd716136a3126 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd771ffb640 1 -- 172.21.15.3:0/3206284891 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c024070 con 0x5606bbeaaa30 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd771ffb640 1 -- 172.21.15.3:0/3206284891 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd77c00e7f0 con 0x5606bbeaaa30 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd771ffb640 1 -- 172.21.15.3:0/3206284891 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd77c025770 con 0x5606bbeaaa30 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 -- 172.21.15.3:0/3206284891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 msgr2=0x5606bbecb150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 0x5606bbecb150 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x5606bbec70d0 tx=0x7fd77c014000 comp rx=0 tx=0).stop 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 -- 172.21.15.3:0/3206284891 shutdown_connections 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeb3f60 0x5606bbeb1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeab3e0 0x5606bbeb1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3206284891 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeaaa30 0x5606bbecb150 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 -- 172.21.15.3:0/3206284891 >> 172.21.15.3:0/3206284891 conn(0x5606bbea5100 msgr2=0x5606bbeaf880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 -- 172.21.15.3:0/3206284891 shutdown_connections 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.064+0000 7fd7955fa740 1 -- 172.21.15.3:0/3206284891 wait complete. 2024-02-09T22:58:41.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 -- start start 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 Processor -- start 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeab3e0 0x5606bbeb9ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 0x5606bbeba3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbf1a8e0 con 0x5606bbeb3f60 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbf1aa20 con 0x5606bbeab3e0 2024-02-09T22:58:41.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606bbf1ab60 con 0x5606bbeaaa30 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd7955fa740 10 client.0 _ll_register_callbacks cb 0x7fd78000bea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd790fc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 0x5606bbeba3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd78b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeab3e0 0x5606bbeb9ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd790fc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 0x5606bbeba3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60818/0 (socket says 172.21.15.3:60818) 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd790fc0640 1 -- 172.21.15.3:0/3820463947 learned_addr learned my addr 172.21.15.3:0/3820463947 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.065+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd790fc0640 1 -- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 msgr2=0x5606bbeb9960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd790fc0640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb9960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd790fc0640 1 -- 172.21.15.3:0/3820463947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeab3e0 msgr2=0x5606bbeb9ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd790fc0640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeab3e0 0x5606bbeb9ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd790fc0640 1 -- 172.21.15.3:0/3820463947 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c017070 con 0x5606bbeb3f60 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd78b7fe640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeab3e0 0x5606bbeb9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd790fc0640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 0x5606bbeba3e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd7840b9b70 tx=0x7fd7840bd070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3eb8a0cc9d8a41 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd76b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3820463947 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7840bde10 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3820463947 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fd7840bd710 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd780004510 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3820463947 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd7840c1860 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd7955fa740 10 client.18186 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x5606bbdd2a70 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3820463947 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd780004510 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3820463947 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fd784004040 con 0x5606bbeb3f60 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd7955fa740 10 client.18186 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd7955fa740 10 client.18186 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.066+0000 7fd7955fa740 20 client.18186 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbf1c060 0x5606bbf1e480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 4 client.18186 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5606bbb9d890 con 0x5606bbf1c060 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbf1c060 0x5606bbf1e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd78bfff640 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbf1c060 0x5606bbf1e480 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=530c1437afd79a53 server_cookie=463b2a4193bb0a12 in_seq=0 out_seq=0 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd76b7fe640 10 client.18186 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd76b7fe640 1 -- 172.21.15.3:0/3820463947 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5606bbb9d890 con 0x5606bbf1c060 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd76b7fe640 10 client.18186 handle_command_reply: tid=0 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 client.18186 shutdown 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 msgr2=0x5606bbeba3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 0x5606bbeba3e0 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7fd7840b9b70 tx=0x7fd7840bd070 comp rx=0 tx=0).stop 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 shutdown_connections 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5606bbf1c060 0x5606bbf1e480 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5606bbeab3e0 0x5606bbeb9ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd76b7fe640 0 client.18186 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5606bbeb3f60 0x5606bbeba3e0 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 --2- 172.21.15.3:0/3820463947 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5606bbeaaa30 0x5606bbeb9960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.067+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 >> 172.21.15.3:0/3820463947 conn(0x5606bbea5100 msgr2=0x5606bbeaf450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 shutdown_connections 2024-02-09T22:58:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd7955fa740 1 -- 172.21.15.3:0/3820463947 wait complete. 2024-02-09T22:58:41.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd7955fa740 20 client.18186 trim_cache size 0 max 16384 2024-02-09T22:58:41.077 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7740a2260 msgr2=0x7fd7740a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7740a2260 0x7fd7740a4720 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fd77c009bd0 tx=0x7fd77c005f90 comp rx=0 tx=0).stop 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 msgr2=0x7fd78c1a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 0x7fd78c1a59e0 secure :-1 s=READY pgs=1227 cs=0 l=1 rev1=1 crypto rx=0x7fd78402c630 tx=0x7fd784034b10 comp rx=0 tx=0).stop 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 shutdown_connections 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7740a2260 0x7fd7740a4720 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd78c1053f0 0x7fd78c1022f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd78c102830 0x7fd78c1a59e0 unknown :-1 s=CLOSED pgs=1227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 --2- 172.21.15.3:0/670103395 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd78c100330 0x7fd78c101db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 >> 172.21.15.3:0/670103395 conn(0x7fd78c0771d0 msgr2=0x7fd78c104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 shutdown_connections 2024-02-09T22:58:41.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.068+0000 7fd792a23640 1 -- 172.21.15.3:0/670103395 wait complete. 2024-02-09T22:58:41.082 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:41.082 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 -- 172.21.15.3:0/412775398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8105410 msgr2=0x7faef810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 --2- 172.21.15.3:0/412775398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8105410 0x7faef810cfe0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7faef0009cd0 tx=0x7faef002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 -- 172.21.15.3:0/412775398 shutdown_connections 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 --2- 172.21.15.3:0/412775398 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8105410 0x7faef810cfe0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 --2- 172.21.15.3:0/412775398 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef8100cf0 0x7faef8104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 --2- 172.21.15.3:0/412775398 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faef8100340 0x7faef8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 -- 172.21.15.3:0/412775398 >> 172.21.15.3:0/412775398 conn(0x7faef80772e0 msgr2=0x7faef80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 -- 172.21.15.3:0/412775398 shutdown_connections 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.186+0000 7faf001c9640 1 -- 172.21.15.3:0/412775398 wait complete. 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 Processor -- start 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 -- start start 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 0x7faef8198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faef8105410 0x7faef8198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef8199210 0x7faef81a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef81a3e60 con 0x7faef8100340 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef81a3fd0 con 0x7faef8105410 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faf001c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faef81a4190 con 0x7faef8199210 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 0x7faef8198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 0x7faef8198740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60836/0 (socket says 172.21.15.3:60836) 2024-02-09T22:58:41.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 -- 172.21.15.3:0/3912600238 learned_addr learned my addr 172.21.15.3:0/3912600238 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefd764640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faef8105410 0x7faef8198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 -- 172.21.15.3:0/3912600238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef8199210 msgr2=0x7faef81a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef8199210 0x7faef81a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 -- 172.21.15.3:0/3912600238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faef8105410 msgr2=0x7faef8198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.187+0000 7faefdf65640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faef8105410 0x7faef8198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faefdf65640 1 -- 172.21.15.3:0/3912600238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef0009900 con 0x7faef8100340 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faefdf65640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 0x7faef8198740 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7faef400e9f0 tx=0x7faef400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faeeeffd640 1 -- 172.21.15.3:0/3912600238 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4009700 con 0x7faef8100340 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faeeeffd640 1 -- 172.21.15.3:0/3912600238 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef400cd80 con 0x7faef8100340 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef81a4470 con 0x7faef8100340 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faeeeffd640 1 -- 172.21.15.3:0/3912600238 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4018920 con 0x7faef8100340 2024-02-09T22:58:41.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.188+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef81a4980 con 0x7faef8100340 2024-02-09T22:58:41.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faeeeffd640 1 -- 172.21.15.3:0/3912600238 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7faef4018a80 con 0x7faef8100340 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 Processor -- start 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 -- start start 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f2ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f33990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f363b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7ee6020 con 0x560ef7f2aa30 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7e99320 con 0x560ef7f2b3e0 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faf02da0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7e96850 con 0x560ef7f33f60 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faefe766640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f2ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faeeeffd640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faee40a2230 0x7faee40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faefdf65640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f33990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faefdf65640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f33990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52070/0 (socket says 172.21.15.3:52070) 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faefdf65640 1 -- 172.21.15.3:0/3252888827 learned_addr learned my addr 172.21.15.3:0/3252888827 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.191+0000 7faefe766640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f2ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60842/0 (socket says 172.21.15.3:60842) 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefd764640 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f363b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefe766640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faee40a2230 0x7faee40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faeeeffd640 1 -- 172.21.15.3:0/3912600238 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7faef4014070 con 0x7faef8100340 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefdf65640 1 -- 172.21.15.3:0/3252888827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 msgr2=0x560ef7f363b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefdf65640 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f363b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefdf65640 1 -- 172.21.15.3:0/3252888827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 msgr2=0x560ef7f2ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefdf65640 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f2ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefdf65640 1 -- 172.21.15.3:0/3252888827 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ef7e54210 con 0x560ef7f2b3e0 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefe766640 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f2ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefe766640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faee40a2230 0x7faee40a46f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faef00098d0 tx=0x7faef00095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faefdf65640 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f33990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faef4000c00 tx=0x7faef401f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11a2f1ebfae605f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faee3fff640 1 -- 172.21.15.3:0/3252888827 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef409b880 con 0x560ef7f2b3e0 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faee3fff640 1 -- 172.21.15.3:0/3252888827 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef409b5f0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.192+0000 7faee3fff640 1 -- 172.21.15.3:0/3252888827 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef4097560 con 0x560ef7f2b3e0 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 -- 172.21.15.3:0/3252888827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 msgr2=0x560ef7f33990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f33990 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faef4000c00 tx=0x7faef401f000 comp rx=0 tx=0).stop 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 -- 172.21.15.3:0/3252888827 shutdown_connections 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f33990 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f363b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 --2- 172.21.15.3:0/3252888827 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f2ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 -- 172.21.15.3:0/3252888827 >> 172.21.15.3:0/3252888827 conn(0x560ef7f25100 msgr2=0x560ef7f254e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 -- 172.21.15.3:0/3252888827 shutdown_connections 2024-02-09T22:58:41.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 -- 172.21.15.3:0/3252888827 wait complete. 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 -- start start 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.193+0000 7faf02da0740 1 Processor -- start 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2aa30 0x560ef7f30f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f31440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f31980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7e92030 con 0x560ef7f2b3e0 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7b65430 con 0x560ef7f2aa30 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7b21e00 con 0x560ef7f33f60 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faf02da0740 10 client.0 _ll_register_callbacks cb 0x560ef7e87030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f31440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f31440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60844/0 (socket says 172.21.15.3:60844) 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 -- 172.21.15.3:0/1313152301 learned_addr learned my addr 172.21.15.3:0/1313152301 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.202 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefdf65640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2aa30 0x560ef7f30f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefe766640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f31980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 -- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 msgr2=0x560ef7f31980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f31980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 -- 172.21.15.3:0/1313152301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2aa30 msgr2=0x560ef7f30f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2aa30 0x560ef7f30f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.194+0000 7faefd764640 1 -- 172.21.15.3:0/1313152301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef81a4980 con 0x560ef7f2b3e0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faefd764640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f31440 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7faee8012730 tx=0x7faee8012bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b4303d66c8d3ec server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faee17fa640 1 -- 172.21.15.3:0/1313152301 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8017070 con 0x560ef7f2b3e0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ef7b917e0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faee17fa640 1 -- 172.21.15.3:0/1313152301 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7faee8012e40 con 0x560ef7f2b3e0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faee17fa640 1 -- 172.21.15.3:0/1313152301 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7faee800f8c0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 10 client.18198 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x560ef7e906b0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faee17fa640 1 -- 172.21.15.3:0/1313152301 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560ef7b917e0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faee17fa640 1 -- 172.21.15.3:0/1313152301 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7faee801ccd0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 10 client.18198 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 10 client.18198 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 20 client.18198 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f444e0 0x560ef7f46900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 4 client.18198 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.195+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ef7bce720 con 0x560ef7f444e0 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.196+0000 7faefdf65640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f444e0 0x560ef7f46900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.196+0000 7faefdf65640 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f444e0 0x560ef7f46900 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=56b7fc7537607a04 server_cookie=68482d949431dda3 in_seq=0 out_seq=0 2024-02-09T22:58:41.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.196+0000 7faee17fa640 10 client.18198 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:41.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faee17fa640 1 -- 172.21.15.3:0/1313152301 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ef7bce720 con 0x560ef7f444e0 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faee17fa640 10 client.18198 handle_command_reply: tid=0 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 client.18198 shutdown 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 msgr2=0x560ef7f31440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f31440 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7faee8012730 tx=0x7faee8012bf0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 shutdown_connections 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f444e0 0x560ef7f46900 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f31980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f31440 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faee17fa640 0 client.18198 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 --2- 172.21.15.3:0/1313152301 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2aa30 0x560ef7f30f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 >> 172.21.15.3:0/1313152301 conn(0x560ef7f25100 msgr2=0x560ef7f26e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 shutdown_connections 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 1 -- 172.21.15.3:0/1313152301 wait complete. 2024-02-09T22:58:41.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.197+0000 7faf02da0740 20 client.18198 trim_cache size 0 max 16384 2024-02-09T22:58:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 Processor -- start 2024-02-09T22:58:41.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 -- start start 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f33630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f317a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f2e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7e17130 con 0x560ef7f2aa30 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7f36580 con 0x560ef7f2b3e0 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faf02da0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7f4bb50 con 0x560ef7f33f60 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faefdf65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f33630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faefdf65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f33630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60860/0 (socket says 172.21.15.3:60860) 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faefdf65640 1 -- 172.21.15.3:0/3672685905 learned_addr learned my addr 172.21.15.3:0/3672685905 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faefd764640 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.209+0000 7faefe766640 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f2e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faefdf65640 1 -- 172.21.15.3:0/3672685905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 msgr2=0x560ef7f2e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faefdf65640 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f2e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faefdf65640 1 -- 172.21.15.3:0/3672685905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 msgr2=0x560ef7f317a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faefdf65640 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f317a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faefdf65640 1 -- 172.21.15.3:0/3672685905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ef7e2cc90 con 0x560ef7f2aa30 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faefdf65640 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f33630 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7faef400c6a0 tx=0x7faef4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c84e46b7f4ed18b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faee37fe640 1 -- 172.21.15.3:0/3672685905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef40ce5d0 con 0x560ef7f2aa30 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faee37fe640 1 -- 172.21.15.3:0/3672685905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef40a3690 con 0x560ef7f2aa30 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.210+0000 7faee37fe640 1 -- 172.21.15.3:0/3672685905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef409be10 con 0x560ef7f2aa30 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- 172.21.15.3:0/3672685905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 msgr2=0x560ef7f33630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f33630 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7faef400c6a0 tx=0x7faef4000bc0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- 172.21.15.3:0/3672685905 shutdown_connections 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f2b3e0 0x560ef7f317a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f33f60 0x560ef7f2e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- 172.21.15.3:0/3672685905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2aa30 0x560ef7f33630 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- 172.21.15.3:0/3672685905 >> 172.21.15.3:0/3672685905 conn(0x560ef7f25100 msgr2=0x560ef7f2c210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- 172.21.15.3:0/3672685905 shutdown_connections 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- 172.21.15.3:0/3672685905 wait complete. 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- start start 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 Processor -- start 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f2aa30 0x560ef7f94ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f95430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f33f60 0x560ef7f3a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7f3a8c0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7f3aa30 con 0x560ef7f33f60 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef7f3abf0 con 0x560ef7f2aa30 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faf02da0740 10 client.0 _ll_register_callbacks cb 0x7faef409acc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faefd764640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f95430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.220 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.211+0000 7faefd764640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f95430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60864/0 (socket says 172.21.15.3:60864) 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 -- 172.21.15.3:0/551954087 learned_addr learned my addr 172.21.15.3:0/551954087 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefdf65640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f2aa30 0x560ef7f94ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefe766640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f33f60 0x560ef7f3a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 -- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f2aa30 msgr2=0x560ef7f94ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f2aa30 0x560ef7f94ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 -- 172.21.15.3:0/551954087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f33f60 msgr2=0x560ef7f3a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f33f60 0x560ef7f3a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 -- 172.21.15.3:0/551954087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ef7e54210 con 0x560ef7f2b3e0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faefd764640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f95430 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x560ef7f324d0 tx=0x7faee8005670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2c8e184197b1472 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faee17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faee17fa640 1 -- 172.21.15.3:0/551954087 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faee8017070 con 0x560ef7f2b3e0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faee17fa640 1 -- 172.21.15.3:0/551954087 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7faee8009d90 con 0x560ef7f2b3e0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faef409b5f0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faee17fa640 1 -- 172.21.15.3:0/551954087 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7faee800fba0 con 0x560ef7f2b3e0 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faf02da0740 10 client.18204 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x560ef7e52a70 con 0x560ef7f2b3e0 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.212+0000 7faee17fa640 1 -- 172.21.15.3:0/551954087 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faee803a020 con 0x560ef7f2b3e0 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faee17fa640 1 -- 172.21.15.3:0/551954087 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7faee802b030 con 0x560ef7f2b3e0 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 10 client.18204 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 10 client.18204 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 20 client.18204 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f9cd70 0x560ef7f9f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 4 client.18204 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560ef7c1d890 con 0x560ef7f9cd70 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faefdf65640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f9cd70 0x560ef7f9f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faefdf65640 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f9cd70 0x560ef7f9f190 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=56d09c14d82d3cb8 server_cookie=a98cdf2e98f00a0c in_seq=0 out_seq=0 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faee17fa640 10 client.18204 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:41.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faee17fa640 1 -- 172.21.15.3:0/551954087 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560ef7c1d890 con 0x560ef7f9cd70 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faee17fa640 10 client.18204 handle_command_reply: tid=0 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 1 client.18204 shutdown 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 msgr2=0x560ef7f95430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.213+0000 7faf02da0740 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f95430 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x560ef7f324d0 tx=0x7faee8005670 comp rx=0 tx=0).stop 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 shutdown_connections 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x560ef7f9cd70 0x560ef7f9f190 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ef7f33f60 0x560ef7f3a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ef7f2b3e0 0x560ef7f95430 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 --2- 172.21.15.3:0/551954087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ef7f2aa30 0x560ef7f94ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faee17fa640 0 client.18204 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 >> 172.21.15.3:0/551954087 conn(0x560ef7f25100 msgr2=0x560ef7f239b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 shutdown_connections 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 1 -- 172.21.15.3:0/551954087 wait complete. 2024-02-09T22:58:41.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf02da0740 20 client.18204 trim_cache size 0 max 16384 2024-02-09T22:58:41.224 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faee40a2230 msgr2=0x7faee40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faee40a2230 0x7faee40a46f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7faef00098d0 tx=0x7faef00095c0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 msgr2=0x7faef8198740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 0x7faef8198740 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7faef400e9f0 tx=0x7faef400eeb0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 shutdown_connections 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faee40a2230 0x7faee40a46f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faef8105410 0x7faef8198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faef8199210 0x7faef81a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 --2- 172.21.15.3:0/3912600238 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faef8100340 0x7faef8198740 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 >> 172.21.15.3:0/3912600238 conn(0x7faef80772e0 msgr2=0x7faef81065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 shutdown_connections 2024-02-09T22:58:41.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.214+0000 7faf001c9640 1 -- 172.21.15.3:0/3912600238 wait complete. 2024-02-09T22:58:41.228 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:41.228 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 -- 172.21.15.3:0/100521186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901061d0 msgr2=0x7f02901065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 --2- 172.21.15.3:0/100521186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901061d0 0x7f02901065b0 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f0280009930 tx=0x7f028002bc20 comp rx=0 tx=0).stop 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 -- 172.21.15.3:0/100521186 shutdown_connections 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 --2- 172.21.15.3:0/100521186 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901000b0 0x7f029010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 --2- 172.21.15.3:0/100521186 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f02900ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 --2- 172.21.15.3:0/100521186 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901061d0 0x7f02901065b0 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 -- 172.21.15.3:0/100521186 >> 172.21.15.3:0/100521186 conn(0x7f02900776a0 msgr2=0x7f02900fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 -- 172.21.15.3:0/100521186 shutdown_connections 2024-02-09T22:58:41.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.335+0000 7f0295eec640 1 -- 172.21.15.3:0/100521186 wait complete. 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 Processor -- start 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 -- start start 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f029019d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 0x7f029019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901061d0 0x7f0290197760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0290197ef0 con 0x7f02901000b0 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0290198060 con 0x7f02901061d0 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f0295eec640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0290198220 con 0x7f02900ff6f0 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f028effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 0x7f029019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f029019d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f028effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 0x7f029019dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60880/0 (socket says 172.21.15.3:60880) 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f028effd640 1 -- 172.21.15.3:0/670203188 learned_addr learned my addr 172.21.15.3:0/670203188 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f028ffff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901061d0 0x7f0290197760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.336+0000 7f028f7fe640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f029019d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35850/0 (socket says 172.21.15.3:35850) 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028effd640 1 -- 172.21.15.3:0/670203188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 msgr2=0x7f029019d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028effd640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f029019d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028effd640 1 -- 172.21.15.3:0/670203188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901061d0 msgr2=0x7f0290197760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028effd640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901061d0 0x7f0290197760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028effd640 1 -- 172.21.15.3:0/670203188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0280009560 con 0x7f02901000b0 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028ffff640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901061d0 0x7f0290197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028f7fe640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f029019d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028effd640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 0x7f029019dbe0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f028400e9f0 tx=0x7f028400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028cff9640 1 -- 172.21.15.3:0/670203188 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284009720 con 0x7f02901000b0 2024-02-09T22:58:41.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028cff9640 1 -- 172.21.15.3:0/670203188 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f028400cd80 con 0x7f02901000b0 2024-02-09T22:58:41.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0290198500 con 0x7f02901000b0 2024-02-09T22:58:41.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f028cff9640 1 -- 172.21.15.3:0/670203188 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0284010690 con 0x7f02901000b0 2024-02-09T22:58:41.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.337+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02901a4620 con 0x7f02901000b0 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.339+0000 7f028cff9640 1 -- 172.21.15.3:0/670203188 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f02840046c0 con 0x7f02901000b0 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.339+0000 7f0298b95740 1 Processor -- start 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 -- start start 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d3a30 0x55f9f54d3e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 0x55f9f54dc990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54df3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f548f020 con 0x55f9f54dcf60 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f5442320 con 0x55f9f54d3a30 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f0298b95740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f543f850 con 0x55f9f54d43e0 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028cff9640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f02780a22b0 0x7f02780a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 0x55f9f54dc990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54df3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 0x55f9f54dc990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35854/0 (socket says 172.21.15.3:35854) 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 -- 172.21.15.3:0/3234224117 learned_addr learned my addr 172.21.15.3:0/3234224117 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028ffff640 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d3a30 0x55f9f54d3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028ffff640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f02780a22b0 0x7f02780a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 -- 172.21.15.3:0/3234224117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d3a30 msgr2=0x55f9f54d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d3a30 0x55f9f54d3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028cff9640 1 -- 172.21.15.3:0/670203188 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0284014070 con 0x7f02901000b0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 -- 172.21.15.3:0/3234224117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 msgr2=0x55f9f54df3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54df3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028f7fe640 1 -- 172.21.15.3:0/3234224117 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9f53fd210 con 0x55f9f54d43e0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028effd640 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54df3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.340+0000 7f028ffff640 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d3a30 0x55f9f54d3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f028f7fe640 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 0x55f9f54dc990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0280036040 tx=0x7f028002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52b29d5cce296f4f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f028ffff640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f02780a22b0 0x7f02780a4770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f02880099e0 tx=0x7f0288009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0275ffb640 1 -- 172.21.15.3:0/3234224117 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028002c7d0 con 0x55f9f54d43e0 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0275ffb640 1 -- 172.21.15.3:0/3234224117 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0280004510 con 0x55f9f54d43e0 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0275ffb640 1 -- 172.21.15.3:0/3234224117 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0280005430 con 0x55f9f54d43e0 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0298b95740 1 -- 172.21.15.3:0/3234224117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 msgr2=0x55f9f54dc990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0298b95740 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 0x55f9f54dc990 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0280036040 tx=0x7f028002c650 comp rx=0 tx=0).stop 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0298b95740 1 -- 172.21.15.3:0/3234224117 shutdown_connections 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0298b95740 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54df3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.341+0000 7f0298b95740 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d43e0 0x55f9f54dc990 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 --2- 172.21.15.3:0/3234224117 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d3a30 0x55f9f54d3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- 172.21.15.3:0/3234224117 >> 172.21.15.3:0/3234224117 conn(0x55f9f54ce100 msgr2=0x55f9f54ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- 172.21.15.3:0/3234224117 shutdown_connections 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- 172.21.15.3:0/3234224117 wait complete. 2024-02-09T22:58:41.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- start start 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 Processor -- start 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54d9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 0x55f9f54da4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54dcf60 0x55f9f54daa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f543b030 con 0x55f9f54d43e0 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f510e430 con 0x55f9f54dcf60 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f50cae00 con 0x55f9f54d3a30 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f0298b95740 10 client.0 _ll_register_callbacks cb 0x55f9f5430030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f028effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 0x55f9f54da4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54d9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.342+0000 7f028effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 0x55f9f54da4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60908/0 (socket says 172.21.15.3:60908) 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028effd640 1 -- 172.21.15.3:0/1953067605 learned_addr learned my addr 172.21.15.3:0/1953067605 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028ffff640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54dcf60 0x55f9f54daa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028f7fe640 1 -- 172.21.15.3:0/1953067605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54dcf60 msgr2=0x55f9f54daa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028f7fe640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54dcf60 0x55f9f54daa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028f7fe640 1 -- 172.21.15.3:0/1953067605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 msgr2=0x55f9f54da4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028f7fe640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 0x55f9f54da4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028f7fe640 1 -- 172.21.15.3:0/1953067605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f029010ea00 con 0x55f9f54d3a30 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028effd640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 0x55f9f54da4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f028f7fe640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54d9f90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55f9f54cf170 tx=0x7f0280005b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fcdf76f8f27da47f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f026effd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f026effd640 1 -- 172.21.15.3:0/1953067605 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02800055c0 con 0x55f9f54d3a30 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f026effd640 1 -- 172.21.15.3:0/1953067605 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f028004d070 con 0x55f9f54d3a30 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f9f513a7e0 con 0x55f9f54d3a30 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f026effd640 1 -- 172.21.15.3:0/1953067605 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f028003f440 con 0x55f9f54d3a30 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f0298b95740 10 client.27379 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.343+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55f9f54396b0 con 0x55f9f54d3a30 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f026effd640 1 -- 172.21.15.3:0/1953067605 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f9f513a7e0 con 0x55f9f54d3a30 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f026effd640 1 -- 172.21.15.3:0/1953067605 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f028003c050 con 0x55f9f54d3a30 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f0298b95740 10 client.27379 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f0298b95740 10 client.27379 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f0298b95740 20 client.27379 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f0298b95740 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f54ed0c0 0x55f9f54ef4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f0298b95740 4 client.27379 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9f5177720 con 0x55f9f54ed0c0 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f028effd640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f54ed0c0 0x55f9f54ef4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f026effd640 10 client.27379 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.344+0000 7f028effd640 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f54ed0c0 0x55f9f54ef4b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7d0ecddcb9a16168 server_cookie=b12f15abcfb60aa8 in_seq=0 out_seq=0 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f026effd640 1 -- 172.21.15.3:0/1953067605 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f9f5177720 con 0x55f9f54ed0c0 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f026effd640 10 client.27379 handle_command_reply: tid=0 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 client.27379 shutdown 2024-02-09T22:58:41.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 msgr2=0x55f9f54d9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54d9f90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x55f9f54cf170 tx=0x7f0280005b50 comp rx=0 tx=0).stop 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 shutdown_connections 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f54ed0c0 0x55f9f54ef4b0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54dcf60 0x55f9f54daa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54d43e0 0x55f9f54da4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 --2- 172.21.15.3:0/1953067605 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54d9f90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f026effd640 0 client.27379 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 >> 172.21.15.3:0/1953067605 conn(0x55f9f54ce100 msgr2=0x55f9f54cfe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 shutdown_connections 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 1 -- 172.21.15.3:0/1953067605 wait complete. 2024-02-09T22:58:41.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.345+0000 7f0298b95740 20 client.27379 trim_cache size 0 max 16384 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 Processor -- start 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 -- start start 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54dc6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54c9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54da830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f53c0130 con 0x55f9f54dcf60 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f54f46a0 con 0x55f9f54d43e0 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f0298b95740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f54cdc30 con 0x55f9f54d3a30 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54c9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54da830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54dc6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54c9180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52130/0 (socket says 172.21.15.3:52130) 2024-02-09T22:58:41.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 -- 172.21.15.3:0/930926100 learned_addr learned my addr 172.21.15.3:0/930926100 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 -- 172.21.15.3:0/930926100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 msgr2=0x55f9f54dc6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54dc6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 -- 172.21.15.3:0/930926100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 msgr2=0x55f9f54da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54da830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.358+0000 7f028effd640 1 -- 172.21.15.3:0/930926100 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9f53d5c90 con 0x55f9f54d43e0 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.359+0000 7f028effd640 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54c9180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0284097760 tx=0x7f02840c9040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cc057d1dc8a89e8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.359+0000 7f02757fa640 1 -- 172.21.15.3:0/930926100 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f028409b490 con 0x55f9f54d43e0 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.359+0000 7f02757fa640 1 -- 172.21.15.3:0/930926100 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f028409ad90 con 0x55f9f54d43e0 2024-02-09T22:58:41.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.359+0000 7f02757fa640 1 -- 172.21.15.3:0/930926100 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02840a3300 con 0x55f9f54d43e0 2024-02-09T22:58:41.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- 172.21.15.3:0/930926100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 msgr2=0x55f9f54c9180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54c9180 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0284097760 tx=0x7f02840c9040 comp rx=0 tx=0).stop 2024-02-09T22:58:41.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- 172.21.15.3:0/930926100 shutdown_connections 2024-02-09T22:58:41.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54c9180 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54da830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- 172.21.15.3:0/930926100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54dc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- 172.21.15.3:0/930926100 >> 172.21.15.3:0/930926100 conn(0x55f9f54ce100 msgr2=0x55f9f54ce4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- 172.21.15.3:0/930926100 shutdown_connections 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- 172.21.15.3:0/930926100 wait complete. 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- start start 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 Processor -- start 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54e4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54e25e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54e2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f54e31f0 con 0x55f9f54dcf60 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f54e3360 con 0x55f9f54d43e0 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9f5543580 con 0x55f9f54d3a30 2024-02-09T22:58:41.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.360+0000 7f0298b95740 10 client.0 _ll_register_callbacks cb 0x7f0284005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54e2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54e4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54e25e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54e2b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60926/0 (socket says 172.21.15.3:60926) 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54e25e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52136/0 (socket says 172.21.15.3:52136) 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54e4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35878/0 (socket says 172.21.15.3:35878) 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 -- 172.21.15.3:0/422267859 learned_addr learned my addr 172.21.15.3:0/422267859 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 -- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 msgr2=0x55f9f54e4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54e4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 -- 172.21.15.3:0/422267859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 msgr2=0x55f9f54e25e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54e25e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 -- 172.21.15.3:0/422267859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0290198500 con 0x55f9f54dcf60 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028effd640 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54e25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f028ffff640 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54e2b20 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f0288003ea0 tx=0x7f0288004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bcfdc8607ef00f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f026effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f026effd640 1 -- 172.21.15.3:0/422267859 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0288026040 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0280004510 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f026effd640 1 -- 172.21.15.3:0/422267859 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f02880115c0 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f026effd640 1 -- 172.21.15.3:0/422267859 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0288004b20 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f0298b95740 10 client.18222 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.361+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x55f9f53fba70 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f026effd640 1 -- 172.21.15.3:0/422267859 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0280004510 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f026effd640 1 -- 172.21.15.3:0/422267859 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0288005020 con 0x55f9f54dcf60 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f0298b95740 10 client.18222 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f0298b95740 10 client.18222 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f0298b95740 20 client.18222 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f0298b95740 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f5544d50 0x55f9f5547170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f0298b95740 4 client.18222 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f9f51c6890 con 0x55f9f5544d50 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.362+0000 7f028f7fe640 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f5544d50 0x55f9f5547170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f026effd640 10 client.18222 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f028f7fe640 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f5544d50 0x55f9f5547170 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=93aeefa53535595f server_cookie=da2d3302b22eb827 in_seq=0 out_seq=0 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f026effd640 1 -- 172.21.15.3:0/422267859 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f9f51c6890 con 0x55f9f5544d50 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f026effd640 10 client.18222 handle_command_reply: tid=0 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 client.18222 shutdown 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 msgr2=0x55f9f54e2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54e2b20 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f0288003ea0 tx=0x7f0288004ab0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 shutdown_connections 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55f9f5544d50 0x55f9f5547170 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9f54d43e0 0x55f9f54e25e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f026effd640 0 client.18222 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9f54dcf60 0x55f9f54e2b20 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 --2- 172.21.15.3:0/422267859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9f54d3a30 0x55f9f54e4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 >> 172.21.15.3:0/422267859 conn(0x55f9f54ce100 msgr2=0x55f9f54d5250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 shutdown_connections 2024-02-09T22:58:41.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 1 -- 172.21.15.3:0/422267859 wait complete. 2024-02-09T22:58:41.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.363+0000 7f0298b95740 20 client.18222 trim_cache size 0 max 16384 2024-02-09T22:58:41.372 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f02780a22b0 msgr2=0x7f02780a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f02780a22b0 0x7f02780a4770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f02880099e0 tx=0x7f0288009500 comp rx=0 tx=0).stop 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 msgr2=0x7f029019dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 0x7f029019dbe0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f028400e9f0 tx=0x7f028400edf0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 shutdown_connections 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f02780a22b0 0x7f02780a4770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f02901061d0 0x7f0290197760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f02901000b0 0x7f029019dbe0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 --2- 172.21.15.3:0/670203188 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f02900ff6f0 0x7f029019d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 >> 172.21.15.3:0/670203188 conn(0x7f02900776a0 msgr2=0x7f0290104370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 shutdown_connections 2024-02-09T22:58:41.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.364+0000 7f0295eec640 1 -- 172.21.15.3:0/670203188 wait complete. 2024-02-09T22:58:41.378 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:41.378 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 -- 172.21.15.3:0/2595967743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb41005a0 msgr2=0x7f0bb4100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2595967743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb41005a0 0x7f0bb4100980 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4009cd0 tx=0x7f0ba402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 -- 172.21.15.3:0/2595967743 shutdown_connections 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2595967743 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0bb4100f50 0x7f0bb4104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2595967743 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb410cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2595967743 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb41005a0 0x7f0bb4100980 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 -- 172.21.15.3:0/2595967743 >> 172.21.15.3:0/2595967743 conn(0x7f0bb4077fa0 msgr2=0x7f0bb40783b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 -- 172.21.15.3:0/2595967743 shutdown_connections 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 -- 172.21.15.3:0/2595967743 wait complete. 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 Processor -- start 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.492+0000 7f0bb9301640 1 -- start start 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0bb41005a0 0x7f0bb4198670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb4198bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 0x7f0bb41a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41a3dc0 con 0x7f0bb4199140 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41a3f30 con 0x7f0bb41005a0 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bb41a40f0 con 0x7f0bb41053e0 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 0x7f0bb41a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb4198bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 0x7f0bb41a36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60954/0 (socket says 172.21.15.3:60954) 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0bb41005a0 0x7f0bb4198670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 -- 172.21.15.3:0/2794681522 learned_addr learned my addr 172.21.15.3:0/2794681522 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb4198bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35898/0 (socket says 172.21.15.3:35898) 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 -- 172.21.15.3:0/2794681522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 msgr2=0x7f0bb4198bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb4198bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 -- 172.21.15.3:0/2794681522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0bb41005a0 msgr2=0x7f0bb4198670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0bb41005a0 0x7f0bb4198670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 -- 172.21.15.3:0/2794681522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba4009900 con 0x7f0bb4199140 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb37fe640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 0x7f0bb41a36f0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e970 tx=0x7f0bac00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0babfff640 1 -- 172.21.15.3:0/2794681522 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac00cab0 con 0x7f0bb4199140 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0babfff640 1 -- 172.21.15.3:0/2794681522 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bac005500 con 0x7f0bb4199140 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bb41a43d0 con 0x7f0bb4199140 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0babfff640 1 -- 172.21.15.3:0/2794681522 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bac004e40 con 0x7f0bb4199140 2024-02-09T22:58:41.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.493+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb41a48e0 con 0x7f0bb4199140 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.496+0000 7f0bbbfa8740 1 Processor -- start 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.496+0000 7f0babfff640 1 -- 172.21.15.3:0/2794681522 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0bac005000 con 0x7f0bb4199140 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.496+0000 7f0bbbfa8740 1 -- start start 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36cfce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d05990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36cb8020 con 0x559e36cfd3e0 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36c6b320 con 0x559e36cfca30 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36c68850 con 0x559e36d05f60 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d05990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d05990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60964/0 (socket says 172.21.15.3:60964) 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/1485382013 learned_addr learned my addr 172.21.15.3:0/1485382013 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36cfce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0babfff640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ba00a2230 0x7f0ba00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/1485382013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 msgr2=0x559e36d083b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d083b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb37fe640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ba00a2230 0x7f0ba00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/1485382013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 msgr2=0x559e36cfce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36cfce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/1485382013 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e36c26210 con 0x559e36cfd3e0 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.497+0000 7f0babfff640 1 -- 172.21.15.3:0/2794681522 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0bac014070 con 0x7f0bb4199140 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.498+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d05990 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f0ba402c700 tx=0x7f0ba4032de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77d77d6869787ce1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.498+0000 7f0ba97fa640 1 -- 172.21.15.3:0/1485382013 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4034440 con 0x559e36cfd3e0 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.498+0000 7f0bb37fe640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ba00a2230 0x7f0ba00a46f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0cbd10 tx=0x7f0bac06a0a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.498+0000 7f0ba97fa640 1 -- 172.21.15.3:0/1485382013 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ba40345a0 con 0x559e36cfd3e0 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.498+0000 7f0ba97fa640 1 -- 172.21.15.3:0/1485382013 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4003bd0 con 0x559e36cfd3e0 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/1485382013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 msgr2=0x559e36d05990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d05990 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7f0ba402c700 tx=0x7f0ba4032de0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/1485382013 shutdown_connections 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d083b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d05990 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/1485382013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36cfce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/1485382013 >> 172.21.15.3:0/1485382013 conn(0x559e36cf7100 msgr2=0x559e36cf74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/1485382013 shutdown_connections 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/1485382013 wait complete. 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- start start 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 Processor -- start 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d02ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d03970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36c64030 con 0x559e36cfd3e0 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36937430 con 0x559e36cfca30 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e368f3e00 con 0x559e36d05f60 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bbbfa8740 10 client.0 _ll_register_callbacks cb 0x559e36c59030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.499+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60980/0 (socket says 172.21.15.3:60980) 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d02ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/3947861281 learned_addr learned my addr 172.21.15.3:0/3947861281 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d03970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/3947861281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 msgr2=0x559e36d03970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d03970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 msgr2=0x559e36d02ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d02ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/3947861281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bb410d800 con 0x559e36cfd3e0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03430 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0ba402c630 tx=0x7f0ba4005c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af496c4abda62e06 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0b9a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/3947861281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4034440 con 0x559e36cfd3e0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/3947861281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0ba40326b0 con 0x559e36cfd3e0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559e369637e0 con 0x559e36cfd3e0 2024-02-09T22:58:41.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/3947861281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ba4034ea0 con 0x559e36cfd3e0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bbbfa8740 10 client.18234 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.500+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x559e36c626b0 con 0x559e36cfd3e0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/3947861281 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559e369637e0 con 0x559e36cfd3e0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/3947861281 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0ba40045d0 con 0x559e36cfd3e0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bbbfa8740 10 client.18234 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bbbfa8740 10 client.18234 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bbbfa8740 20 client.18234 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d160c0 0x559e36d184e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bbbfa8740 4 client.18234 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e369a0720 con 0x559e36d160c0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d160c0 0x559e36d184e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0b9a7fc640 10 client.18234 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.501+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d160c0 0x559e36d184e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=796da06332c50c10 server_cookie=4ab52f6dcfa61161 in_seq=0 out_seq=0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/3947861281 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559e369a0720 con 0x559e36d160c0 2024-02-09T22:58:41.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0b9a7fc640 10 client.18234 handle_command_reply: tid=0 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 client.18234 shutdown 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 msgr2=0x559e36d03430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03430 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7f0ba402c630 tx=0x7f0ba4005c70 comp rx=0 tx=0).stop 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 shutdown_connections 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d160c0 0x559e36d184e0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d03970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03430 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/3947861281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d02ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0b9a7fc640 0 client.18234 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 >> 172.21.15.3:0/3947861281 conn(0x559e36cf7100 msgr2=0x559e36cf8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 shutdown_connections 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/3947861281 wait complete. 2024-02-09T22:58:41.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.502+0000 7f0bbbfa8740 20 client.18234 trim_cache size 0 max 16384 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 Processor -- start 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 -- start start 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d05590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d1ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36cf28b0 con 0x559e36cfd3e0 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36be9130 con 0x559e36cfca30 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36cfeb30 con 0x559e36d05f60 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d05590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bb2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:60996/0 (socket says 172.21.15.3:60996) 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/4175751121 learned_addr learned my addr 172.21.15.3:0/4175751121 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.515+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d1ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/4175751121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 msgr2=0x559e36d1ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d1ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/4175751121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 msgr2=0x559e36d05590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d05590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb2ffd640 1 -- 172.21.15.3:0/4175751121 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e36bfec90 con 0x559e36cfd3e0 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d05590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03790 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4004980 tx=0x7f0ba4034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdcfe909c25d5160 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0ba8ff9640 1 -- 172.21.15.3:0/4175751121 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4005cf0 con 0x559e36cfd3e0 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0ba8ff9640 1 -- 172.21.15.3:0/4175751121 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ba40326b0 con 0x559e36cfd3e0 2024-02-09T22:58:41.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.516+0000 7f0ba8ff9640 1 -- 172.21.15.3:0/4175751121 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ba4061690 con 0x559e36cfd3e0 2024-02-09T22:58:41.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/4175751121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 msgr2=0x559e36d03790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03790 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f0ba4004980 tx=0x7f0ba4034440 comp rx=0 tx=0).stop 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/4175751121 shutdown_connections 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d1ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfd3e0 0x559e36d03790 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/4175751121 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfca30 0x559e36d05590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/4175751121 >> 172.21.15.3:0/4175751121 conn(0x559e36cf7100 msgr2=0x559e36cf74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/4175751121 shutdown_connections 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/4175751121 wait complete. 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- start start 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 Processor -- start 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 0x559e36d0cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfd3e0 0x559e36d0d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d0b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36d0bbf0 con 0x559e36cfca30 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36d0bd60 con 0x559e36cfd3e0 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.517+0000 7f0bbbfa8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e36d0bf20 con 0x559e36d05f60 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bbbfa8740 10 client.0 _ll_register_callbacks cb 0x7f0bac005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 0x559e36d0cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 0x559e36d0cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32770/0 (socket says 172.21.15.3:32770) 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 -- 172.21.15.3:0/2506231299 learned_addr learned my addr 172.21.15.3:0/2506231299 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfd3e0 0x559e36d0d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d0b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 -- 172.21.15.3:0/2506231299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 msgr2=0x559e36d0b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d0b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 -- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfd3e0 msgr2=0x559e36d0d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfd3e0 0x559e36d0d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 -- 172.21.15.3:0/2506231299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b9c019040 con 0x559e36cfca30 2024-02-09T22:58:41.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bb27fc640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 0x559e36d0cec0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c005f10 tx=0x7f0b9c026040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff788bfa887dbfbe server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0b9a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/2506231299 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c00fca0 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/2506231299 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0b9c011e40 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ba40345a0 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/2506231299 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b9c00fe00 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bbbfa8740 10 client.18240 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.518+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x559e36c626b0 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/2506231299 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ba40345a0 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/2506231299 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0b9c0213d0 con 0x559e36cfca30 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 10 client.18240 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 10 client.18240 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 20 client.18240 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d6d960 0x559e36d6fd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 4 client.18240 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559e369ef890 con 0x559e36d6d960 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d6d960 0x559e36d6fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bb2ffd640 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d6d960 0x559e36d6fd80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ffa623386a0ff033 server_cookie=9bb7035ba9b224 in_seq=0 out_seq=0 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0b9a7fc640 10 client.18240 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0b9a7fc640 1 -- 172.21.15.3:0/2506231299 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559e369ef890 con 0x559e36d6d960 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0b9a7fc640 10 client.18240 handle_command_reply: tid=0 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.519+0000 7f0bbbfa8740 1 client.18240 shutdown 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 msgr2=0x559e36d0cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 0x559e36d0cec0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f0b9c005f10 tx=0x7f0b9c026040 comp rx=0 tx=0).stop 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 shutdown_connections 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x559e36d6d960 0x559e36d6fd80 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.528 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559e36cfd3e0 0x559e36d0d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559e36d05f60 0x559e36d0b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0b9a7fc640 0 client.18240 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 --2- 172.21.15.3:0/2506231299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559e36cfca30 0x559e36d0cec0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 >> 172.21.15.3:0/2506231299 conn(0x559e36cf7100 msgr2=0x559e36d1ce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 shutdown_connections 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 1 -- 172.21.15.3:0/2506231299 wait complete. 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bbbfa8740 20 client.18240 trim_cache size 0 max 16384 2024-02-09T22:58:41.529 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:41.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ba00a2230 msgr2=0x7f0ba00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ba00a2230 0x7f0ba00a46f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0bac0cbd10 tx=0x7f0bac06a0a0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 msgr2=0x7f0bb41a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 0x7f0bb41a36f0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7f0bac00e970 tx=0x7f0bac00ed70 comp rx=0 tx=0).stop 2024-02-09T22:58:41.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 shutdown_connections 2024-02-09T22:58:41.530 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0ba00a2230 0x7f0ba00a46f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0bb4199140 0x7f0bb41a36f0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0bb41053e0 0x7f0bb4198bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 --2- 172.21.15.3:0/2794681522 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0bb41005a0 0x7f0bb4198670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.520+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 >> 172.21.15.3:0/2794681522 conn(0x7f0bb4077fa0 msgr2=0x7f0bb40fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.521+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 shutdown_connections 2024-02-09T22:58:41.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.521+0000 7f0bb9301640 1 -- 172.21.15.3:0/2794681522 wait complete. 2024-02-09T22:58:41.533 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:41 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3190615057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:58:41.534 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:58:41.535 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:58:41.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 -- 172.21.15.3:0/4175291808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4101de0 msgr2=0x7f6db4105e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 --2- 172.21.15.3:0/4175291808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4101de0 0x7f6db4105e20 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009930 tx=0x7f6da802bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 -- 172.21.15.3:0/4175291808 shutdown_connections 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 --2- 172.21.15.3:0/4175291808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6db4106500 0x7f6db410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 --2- 172.21.15.3:0/4175291808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4101de0 0x7f6db4105e20 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 --2- 172.21.15.3:0/4175291808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db4101430 0x7f6db4101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 -- 172.21.15.3:0/4175291808 >> 172.21.15.3:0/4175291808 conn(0x7f6db40fccd0 msgr2=0x7f6db40ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 -- 172.21.15.3:0/4175291808 shutdown_connections 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.643+0000 7f6dba257640 1 -- 172.21.15.3:0/4175291808 wait complete. 2024-02-09T22:58:41.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 Processor -- start 2024-02-09T22:58:41.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 -- start start 2024-02-09T22:58:41.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6db4101430 0x7f6db419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 0x7f6db419dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db4197820 0x7f6db4197cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db41982d0 con 0x7f6db4106500 2024-02-09T22:58:41.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db4198440 con 0x7f6db4101430 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6dba257640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db4198600 con 0x7f6db4197820 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6db2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 0x7f6db419dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6db4101430 0x7f6db419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6db2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 0x7f6db419dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32784/0 (socket says 172.21.15.3:32784) 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6db2ffd640 1 -- 172.21.15.3:0/1760295627 learned_addr learned my addr 172.21.15.3:0/1760295627 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.644+0000 7f6db3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db4197820 0x7f6db4197cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db2ffd640 1 -- 172.21.15.3:0/1760295627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db4197820 msgr2=0x7f6db4197cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db4197820 0x7f6db4197cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db2ffd640 1 -- 172.21.15.3:0/1760295627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6db4101430 msgr2=0x7f6db419d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6db4101430 0x7f6db419d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db2ffd640 1 -- 172.21.15.3:0/1760295627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dac009490 con 0x7f6db4106500 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 0x7f6db419dc10 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009230 tx=0x7f6da802c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db0ff9640 1 -- 172.21.15.3:0/1760295627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8004350 con 0x7f6db4106500 2024-02-09T22:58:41.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da8009560 con 0x7f6db4106500 2024-02-09T22:58:41.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db0ff9640 1 -- 172.21.15.3:0/1760295627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da80044b0 con 0x7f6db4106500 2024-02-09T22:58:41.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6db0ff9640 1 -- 172.21.15.3:0/1760295627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da8003d30 con 0x7f6db4106500 2024-02-09T22:58:41.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.645+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db41a48c0 con 0x7f6db4106500 2024-02-09T22:58:41.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 210257 2.57 MB/sec execute 275 sec latency 13.200 ms 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.647+0000 7f6db0ff9640 1 -- 172.21.15.3:0/1760295627 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6da8004df0 con 0x7f6db4106500 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6db0ff9640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d9c0a2230 0x7f6d9c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 Processor -- start 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6db0ff9640 1 -- 172.21.15.3:0/1760295627 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6da80effb0 con 0x7f6db4106500 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6db37fe640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d9c0a2230 0x7f6d9c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 -- start start 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c65ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c77570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c79ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97c21020 con 0x557b97c77ab0 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97bd4320 con 0x557b97c66000 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.648+0000 7f6dbce20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97bd1850 con 0x557b97c656e0 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c65ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c79ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c65ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35960/0 (socket says 172.21.15.3:35960) 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 -- 172.21.15.3:0/2141578848 learned_addr learned my addr 172.21.15.3:0/2141578848 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d9c0a2230 0x7f6d9c0a46f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005fd0 tx=0x7f6da4009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c77570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 -- 172.21.15.3:0/2141578848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 msgr2=0x557b97c77570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c77570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 -- 172.21.15.3:0/2141578848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 msgr2=0x557b97c79ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c79ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 -- 172.21.15.3:0/2141578848 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b97b8f210 con 0x557b97c656e0 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6db37fe640 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c65ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6da400e740 tx=0x7f6da4014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=776d9c571cb178e5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6d99ffb640 1 -- 172.21.15.3:0/2141578848 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4003970 con 0x557b97c656e0 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6d99ffb640 1 -- 172.21.15.3:0/2141578848 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da4004020 con 0x557b97c656e0 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.649+0000 7f6d99ffb640 1 -- 172.21.15.3:0/2141578848 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da401da40 con 0x557b97c656e0 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 -- 172.21.15.3:0/2141578848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 msgr2=0x557b97c65ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c65ac0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6da400e740 tx=0x7f6da4014000 comp rx=0 tx=0).stop 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 -- 172.21.15.3:0/2141578848 shutdown_connections 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c77570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c79ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 --2- 172.21.15.3:0/2141578848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c65ac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 -- 172.21.15.3:0/2141578848 >> 172.21.15.3:0/2141578848 conn(0x557b97c60100 msgr2=0x557b97c604e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 -- 172.21.15.3:0/2141578848 shutdown_connections 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 -- 172.21.15.3:0/2141578848 wait complete. 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.650+0000 7f6dbce20740 1 -- start start 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 Processor -- start 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c74ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c74ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c75530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97bcd030 con 0x557b97c77ab0 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b978a0430 con 0x557b97c66000 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b9785ce00 con 0x557b97c656e0 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6dbce20740 10 client.0 _ll_register_callbacks cb 0x557b97bc2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c75530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c74ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c75530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32806/0 (socket says 172.21.15.3:32806) 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db3fff640 1 -- 172.21.15.3:0/247021311 learned_addr learned my addr 172.21.15.3:0/247021311 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c74ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:35966/0 (socket says 172.21.15.3:35966) 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c74ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 -- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 msgr2=0x557b97c74ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c74ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 -- 172.21.15.3:0/247021311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 msgr2=0x557b97c75530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c75530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.651+0000 7f6db37fe640 1 -- 172.21.15.3:0/247021311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da4017050 con 0x557b97c656e0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6db37fe640 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c74ab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6da400efd0 tx=0x7f6da4004210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f305bc5d2f0fd54c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6d92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6d92ffd640 1 -- 172.21.15.3:0/247021311 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4003970 con 0x557b97c656e0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557b978cc7e0 con 0x557b97c656e0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6d92ffd640 1 -- 172.21.15.3:0/247021311 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6da4003ad0 con 0x557b97c656e0 2024-02-09T22:58:41.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6d92ffd640 1 -- 172.21.15.3:0/247021311 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6da4003c90 con 0x557b97c656e0 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 10 client.27415 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x557b97bcb6b0 con 0x557b97c656e0 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6d92ffd640 1 -- 172.21.15.3:0/247021311 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557b978cc7e0 con 0x557b97c656e0 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6d92ffd640 1 -- 172.21.15.3:0/247021311 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f6da401e730 con 0x557b97c656e0 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 10 client.27415 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 10 client.27415 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 20 client.27415 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97c87c90 0x557b97c8a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 4 client.27415 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.652+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557b97909720 con 0x557b97c87c90 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.653+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97c87c90 0x557b97c8a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.653+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97c87c90 0x557b97c8a0b0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92c3fce13710dfc6 server_cookie=ceef050820adf110 in_seq=0 out_seq=0 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.653+0000 7f6d92ffd640 10 client.27415 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6d92ffd640 1 -- 172.21.15.3:0/247021311 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557b97909720 con 0x557b97c87c90 2024-02-09T22:58:41.662 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6d92ffd640 10 client.27415 handle_command_reply: tid=0 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 client.27415 shutdown 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 msgr2=0x557b97c74ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c74ab0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f6da400efd0 tx=0x7f6da4004210 comp rx=0 tx=0).stop 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 shutdown_connections 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97c87c90 0x557b97c8a0b0 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c74ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c77ab0 0x557b97c75530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 --2- 172.21.15.3:0/247021311 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c656e0 0x557b97c74ab0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6d92ffd640 0 client.27415 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 >> 172.21.15.3:0/247021311 conn(0x557b97c60100 msgr2=0x557b97c61e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 shutdown_connections 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 1 -- 172.21.15.3:0/247021311 wait complete. 2024-02-09T22:58:41.663 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.654+0000 7f6dbce20740 20 client.27415 trim_cache size 0 max 16384 2024-02-09T22:58:41.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.666+0000 7f6dbce20740 1 Processor -- start 2024-02-09T22:58:41.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 -- start start 2024-02-09T22:58:41.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97c61bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c7e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97c7ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97b52130 con 0x557b97c656e0 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97c76010 con 0x557b97c66000 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6dbce20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97c8f060 con 0x557b97c77ab0 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97c61bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97c61bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32822/0 (socket says 172.21.15.3:32822) 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 -- 172.21.15.3:0/1570113965 learned_addr learned my addr 172.21.15.3:0/1570113965 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c7e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db3fff640 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97c7ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 -- 172.21.15.3:0/1570113965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 msgr2=0x557b97c7ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97c7ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 -- 172.21.15.3:0/1570113965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 msgr2=0x557b97c7e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c7e780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 -- 172.21.15.3:0/1570113965 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557b97b67c90 con 0x557b97c656e0 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db3fff640 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97c7ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c7e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6db37fe640 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97c61bf0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f80 tx=0x7f6da400e690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=136796099db890cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6d997fa640 1 -- 172.21.15.3:0/1570113965 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4005500 con 0x557b97c656e0 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.667+0000 7f6d997fa640 1 -- 172.21.15.3:0/1570113965 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da40090b0 con 0x557b97c656e0 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6d997fa640 1 -- 172.21.15.3:0/1570113965 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da401d910 con 0x557b97c656e0 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 -- 172.21.15.3:0/1570113965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 msgr2=0x557b97c61bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97c61bf0 secure :-1 s=READY pgs=1247 cs=0 l=1 rev1=1 crypto rx=0x7f6da4000f80 tx=0x7f6da400e690 comp rx=0 tx=0).stop 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 -- 172.21.15.3:0/1570113965 shutdown_connections 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97c7e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97c7ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 --2- 172.21.15.3:0/1570113965 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97c61bf0 unknown :-1 s=CLOSED pgs=1247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 -- 172.21.15.3:0/1570113965 >> 172.21.15.3:0/1570113965 conn(0x557b97c60100 msgr2=0x557b97c614e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 -- 172.21.15.3:0/1570113965 shutdown_connections 2024-02-09T22:58:41.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.668+0000 7f6dbce20740 1 -- 172.21.15.3:0/1570113965 wait complete. 2024-02-09T22:58:41.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 -- start start 2024-02-09T22:58:41.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 Processor -- start 2024-02-09T22:58:41.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97cd7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97cd8280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97cd2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97cd2720 con 0x557b97c656e0 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97cd2890 con 0x557b97c66000 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557b97cd2a50 con 0x557b97c77ab0 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6dbce20740 10 client.0 _ll_register_callbacks cb 0x7f6da400e3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97cd7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6db37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97cd7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32824/0 (socket says 172.21.15.3:32824) 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6db37fe640 1 -- 172.21.15.3:0/156049684 learned_addr learned my addr 172.21.15.3:0/156049684 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97cd8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.669+0000 7f6db3fff640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97cd2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db37fe640 1 -- 172.21.15.3:0/156049684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 msgr2=0x557b97cd2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db37fe640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97cd2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db37fe640 1 -- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 msgr2=0x557b97cd8280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db37fe640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97cd8280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db37fe640 1 -- 172.21.15.3:0/156049684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db41a48c0 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97cd8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6db37fe640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97cd7d40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f6da401ea90 tx=0x7f6da4005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2d6ea482c1f3bf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6d92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6d92ffd640 1 -- 172.21.15.3:0/156049684 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da4003eb0 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6d92ffd640 1 -- 172.21.15.3:0/156049684 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f6da401e640 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6da4004020 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6d92ffd640 1 -- 172.21.15.3:0/156049684 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6da401e840 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6dbce20740 10 client.18258 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x7f6dac009aa0 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6d92ffd640 1 -- 172.21.15.3:0/156049684 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da4004020 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.670+0000 7f6d92ffd640 1 -- 172.21.15.3:0/156049684 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f6da4041840 con 0x557b97c656e0 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6dbce20740 10 client.18258 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:41.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6dbce20740 10 client.18258 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6dbce20740 20 client.18258 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6dbce20740 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97cdf850 0x557b97ce1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6dbce20740 4 client.18258 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557b97958890 con 0x557b97cdf850 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97cdf850 0x557b97ce1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6d92ffd640 10 client.18258 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.671+0000 7f6db2ffd640 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97cdf850 0x557b97ce1c70 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43cabf1a5866d7a6 server_cookie=98a738f4327dfc4e in_seq=0 out_seq=0 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6d92ffd640 1 -- 172.21.15.3:0/156049684 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557b97958890 con 0x557b97cdf850 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6d92ffd640 10 client.18258 handle_command_reply: tid=0 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 client.18258 shutdown 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 msgr2=0x557b97cd7d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97cd7d40 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f6da401ea90 tx=0x7f6da4005590 comp rx=0 tx=0).stop 2024-02-09T22:58:41.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 shutdown_connections 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557b97cdf850 0x557b97ce1c70 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557b97c66000 0x557b97cd8280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557b97c77ab0 0x557b97cd2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 --2- 172.21.15.3:0/156049684 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557b97c656e0 0x557b97cd7d40 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 >> 172.21.15.3:0/156049684 conn(0x557b97c60100 msgr2=0x557b97c5f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6d92ffd640 0 client.18258 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 shutdown_connections 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 1 -- 172.21.15.3:0/156049684 wait complete. 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.672+0000 7f6dbce20740 20 client.18258 trim_cache size 0 max 16384 2024-02-09T22:58:41.681 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:58:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d9c0a2230 msgr2=0x7f6d9c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d9c0a2230 0x7f6d9c0a46f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6da4005fd0 tx=0x7f6da4009500 comp rx=0 tx=0).stop 2024-02-09T22:58:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 msgr2=0x7f6db419dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:41.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 0x7f6db419dc10 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f6da8009230 tx=0x7f6da802c680 comp rx=0 tx=0).stop 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 shutdown_connections 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d9c0a2230 0x7f6d9c0a46f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6db4197820 0x7f6db4197cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6db4106500 0x7f6db419dc10 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 --2- 172.21.15.3:0/1760295627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6db4101430 0x7f6db419d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 >> 172.21.15.3:0/1760295627 conn(0x7f6db40fccd0 msgr2=0x7f6db40fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 shutdown_connections 2024-02-09T22:58:41.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:41.673+0000 7f6dba257640 1 -- 172.21.15.3:0/1760295627 wait complete. 2024-02-09T22:58:41.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:41 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3190615057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:58:41.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:41 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3190615057' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:58:42.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:42 smithi003 ceph-mon[32678]: pgmap v229: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-02-09T22:58:42.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 210970 2.58 MB/sec execute 276 sec latency 456.148 ms 2024-02-09T22:58:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:42 smithi064 ceph-mon[36638]: pgmap v229: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-02-09T22:58:42.687 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:58:42.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:58:42.688 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:58:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:42 smithi089 ceph-mon[36665]: pgmap v229: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.1 MiB/s wr, 115 op/s 2024-02-09T22:58:42.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 -- 172.21.15.3:0/2683436161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c100b10 msgr2=0x7f0d5c104b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 --2- 172.21.15.3:0/2683436161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c100b10 0x7f0d5c104b50 secure :-1 s=READY pgs=1249 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009930 tx=0x7f0d4c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:42.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 -- 172.21.15.3:0/2683436161 shutdown_connections 2024-02-09T22:58:42.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 --2- 172.21.15.3:0/2683436161 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c105230 0x7f0d5c10ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 --2- 172.21.15.3:0/2683436161 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c100b10 0x7f0d5c104b50 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 --2- 172.21.15.3:0/2683436161 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d5c100160 0x7f0d5c100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 -- 172.21.15.3:0/2683436161 >> 172.21.15.3:0/2683436161 conn(0x7f0d5c0fb9a0 msgr2=0x7f0d5c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 -- 172.21.15.3:0/2683436161 shutdown_connections 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.795+0000 7f0d627a0640 1 -- 172.21.15.3:0/2683436161 wait complete. 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 Processor -- start 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 -- start start 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c100160 0x7f0d5c198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 0x7f0d5c198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d5c198f00 0x7f0d5c1a34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1a3b80 con 0x7f0d5c105230 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1a3cf0 con 0x7f0d5c198f00 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d627a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d5c1a3eb0 con 0x7f0d5c100160 2024-02-09T22:58:42.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 0x7f0d5c198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 0x7f0d5c198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32848/0 (socket says 172.21.15.3:32848) 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1574042811 learned_addr learned my addr 172.21.15.3:0/1574042811 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d5c198f00 0x7f0d5c1a34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d6179e640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c100160 0x7f0d5c198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1574042811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c100160 msgr2=0x7f0d5c198430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c100160 0x7f0d5c198430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1574042811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d5c198f00 msgr2=0x7f0d5c1a34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d5c198f00 0x7f0d5c1a34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.796+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1574042811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d4c009560 con 0x7f0d5c105230 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.797+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 0x7f0d5c198970 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009230 tx=0x7f0d4c004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.797+0000 7f0d527fc640 1 -- 172.21.15.3:0/1574042811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0043d0 con 0x7f0d5c105230 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.797+0000 7f0d527fc640 1 -- 172.21.15.3:0/1574042811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d4c0052d0 con 0x7f0d5c105230 2024-02-09T22:58:42.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.797+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d5c1a4130 con 0x7f0d5c105230 2024-02-09T22:58:42.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.797+0000 7f0d527fc640 1 -- 172.21.15.3:0/1574042811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c003d80 con 0x7f0d5c105230 2024-02-09T22:58:42.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.797+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c1a4640 con 0x7f0d5c105230 2024-02-09T22:58:42.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d527fc640 1 -- 172.21.15.3:0/1574042811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0d4c005440 con 0x7f0d5c105230 2024-02-09T22:58:42.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 Processor -- start 2024-02-09T22:58:42.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 -- start start 2024-02-09T22:58:42.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391aaf6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf69d0 0x56391ab03750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab06080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391aa64000 con 0x56391ab03c90 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391aa61530 con 0x56391aaf69d0 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d665d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391aa5cd10 con 0x56391aaf60b0 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.799+0000 7f0d527fc640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0d480a2220 0x7f0d480a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0d480a2220 0x7f0d480a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab06080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab06080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32858/0 (socket says 172.21.15.3:32858) 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d61f9f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391aaf6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 -- 172.21.15.3:0/3718116628 learned_addr learned my addr 172.21.15.3:0/3718116628 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d527fc640 1 -- 172.21.15.3:0/1574042811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0d4c0f00a0 con 0x7f0d5c105230 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d6179e640 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf69d0 0x56391ab03750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 -- 172.21.15.3:0/3718116628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 msgr2=0x56391aaf6490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391aaf6490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 -- 172.21.15.3:0/3718116628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf69d0 msgr2=0x56391ab03750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf69d0 0x56391ab03750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 -- 172.21.15.3:0/3718116628 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56391aa03210 con 0x56391ab03c90 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0d480a2220 0x7f0d480a46e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0d54007fd0 tx=0x7f0d54007d60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab06080 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0bcc00 tx=0x7f0d4c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba6475dd495d36e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d477fe640 1 -- 172.21.15.3:0/3718116628 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0bc7b0 con 0x56391ab03c90 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d477fe640 1 -- 172.21.15.3:0/3718116628 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d4c0bc5d0 con 0x56391ab03c90 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.800+0000 7f0d477fe640 1 -- 172.21.15.3:0/3718116628 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0b92e0 con 0x56391ab03c90 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 -- 172.21.15.3:0/3718116628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 msgr2=0x56391ab06080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab06080 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c0bcc00 tx=0x7f0d4c009260 comp rx=0 tx=0).stop 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 -- 172.21.15.3:0/3718116628 shutdown_connections 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf69d0 0x56391ab03750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab06080 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 --2- 172.21.15.3:0/3718116628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391aaf6490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 -- 172.21.15.3:0/3718116628 >> 172.21.15.3:0/3718116628 conn(0x56391aaf0ab0 msgr2=0x56391aaf0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 -- 172.21.15.3:0/3718116628 shutdown_connections 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 -- 172.21.15.3:0/3718116628 wait complete. 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.801+0000 7f0d665d4740 1 -- start start 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 Processor -- start 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391ab00b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 0x56391ab01090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391ab03c90 0x56391ab015d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391aa01e60 con 0x56391aaf69d0 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391a7304b0 con 0x56391ab03c90 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391a9e2670 con 0x56391aaf60b0 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d665d4740 10 client.0 _ll_register_callbacks cb 0x56391a9fa190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:42.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 0x56391ab01090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 0x56391ab01090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32872/0 (socket says 172.21.15.3:32872) 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1221843790 learned_addr learned my addr 172.21.15.3:0/1221843790 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d6179e640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391ab00b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391ab03c90 0x56391ab015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1221843790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 msgr2=0x56391ab00b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391ab00b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391ab03c90 msgr2=0x56391ab015d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391ab03c90 0x56391ab015d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 -- 172.21.15.3:0/1221843790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d5c1a4640 con 0x56391aaf69d0 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391ab03c90 0x56391ab015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d6179e640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391ab00b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 0x56391ab01090 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c03e040 tx=0x7f0d4c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa1268b414e91bac server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.802+0000 7f0d44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:42.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d44ff9640 1 -- 172.21.15.3:0/1221843790 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d4c0bc7b0 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d44ff9640 1 -- 172.21.15.3:0/1221843790 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0d4c0036d0 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56391a75c860 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d44ff9640 1 -- 172.21.15.3:0/1221843790 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d4c0bd370 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 10 client.18270 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x56391aa5b390 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d44ff9640 1 -- 172.21.15.3:0/1221843790 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56391a75c860 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d44ff9640 1 -- 172.21.15.3:0/1221843790 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0d4c03c070 con 0x56391aaf69d0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 10 client.18270 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 10 client.18270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 20 client.18270 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab134a0 0x56391ab15890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 4 client.18270 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56391a7ad410 con 0x56391ab134a0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.803+0000 7f0d6179e640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab134a0 0x56391ab15890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.814 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d6179e640 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab134a0 0x56391ab15890 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6387c4833ecf8534 server_cookie=2ca662288c53ca4e in_seq=0 out_seq=0 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d44ff9640 10 client.18270 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d44ff9640 1 -- 172.21.15.3:0/1221843790 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56391a7ad410 con 0x56391ab134a0 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d44ff9640 10 client.18270 handle_command_reply: tid=0 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 client.18270 shutdown 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 msgr2=0x56391ab01090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 0x56391ab01090 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c03e040 tx=0x7f0d4c005fd0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 shutdown_connections 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab134a0 0x56391ab15890 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391ab03c90 0x56391ab015d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d44ff9640 0 client.18270 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391aaf69d0 0x56391ab01090 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 --2- 172.21.15.3:0/1221843790 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf60b0 0x56391ab00b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.804+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 >> 172.21.15.3:0/1221843790 conn(0x56391aaf0ab0 msgr2=0x56391aaf2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.805+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 shutdown_connections 2024-02-09T22:58:42.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.805+0000 7f0d665d4740 1 -- 172.21.15.3:0/1221843790 wait complete. 2024-02-09T22:58:42.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.805+0000 7f0d665d4740 20 client.18270 trim_cache size 0 max 16384 2024-02-09T22:58:42.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 Processor -- start 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 -- start start 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391aaffcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab013f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391aaef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391aafc7e0 con 0x56391ab03c90 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391ab00230 con 0x56391aaf60b0 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d665d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391ab03b00 con 0x56391aaf69d0 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391aaef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391aaef9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32882/0 (socket says 172.21.15.3:32882) 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 -- 172.21.15.3:0/78331641 learned_addr learned my addr 172.21.15.3:0/78331641 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d6179e640 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391aaffcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 -- 172.21.15.3:0/78331641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 msgr2=0x56391ab013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab013f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 -- 172.21.15.3:0/78331641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 msgr2=0x56391aaffcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391aaffcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d61f9f640 1 -- 172.21.15.3:0/78331641 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56391a9f7c70 con 0x56391ab03c90 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.820+0000 7f0d6179e640 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391aaffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391aaef9a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0d540078e0 tx=0x7f0d54004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21b034d2e7f7a233 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d46ffd640 1 -- 172.21.15.3:0/78331641 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d540129a0 con 0x56391ab03c90 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d46ffd640 1 -- 172.21.15.3:0/78331641 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d5401a040 con 0x56391ab03c90 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d46ffd640 1 -- 172.21.15.3:0/78331641 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d5400f630 con 0x56391ab03c90 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 -- 172.21.15.3:0/78331641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 msgr2=0x56391aaef9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391aaef9a0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f0d540078e0 tx=0x7f0d54004ab0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 -- 172.21.15.3:0/78331641 shutdown_connections 2024-02-09T22:58:42.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391aaef9a0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 --2- 172.21.15.3:0/78331641 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391aaffcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 -- 172.21.15.3:0/78331641 >> 172.21.15.3:0/78331641 conn(0x56391aaf0ab0 msgr2=0x56391aaef290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 -- 172.21.15.3:0/78331641 shutdown_connections 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.821+0000 7f0d665d4740 1 -- 172.21.15.3:0/78331641 wait complete. 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 -- start start 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 Processor -- start 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391ab09be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab0a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab63980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391ab63ec0 con 0x56391ab03c90 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391ab64030 con 0x56391aaf60b0 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56391ab641f0 con 0x56391aaf69d0 2024-02-09T22:58:42.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d665d4740 10 client.0 _ll_register_callbacks cb 0x7f0d58009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab63980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab63980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32888/0 (socket says 172.21.15.3:32888) 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 -- 172.21.15.3:0/2175404133 learned_addr learned my addr 172.21.15.3:0/2175404133 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d6179e640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391ab09be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d60f9d640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab0a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 -- 172.21.15.3:0/2175404133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 msgr2=0x56391ab0a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab0a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 -- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 msgr2=0x56391ab09be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.822+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391ab09be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d61f9f640 1 -- 172.21.15.3:0/2175404133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d54014530 con 0x56391ab03c90 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d61f9f640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab63980 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0d540078b0 tx=0x7f0d54021bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4fab99b3d775258 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:42.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d44ff9640 1 -- 172.21.15.3:0/2175404133 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d54004d20 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d44ff9640 1 -- 172.21.15.3:0/2175404133 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0d540129b0 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d4c0bc5d0 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d44ff9640 1 -- 172.21.15.3:0/2175404133 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d54004e80 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 10 client.18276 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x56391aa1dac0 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d44ff9640 1 -- 172.21.15.3:0/2175404133 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d4c0bc5d0 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d44ff9640 1 -- 172.21.15.3:0/2175404133 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f0d5402a450 con 0x56391ab03c90 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 10 client.18276 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 10 client.18276 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 20 client.18276 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab6bb20 0x56391ab6df40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 4 client.18276 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.823+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56391a7997b0 con 0x56391ab6bb20 2024-02-09T22:58:42.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.824+0000 7f0d6179e640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab6bb20 0x56391ab6df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.824+0000 7f0d44ff9640 10 client.18276 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.824+0000 7f0d6179e640 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab6bb20 0x56391ab6df40 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa573ebfe2571f6b server_cookie=c06472f203de65ca in_seq=0 out_seq=0 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d44ff9640 1 -- 172.21.15.3:0/2175404133 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56391a7997b0 con 0x56391ab6bb20 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d44ff9640 10 client.18276 handle_command_reply: tid=0 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 client.18276 shutdown 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 msgr2=0x56391ab63980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab63980 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 crypto rx=0x7f0d540078b0 tx=0x7f0d54021bb0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 shutdown_connections 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56391ab6bb20 0x56391ab6df40 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56391ab03c90 0x56391ab63980 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56391aaf69d0 0x56391ab0a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d44ff9640 0 client.18276 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 --2- 172.21.15.3:0/2175404133 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56391aaf60b0 0x56391ab09be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 >> 172.21.15.3:0/2175404133 conn(0x56391aaf0ab0 msgr2=0x56391aaefb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 shutdown_connections 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 1 -- 172.21.15.3:0/2175404133 wait complete. 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.825+0000 7f0d665d4740 20 client.18276 trim_cache size 0 max 16384 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "ef5f6af4-1f83-46a4-a1ee-98973eaa45ae", 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:58:42.835 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:58:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0d480a2220 msgr2=0x7f0d480a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0d480a2220 0x7f0d480a46e0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0d54007fd0 tx=0x7f0d54007d60 comp rx=0 tx=0).stop 2024-02-09T22:58:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 msgr2=0x7f0d5c198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 0x7f0d5c198970 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f0d4c009230 tx=0x7f0d4c004c00 comp rx=0 tx=0).stop 2024-02-09T22:58:42.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 shutdown_connections 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0d480a2220 0x7f0d480a46e0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0d5c198f00 0x7f0d5c1a34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0d5c105230 0x7f0d5c198970 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 --2- 172.21.15.3:0/1574042811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0d5c100160 0x7f0d5c198430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 >> 172.21.15.3:0/1574042811 conn(0x7f0d5c0fb9a0 msgr2=0x7f0d5c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 shutdown_connections 2024-02-09T22:58:42.837 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.826+0000 7f0d627a0640 1 -- 172.21.15.3:0/1574042811 wait complete. 2024-02-09T22:58:42.842 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:58:42.842 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:58:42.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/151040370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8101440 msgr2=0x7f3db8101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/151040370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8101440 0x7f3db8101820 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009930 tx=0x7f3db402bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:42.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/151040370 shutdown_connections 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/151040370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8106510 0x7f3db810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/151040370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8101df0 0x7f3db8105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/151040370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8101440 0x7f3db8101820 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/151040370 >> 172.21.15.3:0/151040370 conn(0x7f3db80fccc0 msgr2=0x7f3db80ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/151040370 shutdown_connections 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.952+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/151040370 wait complete. 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 Processor -- start 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 -- start start 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8101440 0x7f3db819d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 0x7f3db819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 0x7f3db8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db81981e0 con 0x7f3db8197710 2024-02-09T22:58:42.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db8198350 con 0x7f3db8101440 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbfbe2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db8198510 con 0x7f3db8106510 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbe17f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 0x7f3db8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbd17d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 0x7f3db819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbe17f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 0x7f3db8197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32920/0 (socket says 172.21.15.3:32920) 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbd17d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 0x7f3db819db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:36060/0 (socket says 172.21.15.3:36060) 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbe17f640 1 -- 172.21.15.3:0/4138407988 learned_addr learned my addr 172.21.15.3:0/4138407988 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.953+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8101440 0x7f3db819d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbe17f640 1 -- 172.21.15.3:0/4138407988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 msgr2=0x7f3db819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 0x7f3db819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbe17f640 1 -- 172.21.15.3:0/4138407988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8101440 msgr2=0x7f3db819d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8101440 0x7f3db819d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbe17f640 1 -- 172.21.15.3:0/4138407988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db4009560 con 0x7f3db8197710 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8101440 0x7f3db819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:58:42.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 0x7f3db819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 0x7f3db8197b90 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3db000eb10 tx=0x7f3db000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3daeffd640 1 -- 172.21.15.3:0/4138407988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0004b20 con 0x7f3db8197710 2024-02-09T22:58:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3daeffd640 1 -- 172.21.15.3:0/4138407988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db0004c80 con 0x7f3db8197710 2024-02-09T22:58:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3db81a44b0 con 0x7f3db8197710 2024-02-09T22:58:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3daeffd640 1 -- 172.21.15.3:0/4138407988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0010650 con 0x7f3db8197710 2024-02-09T22:58:42.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.954+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db81a4940 con 0x7f3db8197710 2024-02-09T22:58:42.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.956+0000 7f3daeffd640 1 -- 172.21.15.3:0/4138407988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3db0005020 con 0x7f3db8197710 2024-02-09T22:58:42.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 Processor -- start 2024-02-09T22:58:42.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3daeffd640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3da40a2230 0x7f3da40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 -- start start 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3da40a2230 0x7f3da40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3daeffd640 1 -- 172.21.15.3:0/4138407988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3db0014070 con 0x7f3db8197710 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b746b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b74660d00 con 0x559b746a5820 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b746152f0 con 0x559b746a61d0 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.957+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b74612820 con 0x559b746b32c0 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbd17d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbd97e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbd17d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52348/0 (socket says 172.21.15.3:52348) 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbd17d640 1 -- 172.21.15.3:0/246808968 learned_addr learned my addr 172.21.15.3:0/246808968 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b746b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 -- 172.21.15.3:0/246808968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 msgr2=0x559b746b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b2cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3da40a2230 0x7f3da40a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3da80138b0 tx=0x7f3da8013cb0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 -- 172.21.15.3:0/246808968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 msgr2=0x559b746a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 -- 172.21.15.3:0/246808968 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b745cfff0 con 0x559b746b32c0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b746b56d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3db00cbdb0 tx=0x7f3db009aa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1770d2b4026f5dba server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3da3fff640 1 -- 172.21.15.3:0/246808968 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db009f1a0 con 0x559b746b32c0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3da3fff640 1 -- 172.21.15.3:0/246808968 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db00b1030 con 0x559b746b32c0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.958+0000 7f3da3fff640 1 -- 172.21.15.3:0/246808968 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db00a3360 con 0x559b746b32c0 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 -- 172.21.15.3:0/246808968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 msgr2=0x559b746b56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b746b56d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3db00cbdb0 tx=0x7f3db009aa70 comp rx=0 tx=0).stop 2024-02-09T22:58:42.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 -- 172.21.15.3:0/246808968 shutdown_connections 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b746b56d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/246808968 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 -- 172.21.15.3:0/246808968 >> 172.21.15.3:0/246808968 conn(0x559b746a0050 msgr2=0x559b746a0430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 -- 172.21.15.3:0/246808968 shutdown_connections 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.959+0000 7f3dc27b4740 1 -- 172.21.15.3:0/246808968 wait complete. 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 -- start start 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 Processor -- start 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746b0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b7460e000 con 0x559b746b32c0 2024-02-09T22:58:42.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b742e1430 con 0x559b746a5820 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b7429de00 con 0x559b746a61d0 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dc27b4740 10 client.0 _ll_register_callbacks cb 0x559b746027f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dbe17f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746b0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dbd17d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dbd97e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dbd97e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52352/0 (socket says 172.21.15.3:52352) 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dbd17d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:36088/0 (socket says 172.21.15.3:36088) 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.960+0000 7f3dbd97e640 1 -- 172.21.15.3:0/65838250 learned_addr learned my addr 172.21.15.3:0/65838250 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbe17f640 1 -- 172.21.15.3:0/65838250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 msgr2=0x559b746b0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbe17f640 1 -- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 msgr2=0x559b746b0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbe17f640 1 -- 172.21.15.3:0/65838250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3db81a4940 con 0x559b746b32c0 2024-02-09T22:58:42.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746b0bd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3db00a39b0 tx=0x7f3db000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ef41718fa865ac server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3da17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3da17fa640 1 -- 172.21.15.3:0/65838250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0064070 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3da17fa640 1 -- 172.21.15.3:0/65838250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3db00990f0 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559b7430d7e0 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3da17fa640 1 -- 172.21.15.3:0/65838250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3db00a3360 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dc27b4740 10 client.18288 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x559b7460c680 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.961+0000 7f3da17fa640 1 -- 172.21.15.3:0/65838250 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db007f020 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3da17fa640 1 -- 172.21.15.3:0/65838250 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3db001f060 con 0x559b746b32c0 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dc27b4740 10 client.18288 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dc27b4740 10 client.18288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dc27b4740 20 client.18288 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:42.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b746c3480 0x559b746c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dc27b4740 4 client.18288 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559b7434a720 con 0x559b746c3480 2024-02-09T22:58:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b746c3480 0x559b746c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3da17fa640 10 client.18288 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.962+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b746c3480 0x559b746c58a0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c4d31306e7dd144 server_cookie=3253d101702ee5db in_seq=0 out_seq=0 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3da17fa640 1 -- 172.21.15.3:0/65838250 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559b7434a720 con 0x559b746c3480 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3da17fa640 10 client.18288 handle_command_reply: tid=0 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 client.18288 shutdown 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 msgr2=0x559b746b0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746b0bd0 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7f3db00a39b0 tx=0x7f3db000cf70 comp rx=0 tx=0).stop 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 shutdown_connections 2024-02-09T22:58:42.972 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b746c3480 0x559b746c58a0 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746b0bd0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3da17fa640 0 client.18288 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/65838250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 >> 172.21.15.3:0/65838250 conn(0x559b746a0050 msgr2=0x559b746a1da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 shutdown_connections 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.963+0000 7f3dc27b4740 1 -- 172.21.15.3:0/65838250 wait complete. 2024-02-09T22:58:42.973 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.964+0000 7f3dc27b4740 20 client.18288 trim_cache size 0 max 16384 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.980+0000 7f3dc27b4740 1 Processor -- start 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.980+0000 7f3dc27b4740 1 -- start start 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.980+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.980+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.980+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746ad3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b74592fd0 con 0x559b746b32c0 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b746b56b0 con 0x559b746a5820 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b7469feb0 con 0x559b746a61d0 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746ad3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746ad3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32940/0 (socket says 172.21.15.3:32940) 2024-02-09T22:58:42.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 -- 172.21.15.3:0/2382405824 learned_addr learned my addr 172.21.15.3:0/2382405824 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 -- 172.21.15.3:0/2382405824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 msgr2=0x559b746b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 -- 172.21.15.3:0/2382405824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 msgr2=0x559b746b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 -- 172.21.15.3:0/2382405824 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559b745a8b30 con 0x559b746b32c0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746ad3b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f3db00974b0 tx=0x7f3db0097420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7330e93f55077a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3da37fe640 1 -- 172.21.15.3:0/2382405824 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db00b95d0 con 0x559b746b32c0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3da37fe640 1 -- 172.21.15.3:0/2382405824 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559b7430d7e0 con 0x559b746b32c0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.981+0000 7f3da37fe640 1 -- 172.21.15.3:0/2382405824 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db0095bc0 con 0x559b746b32c0 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 -- 172.21.15.3:0/2382405824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 msgr2=0x559b746ad3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746ad3b0 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f3db00974b0 tx=0x7f3db0097420 comp rx=0 tx=0).stop 2024-02-09T22:58:42.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 -- 172.21.15.3:0/2382405824 shutdown_connections 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746b32c0 0x559b746ad3b0 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746a61d0 0x559b746b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/2382405824 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a5820 0x559b746b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 -- 172.21.15.3:0/2382405824 >> 172.21.15.3:0/2382405824 conn(0x559b746a0050 msgr2=0x559b746af180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 -- 172.21.15.3:0/2382405824 shutdown_connections 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 -- 172.21.15.3:0/2382405824 wait complete. 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 -- start start 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.982+0000 7f3dc27b4740 1 Processor -- start 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746b9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b9660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b74719950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b746b9f30 con 0x559b746a5820 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b74719e90 con 0x559b746a61d0 2024-02-09T22:58:42.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559b74719fd0 con 0x559b746b32c0 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 10 client.0 _ll_register_callbacks cb 0x7f3db00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746b9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746b9120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32946/0 (socket says 172.21.15.3:32946) 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 -- 172.21.15.3:0/1474871172 learned_addr learned my addr 172.21.15.3:0/1474871172 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b9660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbe17f640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b74719950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 -- 172.21.15.3:0/1474871172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 msgr2=0x559b74719950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b74719950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 -- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 msgr2=0x559b746b9660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b9660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 -- 172.21.15.3:0/1474871172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559b7460c680 con 0x559b746a5820 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dbd97e640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746b9120 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3db402c170 tx=0x7f3db402c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e1542d7935a3af3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:42.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3da17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3da17fa640 1 -- 172.21.15.3:0/1474871172 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db4048de0 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3da17fa640 1 -- 172.21.15.3:0/1474871172 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f3db4045040 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.983+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3db00b1030 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3da17fa640 1 -- 172.21.15.3:0/1474871172 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3db4004040 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 10 client.18294 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x559b745ce850 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3da17fa640 1 -- 172.21.15.3:0/1474871172 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3db00b1030 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3da17fa640 1 -- 172.21.15.3:0/1474871172 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7f3db403c040 con 0x559b746a5820 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 10 client.18294 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 10 client.18294 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 20 client.18294 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b7471b8d0 0x559b7471dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 4 client.18294 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559b74399890 con 0x559b7471b8d0 2024-02-09T22:58:42.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.984+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b7471b8d0 0x559b7471dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3da17fa640 10 client.18294 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dbd17d640 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b7471b8d0 0x559b7471dcf0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c140136fa693601 server_cookie=384a6da9dba26e2e in_seq=0 out_seq=0 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3da17fa640 1 -- 172.21.15.3:0/1474871172 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x559b74399890 con 0x559b7471b8d0 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3da17fa640 10 client.18294 handle_command_reply: tid=0 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 client.18294 shutdown 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 msgr2=0x559b746b9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746b9120 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7f3db402c170 tx=0x7f3db402c980 comp rx=0 tx=0).stop 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 shutdown_connections 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559b7471b8d0 0x559b7471dcf0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3da17fa640 0 client.18294 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559b746a61d0 0x559b746b9660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559b746b32c0 0x559b74719950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 --2- 172.21.15.3:0/1474871172 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559b746a5820 0x559b746b9120 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 >> 172.21.15.3:0/1474871172 conn(0x559b746a0050 msgr2=0x559b7469e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 shutdown_connections 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 1 -- 172.21.15.3:0/1474871172 wait complete. 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.985+0000 7f3dc27b4740 20 client.18294 trim_cache size 0 max 16384 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: "ef5f6af4-1f83-46a4-a1ee-98973eaa45ae": { 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "ef5f6af4-1f83-46a4-a1ee-98973eaa45ae", 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:58:42.995 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:58:42.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:58:42.996 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:58:42.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3da40a2230 msgr2=0x7f3da40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3da40a2230 0x7f3da40a46f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3da80138b0 tx=0x7f3da8013cb0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 msgr2=0x7f3db8197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 0x7f3db8197b90 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7f3db000eb10 tx=0x7f3db000c2d0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 shutdown_connections 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3da40a2230 0x7f3da40a46f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3db8197710 0x7f3db8197b90 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3db8106510 0x7f3db819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 --2- 172.21.15.3:0/4138407988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3db8101440 0x7f3db819d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 >> 172.21.15.3:0/4138407988 conn(0x7f3db80fccc0 msgr2=0x7f3db80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 shutdown_connections 2024-02-09T22:58:42.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:58:42.986+0000 7f3dbfbe2640 1 -- 172.21.15.3:0/4138407988 wait complete. 2024-02-09T22:58:43.000 INFO:tasks.cephfs.filesystem:scrub status for tag:ef5f6af4-1f83-46a4-a1ee-98973eaa45ae - {'path': '/', 'tag': 'ef5f6af4-1f83-46a4-a1ee-98973eaa45ae', 'options': 'recursive,force'} 2024-02-09T22:58:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:43 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213048 2.62 MB/sec execute 277 sec latency 659.059 ms 2024-02-09T22:58:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:43 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:43 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 -- 172.21.15.3:0/4175716467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 msgr2=0x7fb36c10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 --2- 172.21.15.3:0/4175716467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c10cfe0 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7fb364009930 tx=0x7fb36402bc30 comp rx=0 tx=0).stop 2024-02-09T22:58:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 -- 172.21.15.3:0/4175716467 shutdown_connections 2024-02-09T22:58:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 --2- 172.21.15.3:0/4175716467 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c10cfe0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 --2- 172.21.15.3:0/4175716467 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c101800 0x7fb36c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 --2- 172.21.15.3:0/4175716467 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 -- 172.21.15.3:0/4175716467 >> 172.21.15.3:0/4175716467 conn(0x7fb36c075450 msgr2=0x7fb36c075860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 -- 172.21.15.3:0/4175716467 shutdown_connections 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.972+0000 7fb373b8a640 1 -- 172.21.15.3:0/4175716467 wait complete. 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 Processor -- start 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 -- start start 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c197670 0x7fb36c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c198140 con 0x7fb36c105410 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c1982b0 con 0x7fb36c100e50 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb373b8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb36c198470 con 0x7fb36c197670 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb371125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52390/0 (socket says 172.21.15.3:52390) 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb371926640 1 -- 172.21.15.3:0/3177532497 learned_addr learned my addr 172.21.15.3:0/3177532497 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.973+0000 7fb372127640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c197670 0x7fb36c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371926640 1 -- 172.21.15.3:0/3177532497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c197670 msgr2=0x7fb36c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371926640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c197670 0x7fb36c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371926640 1 -- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 msgr2=0x7fb36c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371926640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371926640 1 -- 172.21.15.3:0/3177532497 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb364009560 con 0x7fb36c100e50 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371125640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb371926640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c19d5b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb36800ebc0 tx=0x7fb36800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb362ffd640 1 -- 172.21.15.3:0/3177532497 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680099b0 con 0x7fb36c100e50 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb36c1a4410 con 0x7fb36c100e50 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb362ffd640 1 -- 172.21.15.3:0/3177532497 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb36800eec0 con 0x7fb36c100e50 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb362ffd640 1 -- 172.21.15.3:0/3177532497 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368018b60 con 0x7fb36c100e50 2024-02-09T22:58:43.979 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.974+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c1a48f0 con 0x7fb36c100e50 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb362ffd640 1 -- 172.21.15.3:0/3177532497 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb368018cc0 con 0x7fb36c100e50 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 Processor -- start 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 -- start start 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb230b0 0x559dfbb23490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb239d0 0x559dfbb2c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfba91a00 con 0x559dfbb2c7a0 2024-02-09T22:58:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfba8ef30 con 0x559dfbb239d0 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb3763fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfba8a710 con 0x559dfbb230b0 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb239d0 0x559dfbb2c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb371125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb239d0 0x559dfbb2c260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52396/0 (socket says 172.21.15.3:52396) 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb371125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32970/0 (socket says 172.21.15.3:32970) 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb371926640 1 -- 172.21.15.3:0/1762733349 learned_addr learned my addr 172.21.15.3:0/1762733349 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb362ffd640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3580a2220 0x7fb3580a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.977+0000 7fb372127640 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb230b0 0x559dfbb23490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb372127640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3580a2220 0x7fb3580a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb371125640 1 -- 172.21.15.3:0/1762733349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb230b0 msgr2=0x559dfbb23490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb371125640 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb230b0 0x559dfbb23490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb371125640 1 -- 172.21.15.3:0/1762733349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb239d0 msgr2=0x559dfbb2c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb371125640 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb239d0 0x559dfbb2c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb371125640 1 -- 172.21.15.3:0/1762733349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dfba4f3b0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb362ffd640 1 -- 172.21.15.3:0/3177532497 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb368014070 con 0x7fb36c100e50 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb371125640 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2eb90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb36c073920 tx=0x7fb35c0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25240d30af2c4aea server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb357fff640 1 -- 172.21.15.3:0/1762733349 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c004310 con 0x559dfbb2c7a0 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb372127640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3580a2220 0x7fb3580a46e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb364005940 tx=0x7fb3640058d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb357fff640 1 -- 172.21.15.3:0/1762733349 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb35c004470 con 0x559dfbb2c7a0 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.978+0000 7fb357fff640 1 -- 172.21.15.3:0/1762733349 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb35c0035f0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 -- 172.21.15.3:0/1762733349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 msgr2=0x559dfbb2eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2eb90 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7fb36c073920 tx=0x7fb35c0118d0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 -- 172.21.15.3:0/1762733349 shutdown_connections 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb239d0 0x559dfbb2c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2eb90 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1762733349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb230b0 0x559dfbb23490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 -- 172.21.15.3:0/1762733349 >> 172.21.15.3:0/1762733349 conn(0x559dfbb1d9d0 msgr2=0x559dfbb1ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 -- 172.21.15.3:0/1762733349 shutdown_connections 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 -- 172.21.15.3:0/1762733349 wait complete. 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 -- start start 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.979+0000 7fb3763fd740 1 Processor -- start 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb29790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb29cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfba2fb90 con 0x559dfbb2c7a0 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfb75e250 con 0x559dfbb230b0 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfba10200 con 0x559dfbb239d0 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3763fd740 10 client.0 _ll_register_callbacks cb 0x559dfba465d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:43.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb29790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb29790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52400/0 (socket says 172.21.15.3:52400) 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb371926640 1 -- 172.21.15.3:0/3542322779 learned_addr learned my addr 172.21.15.3:0/3542322779 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb371125640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb29cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 -- 172.21.15.3:0/3542322779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 msgr2=0x559dfbb29cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb29cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 -- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 msgr2=0x559dfbb29790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb29790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 -- 172.21.15.3:0/3542322779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb35c009be0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb371926640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb29790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb372127640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2a210 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fb36402c170 tx=0x7fb364009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d296aaa0f8e11e0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.980+0000 7fb3557fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542322779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb36402cb20 con 0x559dfbb2c7a0 2024-02-09T22:58:43.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542322779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb364004df0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559dfbab0ee0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542322779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb364004fb0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 10 client.18306 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x559dfba88d90 con 0x559dfbb2c7a0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542322779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559dfbab0ee0 con 0x559dfbb2c7a0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542322779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fb36403c040 con 0x559dfbb2c7a0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 10 client.18306 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 10 client.18306 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 20 client.18306 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb3ca40 0x559dfbb3ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 4 client.18306 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559dfb78a600 con 0x559dfbb3ca40 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.981+0000 7fb371926640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb3ca40 0x559dfbb3ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:43.991 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.982+0000 7fb3557fa640 10 client.18306 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.982+0000 7fb371926640 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb3ca40 0x559dfbb3ee60 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfe65feb1bb0b2bb server_cookie=91e955461417d972 in_seq=0 out_seq=0 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542322779 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559dfb78a600 con 0x559dfbb3ca40 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3557fa640 10 client.18306 handle_command_reply: tid=0 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 client.18306 shutdown 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 msgr2=0x559dfbb2a210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2a210 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7fb36402c170 tx=0x7fb364009260 comp rx=0 tx=0).stop 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 shutdown_connections 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb3ca40 0x559dfbb3ee60 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb2a210 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb29cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3557fa640 0 client.18306 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542322779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb29790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:43.992 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 >> 172.21.15.3:0/3542322779 conn(0x559dfbb1d9d0 msgr2=0x559dfbb1f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 shutdown_connections 2024-02-09T22:58:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542322779 wait complete. 2024-02-09T22:58:43.993 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.983+0000 7fb3763fd740 20 client.18306 trim_cache size 0 max 16384 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.997+0000 7fb3763fd740 1 Processor -- start 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.997+0000 7fb3763fd740 1 -- start start 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb2bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb239d0 0x559dfbb2a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb2c7a0 0x559dfbb269f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb3763fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfbb19310 con 0x559dfbb239d0 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb3763fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfbb17ec0 con 0x559dfbb230b0 2024-02-09T22:58:44.001 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb3763fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfbb1d500 con 0x559dfbb2c7a0 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb239d0 0x559dfbb2a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb2bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371125640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb239d0 0x559dfbb2a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:32992/0 (socket says 172.21.15.3:32992) 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371125640 1 -- 172.21.15.3:0/1503792411 learned_addr learned my addr 172.21.15.3:0/1503792411 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb372127640 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb2c7a0 0x559dfbb269f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 -- 172.21.15.3:0/1503792411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb2c7a0 msgr2=0x559dfbb269f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb2c7a0 0x559dfbb269f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 -- 172.21.15.3:0/1503792411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb239d0 msgr2=0x559dfbb2a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb239d0 0x559dfbb2a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 -- 172.21.15.3:0/1503792411 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559dfba25820 con 0x559dfbb230b0 2024-02-09T22:58:44.002 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.998+0000 7fb371926640 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb2bec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb368097b50 tx=0x7fb3680d0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5857235ca04a3119 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3577fe640 1 -- 172.21.15.3:0/1503792411 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb368010dc0 con 0x559dfbb230b0 2024-02-09T22:58:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3577fe640 1 -- 172.21.15.3:0/1503792411 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb36809b180 con 0x559dfbb230b0 2024-02-09T22:58:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3577fe640 1 -- 172.21.15.3:0/1503792411 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3680cc0b0 con 0x559dfbb230b0 2024-02-09T22:58:44.006 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 -- 172.21.15.3:0/1503792411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 msgr2=0x559dfbb2bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb2bec0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fb368097b50 tx=0x7fb3680d0040 comp rx=0 tx=0).stop 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 -- 172.21.15.3:0/1503792411 shutdown_connections 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb2c7a0 0x559dfbb269f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb239d0 0x559dfbb2a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 --2- 172.21.15.3:0/1503792411 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb2bec0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 -- 172.21.15.3:0/1503792411 >> 172.21.15.3:0/1503792411 conn(0x559dfbb1d9d0 msgr2=0x559dfbb28060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 -- 172.21.15.3:0/1503792411 shutdown_connections 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:43.999+0000 7fb3763fd740 1 -- 172.21.15.3:0/1503792411 wait complete. 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 -- start start 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 Processor -- start 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb31e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb32390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb328d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.007 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfbb92f70 con 0x559dfbb2c7a0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfbb930b0 con 0x559dfbb230b0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559dfbb931f0 con 0x559dfbb239d0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb3763fd740 10 client.0 _ll_register_callbacks cb 0x7fb35c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb372127640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb328d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb371125640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb32390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb371926640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb31e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb371125640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb32390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:36150/0 (socket says 172.21.15.3:36150) 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb372127640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb328d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33002/0 (socket says 172.21.15.3:33002) 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb371125640 1 -- 172.21.15.3:0/3542545549 learned_addr learned my addr 172.21.15.3:0/3542545549 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.000+0000 7fb371926640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb31e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52428/0 (socket says 172.21.15.3:52428) 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb372127640 1 -- 172.21.15.3:0/3542545549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 msgr2=0x559dfbb32390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb372127640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb32390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.008 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb372127640 1 -- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 msgr2=0x559dfbb31e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb372127640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb31e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb372127640 1 -- 172.21.15.3:0/3542545549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb36c1a48f0 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb371926640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb31e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb372127640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb328d0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fb364004c50 tx=0x7fb364005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffcebccebc64595e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3557fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542545549 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb364003a00 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542545549 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb3640048a0 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb35c004470 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542545549 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb364043480 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3763fd740 10 client.18312 fetch_fsmap learned FSMap version 42 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) v3 -- 0x559dfba4b6f0 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.001+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542545549 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb35c004470 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542545549 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 42) v1 ==== 7910+0+0 (secure 0 0 0) 0x7fb36403c070 con 0x559dfbb2c7a0 2024-02-09T22:58:44.009 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3763fd740 10 client.18312 fetch_fsmap finished waiting for FSMap version 42 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3763fd740 10 client.18312 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3763fd740 20 client.18312 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb94790 0x559dfbb96bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3763fd740 4 client.18312 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559dfb7db1b0 con 0x559dfbb94790 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb371926640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb94790 0x559dfbb96bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb371926640 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb94790 0x559dfbb96bb0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40a70dceac5bf21b server_cookie=e9e18ddf06cfe180 in_seq=0 out_seq=0 2024-02-09T22:58:44.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.002+0000 7fb3557fa640 10 client.18312 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.025+0000 7fb3557fa640 1 -- 172.21.15.3:0/3542545549 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559dfb7db1b0 con 0x559dfbb94790 2024-02-09T22:58:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.025+0000 7fb3557fa640 10 client.18312 handle_command_reply: tid=0 2024-02-09T22:58:44.029 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 client.18312 shutdown 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 msgr2=0x559dfbb328d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb328d0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7fb364004c50 tx=0x7fb364005b50 comp rx=0 tx=0).stop 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 shutdown_connections 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559dfbb94790 0x559dfbb96bb0 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559dfbb2c7a0 0x559dfbb328d0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559dfbb239d0 0x559dfbb32390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3557fa640 0 client.18312 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 --2- 172.21.15.3:0/3542545549 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559dfbb230b0 0x559dfbb31e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 >> 172.21.15.3:0/3542545549 conn(0x559dfbb1d9d0 msgr2=0x559dfbb1c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 shutdown_connections 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 1 -- 172.21.15.3:0/3542545549 wait complete. 2024-02-09T22:58:44.030 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb3763fd740 20 client.18312 trim_cache size 0 max 16384 2024-02-09T22:58:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:58:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:58:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:58:44.031 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3580a2220 msgr2=0x7fb3580a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb373b8a640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3580a2220 0x7fb3580a46e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fb364005940 tx=0x7fb3640058d0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 msgr2=0x7fb36c19d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.026+0000 7fb373b8a640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c19d5b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb36800ebc0 tx=0x7fb36800c6a0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 shutdown_connections 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb3580a2220 0x7fb3580a46e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb36c197670 0x7fb36c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb36c105410 0x7fb36c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 --2- 172.21.15.3:0/3177532497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb36c100e50 0x7fb36c19d5b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:44.032 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 >> 172.21.15.3:0/3177532497 conn(0x7fb36c075450 msgr2=0x7fb36c1041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 shutdown_connections 2024-02-09T22:58:44.033 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:44.027+0000 7fb373b8a640 1 -- 172.21.15.3:0/3177532497 wait complete. 2024-02-09T22:58:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: pgmap v230: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 590 KiB/s wr, 92 op/s 2024-02-09T22:58:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:58:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:58:44.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:44 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213541 2.61 MB/sec execute 278 sec latency 509.452 ms 2024-02-09T22:58:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: pgmap v230: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 590 KiB/s wr, 92 op/s 2024-02-09T22:58:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:58:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:58:44.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:44 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: pgmap v230: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 590 KiB/s wr, 92 op/s 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:44 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:58:45.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213541 2.60 MB/sec execute 279 sec latency 1509.555 ms 2024-02-09T22:58:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:46 smithi003 ceph-mon[32678]: pgmap v231: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 590 KiB/s wr, 92 op/s 2024-02-09T22:58:46.648 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213541 2.59 MB/sec execute 280 sec latency 2509.617 ms 2024-02-09T22:58:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:46 smithi064 ceph-mon[36638]: pgmap v231: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 590 KiB/s wr, 92 op/s 2024-02-09T22:58:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:46 smithi089 ceph-mon[36665]: pgmap v231: 97 pgs: 97 active+clean; 82 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 590 KiB/s wr, 92 op/s 2024-02-09T22:58:47.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213541 2.59 MB/sec execute 281 sec latency 3509.715 ms 2024-02-09T22:58:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:48 smithi003 ceph-mon[32678]: pgmap v232: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 185 op/s 2024-02-09T22:58:48.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213609 2.58 MB/sec execute 282 sec latency 3912.594 ms 2024-02-09T22:58:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:48 smithi064 ceph-mon[36638]: pgmap v232: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 185 op/s 2024-02-09T22:58:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:48 smithi089 ceph-mon[36665]: pgmap v232: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.1 MiB/s wr, 185 op/s 2024-02-09T22:58:49.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213609 2.57 MB/sec execute 283 sec latency 1554.393 ms 2024-02-09T22:58:50.529 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:50 smithi089 ceph-mon[36665]: pgmap v233: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 867 KiB/s wr, 145 op/s 2024-02-09T22:58:50.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:50 smithi003 ceph-mon[32678]: pgmap v233: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 867 KiB/s wr, 145 op/s 2024-02-09T22:58:50.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213609 2.56 MB/sec execute 284 sec latency 2554.467 ms 2024-02-09T22:58:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:50 smithi064 ceph-mon[36638]: pgmap v233: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 867 KiB/s wr, 145 op/s 2024-02-09T22:58:51.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213609 2.55 MB/sec execute 285 sec latency 3554.564 ms 2024-02-09T22:58:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:52 smithi003 ceph-mon[32678]: pgmap v234: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 165 op/s 2024-02-09T22:58:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:52 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:52.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 213609 2.54 MB/sec execute 286 sec latency 4554.672 ms 2024-02-09T22:58:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:52 smithi064 ceph-mon[36638]: pgmap v234: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 165 op/s 2024-02-09T22:58:52.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:52.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:52.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:52.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:52 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:52.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:52 smithi089 ceph-mon[36665]: pgmap v234: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 165 op/s 2024-02-09T22:58:52.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:52.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:52.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:52.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:52 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:53 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:58:53.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 214512 2.55 MB/sec execute 287 sec latency 4955.598 ms 2024-02-09T22:58:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:53 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:58:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:53 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:58:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:54 smithi003 ceph-mon[32678]: pgmap v235: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 823 KiB/s wr, 113 op/s 2024-02-09T22:58:54.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 214512 2.54 MB/sec execute 288 sec latency 1135.075 ms 2024-02-09T22:58:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:54 smithi064 ceph-mon[36638]: pgmap v235: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 823 KiB/s wr, 113 op/s 2024-02-09T22:58:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:54 smithi089 ceph-mon[36665]: pgmap v235: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 823 KiB/s wr, 113 op/s 2024-02-09T22:58:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:55 smithi003 ceph-mon[32678]: pgmap v236: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 823 KiB/s wr, 113 op/s 2024-02-09T22:58:55.649 INFO:tasks.workunit.client.0.smithi003.stdout: 1 214512 2.53 MB/sec execute 289 sec latency 2135.186 ms 2024-02-09T22:58:55.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:55 smithi064 ceph-mon[36638]: pgmap v236: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 823 KiB/s wr, 113 op/s 2024-02-09T22:58:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:55 smithi089 ceph-mon[36665]: pgmap v236: 97 pgs: 97 active+clean; 83 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 823 KiB/s wr, 113 op/s 2024-02-09T22:58:56.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 214512 2.52 MB/sec execute 290 sec latency 3135.231 ms 2024-02-09T22:58:57.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 -- 172.21.15.3:0/1531548280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 msgr2=0x7ff9f410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/1531548280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f410cfe0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec009cd0 tx=0x7ff9ec02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 -- 172.21.15.3:0/1531548280 shutdown_connections 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/1531548280 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9f4100cf0 0x7ff9f4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/1531548280 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f410cfe0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/1531548280 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9f4100340 0x7ff9f4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 -- 172.21.15.3:0/1531548280 >> 172.21.15.3:0/1531548280 conn(0x7ff9f4077220 msgr2=0x7ff9f4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 -- 172.21.15.3:0/1531548280 shutdown_connections 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.119+0000 7ff9fa30d640 1 -- 172.21.15.3:0/1531548280 wait complete. 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 Processor -- start 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 -- start start 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9f4100340 0x7ff9f419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.126 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9f4197730 0x7ff9f4197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f41981e0 con 0x7ff9f4105410 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4198350 con 0x7ff9f4100340 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9fa30d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9f4198510 con 0x7ff9f4197730 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f419db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59140/0 (socket says 172.21.15.3:59140) 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 -- 172.21.15.3:0/2791370111 learned_addr learned my addr 172.21.15.3:0/2791370111 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9f4100340 0x7ff9f419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9f4197730 0x7ff9f4197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 -- 172.21.15.3:0/2791370111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9f4197730 msgr2=0x7ff9f4197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9f4197730 0x7ff9f4197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 -- 172.21.15.3:0/2791370111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9f4100340 msgr2=0x7ff9f419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9f4100340 0x7ff9f419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.127 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.120+0000 7ff9f37fe640 1 -- 172.21.15.3:0/2791370111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec009900 con 0x7ff9f4105410 2024-02-09T22:58:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.121+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f419db20 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800d770 tx=0x7ff9e800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.121+0000 7ff9f17fa640 1 -- 172.21.15.3:0/2791370111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e8009c30 con 0x7ff9f4105410 2024-02-09T22:58:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.121+0000 7ff9f17fa640 1 -- 172.21.15.3:0/2791370111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9e800de80 con 0x7ff9f4105410 2024-02-09T22:58:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.121+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9f41a4450 con 0x7ff9f4105410 2024-02-09T22:58:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.121+0000 7ff9f17fa640 1 -- 172.21.15.3:0/2791370111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e800f6b0 con 0x7ff9f4105410 2024-02-09T22:58:57.128 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.121+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f41a4960 con 0x7ff9f4105410 2024-02-09T22:58:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.123+0000 7ff9f17fa640 1 -- 172.21.15.3:0/2791370111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff9e8010460 con 0x7ff9f4105410 2024-02-09T22:58:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 Processor -- start 2024-02-09T22:58:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f17fa640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9dc0a2220 0x7ff9dc0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 -- start start 2024-02-09T22:58:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9dc0a2220 0x7ff9dc0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f17fa640 1 -- 172.21.15.3:0/2791370111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9e80cddc0 con 0x7ff9f4105410 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628bf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628c8260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628c87a0 0x55ab628cab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6282da00 con 0x55ab628bf9d0 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6282af30 con 0x55ab628bf0b0 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab62826710 con 0x55ab628c87a0 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628bf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628c8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628bf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:56130/0 (socket says 172.21.15.3:56130) 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628c8260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59156/0 (socket says 172.21.15.3:59156) 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f3fff640 1 -- 172.21.15.3:0/2911485619 learned_addr learned my addr 172.21.15.3:0/2911485619 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.124+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628c87a0 0x55ab628cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f3fff640 1 -- 172.21.15.3:0/2911485619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628c87a0 msgr2=0x55ab628cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628c87a0 0x55ab628cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9dc0a2220 0x7ff9dc0a46e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec005cc0 tx=0x7ff9ec032ef0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f3fff640 1 -- 172.21.15.3:0/2911485619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 msgr2=0x55ab628c8260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628c8260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f3fff640 1 -- 172.21.15.3:0/2911485619 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab627eb3b0 con 0x55ab628bf0b0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628bf490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009d20 tx=0x7ff9e4011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a37d62866babc19f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9da7fc640 1 -- 172.21.15.3:0/2911485619 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4019630 con 0x55ab628bf0b0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9da7fc640 1 -- 172.21.15.3:0/2911485619 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9e4019c70 con 0x55ab628bf0b0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.125+0000 7ff9da7fc640 1 -- 172.21.15.3:0/2911485619 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e400ecb0 con 0x55ab628bf0b0 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/2911485619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 msgr2=0x55ab628bf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628bf490 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4009d20 tx=0x7ff9e4011810 comp rx=0 tx=0).stop 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/2911485619 shutdown_connections 2024-02-09T22:58:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628c87a0 0x55ab628cab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628c8260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/2911485619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628bf490 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/2911485619 >> 172.21.15.3:0/2911485619 conn(0x55ab628b99d0 msgr2=0x55ab628b9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/2911485619 shutdown_connections 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/2911485619 wait complete. 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 -- start start 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.126+0000 7ff9fcfbb740 1 Processor -- start 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628c5840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628c5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628c87a0 0x55ab628c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab627cbb90 con 0x55ab628c87a0 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab624fa250 con 0x55ab628bf0b0 2024-02-09T22:58:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab627ac200 con 0x55ab628bf9d0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9fcfbb740 10 client.0 _ll_register_callbacks cb 0x55ab627e25d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f88aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628c87a0 0x55ab628c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628c5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f88aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628c87a0 0x55ab628c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59162/0 (socket says 172.21.15.3:59162) 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f88aa640 1 -- 172.21.15.3:0/4126311458 learned_addr learned my addr 172.21.15.3:0/4126311458 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628c5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 -- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 msgr2=0x55ab628c5840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628c5840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 -- 172.21.15.3:0/4126311458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628c87a0 msgr2=0x55ab628c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628c87a0 0x55ab628c62c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 -- 172.21.15.3:0/4126311458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9e4009950 con 0x55ab628bf9d0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628c5d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80cb4e0 tx=0x7ff9e809a8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d2565a66fd3e4c4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.127+0000 7ff9d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:58:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9d37fe640 1 -- 172.21.15.3:0/4126311458 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e80cb760 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9d37fe640 1 -- 172.21.15.3:0/4126311458 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9e80cb8c0 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ab6284cee0 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9d37fe640 1 -- 172.21.15.3:0/4126311458 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9e8096ea0 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 10 client.27481 fetch_fsmap learned FSMap version 43 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55ab62824d90 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9d37fe640 1 -- 172.21.15.3:0/4126311458 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ab6284cee0 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9d37fe640 1 -- 172.21.15.3:0/4126311458 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 43) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff9e80ba0b0 con 0x55ab628bf9d0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 10 client.27481 fetch_fsmap finished waiting for FSMap version 43 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 10 client.27481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 20 client.27481 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab628d8aa0 0x55ab628daec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 4 client.27481 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.128+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ab62526600 con 0x55ab628d8aa0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.129+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab628d8aa0 0x55ab628daec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.129+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab628d8aa0 0x55ab628daec0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae6aeafb2129a10 server_cookie=1a971368fe9ab447 in_seq=0 out_seq=0 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.129+0000 7ff9d37fe640 10 client.27481 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9d37fe640 1 -- 172.21.15.3:0/4126311458 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ab62526600 con 0x55ab628d8aa0 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9d37fe640 10 client.27481 handle_command_reply: tid=0 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 client.27481 shutdown 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 msgr2=0x55ab628c5d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628c5d80 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff9e80cb4e0 tx=0x7ff9e809a8f0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 shutdown_connections 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab628d8aa0 0x55ab628daec0 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628c87a0 0x55ab628c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628c5d80 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/4126311458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628bf0b0 0x55ab628c5840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9d37fe640 0 client.27481 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 >> 172.21.15.3:0/4126311458 conn(0x55ab628b99d0 msgr2=0x55ab628bb720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 shutdown_connections 2024-02-09T22:58:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/4126311458 wait complete. 2024-02-09T22:58:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.130+0000 7ff9fcfbb740 20 client.27481 trim_cache size 0 max 16384 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.144+0000 7ff9fcfbb740 1 Processor -- start 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.144+0000 7ff9fcfbb740 1 -- start start 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 0x55ab628c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628b4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628c60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab628ba9d0 con 0x55ab628bf0b0 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab628b9500 con 0x55ab628c87a0 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab628ba760 con 0x55ab628bf9d0 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 0x55ab628c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 0x55ab628c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59168/0 (socket says 172.21.15.3:59168) 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 -- 172.21.15.3:0/3456293679 learned_addr learned my addr 172.21.15.3:0/3456293679 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628c60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628b4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 -- 172.21.15.3:0/3456293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 msgr2=0x55ab628b4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628b4cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 -- 172.21.15.3:0/3456293679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 msgr2=0x55ab628c60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628c60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 -- 172.21.15.3:0/3456293679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ab627c1820 con 0x55ab628bf0b0 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.145+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 0x55ab628c7f70 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff9e401c040 tx=0x7ff9e400e8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f7d652425030722 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9d9ffb640 1 -- 172.21.15.3:0/3456293679 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4009ed0 con 0x55ab628bf0b0 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9d9ffb640 1 -- 172.21.15.3:0/3456293679 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9e400ee40 con 0x55ab628bf0b0 2024-02-09T22:58:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9d9ffb640 1 -- 172.21.15.3:0/3456293679 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e4019690 con 0x55ab628bf0b0 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/3456293679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 msgr2=0x55ab628c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 0x55ab628c7f70 secure :-1 s=READY pgs=1266 cs=0 l=1 rev1=1 crypto rx=0x7ff9e401c040 tx=0x7ff9e400e8f0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/3456293679 shutdown_connections 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628c60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf9d0 0x55ab628b4cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/3456293679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf0b0 0x55ab628c7f70 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.155 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/3456293679 >> 172.21.15.3:0/3456293679 conn(0x55ab628b99d0 msgr2=0x55ab628c4110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/3456293679 shutdown_connections 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.146+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/3456293679 wait complete. 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 -- start start 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 Processor -- start 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 0x55ab628cf860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628cdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628ce140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab628ce820 con 0x55ab628bf9d0 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab628ce990 con 0x55ab628c87a0 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ab6292eff0 con 0x55ab628bf0b0 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9fcfbb740 10 client.0 _ll_register_callbacks cb 0x7ff9e800d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:58:57.156 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628cdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 0x55ab628cf860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9f3fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 0x55ab628cf860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:59424/0 (socket says 172.21.15.3:59424) 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9f3fff640 1 -- 172.21.15.3:0/1882503869 learned_addr learned my addr 172.21.15.3:0/1882503869 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9f37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628cdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:59172/0 (socket says 172.21.15.3:59172) 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.147+0000 7ff9f88aa640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628ce140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f3fff640 1 -- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 msgr2=0x55ab628ce140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628ce140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f3fff640 1 -- 172.21.15.3:0/1882503869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 msgr2=0x55ab628cdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628cdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f3fff640 1 -- 172.21.15.3:0/1882503869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9f410d7f0 con 0x55ab628bf0b0 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9f3fff640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 0x55ab628cf860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005890 tx=0x7ff9e40093b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f272c717d1129fc4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:58:57.157 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9d37fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9d37fe640 1 -- 172.21.15.3:0/1882503869 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9e40050b0 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9d37fe640 1 -- 172.21.15.3:0/1882503869 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff9e4005210 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff9e4019c70 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9d37fe640 1 -- 172.21.15.3:0/1882503869 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9e400e590 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9fcfbb740 10 client.27487 fetch_fsmap learned FSMap version 43 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.148+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) v3 -- 0x55ab627eb3b0 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9d37fe640 1 -- 172.21.15.3:0/1882503869 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9e4019c70 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9d37fe640 1 -- 172.21.15.3:0/1882503869 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 43) v1 ==== 7902+0+0 (secure 0 0 0) 0x7ff9e40035f0 con 0x55ab628bf0b0 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9fcfbb740 10 client.27487 fetch_fsmap finished waiting for FSMap version 43 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9fcfbb740 10 client.27487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9fcfbb740 20 client.27487 populate_metadata read hostname 'smithi003' 2024-02-09T22:58:57.158 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab62930450 0x55ab62932870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:58:57.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9fcfbb740 4 client.27487 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:58:57.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ab625771b0 con 0x55ab62930450 2024-02-09T22:58:57.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab62930450 0x55ab62932870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:58:57.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.149+0000 7ff9d37fe640 10 client.27487 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:57.159 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.150+0000 7ff9f37fe640 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab62930450 0x55ab62932870 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eee65dd3709e20d3 server_cookie=4f9c4cdffe863102 in_seq=0 out_seq=0 2024-02-09T22:58:57.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9d37fe640 1 -- 172.21.15.3:0/1882503869 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ab625771b0 con 0x55ab62930450 2024-02-09T22:58:57.178 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9d37fe640 10 client.27487 handle_command_reply: tid=0 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 client.27487 shutdown 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 msgr2=0x55ab628cf860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 0x55ab628cf860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ff9e4005890 tx=0x7ff9e40093b0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 shutdown_connections 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ab62930450 0x55ab62932870 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ab628c87a0 0x55ab628ce140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9d37fe640 0 client.27487 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ab628bf9d0 0x55ab628cdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 --2- 172.21.15.3:0/1882503869 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ab628bf0b0 0x55ab628cf860 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 >> 172.21.15.3:0/1882503869 conn(0x55ab628b99d0 msgr2=0x55ab628b81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 shutdown_connections 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 1 -- 172.21.15.3:0/1882503869 wait complete. 2024-02-09T22:58:57.179 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.174+0000 7ff9fcfbb740 20 client.27487 trim_cache size 0 max 16384 2024-02-09T22:58:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:58:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:58:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:58:57.180 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9dc0a2220 msgr2=0x7ff9dc0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9dc0a2220 0x7ff9dc0a46e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7ff9ec005cc0 tx=0x7ff9ec032ef0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 msgr2=0x7ff9f419db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f419db20 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7ff9e800d770 tx=0x7ff9e800dc30 comp rx=0 tx=0).stop 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 shutdown_connections 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff9dc0a2220 0x7ff9dc0a46e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff9f4197730 0x7ff9f4197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9f4105410 0x7ff9f419db20 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.181 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 --2- 172.21.15.3:0/2791370111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff9f4100340 0x7ff9f419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:58:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 >> 172.21.15.3:0/2791370111 conn(0x7ff9f4077220 msgr2=0x7ff9f4068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:58:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 shutdown_connections 2024-02-09T22:58:57.182 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:58:57.175+0000 7ff9fa30d640 1 -- 172.21.15.3:0/2791370111 wait complete. 2024-02-09T22:58:57.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:57 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:57 smithi003 ceph-mon[32678]: pgmap v237: 97 pgs: 97 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 142 op/s 2024-02-09T22:58:57.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 216731 2.56 MB/sec execute 291 sec latency 3167.990 ms 2024-02-09T22:58:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:57 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:57 smithi064 ceph-mon[36638]: pgmap v237: 97 pgs: 97 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 142 op/s 2024-02-09T22:58:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:57 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:57 smithi089 ceph-mon[36665]: pgmap v237: 97 pgs: 97 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 142 op/s 2024-02-09T22:58:58.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:58:58 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:58.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:58:58 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:58.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 217213 2.55 MB/sec execute 292 sec latency 525.443 ms 2024-02-09T22:58:58.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:58:58 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:58:59.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 217213 2.55 MB/sec execute 293 sec latency 1525.558 ms 2024-02-09T22:59:00.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:00 smithi003 ceph-mon[32678]: pgmap v238: 97 pgs: 97 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 620 KiB/s wr, 49 op/s 2024-02-09T22:59:00.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 217279 2.54 MB/sec execute 294 sec latency 2495.664 ms 2024-02-09T22:59:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:00 smithi064 ceph-mon[36638]: pgmap v238: 97 pgs: 97 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 620 KiB/s wr, 49 op/s 2024-02-09T22:59:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:00 smithi089 ceph-mon[36665]: pgmap v238: 97 pgs: 97 active+clean; 84 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 416 KiB/s rd, 620 KiB/s wr, 49 op/s 2024-02-09T22:59:01.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 218114 2.54 MB/sec execute 295 sec latency 250.032 ms 2024-02-09T22:59:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:02 smithi003 ceph-mon[32678]: pgmap v239: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 860 KiB/s wr, 91 op/s 2024-02-09T22:59:02.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 220252 2.58 MB/sec execute 296 sec latency 281.253 ms 2024-02-09T22:59:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:02 smithi064 ceph-mon[36638]: pgmap v239: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 860 KiB/s wr, 91 op/s 2024-02-09T22:59:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:02 smithi089 ceph-mon[36665]: pgmap v239: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 860 KiB/s wr, 91 op/s 2024-02-09T22:59:03.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:03 smithi003 ceph-mon[32678]: osdmap e93: 12 total, 12 up, 12 in 2024-02-09T22:59:03.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 221559 2.59 MB/sec execute 297 sec latency 22.476 ms 2024-02-09T22:59:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:03 smithi064 ceph-mon[36638]: osdmap e93: 12 total, 12 up, 12 in 2024-02-09T22:59:03.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:03 smithi089 ceph-mon[36665]: osdmap e93: 12 total, 12 up, 12 in 2024-02-09T22:59:04.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:04 smithi003 ceph-mon[32678]: pgmap v241: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 682 KiB/s wr, 85 op/s 2024-02-09T22:59:04.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 221657 2.58 MB/sec execute 298 sec latency 914.876 ms 2024-02-09T22:59:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:04 smithi064 ceph-mon[36638]: pgmap v241: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 682 KiB/s wr, 85 op/s 2024-02-09T22:59:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:04 smithi089 ceph-mon[36665]: pgmap v241: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 682 KiB/s wr, 85 op/s 2024-02-09T22:59:05.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 222975 2.60 MB/sec execute 299 sec latency 1361.959 ms 2024-02-09T22:59:06.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:06 smithi003 ceph-mon[32678]: pgmap v242: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 682 KiB/s wr, 85 op/s 2024-02-09T22:59:06.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 224180 2.61 MB/sec execute 300 sec latency 210.745 ms 2024-02-09T22:59:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:06 smithi064 ceph-mon[36638]: pgmap v242: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 682 KiB/s wr, 85 op/s 2024-02-09T22:59:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:06 smithi089 ceph-mon[36665]: pgmap v242: 97 pgs: 97 active+clean; 86 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 682 KiB/s wr, 85 op/s 2024-02-09T22:59:07.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 224180 2.60 MB/sec execute 301 sec latency 1210.829 ms 2024-02-09T22:59:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:08 smithi003 ceph-mon[32678]: pgmap v243: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 168 op/s 2024-02-09T22:59:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:08 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:59:08.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 225205 2.61 MB/sec execute 302 sec latency 1345.178 ms 2024-02-09T22:59:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:08 smithi064 ceph-mon[36638]: pgmap v243: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 168 op/s 2024-02-09T22:59:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:08 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:59:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:08 smithi089 ceph-mon[36665]: pgmap v243: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 168 op/s 2024-02-09T22:59:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:08 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:59:09.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 225205 2.60 MB/sec execute 303 sec latency 1320.034 ms 2024-02-09T22:59:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 -- 172.21.15.3:0/4143365300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100cf0 msgr2=0x7f5df8104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/4143365300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100cf0 0x7f5df8104d30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5de8009900 tx=0x7f5de802bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 -- 172.21.15.3:0/4143365300 shutdown_connections 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/4143365300 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/4143365300 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100cf0 0x7f5df8104d30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/4143365300 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8100340 0x7f5df8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 -- 172.21.15.3:0/4143365300 >> 172.21.15.3:0/4143365300 conn(0x7f5df8077220 msgr2=0x7f5df8077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 -- 172.21.15.3:0/4143365300 shutdown_connections 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.277+0000 7f5dfffe9640 1 -- 172.21.15.3:0/4143365300 wait complete. 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 Processor -- start 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 -- start start 2024-02-09T22:59:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100340 0x7f5df8101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df8102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8102850 0x7f5df81a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81a6070 con 0x7f5df8105410 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81a61e0 con 0x7f5df8102850 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfffe9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5df81a63a0 con 0x7f5df8100340 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df8102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df8102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55342/0 (socket says 172.21.15.3:55342) 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfe586640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8102850 0x7f5df81a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 -- 172.21.15.3:0/930672020 learned_addr learned my addr 172.21.15.3:0/930672020 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100340 0x7f5df8101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 -- 172.21.15.3:0/930672020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100340 msgr2=0x7f5df8101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100340 0x7f5df8101dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 -- 172.21.15.3:0/930672020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8102850 msgr2=0x7f5df81a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8102850 0x7f5df81a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfd584640 1 -- 172.21.15.3:0/930672020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de8009560 con 0x7f5df8105410 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfe586640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8102850 0x7f5df81a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.278+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100340 0x7f5df8101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.279+0000 7f5dfd584640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df8102310 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5de8036040 tx=0x7f5de802c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.279+0000 7f5deeffd640 1 -- 172.21.15.3:0/930672020 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8004350 con 0x7f5df8105410 2024-02-09T22:59:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.279+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5df81a6620 con 0x7f5df8105410 2024-02-09T22:59:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.279+0000 7f5deeffd640 1 -- 172.21.15.3:0/930672020 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5de80044b0 con 0x7f5df8105410 2024-02-09T22:59:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.279+0000 7f5deeffd640 1 -- 172.21.15.3:0/930672020 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de8003da0 con 0x7f5df8105410 2024-02-09T22:59:10.287 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.279+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df81a6b30 con 0x7f5df8105410 2024-02-09T22:59:10.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.281+0000 7f5deeffd640 1 -- 172.21.15.3:0/930672020 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5de8004df0 con 0x7f5df8105410 2024-02-09T22:59:10.294 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 Processor -- start 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 -- start start 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a41d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a426260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a4267a0 0x55c59a428b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a38ba00 con 0x55c59a41d9d0 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a388f30 con 0x55c59a4267a0 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a384710 con 0x55c59a41d0b0 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5dfdd85640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a426260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5dfe586640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a41d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a4267a0 0x55c59a428b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5dfdd85640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a426260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55354/0 (socket says 172.21.15.3:55354) 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5dfdd85640 1 -- 172.21.15.3:0/137159408 learned_addr learned my addr 172.21.15.3:0/137159408 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.282+0000 7f5deeffd640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5de40a2250 0x7f5de40a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfe586640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5de40a2250 0x7f5de40a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfdd85640 1 -- 172.21.15.3:0/137159408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 msgr2=0x55c59a41d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a41d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfdd85640 1 -- 172.21.15.3:0/137159408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a4267a0 msgr2=0x55c59a428b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a4267a0 0x55c59a428b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfdd85640 1 -- 172.21.15.3:0/137159408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c59a3493b0 con 0x55c59a41d9d0 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfd584640 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a4267a0 0x55c59a428b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5deeffd640 1 -- 172.21.15.3:0/930672020 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5de80f0000 con 0x7f5df8105410 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a426260 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5df40115f0 tx=0x7f5df4011ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a47f2caea28aed server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:10.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5dfe586640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5de40a2250 0x7f5de40a4710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5df00094d0 tx=0x7f5df0004ab0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5de3fff640 1 -- 172.21.15.3:0/137159408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df400f640 con 0x55c59a41d9d0 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5de3fff640 1 -- 172.21.15.3:0/137159408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5df400fc80 con 0x55c59a41d9d0 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.283+0000 7f5de3fff640 1 -- 172.21.15.3:0/137159408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df4019d20 con 0x55c59a41d9d0 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 -- 172.21.15.3:0/137159408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 msgr2=0x55c59a426260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a426260 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f5df40115f0 tx=0x7f5df4011ab0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 -- 172.21.15.3:0/137159408 shutdown_connections 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a4267a0 0x55c59a428b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a426260 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/137159408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a41d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 -- 172.21.15.3:0/137159408 >> 172.21.15.3:0/137159408 conn(0x55c59a4179d0 msgr2=0x55c59a417db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 -- 172.21.15.3:0/137159408 shutdown_connections 2024-02-09T22:59:10.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 -- 172.21.15.3:0/137159408 wait complete. 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.284+0000 7f5e02bc2740 1 -- start start 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 Processor -- start 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a4237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a423d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a424250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a329b90 con 0x55c59a4267a0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a058250 con 0x55c59a41d9d0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a30a200 con 0x55c59a41d0b0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5e02bc2740 10 client.0 _ll_register_callbacks cb 0x55c59a3405d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a424250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfdd85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a4237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a423d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a424250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55362/0 (socket says 172.21.15.3:55362) 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 -- 172.21.15.3:0/1923916447 learned_addr learned my addr 172.21.15.3:0/1923916447 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 -- 172.21.15.3:0/1923916447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 msgr2=0x55c59a4237d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a4237d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 -- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 msgr2=0x55c59a423d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a423d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfe586640 1 -- 172.21.15.3:0/1923916447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df4009d40 con 0x55c59a4267a0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.285+0000 7f5dfd584640 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a423d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5dfe586640 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a424250 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f5df000ffa0 tx=0x7f5df0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8c1e86e0214b80 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5de17fa640 1 -- 172.21.15.3:0/1923916447 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0022480 con 0x55c59a4267a0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5de17fa640 1 -- 172.21.15.3:0/1923916447 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5df00225e0 con 0x55c59a4267a0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c59a3aaee0 con 0x55c59a4267a0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5de17fa640 1 -- 172.21.15.3:0/1923916447 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5df00227a0 con 0x55c59a4267a0 2024-02-09T22:59:10.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 10 client.18342 fetch_fsmap learned FSMap version 45 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x55c59a382d90 con 0x55c59a4267a0 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5de17fa640 1 -- 172.21.15.3:0/1923916447 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c59a3aaee0 con 0x55c59a4267a0 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5de17fa640 1 -- 172.21.15.3:0/1923916447 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 45) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f5df0010ae0 con 0x55c59a4267a0 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 10 client.18342 fetch_fsmap finished waiting for FSMap version 45 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 10 client.18342 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 20 client.18342 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a436a90 0x55c59a438eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 4 client.18342 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.286+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c59a084600 con 0x55c59a436a90 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.287+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a436a90 0x55c59a438eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.287+0000 7f5de17fa640 10 client.18342 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.287+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a436a90 0x55c59a438eb0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b118b830d792a0 server_cookie=30da7a316baf5c38 in_seq=0 out_seq=0 2024-02-09T22:59:10.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5de17fa640 1 -- 172.21.15.3:0/1923916447 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c59a084600 con 0x55c59a436a90 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5de17fa640 10 client.18342 handle_command_reply: tid=0 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 client.18342 shutdown 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 msgr2=0x55c59a424250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a424250 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f5df000ffa0 tx=0x7f5df0004a40 comp rx=0 tx=0).stop 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 shutdown_connections 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a436a90 0x55c59a438eb0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a423d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a424250 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1923916447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a4237d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5de17fa640 0 client.18342 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 >> 172.21.15.3:0/1923916447 conn(0x55c59a4179d0 msgr2=0x55c59a419720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 shutdown_connections 2024-02-09T22:59:10.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1923916447 wait complete. 2024-02-09T22:59:10.302 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.288+0000 7f5e02bc2740 20 client.18342 trim_cache size 0 max 16384 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 Processor -- start 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 -- start start 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a425f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a424070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a4220a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a417270 con 0x55c59a4267a0 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a421d20 con 0x55c59a41d9d0 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a41d7c0 con 0x55c59a41d0b0 2024-02-09T22:59:10.312 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a4220a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfdd85640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a425f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a424070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a4220a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55376/0 (socket says 172.21.15.3:55376) 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 -- 172.21.15.3:0/597272408 learned_addr learned my addr 172.21.15.3:0/597272408 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 -- 172.21.15.3:0/597272408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 msgr2=0x55c59a425f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a425f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 -- 172.21.15.3:0/597272408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 msgr2=0x55c59a424070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a424070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfe586640 1 -- 172.21.15.3:0/597272408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c59a31f820 con 0x55c59a4267a0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.303+0000 7f5dfd584640 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a424070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.304+0000 7f5dfe586640 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a4220a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f5df0000f80 tx=0x7f5df0013610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=568205090a2a6eef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.304+0000 7f5de37fe640 1 -- 172.21.15.3:0/597272408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0013b60 con 0x55c59a4267a0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.304+0000 7f5de37fe640 1 -- 172.21.15.3:0/597272408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5df0013e40 con 0x55c59a4267a0 2024-02-09T22:59:10.313 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.304+0000 7f5de37fe640 1 -- 172.21.15.3:0/597272408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5df0010630 con 0x55c59a4267a0 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 -- 172.21.15.3:0/597272408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 msgr2=0x55c59a4220a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a4220a0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f5df0000f80 tx=0x7f5df0013610 comp rx=0 tx=0).stop 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 -- 172.21.15.3:0/597272408 shutdown_connections 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d9d0 0x55c59a424070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a4267a0 0x55c59a4220a0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/597272408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a41d0b0 0x55c59a425f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 -- 172.21.15.3:0/597272408 >> 172.21.15.3:0/597272408 conn(0x55c59a4179d0 msgr2=0x55c59a41eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 -- 172.21.15.3:0/597272408 shutdown_connections 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 -- 172.21.15.3:0/597272408 wait complete. 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 -- start start 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.305+0000 7f5e02bc2740 1 Processor -- start 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d0b0 0x55c59a42d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a42d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.314 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a4267a0 0x55c59a42ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a42c0d0 con 0x55c59a41d9d0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a42c240 con 0x55c59a41d0b0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59a42c400 con 0x55c59a4267a0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 10 client.0 _ll_register_callbacks cb 0x7f5df00136e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a42d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a42d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55382/0 (socket says 172.21.15.3:55382) 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 -- 172.21.15.3:0/1040660793 learned_addr learned my addr 172.21.15.3:0/1040660793 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d0b0 0x55c59a42d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfe586640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a4267a0 0x55c59a42ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 -- 172.21.15.3:0/1040660793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a4267a0 msgr2=0x55c59a42ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a4267a0 0x55c59a42ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 -- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d0b0 msgr2=0x55c59a42d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d0b0 0x55c59a42d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.315 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 -- 172.21.15.3:0/1040660793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5df81a6620 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d0b0 0x55c59a42d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5dfd584640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a42d9a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f5de8005ec0 tx=0x7f5de80b8ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16ec80efa684487 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5de17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5de17fa640 1 -- 172.21.15.3:0/1040660793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5de80edea0 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5de17fa640 1 -- 172.21.15.3:0/1040660793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5de80bd200 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5df400fc80 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5de17fa640 1 -- 172.21.15.3:0/1040660793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5de80bc460 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 10 client.18348 fetch_fsmap learned FSMap version 45 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.306+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x55c59a382d90 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5de17fa640 1 -- 172.21.15.3:0/1040660793 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5df400fc80 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5de17fa640 1 -- 172.21.15.3:0/1040660793 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 45) v1 ==== 7886+0+0 (secure 0 0 0) 0x7f5de803c070 con 0x55c59a41d9d0 2024-02-09T22:59:10.316 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5e02bc2740 10 client.18348 fetch_fsmap finished waiting for FSMap version 45 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5e02bc2740 10 client.18348 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5e02bc2740 20 client.18348 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a48e1b0 0x55c59a4905a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5e02bc2740 4 client.18348 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c59a0d51b0 con 0x55c59a48e1b0 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a48e1b0 0x55c59a4905a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5de17fa640 10 client.18348 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:10.317 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.307+0000 7f5dfdd85640 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a48e1b0 0x55c59a4905a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bae5592b62f538a server_cookie=624ef8c4d4222362 in_seq=0 out_seq=0 2024-02-09T22:59:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.338+0000 7f5de17fa640 1 -- 172.21.15.3:0/1040660793 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c59a0d51b0 con 0x55c59a48e1b0 2024-02-09T22:59:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.338+0000 7f5de17fa640 10 client.18348 handle_command_reply: tid=0 2024-02-09T22:59:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.338+0000 7f5e02bc2740 1 client.18348 shutdown 2024-02-09T22:59:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 msgr2=0x55c59a42d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a42d9a0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f5de8005ec0 tx=0x7f5de80b8ed0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.343 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 shutdown_connections 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c59a48e1b0 0x55c59a4905a0 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5de17fa640 0 client.18348 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c59a4267a0 0x55c59a42ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c59a41d9d0 0x55c59a42d9a0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 --2- 172.21.15.3:0/1040660793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c59a41d0b0 0x55c59a42d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 >> 172.21.15.3:0/1040660793 conn(0x55c59a4179d0 msgr2=0x55c59a422000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 shutdown_connections 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 1 -- 172.21.15.3:0/1040660793 wait complete. 2024-02-09T22:59:10.344 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.339+0000 7f5e02bc2740 20 client.18348 trim_cache size 0 max 16384 2024-02-09T22:59:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:59:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:59:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:59:10.345 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5de40a2250 msgr2=0x7f5de40a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5de40a2250 0x7f5de40a4710 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f5df00094d0 tx=0x7f5df0004ab0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 msgr2=0x7f5df8102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df8102310 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f5de8036040 tx=0x7f5de802c680 comp rx=0 tx=0).stop 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 shutdown_connections 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5de40a2250 0x7f5de40a4710 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5df8102850 0x7f5df81a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5df8105410 0x7f5df8102310 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 --2- 172.21.15.3:0/930672020 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5df8100340 0x7f5df8101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 >> 172.21.15.3:0/930672020 conn(0x7f5df8077220 msgr2=0x7f5df8068d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 shutdown_connections 2024-02-09T22:59:10.346 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:10.340+0000 7f5dfffe9640 1 -- 172.21.15.3:0/930672020 wait complete. 2024-02-09T22:59:10.589 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:10 smithi064 ceph-mon[36638]: pgmap v244: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 168 op/s 2024-02-09T22:59:10.589 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:10 smithi064 ceph-mon[36638]: osdmap e94: 12 total, 12 up, 12 in 2024-02-09T22:59:10.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:10 smithi003 ceph-mon[32678]: pgmap v244: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 168 op/s 2024-02-09T22:59:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:10 smithi003 ceph-mon[32678]: osdmap e94: 12 total, 12 up, 12 in 2024-02-09T22:59:10.650 INFO:tasks.workunit.client.0.smithi003.stdout: 1 226523 2.62 MB/sec execute 304 sec latency 1766.644 ms 2024-02-09T22:59:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:10 smithi089 ceph-mon[36665]: pgmap v244: 97 pgs: 97 active+clean; 90 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 980 KiB/s wr, 168 op/s 2024-02-09T22:59:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:10 smithi089 ceph-mon[36665]: osdmap e94: 12 total, 12 up, 12 in 2024-02-09T22:59:10.935 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:59:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setuser ceph since I am not root 2024-02-09T22:59:10.936 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:59:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setgroup ceph since I am not root 2024-02-09T22:59:10.936 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:59:10 smithi064 ceph-mds[66607]: -- 172.21.15.64:0/2078677027 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b6327321a0 con 0x55b632709000 2024-02-09T22:59:10.936 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 22:59:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: starting mds.e at 2024-02-09T22:59:11.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:11 smithi003 ceph-mon[32678]: daemon mds.e restarted 2024-02-09T22:59:11.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 227722 2.63 MB/sec execute 305 sec latency 104.184 ms 2024-02-09T22:59:11.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:11 smithi064 ceph-mon[36638]: daemon mds.e restarted 2024-02-09T22:59:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:11 smithi089 ceph-mon[36665]: daemon mds.e restarted 2024-02-09T22:59:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:12 smithi003 ceph-mon[32678]: pgmap v246: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 235 op/s 2024-02-09T22:59:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:12 smithi003 ceph-mon[32678]: osdmap e95: 12 total, 12 up, 12 in 2024-02-09T22:59:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:12 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6834/3325656265,v1:172.21.15.64:6835/3325656265] up:boot 2024-02-09T22:59:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:12 smithi003 ceph-mon[32678]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T22:59:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:12 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:59:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:12 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:59:12.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 227722 2.62 MB/sec execute 306 sec latency 1104.274 ms 2024-02-09T22:59:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:12 smithi064 ceph-mon[36638]: pgmap v246: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 235 op/s 2024-02-09T22:59:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:12 smithi064 ceph-mon[36638]: osdmap e95: 12 total, 12 up, 12 in 2024-02-09T22:59:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:12 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6834/3325656265,v1:172.21.15.64:6835/3325656265] up:boot 2024-02-09T22:59:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:12 smithi064 ceph-mon[36638]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T22:59:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:12 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:59:12.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:12 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:59:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:12 smithi089 ceph-mon[36665]: pgmap v246: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 235 op/s 2024-02-09T22:59:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:12 smithi089 ceph-mon[36665]: osdmap e95: 12 total, 12 up, 12 in 2024-02-09T22:59:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:12 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6834/3325656265,v1:172.21.15.64:6835/3325656265] up:boot 2024-02-09T22:59:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:12 smithi089 ceph-mon[36665]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T22:59:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:12 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T22:59:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:12 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T22:59:13.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:59:13.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:59:13.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 -- 172.21.15.3:0/1564066556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 msgr2=0x7fc1e80ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 --2- 172.21.15.3:0/1564066556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e80ffb20 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009930 tx=0x7fc1d802bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:13.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 -- 172.21.15.3:0/1564066556 shutdown_connections 2024-02-09T22:59:13.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 --2- 172.21.15.3:0/1564066556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 0x7fc1e810e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 --2- 172.21.15.3:0/1564066556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e80ffb20 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 --2- 172.21.15.3:0/1564066556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8106200 0x7fc1e81065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.121+0000 7fc1f0240640 1 -- 172.21.15.3:0/1564066556 >> 172.21.15.3:0/1564066556 conn(0x7fc1e8076630 msgr2=0x7fc1e80fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 -- 172.21.15.3:0/1564066556 shutdown_connections 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 -- 172.21.15.3:0/1564066556 wait complete. 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 Processor -- start 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 -- start start 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e819f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 0x7fc1e819fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8106200 0x7fc1e81999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e819a010 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e819a180 con 0x7fc1e8100060 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.122+0000 7fc1f0240640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e819a340 con 0x7fc1e8106200 2024-02-09T22:59:13.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e819f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1ed7db640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 0x7fc1e819fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1ee7dd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8106200 0x7fc1e81999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e819f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55390/0 (socket says 172.21.15.3:55390) 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1ed7db640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 0x7fc1e819fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:34226/0 (socket says 172.21.15.3:34226) 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 -- 172.21.15.3:0/3632723554 learned_addr learned my addr 172.21.15.3:0/3632723554 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 -- 172.21.15.3:0/3632723554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8106200 msgr2=0x7fc1e81999c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8106200 0x7fc1e81999c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 -- 172.21.15.3:0/3632723554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 msgr2=0x7fc1e819fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 0x7fc1e819fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 -- 172.21.15.3:0/3632723554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1d8009560 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e819f810 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400e9f0 tx=0x7fc1e400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1deffd640 1 -- 172.21.15.3:0/3632723554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4009720 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e819a620 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1deffd640 1 -- 172.21.15.3:0/3632723554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e400cd80 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1deffd640 1 -- 172.21.15.3:0/3632723554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4010690 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.129 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.123+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e81a68b0 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.126+0000 7fc1deffd640 1 -- 172.21.15.3:0/3632723554 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc1e40046c0 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.126+0000 7fc1f2e14740 1 Processor -- start 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 -- start start 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce45d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce51300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cce00d00 con 0x5644cce45980 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ccdb52f0 con 0x5644cce46330 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ccdb2820 con 0x5644cce4eeb0 2024-02-09T22:59:13.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ee7dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce45d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ee7dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce45d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55404/0 (socket says 172.21.15.3:55404) 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ee7dd640 1 -- 172.21.15.3:0/2229702756 learned_addr learned my addr 172.21.15.3:0/2229702756 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1deffd640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1d40a2230 0x7fc1d40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce51300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1d40a2230 0x7fc1d40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1deffd640 1 -- 172.21.15.3:0/3632723554 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc1e4014070 con 0x7fc1e80ff6a0 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ed7db640 1 -- 172.21.15.3:0/2229702756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 msgr2=0x5644cce4e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ed7db640 1 -- 172.21.15.3:0/2229702756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 msgr2=0x5644cce45d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce45d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ed7db640 1 -- 172.21.15.3:0/2229702756 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644ccd6fff0 con 0x5644cce4eeb0 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce45d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:13.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.127+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.128+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce51300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009230 tx=0x7fc1d802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76e5f3a68a2863d5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.128+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1d40a2230 0x7fc1d40a46f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009fd0 tx=0x7fc1e0004ab0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.128+0000 7fc1d3fff640 1 -- 172.21.15.3:0/2229702756 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d802c8d0 con 0x5644cce4eeb0 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.128+0000 7fc1d3fff640 1 -- 172.21.15.3:0/2229702756 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1d802ca30 con 0x5644cce4eeb0 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.128+0000 7fc1d3fff640 1 -- 172.21.15.3:0/2229702756 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1d8005050 con 0x5644cce4eeb0 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2229702756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 msgr2=0x5644cce51300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce51300 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc1d8009230 tx=0x7fc1d802c5d0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2229702756 shutdown_connections 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce51300 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2229702756 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce45d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2229702756 >> 172.21.15.3:0/2229702756 conn(0x5644cce40050 msgr2=0x5644cce40430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2229702756 shutdown_connections 2024-02-09T22:59:13.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2229702756 wait complete. 2024-02-09T22:59:13.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 -- start start 2024-02-09T22:59:13.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.129+0000 7fc1f2e14740 1 Processor -- start 2024-02-09T22:59:13.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce4bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce4c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ccdae000 con 0x5644cce45980 2024-02-09T22:59:13.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cca81430 con 0x5644cce46330 2024-02-09T22:59:13.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cca3de00 con 0x5644cce4eeb0 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 10 client.0 _ll_register_callbacks cb 0x5644ccda27f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce4bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce4bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55410/0 (socket says 172.21.15.3:55410) 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 -- 172.21.15.3:0/26744177 learned_addr learned my addr 172.21.15.3:0/26744177 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce4c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 -- 172.21.15.3:0/26744177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 msgr2=0x5644cce4c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce4c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 -- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 msgr2=0x5644cce4c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 -- 172.21.15.3:0/26744177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e810e9e0 con 0x5644cce45980 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce4c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce4bd60 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc1e409ad30 tx=0x7fc1e401da50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555cbb8620c524e4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1d17fa640 1 -- 172.21.15.3:0/26744177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e409e070 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1d17fa640 1 -- 172.21.15.3:0/26744177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc1e409ada0 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.130+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5644ccaad7e0 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1d17fa640 1 -- 172.21.15.3:0/26744177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1e400ace0 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 10 client.18366 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5644ccdac680 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1d17fa640 1 -- 172.21.15.3:0/26744177 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4064020 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1d17fa640 1 -- 172.21.15.3:0/26744177 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc1e40ce070 con 0x5644cce45980 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 10 client.18366 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 10 client.18366 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 20 client.18366 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cce5f260 0x5644cce61680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 4 client.18366 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5644ccaea720 con 0x5644cce5f260 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.131+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cce5f260 0x5644cce61680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cce5f260 0x5644cce61680 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=122f155b598c608 server_cookie=a49fe0e86650711d in_seq=0 out_seq=0 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1d17fa640 10 client.18366 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1d17fa640 1 -- 172.21.15.3:0/26744177 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5644ccaea720 con 0x5644cce5f260 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1d17fa640 10 client.18366 handle_command_reply: tid=0 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 client.18366 shutdown 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 msgr2=0x5644cce4bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce4bd60 secure :-1 s=READY pgs=1274 cs=0 l=1 rev1=1 crypto rx=0x7fc1e409ad30 tx=0x7fc1e401da50 comp rx=0 tx=0).stop 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 shutdown_connections 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cce5f260 0x5644cce61680 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce4c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce4eeb0 0x5644cce4c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/26744177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce4bd60 unknown :-1 s=CLOSED pgs=1274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1d17fa640 0 client.18366 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:13.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 >> 172.21.15.3:0/26744177 conn(0x5644cce40050 msgr2=0x5644cce41da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 shutdown_connections 2024-02-09T22:59:13.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.132+0000 7fc1f2e14740 1 -- 172.21.15.3:0/26744177 wait complete. 2024-02-09T22:59:13.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.133+0000 7fc1f2e14740 20 client.18366 trim_cache size 0 max 16384 2024-02-09T22:59:13.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 Processor -- start 2024-02-09T22:59:13.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 -- start start 2024-02-09T22:59:13.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce47020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce46330 0x5644cce47560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce4eeb0 0x5644cce4a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644ccd32fd0 con 0x5644cce45980 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cce5bde0 con 0x5644cce4eeb0 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cce46b70 con 0x5644cce46330 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce47020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce47020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55422/0 (socket says 172.21.15.3:55422) 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 -- 172.21.15.3:0/4129337855 learned_addr learned my addr 172.21.15.3:0/4129337855 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce4eeb0 0x5644cce4a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 -- 172.21.15.3:0/4129337855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce46330 msgr2=0x5644cce47560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce46330 0x5644cce47560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce46330 0x5644cce47560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 -- 172.21.15.3:0/4129337855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce4eeb0 msgr2=0x5644cce4a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce4eeb0 0x5644cce4a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1edfdc640 1 -- 172.21.15.3:0/4129337855 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5644ccd48b30 con 0x5644cce45980 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce4eeb0 0x5644cce4a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.148+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce46330 0x5644cce47560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce47020 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4097200 tx=0x7fc1e4005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40052d83a7cabac7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1d37fe640 1 -- 172.21.15.3:0/4129337855 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e409e070 con 0x5644cce45980 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1d37fe640 1 -- 172.21.15.3:0/4129337855 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1e4064030 con 0x5644cce45980 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1d37fe640 1 -- 172.21.15.3:0/4129337855 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e409ada0 con 0x5644cce45980 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 -- 172.21.15.3:0/4129337855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 msgr2=0x5644cce47020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce47020 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4097200 tx=0x7fc1e4005720 comp rx=0 tx=0).stop 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 -- 172.21.15.3:0/4129337855 shutdown_connections 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce4eeb0 0x5644cce4a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce46330 0x5644cce47560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/4129337855 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce45980 0x5644cce47020 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 -- 172.21.15.3:0/4129337855 >> 172.21.15.3:0/4129337855 conn(0x5644cce40050 msgr2=0x5644cce40430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 -- 172.21.15.3:0/4129337855 shutdown_connections 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.149+0000 7fc1f2e14740 1 -- 172.21.15.3:0/4129337855 wait complete. 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 -- start start 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 Processor -- start 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce45980 0x5644cce56060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce54400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 0x5644cce54940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cce54e80 con 0x5644cce4eeb0 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cce54ff0 con 0x5644cce46330 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5644cce55160 con 0x5644cce45980 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1f2e14740 10 client.0 _ll_register_callbacks cb 0x7fc1e4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1ee7dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 0x5644cce54940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1ee7dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 0x5644cce54940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55424/0 (socket says 172.21.15.3:55424) 2024-02-09T22:59:13.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1ee7dd640 1 -- 172.21.15.3:0/2244856070 learned_addr learned my addr 172.21.15.3:0/2244856070 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce45980 0x5644cce56060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.150+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce54400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ee7dd640 1 -- 172.21.15.3:0/2244856070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce45980 msgr2=0x5644cce56060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce45980 0x5644cce56060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ee7dd640 1 -- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 msgr2=0x5644cce54400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce54400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ee7dd640 1 -- 172.21.15.3:0/2244856070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5644ccdac680 con 0x5644cce4eeb0 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ed7db640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce54400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce45980 0x5644cce56060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1ee7dd640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 0x5644cce54940 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0004760 tx=0x7fc1e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37a9280e2665bc16 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1d17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1d17fa640 1 -- 172.21.15.3:0/2244856070 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1e0027070 con 0x5644cce4eeb0 2024-02-09T22:59:13.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1d17fa640 1 -- 172.21.15.3:0/2244856070 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fc1e002c470 con 0x5644cce4eeb0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc1d802ca30 con 0x5644cce4eeb0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1d17fa640 1 -- 172.21.15.3:0/2244856070 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc1e002c630 con 0x5644cce4eeb0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 10 client.18372 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5644ccd6e850 con 0x5644cce4eeb0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1d17fa640 1 -- 172.21.15.3:0/2244856070 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc1d802ca30 con 0x5644cce4eeb0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1d17fa640 1 -- 172.21.15.3:0/2244856070 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fc1e00227e0 con 0x5644cce4eeb0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 10 client.18372 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 10 client.18372 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 20 client.18372 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.151+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cceb6d60 0x5644cceb9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1f2e14740 4 client.18372 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5644ccb39890 con 0x5644cceb6d60 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cceb6d60 0x5644cceb9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1d17fa640 10 client.18372 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1edfdc640 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cceb6d60 0x5644cceb9180 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25f4ceeb3ea6bb29 server_cookie=7bbc5b9ba7551643 in_seq=0 out_seq=0 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1d17fa640 1 -- 172.21.15.3:0/2244856070 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5644ccb39890 con 0x5644cceb6d60 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1d17fa640 10 client.18372 handle_command_reply: tid=0 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1f2e14740 1 client.18372 shutdown 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 msgr2=0x5644cce54940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.152+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 0x5644cce54940 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0004760 tx=0x7fc1e0004a40 comp rx=0 tx=0).stop 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 shutdown_connections 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5644cceb6d60 0x5644cceb9180 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5644cce46330 0x5644cce54400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5644cce4eeb0 0x5644cce54940 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1d17fa640 0 client.18372 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 --2- 172.21.15.3:0/2244856070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5644cce45980 0x5644cce56060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 >> 172.21.15.3:0/2244856070 conn(0x5644cce40050 msgr2=0x5644cce47020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 shutdown_connections 2024-02-09T22:59:13.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 1 -- 172.21.15.3:0/2244856070 wait complete. 2024-02-09T22:59:13.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f2e14740 20 client.18372 trim_cache size 0 max 16384 2024-02-09T22:59:13.163 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:59:13.163 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:59:13.163 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:59:13.163 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:59:13.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1d40a2230 msgr2=0x7fc1d40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1d40a2230 0x7fc1d40a46f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc1e0009fd0 tx=0x7fc1e0004ab0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 msgr2=0x7fc1e819f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e819f810 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7fc1e400e9f0 tx=0x7fc1e400edf0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 shutdown_connections 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc1d40a2230 0x7fc1d40a46f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc1e8100060 0x7fc1e819fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc1e8106200 0x7fc1e81999c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 --2- 172.21.15.3:0/3632723554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc1e80ff6a0 0x7fc1e819f810 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 >> 172.21.15.3:0/3632723554 conn(0x7fc1e8076630 msgr2=0x7fc1e8104340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 shutdown_connections 2024-02-09T22:59:13.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.153+0000 7fc1f0240640 1 -- 172.21.15.3:0/3632723554 wait complete. 2024-02-09T22:59:13.167 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:59:13.167 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 -- 172.21.15.3:0/4152317150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 msgr2=0x7f823010e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 --2- 172.21.15.3:0/4152317150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823010e1c0 secure :-1 s=READY pgs=1277 cs=0 l=1 rev1=1 crypto rx=0x7f8228009ca0 tx=0x7f822802c180 comp rx=0 tx=0).stop 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 -- 172.21.15.3:0/4152317150 shutdown_connections 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 --2- 172.21.15.3:0/4152317150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82300ffaa0 0x7f82300fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 --2- 172.21.15.3:0/4152317150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823010e1c0 unknown :-1 s=CLOSED pgs=1277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 --2- 172.21.15.3:0/4152317150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8230106370 0x7f8230106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 -- 172.21.15.3:0/4152317150 >> 172.21.15.3:0/4152317150 conn(0x7f82300765f0 msgr2=0x7f8230076a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 -- 172.21.15.3:0/4152317150 shutdown_connections 2024-02-09T22:59:13.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 -- 172.21.15.3:0/4152317150 wait complete. 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 Processor -- start 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.268+0000 7f823601c640 1 -- start start 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82300ffaa0 0x7f823019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8230106370 0x7f8230197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230197de0 con 0x7f8230100460 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230197f50 con 0x7f8230106370 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8230198110 con 0x7f82300ffaa0 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82300ffaa0 0x7f823019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823019dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55436/0 (socket says 172.21.15.3:55436) 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 -- 172.21.15.3:0/2060061374 learned_addr learned my addr 172.21.15.3:0/2060061374 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822ffff640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8230106370 0x7f8230197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 -- 172.21.15.3:0/2060061374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82300ffaa0 msgr2=0x7f823019d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82300ffaa0 0x7f823019d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 -- 172.21.15.3:0/2060061374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8230106370 msgr2=0x7f8230197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8230106370 0x7f8230197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 -- 172.21.15.3:0/2060061374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8228009900 con 0x7f8230100460 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822effd640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823019dad0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f822400c920 tx=0x7f822400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8224007c70 con 0x7f8230100460 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82301983f0 con 0x7f8230100460 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8224007dd0 con 0x7f8230100460 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.269+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8224018860 con 0x7f8230100460 2024-02-09T22:59:13.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.270+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82301a4510 con 0x7f8230100460 2024-02-09T22:59:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.270+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8230069520 con 0x7f8230100460 2024-02-09T22:59:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.272+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f822400fac0 con 0x7f8230100460 2024-02-09T22:59:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.272+0000 7f822cff9640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82180a2230 0x7f82180a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.272+0000 7f822f7fe640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82180a2230 0x7f82180a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.272+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f82240ce3a0 con 0x7f8230100460 2024-02-09T22:59:13.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.273+0000 7f822f7fe640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82180a2230 0x7f82180a46f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8220009bd0 tx=0x7f8220005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.276+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f82240970f0 con 0x7f8230100460 2024-02-09T22:59:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.505+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8230198bb0 con 0x7f8230100460 2024-02-09T22:59:13.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.506+0000 7f822cff9640 1 -- 172.21.15.3:0/2060061374 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f822409a7b0 con 0x7f8230100460 2024-02-09T22:59:13.509 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:59:13.509 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:59:11.242420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:59:13.509 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 47 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82180a2230 msgr2=0x7f82180a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82180a2230 0x7f82180a46f0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f8220009bd0 tx=0x7f8220005f90 comp rx=0 tx=0).stop 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 msgr2=0x7f823019dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823019dad0 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7f822400c920 tx=0x7f822400cde0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 shutdown_connections 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82180a2230 0x7f82180a46f0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8230106370 0x7f8230197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8230100460 0x7f823019dad0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 --2- 172.21.15.3:0/2060061374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82300ffaa0 0x7f823019d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 >> 172.21.15.3:0/2060061374 conn(0x7f82300765f0 msgr2=0x7f82300fd500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 shutdown_connections 2024-02-09T22:59:13.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.508+0000 7f823601c640 1 -- 172.21.15.3:0/2060061374 wait complete. 2024-02-09T22:59:13.521 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:13.521 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:59:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/902068288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc100440 msgr2=0x7fa9cc100820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/902068288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc100440 0x7fa9cc100820 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7fa9c8009930 tx=0x7fa9c802bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/902068288 shutdown_connections 2024-02-09T22:59:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/902068288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9cc101d20 0x7fa9cc102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/902068288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc10d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/902068288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc100440 0x7fa9cc100820 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/902068288 >> 172.21.15.3:0/902068288 conn(0x7fa9cc077310 msgr2=0x7fa9cc077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/902068288 shutdown_connections 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.622+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/902068288 wait complete. 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 Processor -- start 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 -- start start 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9cc100440 0x7fa9cc19d640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9cc101d20 0x7fa9cc19db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc197ee0 con 0x7fa9cc1026c0 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc198050 con 0x7fa9cc101d20 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d3f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9cc198210 con 0x7fa9cc100440 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc197750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55448/0 (socket says 172.21.15.3:55448) 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 -- 172.21.15.3:0/3999752926 learned_addr learned my addr 172.21.15.3:0/3999752926 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9cc100440 0x7fa9cc19d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9cc101d20 0x7fa9cc19db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 -- 172.21.15.3:0/3999752926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9cc100440 msgr2=0x7fa9cc19d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9cc100440 0x7fa9cc19d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 -- 172.21.15.3:0/3999752926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9cc101d20 msgr2=0x7fa9cc19db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9cc101d20 0x7fa9cc19db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.623+0000 7fa9d24db640 1 -- 172.21.15.3:0/3999752926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c8009560 con 0x7fa9cc1026c0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.624+0000 7fa9d24db640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc197750 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400e960 tx=0x7fa9c400ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.624+0000 7fa9c2ffd640 1 -- 172.21.15.3:0/3999752926 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4004d20 con 0x7fa9cc1026c0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.624+0000 7fa9c2ffd640 1 -- 172.21.15.3:0/3999752926 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9c4004e80 con 0x7fa9cc1026c0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.624+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9cc1984f0 con 0x7fa9cc1026c0 2024-02-09T22:59:13.629 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.624+0000 7fa9c2ffd640 1 -- 172.21.15.3:0/3999752926 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c40116d0 con 0x7fa9cc1026c0 2024-02-09T22:59:13.630 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.624+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9cc1a45c0 con 0x7fa9cc1026c0 2024-02-09T22:59:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.626+0000 7fa9c2ffd640 1 -- 172.21.15.3:0/3999752926 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa9c4005340 con 0x7fa9cc1026c0 2024-02-09T22:59:13.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.626+0000 7fa9d6bed740 1 Processor -- start 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.626+0000 7fa9d6bed740 1 -- start start 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56e910 0x55655f56ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f56f2c0 0x55655f577df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 0x55655f578820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f52a020 con 0x55655f5783c0 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f4dd320 con 0x55655f56f2c0 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f4da850 con 0x55655f56e910 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9c2ffd640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa9b80a2230 0x7fa9b80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 0x55655f578820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 0x55655f578820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55464/0 (socket says 172.21.15.3:55464) 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2810234363 learned_addr learned my addr 172.21.15.3:0/2810234363 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d24db640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa9b80a2230 0x7fa9b80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56e910 0x55655f56ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f56f2c0 0x55655f577df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9c2ffd640 1 -- 172.21.15.3:0/3999752926 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa9c40ce5f0 con 0x7fa9cc1026c0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2810234363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56e910 msgr2=0x55655f56ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56e910 0x55655f56ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2810234363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f56f2c0 msgr2=0x55655f577df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f56f2c0 0x55655f577df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2810234363 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655f498210 con 0x55655f5783c0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 0x55655f578820 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc013860 tx=0x7fa9bc00b400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d59fcae2888ad8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9d24db640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa9b80a2230 0x7fa9b80a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40cbe20 tx=0x7fa9c4085b40 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.627+0000 7fa9b7fff640 1 -- 172.21.15.3:0/2810234363 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc013ea0 con 0x55655f5783c0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9b7fff640 1 -- 172.21.15.3:0/2810234363 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9bc019070 con 0x55655f5783c0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9b7fff640 1 -- 172.21.15.3:0/2810234363 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc00e8f0 con 0x55655f5783c0 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2810234363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 msgr2=0x55655f578820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 0x55655f578820 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc013860 tx=0x7fa9bc00b400 comp rx=0 tx=0).stop 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2810234363 shutdown_connections 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f56f2c0 0x55655f577df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f5783c0 0x55655f578820 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2810234363 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56e910 0x55655f56ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2810234363 >> 172.21.15.3:0/2810234363 conn(0x55655f569100 msgr2=0x55655f5694e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2810234363 shutdown_connections 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.628+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2810234363 wait complete. 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 -- start start 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 Processor -- start 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f57c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f57c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f4d6030 con 0x55655f56e910 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f1a9430 con 0x55655f5783c0 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f165e00 con 0x55655f56f2c0 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d6bed740 10 client.0 _ll_register_callbacks cb 0x55655f4cb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55468/0 (socket says 172.21.15.3:55468) 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d14d9640 1 -- 172.21.15.3:0/447715569 learned_addr learned my addr 172.21.15.3:0/447715569 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.629+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f57c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d14d9640 1 -- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 msgr2=0x55655f57c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f57c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d14d9640 1 -- 172.21.15.3:0/447715569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 msgr2=0x55655f57c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f57c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d14d9640 1 -- 172.21.15.3:0/447715569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9bc013540 con 0x55655f56e910 2024-02-09T22:59:13.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f57c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57be60 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc004650 tx=0x7fa9bc004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a04b9f7f0bd1b5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9b57fa640 1 -- 172.21.15.3:0/447715569 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc023070 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9b57fa640 1 -- 172.21.15.3:0/447715569 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa9bc013050 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55655f1d57e0 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9b57fa640 1 -- 172.21.15.3:0/447715569 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9bc013ea0 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 10 client.18390 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55655f4d46b0 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9b57fa640 1 -- 172.21.15.3:0/447715569 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55655f1d57e0 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9b57fa640 1 -- 172.21.15.3:0/447715569 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa9bc00e6e0 con 0x55655f56e910 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 10 client.18390 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 10 client.18390 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 20 client.18390 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f58c7e0 0x55655f58ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 4 client.18390 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.630+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55655f212720 con 0x55655f58c7e0 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.631+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f58c7e0 0x55655f58ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.631+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f58c7e0 0x55655f58ebd0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6820d465662f08e5 server_cookie=83ddd45321c82bcc in_seq=0 out_seq=0 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.631+0000 7fa9b57fa640 10 client.18390 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9b57fa640 1 -- 172.21.15.3:0/447715569 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55655f212720 con 0x55655f58c7e0 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9b57fa640 10 client.18390 handle_command_reply: tid=0 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 client.18390 shutdown 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 msgr2=0x55655f57be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57be60 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc004650 tx=0x7fa9bc004b00 comp rx=0 tx=0).stop 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 shutdown_connections 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f58c7e0 0x55655f58ebd0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f57c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f57c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9b57fa640 0 client.18390 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/447715569 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57be60 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 >> 172.21.15.3:0/447715569 conn(0x55655f569100 msgr2=0x55655f56ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 shutdown_connections 2024-02-09T22:59:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 1 -- 172.21.15.3:0/447715569 wait complete. 2024-02-09T22:59:13.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.632+0000 7fa9d6bed740 20 client.18390 trim_cache size 0 max 16384 2024-02-09T22:59:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 Processor -- start 2024-02-09T22:59:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 -- start start 2024-02-09T22:59:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f5797b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f57a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f45b130 con 0x55655f56e910 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f56a180 con 0x55655f5783c0 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f56dd40 con 0x55655f56f2c0 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55476/0 (socket says 172.21.15.3:55476) 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 -- 172.21.15.3:0/665247903 learned_addr learned my addr 172.21.15.3:0/665247903 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f5797b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 -- 172.21.15.3:0/665247903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 msgr2=0x55655f5797b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f5797b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 -- 172.21.15.3:0/665247903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 msgr2=0x55655f57a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f57a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 -- 172.21.15.3:0/665247903 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55655f470c90 con 0x55655f56e910 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.644+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57e590 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc013860 tx=0x7fa9bc005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04c80681bd61eaf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9b77fe640 1 -- 172.21.15.3:0/665247903 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc023040 con 0x55655f56e910 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9b77fe640 1 -- 172.21.15.3:0/665247903 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9bc004b70 con 0x55655f56e910 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9b77fe640 1 -- 172.21.15.3:0/665247903 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc005000 con 0x55655f56e910 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 -- 172.21.15.3:0/665247903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 msgr2=0x55655f57e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57e590 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc013860 tx=0x7fa9bc005cc0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 -- 172.21.15.3:0/665247903 shutdown_connections 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f57a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f5797b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/665247903 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f57e590 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 -- 172.21.15.3:0/665247903 >> 172.21.15.3:0/665247903 conn(0x55655f569100 msgr2=0x55655f576c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 -- 172.21.15.3:0/665247903 shutdown_connections 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 -- 172.21.15.3:0/665247903 wait complete. 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 -- start start 2024-02-09T22:59:13.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.645+0000 7fa9d6bed740 1 Processor -- start 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f581d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f582290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f5827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f583ca0 con 0x55655f56e910 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f5e2c50 con 0x55655f5783c0 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55655f5e2d90 con 0x55655f56f2c0 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d6bed740 10 client.0 _ll_register_callbacks cb 0x7fa9bc00b9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f581d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f581d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55490/0 (socket says 172.21.15.3:55490) 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2808604964 learned_addr learned my addr 172.21.15.3:0/2808604964 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f582290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f5827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 msgr2=0x55655f582290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f582290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2808604964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 msgr2=0x55655f5827d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f5827d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 -- 172.21.15.3:0/2808604964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55655f4d46b0 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9d14d9640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f581d50 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc005940 tx=0x7fa9bc00e800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92502eef5334815b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9b57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.646+0000 7fa9b57fa640 1 -- 172.21.15.3:0/2808604964 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9bc023070 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9b57fa640 1 -- 172.21.15.3:0/2808604964 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fa9bc000e40 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9bc019070 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9b57fa640 1 -- 172.21.15.3:0/2808604964 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9bc00b7e0 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 10 client.18396 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55655f496a70 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9b57fa640 1 -- 172.21.15.3:0/2808604964 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9bc019070 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9b57fa640 1 -- 172.21.15.3:0/2808604964 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa9bc012480 con 0x55655f56e910 2024-02-09T22:59:13.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 10 client.18396 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 10 client.18396 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 20 client.18396 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f5e4000 0x55655f5e63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 4 client.18396 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55655f261890 con 0x55655f5e4000 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.647+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f5e4000 0x55655f5e63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9b57fa640 10 client.18396 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d1cda640 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f5e4000 0x55655f5e63f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7fc34337d9d8e17c server_cookie=af2c33c641480a1b in_seq=0 out_seq=0 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9b57fa640 1 -- 172.21.15.3:0/2808604964 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55655f261890 con 0x55655f5e4000 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9b57fa640 10 client.18396 handle_command_reply: tid=0 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 client.18396 shutdown 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 msgr2=0x55655f581d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f581d50 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7fa9bc005940 tx=0x7fa9bc00e800 comp rx=0 tx=0).stop 2024-02-09T22:59:13.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 227846 2.62 MB/sec execute 307 sec latency 1504.353 ms 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 shutdown_connections 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55655f5e4000 0x55655f5e63f0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55655f5783c0 0x55655f5827d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55655f56f2c0 0x55655f582290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9b57fa640 0 client.18396 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 --2- 172.21.15.3:0/2808604964 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55655f56e910 0x55655f581d50 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 >> 172.21.15.3:0/2808604964 conn(0x55655f569100 msgr2=0x55655f57a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 shutdown_connections 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 1 -- 172.21.15.3:0/2808604964 wait complete. 2024-02-09T22:59:13.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.648+0000 7fa9d6bed740 20 client.18396 trim_cache size 0 max 16384 2024-02-09T22:59:13.659 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:13.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa9b80a2230 msgr2=0x7fa9b80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa9b80a2230 0x7fa9b80a46f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fa9c40cbe20 tx=0x7fa9c4085b40 comp rx=0 tx=0).stop 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 msgr2=0x7fa9cc197750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc197750 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7fa9c400e960 tx=0x7fa9c400ee20 comp rx=0 tx=0).stop 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 shutdown_connections 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa9b80a2230 0x7fa9b80a46f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa9cc101d20 0x7fa9cc19db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa9cc1026c0 0x7fa9cc197750 unknown :-1 s=CLOSED pgs=1280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 --2- 172.21.15.3:0/3999752926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa9cc100440 0x7fa9cc19d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 >> 172.21.15.3:0/3999752926 conn(0x7fa9cc077310 msgr2=0x7fa9cc103f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 shutdown_connections 2024-02-09T22:59:13.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.649+0000 7fa9d3f3e640 1 -- 172.21.15.3:0/3999752926 wait complete. 2024-02-09T22:59:13.662 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:13.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:59:13.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.763+0000 7efe84513640 1 -- 172.21.15.3:0/1800844164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 msgr2=0x7efe7c10dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.763+0000 7efe84513640 1 --2- 172.21.15.3:0/1800844164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c10dc50 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7efe74009900 tx=0x7efe7402bcd0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- 172.21.15.3:0/1800844164 shutdown_connections 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 --2- 172.21.15.3:0/1800844164 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe7c102d70 0x7efe7c10b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 --2- 172.21.15.3:0/1800844164 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c10dc50 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 --2- 172.21.15.3:0/1800844164 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe7c102450 0x7efe7c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- 172.21.15.3:0/1800844164 >> 172.21.15.3:0/1800844164 conn(0x7efe7c0772e0 msgr2=0x7efe7c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- 172.21.15.3:0/1800844164 shutdown_connections 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- 172.21.15.3:0/1800844164 wait complete. 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 Processor -- start 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- start start 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe7c102450 0x7efe7c19d460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe7c102d70 0x7efe7c19d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c197c60 con 0x7efe7c10b820 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c197dd0 con 0x7efe7c102d70 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.764+0000 7efe84513640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe7c197f90 con 0x7efe7c102450 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55516/0 (socket says 172.21.15.3:55516) 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 -- 172.21.15.3:0/1429806843 learned_addr learned my addr 172.21.15.3:0/1429806843 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe822af640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe7c102450 0x7efe7c19d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe81aae640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe7c102d70 0x7efe7c19d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 -- 172.21.15.3:0/1429806843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe7c102450 msgr2=0x7efe7c19d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe7c102450 0x7efe7c19d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 -- 172.21.15.3:0/1429806843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe7c102d70 msgr2=0x7efe7c19d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe7c102d70 0x7efe7c19d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 -- 172.21.15.3:0/1429806843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe74009560 con 0x7efe7c10b820 2024-02-09T22:59:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe82ab0640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c197610 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7efe74009230 tx=0x7efe7402c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe7b7fe640 1 -- 172.21.15.3:0/1429806843 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74004350 con 0x7efe7c10b820 2024-02-09T22:59:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe7b7fe640 1 -- 172.21.15.3:0/1429806843 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe740044b0 con 0x7efe7c10b820 2024-02-09T22:59:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe7c198210 con 0x7efe7c10b820 2024-02-09T22:59:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe7b7fe640 1 -- 172.21.15.3:0/1429806843 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe74003b50 con 0x7efe7c10b820 2024-02-09T22:59:13.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.765+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe7c1a4500 con 0x7efe7c10b820 2024-02-09T22:59:13.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.768+0000 7efe7b7fe640 1 -- 172.21.15.3:0/1429806843 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efe74003cb0 con 0x7efe7c10b820 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.768+0000 7efe871c7740 1 Processor -- start 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 -- start start 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10ce6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cefdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10ca2020 con 0x55ad10cf03c0 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10c55320 con 0x55ad10ce6910 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe871c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10c52850 con 0x55ad10ce72c0 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe7b7fe640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe640a22b0 0x7efe640a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55520/0 (socket says 172.21.15.3:55520) 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 -- 172.21.15.3:0/1437815229 learned_addr learned my addr 172.21.15.3:0/1437815229 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe82ab0640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe640a22b0 0x7efe640a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe822af640 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cefdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10ce6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe7b7fe640 1 -- 172.21.15.3:0/1429806843 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe740f06a0 con 0x7efe7c10b820 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 -- 172.21.15.3:0/1437815229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 msgr2=0x55ad10cefdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cefdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 -- 172.21.15.3:0/1437815229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 msgr2=0x55ad10ce6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10ce6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.769+0000 7efe81aae640 1 -- 172.21.15.3:0/1437815229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad10c10210 con 0x55ad10cf03c0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe82ab0640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe640a22b0 0x7efe640a4770 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efe740bced0 tx=0x7efe740b28a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe81aae640 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf0820 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7efe70014920 tx=0x7efe7000c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1aaf043e45c39297 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe78bfe640 1 -- 172.21.15.3:0/1437815229 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70005c50 con 0x55ad10cf03c0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe78bfe640 1 -- 172.21.15.3:0/1437815229 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe7001a020 con 0x55ad10cf03c0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe78bfe640 1 -- 172.21.15.3:0/1437815229 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe70012810 con 0x55ad10cf03c0 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe871c7740 1 -- 172.21.15.3:0/1437815229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 msgr2=0x55ad10cf0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe871c7740 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf0820 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 crypto rx=0x7efe70014920 tx=0x7efe7000c430 comp rx=0 tx=0).stop 2024-02-09T22:59:13.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe871c7740 1 -- 172.21.15.3:0/1437815229 shutdown_connections 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe871c7740 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf0820 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.770+0000 7efe871c7740 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cefdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 --2- 172.21.15.3:0/1437815229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10ce6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- 172.21.15.3:0/1437815229 >> 172.21.15.3:0/1437815229 conn(0x55ad10ce1100 msgr2=0x55ad10ce14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- 172.21.15.3:0/1437815229 shutdown_connections 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- 172.21.15.3:0/1437815229 wait complete. 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- start start 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 Processor -- start 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cf43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf4910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10c4e030 con 0x55ad10cf03c0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10921430 con 0x55ad10ce6910 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad108dde00 con 0x55ad10ce72c0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe871c7740 10 client.0 _ll_register_callbacks cb 0x55ad10c43030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe81aae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe81aae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf4910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55530/0 (socket says 172.21.15.3:55530) 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe81aae640 1 -- 172.21.15.3:0/1165227419 learned_addr learned my addr 172.21.15.3:0/1165227419 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.771+0000 7efe822af640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cf43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 -- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 msgr2=0x55ad10cf43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cf43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 -- 172.21.15.3:0/1165227419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 msgr2=0x55ad10cf3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 -- 172.21.15.3:0/1165227419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe70014600 con 0x55ad10cf03c0 2024-02-09T22:59:13.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe822af640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cf43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe81aae640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf4910 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7efe70005fd0 tx=0x7efe70014f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25aaadf4ba5b101c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe61ffb640 1 -- 172.21.15.3:0/1165227419 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe7000f940 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe61ffb640 1 -- 172.21.15.3:0/1165227419 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe7001b030 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ad1094d7e0 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe61ffb640 1 -- 172.21.15.3:0/1165227419 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe7000faa0 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe871c7740 10 client.18408 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55ad10c4c6b0 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.772+0000 7efe61ffb640 1 -- 172.21.15.3:0/1165227419 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ad1094d7e0 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe61ffb640 1 -- 172.21.15.3:0/1165227419 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efe70011440 con 0x55ad10cf03c0 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe871c7740 10 client.18408 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe871c7740 10 client.18408 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe871c7740 20 client.18408 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe871c7740 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d047a0 0x55ad10d06bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe871c7740 4 client.18408 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad1098a720 con 0x55ad10d047a0 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe822af640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d047a0 0x55ad10d06bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe61ffb640 10 client.18408 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.773+0000 7efe822af640 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d047a0 0x55ad10d06bc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a151aa78ebd9a2fe server_cookie=9b95938c83bff73f in_seq=0 out_seq=0 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe61ffb640 1 -- 172.21.15.3:0/1165227419 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad1098a720 con 0x55ad10d047a0 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe61ffb640 10 client.18408 handle_command_reply: tid=0 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 client.18408 shutdown 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 msgr2=0x55ad10cf4910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf4910 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7efe70005fd0 tx=0x7efe70014f60 comp rx=0 tx=0).stop 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 shutdown_connections 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d047a0 0x55ad10d06bc0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10cf03c0 0x55ad10cf4910 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe61ffb640 0 client.18408 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10ce72c0 0x55ad10cf43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 --2- 172.21.15.3:0/1165227419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.783 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 >> 172.21.15.3:0/1165227419 conn(0x55ad10ce1100 msgr2=0x55ad10ce2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 shutdown_connections 2024-02-09T22:59:13.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 1 -- 172.21.15.3:0/1165227419 wait complete. 2024-02-09T22:59:13.784 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.774+0000 7efe871c7740 20 client.18408 trim_cache size 0 max 16384 2024-02-09T22:59:13.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 Processor -- start 2024-02-09T22:59:13.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 -- start start 2024-02-09T22:59:13.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf65c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cf17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 0x55ad10ceec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10bd3130 con 0x55ad10ce72c0 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10ce5d40 con 0x55ad10ce6910 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe871c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10d0bdb0 con 0x55ad10cf03c0 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cf17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cf17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55546/0 (socket says 172.21.15.3:55546) 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 -- 172.21.15.3:0/3834348100 learned_addr learned my addr 172.21.15.3:0/3834348100 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 -- 172.21.15.3:0/3834348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 msgr2=0x55ad10ceec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 0x55ad10ceec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 -- 172.21.15.3:0/3834348100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 msgr2=0x55ad10cf65c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf65c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 -- 172.21.15.3:0/3834348100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad10be8c90 con 0x55ad10ce72c0 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.786+0000 7efe822af640 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cf17b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x55ad10ce7700 tx=0x7efe6c009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efe78f3ed7406d6e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe63fff640 1 -- 172.21.15.3:0/3834348100 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c016070 con 0x55ad10ce72c0 2024-02-09T22:59:13.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe63fff640 1 -- 172.21.15.3:0/3834348100 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe6c024070 con 0x55ad10ce72c0 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe63fff640 1 -- 172.21.15.3:0/3834348100 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c0208a0 con 0x55ad10ce72c0 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 -- 172.21.15.3:0/3834348100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 msgr2=0x55ad10cf17b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cf17b0 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x55ad10ce7700 tx=0x7efe6c009500 comp rx=0 tx=0).stop 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 -- 172.21.15.3:0/3834348100 shutdown_connections 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 0x55ad10ceec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cf17b0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 --2- 172.21.15.3:0/3834348100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf65c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 -- 172.21.15.3:0/3834348100 >> 172.21.15.3:0/3834348100 conn(0x55ad10ce1100 msgr2=0x55ad10cee140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 -- 172.21.15.3:0/3834348100 shutdown_connections 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 -- 172.21.15.3:0/3834348100 wait complete. 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.787+0000 7efe871c7740 1 -- start start 2024-02-09T22:59:13.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 Processor -- start 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cfa380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 0x55ad10cfa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10d5acb0 con 0x55ad10ce72c0 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10d5adf0 con 0x55ad10ce6910 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad10d5af30 con 0x55ad10cf03c0 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe871c7740 10 client.0 _ll_register_callbacks cb 0x7efe7000ca10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cfa380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cfa380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55556/0 (socket says 172.21.15.3:55556) 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 -- 172.21.15.3:0/3191143955 learned_addr learned my addr 172.21.15.3:0/3191143955 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe81aae640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 -- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 msgr2=0x55ad10cfa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 0x55ad10cfa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 -- 172.21.15.3:0/3191143955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 msgr2=0x55ad10cf9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf9e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 -- 172.21.15.3:0/3191143955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad10c0ea70 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe81aae640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe822af640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cfa380 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7efe6c013010 tx=0x7efe6c020940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c681d986d3dc107c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe61ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.788+0000 7efe61ffb640 1 -- 172.21.15.3:0/3191143955 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6c016070 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe61ffb640 1 -- 172.21.15.3:0/3191143955 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7efe6c004a80 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efe7001a020 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe61ffb640 1 -- 172.21.15.3:0/3191143955 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7efe6c004c80 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 10 client.18414 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55ad10bf5170 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe61ffb640 1 -- 172.21.15.3:0/3191143955 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe7001a020 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe61ffb640 1 -- 172.21.15.3:0/3191143955 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efe6c0056b0 con 0x55ad10ce72c0 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 10 client.18414 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 10 client.18414 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 20 client.18414 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d5c660 0x55ad10d5ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 4 client.18414 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ad109d9890 con 0x55ad10d5c660 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.789+0000 7efe81aae640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d5c660 0x55ad10d5ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe61ffb640 10 client.18414 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe81aae640 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d5c660 0x55ad10d5ea80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e182f66cb3f87a51 server_cookie=126e45adc18882db in_seq=0 out_seq=0 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe61ffb640 1 -- 172.21.15.3:0/3191143955 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ad109d9890 con 0x55ad10d5c660 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe61ffb640 10 client.18414 handle_command_reply: tid=0 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 client.18414 shutdown 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 msgr2=0x55ad10cfa380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cfa380 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7efe6c013010 tx=0x7efe6c020940 comp rx=0 tx=0).stop 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 shutdown_connections 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55ad10d5c660 0x55ad10d5ea80 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ad10cf03c0 0x55ad10cfa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe61ffb640 0 client.18414 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ad10ce72c0 0x55ad10cfa380 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 --2- 172.21.15.3:0/3191143955 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ad10ce6910 0x55ad10cf9e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 >> 172.21.15.3:0/3191143955 conn(0x55ad10ce1100 msgr2=0x55ad10ceec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 shutdown_connections 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 1 -- 172.21.15.3:0/3191143955 wait complete. 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe871c7740 20 client.18414 trim_cache size 0 max 16384 2024-02-09T22:59:13.800 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:13.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe640a22b0 msgr2=0x7efe640a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.790+0000 7efe84513640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe640a22b0 0x7efe640a4770 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7efe740bced0 tx=0x7efe740b28a0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 msgr2=0x7efe7c197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c197610 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7efe74009230 tx=0x7efe7402c6f0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 shutdown_connections 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efe640a22b0 0x7efe640a4770 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe7c102d70 0x7efe7c19d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe7c10b820 0x7efe7c197610 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 --2- 172.21.15.3:0/1429806843 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe7c102450 0x7efe7c19d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 >> 172.21.15.3:0/1429806843 conn(0x7efe7c0772e0 msgr2=0x7efe7c0fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 shutdown_connections 2024-02-09T22:59:13.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.791+0000 7efe84513640 1 -- 172.21.15.3:0/1429806843 wait complete. 2024-02-09T22:59:13.804 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:13.804 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 -- 172.21.15.3:0/1180544416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 msgr2=0x7fe99410b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 --2- 172.21.15.3:0/1180544416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99410b190 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7fe97c009900 tx=0x7fe97c02bcd0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 -- 172.21.15.3:0/1180544416 shutdown_connections 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 --2- 172.21.15.3:0/1180544416 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99410b7b0 0x7fe99410dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 --2- 172.21.15.3:0/1180544416 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99410b190 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 --2- 172.21.15.3:0/1180544416 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe994102390 0x7fe994102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 -- 172.21.15.3:0/1180544416 >> 172.21.15.3:0/1180544416 conn(0x7fe9940fbb90 msgr2=0x7fe9940fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 -- 172.21.15.3:0/1180544416 shutdown_connections 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.906+0000 7fe999bb5640 1 -- 172.21.15.3:0/1180544416 wait complete. 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 Processor -- start 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 -- start start 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe994102390 0x7fe99419d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99419d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99410b7b0 0x7fe994197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994197cc0 con 0x7fe994102cb0 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994197e30 con 0x7fe994102390 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe999bb5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe994197ff0 con 0x7fe99410b7b0 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99419d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99419d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55560/0 (socket says 172.21.15.3:55560) 2024-02-09T22:59:13.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 -- 172.21.15.3:0/512717805 learned_addr learned my addr 172.21.15.3:0/512717805 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe9937fe640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe994102390 0x7fe99419d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe993fff640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99410b7b0 0x7fe994197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 -- 172.21.15.3:0/512717805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99410b7b0 msgr2=0x7fe994197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99410b7b0 0x7fe994197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 -- 172.21.15.3:0/512717805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe994102390 msgr2=0x7fe99419d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe994102390 0x7fe99419d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.907+0000 7fe98bfff640 1 -- 172.21.15.3:0/512717805 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe97c009560 con 0x7fe994102cb0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.908+0000 7fe98bfff640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99419d9b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe97c005940 tx=0x7fe97c02c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.908+0000 7fe9917fa640 1 -- 172.21.15.3:0/512717805 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c004350 con 0x7fe994102cb0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.908+0000 7fe9917fa640 1 -- 172.21.15.3:0/512717805 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe97c0044b0 con 0x7fe994102cb0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.908+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe994198270 con 0x7fe994102cb0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.908+0000 7fe9917fa640 1 -- 172.21.15.3:0/512717805 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c003d30 con 0x7fe994102cb0 2024-02-09T22:59:13.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.908+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9941a4510 con 0x7fe994102cb0 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.910+0000 7fe9917fa640 1 -- 172.21.15.3:0/512717805 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe97c004df0 con 0x7fe994102cb0 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.910+0000 7fe99c781740 1 Processor -- start 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.910+0000 7fe99c781740 1 -- start start 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe99c781740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe99c781740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe99c781740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe99c781740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a08486020 con 0x562a084cb2c0 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9917fa640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe9800a2230 0x7fe9800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe99c781740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a08439320 con 0x562a084ca910 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe99c781740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a08436850 con 0x562a084d43c0 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.920 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55570/0 (socket says 172.21.15.3:55570) 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe993fff640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe9800a2230 0x7fe9800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 -- 172.21.15.3:0/887867263 learned_addr learned my addr 172.21.15.3:0/887867263 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9917fa640 1 -- 172.21.15.3:0/512717805 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe97c0f0620 con 0x7fe994102cb0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe98bfff640 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe993fff640 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 -- 172.21.15.3:0/887867263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 msgr2=0x562a084d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 -- 172.21.15.3:0/887867263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 msgr2=0x562a084cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 -- 172.21.15.3:0/887867263 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a083f4210 con 0x562a084cb2c0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe98bfff640 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe993fff640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe9800a2230 0x7fe9800a46f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe98c007310 tx=0x7fe98c013ac0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe9937fe640 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d3df0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe984009cf0 tx=0x7fe984011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9025aaee715dd3d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe989ffb640 1 -- 172.21.15.3:0/887867263 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe984009ea0 con 0x562a084cb2c0 2024-02-09T22:59:13.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe989ffb640 1 -- 172.21.15.3:0/887867263 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe984004230 con 0x562a084cb2c0 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.911+0000 7fe989ffb640 1 -- 172.21.15.3:0/887867263 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9840198c0 con 0x562a084cb2c0 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 -- 172.21.15.3:0/887867263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 msgr2=0x562a084d3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d3df0 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fe984009cf0 tx=0x7fe984011810 comp rx=0 tx=0).stop 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 -- 172.21.15.3:0/887867263 shutdown_connections 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d3df0 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 --2- 172.21.15.3:0/887867263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 -- 172.21.15.3:0/887867263 >> 172.21.15.3:0/887867263 conn(0x562a084c5100 msgr2=0x562a084c54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 -- 172.21.15.3:0/887867263 shutdown_connections 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 -- 172.21.15.3:0/887867263 wait complete. 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.912+0000 7fe99c781740 1 -- start start 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 Processor -- start 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084d7df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d8330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d8870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a08432030 con 0x562a084cb2c0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a08105430 con 0x562a084ca910 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a080c1e00 con 0x562a084d43c0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe99c781740 10 client.0 _ll_register_callbacks cb 0x562a08427030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d8330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d8330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55582/0 (socket says 172.21.15.3:55582) 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 -- 172.21.15.3:0/2129138708 learned_addr learned my addr 172.21.15.3:0/2129138708 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe9937fe640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084d7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe993fff640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 -- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 msgr2=0x562a084d8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 -- 172.21.15.3:0/2129138708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 msgr2=0x562a084d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084d7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 -- 172.21.15.3:0/2129138708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe984009950 con 0x562a084cb2c0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe98bfff640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d8330 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0b9b20 tx=0x7fe97c0bcd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1949ea50227a1c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.913+0000 7fe972ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe972ffd640 1 -- 172.21.15.3:0/2129138708 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe97c0bd150 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe972ffd640 1 -- 172.21.15.3:0/2129138708 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe97c0c12a0 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562a081317e0 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe972ffd640 1 -- 172.21.15.3:0/2129138708 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe97c0c58f0 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 10 client.18426 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x562a084306b0 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe972ffd640 1 -- 172.21.15.3:0/2129138708 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562a081317e0 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe972ffd640 1 -- 172.21.15.3:0/2129138708 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe97c03c070 con 0x562a084cb2c0 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 10 client.18426 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 10 client.18426 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 20 client.18426 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a084e83f0 0x562a084ea800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 4 client.18426 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562a0816e720 con 0x562a084e83f0 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.914+0000 7fe9937fe640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a084e83f0 0x562a084ea800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe972ffd640 10 client.18426 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe9937fe640 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a084e83f0 0x562a084ea800 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ca3b9693f34aa2b6 server_cookie=ee2c9b45ac5d3908 in_seq=0 out_seq=0 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe972ffd640 1 -- 172.21.15.3:0/2129138708 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562a0816e720 con 0x562a084e83f0 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe972ffd640 10 client.18426 handle_command_reply: tid=0 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe99c781740 1 client.18426 shutdown 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 msgr2=0x562a084d8330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.915+0000 7fe99c781740 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d8330 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fe97c0b9b20 tx=0x7fe97c0bcd50 comp rx=0 tx=0).stop 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 shutdown_connections 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a084e83f0 0x562a084ea800 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084d43c0 0x562a084d8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084cb2c0 0x562a084d8330 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe972ffd640 0 client.18426 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 --2- 172.21.15.3:0/2129138708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084d7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 >> 172.21.15.3:0/2129138708 conn(0x562a084c5100 msgr2=0x562a084c6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 shutdown_connections 2024-02-09T22:59:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 1 -- 172.21.15.3:0/2129138708 wait complete. 2024-02-09T22:59:13.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.916+0000 7fe99c781740 20 client.18426 trim_cache size 0 max 16384 2024-02-09T22:59:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 Processor -- start 2024-02-09T22:59:13.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 -- start start 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084ca910 0x562a084da520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084cb2c0 0x562a084d57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084d8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a083b7130 con 0x562a084d43c0 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0816e720 con 0x562a084cb2c0 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.927+0000 7fe99c781740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a081317e0 con 0x562a084ca910 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084d8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084d8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55594/0 (socket says 172.21.15.3:55594) 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 -- 172.21.15.3:0/2554304680 learned_addr learned my addr 172.21.15.3:0/2554304680 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe9937fe640 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084ca910 0x562a084da520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 -- 172.21.15.3:0/2554304680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084ca910 msgr2=0x562a084da520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084ca910 0x562a084da520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 -- 172.21.15.3:0/2554304680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084cb2c0 msgr2=0x562a084d57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084cb2c0 0x562a084d57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 -- 172.21.15.3:0/2554304680 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562a083ccc90 con 0x562a084d43c0 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe9937fe640 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084ca910 0x562a084da520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe993fff640 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084d8690 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe98c013320 tx=0x7fe98c009000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e66bc1dfbfc11e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe9897fa640 1 -- 172.21.15.3:0/2554304680 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c004d20 con 0x562a084d43c0 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe9897fa640 1 -- 172.21.15.3:0/2554304680 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe98c019070 con 0x562a084d43c0 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.928+0000 7fe9897fa640 1 -- 172.21.15.3:0/2554304680 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c0053e0 con 0x562a084d43c0 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 -- 172.21.15.3:0/2554304680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 msgr2=0x562a084d8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084d8690 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe98c013320 tx=0x7fe98c009000 comp rx=0 tx=0).stop 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 -- 172.21.15.3:0/2554304680 shutdown_connections 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084cb2c0 0x562a084d57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084d8690 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 --2- 172.21.15.3:0/2554304680 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084ca910 0x562a084da520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 -- 172.21.15.3:0/2554304680 >> 172.21.15.3:0/2554304680 conn(0x562a084c5100 msgr2=0x562a084d66c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 -- 172.21.15.3:0/2554304680 shutdown_connections 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 -- 172.21.15.3:0/2554304680 wait complete. 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 -- start start 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.929+0000 7fe99c781740 1 Processor -- start 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084de060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084cb2c0 0x562a084de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084deb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0853eaa0 con 0x562a084d43c0 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0853ec10 con 0x562a084ca910 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562a0853edd0 con 0x562a084cb2c0 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 10 client.0 _ll_register_callbacks cb 0x7fe9840097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084deb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55610/0 (socket says 172.21.15.3:55610) 2024-02-09T22:59:13.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 -- 172.21.15.3:0/3063265153 learned_addr learned my addr 172.21.15.3:0/3063265153 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe98bfff640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084cb2c0 0x562a084de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe9937fe640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084de060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 -- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084cb2c0 msgr2=0x562a084de5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084cb2c0 0x562a084de5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 -- 172.21.15.3:0/3063265153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 msgr2=0x562a084de060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084de060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 -- 172.21.15.3:0/3063265153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe98c0130e0 con 0x562a084d43c0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe98bfff640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084cb2c0 0x562a084de5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe993fff640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084deb00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe98c015040 tx=0x7fe98c004e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e339e3914e32b620 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe972ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe972ffd640 1 -- 172.21.15.3:0/3063265153 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98c02f070 con 0x562a084d43c0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe972ffd640 1 -- 172.21.15.3:0/3063265153 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fe98c0127e0 con 0x562a084d43c0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe984004230 con 0x562a084d43c0 2024-02-09T22:59:13.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.930+0000 7fe972ffd640 1 -- 172.21.15.3:0/3063265153 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe98c0129a0 con 0x562a084d43c0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 10 client.18432 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x562a083f2a70 con 0x562a084d43c0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe972ffd640 1 -- 172.21.15.3:0/3063265153 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe98c042020 con 0x562a084d43c0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe972ffd640 1 -- 172.21.15.3:0/3063265153 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fe98c027030 con 0x562a084d43c0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 10 client.18432 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 10 client.18432 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 20 client.18432 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a085403c0 0x562a085427e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 4 client.18432 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562a081bd890 con 0x562a085403c0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe9937fe640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a085403c0 0x562a085427e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe972ffd640 10 client.18432 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.931+0000 7fe9937fe640 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a085403c0 0x562a085427e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eca9cee6cdf9c8ab server_cookie=b9e8ae1a4e5caf0c in_seq=0 out_seq=0 2024-02-09T22:59:13.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe972ffd640 1 -- 172.21.15.3:0/3063265153 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562a081bd890 con 0x562a085403c0 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe972ffd640 10 client.18432 handle_command_reply: tid=0 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 client.18432 shutdown 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 msgr2=0x562a084deb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084deb00 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7fe98c015040 tx=0x7fe98c004e90 comp rx=0 tx=0).stop 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 shutdown_connections 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x562a085403c0 0x562a085427e0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562a084d43c0 0x562a084deb00 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe972ffd640 0 client.18432 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562a084cb2c0 0x562a084de5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 --2- 172.21.15.3:0/3063265153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562a084ca910 0x562a084de060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 >> 172.21.15.3:0/3063265153 conn(0x562a084c5100 msgr2=0x562a084c39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 shutdown_connections 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 1 -- 172.21.15.3:0/3063265153 wait complete. 2024-02-09T22:59:13.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe99c781740 20 client.18432 trim_cache size 0 max 16384 2024-02-09T22:59:13.942 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe9800a2230 msgr2=0x7fe9800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe9800a2230 0x7fe9800a46f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe98c007310 tx=0x7fe98c013ac0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 msgr2=0x7fe99419d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99419d9b0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7fe97c005940 tx=0x7fe97c02c710 comp rx=0 tx=0).stop 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 shutdown_connections 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe9800a2230 0x7fe9800a46f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe99410b7b0 0x7fe994197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe994102cb0 0x7fe99419d9b0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 --2- 172.21.15.3:0/512717805 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe994102390 0x7fe99419d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 >> 172.21.15.3:0/512717805 conn(0x7fe9940fbb90 msgr2=0x7fe9940fd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.932+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 shutdown_connections 2024-02-09T22:59:13.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:13.933+0000 7fe999bb5640 1 -- 172.21.15.3:0/512717805 wait complete. 2024-02-09T22:59:13.946 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:13.946 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.047+0000 7f14751e2640 1 -- 172.21.15.3:0/1489868312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14701000b0 msgr2=0x7f147010e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.047+0000 7f14751e2640 1 --2- 172.21.15.3:0/1489868312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14701000b0 0x7f147010e080 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f1460009cd0 tx=0x7f146002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 -- 172.21.15.3:0/1489868312 shutdown_connections 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 --2- 172.21.15.3:0/1489868312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14701000b0 0x7f147010e080 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 --2- 172.21.15.3:0/1489868312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14700ff6f0 0x7f14700ffb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 --2- 172.21.15.3:0/1489868312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14701061d0 0x7f14701065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 -- 172.21.15.3:0/1489868312 >> 172.21.15.3:0/1489868312 conn(0x7f14700776a0 msgr2=0x7f14700fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 -- 172.21.15.3:0/1489868312 shutdown_connections 2024-02-09T22:59:14.051 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 -- 172.21.15.3:0/1489868312 wait complete. 2024-02-09T22:59:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 Processor -- start 2024-02-09T22:59:14.053 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.048+0000 7f14751e2640 1 -- start start 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 0x7f147019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14701000b0 0x7f147019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14701061d0 0x7f1470197660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470197df0 con 0x7f14700ff6f0 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470197f60 con 0x7f14701000b0 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470198120 con 0x7f14701061d0 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 0x7f147019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 0x7f147019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55626/0 (socket says 172.21.15.3:55626) 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 -- 172.21.15.3:0/3933614938 learned_addr learned my addr 172.21.15.3:0/3933614938 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146e59c640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14701000b0 0x7f147019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146f59e640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14701061d0 0x7f1470197660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 -- 172.21.15.3:0/3933614938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14701061d0 msgr2=0x7f1470197660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14701061d0 0x7f1470197660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.054 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 -- 172.21.15.3:0/3933614938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14701000b0 msgr2=0x7f147019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14701000b0 0x7f147019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 -- 172.21.15.3:0/3933614938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1460009900 con 0x7f14700ff6f0 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f146ed9d640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 0x7f147019d5a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f146800d950 tx=0x7f146800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f1467fff640 1 -- 172.21.15.3:0/3933614938 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468009c30 con 0x7f14700ff6f0 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f1467fff640 1 -- 172.21.15.3:0/3933614938 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1468009d90 con 0x7f14700ff6f0 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1470198400 con 0x7f14700ff6f0 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.049+0000 7f1467fff640 1 -- 172.21.15.3:0/3933614938 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146800f7e0 con 0x7f14700ff6f0 2024-02-09T22:59:14.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.050+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701a4520 con 0x7f14700ff6f0 2024-02-09T22:59:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.052+0000 7f1467fff640 1 -- 172.21.15.3:0/3933614938 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1468010ad0 con 0x7f14700ff6f0 2024-02-09T22:59:14.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.052+0000 7f1477db2740 1 Processor -- start 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.052+0000 7f1477db2740 1 -- start start 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.052+0000 7f1467fff640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f145c0a2230 0x7f145c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1477db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f151e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1477db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f15a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1477db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 0x56540f15d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1477db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f10d020 con 0x56540f1523e0 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1477db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f0c0320 con 0x56540f15af60 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1477db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f0bd850 con 0x56540f151a30 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f1467fff640 1 -- 172.21.15.3:0/3933614938 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14680cdce0 con 0x7f14700ff6f0 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f15a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146f59e640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f145c0a2230 0x7f145c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f15a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55640/0 (socket says 172.21.15.3:55640) 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146e59c640 1 -- 172.21.15.3:0/367627442 learned_addr learned my addr 172.21.15.3:0/367627442 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 -- 172.21.15.3:0/367627442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 msgr2=0x56540f15d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 0x56540f15d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 -- 172.21.15.3:0/367627442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 msgr2=0x56540f15a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f15a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 -- 172.21.15.3:0/367627442 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56540f07b210 con 0x56540f151a30 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146e59c640 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146f59e640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f145c0a2230 0x7f145c0a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1460006010 tx=0x7f1460009890 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f146ed9d640 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f151e10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14680cb460 tx=0x7f1468000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d90ebae647e96944 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.053+0000 7f14657fa640 1 -- 172.21.15.3:0/367627442 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146809a700 con 0x56540f151a30 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f14657fa640 1 -- 172.21.15.3:0/367627442 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f146809e850 con 0x56540f151a30 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f14657fa640 1 -- 172.21.15.3:0/367627442 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468096d70 con 0x56540f151a30 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 -- 172.21.15.3:0/367627442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 msgr2=0x56540f151e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f151e10 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f14680cb460 tx=0x7f1468000bc0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 -- 172.21.15.3:0/367627442 shutdown_connections 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 0x56540f15d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f15a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 --2- 172.21.15.3:0/367627442 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f151e10 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 -- 172.21.15.3:0/367627442 >> 172.21.15.3:0/367627442 conn(0x56540f14c100 msgr2=0x56540f14c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 -- 172.21.15.3:0/367627442 shutdown_connections 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.054+0000 7f1477db2740 1 -- 172.21.15.3:0/367627442 wait complete. 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 -- start start 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 Processor -- start 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f151a30 0x56540f157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f1523e0 0x56540f1584f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 0x56540f158a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f0b9030 con 0x56540f15af60 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540ed8c430 con 0x56540f151a30 2024-02-09T22:59:14.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540ed48e00 con 0x56540f1523e0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f1477db2740 10 client.0 _ll_register_callbacks cb 0x56540f0ae030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 0x56540f158a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 0x56540f158a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55642/0 (socket says 172.21.15.3:55642) 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 -- 172.21.15.3:0/534584385 learned_addr learned my addr 172.21.15.3:0/534584385 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146e59c640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f1523e0 0x56540f1584f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146ed9d640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f151a30 0x56540f157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 -- 172.21.15.3:0/534584385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f1523e0 msgr2=0x56540f1584f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f1523e0 0x56540f1584f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 -- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f151a30 msgr2=0x56540f157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f151a30 0x56540f157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.055+0000 7f146f59e640 1 -- 172.21.15.3:0/534584385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701a4520 con 0x56540f15af60 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f146ed9d640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f151a30 0x56540f157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f146f59e640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 0x56540f158a30 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f1460036000 tx=0x7f14600095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fee384ad1aacfa3e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f14567fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:14.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f14567fc640 1 -- 172.21.15.3:0/534584385 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1460032e90 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f14567fc640 1 -- 172.21.15.3:0/534584385 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f14600324c0 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56540edb87e0 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f14567fc640 1 -- 172.21.15.3:0/534584385 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1460003970 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 10 client.18444 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x56540f0b76b0 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f14567fc640 1 -- 172.21.15.3:0/534584385 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56540edb87e0 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f14567fc640 1 -- 172.21.15.3:0/534584385 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f146003e040 con 0x56540f15af60 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 10 client.18444 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 10 client.18444 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 20 client.18444 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f16d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 4 client.18444 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.056+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56540edf5720 con 0x56540f16b5d0 2024-02-09T22:59:14.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.057+0000 7f146ed9d640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f16d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.057+0000 7f146ed9d640 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f16d9f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=356daffb12c309a8 server_cookie=46eaa4a99217a4d6 in_seq=0 out_seq=0 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.057+0000 7f14567fc640 10 client.18444 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f14567fc640 1 -- 172.21.15.3:0/534584385 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56540edf5720 con 0x56540f16b5d0 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f14567fc640 10 client.18444 handle_command_reply: tid=0 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 client.18444 shutdown 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 msgr2=0x56540f158a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 0x56540f158a30 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f1460036000 tx=0x7f14600095c0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 shutdown_connections 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f16d9f0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f15af60 0x56540f158a30 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f1523e0 0x56540f1584f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f14567fc640 0 client.18444 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 --2- 172.21.15.3:0/534584385 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f151a30 0x56540f157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 >> 172.21.15.3:0/534584385 conn(0x56540f14c100 msgr2=0x56540f14de50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 shutdown_connections 2024-02-09T22:59:14.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 1 -- 172.21.15.3:0/534584385 wait complete. 2024-02-09T22:59:14.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.058+0000 7f1477db2740 20 client.18444 trim_cache size 0 max 16384 2024-02-09T22:59:14.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 Processor -- start 2024-02-09T22:59:14.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 -- start start 2024-02-09T22:59:14.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 0x56540f15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f1523e0 0x56540f158850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f15af60 0x56540f153250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f03e130 con 0x56540f151a30 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f15d580 con 0x56540f1523e0 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f1477db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f151530 con 0x56540f15af60 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 0x56540f15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 0x56540f15a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55644/0 (socket says 172.21.15.3:55644) 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 -- 172.21.15.3:0/2080106163 learned_addr learned my addr 172.21.15.3:0/2080106163 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146f59e640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f15af60 0x56540f153250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 -- 172.21.15.3:0/2080106163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f15af60 msgr2=0x56540f153250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f15af60 0x56540f153250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 -- 172.21.15.3:0/2080106163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f1523e0 msgr2=0x56540f158850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146e59c640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f1523e0 0x56540f158850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f1523e0 0x56540f158850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146ed9d640 1 -- 172.21.15.3:0/2080106163 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56540f053c90 con 0x56540f151a30 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146f59e640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f15af60 0x56540f153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.070+0000 7f146e59c640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f1523e0 0x56540f158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f146ed9d640 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 0x56540f15a6e0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f14680cb4c0 tx=0x7f1468005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2128efd4888fb63d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1464ff9640 1 -- 172.21.15.3:0/2080106163 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468085130 con 0x56540f151a30 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1464ff9640 1 -- 172.21.15.3:0/2080106163 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14680b3210 con 0x56540f151a30 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1464ff9640 1 -- 172.21.15.3:0/2080106163 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1468096d70 con 0x56540f151a30 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 -- 172.21.15.3:0/2080106163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 msgr2=0x56540f15a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 0x56540f15a6e0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7f14680cb4c0 tx=0x7f1468005880 comp rx=0 tx=0).stop 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 -- 172.21.15.3:0/2080106163 shutdown_connections 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f1523e0 0x56540f158850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f15af60 0x56540f153250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 --2- 172.21.15.3:0/2080106163 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f151a30 0x56540f15a6e0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 -- 172.21.15.3:0/2080106163 >> 172.21.15.3:0/2080106163 conn(0x56540f14c100 msgr2=0x56540f147210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 -- 172.21.15.3:0/2080106163 shutdown_connections 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.071+0000 7f1477db2740 1 -- 172.21.15.3:0/2080106163 wait complete. 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 -- start start 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 Processor -- start 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f1623b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f1607a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 0x56540f160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f1615b0 con 0x56540f1523e0 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f1c1ae0 con 0x56540f15af60 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56540f1c1c20 con 0x56540f151a30 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f1477db2740 10 client.0 _ll_register_callbacks cb 0x7f146800df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f1607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f1607a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55646/0 (socket says 172.21.15.3:55646) 2024-02-09T22:59:14.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 -- 172.21.15.3:0/2109427373 learned_addr learned my addr 172.21.15.3:0/2109427373 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 -- 172.21.15.3:0/2109427373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 msgr2=0x56540f1623b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f1623b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 -- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 msgr2=0x56540f160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 0x56540f160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.072+0000 7f146e59c640 1 -- 172.21.15.3:0/2109427373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56540f07b210 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f146e59c640 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f1607a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x56540f1527e0 tx=0x7f14580039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32bf46230c2ee678 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f14567fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f14567fc640 1 -- 172.21.15.3:0/2109427373 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458020070 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f14567fc640 1 -- 172.21.15.3:0/2109427373 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f1458005020 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f146809e850 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f14567fc640 1 -- 172.21.15.3:0/2109427373 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1458005ad0 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 10 client.18450 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x56540f079a70 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f14567fc640 1 -- 172.21.15.3:0/2109427373 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f146809e850 con 0x56540f1523e0 2024-02-09T22:59:14.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f14567fc640 1 -- 172.21.15.3:0/2109427373 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1458023020 con 0x56540f1523e0 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 10 client.18450 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 10 client.18450 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 20 client.18450 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f1c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 4 client.18450 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56540ee44890 con 0x56540f16b5d0 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.073+0000 7f146ed9d640 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f1c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f14567fc640 10 client.18450 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f146ed9d640 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f1c5460 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=778387245a2e80e3 server_cookie=7fb70e1797222630 in_seq=0 out_seq=0 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f14567fc640 1 -- 172.21.15.3:0/2109427373 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56540ee44890 con 0x56540f16b5d0 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f14567fc640 10 client.18450 handle_command_reply: tid=0 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 client.18450 shutdown 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 msgr2=0x56540f1607a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f1607a0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x56540f1527e0 tx=0x7f14580039c0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 shutdown_connections 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56540f16b5d0 0x56540f1c5460 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56540f15af60 0x56540f160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56540f1523e0 0x56540f1607a0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f14567fc640 0 client.18450 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 --2- 172.21.15.3:0/2109427373 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56540f151a30 0x56540f1623b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 >> 172.21.15.3:0/2109427373 conn(0x56540f14c100 msgr2=0x56540f153250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 shutdown_connections 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 1 -- 172.21.15.3:0/2109427373 wait complete. 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.074+0000 7f1477db2740 20 client.18450 trim_cache size 0 max 16384 2024-02-09T22:59:14.083 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:14.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f145c0a2230 msgr2=0x7f145c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f145c0a2230 0x7f145c0a46f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1460006010 tx=0x7f1460009890 comp rx=0 tx=0).stop 2024-02-09T22:59:14.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 msgr2=0x7f147019d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 0x7f147019d5a0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f146800d950 tx=0x7f146800de10 comp rx=0 tx=0).stop 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 shutdown_connections 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f145c0a2230 0x7f145c0a46f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f14701000b0 0x7f147019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f14701061d0 0x7f1470197660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 --2- 172.21.15.3:0/3933614938 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f14700ff6f0 0x7f147019d5a0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 >> 172.21.15.3:0/3933614938 conn(0x7f14700776a0 msgr2=0x7f14701041b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 shutdown_connections 2024-02-09T22:59:14.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.075+0000 7f14751e2640 1 -- 172.21.15.3:0/3933614938 wait complete. 2024-02-09T22:59:14.088 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:14.088 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:59:14.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 -- 172.21.15.3:0/2707160218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c10b7b0 msgr2=0x7f492c10dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 --2- 172.21.15.3:0/2707160218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c10b7b0 0x7f492c10dc50 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7f4924009ca0 tx=0x7f492402c180 comp rx=0 tx=0).stop 2024-02-09T22:59:14.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 -- 172.21.15.3:0/2707160218 shutdown_connections 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 --2- 172.21.15.3:0/2707160218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c10b7b0 0x7f492c10dc50 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 --2- 172.21.15.3:0/2707160218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f492c102cb0 0x7f492c10b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 --2- 172.21.15.3:0/2707160218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f492c102390 0x7f492c102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 -- 172.21.15.3:0/2707160218 >> 172.21.15.3:0/2707160218 conn(0x7f492c0fbb90 msgr2=0x7f492c0fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 -- 172.21.15.3:0/2707160218 shutdown_connections 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.190+0000 7f4930fd7640 1 -- 172.21.15.3:0/2707160218 wait complete. 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 Processor -- start 2024-02-09T22:59:14.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 -- start start 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f492c102390 0x7f492c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 0x7f492c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f492c10b7b0 0x7f492c197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c197c60 con 0x7f492c102cb0 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c197dd0 con 0x7f492c10b7b0 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4930fd7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f492c197f90 con 0x7f492c102390 2024-02-09T22:59:14.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 0x7f492c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 0x7f492c19d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55668/0 (socket says 172.21.15.3:55668) 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 -- 172.21.15.3:0/1049368799 learned_addr learned my addr 172.21.15.3:0/1049368799 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f492a59c640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f492c102390 0x7f492c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f492c10b7b0 0x7f492c197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 -- 172.21.15.3:0/1049368799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f492c102390 msgr2=0x7f492c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f492c102390 0x7f492c19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 -- 172.21.15.3:0/1049368799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f492c10b7b0 msgr2=0x7f492c197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f492c10b7b0 0x7f492c197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.191+0000 7f4923fff640 1 -- 172.21.15.3:0/1049368799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4924009900 con 0x7f492c102cb0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.192+0000 7f4923fff640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 0x7f492c19d950 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f491400d740 tx=0x7f491400db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.192+0000 7f49237fe640 1 -- 172.21.15.3:0/1049368799 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4914009a20 con 0x7f492c102cb0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.192+0000 7f49237fe640 1 -- 172.21.15.3:0/1049368799 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4914009b80 con 0x7f492c102cb0 2024-02-09T22:59:14.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.192+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492c198270 con 0x7f492c102cb0 2024-02-09T22:59:14.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.192+0000 7f49237fe640 1 -- 172.21.15.3:0/1049368799 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491400f620 con 0x7f492c102cb0 2024-02-09T22:59:14.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.192+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c1a44b0 con 0x7f492c102cb0 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.194+0000 7f49237fe640 1 -- 172.21.15.3:0/1049368799 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f4914010460 con 0x7f492c102cb0 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.194+0000 7f4933baf740 1 Processor -- start 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.194+0000 7f4933baf740 1 -- start start 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4933baf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f57383be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4933baf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573844990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4933baf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f573844f60 0x55f5738473b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4933baf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5737f7020 con 0x55f57383ba30 2024-02-09T22:59:14.205 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4933baf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5737aa320 con 0x55f573844f60 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4933baf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5737a7850 con 0x55f57383c3e0 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f49237fe640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49180a2230 0x7f49180a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f57383be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f57383be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55680/0 (socket says 172.21.15.3:55680) 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 -- 172.21.15.3:0/1810793316 learned_addr learned my addr 172.21.15.3:0/1810793316 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f49237fe640 1 -- 172.21.15.3:0/1049368799 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f49140cdb50 con 0x7f492c102cb0 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4923fff640 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f573844f60 0x55f5738473b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49180a2230 0x7f49180a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492a59c640 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573844990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 -- 172.21.15.3:0/1810793316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 msgr2=0x55f573844990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573844990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 -- 172.21.15.3:0/1810793316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f573844f60 msgr2=0x55f5738473b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f573844f60 0x55f5738473b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f492ad9d640 1 -- 172.21.15.3:0/1810793316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f573765210 con 0x55f57383ba30 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.195+0000 7f4923fff640 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f573844f60 0x55f5738473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.196+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f57383be10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4924037040 tx=0x7f4924003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ae81ab18cc2f63f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.196+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49180a2230 0x7f49180a46f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4924009ca0 tx=0x7f492403c000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.196+0000 7f4920ff9640 1 -- 172.21.15.3:0/1810793316 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924048070 con 0x55f57383ba30 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.196+0000 7f4920ff9640 1 -- 172.21.15.3:0/1810793316 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4924032480 con 0x55f57383ba30 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.196+0000 7f4920ff9640 1 -- 172.21.15.3:0/1810793316 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924032e80 con 0x55f57383ba30 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- 172.21.15.3:0/1810793316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 msgr2=0x55f57383be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f57383be10 secure :-1 s=READY pgs=1304 cs=0 l=1 rev1=1 crypto rx=0x7f4924037040 tx=0x7f4924003aa0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- 172.21.15.3:0/1810793316 shutdown_connections 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f573844f60 0x55f5738473b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573844990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- 172.21.15.3:0/1810793316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f57383be10 unknown :-1 s=CLOSED pgs=1304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- 172.21.15.3:0/1810793316 >> 172.21.15.3:0/1810793316 conn(0x55f573836100 msgr2=0x55f5738364e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- 172.21.15.3:0/1810793316 shutdown_connections 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- 172.21.15.3:0/1810793316 wait complete. 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- start start 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 Processor -- start 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f573841fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383c3e0 0x55f573842500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f573842a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5737a3030 con 0x55f57383ba30 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f573476430 con 0x55f57383c3e0 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f573432e00 con 0x55f573844f60 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f4933baf740 10 client.0 _ll_register_callbacks cb 0x55f573798030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f492a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f573841fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f492a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f573841fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55686/0 (socket says 172.21.15.3:55686) 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f492a59c640 1 -- 172.21.15.3:0/3121543520 learned_addr learned my addr 172.21.15.3:0/3121543520 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.197+0000 7f492ad9d640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f573842a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4923fff640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383c3e0 0x55f573842500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f492a59c640 1 -- 172.21.15.3:0/3121543520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 msgr2=0x55f573842a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f492a59c640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f573842a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f492a59c640 1 -- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383c3e0 msgr2=0x55f573842500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f492a59c640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383c3e0 0x55f573842500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f492a59c640 1 -- 172.21.15.3:0/3121543520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f492c10e8d0 con 0x55f57383ba30 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4923fff640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383c3e0 0x55f573842500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f492a59c640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f573841fc0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f491c010820 tx=0x7f491c010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d56e0686ff796d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4911ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4911ffb640 1 -- 172.21.15.3:0/3121543520 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491c015070 con 0x55f57383ba30 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4911ffb640 1 -- 172.21.15.3:0/3121543520 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f491c00fb90 con 0x55f57383ba30 2024-02-09T22:59:14.209 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f5734a27e0 con 0x55f57383ba30 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4911ffb640 1 -- 172.21.15.3:0/3121543520 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f491c00fd50 con 0x55f57383ba30 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4933baf740 10 client.18462 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55f5737a16b0 con 0x55f57383ba30 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4911ffb640 1 -- 172.21.15.3:0/3121543520 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f5734a27e0 con 0x55f57383ba30 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.198+0000 7f4911ffb640 1 -- 172.21.15.3:0/3121543520 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f491c0214a0 con 0x55f57383ba30 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4933baf740 10 client.18462 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4933baf740 10 client.18462 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4933baf740 20 client.18462 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4933baf740 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f573855670 0x55f573857a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4933baf740 4 client.18462 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5734df720 con 0x55f573855670 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4923fff640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f573855670 0x55f573857a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4911ffb640 10 client.18462 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:14.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.199+0000 7f4923fff640 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f573855670 0x55f573857a90 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4c1e2c01718afc server_cookie=20677428ce58f1be in_seq=0 out_seq=0 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4911ffb640 1 -- 172.21.15.3:0/3121543520 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f5734df720 con 0x55f573855670 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4911ffb640 10 client.18462 handle_command_reply: tid=0 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 client.18462 shutdown 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 msgr2=0x55f573841fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f573841fc0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f491c010820 tx=0x7f491c010ce0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 shutdown_connections 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f573855670 0x55f573857a90 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383c3e0 0x55f573842500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4911ffb640 0 client.18462 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f573842a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 --2- 172.21.15.3:0/3121543520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383ba30 0x55f573841fc0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 >> 172.21.15.3:0/3121543520 conn(0x55f573836100 msgr2=0x55f573837e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 shutdown_connections 2024-02-09T22:59:14.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.200+0000 7f4933baf740 1 -- 172.21.15.3:0/3121543520 wait complete. 2024-02-09T22:59:14.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.201+0000 7f4933baf740 20 client.18462 trim_cache size 0 max 16384 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 Processor -- start 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 -- start start 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f5738446f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573842860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 0x55f57383f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f573728130 con 0x55f573844f60 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f573843310 con 0x55f57383ba30 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f4933baf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f57385cce0 con 0x55f57383c3e0 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f492ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 0x55f57383f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f492ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 0x55f57383f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55694/0 (socket says 172.21.15.3:55694) 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.212+0000 7f492ad9d640 1 -- 172.21.15.3:0/1378225408 learned_addr learned my addr 172.21.15.3:0/1378225408 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f492ad9d640 1 -- 172.21.15.3:0/1378225408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 msgr2=0x55f573842860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573842860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f492ad9d640 1 -- 172.21.15.3:0/1378225408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 msgr2=0x55f5738446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f5738446f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f492ad9d640 1 -- 172.21.15.3:0/1378225408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f57373dc90 con 0x55f573844f60 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f492ad9d640 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 0x55f57383f220 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f492402c6c0 tx=0x7f4924034550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a18640a8648a74 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4913fff640 1 -- 172.21.15.3:0/1378225408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924048050 con 0x55f573844f60 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4913fff640 1 -- 172.21.15.3:0/1378225408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4924049040 con 0x55f573844f60 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4913fff640 1 -- 172.21.15.3:0/1378225408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4924004520 con 0x55f573844f60 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 -- 172.21.15.3:0/1378225408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 msgr2=0x55f57383f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 0x55f57383f220 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f492402c6c0 tx=0x7f4924034550 comp rx=0 tx=0).stop 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 -- 172.21.15.3:0/1378225408 shutdown_connections 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f573844f60 0x55f57383f220 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f57383c3e0 0x55f573842860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 --2- 172.21.15.3:0/1378225408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f5738446f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.213+0000 7f4933baf740 1 -- 172.21.15.3:0/1378225408 >> 172.21.15.3:0/1378225408 conn(0x55f573836100 msgr2=0x55f57383d2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 -- 172.21.15.3:0/1378225408 shutdown_connections 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 -- 172.21.15.3:0/1378225408 wait complete. 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 -- start start 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 Processor -- start 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f57384b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 0x55f57384b680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f5738abbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5738ac100 con 0x55f57383c3e0 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5738ac240 con 0x55f57383ba30 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5738ac380 con 0x55f573844f60 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4933baf740 10 client.0 _ll_register_callbacks cb 0x7f4924009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 0x55f57384b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 0x55f57384b680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55710/0 (socket says 172.21.15.3:55710) 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 -- 172.21.15.3:0/3916467755 learned_addr learned my addr 172.21.15.3:0/3916467755 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f492ad9d640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f5738abbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 -- 172.21.15.3:0/3916467755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 msgr2=0x55f5738abbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f5738abbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 -- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 msgr2=0x55f57384b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f492a59c640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f57384b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f57384b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f4923fff640 1 -- 172.21.15.3:0/3916467755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f573765210 con 0x55f57383c3e0 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f492ad9d640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f5738abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.214+0000 7f492a59c640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f57384b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4923fff640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 0x55f57384b680 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f491409a4a0 tx=0x7f49140968d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e5c6f3eb3a6182 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4911ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:14.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4911ffb640 1 -- 172.21.15.3:0/3916467755 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f491409e660 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4911ffb640 1 -- 172.21.15.3:0/3916467755 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f491409f900 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4924032480 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4911ffb640 1 -- 172.21.15.3:0/3916467755 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f49140a2cb0 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 10 client.18468 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55f573763a70 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4911ffb640 1 -- 172.21.15.3:0/3916467755 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4924032480 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4911ffb640 1 -- 172.21.15.3:0/3916467755 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f49140cb580 con 0x55f57383c3e0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 10 client.18468 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 10 client.18468 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 20 client.18468 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f5738adaf0 0x55f5738afee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 4 client.18468 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f57352e890 con 0x55f5738adaf0 2024-02-09T22:59:14.225 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.215+0000 7f492a59c640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f5738adaf0 0x55f5738afee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4911ffb640 10 client.18468 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f492a59c640 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f5738adaf0 0x55f5738afee0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f575f0f84e8757e server_cookie=ebe3efef53aeadd5 in_seq=0 out_seq=0 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4911ffb640 1 -- 172.21.15.3:0/3916467755 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f57352e890 con 0x55f5738adaf0 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4911ffb640 10 client.18468 handle_command_reply: tid=0 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 client.18468 shutdown 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 msgr2=0x55f57384b680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 0x55f57384b680 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 crypto rx=0x7f491409a4a0 tx=0x7f49140968d0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 shutdown_connections 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f5738adaf0 0x55f5738afee0 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f573844f60 0x55f5738abbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f57383c3e0 0x55f57384b680 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4911ffb640 0 client.18468 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 --2- 172.21.15.3:0/3916467755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f57383ba30 0x55f57384b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 >> 172.21.15.3:0/3916467755 conn(0x55f573836100 msgr2=0x55f5738349b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 shutdown_connections 2024-02-09T22:59:14.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 1 -- 172.21.15.3:0/3916467755 wait complete. 2024-02-09T22:59:14.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.216+0000 7f4933baf740 20 client.18468 trim_cache size 0 max 16384 2024-02-09T22:59:14.227 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49180a2230 msgr2=0x7f49180a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49180a2230 0x7f49180a46f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4924009ca0 tx=0x7f492403c000 comp rx=0 tx=0).stop 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 msgr2=0x7f492c19d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 0x7f492c19d950 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f491400d740 tx=0x7f491400db40 comp rx=0 tx=0).stop 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 shutdown_connections 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f49180a2230 0x7f49180a46f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f492c10b7b0 0x7f492c197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f492c102cb0 0x7f492c19d950 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 --2- 172.21.15.3:0/1049368799 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f492c102390 0x7f492c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:14.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 >> 172.21.15.3:0/1049368799 conn(0x7f492c0fbb90 msgr2=0x7f492c0fd2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:14.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 shutdown_connections 2024-02-09T22:59:14.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:14.217+0000 7f4930fd7640 1 -- 172.21.15.3:0/1049368799 wait complete. 2024-02-09T22:59:14.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:14 smithi003 ceph-mon[32678]: pgmap v248: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 240 op/s 2024-02-09T22:59:14.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:14 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2060061374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:59:14.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 227846 2.61 MB/sec execute 308 sec latency 1513.933 ms 2024-02-09T22:59:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:14 smithi064 ceph-mon[36638]: pgmap v248: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 240 op/s 2024-02-09T22:59:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:14 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2060061374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:59:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:14 smithi089 ceph-mon[36665]: pgmap v248: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.7 MiB/s wr, 240 op/s 2024-02-09T22:59:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:14 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2060061374' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:59:15.231 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:59:15.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:59:15.232 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:59:15.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/3877216177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 msgr2=0x7fb2dc0fff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/3877216177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc0fff00 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0009930 tx=0x7fb2d002bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/3877216177 shutdown_connections 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/3877216177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2dc100440 0x7fb2dc10e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/3877216177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc0fff00 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/3877216177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc106220 0x7fb2dc106600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/3877216177 >> 172.21.15.3:0/3877216177 conn(0x7fb2dc076410 msgr2=0x7fb2dc076820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/3877216177 shutdown_connections 2024-02-09T22:59:15.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.351+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/3877216177 wait complete. 2024-02-09T22:59:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 Processor -- start 2024-02-09T22:59:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 -- start start 2024-02-09T22:59:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc100440 0x7fb2dc19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2dc106220 0x7fb2dc197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc197e10 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc197f80 con 0x7fb2dc100440 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2e0b3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc198140 con 0x7fb2dc106220 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.352+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55726/0 (socket says 172.21.15.3:55726) 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 -- 172.21.15.3:0/2952927480 learned_addr learned my addr 172.21.15.3:0/2952927480 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2dc106220 0x7fb2dc197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc100440 0x7fb2dc19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 -- 172.21.15.3:0/2952927480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2dc106220 msgr2=0x7fb2dc197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2dc106220 0x7fb2dc197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 -- 172.21.15.3:0/2952927480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc100440 msgr2=0x7fb2dc19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc100440 0x7fb2dc19db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 -- 172.21.15.3:0/2952927480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d0009560 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc100440 0x7fb2dc19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2da59c640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc19d5c0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00d950 tx=0x7fb2cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2cb7fe640 1 -- 172.21.15.3:0/2952927480 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc009c30 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2cb7fe640 1 -- 172.21.15.3:0/2952927480 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2cc009d90 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc198420 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2cb7fe640 1 -- 172.21.15.3:0/2952927480 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc00f7e0 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.353+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc1a4540 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.355+0000 7fb2cb7fe640 1 -- 172.21.15.3:0/2952927480 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb2cc010ad0 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 Processor -- start 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 -- start start 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 0x55cc7affba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7af5e000 con 0x55cc7aff9690 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7af5b530 con 0x55cc7aff0ab0 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7af56d10 con 0x55cc7aff0190 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.356+0000 7fb2cb7fe640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2c40a2230 0x7fb2c40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 0x55cc7affba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 0x55cc7affba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55734/0 (socket says 172.21.15.3:55734) 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1954867332 learned_addr learned my addr 172.21.15.3:0/1954867332 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2da59c640 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2c40a2230 0x7fb2c40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2cb7fe640 1 -- 172.21.15.3:0/2952927480 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb2cc0cdd90 con 0x7fb2dc0ffa80 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1954867332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 msgr2=0x55cc7aff0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1954867332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 msgr2=0x55cc7aff9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1954867332 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc7aefd210 con 0x55cc7aff9690 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2da59c640 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 0x55cc7affba80 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0006010 tx=0x7fb2d0005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7345a5d5f6003058 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2c40a2230 0x7fb2c40a46f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4005c40 tx=0x7fb2d400a3c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2c8ff9640 1 -- 172.21.15.3:0/1954867332 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0004b00 con 0x55cc7aff9690 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2c8ff9640 1 -- 172.21.15.3:0/1954867332 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2d0004030 con 0x55cc7aff9690 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.357+0000 7fb2c8ff9640 1 -- 172.21.15.3:0/1954867332 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0005330 con 0x55cc7aff9690 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1954867332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 msgr2=0x55cc7affba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 0x55cc7affba80 secure :-1 s=READY pgs=1310 cs=0 l=1 rev1=1 crypto rx=0x7fb2d0006010 tx=0x7fb2d0005e80 comp rx=0 tx=0).stop 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1954867332 shutdown_connections 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff9690 0x55cc7affba80 unknown :-1 s=CLOSED pgs=1310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1954867332 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1954867332 >> 172.21.15.3:0/1954867332 conn(0x55cc7afea8c0 msgr2=0x55cc7afeaca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1954867332 shutdown_connections 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1954867332 wait complete. 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.358+0000 7fb2e37ee740 1 -- start start 2024-02-09T22:59:15.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 Processor -- start 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 0x55cc7aff6590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7aff7010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7aefbe60 con 0x55cc7aff0190 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7ac2a4b0 con 0x55cc7aff0ab0 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7aedc670 con 0x55cc7aff9690 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2e37ee740 10 client.0 _ll_register_callbacks cb 0x55cc7aef4190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 0x55cc7aff6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 0x55cc7aff6590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55736/0 (socket says 172.21.15.3:55736) 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 -- 172.21.15.3:0/2051207098 learned_addr learned my addr 172.21.15.3:0/2051207098 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7aff7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 -- 172.21.15.3:0/2051207098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 msgr2=0x55cc7aff7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7aff7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 -- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 msgr2=0x55cc7aff6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 -- 172.21.15.3:0/2051207098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc10e9b0 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2da59c640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 0x55cc7aff6590 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc09a740 tx=0x7fb2cc065ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126de1ee8e60aeff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.359+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/2051207098 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc096a50 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/2051207098 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb2cc065d10 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cc7ac56860 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/2051207098 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2cc09aeb0 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 10 client.18480 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55cc7af55390 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/2051207098 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55cc7ac56860 con 0x55cc7aff0190 2024-02-09T22:59:15.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/2051207098 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb2cc0cb790 con 0x55cc7aff0190 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 10 client.18480 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 10 client.18480 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 20 client.18480 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b0094a0 0x55cc7b00b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 4 client.18480 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cc7aca7410 con 0x55cc7b0094a0 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.360+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b0094a0 0x55cc7b00b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.361+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b0094a0 0x55cc7b00b8d0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1654302308f82bac server_cookie=304e4a1c868eb696 in_seq=0 out_seq=0 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.361+0000 7fb2c1ffb640 10 client.18480 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.361+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/2051207098 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cc7aca7410 con 0x55cc7b0094a0 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.361+0000 7fb2c1ffb640 10 client.18480 handle_command_reply: tid=0 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.361+0000 7fb2e37ee740 1 client.18480 shutdown 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 msgr2=0x55cc7aff6590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 0x55cc7aff6590 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc09a740 tx=0x7fb2cc065ca0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 shutdown_connections 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b0094a0 0x55cc7b00b8d0 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7aff7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2c1ffb640 0 client.18480 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/2051207098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0190 0x55cc7aff6590 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 >> 172.21.15.3:0/2051207098 conn(0x55cc7afea8c0 msgr2=0x55cc7afec610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 shutdown_connections 2024-02-09T22:59:15.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 1 -- 172.21.15.3:0/2051207098 wait complete. 2024-02-09T22:59:15.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.362+0000 7fb2e37ee740 20 client.18480 trim_cache size 0 max 16384 2024-02-09T22:59:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 Processor -- start 2024-02-09T22:59:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 -- start start 2024-02-09T22:59:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff56f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff9690 0x55cc7b000510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7afefb60 con 0x55cc7aff0ab0 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7afef280 con 0x55cc7aff9690 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7b0092a0 con 0x55cc7aff0190 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55750/0 (socket says 172.21.15.3:55750) 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1849779176 learned_addr learned my addr 172.21.15.3:0/1849779176 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2da59c640 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.377+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff9690 0x55cc7b000510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1849779176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 msgr2=0x55cc7aff56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1849779176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff9690 msgr2=0x55cc7b000510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff9690 0x55cc7b000510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/1849779176 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cc7afe5b30 con 0x55cc7aff0ab0 2024-02-09T22:59:15.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6e30 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002c170 tx=0x7fb2d002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6923076a50ed16ba server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2c3fff640 1 -- 172.21.15.3:0/1849779176 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d00043a0 con 0x55cc7aff0ab0 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2c3fff640 1 -- 172.21.15.3:0/1849779176 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2d002c2b0 con 0x55cc7aff0ab0 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2c3fff640 1 -- 172.21.15.3:0/1849779176 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d004f780 con 0x55cc7aff0ab0 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1849779176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 msgr2=0x55cc7aff6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.378+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6e30 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002c170 tx=0x7fb2d002bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1849779176 shutdown_connections 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff9690 0x55cc7b000510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7aff6e30 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/1849779176 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff0190 0x55cc7aff56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1849779176 >> 172.21.15.3:0/1849779176 conn(0x55cc7afea8c0 msgr2=0x55cc7afe9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1849779176 shutdown_connections 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- 172.21.15.3:0/1849779176 wait complete. 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- start start 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 Processor -- start 2024-02-09T22:59:15.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0190 0x55cc7afff5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7afffb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7b05fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7b000260 con 0x55cc7aff0ab0 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7b05ff40 con 0x55cc7aff0190 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cc7b0600d0 con 0x55cc7aff9690 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2e37ee740 10 client.0 _ll_register_callbacks cb 0x7fb2d0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7afffb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2d9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7afffb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55754/0 (socket says 172.21.15.3:55754) 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/3779426120 learned_addr learned my addr 172.21.15.3:0/3779426120 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2dad9d640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7b05fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.379+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/3779426120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 msgr2=0x55cc7b05fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7b05fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2da59c640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0190 0x55cc7afff5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0190 msgr2=0x55cc7afff5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0190 0x55cc7afff5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2d9d9b640 1 -- 172.21.15.3:0/3779426120 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cc7aefd210 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2d9d9b640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7afffb20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002e010 tx=0x7fb2d00047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a725801c817ea5d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2c1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/3779426120 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d002c9e0 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/3779426120 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fb2d002cb40 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb2d0004030 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/3779426120 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2d0004e90 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 10 client.18486 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55cc7aef1c70 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/3779426120 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2d0004030 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/3779426120 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb2d003c040 con 0x55cc7aff0ab0 2024-02-09T22:59:15.388 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 10 client.18486 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 10 client.18486 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 20 client.18486 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b061510 0x55cc7b063930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 4 client.18486 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.380+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cc7ac937b0 con 0x55cc7b061510 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.381+0000 7fb2da59c640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b061510 0x55cc7b063930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.381+0000 7fb2da59c640 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b061510 0x55cc7b063930 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebe10ec359a0131a server_cookie=3bbf2dbf19acf871 in_seq=0 out_seq=0 2024-02-09T22:59:15.389 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.381+0000 7fb2c1ffb640 10 client.18486 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2c1ffb640 1 -- 172.21.15.3:0/3779426120 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cc7ac937b0 con 0x55cc7b061510 2024-02-09T22:59:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2c1ffb640 10 client.18486 handle_command_reply: tid=0 2024-02-09T22:59:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 client.18486 shutdown 2024-02-09T22:59:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 msgr2=0x55cc7afffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7afffb20 secure :-1 s=READY pgs=1313 cs=0 l=1 rev1=1 crypto rx=0x7fb2d002e010 tx=0x7fb2d00047d0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 shutdown_connections 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cc7b061510 0x55cc7b063930 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2c1ffb640 0 client.18486 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cc7aff9690 0x55cc7b05fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cc7aff0ab0 0x55cc7afffb20 unknown :-1 s=CLOSED pgs=1313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.382+0000 7fb2e37ee740 1 --2- 172.21.15.3:0/3779426120 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cc7aff0190 0x55cc7afff5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 >> 172.21.15.3:0/3779426120 conn(0x55cc7afea8c0 msgr2=0x55cc7aff6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 shutdown_connections 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e37ee740 1 -- 172.21.15.3:0/3779426120 wait complete. 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e37ee740 20 client.18486 trim_cache size 0 max 16384 2024-02-09T22:59:15.391 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:59:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:59:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "e736773d-dced-4ce2-b85e-0619031f84b2", 2024-02-09T22:59:15.392 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:59:15.392 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2c40a2230 msgr2=0x7fb2c40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2c40a2230 0x7fb2c40a46f0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4005c40 tx=0x7fb2d400a3c0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 msgr2=0x7fb2dc19d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc19d5c0 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc00d950 tx=0x7fb2cc00de10 comp rx=0 tx=0).stop 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 shutdown_connections 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2c40a2230 0x7fb2c40a46f0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2dc100440 0x7fb2dc19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb2dc106220 0x7fb2dc197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 --2- 172.21.15.3:0/2952927480 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2dc0ffa80 0x7fb2dc19d5c0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 >> 172.21.15.3:0/2952927480 conn(0x7fb2dc076410 msgr2=0x7fb2dc0fdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 shutdown_connections 2024-02-09T22:59:15.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.383+0000 7fb2e0b3e640 1 -- 172.21.15.3:0/2952927480 wait complete. 2024-02-09T22:59:15.397 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:59:15.397 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:59:15.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 -- 172.21.15.3:0/1159980970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4194100060 msgr2=0x7f419410e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 --2- 172.21.15.3:0/1159980970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4194100060 0x7f419410e090 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x7f418c009cd0 tx=0x7f418c02c0f0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 -- 172.21.15.3:0/1159980970 shutdown_connections 2024-02-09T22:59:15.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 --2- 172.21.15.3:0/1159980970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f4194100060 0x7f419410e090 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 --2- 172.21.15.3:0/1159980970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f41940ff6a0 0x7f41940ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 --2- 172.21.15.3:0/1159980970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4194106200 0x7f41941065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.507+0000 7f419bf63640 1 -- 172.21.15.3:0/1159980970 >> 172.21.15.3:0/1159980970 conn(0x7f4194076630 msgr2=0x7f41940fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 -- 172.21.15.3:0/1159980970 shutdown_connections 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 -- 172.21.15.3:0/1159980970 wait complete. 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 Processor -- start 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 -- start start 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 0x7f419419f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4194100060 0x7f419419fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4194106200 0x7f4194199880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4194199ed0 con 0x7f41940ff6a0 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419419a040 con 0x7f4194100060 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419bf63640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419419a200 con 0x7f4194106200 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f4199cff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 0x7f419419f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f4199cff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 0x7f419419f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55768/0 (socket says 172.21.15.3:55768) 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f4199cff640 1 -- 172.21.15.3:0/666023180 learned_addr learned my addr 172.21.15.3:0/666023180 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f41994fe640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4194100060 0x7f419419fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.508+0000 7f419a500640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4194106200 0x7f4194199880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f4199cff640 1 -- 172.21.15.3:0/666023180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4194106200 msgr2=0x7f4194199880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f4199cff640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4194106200 0x7f4194199880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f4199cff640 1 -- 172.21.15.3:0/666023180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4194100060 msgr2=0x7f419419fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f4199cff640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4194100060 0x7f419419fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f4199cff640 1 -- 172.21.15.3:0/666023180 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4190009490 con 0x7f41940ff6a0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f4199cff640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 0x7f419419f6d0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f419000ea60 tx=0x7f419000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f418affd640 1 -- 172.21.15.3:0/666023180 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419000cde0 con 0x7f41940ff6a0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f418c009900 con 0x7f41940ff6a0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f418affd640 1 -- 172.21.15.3:0/666023180 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41900035f0 con 0x7f41940ff6a0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f418affd640 1 -- 172.21.15.3:0/666023180 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4190010690 con 0x7f41940ff6a0 2024-02-09T22:59:15.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.509+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41941a6590 con 0x7f41940ff6a0 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.511+0000 7f418affd640 1 -- 172.21.15.3:0/666023180 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f4190003af0 con 0x7f41940ff6a0 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 Processor -- start 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 -- start start 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f418affd640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f41800a2230 0x7f41800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ad7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad8330 0x55b990ae08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ae0eb0 0x55b990ae3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990a92d00 con 0x55b990ae0eb0 2024-02-09T22:59:15.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990a472f0 con 0x55b990ad7980 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419ec0f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990a44820 con 0x55b990ad8330 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419a500640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f41800a2230 0x7f41800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f418affd640 1 -- 172.21.15.3:0/666023180 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4190014070 con 0x7f41940ff6a0 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f4199cff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ad7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f41994fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad8330 0x55b990ae08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f419a500640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ae0eb0 0x55b990ae3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f4199cff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ad7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:34624/0 (socket says 172.21.15.3:34624) 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f41994fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad8330 0x55b990ae08e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:46718/0 (socket says 172.21.15.3:46718) 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.512+0000 7f4199cff640 1 -- 172.21.15.3:0/1324054291 learned_addr learned my addr 172.21.15.3:0/1324054291 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f4199cff640 1 -- 172.21.15.3:0/1324054291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad8330 msgr2=0x55b990ae08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f4199cff640 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad8330 0x55b990ae08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f4199cff640 1 -- 172.21.15.3:0/1324054291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ae0eb0 msgr2=0x55b990ae3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f4199cff640 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ae0eb0 0x55b990ae3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f4199cff640 1 -- 172.21.15.3:0/1324054291 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b990a01ff0 con 0x55b990ad7980 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f419a500640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f41800a2230 0x7f41800a46f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f418c005cc0 tx=0x7f418c032e90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.522 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f4199cff640 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ad7d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4190097910 tx=0x7f419009ab40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2a06555b63c17df server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f417ffff640 1 -- 172.21.15.3:0/1324054291 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f419009af40 con 0x55b990ad7980 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f417ffff640 1 -- 172.21.15.3:0/1324054291 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f419009f090 con 0x55b990ad7980 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.513+0000 7f417ffff640 1 -- 172.21.15.3:0/1324054291 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41900975b0 con 0x55b990ad7980 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 -- 172.21.15.3:0/1324054291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 msgr2=0x55b990ad7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ad7d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4190097910 tx=0x7f419009ab40 comp rx=0 tx=0).stop 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 -- 172.21.15.3:0/1324054291 shutdown_connections 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ae0eb0 0x55b990ae3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad8330 0x55b990ae08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 --2- 172.21.15.3:0/1324054291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ad7d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 -- 172.21.15.3:0/1324054291 >> 172.21.15.3:0/1324054291 conn(0x55b990ad2050 msgr2=0x55b990ad2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 -- 172.21.15.3:0/1324054291 shutdown_connections 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 -- 172.21.15.3:0/1324054291 wait complete. 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 -- start start 2024-02-09T22:59:15.523 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.514+0000 7f419ec0f740 1 Processor -- start 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990adddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ade330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ae0eb0 0x55b990ade870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990a40000 con 0x55b990ad8330 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990713430 con 0x55b990ad7980 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9906cfe00 con 0x55b990ae0eb0 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419ec0f740 10 client.0 _ll_register_callbacks cb 0x55b990a347f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ade330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ade330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55788/0 (socket says 172.21.15.3:55788) 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 -- 172.21.15.3:0/3004686032 learned_addr learned my addr 172.21.15.3:0/3004686032 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f4199cff640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990adddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f419a500640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ae0eb0 0x55b990ade870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 -- 172.21.15.3:0/3004686032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ae0eb0 msgr2=0x55b990ade870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.524 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ae0eb0 0x55b990ade870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 -- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 msgr2=0x55b990adddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990adddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 -- 172.21.15.3:0/3004686032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41941a6590 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f41994fe640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ade330 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4184012700 tx=0x7f4184012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a1b733457e3e994 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f417d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.515+0000 7f417d7fa640 1 -- 172.21.15.3:0/3004686032 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4184017070 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f417d7fa640 1 -- 172.21.15.3:0/3004686032 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f4184004020 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b99073f7e0 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f417d7fa640 1 -- 172.21.15.3:0/3004686032 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41840041e0 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 10 client.18498 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55b990a3e680 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f417d7fa640 1 -- 172.21.15.3:0/3004686032 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b99073f7e0 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f417d7fa640 1 -- 172.21.15.3:0/3004686032 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f4184003a40 con 0x55b990ad8330 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 10 client.18498 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:15.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 10 client.18498 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 20 client.18498 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990af1430 0x55b990af3850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 4 client.18498 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b99077c720 con 0x55b990af1430 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.516+0000 7f4199cff640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990af1430 0x55b990af3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.517+0000 7f4199cff640 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990af1430 0x55b990af3850 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ae42eddc9f2ec79 server_cookie=a98bbc7983d962b9 in_seq=0 out_seq=0 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.517+0000 7f417d7fa640 10 client.18498 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.517+0000 7f417d7fa640 1 -- 172.21.15.3:0/3004686032 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b99077c720 con 0x55b990af1430 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.517+0000 7f417d7fa640 10 client.18498 handle_command_reply: tid=0 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.517+0000 7f419ec0f740 1 client.18498 shutdown 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 msgr2=0x55b990ade330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ade330 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7f4184012700 tx=0x7f4184012bc0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 shutdown_connections 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990af1430 0x55b990af3850 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ae0eb0 0x55b990ade870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ade330 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f417d7fa640 0 client.18498 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 --2- 172.21.15.3:0/3004686032 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990adddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 >> 172.21.15.3:0/3004686032 conn(0x55b990ad2050 msgr2=0x55b990ad3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 shutdown_connections 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 1 -- 172.21.15.3:0/3004686032 wait complete. 2024-02-09T22:59:15.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.518+0000 7f419ec0f740 20 client.18498 trim_cache size 0 max 16384 2024-02-09T22:59:15.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 Processor -- start 2024-02-09T22:59:15.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 -- start start 2024-02-09T22:59:15.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad2050 0x55b990ad9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad7980 0x55b990ad96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990adc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9909c4fd0 con 0x55b990ad8330 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990aedee0 con 0x55b990ad2050 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419ec0f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990ae0880 con 0x55b990ad7980 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990adc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990adc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55798/0 (socket says 172.21.15.3:55798) 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 -- 172.21.15.3:0/4279113239 learned_addr learned my addr 172.21.15.3:0/4279113239 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f4199cff640 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad2050 0x55b990ad9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f41994fe640 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad7980 0x55b990ad96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 -- 172.21.15.3:0/4279113239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad7980 msgr2=0x55b990ad96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad7980 0x55b990ad96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 -- 172.21.15.3:0/4279113239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad2050 msgr2=0x55b990ad9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad2050 0x55b990ad9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 -- 172.21.15.3:0/4279113239 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9909dab30 con 0x55b990ad8330 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.533+0000 7f419a500640 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990adc700 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f418c009cd0 tx=0x7f418c03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3966389366809350 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f417f7fe640 1 -- 172.21.15.3:0/4279113239 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c048070 con 0x55b990ad8330 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f417f7fe640 1 -- 172.21.15.3:0/4279113239 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f418c034e50 con 0x55b990ad8330 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f417f7fe640 1 -- 172.21.15.3:0/4279113239 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c0042a0 con 0x55b990ad8330 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 -- 172.21.15.3:0/4279113239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 msgr2=0x55b990adc700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990adc700 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7f418c009cd0 tx=0x7f418c03c000 comp rx=0 tx=0).stop 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 -- 172.21.15.3:0/4279113239 shutdown_connections 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990adc700 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad7980 0x55b990ad96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4279113239 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad2050 0x55b990ad9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 -- 172.21.15.3:0/4279113239 >> 172.21.15.3:0/4279113239 conn(0x55b990ad8a80 msgr2=0x55b990ad2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 -- 172.21.15.3:0/4279113239 shutdown_connections 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.534+0000 7f419ec0f740 1 -- 172.21.15.3:0/4279113239 wait complete. 2024-02-09T22:59:15.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 -- start start 2024-02-09T22:59:15.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 Processor -- start 2024-02-09T22:59:15.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad2050 0x55b990ae8340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ae66e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ae6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990ae7400 con 0x55b990ad8330 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990ae7570 con 0x55b990ad7980 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b990b479d0 con 0x55b990ad2050 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419ec0f740 10 client.0 _ll_register_callbacks cb 0x7f419000ef90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419a500640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ae6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419a500640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ae6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55812/0 (socket says 172.21.15.3:55812) 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f4199cff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad2050 0x55b990ae8340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f419a500640 1 -- 172.21.15.3:0/4132533464 learned_addr learned my addr 172.21.15.3:0/4132533464 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.535+0000 7f41994fe640 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ae66e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419a500640 1 -- 172.21.15.3:0/4132533464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad2050 msgr2=0x55b990ae8340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419a500640 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad2050 0x55b990ae8340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419a500640 1 -- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 msgr2=0x55b990ae66e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:15.543 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419a500640 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ae66e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419a500640 1 -- 172.21.15.3:0/4132533464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f419410e9e0 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419a500640 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ae6d60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f418c02c7b0 tx=0x7f418c0324e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82171846ec1b68dc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f417d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f417d7fa640 1 -- 172.21.15.3:0/4132533464 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c048040 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f417d7fa640 1 -- 172.21.15.3:0/4132533464 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f418c0326a0 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f419009f090 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f417d7fa640 1 -- 172.21.15.3:0/4132533464 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f418c032860 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 10 client.18504 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55b990a00850 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f417d7fa640 1 -- 172.21.15.3:0/4132533464 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f419009f090 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f417d7fa640 1 -- 172.21.15.3:0/4132533464 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f418c04e460 con 0x55b990ad8330 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 10 client.18504 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:15.544 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 10 client.18504 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 20 client.18504 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990ae0eb0 0x55b990b4ae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 4 client.18504 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.536+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b9907cb890 con 0x55b990ae0eb0 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f4199cff640 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990ae0eb0 0x55b990b4ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f417d7fa640 10 client.18504 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f4199cff640 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990ae0eb0 0x55b990b4ae90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8be7c64d8b7b55f4 server_cookie=27e9b07a1edc1d81 in_seq=0 out_seq=0 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f417d7fa640 1 -- 172.21.15.3:0/4132533464 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b9907cb890 con 0x55b990ae0eb0 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f417d7fa640 10 client.18504 handle_command_reply: tid=0 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 client.18504 shutdown 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 msgr2=0x55b990ae6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ae6d60 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7f418c02c7b0 tx=0x7f418c0324e0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 shutdown_connections 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55b990ae0eb0 0x55b990b4ae90 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b990ad7980 0x55b990ae66e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.545 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b990ad8330 0x55b990ae6d60 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f417d7fa640 0 client.18504 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 --2- 172.21.15.3:0/4132533464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b990ad2050 0x55b990ae8340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 >> 172.21.15.3:0/4132533464 conn(0x55b990ad8a80 msgr2=0x55b990ad9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 shutdown_connections 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 1 -- 172.21.15.3:0/4132533464 wait complete. 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.537+0000 7f419ec0f740 20 client.18504 trim_cache size 0 max 16384 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "e736773d-dced-4ce2-b85e-0619031f84b2": { 2024-02-09T22:59:15.546 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:59:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "e736773d-dced-4ce2-b85e-0619031f84b2", 2024-02-09T22:59:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:59:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:59:15.547 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:59:15.547 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f41800a2230 msgr2=0x7f41800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f41800a2230 0x7f41800a46f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f418c005cc0 tx=0x7f418c032e90 comp rx=0 tx=0).stop 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 msgr2=0x7f419419f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 0x7f419419f6d0 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x7f419000ea60 tx=0x7f419000ef20 comp rx=0 tx=0).stop 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 shutdown_connections 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f41800a2230 0x7f41800a46f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4194100060 0x7f419419fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4194106200 0x7f4194199880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 --2- 172.21.15.3:0/666023180 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f41940ff6a0 0x7f419419f6d0 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:15.548 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 >> 172.21.15.3:0/666023180 conn(0x7f4194076630 msgr2=0x7f4194103a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:15.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 shutdown_connections 2024-02-09T22:59:15.549 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:15.538+0000 7f419bf63640 1 -- 172.21.15.3:0/666023180 wait complete. 2024-02-09T22:59:15.552 INFO:tasks.cephfs.filesystem:scrub status for tag:e736773d-dced-4ce2-b85e-0619031f84b2 - {'path': '/', 'tag': 'e736773d-dced-4ce2-b85e-0619031f84b2', 'options': 'recursive,force'} 2024-02-09T22:59:15.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 228243 2.60 MB/sec execute 309 sec latency 2281.079 ms 2024-02-09T22:59:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: pgmap v249: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 866 KiB/s wr, 92 op/s 2024-02-09T22:59:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:59:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:16.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:16 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 228748 2.61 MB/sec execute 310 sec latency 770.634 ms 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: pgmap v249: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 866 KiB/s wr, 92 op/s 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:16 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: pgmap v249: 97 pgs: 97 active+clean; 94 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 610 KiB/s rd, 866 KiB/s wr, 92 op/s 2024-02-09T22:59:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:59:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:16.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:16 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:17.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 230912 2.64 MB/sec execute 311 sec latency 814.209 ms 2024-02-09T22:59:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:18 smithi003 ceph-mon[32678]: pgmap v250: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T22:59:18.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 232296 2.65 MB/sec execute 312 sec latency 79.122 ms 2024-02-09T22:59:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:18 smithi064 ceph-mon[36638]: pgmap v250: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T22:59:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:18 smithi089 ceph-mon[36665]: pgmap v250: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T22:59:19.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 232296 2.64 MB/sec execute 313 sec latency 1079.203 ms 2024-02-09T22:59:20.529 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:20 smithi089 ceph-mon[36665]: pgmap v251: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 153 op/s 2024-02-09T22:59:20.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:20 smithi003 ceph-mon[32678]: pgmap v251: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 153 op/s 2024-02-09T22:59:20.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 232296 2.63 MB/sec execute 314 sec latency 2079.315 ms 2024-02-09T22:59:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:20 smithi064 ceph-mon[36638]: pgmap v251: 97 pgs: 97 active+clean; 88 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 1.5 MiB/s wr, 153 op/s 2024-02-09T22:59:21.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 232296 2.63 MB/sec execute 315 sec latency 3079.363 ms 2024-02-09T22:59:22.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:22 smithi003 ceph-mon[32678]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-02-09T22:59:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:22 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:22 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:22 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:22 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:22.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 234423 2.66 MB/sec execute 316 sec latency 3136.745 ms 2024-02-09T22:59:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:22 smithi064 ceph-mon[36638]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-02-09T22:59:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:22 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:22 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:22 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:22 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:22 smithi089 ceph-mon[36665]: pgmap v252: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 954 KiB/s rd, 1.5 MiB/s wr, 155 op/s 2024-02-09T22:59:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:22 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:22 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:22 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:22 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 -- 172.21.15.3:0/3671112506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100380 msgr2=0x7f5a10100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 --2- 172.21.15.3:0/3671112506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100380 0x7f5a10100800 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009930 tx=0x7f5a0402bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 -- 172.21.15.3:0/3671112506 shutdown_connections 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 --2- 172.21.15.3:0/3671112506 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a10100d40 0x7f5a1010cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 --2- 172.21.15.3:0/3671112506 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100380 0x7f5a10100800 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 --2- 172.21.15.3:0/3671112506 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a101062c0 0x7f5a101066a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 -- 172.21.15.3:0/3671112506 >> 172.21.15.3:0/3671112506 conn(0x7f5a10076ef0 msgr2=0x7f5a10077300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 -- 172.21.15.3:0/3671112506 shutdown_connections 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.431+0000 7f5a15823640 1 -- 172.21.15.3:0/3671112506 wait complete. 2024-02-09T22:59:23.435 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 Processor -- start 2024-02-09T22:59:23.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 -- start start 2024-02-09T22:59:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a10100380 0x7f5a1019d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 0x7f5a1019db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a101062c0 0x7f5a10197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a10197e20 con 0x7f5a10100d40 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a10197f90 con 0x7f5a101062c0 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a15823640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a10198150 con 0x7f5a10100380 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 0x7f5a1019db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 0x7f5a1019db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49998/0 (socket says 172.21.15.3:49998) 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a0effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a10100380 0x7f5a1019d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/3426649649 learned_addr learned my addr 172.21.15.3:0/3426649649 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.432+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a101062c0 0x7f5a10197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/3426649649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a10100380 msgr2=0x7f5a1019d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a10100380 0x7f5a1019d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/3426649649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a101062c0 msgr2=0x7f5a10197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a101062c0 0x7f5a10197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/3426649649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a04009560 con 0x7f5a10100d40 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 0x7f5a1019db10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009230 tx=0x7f5a0402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:23.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a14821640 1 -- 172.21.15.3:0/3426649649 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04004350 con 0x7f5a10100d40 2024-02-09T22:59:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a14821640 1 -- 172.21.15.3:0/3426649649 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a040044b0 con 0x7f5a10100d40 2024-02-09T22:59:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a101983d0 con 0x7f5a10100d40 2024-02-09T22:59:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a101a4550 con 0x7f5a10100d40 2024-02-09T22:59:23.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.433+0000 7f5a14821640 1 -- 172.21.15.3:0/3426649649 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04003d00 con 0x7f5a10100d40 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a14821640 1 -- 172.21.15.3:0/3426649649 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5a04004df0 con 0x7f5a10100d40 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 Processor -- start 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 -- start start 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f695490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69e260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f6a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f603a00 con 0x55c46f69e7a0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f600f30 con 0x55c46f6959d0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a183fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f5fc710 con 0x55c46f6950b0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f695490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f6a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69e260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52310/0 (socket says 172.21.15.3:52310) 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f695490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:58588/0 (socket says 172.21.15.3:58588) 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a0effd640 1 -- 172.21.15.3:0/1876924509 learned_addr learned my addr 172.21.15.3:0/1876924509 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.437+0000 7f5a14821640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f59f80a22a0 0x7f59f80a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0effd640 1 -- 172.21.15.3:0/1876924509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 msgr2=0x55c46f695490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0effd640 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f695490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0effd640 1 -- 172.21.15.3:0/1876924509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 msgr2=0x55c46f6a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0effd640 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f6a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0effd640 1 -- 172.21.15.3:0/1876924509 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c46f5c13b0 con 0x55c46f6959d0 2024-02-09T22:59:23.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f695490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a14821640 1 -- 172.21.15.3:0/3426649649 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a040efed0 con 0x7f5a10100d40 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f59f80a22a0 0x7f59f80a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0effd640 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69e260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009cf0 tx=0x7f5a00011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6395f7c387245768 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f59f5ffb640 1 -- 172.21.15.3:0/1876924509 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00009ea0 con 0x55c46f6959d0 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f59f5ffb640 1 -- 172.21.15.3:0/1876924509 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a00004230 con 0x55c46f6959d0 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f59f5ffb640 1 -- 172.21.15.3:0/1876924509 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00019820 con 0x55c46f6959d0 2024-02-09T22:59:23.443 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.438+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f59f80a22a0 0x7f59f80a4760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5a08005fd0 tx=0x7f5a08004a40 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 -- 172.21.15.3:0/1876924509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 msgr2=0x55c46f69e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69e260 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009cf0 tx=0x7f5a00011810 comp rx=0 tx=0).stop 2024-02-09T22:59:23.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 -- 172.21.15.3:0/1876924509 shutdown_connections 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69e260 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f6a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1876924509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f695490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 -- 172.21.15.3:0/1876924509 >> 172.21.15.3:0/1876924509 conn(0x55c46f68f9d0 msgr2=0x55c46f68fdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 -- 172.21.15.3:0/1876924509 shutdown_connections 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 -- 172.21.15.3:0/1876924509 wait complete. 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 -- start start 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.439+0000 7f5a183fc740 1 Processor -- start 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f69b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 0x55c46f69bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f5a1b90 con 0x55c46f69e7a0 2024-02-09T22:59:23.452 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f2d0250 con 0x55c46f6950b0 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f582200 con 0x55c46f6959d0 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a183fc740 10 client.0 _ll_register_callbacks cb 0x55c46f5b85d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 0x55c46f69bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f69b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 0x55c46f69bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:58594/0 (socket says 172.21.15.3:58594) 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50002/0 (socket says 172.21.15.3:50002) 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/1031218130 learned_addr learned my addr 172.21.15.3:0/1031218130 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f69b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52320/0 (socket says 172.21.15.3:52320) 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 msgr2=0x55c46f69b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f69b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/1031218130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 msgr2=0x55c46f69c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.453 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/1031218130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a00009950 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 0x55c46f69bc90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a040b93d0 tx=0x7f5a040f35d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f838a8ea3ed4a409 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.440+0000 7f59eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f59eeffd640 1 -- 172.21.15.3:0/1031218130 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040bc360 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c46f622ee0 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f59eeffd640 1 -- 172.21.15.3:0/1031218130 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a040035f0 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f59eeffd640 1 -- 172.21.15.3:0/1031218130 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a04003e60 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 10 client.27640 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55c46f5fad90 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f59eeffd640 1 -- 172.21.15.3:0/1031218130 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c46f622ee0 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f59eeffd640 1 -- 172.21.15.3:0/1031218130 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a0403c070 con 0x55c46f6959d0 2024-02-09T22:59:23.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 10 client.27640 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 10 client.27640 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 20 client.27640 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f6ae7c0 0x55c46f6b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 4 client.27640 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c46f2fc600 con 0x55c46f6ae7c0 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.441+0000 7f5a0effd640 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f6ae7c0 0x55c46f6b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.442+0000 7f59eeffd640 10 client.27640 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.442+0000 7f5a0effd640 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f6ae7c0 0x55c46f6b0be0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9a5221b0a332d0 server_cookie=3dbec69df8c9064e in_seq=0 out_seq=0 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.442+0000 7f59eeffd640 1 -- 172.21.15.3:0/1031218130 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c46f2fc600 con 0x55c46f6ae7c0 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.442+0000 7f59eeffd640 10 client.27640 handle_command_reply: tid=0 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.442+0000 7f5a183fc740 1 client.27640 shutdown 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 msgr2=0x55c46f69bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 0x55c46f69bc90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a040b93d0 tx=0x7f5a040f35d0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 shutdown_connections 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f6ae7c0 0x55c46f6b0be0 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6959d0 0x55c46f69bc90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1031218130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f69b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f59eeffd640 0 client.27640 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 >> 172.21.15.3:0/1031218130 conn(0x55c46f68f9d0 msgr2=0x55c46f691720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 shutdown_connections 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 1 -- 172.21.15.3:0/1031218130 wait complete. 2024-02-09T22:59:23.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.443+0000 7f5a183fc740 20 client.27640 trim_cache size 0 max 16384 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 Processor -- start 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 -- start start 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f69de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69a020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f622ee0 con 0x55c46f69e7a0 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f6a0b90 con 0x55c46f6959d0 2024-02-09T22:59:23.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a183fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f690c00 con 0x55c46f6950b0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f69de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69a560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50014/0 (socket says 172.21.15.3:50014) 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 -- 172.21.15.3:0/1800402468 learned_addr learned my addr 172.21.15.3:0/1800402468 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 -- 172.21.15.3:0/1800402468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 msgr2=0x55c46f69de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f69de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 -- 172.21.15.3:0/1800402468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 msgr2=0x55c46f69a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 -- 172.21.15.3:0/1800402468 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c46f597820 con 0x55c46f69e7a0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0effd640 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f69de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.457+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69a560 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f5a08020b40 tx=0x7f5a08013030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=508ebde854750a2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f59f57fa640 1 -- 172.21.15.3:0/1800402468 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a08025040 con 0x55c46f69e7a0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f59f57fa640 1 -- 172.21.15.3:0/1800402468 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a08017070 con 0x55c46f69e7a0 2024-02-09T22:59:23.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f59f57fa640 1 -- 172.21.15.3:0/1800402468 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a080057d0 con 0x55c46f69e7a0 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 -- 172.21.15.3:0/1800402468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 msgr2=0x55c46f69a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69a560 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f5a08020b40 tx=0x7f5a08013030 comp rx=0 tx=0).stop 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 -- 172.21.15.3:0/1800402468 shutdown_connections 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6959d0 0x55c46f69a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f69e7a0 0x55c46f69a560 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 --2- 172.21.15.3:0/1800402468 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f6950b0 0x55c46f69de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 -- 172.21.15.3:0/1800402468 >> 172.21.15.3:0/1800402468 conn(0x55c46f68f9d0 msgr2=0x55c46f696a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 -- 172.21.15.3:0/1800402468 shutdown_connections 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.458+0000 7f5a183fc740 1 -- 172.21.15.3:0/1800402468 wait complete. 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 -- start start 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 Processor -- start 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f6a5520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 0x55c46f6a38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.466 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f69e7a0 0x55c46f6a3e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f6a4340 con 0x55c46f6959d0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f6a44b0 con 0x55c46f6950b0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c46f6a4670 con 0x55c46f69e7a0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a183fc740 10 client.0 _ll_register_callbacks cb 0x7f5a04009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 0x55c46f6a38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 0x55c46f6a38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:50026/0 (socket says 172.21.15.3:50026) 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/2915895778 learned_addr learned my addr 172.21.15.3:0/2915895778 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0effd640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f6a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0f7fe640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f69e7a0 0x55c46f6a3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/2915895778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f69e7a0 msgr2=0x55c46f6a3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f69e7a0 0x55c46f6a3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 msgr2=0x55c46f6a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f6a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.459+0000 7f5a0e7fc640 1 -- 172.21.15.3:0/2915895778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c46f5c13b0 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a0e7fc640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 0x55c46f6a38c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x55c46f69cd20 tx=0x7f5a04005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c245ef83f1cade46 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f59eeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f59eeffd640 1 -- 172.21.15.3:0/2915895778 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040035f0 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f59eeffd640 1 -- 172.21.15.3:0/2915895778 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f5a040d4070 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a00004230 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f59eeffd640 1 -- 172.21.15.3:0/2915895778 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a040bd100 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 10 client.18522 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x7f5a08020800 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f59eeffd640 1 -- 172.21.15.3:0/2915895778 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a00004230 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f59eeffd640 1 -- 172.21.15.3:0/2915895778 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5a0403c070 con 0x55c46f6959d0 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 10 client.18522 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 10 client.18522 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:23.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 20 client.18522 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f705bd0 0x55c46f707fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 4 client.18522 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:59:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.460+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c46f34d1b0 con 0x55c46f705bd0 2024-02-09T22:59:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.461+0000 7f5a0effd640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f705bd0 0x55c46f707fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.461+0000 7f59eeffd640 10 client.18522 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:23.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.461+0000 7f5a0effd640 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f705bd0 0x55c46f707fc0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53a92a6e3f9e2bdb server_cookie=8e79dfdb0cca23fe in_seq=0 out_seq=0 2024-02-09T22:59:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.494+0000 7f59eeffd640 1 -- 172.21.15.3:0/2915895778 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c46f34d1b0 con 0x55c46f705bd0 2024-02-09T22:59:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.494+0000 7f59eeffd640 10 client.18522 handle_command_reply: tid=0 2024-02-09T22:59:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 client.18522 shutdown 2024-02-09T22:59:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 msgr2=0x55c46f6a38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 0x55c46f6a38c0 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x55c46f69cd20 tx=0x7f5a04005fd0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.498 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 shutdown_connections 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c46f705bd0 0x55c46f707fc0 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c46f69e7a0 0x55c46f6a3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c46f6959d0 0x55c46f6a38c0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f59eeffd640 0 client.18522 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 --2- 172.21.15.3:0/2915895778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c46f6950b0 0x55c46f6a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 >> 172.21.15.3:0/2915895778 conn(0x55c46f68f9d0 msgr2=0x55c46f699fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 shutdown_connections 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 1 -- 172.21.15.3:0/2915895778 wait complete. 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a183fc740 20 client.18522 trim_cache size 0 max 16384 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:59:23.499 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:59:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:59:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:59:23.500 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f59f80a22a0 msgr2=0x7f59f80a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a15823640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f59f80a22a0 0x7f59f80a4760 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5a08005fd0 tx=0x7f5a08004a40 comp rx=0 tx=0).stop 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 msgr2=0x7f5a1019db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.495+0000 7f5a15823640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 0x7f5a1019db10 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 crypto rx=0x7f5a04009230 tx=0x7f5a0402c650 comp rx=0 tx=0).stop 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 shutdown_connections 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f59f80a22a0 0x7f59f80a4760 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5a101062c0 0x7f5a10197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5a10100d40 0x7f5a1019db10 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 --2- 172.21.15.3:0/3426649649 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5a10100380 0x7f5a1019d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 >> 172.21.15.3:0/3426649649 conn(0x7f5a10076ef0 msgr2=0x7f5a10104330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 shutdown_connections 2024-02-09T22:59:23.501 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:23.496+0000 7f5a15823640 1 -- 172.21.15.3:0/3426649649 wait complete. 2024-02-09T22:59:23.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:23 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:23.651 INFO:tasks.workunit.client.0.smithi003.stdout: 1 235709 2.67 MB/sec execute 317 sec latency 16.986 ms 2024-02-09T22:59:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:23 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:23.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:23 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:24 smithi003 ceph-mon[32678]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-02-09T22:59:24.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 235839 2.66 MB/sec execute 318 sec latency 883.165 ms 2024-02-09T22:59:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:24 smithi064 ceph-mon[36638]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-02-09T22:59:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:24 smithi089 ceph-mon[36665]: pgmap v253: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 1.3 MiB/s wr, 139 op/s 2024-02-09T22:59:25.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 236751 2.66 MB/sec execute 319 sec latency 1337.315 ms 2024-02-09T22:59:26.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:26 smithi003 ceph-mon[32678]: pgmap v254: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-02-09T22:59:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:26 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:59:26.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 238316 2.69 MB/sec execute 320 sec latency 14.486 ms 2024-02-09T22:59:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:26 smithi064 ceph-mon[36638]: pgmap v254: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-02-09T22:59:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:26 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:59:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:26 smithi089 ceph-mon[36665]: pgmap v254: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 795 KiB/s rd, 1.2 MiB/s wr, 129 op/s 2024-02-09T22:59:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:26 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T22:59:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:27 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:59:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:27 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:59:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:27 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:59:27.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 239388 2.70 MB/sec execute 321 sec latency 430.212 ms 2024-02-09T22:59:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:27 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:59:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:27 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:59:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:27 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:59:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:27 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T22:59:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:27 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T22:59:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:27 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T22:59:28.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:28 smithi064 ceph-mon[36638]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 189 op/s 2024-02-09T22:59:28.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:28 smithi003 ceph-mon[32678]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 189 op/s 2024-02-09T22:59:28.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241044 2.72 MB/sec execute 322 sec latency 827.528 ms 2024-02-09T22:59:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:28 smithi089 ceph-mon[36665]: pgmap v255: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.6 MiB/s wr, 189 op/s 2024-02-09T22:59:29.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241905 2.72 MB/sec execute 323 sec latency 305.406 ms 2024-02-09T22:59:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:30 smithi003 ceph-mon[32678]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-02-09T22:59:30.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241905 2.71 MB/sec execute 324 sec latency 1305.518 ms 2024-02-09T22:59:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:30 smithi064 ceph-mon[36638]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-02-09T22:59:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:30 smithi089 ceph-mon[36665]: pgmap v256: 97 pgs: 97 active+clean; 91 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 1.1 MiB/s wr, 133 op/s 2024-02-09T22:59:31.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241905 2.71 MB/sec execute 325 sec latency 2305.638 ms 2024-02-09T22:59:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:32 smithi003 ceph-mon[32678]: pgmap v257: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-02-09T22:59:32.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241905 2.70 MB/sec execute 326 sec latency 3305.760 ms 2024-02-09T22:59:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:32 smithi064 ceph-mon[36638]: pgmap v257: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-02-09T22:59:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:32 smithi089 ceph-mon[36665]: pgmap v257: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 214 op/s 2024-02-09T22:59:33.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241961 2.69 MB/sec execute 327 sec latency 3706.964 ms 2024-02-09T22:59:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:34 smithi003 ceph-mon[32678]: pgmap v258: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:59:34.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241961 2.68 MB/sec execute 328 sec latency 1558.674 ms 2024-02-09T22:59:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:34 smithi064 ceph-mon[36638]: pgmap v258: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:59:34.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:34 smithi089 ceph-mon[36665]: pgmap v258: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:59:35.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241961 2.67 MB/sec execute 329 sec latency 2558.766 ms 2024-02-09T22:59:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2823041794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100cf0 msgr2=0x7f9d44104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2823041794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100cf0 0x7f9d44104d30 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f9d34009930 tx=0x7f9d3402bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2823041794 shutdown_connections 2024-02-09T22:59:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2823041794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d44105410 0x7f9d4410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.599 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2823041794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100cf0 0x7f9d44104d30 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2823041794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d44100340 0x7f9d44100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2823041794 >> 172.21.15.3:0/2823041794 conn(0x7f9d44077220 msgr2=0x7f9d44077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2823041794 shutdown_connections 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.594+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2823041794 wait complete. 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 Processor -- start 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 -- start start 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 0x7f9d4419c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d44105410 0x7f9d4419c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d44196610 0x7f9d44196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44197000 con 0x7f9d44100340 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44197170 con 0x7f9d44105410 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4bb27640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d44197330 con 0x7f9d44196610 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 0x7f9d4419c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.600 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 0x7f9d4419c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55248/0 (socket says 172.21.15.3:55248) 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 -- 172.21.15.3:0/2225173951 learned_addr learned my addr 172.21.15.3:0/2225173951 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d490c2640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d44105410 0x7f9d4419c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d44196610 0x7f9d44196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 -- 172.21.15.3:0/2225173951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d44196610 msgr2=0x7f9d44196ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d44196610 0x7f9d44196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 -- 172.21.15.3:0/2225173951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d44105410 msgr2=0x7f9d4419c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d44105410 0x7f9d4419c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.595+0000 7f9d498c3640 1 -- 172.21.15.3:0/2225173951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d34009560 con 0x7f9d44100340 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.596+0000 7f9d498c3640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 0x7f9d4419c460 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000e9f0 tx=0x7f9d4000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.596+0000 7f9d3affd640 1 -- 172.21.15.3:0/2225173951 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40009700 con 0x7f9d44100340 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.596+0000 7f9d3affd640 1 -- 172.21.15.3:0/2225173951 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d4000cd80 con 0x7f9d44100340 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.596+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d441a3320 con 0x7f9d44100340 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.596+0000 7f9d3affd640 1 -- 172.21.15.3:0/2225173951 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d40018920 con 0x7f9d44100340 2024-02-09T22:59:36.601 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.596+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d441a3850 con 0x7f9d44100340 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.599+0000 7f9d3affd640 1 -- 172.21.15.3:0/2225173951 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9d40018a80 con 0x7f9d44100340 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.599+0000 7f9d4e7d0740 1 Processor -- start 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.599+0000 7f9d4e7d0740 1 -- start start 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfba490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba9d0 0x560a3cfc5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfc7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cf28a00 con 0x560a3cfba0b0 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cf25f30 con 0x560a3cfc59b0 2024-02-09T22:59:36.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cf21710 con 0x560a3cfba9d0 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4a0c4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfba490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d3affd640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d300a22a0 0x7f9d300a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4a0c4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfba490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55254/0 (socket says 172.21.15.3:55254) 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4a0c4640 1 -- 172.21.15.3:0/1410928440 learned_addr learned my addr 172.21.15.3:0/1410928440 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d300a22a0 0x7f9d300a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d3affd640 1 -- 172.21.15.3:0/2225173951 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d40014070 con 0x7f9d44100340 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d498c3640 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba9d0 0x560a3cfc5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d490c2640 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfc7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d498c3640 1 -- 172.21.15.3:0/1410928440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 msgr2=0x560a3cfc7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d498c3640 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfc7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d498c3640 1 -- 172.21.15.3:0/1410928440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 msgr2=0x560a3cfba490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d498c3640 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfba490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d498c3640 1 -- 172.21.15.3:0/1410928440 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a3cee63b0 con 0x560a3cfba9d0 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:36.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.600+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d300a22a0 0x7f9d300a4760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c013b70 tx=0x7f9d3c009950 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.601+0000 7f9d498c3640 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba9d0 0x560a3cfc5470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9d400cbbb0 tx=0x7f9d4009a870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=768ab0c7bc8f33a7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.601+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1410928440 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d400aeea0 con 0x560a3cfba9d0 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.601+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1410928440 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d400a35f0 con 0x560a3cfba9d0 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.601+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1410928440 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4009a2a0 con 0x560a3cfba9d0 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410928440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba9d0 msgr2=0x560a3cfc5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba9d0 0x560a3cfc5470 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f9d400cbbb0 tx=0x7f9d4009a870 comp rx=0 tx=0).stop 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410928440 shutdown_connections 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfc7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba9d0 0x560a3cfc5470 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410928440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfba490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410928440 >> 172.21.15.3:0/1410928440 conn(0x560a3cfb49d0 msgr2=0x560a3cfb4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410928440 shutdown_connections 2024-02-09T22:59:36.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410928440 wait complete. 2024-02-09T22:59:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 -- start start 2024-02-09T22:59:36.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 Processor -- start 2024-02-09T22:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.602+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfc29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba9d0 0x560a3cfc2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfc3420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cec6b90 con 0x560a3cfba0b0 2024-02-09T22:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cbf5250 con 0x560a3cfba9d0 2024-02-09T22:59:36.611 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cea7200 con 0x560a3cfc59b0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 10 client.0 _ll_register_callbacks cb 0x560a3cedd5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfc29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfc29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55256/0 (socket says 172.21.15.3:55256) 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 -- 172.21.15.3:0/2600813907 learned_addr learned my addr 172.21.15.3:0/2600813907 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfc3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d490c2640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba9d0 0x560a3cfc2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 -- 172.21.15.3:0/2600813907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 msgr2=0x560a3cfc3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfc3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 -- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba9d0 msgr2=0x560a3cfc2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba9d0 0x560a3cfc2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 -- 172.21.15.3:0/2600813907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d441a3850 con 0x560a3cfba0b0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d498c3640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfc29a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000e9c0 tx=0x7f9d40097220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa70538d5f3d626d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/2600813907 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d400b7070 con 0x560a3cfba0b0 2024-02-09T22:59:36.612 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/2600813907 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d400cbe30 con 0x560a3cfba0b0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560a3cf47ee0 con 0x560a3cfba0b0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/2600813907 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d400aeea0 con 0x560a3cfba0b0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 10 client.18534 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.603+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x560a3cf1fd90 con 0x560a3cfba0b0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/2600813907 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560a3cf47ee0 con 0x560a3cfba0b0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/2600813907 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9d4009a2a0 con 0x560a3cfba0b0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d4e7d0740 10 client.18534 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d4e7d0740 10 client.18534 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d4e7d0740 20 client.18534 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3cfd5820 0x560a3cfd7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d4e7d0740 4 client.18534 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560a3cc21600 con 0x560a3cfd5820 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.604+0000 7f9d490c2640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3cfd5820 0x560a3cfd7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.613 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d2d7fa640 10 client.18534 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d490c2640 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3cfd5820 0x560a3cfd7c10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e38ff84ef3b1ef65 server_cookie=8b6d9c20e4965f1a in_seq=0 out_seq=0 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/2600813907 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560a3cc21600 con 0x560a3cfd5820 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d2d7fa640 10 client.18534 handle_command_reply: tid=0 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 client.18534 shutdown 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 msgr2=0x560a3cfc29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfc29a0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000e9c0 tx=0x7f9d40097220 comp rx=0 tx=0).stop 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 shutdown_connections 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3cfd5820 0x560a3cfd7c10 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba9d0 0x560a3cfc2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfc3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d2d7fa640 0 client.18534 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/2600813907 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba0b0 0x560a3cfc29a0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.605+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 >> 172.21.15.3:0/2600813907 conn(0x560a3cfb49d0 msgr2=0x560a3cfb6720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.606+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 shutdown_connections 2024-02-09T22:59:36.614 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.606+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/2600813907 wait complete. 2024-02-09T22:59:36.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.606+0000 7f9d4e7d0740 20 client.18534 trim_cache size 0 max 16384 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 Processor -- start 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 -- start start 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba0b0 0x560a3cfc50d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfc3240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfbfc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cfb9f60 con 0x560a3cfba9d0 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cfb0340 con 0x560a3cfba0b0 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3cfb4240 con 0x560a3cfc59b0 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d490c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfc3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d490c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfc3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55260/0 (socket says 172.21.15.3:55260) 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.619+0000 7f9d490c2640 1 -- 172.21.15.3:0/1410569359 learned_addr learned my addr 172.21.15.3:0/1410569359 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:36.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d498c3640 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba0b0 0x560a3cfc50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfbfc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d490c2640 1 -- 172.21.15.3:0/1410569359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 msgr2=0x560a3cfbfc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d490c2640 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfbfc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d490c2640 1 -- 172.21.15.3:0/1410569359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba0b0 msgr2=0x560a3cfc50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d490c2640 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba0b0 0x560a3cfc50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d490c2640 1 -- 172.21.15.3:0/1410569359 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560a3cebc820 con 0x560a3cfba9d0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfbfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d490c2640 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfc3240 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000c00 tx=0x7f9d34005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c0a5046a247ae8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d2f7fe640 1 -- 172.21.15.3:0/1410569359 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340049e0 con 0x560a3cfba9d0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d2f7fe640 1 -- 172.21.15.3:0/1410569359 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d34048be0 con 0x560a3cfba9d0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.620+0000 7f9d2f7fe640 1 -- 172.21.15.3:0/1410569359 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d340483a0 con 0x560a3cfba9d0 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410569359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 msgr2=0x560a3cfc3240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfc3240 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f9d34000c00 tx=0x7f9d34005fd0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410569359 shutdown_connections 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfc59b0 0x560a3cfbfc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfc3240 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/1410569359 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfba0b0 0x560a3cfc50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410569359 >> 172.21.15.3:0/1410569359 conn(0x560a3cfb49d0 msgr2=0x560a3cfc1290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410569359 shutdown_connections 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/1410569359 wait complete. 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- start start 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 Processor -- start 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba0b0 0x560a3cfcb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfcb770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfcbcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3d02bd00 con 0x560a3cfba9d0 2024-02-09T22:59:36.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3d02be70 con 0x560a3cfc59b0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560a3d02c030 con 0x560a3cfba0b0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d4e7d0740 10 client.0 _ll_register_callbacks cb 0x7f9d4009a990 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.621+0000 7f9d490c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfcb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfcb770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55264/0 (socket says 172.21.15.3:55264) 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 -- 172.21.15.3:0/3070006705 learned_addr learned my addr 172.21.15.3:0/3070006705 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d498c3640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba0b0 0x560a3cfcb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d4a0c4640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfcbcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 -- 172.21.15.3:0/3070006705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba0b0 msgr2=0x560a3cfcb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba0b0 0x560a3cfcb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 -- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 msgr2=0x560a3cfcbcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfcbcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 -- 172.21.15.3:0/3070006705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d4410d7f0 con 0x560a3cfba9d0 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d498c3640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba0b0 0x560a3cfcb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:36.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d490c2640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfcb770 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9d3402ca90 tx=0x7f9d3402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c0d8c3e2aaef468 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d2d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/3070006705 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3402cb30 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/3070006705 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d34045040 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d400a35f0 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/3070006705 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d3402cc90 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d4e7d0740 10 client.18540 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.622+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x560a3cee26f0 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/3070006705 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d400a35f0 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/3070006705 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9d3403c040 con 0x560a3cfba9d0 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d4e7d0740 10 client.18540 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d4e7d0740 10 client.18540 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d4e7d0740 20 client.18540 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3d02d580 0x560a3d02f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:36.631 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d4e7d0740 4 client.18540 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:59:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560a3cc721b0 con 0x560a3d02d580 2024-02-09T22:59:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d498c3640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3d02d580 0x560a3d02f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d2d7fa640 10 client.18540 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:36.632 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.623+0000 7f9d498c3640 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3d02d580 0x560a3d02f9a0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cae390cf1909104 server_cookie=c78c44d075c5f2f2 in_seq=0 out_seq=0 2024-02-09T22:59:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:36 smithi003 ceph-mon[32678]: pgmap v259: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:59:36.652 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241961 2.67 MB/sec execute 330 sec latency 3558.815 ms 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d2d7fa640 1 -- 172.21.15.3:0/3070006705 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560a3cc721b0 con 0x560a3d02d580 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d2d7fa640 10 client.18540 handle_command_reply: tid=0 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 client.18540 shutdown 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 msgr2=0x560a3cfcb770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfcb770 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f9d3402ca90 tx=0x7f9d3402bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 shutdown_connections 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560a3d02d580 0x560a3d02f9a0 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560a3cfc59b0 0x560a3cfcbcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.655 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560a3cfba9d0 0x560a3cfcb770 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 --2- 172.21.15.3:0/3070006705 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560a3cfba0b0 0x560a3cfcb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d2d7fa640 0 client.18540 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 >> 172.21.15.3:0/3070006705 conn(0x560a3cfb49d0 msgr2=0x560a3cfb31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 shutdown_connections 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 1 -- 172.21.15.3:0/3070006705 wait complete. 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.652+0000 7f9d4e7d0740 20 client.18540 trim_cache size 0 max 16384 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:59:36.656 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:59:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d300a22a0 msgr2=0x7f9d300a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d300a22a0 0x7f9d300a4760 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c013b70 tx=0x7f9d3c009950 comp rx=0 tx=0).stop 2024-02-09T22:59:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 msgr2=0x7f9d4419c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:36.657 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 0x7f9d4419c460 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f9d4000e9f0 tx=0x7f9d4000eeb0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 shutdown_connections 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d300a22a0 0x7f9d300a4760 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d44105410 0x7f9d4419c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d44196610 0x7f9d44196ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 --2- 172.21.15.3:0/2225173951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d44100340 0x7f9d4419c460 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 >> 172.21.15.3:0/2225173951 conn(0x7f9d44077220 msgr2=0x7f9d440fe4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 shutdown_connections 2024-02-09T22:59:36.658 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:36.653+0000 7f9d4bb27640 1 -- 172.21.15.3:0/2225173951 wait complete. 2024-02-09T22:59:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:36 smithi064 ceph-mon[36638]: pgmap v259: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:59:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:36 smithi089 ceph-mon[36665]: pgmap v259: 97 pgs: 97 active+clean; 95 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.1 MiB/s wr, 140 op/s 2024-02-09T22:59:37.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 241961 2.66 MB/sec execute 331 sec latency 4558.863 ms 2024-02-09T22:59:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:38 smithi003 ceph-mon[32678]: pgmap v260: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-02-09T22:59:38.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242029 2.65 MB/sec execute 332 sec latency 4962.029 ms 2024-02-09T22:59:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:38 smithi064 ceph-mon[36638]: pgmap v260: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-02-09T22:59:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:38 smithi089 ceph-mon[36665]: pgmap v260: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1012 KiB/s rd, 1.4 MiB/s wr, 159 op/s 2024-02-09T22:59:39.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242029 2.64 MB/sec execute 333 sec latency 1554.295 ms 2024-02-09T22:59:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:40 smithi003 ceph-mon[32678]: pgmap v261: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-02-09T22:59:40.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242029 2.63 MB/sec execute 334 sec latency 2554.409 ms 2024-02-09T22:59:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:40 smithi064 ceph-mon[36638]: pgmap v261: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-02-09T22:59:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:40 smithi089 ceph-mon[36665]: pgmap v261: 97 pgs: 97 active+clean; 97 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 620 KiB/s rd, 1.0 MiB/s wr, 99 op/s 2024-02-09T22:59:41.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242029 2.63 MB/sec execute 335 sec latency 3554.559 ms 2024-02-09T22:59:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:42 smithi003 ceph-mon[32678]: pgmap v262: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-02-09T22:59:42.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242029 2.62 MB/sec execute 336 sec latency 4554.659 ms 2024-02-09T22:59:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:42 smithi064 ceph-mon[36638]: pgmap v262: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-02-09T22:59:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:42 smithi089 ceph-mon[36665]: pgmap v262: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 1.4 MiB/s wr, 128 op/s 2024-02-09T22:59:43.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242931 2.62 MB/sec execute 337 sec latency 4956.130 ms 2024-02-09T22:59:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:44 smithi003 ceph-mon[32678]: pgmap v263: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 672 KiB/s wr, 48 op/s 2024-02-09T22:59:44.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242931 2.62 MB/sec execute 338 sec latency 1105.898 ms 2024-02-09T22:59:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:44 smithi064 ceph-mon[36638]: pgmap v263: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 672 KiB/s wr, 48 op/s 2024-02-09T22:59:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:44 smithi089 ceph-mon[36665]: pgmap v263: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 672 KiB/s wr, 48 op/s 2024-02-09T22:59:45.553 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:59:45.553 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:59:45.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242931 2.61 MB/sec execute 339 sec latency 2105.951 ms 2024-02-09T22:59:45.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 -- 172.21.15.3:0/1276773261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260102e00 msgr2=0x7f2260103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 --2- 172.21.15.3:0/1276773261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260102e00 0x7f2260103260 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f2254009930 tx=0x7f225402bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 -- 172.21.15.3:0/1276773261 shutdown_connections 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 --2- 172.21.15.3:0/1276773261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f22601037a0 0x7f226010e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 --2- 172.21.15.3:0/1276773261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260102e00 0x7f2260103260 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 --2- 172.21.15.3:0/1276773261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2260101520 0x7f2260101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 -- 172.21.15.3:0/1276773261 >> 172.21.15.3:0/1276773261 conn(0x7f22600fccc0 msgr2=0x7f22600ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 -- 172.21.15.3:0/1276773261 shutdown_connections 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.668+0000 7f2267713640 1 -- 172.21.15.3:0/1276773261 wait complete. 2024-02-09T22:59:45.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 Processor -- start 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 -- start start 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 0x7f226019f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2260102e00 0x7f226019fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22601037a0 0x7f2260199930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2260199f80 con 0x7f2260101520 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226019a0f0 con 0x7f22601037a0 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2267713640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226019a2b0 con 0x7f2260102e00 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 0x7f226019f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 0x7f226019f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55278/0 (socket says 172.21.15.3:55278) 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 -- 172.21.15.3:0/1533896724 learned_addr learned my addr 172.21.15.3:0/1533896724 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2264cae640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2260102e00 0x7f226019fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f2265cb0640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22601037a0 0x7f2260199930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 -- 172.21.15.3:0/1533896724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2260102e00 msgr2=0x7f226019fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2260102e00 0x7f226019fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 -- 172.21.15.3:0/1533896724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22601037a0 msgr2=0x7f2260199930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22601037a0 0x7f2260199930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.669+0000 7f22654af640 1 -- 172.21.15.3:0/1533896724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2254009560 con 0x7f2260101520 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.670+0000 7f22654af640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 0x7f226019f780 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f225800e9f0 tx=0x7f225800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.670+0000 7f225e7fc640 1 -- 172.21.15.3:0/1533896724 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258009700 con 0x7f2260101520 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.670+0000 7f225e7fc640 1 -- 172.21.15.3:0/1533896724 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f225800cd80 con 0x7f2260101520 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.670+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226019a590 con 0x7f2260101520 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.670+0000 7f225e7fc640 1 -- 172.21.15.3:0/1533896724 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2258018920 con 0x7f2260101520 2024-02-09T22:59:45.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.670+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22601a6820 con 0x7f2260101520 2024-02-09T22:59:45.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.672+0000 7f225e7fc640 1 -- 172.21.15.3:0/1533896724 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f2258018a80 con 0x7f2260101520 2024-02-09T22:59:45.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 Processor -- start 2024-02-09T22:59:45.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 -- start start 2024-02-09T22:59:45.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f225e7fc640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22480a2230 0x7f22480a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576d4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576dd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e0300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5768fd00 con 0x558b576d5330 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b576442f0 con 0x558b576ddeb0 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f226a3c6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b57641820 con 0x558b576d4980 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2265cb0640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22480a2230 0x7f22480a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576dd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576dd8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55288/0 (socket says 172.21.15.3:55288) 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 -- 172.21.15.3:0/4235854982 learned_addr learned my addr 172.21.15.3:0/4235854982 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f225e7fc640 1 -- 172.21.15.3:0/1533896724 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2258014070 con 0x7f2260101520 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f22654af640 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576d4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2265cb0640 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 -- 172.21.15.3:0/4235854982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 msgr2=0x558b576d4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.685 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576d4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 -- 172.21.15.3:0/4235854982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 msgr2=0x558b576e0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.673+0000 7f2264cae640 1 -- 172.21.15.3:0/4235854982 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b575feff0 con 0x558b576d5330 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.674+0000 7f2265cb0640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22480a2230 0x7f22480a46f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2250005c40 tx=0x7f2250012b10 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.674+0000 7f2264cae640 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576dd8e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f2254009930 tx=0x7f225402c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03c8a7dec614463 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.674+0000 7f22477fe640 1 -- 172.21.15.3:0/4235854982 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254004940 con 0x558b576d5330 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.674+0000 7f22477fe640 1 -- 172.21.15.3:0/4235854982 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2254004aa0 con 0x558b576d5330 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.674+0000 7f22477fe640 1 -- 172.21.15.3:0/4235854982 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254005170 con 0x558b576d5330 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 -- 172.21.15.3:0/4235854982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 msgr2=0x558b576dd8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576dd8e0 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f2254009930 tx=0x7f225402c5d0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 -- 172.21.15.3:0/4235854982 shutdown_connections 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.686 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576dd8e0 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 --2- 172.21.15.3:0/4235854982 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576d4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 -- 172.21.15.3:0/4235854982 >> 172.21.15.3:0/4235854982 conn(0x558b576cf050 msgr2=0x558b576cf430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 -- 172.21.15.3:0/4235854982 shutdown_connections 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 -- 172.21.15.3:0/4235854982 wait complete. 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 -- start start 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.675+0000 7f226a3c6740 1 Processor -- start 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576dad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576db290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b5763d000 con 0x558b576d5330 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b57310430 con 0x558b576ddeb0 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b572cce00 con 0x558b576d4980 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 10 client.0 _ll_register_callbacks cb 0x558b576317f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:45.687 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576db290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576db290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55292/0 (socket says 172.21.15.3:55292) 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 -- 172.21.15.3:0/2663764643 learned_addr learned my addr 172.21.15.3:0/2663764643 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f22654af640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576dad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2265cb0640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 -- 172.21.15.3:0/2663764643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 msgr2=0x558b576dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576dad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 -- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 msgr2=0x558b576db7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 -- 172.21.15.3:0/2663764643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226010eaa0 con 0x558b576d5330 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2264cae640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576db290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f2254005ab0 tx=0x7f225402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dae197e6b5f48ec8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2244ff9640 1 -- 172.21.15.3:0/2663764643 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540050f0 con 0x558b576d5330 2024-02-09T22:59:45.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f2244ff9640 1 -- 172.21.15.3:0/2663764643 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f2254005250 con 0x558b576d5330 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.676+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558b5733c7e0 con 0x558b576d5330 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f2244ff9640 1 -- 172.21.15.3:0/2663764643 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2254005450 con 0x558b576d5330 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 10 client.18552 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x558b5763b680 con 0x558b576d5330 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f2244ff9640 1 -- 172.21.15.3:0/2663764643 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558b5733c7e0 con 0x558b576d5330 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f2244ff9640 1 -- 172.21.15.3:0/2663764643 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f225403c040 con 0x558b576d5330 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 10 client.18552 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 10 client.18552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 20 client.18552 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b576eded0 0x558b576f02f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 4 client.18552 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558b57379720 con 0x558b576eded0 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.677+0000 7f22654af640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b576eded0 0x558b576f02f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.678+0000 7f22654af640 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b576eded0 0x558b576f02f0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b2e856de9bba265 server_cookie=bf8bd6966d3ddcac in_seq=0 out_seq=0 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.678+0000 7f2244ff9640 10 client.18552 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.678+0000 7f2244ff9640 1 -- 172.21.15.3:0/2663764643 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558b57379720 con 0x558b576eded0 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.678+0000 7f2244ff9640 10 client.18552 handle_command_reply: tid=0 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 client.18552 shutdown 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 msgr2=0x558b576db290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576db290 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7f2254005ab0 tx=0x7f225402bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 shutdown_connections 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b576eded0 0x558b576f02f0 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d5330 0x558b576db290 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f2244ff9640 0 client.18552 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 --2- 172.21.15.3:0/2663764643 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d4980 0x558b576dad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 >> 172.21.15.3:0/2663764643 conn(0x558b576cf050 msgr2=0x558b576d0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 shutdown_connections 2024-02-09T22:59:45.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 1 -- 172.21.15.3:0/2663764643 wait complete. 2024-02-09T22:59:45.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.679+0000 7f226a3c6740 20 client.18552 trim_cache size 0 max 16384 2024-02-09T22:59:45.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 Processor -- start 2024-02-09T22:59:45.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 -- start start 2024-02-09T22:59:45.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576dd480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576d9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.704 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b575c1fd0 con 0x558b576d4980 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b576cfe50 con 0x558b576ddeb0 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f226a3c6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b576ca980 con 0x558b576d5330 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f22654af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576dd480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f22654af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576dd480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55304/0 (socket says 172.21.15.3:55304) 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.694+0000 7f22654af640 1 -- 172.21.15.3:0/30266895 learned_addr learned my addr 172.21.15.3:0/30266895 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2264cae640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576d9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2265cb0640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f22654af640 1 -- 172.21.15.3:0/30266895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 msgr2=0x558b576d9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f22654af640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576d9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f22654af640 1 -- 172.21.15.3:0/30266895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 msgr2=0x558b576db5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f22654af640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f22654af640 1 -- 172.21.15.3:0/30266895 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558b575d7b30 con 0x558b576d4980 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2264cae640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576d9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2265cb0640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f22654af640 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576dd480 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f225800c6a0 tx=0x7f2258096b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171b869ec3d8a051 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2246ffd640 1 -- 172.21.15.3:0/30266895 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225809f0d0 con 0x558b576d4980 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2246ffd640 1 -- 172.21.15.3:0/30266895 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22580a3720 con 0x558b576d4980 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f2246ffd640 1 -- 172.21.15.3:0/30266895 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225806b290 con 0x558b576d4980 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f226a3c6740 1 -- 172.21.15.3:0/30266895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 msgr2=0x558b576dd480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.695+0000 7f226a3c6740 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576dd480 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f225800c6a0 tx=0x7f2258096b80 comp rx=0 tx=0).stop 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- 172.21.15.3:0/30266895 shutdown_connections 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576db5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576d9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 --2- 172.21.15.3:0/30266895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576dd480 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- 172.21.15.3:0/30266895 >> 172.21.15.3:0/30266895 conn(0x558b576cf050 msgr2=0x558b576d9620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- 172.21.15.3:0/30266895 shutdown_connections 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- 172.21.15.3:0/30266895 wait complete. 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- start start 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 Processor -- start 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576e3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576e3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b577444c0 con 0x558b576d4980 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b57744630 con 0x558b576ddeb0 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558b577447f0 con 0x558b576d5330 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f226a3c6740 10 client.0 _ll_register_callbacks cb 0x7f2258005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f22654af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576e3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f22654af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576e3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55314/0 (socket says 172.21.15.3:55314) 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f22654af640 1 -- 172.21.15.3:0/842382718 learned_addr learned my addr 172.21.15.3:0/842382718 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.696+0000 7f2264cae640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576e3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2265cb0640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f22654af640 1 -- 172.21.15.3:0/842382718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 msgr2=0x558b576e3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f22654af640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576e3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f22654af640 1 -- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 msgr2=0x558b576e4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f22654af640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e4530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f22654af640 1 -- 172.21.15.3:0/842382718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22601a6820 con 0x558b576d4980 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2264cae640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f22654af640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576e3ab0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f22580973e0 tx=0x7f2258099280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cf141d5526eae26 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2244ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2244ff9640 1 -- 172.21.15.3:0/842382718 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22580cbf60 con 0x558b576d4980 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2244ff9640 1 -- 172.21.15.3:0/842382718 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f22580cc0c0 con 0x558b576d4980 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2254004aa0 con 0x558b576d4980 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2244ff9640 1 -- 172.21.15.3:0/842382718 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f225806b290 con 0x558b576d4980 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f226a3c6740 10 client.18558 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:45.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x558b575fd850 con 0x558b576d4980 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2244ff9640 1 -- 172.21.15.3:0/842382718 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2254004aa0 con 0x558b576d4980 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.697+0000 7f2244ff9640 1 -- 172.21.15.3:0/842382718 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f225809d070 con 0x558b576d4980 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 10 client.18558 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 10 client.18558 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 20 client.18558 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b57745db0 0x558b577481d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 4 client.18558 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558b573c8890 con 0x558b57745db0 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f2264cae640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b57745db0 0x558b577481d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f2244ff9640 10 client.18558 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:45.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f2264cae640 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b57745db0 0x558b577481d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a639f169b351e5f server_cookie=fae5f4793296ab1e in_seq=0 out_seq=0 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f2244ff9640 1 -- 172.21.15.3:0/842382718 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x558b573c8890 con 0x558b57745db0 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f2244ff9640 10 client.18558 handle_command_reply: tid=0 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.698+0000 7f226a3c6740 1 client.18558 shutdown 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 msgr2=0x558b576e3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576e3ab0 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f22580973e0 tx=0x7f2258099280 comp rx=0 tx=0).stop 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 shutdown_connections 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558b57745db0 0x558b577481d0 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558b576ddeb0 0x558b576e4530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558b576d5330 0x558b576e3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 --2- 172.21.15.3:0/842382718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558b576d4980 0x558b576e3ab0 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2244ff9640 0 client.18558 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 >> 172.21.15.3:0/842382718 conn(0x558b576cf050 msgr2=0x558b576cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 shutdown_connections 2024-02-09T22:59:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 1 -- 172.21.15.3:0/842382718 wait complete. 2024-02-09T22:59:45.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f226a3c6740 20 client.18558 trim_cache size 0 max 16384 2024-02-09T22:59:45.712 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:59:45.712 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T22:59:45.712 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T22:59:45.712 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22480a2230 msgr2=0x7f22480a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22480a2230 0x7f22480a46f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f2250005c40 tx=0x7f2250012b10 comp rx=0 tx=0).stop 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 msgr2=0x7f226019f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 0x7f226019f780 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f225800e9f0 tx=0x7f225800eeb0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 shutdown_connections 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22480a2230 0x7f22480a46f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f22601037a0 0x7f2260199930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2260102e00 0x7f226019fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 --2- 172.21.15.3:0/1533896724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2260101520 0x7f226019f780 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 >> 172.21.15.3:0/1533896724 conn(0x7f22600fccc0 msgr2=0x7f226010ca50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 shutdown_connections 2024-02-09T22:59:45.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.699+0000 7f2267713640 1 -- 172.21.15.3:0/1533896724 wait complete. 2024-02-09T22:59:45.715 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T22:59:45.715 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 -- 172.21.15.3:0/329895296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 msgr2=0x7f368c101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 --2- 172.21.15.3:0/329895296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c101810 secure :-1 s=READY pgs=1334 cs=0 l=1 rev1=1 crypto rx=0x7f367c009930 tx=0x7f367c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 -- 172.21.15.3:0/329895296 shutdown_connections 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 --2- 172.21.15.3:0/329895296 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f368c101de0 0x7f368c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 --2- 172.21.15.3:0/329895296 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 0x7f368c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 --2- 172.21.15.3:0/329895296 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c101810 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 -- 172.21.15.3:0/329895296 >> 172.21.15.3:0/329895296 conn(0x7f368c0fccd0 msgr2=0x7f368c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:45.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 -- 172.21.15.3:0/329895296 shutdown_connections 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.815+0000 7f36921f4640 1 -- 172.21.15.3:0/329895296 wait complete. 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 Processor -- start 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 -- start start 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c19d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 0x7f368c19dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f368c197690 0x7f368c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c198160 con 0x7f368c101430 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c1982d0 con 0x7f368c197690 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f36921f4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c198490 con 0x7f368c106500 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f368b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c19d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f368b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c19d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55334/0 (socket says 172.21.15.3:55334) 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.816+0000 7f368b7fe640 1 -- 172.21.15.3:0/4012900985 learned_addr learned my addr 172.21.15.3:0/4012900985 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f3682ffd640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 0x7f368c19dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368b7fe640 1 -- 172.21.15.3:0/4012900985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 msgr2=0x7f368c19dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368b7fe640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 0x7f368c19dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368bfff640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f368c197690 0x7f368c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368b7fe640 1 -- 172.21.15.3:0/4012900985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f368c197690 msgr2=0x7f368c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368b7fe640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f368c197690 0x7f368c197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368b7fe640 1 -- 172.21.15.3:0/4012900985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c009560 con 0x7f368c101430 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f3682ffd640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 0x7f368c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f368b7fe640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c19d580 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f367c009230 tx=0x7f367c004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c004440 con 0x7f368c101430 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c1a4380 con 0x7f368c101430 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f367c0052d0 con 0x7f368c101430 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367c003d70 con 0x7f368c101430 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.817+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c1a4860 con 0x7f368c101430 2024-02-09T22:59:45.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.818+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368c102900 con 0x7f368c101430 2024-02-09T22:59:45.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.819+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f367c005440 con 0x7f368c101430 2024-02-09T22:59:45.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.820+0000 7f36897fa640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36780a2230 0x7f36780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:45.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.820+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f367c0f0970 con 0x7f368c101430 2024-02-09T22:59:45.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.820+0000 7f3682ffd640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36780a2230 0x7f36780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:45.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.820+0000 7f3682ffd640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36780a2230 0x7f36780a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f368c198e90 tx=0x7f3674005d90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:45.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:45.823+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f367c0b9740 con 0x7f368c101430 2024-02-09T22:59:46.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.042+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f368c109730 con 0x7f368c101430 2024-02-09T22:59:46.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.042+0000 7f36897fa640 1 -- 172.21.15.3:0/4012900985 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f367c047e90 con 0x7f368c101430 2024-02-09T22:59:46.045 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T22:59:46.045 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:59:11.242420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T22:59:46.046 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 47 2024-02-09T22:59:46.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36780a2230 msgr2=0x7f36780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36780a2230 0x7f36780a46f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f368c198e90 tx=0x7f3674005d90 comp rx=0 tx=0).stop 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 msgr2=0x7f368c19d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c19d580 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 crypto rx=0x7f367c009230 tx=0x7f367c004040 comp rx=0 tx=0).stop 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 shutdown_connections 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f36780a2230 0x7f36780a46f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f368c197690 0x7f368c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f368c106500 0x7f368c19dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 --2- 172.21.15.3:0/4012900985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f368c101430 0x7f368c19d580 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.044+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 >> 172.21.15.3:0/4012900985 conn(0x7f368c0fccd0 msgr2=0x7f368c0fefe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.045+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 shutdown_connections 2024-02-09T22:59:46.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.045+0000 7f36921f4640 1 -- 172.21.15.3:0/4012900985 wait complete. 2024-02-09T22:59:46.058 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:46.058 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T22:59:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 -- 172.21.15.3:0/3249801016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b8100330 msgr2=0x7f60b8100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/3249801016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b8100330 0x7f60b8100710 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009930 tx=0x7f60b402bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 -- 172.21.15.3:0/3249801016 shutdown_connections 2024-02-09T22:59:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/3249801016 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100ce0 0x7f60b8104d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/3249801016 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 0x7f60b810cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/3249801016 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b8100330 0x7f60b8100710 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 -- 172.21.15.3:0/3249801016 >> 172.21.15.3:0/3249801016 conn(0x7f60b80771d0 msgr2=0x7f60b80775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 -- 172.21.15.3:0/3249801016 shutdown_connections 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 -- 172.21.15.3:0/3249801016 wait complete. 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 Processor -- start 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.160+0000 7f60bfe2a640 1 -- start start 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100330 0x7f60b819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 0x7f60b819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 0x7f60b8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b81981b0 con 0x7f60b81976e0 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8198320 con 0x7f60b8100330 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b81984e0 con 0x7f60b81053f0 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 0x7f60b8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 0x7f60b8197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55338/0 (socket says 172.21.15.3:55338) 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 -- 172.21.15.3:0/1545294350 learned_addr learned my addr 172.21.15.3:0/1545294350 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 0x7f60b819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100330 0x7f60b819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 -- 172.21.15.3:0/1545294350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 msgr2=0x7f60b819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 0x7f60b819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 -- 172.21.15.3:0/1545294350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100330 msgr2=0x7f60b819d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100330 0x7f60b819d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 -- 172.21.15.3:0/1545294350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b4009560 con 0x7f60b81976e0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100330 0x7f60b819d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 0x7f60b819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60be3c7640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 0x7f60b8197b60 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f60b000eaa0 tx=0x7f60b000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60aeffd640 1 -- 172.21.15.3:0/1545294350 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b000cdb0 con 0x7f60b81976e0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60aeffd640 1 -- 172.21.15.3:0/1545294350 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60b0005020 con 0x7f60b81976e0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b81a4480 con 0x7f60b81976e0 2024-02-09T22:59:46.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60aeffd640 1 -- 172.21.15.3:0/1545294350 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b00116b0 con 0x7f60b81976e0 2024-02-09T22:59:46.168 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.161+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b81a4960 con 0x7f60b81976e0 2024-02-09T22:59:46.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.163+0000 7f60aeffd640 1 -- 172.21.15.3:0/1545294350 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f60b0005520 con 0x7f60b81976e0 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 Processor -- start 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 -- start start 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae95ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae967990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae967f60 0x55b4ae96a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60aeffd640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f60a40a2230 0x7f60a40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae91a020 con 0x55b4ae95ea30 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae8cd320 con 0x55b4ae95f3e0 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60c29fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae8ca850 con 0x55b4ae967f60 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60bd3c5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae95ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60be3c7640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f60a40a2230 0x7f60a40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.164+0000 7f60bd3c5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae95ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55344/0 (socket says 172.21.15.3:55344) 2024-02-09T22:59:46.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 -- 172.21.15.3:0/768254624 learned_addr learned my addr 172.21.15.3:0/768254624 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60aeffd640 1 -- 172.21.15.3:0/1545294350 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f60b00ce300 con 0x7f60b81976e0 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae967990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae967f60 0x55b4ae96a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 -- 172.21.15.3:0/768254624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae967f60 msgr2=0x55b4ae96a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae967f60 0x55b4ae96a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 -- 172.21.15.3:0/768254624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 msgr2=0x55b4ae967990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae967990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 -- 172.21.15.3:0/768254624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4ae888210 con 0x55b4ae95ea30 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60be3c7640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f60a40a2230 0x7f60a40a46f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60b0097690 tx=0x7f60b009e040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae95ee10 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f60a800f390 tx=0x7f60a800b500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=722e7ddecbdb3cd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60a3fff640 1 -- 172.21.15.3:0/768254624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8012860 con 0x55b4ae95ea30 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60a3fff640 1 -- 172.21.15.3:0/768254624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60a8018070 con 0x55b4ae95ea30 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.165+0000 7f60a3fff640 1 -- 172.21.15.3:0/768254624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8011cf0 con 0x55b4ae95ea30 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 -- 172.21.15.3:0/768254624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 msgr2=0x55b4ae95ee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae95ee10 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7f60a800f390 tx=0x7f60a800b500 comp rx=0 tx=0).stop 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 -- 172.21.15.3:0/768254624 shutdown_connections 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae967990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae967f60 0x55b4ae96a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 --2- 172.21.15.3:0/768254624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae95ee10 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 -- 172.21.15.3:0/768254624 >> 172.21.15.3:0/768254624 conn(0x55b4ae959100 msgr2=0x55b4ae9594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 -- 172.21.15.3:0/768254624 shutdown_connections 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 -- 172.21.15.3:0/768254624 wait complete. 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 -- start start 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.166+0000 7f60c29fe740 1 Processor -- start 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae95ea30 0x55b4ae964eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae9653f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 0x55b4ae965930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae8c6030 con 0x55b4ae967f60 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae599430 con 0x55b4ae95f3e0 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae555e00 con 0x55b4ae95ea30 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60c29fe740 10 client.0 _ll_register_callbacks cb 0x55b4ae8bb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 0x55b4ae965930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 0x55b4ae965930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55356/0 (socket says 172.21.15.3:55356) 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 -- 172.21.15.3:0/1969924323 learned_addr learned my addr 172.21.15.3:0/1969924323 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae9653f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae95ea30 0x55b4ae964eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 -- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae95ea30 msgr2=0x55b4ae964eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae95ea30 0x55b4ae964eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 -- 172.21.15.3:0/1969924323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 msgr2=0x55b4ae9653f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae9653f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 -- 172.21.15.3:0/1969924323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a800bc30 con 0x55b4ae967f60 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 0x55b4ae965930 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f60a80058c0 tx=0x7f60a8016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cda806c2713ca9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.167+0000 7f60a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60a17fa640 1 -- 172.21.15.3:0/1969924323 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8005e70 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b4ae5c57e0 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60a17fa640 1 -- 172.21.15.3:0/1969924323 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f60a8012a70 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60a17fa640 1 -- 172.21.15.3:0/1969924323 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60a8012c30 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 10 client.18576 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55b4ae8c46b0 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60a17fa640 1 -- 172.21.15.3:0/1969924323 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b4ae5c57e0 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60a17fa640 1 -- 172.21.15.3:0/1969924323 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f60a802a430 con 0x55b4ae967f60 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 10 client.18576 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 10 client.18576 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 20 client.18576 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae9782f0 0x55b4ae97a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 4 client.18576 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b4ae602720 con 0x55b4ae9782f0 2024-02-09T22:59:46.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.168+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae9782f0 0x55b4ae97a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.169+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae9782f0 0x55b4ae97a710 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9d56f391f9c88b23 server_cookie=f90622c91f96fc52 in_seq=0 out_seq=0 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.169+0000 7f60a17fa640 10 client.18576 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60a17fa640 1 -- 172.21.15.3:0/1969924323 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b4ae602720 con 0x55b4ae9782f0 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60a17fa640 10 client.18576 handle_command_reply: tid=0 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 client.18576 shutdown 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 msgr2=0x55b4ae965930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 0x55b4ae965930 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7f60a80058c0 tx=0x7f60a8016040 comp rx=0 tx=0).stop 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 shutdown_connections 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae9782f0 0x55b4ae97a710 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae9653f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae967f60 0x55b4ae965930 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60a17fa640 0 client.18576 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 --2- 172.21.15.3:0/1969924323 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae95ea30 0x55b4ae964eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 >> 172.21.15.3:0/1969924323 conn(0x55b4ae959100 msgr2=0x55b4ae95ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 shutdown_connections 2024-02-09T22:59:46.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 1 -- 172.21.15.3:0/1969924323 wait complete. 2024-02-09T22:59:46.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.170+0000 7f60c29fe740 20 client.18576 trim_cache size 0 max 16384 2024-02-09T22:59:46.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 Processor -- start 2024-02-09T22:59:46.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 -- start start 2024-02-09T22:59:46.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 0x55b4ae9675e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae963e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae962130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae84b130 con 0x55b4ae95ea30 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae96aa70 con 0x55b4ae95f3e0 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60c29fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae960ac0 con 0x55b4ae959100 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae963e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae963e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55358/0 (socket says 172.21.15.3:55358) 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 -- 172.21.15.3:0/577752687 learned_addr learned my addr 172.21.15.3:0/577752687 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae962130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 -- 172.21.15.3:0/577752687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 msgr2=0x55b4ae9675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 0x55b4ae9675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 -- 172.21.15.3:0/577752687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 msgr2=0x55b4ae962130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae962130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bdbc6640 1 -- 172.21.15.3:0/577752687 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b4ae860c90 con 0x55b4ae95ea30 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.182+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae962130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae963e90 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f60b402c210 tx=0x7f60b402e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3361655531028192 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60a37fe640 1 -- 172.21.15.3:0/577752687 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b402cea0 con 0x55b4ae95ea30 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60a37fe640 1 -- 172.21.15.3:0/577752687 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60b4048de0 con 0x55b4ae95ea30 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60a37fe640 1 -- 172.21.15.3:0/577752687 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b4048690 con 0x55b4ae95ea30 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 -- 172.21.15.3:0/577752687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 msgr2=0x55b4ae963e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae963e90 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7f60b402c210 tx=0x7f60b402e040 comp rx=0 tx=0).stop 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 -- 172.21.15.3:0/577752687 shutdown_connections 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae962130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae963e90 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 --2- 172.21.15.3:0/577752687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 0x55b4ae9675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 -- 172.21.15.3:0/577752687 >> 172.21.15.3:0/577752687 conn(0x55b4ae95fb30 msgr2=0x55b4ae963780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 -- 172.21.15.3:0/577752687 shutdown_connections 2024-02-09T22:59:46.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.183+0000 7f60c29fe740 1 -- 172.21.15.3:0/577752687 wait complete. 2024-02-09T22:59:46.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 -- start start 2024-02-09T22:59:46.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 Processor -- start 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 0x55b4ae96da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae96dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae96e510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae9d2bd0 con 0x55b4ae95ea30 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae9d2d10 con 0x55b4ae95f3e0 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b4ae9d2ed0 con 0x55b4ae959100 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60c29fe740 10 client.0 _ll_register_callbacks cb 0x7f60a800b3b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae96dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae96dfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55360/0 (socket says 172.21.15.3:55360) 2024-02-09T22:59:46.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 -- 172.21.15.3:0/3122604148 learned_addr learned my addr 172.21.15.3:0/3122604148 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 -- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 msgr2=0x55b4ae96da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 0x55b4ae96da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 -- 172.21.15.3:0/3122604148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 msgr2=0x55b4ae96e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae96e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.184+0000 7f60bdbc6640 1 -- 172.21.15.3:0/3122604148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b810d7d0 con 0x55b4ae95ea30 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60bdbc6640 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae96dfd0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f60b402c170 tx=0x7f60b4048cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67a675baa45e29f9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60a17fa640 1 -- 172.21.15.3:0/3122604148 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b402cea0 con 0x55b4ae95ea30 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60a17fa640 1 -- 172.21.15.3:0/3122604148 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f60b4057030 con 0x55b4ae95ea30 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60a8018070 con 0x55b4ae95ea30 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60a17fa640 1 -- 172.21.15.3:0/3122604148 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60b402c7e0 con 0x55b4ae95ea30 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 10 client.18582 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55b4ae886a70 con 0x55b4ae95ea30 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60a17fa640 1 -- 172.21.15.3:0/3122604148 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60a8018070 con 0x55b4ae95ea30 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60a17fa640 1 -- 172.21.15.3:0/3122604148 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f60b403c040 con 0x55b4ae95ea30 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 10 client.18582 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 10 client.18582 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 20 client.18582 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae967f60 0x55b4ae9d6560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 4 client.18582 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.185+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b4ae651890 con 0x55b4ae967f60 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae967f60 0x55b4ae9d6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60bd3c5640 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae967f60 0x55b4ae9d6560 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=76e403810bb5c87c server_cookie=424aa1f7da75c2a3 in_seq=0 out_seq=0 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60a17fa640 10 client.18582 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60a17fa640 1 -- 172.21.15.3:0/3122604148 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b4ae651890 con 0x55b4ae967f60 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60a17fa640 10 client.18582 handle_command_reply: tid=0 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 client.18582 shutdown 2024-02-09T22:59:46.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 msgr2=0x55b4ae96dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae96dfd0 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f60b402c170 tx=0x7f60b4048cd0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 shutdown_connections 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55b4ae967f60 0x55b4ae9d6560 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b4ae95f3e0 0x55b4ae96e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b4ae95ea30 0x55b4ae96dfd0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 --2- 172.21.15.3:0/3122604148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b4ae959100 0x55b4ae96da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60a17fa640 0 client.18582 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 >> 172.21.15.3:0/3122604148 conn(0x55b4ae95fb30 msgr2=0x55b4ae964010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 shutdown_connections 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.186+0000 7f60c29fe740 1 -- 172.21.15.3:0/3122604148 wait complete. 2024-02-09T22:59:46.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60c29fe740 20 client.18582 trim_cache size 0 max 16384 2024-02-09T22:59:46.196 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:46.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f60a40a2230 msgr2=0x7f60a40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f60a40a2230 0x7f60a40a46f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f60b0097690 tx=0x7f60b009e040 comp rx=0 tx=0).stop 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 msgr2=0x7f60b8197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 0x7f60b8197b60 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f60b000eaa0 tx=0x7f60b000ef60 comp rx=0 tx=0).stop 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 shutdown_connections 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f60a40a2230 0x7f60a40a46f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f60b81976e0 0x7f60b8197b60 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f60b81053f0 0x7f60b819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 --2- 172.21.15.3:0/1545294350 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f60b8100330 0x7f60b819d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 >> 172.21.15.3:0/1545294350 conn(0x7f60b80771d0 msgr2=0x7f60b8069c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 shutdown_connections 2024-02-09T22:59:46.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.187+0000 7f60bfe2a640 1 -- 172.21.15.3:0/1545294350 wait complete. 2024-02-09T22:59:46.200 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:46.200 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 -- 172.21.15.3:0/3099451739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c101490 msgr2=0x7fcf9c104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3099451739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c101490 0x7fcf9c104d40 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009930 tx=0x7fcf9002bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 -- 172.21.15.3:0/3099451739 shutdown_connections 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3099451739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf9c105420 0x7fcf9c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3099451739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c101490 0x7fcf9c104d40 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3099451739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf9c100ae0 0x7fcf9c100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 -- 172.21.15.3:0/3099451739 >> 172.21.15.3:0/3099451739 conn(0x7fcf9c076630 msgr2=0x7fcf9c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 -- 172.21.15.3:0/3099451739 shutdown_connections 2024-02-09T22:59:46.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 -- 172.21.15.3:0/3099451739 wait complete. 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.304+0000 7fcfa2264640 1 Processor -- start 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 -- start start 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf9c100ae0 0x7fcf9c19d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 0x7fcf9c19dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf9c197760 0x7fcf9c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c198230 con 0x7fcf9c105420 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c1983a0 con 0x7fcf9c197760 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa2264640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf9c198560 con 0x7fcf9c100ae0 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 0x7fcf9c19dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 0x7fcf9c19dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55378/0 (socket says 172.21.15.3:55378) 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/3405509158 learned_addr learned my addr 172.21.15.3:0/3405509158 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf9c100ae0 0x7fcf9c19d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/3405509158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf9c100ae0 msgr2=0x7fcf9c19d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf9c100ae0 0x7fcf9c19d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/3405509158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf9c197760 msgr2=0x7fcf9c197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcfa0801640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf9c197760 0x7fcf9c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf9c197760 0x7fcf9c197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/3405509158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009560 con 0x7fcf9c105420 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.305+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 0x7fcf9c19dbe0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcf90000c00 tx=0x7fcf9002c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.306+0000 7fcf997fa640 1 -- 172.21.15.3:0/3405509158 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900043d0 con 0x7fcf9c105420 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.306+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9c1a44a0 con 0x7fcf9c105420 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.306+0000 7fcf997fa640 1 -- 172.21.15.3:0/3405509158 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf900052d0 con 0x7fcf9c105420 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.306+0000 7fcf997fa640 1 -- 172.21.15.3:0/3405509158 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90003d40 con 0x7fcf9c105420 2024-02-09T22:59:46.311 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.306+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c1a4980 con 0x7fcf9c105420 2024-02-09T22:59:46.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.308+0000 7fcf997fa640 1 -- 172.21.15.3:0/3405509158 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fcf90005440 con 0x7fcf9c105420 2024-02-09T22:59:46.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 Processor -- start 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 -- start start 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192436cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924372c0 0x55d19243fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d192440820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1923f2020 con 0x55d192436910 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1923a5320 con 0x55d1924372c0 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d1923a2850 con 0x55d1924403c0 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192436cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192436cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55380/0 (socket says 172.21.15.3:55380) 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 -- 172.21.15.3:0/1632903495 learned_addr learned my addr 172.21.15.3:0/1632903495 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcf997fa640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf840a2230 0x7fcf840a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d192440820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924372c0 0x55d19243fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 -- 172.21.15.3:0/1632903495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 msgr2=0x55d192440820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d192440820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 -- 172.21.15.3:0/1632903495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924372c0 msgr2=0x55d19243fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924372c0 0x55d19243fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 -- 172.21.15.3:0/1632903495 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d192360210 con 0x55d192436910 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d192440820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924372c0 0x55d19243fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcf997fa640 1 -- 172.21.15.3:0/3405509158 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf900f0210 con 0x7fcf9c105420 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.309+0000 7fcfa0801640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf840a2230 0x7fcf840a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa0801640 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192436cf0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007f10 tx=0x7fcf9400c560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9c2d1ca63cea2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcf827fc640 1 -- 172.21.15.3:0/1632903495 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9401eb20 con 0x55d192436910 2024-02-09T22:59:46.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcf827fc640 1 -- 172.21.15.3:0/1632903495 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf94022070 con 0x55d192436910 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcf827fc640 1 -- 172.21.15.3:0/1632903495 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9401dc80 con 0x55d192436910 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa0801640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf840a2230 0x7fcf840a46f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcf9401d000 tx=0x7fcf9401e000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1632903495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 msgr2=0x55d192436cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192436cf0 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fcf94007f10 tx=0x7fcf9400c560 comp rx=0 tx=0).stop 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1632903495 shutdown_connections 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924372c0 0x55d19243fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d192440820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1632903495 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192436cf0 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1632903495 >> 172.21.15.3:0/1632903495 conn(0x55d192431100 msgr2=0x55d1924314e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.310+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1632903495 shutdown_connections 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1632903495 wait complete. 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 -- start start 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 Processor -- start 2024-02-09T22:59:46.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192443de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924372c0 0x55d192444320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924403c0 0x55d192444860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19239e030 con 0x55d192436910 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d192071430 con 0x55d1924403c0 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19202de00 con 0x55d1924372c0 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcfa4e35740 10 client.0 _ll_register_callbacks cb 0x55d192393030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcf9bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192443de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcf9bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192443de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55388/0 (socket says 172.21.15.3:55388) 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.311+0000 7fcf9bfff640 1 -- 172.21.15.3:0/4025235456 learned_addr learned my addr 172.21.15.3:0/4025235456 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9bfff640 1 -- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924372c0 msgr2=0x55d192444320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924372c0 0x55d192444320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa0801640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924403c0 0x55d192444860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924372c0 0x55d192444320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9bfff640 1 -- 172.21.15.3:0/4025235456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924403c0 msgr2=0x55d192444860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924403c0 0x55d192444860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9bfff640 1 -- 172.21.15.3:0/4025235456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf94007a10 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924372c0 0x55d192444320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa0801640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924403c0 0x55d192444860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192443de0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00de20 tx=0x7fcf8c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad16473feee9173c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/4025235456 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c005220 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/4025235456 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf8c015070 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d19209d7e0 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/4025235456 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf8c00fea0 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa4e35740 10 client.18594 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55d19239c6b0 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/4025235456 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d19209d7e0 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/4025235456 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcf8c011770 con 0x55d192436910 2024-02-09T22:59:46.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa4e35740 10 client.18594 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.312+0000 7fcfa4e35740 10 client.18594 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcfa4e35740 20 client.18594 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924548e0 0x55d192456cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcfa4e35740 4 client.18594 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d1920da720 con 0x55d1924548e0 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924548e0 0x55d192456cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924548e0 0x55d192456cf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7f81a0c6b0c0198f server_cookie=2a075dc36b1cd36c in_seq=0 out_seq=0 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.313+0000 7fcf7b7fe640 10 client.18594 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/4025235456 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d1920da720 con 0x55d1924548e0 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcf7b7fe640 10 client.18594 handle_command_reply: tid=0 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 client.18594 shutdown 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 msgr2=0x55d192443de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192443de0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00de20 tx=0x7fcf8c00f730 comp rx=0 tx=0).stop 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 shutdown_connections 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924548e0 0x55d192456cf0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d1924403c0 0x55d192444860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924372c0 0x55d192444320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcf7b7fe640 0 client.18594 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/4025235456 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d192436910 0x55d192443de0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 >> 172.21.15.3:0/4025235456 conn(0x55d192431100 msgr2=0x55d192432e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 shutdown_connections 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 1 -- 172.21.15.3:0/4025235456 wait complete. 2024-02-09T22:59:46.325 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.314+0000 7fcfa4e35740 20 client.18594 trim_cache size 0 max 16384 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 Processor -- start 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 -- start start 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d192446510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d19244b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d19244b940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19242c8b0 con 0x55d1924372c0 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d192323130 con 0x55d192436910 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d192435d40 con 0x55d1924403c0 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d19244b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d19244b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55402/0 (socket says 172.21.15.3:55402) 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/1185553435 learned_addr learned my addr 172.21.15.3:0/1185553435 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.334 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d192446510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/1185553435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 msgr2=0x55d19244b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d19244b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/1185553435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 msgr2=0x55d192446510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d192446510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/1185553435 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d192338c90 con 0x55d1924372c0 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.326+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d19244b400 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcf900edb00 tx=0x7fcf9008fdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad135d4d705a5fa0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcf81ffb640 1 -- 172.21.15.3:0/1185553435 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900bd440 con 0x55d1924372c0 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcf81ffb640 1 -- 172.21.15.3:0/1185553435 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf900bcd40 con 0x55d1924372c0 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcf81ffb640 1 -- 172.21.15.3:0/1185553435 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900bb110 con 0x55d1924372c0 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1185553435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 msgr2=0x55d19244b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d19244b400 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fcf900edb00 tx=0x7fcf9008fdb0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1185553435 shutdown_connections 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d19244b940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d19244b400 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/1185553435 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d192446510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1185553435 >> 172.21.15.3:0/1185553435 conn(0x55d192431100 msgr2=0x55d1924446c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1185553435 shutdown_connections 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 -- 172.21.15.3:0/1185553435 wait complete. 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.327+0000 7fcfa4e35740 1 -- start start 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 Processor -- start 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d1924a4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d1924a49b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d19249ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19249f330 con 0x55d1924372c0 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19249f4a0 con 0x55d192436910 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d19249f660 con 0x55d1924403c0 2024-02-09T22:59:46.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcfa4e35740 10 client.0 _ll_register_callbacks cb 0x7fcf94008040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d1924a49b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d1924a49b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55408/0 (socket says 172.21.15.3:55408) 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/652670226 learned_addr learned my addr 172.21.15.3:0/652670226 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d1924a4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 msgr2=0x55d19249ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d19249ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/652670226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 msgr2=0x55d1924a4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d1924a4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 -- 172.21.15.3:0/652670226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf9c1a4980 con 0x55d1924372c0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf9b7fe640 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d1924a49b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcf900bccd0 tx=0x7fcf900eda10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b466cfcc42107cda server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf7b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.328+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/652670226 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900c54e0 con 0x55d1924372c0 2024-02-09T22:59:46.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/652670226 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fcf900bade0 con 0x55d1924372c0 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf94022070 con 0x55d1924372c0 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/652670226 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf900bd440 con 0x55d1924372c0 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 10 client.18600 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55d19235ea70 con 0x55d1924372c0 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/652670226 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf94022070 con 0x55d1924372c0 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/652670226 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fcf9003c070 con 0x55d1924372c0 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 10 client.18600 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 10 client.18600 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 20 client.18600 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924abf30 0x55d1924ae320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 4 client.18600 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:46.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d192129890 con 0x55d1924abf30 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.329+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924abf30 0x55d1924ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcf9bfff640 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924abf30 0x55d1924ae320 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e4292da60646e028 server_cookie=ea77d72619c065a0 in_seq=0 out_seq=0 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcf7b7fe640 10 client.18600 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcf7b7fe640 1 -- 172.21.15.3:0/652670226 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d192129890 con 0x55d1924abf30 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcf7b7fe640 10 client.18600 handle_command_reply: tid=0 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 client.18600 shutdown 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 msgr2=0x55d1924a49b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d1924a49b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fcf900bccd0 tx=0x7fcf900eda10 comp rx=0 tx=0).stop 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 shutdown_connections 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55d1924abf30 0x55d1924ae320 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d1924403c0 0x55d19249ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d1924372c0 0x55d1924a49b0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcf7b7fe640 0 client.18600 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T22:59:46.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 --2- 172.21.15.3:0/652670226 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d192436910 0x55d1924a4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 >> 172.21.15.3:0/652670226 conn(0x55d192431100 msgr2=0x55d19243f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 shutdown_connections 2024-02-09T22:59:46.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 1 -- 172.21.15.3:0/652670226 wait complete. 2024-02-09T22:59:46.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.330+0000 7fcfa4e35740 20 client.18600 trim_cache size 0 max 16384 2024-02-09T22:59:46.340 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf840a2230 msgr2=0x7fcf840a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf840a2230 0x7fcf840a46f0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fcf9401d000 tx=0x7fcf9401e000 comp rx=0 tx=0).stop 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 msgr2=0x7fcf9c19dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 0x7fcf9c19dbe0 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fcf90000c00 tx=0x7fcf9002c710 comp rx=0 tx=0).stop 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 shutdown_connections 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf840a2230 0x7fcf840a46f0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf9c197760 0x7fcf9c197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf9c105420 0x7fcf9c19dbe0 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 --2- 172.21.15.3:0/3405509158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf9c100ae0 0x7fcf9c19d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 >> 172.21.15.3:0/3405509158 conn(0x7fcf9c076630 msgr2=0x7fcf9c103700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 shutdown_connections 2024-02-09T22:59:46.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.331+0000 7fcfa2264640 1 -- 172.21.15.3:0/3405509158 wait complete. 2024-02-09T22:59:46.343 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:46 smithi003 ceph-mon[32678]: pgmap v264: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 672 KiB/s wr, 48 op/s 2024-02-09T22:59:46.343 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:46 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4012900985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:59:46.344 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:46.344 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T22:59:46.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 -- 172.21.15.3:0/1977652477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0100330 msgr2=0x7f03d0100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/1977652477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0100330 0x7f03d0100710 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009cd0 tx=0x7f03c002c0f0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 -- 172.21.15.3:0/1977652477 shutdown_connections 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/1977652477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03d0100ce0 0x7f03d0104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/1977652477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 0x7f03d010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/1977652477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0100330 0x7f03d0100710 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 -- 172.21.15.3:0/1977652477 >> 172.21.15.3:0/1977652477 conn(0x7f03d0077250 msgr2=0x7f03d0077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 -- 172.21.15.3:0/1977652477 shutdown_connections 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.446+0000 7f03d5ac9640 1 -- 172.21.15.3:0/1977652477 wait complete. 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 Processor -- start 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 -- start start 2024-02-09T22:59:46.452 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03d0100330 0x7f03d0198740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 0x7f03d0198c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 0x7f03d01a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01a3e90 con 0x7f03d0199210 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01a4000 con 0x7f03d0100330 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03d5ac9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d01a41c0 con 0x7f03d0105400 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03cffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 0x7f03d01a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03cffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 0x7f03d01a37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55432/0 (socket says 172.21.15.3:55432) 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03cffff640 1 -- 172.21.15.3:0/3783290652 learned_addr learned my addr 172.21.15.3:0/3783290652 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03d0100330 0x7f03d0198740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.447+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 0x7f03d0198c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03cffff640 1 -- 172.21.15.3:0/3783290652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 msgr2=0x7f03d0198c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03cffff640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 0x7f03d0198c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03cffff640 1 -- 172.21.15.3:0/3783290652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03d0100330 msgr2=0x7f03d0198740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03cffff640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03d0100330 0x7f03d0198740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03cffff640 1 -- 172.21.15.3:0/3783290652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c0009900 con 0x7f03d0199210 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 0x7f03d0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03cffff640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 0x7f03d01a37c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009fd0 tx=0x7f03c800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03ccff9640 1 -- 172.21.15.3:0/3783290652 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8011830 con 0x7f03d0199210 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03ccff9640 1 -- 172.21.15.3:0/3783290652 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c8011e70 con 0x7f03d0199210 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d01a44a0 con 0x7f03d0199210 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03ccff9640 1 -- 172.21.15.3:0/3783290652 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c800f2e0 con 0x7f03d0199210 2024-02-09T22:59:46.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.448+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d01a49b0 con 0x7f03d0199210 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.450+0000 7f03ccff9640 1 -- 172.21.15.3:0/3783290652 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f03c80105f0 con 0x7f03d0199210 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.450+0000 7f03d877c740 1 Processor -- start 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 -- start start 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03ccff9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f03b80a22b0 0x7f03b80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 0x555cedd76e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd773e0 0x555cedd7f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd823b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f03b80a22b0 0x7f03b80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedd32020 con 0x555cedd76a30 2024-02-09T22:59:46.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03ccff9640 1 -- 172.21.15.3:0/3783290652 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f03c80cd7c0 con 0x7f03d0199210 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedce5320 con 0x555cedd773e0 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03d877c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedce2850 con 0x555cedd7ff60 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 0x555cedd76e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 0x555cedd76e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55442/0 (socket says 172.21.15.3:55442) 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 -- 172.21.15.3:0/3847875312 learned_addr learned my addr 172.21.15.3:0/3847875312 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd773e0 0x555cedd7f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 -- 172.21.15.3:0/3847875312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 msgr2=0x555cedd823b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cffff640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd823b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd823b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 -- 172.21.15.3:0/3847875312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd773e0 msgr2=0x555cedd7f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd773e0 0x555cedd7f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 -- 172.21.15.3:0/3847875312 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cedca0210 con 0x555cedd76a30 2024-02-09T22:59:46.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cffff640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd823b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd773e0 0x555cedd7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f03b80a22b0 0x7f03b80a4770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f03c4005fd0 tx=0x7f03c40128d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.451+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 0x555cedd76e10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009430 tx=0x7f03c0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fea77feb5309c4a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03b5ffb640 1 -- 172.21.15.3:0/3847875312 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0034440 con 0x555cedd76a30 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03b5ffb640 1 -- 172.21.15.3:0/3847875312 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c00345a0 con 0x555cedd76a30 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03b5ffb640 1 -- 172.21.15.3:0/3847875312 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0003c60 con 0x555cedd76a30 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 -- 172.21.15.3:0/3847875312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 msgr2=0x555cedd76e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 0x555cedd76e10 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009430 tx=0x7f03c0005fd0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 -- 172.21.15.3:0/3847875312 shutdown_connections 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd773e0 0x555cedd7f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd823b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 --2- 172.21.15.3:0/3847875312 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd76a30 0x555cedd76e10 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 -- 172.21.15.3:0/3847875312 >> 172.21.15.3:0/3847875312 conn(0x555cedd71100 msgr2=0x555cedd714e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 -- 172.21.15.3:0/3847875312 shutdown_connections 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.452+0000 7f03d877c740 1 -- 172.21.15.3:0/3847875312 wait complete. 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 -- start start 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 Processor -- start 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd7d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd7d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedcde030 con 0x555cedd7ff60 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ced9b1430 con 0x555cedd76a30 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ced96de00 con 0x555cedd773e0 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03d877c740 10 client.0 _ll_register_callbacks cb 0x555cedcd3030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd7d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd7d9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55450/0 (socket says 172.21.15.3:55450) 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 -- 172.21.15.3:0/1718821970 learned_addr learned my addr 172.21.15.3:0/1718821970 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03ceffd640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd7d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 -- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 msgr2=0x555cedd7d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd7d460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 -- 172.21.15.3:0/1718821970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 msgr2=0x555cedd7cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.453+0000 7f03cffff640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03cffff640 1 -- 172.21.15.3:0/1718821970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d010d7e0 con 0x555cedd7ff60 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03cffff640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd7d9a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f03c8096c60 tx=0x7f03c809a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25bde46794c5a66e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03aeffd640 1 -- 172.21.15.3:0/1718821970 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c80a2a30 con 0x555cedd7ff60 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03aeffd640 1 -- 172.21.15.3:0/1718821970 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f03c809e120 con 0x555cedd7ff60 2024-02-09T22:59:46.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555ced9dd7e0 con 0x555cedd7ff60 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03aeffd640 1 -- 172.21.15.3:0/1718821970 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03c80cb160 con 0x555cedd7ff60 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 10 client.18612 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555cedcdc6b0 con 0x555cedd7ff60 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03aeffd640 1 -- 172.21.15.3:0/1718821970 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555ced9dd7e0 con 0x555cedd7ff60 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03aeffd640 1 -- 172.21.15.3:0/1718821970 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f03c8095fe0 con 0x555cedd7ff60 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 10 client.18612 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 10 client.18612 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 20 client.18612 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedd90470 0x555cedd92880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 4 client.18612 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.454+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555ceda1a720 con 0x555cedd90470 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.455+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedd90470 0x555cedd92880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.455+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedd90470 0x555cedd92880 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c887e78602c5dceb server_cookie=1f7ede26065a4c97 in_seq=0 out_seq=0 2024-02-09T22:59:46.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.455+0000 7f03aeffd640 10 client.18612 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03aeffd640 1 -- 172.21.15.3:0/1718821970 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555ceda1a720 con 0x555cedd90470 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03aeffd640 10 client.18612 handle_command_reply: tid=0 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 client.18612 shutdown 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 msgr2=0x555cedd7d9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd7d9a0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f03c8096c60 tx=0x7f03c809a290 comp rx=0 tx=0).stop 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 shutdown_connections 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedd90470 0x555cedd92880 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd7d9a0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd7d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03aeffd640 0 client.18612 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 --2- 172.21.15.3:0/1718821970 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 >> 172.21.15.3:0/1718821970 conn(0x555cedd71100 msgr2=0x555cedd72e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 shutdown_connections 2024-02-09T22:59:46.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 1 -- 172.21.15.3:0/1718821970 wait complete. 2024-02-09T22:59:46.468 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.456+0000 7f03d877c740 20 client.18612 trim_cache size 0 max 16384 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.467+0000 7f03d877c740 1 Processor -- start 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 -- start start 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 0x555cedd7d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd7a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedd6c8b0 con 0x555cedd773e0 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedc63130 con 0x555cedd76a30 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03d877c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedd8cfe0 con 0x555cedd7ff60 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 0x555cedd7d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 0x555cedd7d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55452/0 (socket says 172.21.15.3:55452) 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 -- 172.21.15.3:0/1941582501 learned_addr learned my addr 172.21.15.3:0/1941582501 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03cffff640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd7a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 -- 172.21.15.3:0/1941582501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 msgr2=0x555cedd7a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd7a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 -- 172.21.15.3:0/1941582501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 msgr2=0x555cedd7f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 -- 172.21.15.3:0/1941582501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555cedc78c90 con 0x555cedd773e0 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03cffff640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd7a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03ceffd640 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 0x555cedd7d7e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f03d006a440 tx=0x7f03c4005ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caa75828367f53ec server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03b57fa640 1 -- 172.21.15.3:0/1941582501 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4005220 con 0x555cedd773e0 2024-02-09T22:59:46.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03b57fa640 1 -- 172.21.15.3:0/1941582501 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c4018070 con 0x555cedd773e0 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.468+0000 7f03b57fa640 1 -- 172.21.15.3:0/1941582501 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c401f8d0 con 0x555cedd773e0 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 -- 172.21.15.3:0/1941582501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 msgr2=0x555cedd7d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 0x555cedd7d7e0 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x7f03d006a440 tx=0x7f03c4005ee0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 -- 172.21.15.3:0/1941582501 shutdown_connections 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd7ff60 0x555cedd7a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd773e0 0x555cedd7d7e0 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 --2- 172.21.15.3:0/1941582501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd7f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 -- 172.21.15.3:0/1941582501 >> 172.21.15.3:0/1941582501 conn(0x555cedd71100 msgr2=0x555cedd782b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 -- 172.21.15.3:0/1941582501 shutdown_connections 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 -- 172.21.15.3:0/1941582501 wait complete. 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.469+0000 7f03d877c740 1 -- start start 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 Processor -- start 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd872d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd856c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd85c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedd86300 con 0x555cedd7ff60 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedd86470 con 0x555cedd76a30 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555cedde6970 con 0x555cedd773e0 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 10 client.0 _ll_register_callbacks cb 0x7f03c0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd85c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd85c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:55466/0 (socket says 172.21.15.3:55466) 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 -- 172.21.15.3:0/3608395732 learned_addr learned my addr 172.21.15.3:0/3608395732 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd872d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03ceffd640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd856c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 -- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 msgr2=0x555cedd856c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd856c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 -- 172.21.15.3:0/3608395732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 msgr2=0x555cedd872d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd872d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 -- 172.21.15.3:0/3608395732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c4010490 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03cffff640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd85c00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f03c8096310 tx=0x7f03c800d130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6589ea4eecc5d83c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03aeffd640 1 -- 172.21.15.3:0/3608395732 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8096530 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03aeffd640 1 -- 172.21.15.3:0/3608395732 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f03c8009200 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.470+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03c00345a0 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03aeffd640 1 -- 172.21.15.3:0/3608395732 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03c80a2a30 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 10 client.18618 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555cedc9ea70 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03aeffd640 1 -- 172.21.15.3:0/3608395732 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c00345a0 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03aeffd640 1 -- 172.21.15.3:0/3608395732 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f03c8099150 con 0x555cedd7ff60 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 10 client.18618 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 10 client.18618 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 20 client.18618 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedde7c90 0x555ceddea0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 4 client.18618 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555ceda69890 con 0x555cedde7c90 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.471+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedde7c90 0x555ceddea0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03cf7fe640 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedde7c90 0x555ceddea0b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b283ba5902722d07 server_cookie=7579db9250973098 in_seq=0 out_seq=0 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03aeffd640 10 client.18618 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03aeffd640 1 -- 172.21.15.3:0/3608395732 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555ceda69890 con 0x555cedde7c90 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03aeffd640 10 client.18618 handle_command_reply: tid=0 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 client.18618 shutdown 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 msgr2=0x555cedd85c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd85c00 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f03c8096310 tx=0x7f03c800d130 comp rx=0 tx=0).stop 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 shutdown_connections 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x555cedde7c90 0x555ceddea0b0 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555cedd7ff60 0x555cedd85c00 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555cedd773e0 0x555cedd856c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 --2- 172.21.15.3:0/3608395732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555cedd76a30 0x555cedd872d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 >> 172.21.15.3:0/3608395732 conn(0x555cedd71100 msgr2=0x555cedd6f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03aeffd640 0 client.18618 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 shutdown_connections 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 1 -- 172.21.15.3:0/3608395732 wait complete. 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.472+0000 7f03d877c740 20 client.18618 trim_cache size 0 max 16384 2024-02-09T22:59:46.482 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f03b80a22b0 msgr2=0x7f03b80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f03b80a22b0 0x7f03b80a4770 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f03c4005fd0 tx=0x7f03c40128d0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 msgr2=0x7f03d01a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 0x7f03d01a37c0 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009fd0 tx=0x7f03c800b8e0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 shutdown_connections 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f03b80a22b0 0x7f03b80a4770 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f03d0199210 0x7f03d01a37c0 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f03d0105400 0x7f03d0198c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 --2- 172.21.15.3:0/3783290652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f03d0100330 0x7f03d0198740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 >> 172.21.15.3:0/3783290652 conn(0x7f03d0077250 msgr2=0x7f03d00fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 shutdown_connections 2024-02-09T22:59:46.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.473+0000 7f03d5ac9640 1 -- 172.21.15.3:0/3783290652 wait complete. 2024-02-09T22:59:46.486 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:46.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- 172.21.15.3:0/791985309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69501026c0 msgr2=0x7f695010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- 172.21.15.3:0/791985309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69501026c0 0x7f695010d0c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f6948009cd0 tx=0x7f694802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- 172.21.15.3:0/791985309 shutdown_connections 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- 172.21.15.3:0/791985309 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6950101d20 0x7f6950102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- 172.21.15.3:0/791985309 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f69501026c0 0x7f695010d0c0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- 172.21.15.3:0/791985309 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6950100440 0x7f6950100820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- 172.21.15.3:0/791985309 >> 172.21.15.3:0/791985309 conn(0x7f6950077310 msgr2=0x7f6950077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- 172.21.15.3:0/791985309 shutdown_connections 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- 172.21.15.3:0/791985309 wait complete. 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 Processor -- start 2024-02-09T22:59:46.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- start start 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 0x7f695019d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6950101d20 0x7f695019dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69501026c0 0x7f69501976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6950197e30 con 0x7f6950100440 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6950197fa0 con 0x7f6950101d20 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6956f8a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6950198160 con 0x7f69501026c0 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6954d26640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 0x7f695019d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.588+0000 7f6954d26640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 0x7f695019d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33622/0 (socket says 172.21.15.3:33622) 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 -- 172.21.15.3:0/101030555 learned_addr learned my addr 172.21.15.3:0/101030555 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f694ffff640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6950101d20 0x7f695019dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6955527640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69501026c0 0x7f69501976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 -- 172.21.15.3:0/101030555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69501026c0 msgr2=0x7f69501976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69501026c0 0x7f69501976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 -- 172.21.15.3:0/101030555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6950101d20 msgr2=0x7f695019dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6950101d20 0x7f695019dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 -- 172.21.15.3:0/101030555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6948009900 con 0x7f6950100440 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6955527640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69501026c0 0x7f69501976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6954d26640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 0x7f695019d590 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f694400c920 tx=0x7f694400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f694dffb640 1 -- 172.21.15.3:0/101030555 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944007c70 con 0x7f6950100440 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f694dffb640 1 -- 172.21.15.3:0/101030555 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6944007dd0 con 0x7f6950100440 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6950198440 con 0x7f6950100440 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f694dffb640 1 -- 172.21.15.3:0/101030555 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6944018860 con 0x7f6950100440 2024-02-09T22:59:46.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.589+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69501a4510 con 0x7f6950100440 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.591+0000 7f694dffb640 1 -- 172.21.15.3:0/101030555 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f694400fac0 con 0x7f6950100440 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 Processor -- start 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 -- start start 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694dffb640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69400a2230 0x7f69400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfc5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbf81020 con 0x55f1fbfc63e0 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbf34320 con 0x55f1fbfc5a30 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6955527640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69400a2230 0x7f69400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6959c38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbf31850 con 0x55f1fbfcef60 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694dffb640 1 -- 172.21.15.3:0/101030555 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f69440ce1f0 con 0x7f6950100440 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33632/0 (socket says 172.21.15.3:33632) 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 -- 172.21.15.3:0/1994830570 learned_addr learned my addr 172.21.15.3:0/1994830570 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6954d26640 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfc5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 -- 172.21.15.3:0/1994830570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 msgr2=0x55f1fbfd13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 -- 172.21.15.3:0/1994830570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc5a30 msgr2=0x55f1fbfc5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfc5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f694ffff640 1 -- 172.21.15.3:0/1994830570 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1fbeef210 con 0x55f1fbfc63e0 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.592+0000 7f6955527640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69400a2230 0x7f69400a46f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6948006010 tx=0x7f6948032420 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f694ffff640 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfce990 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f693c009c10 tx=0x7f693c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45b2ed60b5324d7c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f693affd640 1 -- 172.21.15.3:0/1994830570 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c004050 con 0x55f1fbfc63e0 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f693affd640 1 -- 172.21.15.3:0/1994830570 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f693c005500 con 0x55f1fbfc63e0 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f693affd640 1 -- 172.21.15.3:0/1994830570 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c004d10 con 0x55f1fbfc63e0 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 -- 172.21.15.3:0/1994830570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 msgr2=0x55f1fbfce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfce990 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f693c009c10 tx=0x7f693c011810 comp rx=0 tx=0).stop 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 -- 172.21.15.3:0/1994830570 shutdown_connections 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfce990 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 --2- 172.21.15.3:0/1994830570 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfc5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 -- 172.21.15.3:0/1994830570 >> 172.21.15.3:0/1994830570 conn(0x55f1fbfc0100 msgr2=0x55f1fbfc04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 -- 172.21.15.3:0/1994830570 shutdown_connections 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.593+0000 7f6959c38740 1 -- 172.21.15.3:0/1994830570 wait complete. 2024-02-09T22:59:46.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 -- start start 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 Processor -- start 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfcbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfcc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbf2d030 con 0x55f1fbfc63e0 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbc00430 con 0x55f1fbfcef60 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbbbce00 con 0x55f1fbfc5a30 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f6959c38740 10 client.0 _ll_register_callbacks cb 0x55f1fbf22030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f694ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfcc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f694ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfcc430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33642/0 (socket says 172.21.15.3:33642) 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.594+0000 7f694ffff640 1 -- 172.21.15.3:0/2749025027 learned_addr learned my addr 172.21.15.3:0/2749025027 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f694ffff640 1 -- 172.21.15.3:0/2749025027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 msgr2=0x55f1fbfcbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f6954d26640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfcbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f694ffff640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfcbef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f6955527640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f694ffff640 1 -- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 msgr2=0x55f1fbfcc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f694ffff640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f694ffff640 1 -- 172.21.15.3:0/2749025027 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c009870 con 0x55f1fbfc63e0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f694ffff640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfcc430 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f693c000c00 tx=0x7f693c0045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37813564a9acda6e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f692ffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f692ffff640 1 -- 172.21.15.3:0/2749025027 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c003a00 con 0x55f1fbfc63e0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f692ffff640 1 -- 172.21.15.3:0/2749025027 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f693c0046c0 con 0x55f1fbfc63e0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f1fbc2c7e0 con 0x55f1fbfc63e0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f692ffff640 1 -- 172.21.15.3:0/2749025027 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f693c003b60 con 0x55f1fbfc63e0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f6959c38740 10 client.18630 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55f1fbf2b6b0 con 0x55f1fbfc63e0 2024-02-09T22:59:46.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.595+0000 7f692ffff640 1 -- 172.21.15.3:0/2749025027 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f1fbc2c7e0 con 0x55f1fbfc63e0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f692ffff640 1 -- 172.21.15.3:0/2749025027 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f693c021040 con 0x55f1fbfc63e0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6959c38740 10 client.18630 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6959c38740 10 client.18630 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6959c38740 20 client.18630 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6959c38740 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fbfdf3a0 0x55f1fbfe1790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6959c38740 4 client.18630 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f1fbc69720 con 0x55f1fbfdf3a0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6954d26640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fbfdf3a0 0x55f1fbfe1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f692ffff640 10 client.18630 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.596+0000 7f6954d26640 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fbfdf3a0 0x55f1fbfe1790 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5a9020e2ee7f0e1c server_cookie=2968b0958aa0397d in_seq=0 out_seq=0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f692ffff640 1 -- 172.21.15.3:0/2749025027 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f1fbc69720 con 0x55f1fbfdf3a0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f692ffff640 10 client.18630 handle_command_reply: tid=0 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 client.18630 shutdown 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 msgr2=0x55f1fbfcc430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfcc430 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f693c000c00 tx=0x7f693c0045a0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 shutdown_connections 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fbfdf3a0 0x55f1fbfe1790 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfcc430 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 --2- 172.21.15.3:0/2749025027 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfcbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f692ffff640 0 client.18630 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 >> 172.21.15.3:0/2749025027 conn(0x55f1fbfc0100 msgr2=0x55f1fbfc1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 shutdown_connections 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 1 -- 172.21.15.3:0/2749025027 wait complete. 2024-02-09T22:59:46.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.597+0000 7f6959c38740 20 client.18630 trim_cache size 0 max 16384 2024-02-09T22:59:46.616 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 Processor -- start 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 -- start start 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfce620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfc04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbeb2130 con 0x55f1fbfc63e0 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbfbba40 con 0x55f1fbfcef60 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6959c38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbfca440 con 0x55f1fbfc5a30 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfc04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfc04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33652/0 (socket says 172.21.15.3:33652) 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 -- 172.21.15.3:0/4159431826 learned_addr learned my addr 172.21.15.3:0/4159431826 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6954d26640 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfce620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 -- 172.21.15.3:0/4159431826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 msgr2=0x55f1fbfce620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfce620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.617 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 -- 172.21.15.3:0/4159431826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 msgr2=0x55f1fbfcc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 -- 172.21.15.3:0/4159431826 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1fbec7c90 con 0x55f1fbfc63e0 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f6954d26640 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.609+0000 7f694ffff640 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfc04e0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f693c005480 tx=0x7f693c005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caffb28d064b2479 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f693a7fc640 1 -- 172.21.15.3:0/4159431826 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c004410 con 0x55f1fbfc63e0 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f693a7fc640 1 -- 172.21.15.3:0/4159431826 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f693c023e30 con 0x55f1fbfc63e0 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f693a7fc640 1 -- 172.21.15.3:0/4159431826 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f693c003ee0 con 0x55f1fbfc63e0 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 -- 172.21.15.3:0/4159431826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 msgr2=0x55f1fbfc04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfc04e0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f693c005480 tx=0x7f693c005b90 comp rx=0 tx=0).stop 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 -- 172.21.15.3:0/4159431826 shutdown_connections 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfcef60 0x55f1fbfcc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfc04e0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 --2- 172.21.15.3:0/4159431826 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfce620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 -- 172.21.15.3:0/4159431826 >> 172.21.15.3:0/4159431826 conn(0x55f1fbfc0100 msgr2=0x55f1fbfc67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 -- 172.21.15.3:0/4159431826 shutdown_connections 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 -- 172.21.15.3:0/4159431826 wait complete. 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.610+0000 7f6959c38740 1 -- start start 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 Processor -- start 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfd60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfd4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbfd4ff0 con 0x55f1fbfc5a30 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbfd5160 con 0x55f1fbfc63e0 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1fbfd5320 con 0x55f1fbfcef60 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6959c38740 10 client.0 _ll_register_callbacks cb 0x7f693c0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfd60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfd60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33654/0 (socket says 172.21.15.3:33654) 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 -- 172.21.15.3:0/3648414842 learned_addr learned my addr 172.21.15.3:0/3648414842 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6955527640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f694ffff640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfd4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 -- 172.21.15.3:0/3648414842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 msgr2=0x55f1fbfd4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 -- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc63e0 msgr2=0x55f1fbfd4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfd4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 -- 172.21.15.3:0/3648414842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1fbf2b6b0 con 0x55f1fbfc5a30 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6955527640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f6954d26640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfd60d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f694409a910 tx=0x7f694409a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61cf15a3bdf09855 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.611+0000 7f692ffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f692ffff640 1 -- 172.21.15.3:0/3648414842 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69440ce040 con 0x55f1fbfc5a30 2024-02-09T22:59:46.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f692ffff640 1 -- 172.21.15.3:0/3648414842 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f694409ed60 con 0x55f1fbfc5a30 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f693c005500 con 0x55f1fbfc5a30 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f692ffff640 1 -- 172.21.15.3:0/3648414842 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69440a33b0 con 0x55f1fbfc5a30 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 10 client.18636 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55f1fbeeda70 con 0x55f1fbfc5a30 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f692ffff640 1 -- 172.21.15.3:0/3648414842 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f693c005500 con 0x55f1fbfc5a30 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f692ffff640 1 -- 172.21.15.3:0/3648414842 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6944016020 con 0x55f1fbfc5a30 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 10 client.18636 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 10 client.18636 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 20 client.18636 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fc036b60 0x55f1fc038f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 4 client.18636 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f1fbcb8890 con 0x55f1fc036b60 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.612+0000 7f694ffff640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fc036b60 0x55f1fc038f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f694ffff640 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fc036b60 0x55f1fc038f80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1e798760ff6825f3 server_cookie=be2bae8a52f423cb in_seq=0 out_seq=0 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f692ffff640 10 client.18636 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:46.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f692ffff640 1 -- 172.21.15.3:0/3648414842 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f1fbcb8890 con 0x55f1fc036b60 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f692ffff640 10 client.18636 handle_command_reply: tid=0 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 client.18636 shutdown 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 msgr2=0x55f1fbfd60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfd60d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f694409a910 tx=0x7f694409a6a0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 shutdown_connections 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55f1fc036b60 0x55f1fc038f80 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f1fbfc63e0 0x55f1fbfd4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f692ffff640 0 client.18636 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f1fbfcef60 0x55f1fbfd4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 --2- 172.21.15.3:0/3648414842 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f1fbfc5a30 0x55f1fbfd60d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 >> 172.21.15.3:0/3648414842 conn(0x55f1fbfc0100 msgr2=0x55f1fbfc04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 shutdown_connections 2024-02-09T22:59:46.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 1 -- 172.21.15.3:0/3648414842 wait complete. 2024-02-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.613+0000 7f6959c38740 20 client.18636 trim_cache size 0 max 16384 2024-02-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:46.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69400a2230 msgr2=0x7f69400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69400a2230 0x7f69400a46f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6948006010 tx=0x7f6948032420 comp rx=0 tx=0).stop 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 msgr2=0x7f695019d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 0x7f695019d590 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f694400c920 tx=0x7f694400cde0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 shutdown_connections 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f69400a2230 0x7f69400a46f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6950101d20 0x7f695019dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f69501026c0 0x7f69501976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 --2- 172.21.15.3:0/101030555 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6950100440 0x7f695019d590 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 >> 172.21.15.3:0/101030555 conn(0x7f6950077310 msgr2=0x7f69500ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 shutdown_connections 2024-02-09T22:59:46.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.614+0000 7f6956f8a640 1 -- 172.21.15.3:0/101030555 wait complete. 2024-02-09T22:59:46.627 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T22:59:46.627 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T22:59:46.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 242931 2.60 MB/sec execute 340 sec latency 3105.983 ms 2024-02-09T22:59:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:46 smithi064 ceph-mon[36638]: pgmap v264: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 672 KiB/s wr, 48 op/s 2024-02-09T22:59:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:46 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4012900985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:59:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:46 smithi089 ceph-mon[36665]: pgmap v264: 97 pgs: 97 active+clean; 96 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 672 KiB/s wr, 48 op/s 2024-02-09T22:59:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:46 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4012900985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 -- 172.21.15.3:0/3270143466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 msgr2=0x7fab7010e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 --2- 172.21.15.3:0/3270143466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab7010e2d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 crypto rx=0x7fab68009cd0 tx=0x7fab6802c0f0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 -- 172.21.15.3:0/3270143466 shutdown_connections 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 --2- 172.21.15.3:0/3270143466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab70102df0 0x7fab70107220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 --2- 172.21.15.3:0/3270143466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab7010e2d0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 --2- 172.21.15.3:0/3270143466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab70102440 0x7fab70102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 -- 172.21.15.3:0/3270143466 >> 172.21.15.3:0/3270143466 conn(0x7fab70077310 msgr2=0x7fab70077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 -- 172.21.15.3:0/3270143466 shutdown_connections 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 -- 172.21.15.3:0/3270143466 wait complete. 2024-02-09T22:59:46.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 Processor -- start 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.733+0000 7fab78577640 1 -- start start 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab70102440 0x7fab70100090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab701005d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab70100b10 0x7fab701a5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab701a60f0 con 0x7fab70107930 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab701a6260 con 0x7fab70102440 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab701a6420 con 0x7fab70100b10 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab701005d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab701005d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33680/0 (socket says 172.21.15.3:33680) 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 -- 172.21.15.3:0/4212665911 learned_addr learned my addr 172.21.15.3:0/4212665911 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab76b14640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab70100b10 0x7fab701a5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab76313640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab70102440 0x7fab70100090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 -- 172.21.15.3:0/4212665911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab70100b10 msgr2=0x7fab701a5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab70100b10 0x7fab701a5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 -- 172.21.15.3:0/4212665911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab70102440 msgr2=0x7fab70100090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab70102440 0x7fab70100090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 -- 172.21.15.3:0/4212665911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab68009900 con 0x7fab70107930 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab75b12640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab701005d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fab6000e9f0 tx=0x7fab6000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab677fe640 1 -- 172.21.15.3:0/4212665911 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600096a0 con 0x7fab70107930 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab701a6700 con 0x7fab70107930 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab677fe640 1 -- 172.21.15.3:0/4212665911 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab6000ccc0 con 0x7fab70107930 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.734+0000 7fab677fe640 1 -- 172.21.15.3:0/4212665911 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab60018860 con 0x7fab70107930 2024-02-09T22:59:46.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.735+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab701a6c10 con 0x7fab70107930 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 Processor -- start 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 -- start start 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab677fe640 1 -- 172.21.15.3:0/4212665911 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fab60010aa0 con 0x7fab70107930 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508009a30 0x563508009e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 0x563508012990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508012f60 0x5635080153b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507fc5020 con 0x563508009a30 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507f78320 con 0x563508012f60 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab7b225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507f75850 con 0x56350800a3e0 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab76313640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 0x563508012990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab76b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508009a30 0x563508009e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab76313640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 0x563508012990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:49206/0 (socket says 172.21.15.3:49206) 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab76313640 1 -- 172.21.15.3:0/1150693887 learned_addr learned my addr 172.21.15.3:0/1150693887 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.738+0000 7fab75b12640 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508012f60 0x5635080153b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab677fe640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fab5c0a2230 0x7fab5c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 -- 172.21.15.3:0/1150693887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 msgr2=0x563508012990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 0x563508012990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 -- 172.21.15.3:0/1150693887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508012f60 msgr2=0x5635080153b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508012f60 0x5635080153b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 -- 172.21.15.3:0/1150693887 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563507f33210 con 0x563508009a30 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76313640 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 0x563508012990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab677fe640 1 -- 172.21.15.3:0/4212665911 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fab60014070 con 0x7fab70107930 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fab5c0a2230 0x7fab5c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab76b14640 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508009a30 0x563508009e10 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fab68037040 tx=0x7fab68005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=546cdfbe1df8441e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab64bfe640 1 -- 172.21.15.3:0/1150693887 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68034640 con 0x563508009a30 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab64bfe640 1 -- 172.21.15.3:0/1150693887 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab68003970 con 0x563508009a30 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.739+0000 7fab64bfe640 1 -- 172.21.15.3:0/1150693887 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68050070 con 0x563508009a30 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 -- 172.21.15.3:0/1150693887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508009a30 msgr2=0x563508009e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab76b14640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fab5c0a2230 0x7fab5c0a46f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x563508015aa0 tx=0x7fab6804e000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508009a30 0x563508009e10 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fab68037040 tx=0x7fab68005f80 comp rx=0 tx=0).stop 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 -- 172.21.15.3:0/1150693887 shutdown_connections 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508012f60 0x5635080153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56350800a3e0 0x563508012990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 --2- 172.21.15.3:0/1150693887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508009a30 0x563508009e10 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 -- 172.21.15.3:0/1150693887 >> 172.21.15.3:0/1150693887 conn(0x563508004100 msgr2=0x5635080044e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 -- 172.21.15.3:0/1150693887 shutdown_connections 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 -- 172.21.15.3:0/1150693887 wait complete. 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.740+0000 7fab7b225740 1 -- start start 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 Processor -- start 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508009a30 0x56350800ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56350800a3e0 0x563508010440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 0x563508010980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507f71030 con 0x563508012f60 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507c44430 con 0x56350800a3e0 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507c00e00 con 0x563508009a30 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab7b225740 10 client.0 _ll_register_callbacks cb 0x563507f66030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 0x563508010980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 0x563508010980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33702/0 (socket says 172.21.15.3:33702) 2024-02-09T22:59:46.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 -- 172.21.15.3:0/4081100095 learned_addr learned my addr 172.21.15.3:0/4081100095 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab75b12640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56350800a3e0 0x563508010440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76313640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508009a30 0x56350800ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 -- 172.21.15.3:0/4081100095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508009a30 msgr2=0x56350800ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508009a30 0x56350800ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 -- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56350800a3e0 msgr2=0x563508010440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56350800a3e0 0x563508010440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 -- 172.21.15.3:0/4081100095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7010ea90 con 0x563508012f60 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab76b14640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 0x563508010980 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fab680347b0 tx=0x7fab6802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37f3f8338488d614 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.741+0000 7fab59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab59ffb640 1 -- 172.21.15.3:0/4081100095 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab68050050 con 0x563508012f60 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab59ffb640 1 -- 172.21.15.3:0/4081100095 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fab68003ae0 con 0x563508012f60 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563507c707e0 con 0x563508012f60 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab59ffb640 1 -- 172.21.15.3:0/4081100095 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab68003ca0 con 0x563508012f60 2024-02-09T22:59:46.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 10 client.18648 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x563507f6f6b0 con 0x563508012f60 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab59ffb640 1 -- 172.21.15.3:0/4081100095 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563507c707e0 con 0x563508012f60 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab59ffb640 1 -- 172.21.15.3:0/4081100095 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab6804c7a0 con 0x563508012f60 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 10 client.18648 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 10 client.18648 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 20 client.18648 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563508023100 0x5635080254f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 4 client.18648 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563507cad720 con 0x563508023100 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.742+0000 7fab76313640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563508023100 0x5635080254f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.743+0000 7fab76313640 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563508023100 0x5635080254f0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6c86237c0328967 server_cookie=18878f1cf56b9032 in_seq=0 out_seq=0 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.743+0000 7fab59ffb640 10 client.18648 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab59ffb640 1 -- 172.21.15.3:0/4081100095 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563507cad720 con 0x563508023100 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab59ffb640 10 client.18648 handle_command_reply: tid=0 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 client.18648 shutdown 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 msgr2=0x563508010980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 0x563508010980 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fab680347b0 tx=0x7fab6802c890 comp rx=0 tx=0).stop 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 shutdown_connections 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563508023100 0x5635080254f0 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56350800a3e0 0x563508010440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563508012f60 0x563508010980 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab59ffb640 0 client.18648 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 --2- 172.21.15.3:0/4081100095 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508009a30 0x56350800ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 >> 172.21.15.3:0/4081100095 conn(0x563508004100 msgr2=0x563508005e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 shutdown_connections 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 1 -- 172.21.15.3:0/4081100095 wait complete. 2024-02-09T22:59:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.745+0000 7fab7b225740 20 client.18648 trim_cache size 0 max 16384 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 Processor -- start 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 -- start start 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508012630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080107a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x56350800d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507ef6130 con 0x56350800a3e0 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563507cad720 con 0x563508009a30 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab7b225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563508009530 con 0x563508012f60 2024-02-09T22:59:46.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080107a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080107a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33704/0 (socket says 172.21.15.3:33704) 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 -- 172.21.15.3:0/1988225727 learned_addr learned my addr 172.21.15.3:0/1988225727 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab76b14640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x56350800d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab76313640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508012630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 -- 172.21.15.3:0/1988225727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 msgr2=0x56350800d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x56350800d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 -- 172.21.15.3:0/1988225727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 msgr2=0x563508012630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508012630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab75b12640 1 -- 172.21.15.3:0/1988225727 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563507f0bc90 con 0x56350800a3e0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.759+0000 7fab76b14640 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x56350800d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab75b12640 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080107a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fab600d1a20 tx=0x7fab60010ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fa5af2d0e895e2e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab5bfff640 1 -- 172.21.15.3:0/1988225727 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6009ac10 con 0x56350800a3e0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab5bfff640 1 -- 172.21.15.3:0/1988225727 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab6009a980 con 0x56350800a3e0 2024-02-09T22:59:46.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab5bfff640 1 -- 172.21.15.3:0/1988225727 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab600cbd70 con 0x56350800a3e0 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab7b225740 1 -- 172.21.15.3:0/1988225727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 msgr2=0x5635080107a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab7b225740 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080107a0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7fab600d1a20 tx=0x7fab60010ea0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab7b225740 1 -- 172.21.15.3:0/1988225727 shutdown_connections 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.760+0000 7fab7b225740 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x56350800d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080107a0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 --2- 172.21.15.3:0/1988225727 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508012630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- 172.21.15.3:0/1988225727 >> 172.21.15.3:0/1988225727 conn(0x563508004100 msgr2=0x56350800b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- 172.21.15.3:0/1988225727 shutdown_connections 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- 172.21.15.3:0/1988225727 wait complete. 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- start start 2024-02-09T22:59:46.769 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 Processor -- start 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508018ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080191e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x5635080795a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5635080199a0 con 0x56350800a3e0 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563508019b10 con 0x563508009a30 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563508079b10 con 0x563508012f60 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab7b225740 10 client.0 _ll_register_callbacks cb 0x7fab68037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab75b12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080191e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab75b12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080191e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33720/0 (socket says 172.21.15.3:33720) 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab75b12640 1 -- 172.21.15.3:0/1004624349 learned_addr learned my addr 172.21.15.3:0/1004624349 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab76b14640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x5635080795a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.761+0000 7fab76313640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508018ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.770 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab75b12640 1 -- 172.21.15.3:0/1004624349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 msgr2=0x5635080795a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab75b12640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x5635080795a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab75b12640 1 -- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 msgr2=0x563508018ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab75b12640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508018ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab75b12640 1 -- 172.21.15.3:0/1004624349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563507f6f6b0 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab75b12640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080191e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fab60097810 tx=0x7fab600a3530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20b017578f03fdf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab59ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab59ffb640 1 -- 172.21.15.3:0/1004624349 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6009a490 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab59ffb640 1 -- 172.21.15.3:0/1004624349 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7fab6009ae40 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab68003970 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab59ffb640 1 -- 172.21.15.3:0/1004624349 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab6009a7a0 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 10 client.18654 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x7fab701a6c10 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab59ffb640 1 -- 172.21.15.3:0/1004624349 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab6007d080 con 0x56350800a3e0 2024-02-09T22:59:46.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab59ffb640 1 -- 172.21.15.3:0/1004624349 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fab60099d00 con 0x56350800a3e0 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 10 client.18654 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 10 client.18654 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 20 client.18654 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56350807b010 0x56350807d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 4 client.18654 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.762+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563507cfc890 con 0x56350807b010 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab76313640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56350807b010 0x56350807d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab59ffb640 10 client.18654 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab76313640 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56350807b010 0x56350807d430 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f80dff5d5dbc7a1b server_cookie=faf8045575ef655 in_seq=0 out_seq=0 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab59ffb640 1 -- 172.21.15.3:0/1004624349 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563507cfc890 con 0x56350807b010 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab59ffb640 10 client.18654 handle_command_reply: tid=0 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 client.18654 shutdown 2024-02-09T22:59:46.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 msgr2=0x5635080191e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080191e0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7fab60097810 tx=0x7fab600a3530 comp rx=0 tx=0).stop 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 shutdown_connections 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56350807b010 0x56350807d430 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563508012f60 0x5635080795a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab59ffb640 0 client.18654 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56350800a3e0 0x5635080191e0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 --2- 172.21.15.3:0/1004624349 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563508009a30 0x563508018ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 >> 172.21.15.3:0/1004624349 conn(0x563508004100 msgr2=0x5635080029b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 shutdown_connections 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.763+0000 7fab7b225740 1 -- 172.21.15.3:0/1004624349 wait complete. 2024-02-09T22:59:46.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab7b225740 20 client.18654 trim_cache size 0 max 16384 2024-02-09T22:59:46.775 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fab5c0a2230 msgr2=0x7fab5c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fab5c0a2230 0x7fab5c0a46f0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x563508015aa0 tx=0x7fab6804e000 comp rx=0 tx=0).stop 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 msgr2=0x7fab701005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab701005d0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fab6000e9f0 tx=0x7fab6000eeb0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 shutdown_connections 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fab5c0a2230 0x7fab5c0a46f0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fab70100b10 0x7fab701a5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fab70107930 0x7fab701005d0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 --2- 172.21.15.3:0/4212665911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fab70102440 0x7fab70100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 >> 172.21.15.3:0/4212665911 conn(0x7fab70077310 msgr2=0x7fab700ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 shutdown_connections 2024-02-09T22:59:46.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:46.764+0000 7fab78577640 1 -- 172.21.15.3:0/4212665911 wait complete. 2024-02-09T22:59:47.653 INFO:tasks.workunit.client.0.smithi003.stdout: 1 245159 2.63 MB/sec execute 341 sec latency 3137.506 ms 2024-02-09T22:59:47.778 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T22:59:47.779 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T22:59:47.779 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 -- 172.21.15.3:0/3136382862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c106570 msgr2=0x7f0c0c10e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 --2- 172.21.15.3:0/3136382862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c106570 0x7f0c0c10e120 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f0c04009cd0 tx=0x7f0c0402c0f0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 -- 172.21.15.3:0/3136382862 shutdown_connections 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 --2- 172.21.15.3:0/3136382862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c106570 0x7f0c0c10e120 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 --2- 172.21.15.3:0/3136382862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c0c06a5b0 0x7f0c0c105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 --2- 172.21.15.3:0/3136382862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c0c069c00 0x7f0c0c069fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 -- 172.21.15.3:0/3136382862 >> 172.21.15.3:0/3136382862 conn(0x7f0c0c0771d0 msgr2=0x7f0c0c0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 -- 172.21.15.3:0/3136382862 shutdown_connections 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.890+0000 7f0c1350c640 1 -- 172.21.15.3:0/3136382862 wait complete. 2024-02-09T22:59:47.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 Processor -- start 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 -- start start 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c0c069c00 0x7f0c0c19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c0c106570 0x7f0c0c19d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 0x7f0c0c197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c198000 con 0x7f0c0c197610 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c198170 con 0x7f0c0c106570 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c1350c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0c198330 con 0x7f0c0c069c00 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 0x7f0c0c197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 0x7f0c0c197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33746/0 (socket says 172.21.15.3:33746) 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 -- 172.21.15.3:0/837552825 learned_addr learned my addr 172.21.15.3:0/837552825 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c0c106570 0x7f0c0c19d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c112a8640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c0c069c00 0x7f0c0c19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.896 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 -- 172.21.15.3:0/837552825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c0c069c00 msgr2=0x7f0c0c19d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c0c069c00 0x7f0c0c19d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 -- 172.21.15.3:0/837552825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c0c106570 msgr2=0x7f0c0c19d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c0c106570 0x7f0c0c19d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.891+0000 7f0c11aa9640 1 -- 172.21.15.3:0/837552825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c04009900 con 0x7f0c0c197610 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.892+0000 7f0c11aa9640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 0x7f0c0c197ac0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f0c04004af0 tx=0x7f0c04032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.892+0000 7f0c027fc640 1 -- 172.21.15.3:0/837552825 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c040328b0 con 0x7f0c0c197610 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.892+0000 7f0c027fc640 1 -- 172.21.15.3:0/837552825 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c04003e50 con 0x7f0c0c197610 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.892+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c0c1a42d0 con 0x7f0c0c197610 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.892+0000 7f0c027fc640 1 -- 172.21.15.3:0/837552825 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c04004740 con 0x7f0c0c197610 2024-02-09T22:59:47.897 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.892+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c0c1a4780 con 0x7f0c0c197610 2024-02-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.894+0000 7f0c027fc640 1 -- 172.21.15.3:0/837552825 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0c0400a040 con 0x7f0c0c197610 2024-02-09T22:59:47.902 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.894+0000 7f0c161c3740 1 Processor -- start 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.894+0000 7f0c161c3740 1 -- start start 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c027fc640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0bf80a22b0 0x7f0bf80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88d1220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88de560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88e0e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c161c3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac883f000 con 0x560ac88d1760 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c161c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac883c530 con 0x560ac88d0e40 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c161c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac8837d10 con 0x560ac88deaa0 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0bf80a22b0 0x7f0bf80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c027fc640 1 -- 172.21.15.3:0/837552825 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0c040f0b00 con 0x7f0c0c197610 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88d1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88d1220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37438/0 (socket says 172.21.15.3:37438) 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 -- 172.21.15.3:0/1463879607 learned_addr learned my addr 172.21.15.3:0/1463879607 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:47.903 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c11aa9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88e0e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c10aa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88de560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 -- 172.21.15.3:0/1463879607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 msgr2=0x560ac88e0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88e0e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 -- 172.21.15.3:0/1463879607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 msgr2=0x560ac88de560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88de560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c112a8640 1 -- 172.21.15.3:0/1463879607 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ac87de210 con 0x560ac88d0e40 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.895+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0bf80a22b0 0x7f0bf80a4770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009fd0 tx=0x7f0bfc009b80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.896+0000 7f0c112a8640 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88d1220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009c80 tx=0x7f0c080117b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=42b2ad7e366eeb6f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.896+0000 7f0bf77fe640 1 -- 172.21.15.3:0/1463879607 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c08009750 con 0x560ac88d0e40 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.896+0000 7f0bf77fe640 1 -- 172.21.15.3:0/1463879607 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c08012930 con 0x560ac88d0e40 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.896+0000 7f0bf77fe640 1 -- 172.21.15.3:0/1463879607 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0800f9f0 con 0x560ac88d0e40 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.896+0000 7f0c161c3740 1 -- 172.21.15.3:0/1463879607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 msgr2=0x560ac88d1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.896+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88d1220 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f0c08009c80 tx=0x7f0c080117b0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- 172.21.15.3:0/1463879607 shutdown_connections 2024-02-09T22:59:47.904 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88e0e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88de560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1463879607 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88d1220 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- 172.21.15.3:0/1463879607 >> 172.21.15.3:0/1463879607 conn(0x560ac88cb8c0 msgr2=0x560ac88cbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- 172.21.15.3:0/1463879607 shutdown_connections 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- 172.21.15.3:0/1463879607 wait complete. 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- start start 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 Processor -- start 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d0e40 0x560ac88db9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88dbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac88dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac87dce60 con 0x560ac88d1760 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac850b4b0 con 0x560ac88deaa0 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac87bd670 con 0x560ac88d0e40 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.897+0000 7f0c161c3740 10 client.0 _ll_register_callbacks cb 0x560ac87d5190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:47.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88dbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88dbf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33762/0 (socket says 172.21.15.3:33762) 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 -- 172.21.15.3:0/1697188699 learned_addr learned my addr 172.21.15.3:0/1697188699 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c112a8640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d0e40 0x560ac88db9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c11aa9640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac88dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 -- 172.21.15.3:0/1697188699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d0e40 msgr2=0x560ac88db9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d0e40 0x560ac88db9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 -- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 msgr2=0x560ac88dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac88dc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 -- 172.21.15.3:0/1697188699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c080098b0 con 0x560ac88d1760 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88dbf30 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c102340 tx=0x7f0bfc01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd9996718664f4b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0bf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/1697188699 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc016040 con 0x560ac88d1760 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/1697188699 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0bfc0044c0 con 0x560ac88d1760 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ac8537860 con 0x560ac88d1760 2024-02-09T22:59:47.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/1697188699 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0bfc009e70 con 0x560ac88d1760 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c161c3740 10 client.18666 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.898+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x560ac8836390 con 0x560ac88d1760 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/1697188699 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560ac8537860 con 0x560ac88d1760 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/1697188699 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0bfc026070 con 0x560ac88d1760 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c161c3740 10 client.18666 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c161c3740 10 client.18666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c161c3740 20 client.18666 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac88eeb00 0x560ac88f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c161c3740 4 client.18666 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ac8588410 con 0x560ac88eeb00 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c112a8640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac88eeb00 0x560ac88f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0bf4ff9640 10 client.18666 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:47.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.899+0000 7f0c112a8640 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac88eeb00 0x560ac88f0f20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2f1344d878081d3 server_cookie=b3d6e59cf3a07c0b in_seq=0 out_seq=0 2024-02-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.900+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/1697188699 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ac8588410 con 0x560ac88eeb00 2024-02-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.900+0000 7f0bf4ff9640 10 client.18666 handle_command_reply: tid=0 2024-02-09T22:59:47.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.900+0000 7f0c161c3740 1 client.18666 shutdown 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 msgr2=0x560ac88dbf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88dbf30 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c102340 tx=0x7f0bfc01d000 comp rx=0 tx=0).stop 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 shutdown_connections 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac88eeb00 0x560ac88f0f20 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac88dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88dbf30 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0bf4ff9640 0 client.18666 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 --2- 172.21.15.3:0/1697188699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d0e40 0x560ac88db9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 >> 172.21.15.3:0/1697188699 conn(0x560ac88cb8c0 msgr2=0x560ac88cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 shutdown_connections 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 1 -- 172.21.15.3:0/1697188699 wait complete. 2024-02-09T22:59:47.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.901+0000 7f0c161c3740 20 client.18666 trim_cache size 0 max 16384 2024-02-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 Processor -- start 2024-02-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 -- start start 2024-02-09T22:59:47.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88de120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88da9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88dc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac88d04e0 con 0x560ac88d1760 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac88c70a0 con 0x560ac88d0e40 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c161c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac88f6130 con 0x560ac88deaa0 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c10aa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88da9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c10aa7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88da9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33764/0 (socket says 172.21.15.3:33764) 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c10aa7640 1 -- 172.21.15.3:0/477715523 learned_addr learned my addr 172.21.15.3:0/477715523 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c112a8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88de120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.917+0000 7f0c11aa9640 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88dc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c10aa7640 1 -- 172.21.15.3:0/477715523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 msgr2=0x560ac88dc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88dc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c10aa7640 1 -- 172.21.15.3:0/477715523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 msgr2=0x560ac88de120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88de120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c10aa7640 1 -- 172.21.15.3:0/477715523 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ac87d2c70 con 0x560ac88d1760 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c112a8640 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88de120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88da9d0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00f7b0 tx=0x7f0bfc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ea1115886a5a93 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0bf6ffd640 1 -- 172.21.15.3:0/477715523 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc016040 con 0x560ac88d1760 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0bf6ffd640 1 -- 172.21.15.3:0/477715523 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0bfc00fd60 con 0x560ac88d1760 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.918+0000 7f0bf6ffd640 1 -- 172.21.15.3:0/477715523 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bfc0316b0 con 0x560ac88d1760 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 -- 172.21.15.3:0/477715523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 msgr2=0x560ac88da9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88da9d0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc00f7b0 tx=0x7f0bfc005f90 comp rx=0 tx=0).stop 2024-02-09T22:59:47.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 -- 172.21.15.3:0/477715523 shutdown_connections 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88deaa0 0x560ac88dc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d1760 0x560ac88da9d0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 --2- 172.21.15.3:0/477715523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88d0e40 0x560ac88de120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 -- 172.21.15.3:0/477715523 >> 172.21.15.3:0/477715523 conn(0x560ac88cb8c0 msgr2=0x560ac88da2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 -- 172.21.15.3:0/477715523 shutdown_connections 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 -- 172.21.15.3:0/477715523 wait complete. 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 -- start start 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.919+0000 7f0c161c3740 1 Processor -- start 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 0x560ac88e4810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d1760 0x560ac88e4d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac8945010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac88e5620 con 0x560ac88d0e40 2024-02-09T22:59:47.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac8945550 con 0x560ac88deaa0 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ac8945690 con 0x560ac88d1760 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 10 client.0 _ll_register_callbacks cb 0x7f0c080096a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 0x560ac88e4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 0x560ac88e4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33772/0 (socket says 172.21.15.3:33772) 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 -- 172.21.15.3:0/2448887002 learned_addr learned my addr 172.21.15.3:0/2448887002 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d1760 0x560ac88e4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c11aa9640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac8945010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 -- 172.21.15.3:0/2448887002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d1760 msgr2=0x560ac88e4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d1760 0x560ac88e4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 -- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 msgr2=0x560ac8945010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac8945010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 -- 172.21.15.3:0/2448887002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ac87de210 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c112a8640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 0x560ac88e4810 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f0c08005fd0 tx=0x7f0c0800f480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5236d8f5d69f22 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0bf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/2448887002 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c0802a070 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c08012930 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.920+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/2448887002 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f0c08012760 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/2448887002 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c08009d70 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 10 client.18672 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x560ac8836390 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/2448887002 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c08012930 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/2448887002 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0c08022040 con 0x560ac88d0e40 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 10 client.18672 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:47.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 10 client.18672 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 20 client.18672 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac8946ee0 0x560ac8949300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 4 client.18672 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x560ac85747b0 con 0x560ac8946ee0 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac8946ee0 0x560ac8949300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0bf4ff9640 10 client.18672 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.921+0000 7f0c10aa7640 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac8946ee0 0x560ac8949300 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc737a99d121b122 server_cookie=616b0dc5f0ffb7c2 in_seq=0 out_seq=0 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.922+0000 7f0bf4ff9640 1 -- 172.21.15.3:0/2448887002 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x560ac85747b0 con 0x560ac8946ee0 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.922+0000 7f0bf4ff9640 10 client.18672 handle_command_reply: tid=0 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.922+0000 7f0c161c3740 1 client.18672 shutdown 2024-02-09T22:59:47.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.922+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 msgr2=0x560ac88e4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.922+0000 7f0c161c3740 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 0x560ac88e4810 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7f0c08005fd0 tx=0x7f0c0800f480 comp rx=0 tx=0).stop 2024-02-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 shutdown_connections 2024-02-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560ac8946ee0 0x560ac8949300 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ac88deaa0 0x560ac8945010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ac88d1760 0x560ac88e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0bf4ff9640 0 client.18672 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 --2- 172.21.15.3:0/2448887002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ac88d0e40 0x560ac88e4810 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 >> 172.21.15.3:0/2448887002 conn(0x560ac88cb8c0 msgr2=0x560ac88ca920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 shutdown_connections 2024-02-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 1 -- 172.21.15.3:0/2448887002 wait complete. 2024-02-09T22:59:47.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c161c3740 20 client.18672 trim_cache size 0 max 16384 2024-02-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "8de2f341-e18f-45ab-9a21-9022d1a3cba1", 2024-02-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T22:59:47.937 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:59:47.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0bf80a22b0 msgr2=0x7f0bf80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0bf80a22b0 0x7f0bf80a4770 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc009fd0 tx=0x7f0bfc009b80 comp rx=0 tx=0).stop 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 msgr2=0x7f0c0c197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 0x7f0c0c197ac0 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f0c04004af0 tx=0x7f0c04032570 comp rx=0 tx=0).stop 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 shutdown_connections 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0bf80a22b0 0x7f0bf80a4770 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0c0c106570 0x7f0c0c19d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0c0c197610 0x7f0c0c197ac0 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 --2- 172.21.15.3:0/837552825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0c0c069c00 0x7f0c0c19d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.923+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 >> 172.21.15.3:0/837552825 conn(0x7f0c0c0771d0 msgr2=0x7f0c0c0fe280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.924+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 shutdown_connections 2024-02-09T22:59:47.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:47.924+0000 7f0c1350c640 1 -- 172.21.15.3:0/837552825 wait complete. 2024-02-09T22:59:47.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T22:59:47.940 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 -- 172.21.15.3:0/4145329658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 msgr2=0x7f9d0c100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 --2- 172.21.15.3:0/4145329658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c100720 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc009890 tx=0x7f9cfc02bb70 comp rx=0 tx=0).stop 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 -- 172.21.15.3:0/4145329658 shutdown_connections 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 --2- 172.21.15.3:0/4145329658 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d0c105410 0x7f9d0c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 --2- 172.21.15.3:0/4145329658 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c100cf0 0x7f9d0c104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 --2- 172.21.15.3:0/4145329658 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c100720 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 -- 172.21.15.3:0/4145329658 >> 172.21.15.3:0/4145329658 conn(0x7f9d0c0772e0 msgr2=0x7f9d0c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 -- 172.21.15.3:0/4145329658 shutdown_connections 2024-02-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 -- 172.21.15.3:0/4145329658 wait complete. 2024-02-09T22:59:48.057 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.053+0000 7f9d117bc640 1 Processor -- start 2024-02-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 -- start start 2024-02-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d0c105410 0x7f9d0c19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c1977f0 0x7f9d0c197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c1982a0 con 0x7f9d0c1977f0 2024-02-09T22:59:48.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c198410 con 0x7f9d0c100340 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d117bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0c198580 con 0x7f9d0c105410 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d0affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d0affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c19d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37474/0 (socket says 172.21.15.3:37474) 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d0affd640 1 -- 172.21.15.3:0/1177851066 learned_addr learned my addr 172.21.15.3:0/1177851066 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.054+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d0c105410 0x7f9d0c19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c1977f0 0x7f9d0c197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0affd640 1 -- 172.21.15.3:0/1177851066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d0c105410 msgr2=0x7f9d0c19dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0affd640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d0c105410 0x7f9d0c19dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0affd640 1 -- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c1977f0 msgr2=0x7f9d0c197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0affd640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c1977f0 0x7f9d0c197ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0affd640 1 -- 172.21.15.3:0/1177851066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d00009560 con 0x7f9d0c100340 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c1977f0 0x7f9d0c197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d0affd640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c19d680 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc037040 tx=0x7f9cfc0045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9cf3fff640 1 -- 172.21.15.3:0/1177851066 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0349d0 con 0x7f9d0c100340 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9cf3fff640 1 -- 172.21.15.3:0/1177851066 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cfc02c9f0 con 0x7f9d0c100340 2024-02-09T22:59:48.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9cf3fff640 1 -- 172.21.15.3:0/1177851066 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0035f0 con 0x7f9d0c100340 2024-02-09T22:59:48.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d04009840 con 0x7f9d0c100340 2024-02-09T22:59:48.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.055+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cfc009490 con 0x7f9d0c100340 2024-02-09T22:59:48.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.058+0000 7f9cf3fff640 1 -- 172.21.15.3:0/1177851066 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9cfc03b040 con 0x7f9d0c100340 2024-02-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9cf3fff640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9cf40a66c0 0x7f9cf40a8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9cf3fff640 1 -- 172.21.15.3:0/1177851066 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9cfc0efc60 con 0x7f9d0c100340 2024-02-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9cf40a66c0 0x7f9cf40a8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 Processor -- start 2024-02-09T22:59:48.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 -- start start 2024-02-09T22:59:48.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cceed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 0x555f1ccf78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccfa300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9cf40a66c0 0x7f9cf40a8b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c105030 tx=0x7f9d00009260 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cca9d00 con 0x555f1ccf7eb0 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cc5e2f0 con 0x555f1ccee980 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.059+0000 7f9d1438e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cc5b820 con 0x555f1ccef330 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 0x555f1ccf78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccfa300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cceed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0a7fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 0x555f1ccf78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:49322/0 (socket says 172.21.15.3:49322) 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cceed60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37486/0 (socket says 172.21.15.3:37486) 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/3945768000 learned_addr learned my addr 172.21.15.3:0/3945768000 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3945768000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 msgr2=0x555f1ccf78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 0x555f1ccf78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3945768000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 msgr2=0x555f1cceed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cceed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3945768000 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f1cc18ff0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0affd640 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cceed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 0x555f1ccf78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccfa300 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9d04011460 tx=0x7f9d04011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161ed8f52df0afcf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9cf17fa640 1 -- 172.21.15.3:0/3945768000 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04012770 con 0x555f1ccf7eb0 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9cf17fa640 1 -- 172.21.15.3:0/3945768000 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d04012db0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.060+0000 7f9cf17fa640 1 -- 172.21.15.3:0/3945768000 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0400f3c0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 -- 172.21.15.3:0/3945768000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 msgr2=0x555f1ccfa300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccfa300 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7f9d04011460 tx=0x7f9d04011920 comp rx=0 tx=0).stop 2024-02-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 -- 172.21.15.3:0/3945768000 shutdown_connections 2024-02-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccfa300 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccef330 0x555f1ccf78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3945768000 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cceed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 -- 172.21.15.3:0/3945768000 >> 172.21.15.3:0/3945768000 conn(0x555f1cce9050 msgr2=0x555f1cce9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 -- 172.21.15.3:0/3945768000 shutdown_connections 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.061+0000 7f9d1438e740 1 -- 172.21.15.3:0/3945768000 wait complete. 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 -- start start 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 Processor -- start 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccf4d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccef330 0x555f1ccf52c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccf5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cc57000 con 0x555f1ccf7eb0 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1c92a430 con 0x555f1ccef330 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1c8e6e00 con 0x555f1ccee980 2024-02-09T22:59:48.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d1438e740 10 client.0 _ll_register_callbacks cb 0x555f1cc4b7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccf5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccf5800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33804/0 (socket says 172.21.15.3:33804) 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3988235153 learned_addr learned my addr 172.21.15.3:0/3988235153 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccef330 0x555f1ccf52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0affd640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccf4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3988235153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 msgr2=0x555f1ccf4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccf4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccef330 msgr2=0x555f1ccf52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccef330 0x555f1ccf52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 -- 172.21.15.3:0/3988235153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d04009c90 con 0x555f1ccf7eb0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccf5800 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9d04000f80 tx=0x7f9d0401a670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4e38c2f3a882b8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.062+0000 7f9ceeff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9ceeff5640 1 -- 172.21.15.3:0/3988235153 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d04004c50 con 0x555f1ccf7eb0 2024-02-09T22:59:48.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9ceeff5640 1 -- 172.21.15.3:0/3988235153 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d04004db0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555f1c9567e0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9ceeff5640 1 -- 172.21.15.3:0/3988235153 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d04012570 con 0x555f1ccf7eb0 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 10 client.18684 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555f1cc55680 con 0x555f1ccf7eb0 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9ceeff5640 1 -- 172.21.15.3:0/3988235153 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555f1c9567e0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9ceeff5640 1 -- 172.21.15.3:0/3988235153 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9d04005620 con 0x555f1ccf7eb0 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 10 client.18684 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 10 client.18684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 20 client.18684 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd08240 0x555f1cd0a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 4 client.18684 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.063+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f1c993720 con 0x555f1cd08240 2024-02-09T22:59:48.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.064+0000 7f9d0affd640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd08240 0x555f1cd0a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.064+0000 7f9d0affd640 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd08240 0x555f1cd0a630 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a243e92a27f497c6 server_cookie=2495237864435182 in_seq=0 out_seq=0 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.064+0000 7f9ceeff5640 10 client.18684 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9ceeff5640 1 -- 172.21.15.3:0/3988235153 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555f1c993720 con 0x555f1cd08240 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9ceeff5640 10 client.18684 handle_command_reply: tid=0 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 client.18684 shutdown 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 msgr2=0x555f1ccf5800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccf5800 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7f9d04000f80 tx=0x7f9d0401a670 comp rx=0 tx=0).stop 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 shutdown_connections 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd08240 0x555f1cd0a630 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccef330 0x555f1ccf52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccf7eb0 0x555f1ccf5800 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9ceeff5640 0 client.18684 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 --2- 172.21.15.3:0/3988235153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccf4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 >> 172.21.15.3:0/3988235153 conn(0x555f1cce9050 msgr2=0x555f1cceada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 shutdown_connections 2024-02-09T22:59:48.073 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 1 -- 172.21.15.3:0/3988235153 wait complete. 2024-02-09T22:59:48.074 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.065+0000 7f9d1438e740 20 client.18684 trim_cache size 0 max 16384 2024-02-09T22:59:48.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 Processor -- start 2024-02-09T22:59:48.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 -- start start 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cce8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccf7eb0 0x555f1ccff420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cbdbfd0 con 0x555f1ccef330 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1ccf7c90 con 0x555f1ccee980 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d1438e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cd0f7f0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cce8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfeee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33820/0 (socket says 172.21.15.3:33820) 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/464005541 learned_addr learned my addr 172.21.15.3:0/464005541 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0affd640 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cce8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37504/0 (socket says 172.21.15.3:37504) 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccf7eb0 0x555f1ccff420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/464005541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccf7eb0 msgr2=0x555f1ccff420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccf7eb0 0x555f1ccff420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/464005541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 msgr2=0x555f1cce8070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cce8070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.081+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/464005541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f1cbf1b30 con 0x555f1ccef330 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.082+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfeee0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x555f1ccee450 tx=0x7f9d00014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd3daad48241925a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.082+0000 7f9cf0ff9640 1 -- 172.21.15.3:0/464005541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00025070 con 0x555f1ccef330 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.082+0000 7f9cf0ff9640 1 -- 172.21.15.3:0/464005541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d00016070 con 0x555f1ccef330 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.082+0000 7f9cf0ff9640 1 -- 172.21.15.3:0/464005541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00029ac0 con 0x555f1ccef330 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.082+0000 7f9d1438e740 1 -- 172.21.15.3:0/464005541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 msgr2=0x555f1ccfeee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.082+0000 7f9d1438e740 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfeee0 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x555f1ccee450 tx=0x7f9d00014000 comp rx=0 tx=0).stop 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- 172.21.15.3:0/464005541 shutdown_connections 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccf7eb0 0x555f1ccff420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfeee0 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 --2- 172.21.15.3:0/464005541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccee980 0x555f1cce8070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- 172.21.15.3:0/464005541 >> 172.21.15.3:0/464005541 conn(0x555f1cce9050 msgr2=0x555f1cce7960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- 172.21.15.3:0/464005541 shutdown_connections 2024-02-09T22:59:48.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- 172.21.15.3:0/464005541 wait complete. 2024-02-09T22:59:48.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- start start 2024-02-09T22:59:48.088 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 Processor -- start 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccfe1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccf7eb0 0x555f1cd58730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cd58c70 con 0x555f1ccef330 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cd58de0 con 0x555f1ccf7eb0 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f1cd58fa0 con 0x555f1ccee980 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d1438e740 10 client.0 _ll_register_callbacks cb 0x7f9d0c06aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d0a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d0a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33832/0 (socket says 172.21.15.3:33832) 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/1710202419 learned_addr learned my addr 172.21.15.3:0/1710202419 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d0affd640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccfe1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.083+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccf7eb0 0x555f1cd58730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/1710202419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 msgr2=0x555f1ccfe1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccfe1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccf7eb0 msgr2=0x555f1cd58730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccf7eb0 0x555f1cd58730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0a7fc640 1 -- 172.21.15.3:0/1710202419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d00012450 con 0x555f1ccef330 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0b7fe640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccf7eb0 0x555f1cd58730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d0a7fc640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfe730 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f9d0002c040 tx=0x7f9d00005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97350751d8157a59 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9ceeff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9ceeff5640 1 -- 172.21.15.3:0/1710202419 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d00025040 con 0x555f1ccef330 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d04012db0 con 0x555f1ccef330 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9ceeff5640 1 -- 172.21.15.3:0/1710202419 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7f9d00018070 con 0x555f1ccef330 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9ceeff5640 1 -- 172.21.15.3:0/1710202419 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d00029410 con 0x555f1ccef330 2024-02-09T22:59:48.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d1438e740 10 client.18690 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555f1cc17850 con 0x555f1ccef330 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.084+0000 7f9ceeff5640 1 -- 172.21.15.3:0/1710202419 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d04012db0 con 0x555f1ccef330 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9ceeff5640 1 -- 172.21.15.3:0/1710202419 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f9d00030030 con 0x555f1ccef330 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 10 client.18690 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 10 client.18690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 20 client.18690 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd607b0 0x555f1cd62bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 4 client.18690 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x555f1c9e2890 con 0x555f1cd607b0 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d0affd640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd607b0 0x555f1cd62bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9ceeff5640 10 client.18690 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d0affd640 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd607b0 0x555f1cd62bd0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a879628750844a06 server_cookie=71bdf2d1142ae054 in_seq=0 out_seq=0 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9ceeff5640 1 -- 172.21.15.3:0/1710202419 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x555f1c9e2890 con 0x555f1cd607b0 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9ceeff5640 10 client.18690 handle_command_reply: tid=0 2024-02-09T22:59:48.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 client.18690 shutdown 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 msgr2=0x555f1ccfe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfe730 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f9d0002c040 tx=0x7f9d00005720 comp rx=0 tx=0).stop 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 shutdown_connections 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x555f1cd607b0 0x555f1cd62bd0 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555f1ccf7eb0 0x555f1cd58730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555f1ccef330 0x555f1ccfe730 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 --2- 172.21.15.3:0/1710202419 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555f1ccee980 0x555f1ccfe1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 >> 172.21.15.3:0/1710202419 conn(0x555f1cce9050 msgr2=0x555f1cce81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.085+0000 7f9ceeff5640 0 client.18690 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 shutdown_connections 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d1438e740 1 -- 172.21.15.3:0/1710202419 wait complete. 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d1438e740 20 client.18690 trim_cache size 0 max 16384 2024-02-09T22:59:48.094 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: "8de2f341-e18f-45ab-9a21-9022d1a3cba1": { 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "8de2f341-e18f-45ab-9a21-9022d1a3cba1", 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T22:59:48.095 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9cf40a66c0 msgr2=0x7f9cf40a8b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9cf40a66c0 0x7f9cf40a8b80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c105030 tx=0x7f9d00009260 comp rx=0 tx=0).stop 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 msgr2=0x7f9d0c19d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c19d680 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc037040 tx=0x7f9cfc0045b0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 shutdown_connections 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9cf40a66c0 0x7f9cf40a8b80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d0c1977f0 0x7f9d0c197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d0c105410 0x7f9d0c19dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 --2- 172.21.15.3:0/1177851066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d0c100340 0x7f9d0c19d680 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 >> 172.21.15.3:0/1177851066 conn(0x7f9d0c0772e0 msgr2=0x7f9d0c068a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 shutdown_connections 2024-02-09T22:59:48.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T22:59:48.086+0000 7f9d117bc640 1 -- 172.21.15.3:0/1177851066 wait complete. 2024-02-09T22:59:48.100 INFO:tasks.cephfs.filesystem:scrub status for tag:8de2f341-e18f-45ab-9a21-9022d1a3cba1 - {'path': '/', 'tag': '8de2f341-e18f-45ab-9a21-9022d1a3cba1', 'options': 'recursive,force'} 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: pgmap v265: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1011 KiB/s wr, 79 op/s 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:48.382 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:48.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:48.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.383 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:48 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 246478 2.64 MB/sec execute 342 sec latency 70.942 ms 2024-02-09T22:59:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: pgmap v265: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1011 KiB/s wr, 79 op/s 2024-02-09T22:59:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:48 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: pgmap v265: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 1011 KiB/s wr, 79 op/s 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:48.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:48 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T22:59:49.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.635 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.636 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:49 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 246478 2.63 MB/sec execute 343 sec latency 1071.038 ms 2024-02-09T22:59:49.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:49 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T22:59:49.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:49 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T22:59:49.753 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 -- 172.21.15.3:0/1001351651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078101e00 msgr2=0x7ff078105e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 --2- 172.21.15.3:0/1001351651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078101e00 0x7ff078105e40 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009930 tx=0x7ff06c02bc30 comp rx=0 tx=0).stop 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 -- 172.21.15.3:0/1001351651 shutdown_connections 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 --2- 172.21.15.3:0/1001351651 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078106520 0x7ff07810e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 --2- 172.21.15.3:0/1001351651 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078101e00 0x7ff078105e40 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 --2- 172.21.15.3:0/1001351651 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078101450 0x7ff078101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 -- 172.21.15.3:0/1001351651 >> 172.21.15.3:0/1001351651 conn(0x7ff0780fccb0 msgr2=0x7ff0780ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 -- 172.21.15.3:0/1001351651 shutdown_connections 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.750+0000 7ff07de8d640 1 -- 172.21.15.3:0/1001351651 wait complete. 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 Processor -- start 2024-02-09T22:59:49.754 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 -- start start 2024-02-09T22:59:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078101450 0x7ff078073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078106520 0x7ff078073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 0x7ff0781a5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0781a6120 con 0x7ff078073fb0 2024-02-09T22:59:49.756 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0781a6290 con 0x7ff078101450 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff07de8d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0781a6450 con 0x7ff078106520 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff077fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 0x7ff0781a5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff077fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 0x7ff0781a5bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33856/0 (socket says 172.21.15.3:33856) 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff077fff640 1 -- 172.21.15.3:0/984433200 learned_addr learned my addr 172.21.15.3:0/984433200 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff0777fe640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078101450 0x7ff078073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.751+0000 7ff076ffd640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078106520 0x7ff078073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff077fff640 1 -- 172.21.15.3:0/984433200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078106520 msgr2=0x7ff078073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff077fff640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078106520 0x7ff078073a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff077fff640 1 -- 172.21.15.3:0/984433200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078101450 msgr2=0x7ff078073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff077fff640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078101450 0x7ff078073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff077fff640 1 -- 172.21.15.3:0/984433200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff06c009560 con 0x7ff078073fb0 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff0777fe640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078101450 0x7ff078073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:49.757 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff076ffd640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078106520 0x7ff078073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff077fff640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 0x7ff0781a5bb0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff07000cad0 tx=0x7ff07000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff074ff9640 1 -- 172.21.15.3:0/984433200 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000ae20 con 0x7ff078073fb0 2024-02-09T22:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff074ff9640 1 -- 172.21.15.3:0/984433200 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff070005020 con 0x7ff078073fb0 2024-02-09T22:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0781a6730 con 0x7ff078073fb0 2024-02-09T22:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff074ff9640 1 -- 172.21.15.3:0/984433200 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07000f6b0 con 0x7ff078073fb0 2024-02-09T22:59:49.758 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.752+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0781a6c10 con 0x7ff078073fb0 2024-02-09T22:59:49.764 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.754+0000 7ff074ff9640 1 -- 172.21.15.3:0/984433200 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff070005520 con 0x7ff078073fb0 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 Processor -- start 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff074ff9640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0600a22a0 0x7ff0600a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 -- start start 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff076ffd640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0600a22a0 0x7ff0600a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07620b0 0x5632b0762490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff074ff9640 1 -- 172.21.15.3:0/984433200 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff0700ce0e0 con 0x7ff078073fb0 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 0x5632b076b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b076b7a0 0x5632b076db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b06d0a00 con 0x5632b07629d0 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b06cdf30 con 0x5632b07620b0 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff080a59740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b06c9710 con 0x5632b076b7a0 2024-02-09T22:59:49.765 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff076ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 0x5632b076b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff076ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 0x5632b076b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33872/0 (socket says 172.21.15.3:33872) 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff076ffd640 1 -- 172.21.15.3:0/3128089466 learned_addr learned my addr 172.21.15.3:0/3128089466 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff0777fe640 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07620b0 0x5632b0762490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff077fff640 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b076b7a0 0x5632b076db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff076ffd640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0600a22a0 0x7ff0600a4760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff06c000c00 tx=0x7ff06c004bd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.755+0000 7ff076ffd640 1 -- 172.21.15.3:0/3128089466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b076b7a0 msgr2=0x5632b076db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff076ffd640 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b076b7a0 0x5632b076db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff076ffd640 1 -- 172.21.15.3:0/3128089466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07620b0 msgr2=0x5632b0762490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff076ffd640 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07620b0 0x5632b0762490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff076ffd640 1 -- 172.21.15.3:0/3128089466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632b068e3b0 con 0x5632b07629d0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff076ffd640 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 0x5632b076b260 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009930 tx=0x7ff06c04a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8162e59202126277 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff05dffb640 1 -- 172.21.15.3:0/3128089466 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c004df0 con 0x5632b07629d0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff05dffb640 1 -- 172.21.15.3:0/3128089466 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff06c005430 con 0x5632b07629d0 2024-02-09T22:59:49.766 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.756+0000 7ff05dffb640 1 -- 172.21.15.3:0/3128089466 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff06c003c70 con 0x5632b07629d0 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- 172.21.15.3:0/3128089466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 msgr2=0x5632b076b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 0x5632b076b260 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7ff06c009930 tx=0x7ff06c04a000 comp rx=0 tx=0).stop 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- 172.21.15.3:0/3128089466 shutdown_connections 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b076b7a0 0x5632b076db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07629d0 0x5632b076b260 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- 172.21.15.3:0/3128089466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07620b0 0x5632b0762490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- 172.21.15.3:0/3128089466 >> 172.21.15.3:0/3128089466 conn(0x5632b075c9d0 msgr2=0x5632b075cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- 172.21.15.3:0/3128089466 shutdown_connections 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- 172.21.15.3:0/3128089466 wait complete. 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- start start 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 Processor -- start 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0768790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.767 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0768cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0769210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b066eb90 con 0x5632b07620b0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.757+0000 7ff080a59740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b039d250 con 0x5632b076b7a0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff080a59740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b064f200 con 0x5632b07629d0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff080a59740 10 client.0 _ll_register_callbacks cb 0x5632b06855d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0768790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0768790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33876/0 (socket says 172.21.15.3:33876) 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 -- 172.21.15.3:0/2109525197 learned_addr learned my addr 172.21.15.3:0/2109525197 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff076ffd640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0768cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff077fff640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0769210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 -- 172.21.15.3:0/2109525197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 msgr2=0x5632b0768cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0768cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.768 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 -- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 msgr2=0x5632b0769210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0769210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 -- 172.21.15.3:0/2109525197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff07810e900 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff076ffd640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0768cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff0777fe640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0768790 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7ff068012480 tx=0x7ff068012940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e414abd881fea25 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff056ffd640 1 -- 172.21.15.3:0/2109525197 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068010aa0 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff056ffd640 1 -- 172.21.15.3:0/2109525197 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff068010c00 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5632b06efee0 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.758+0000 7ff056ffd640 1 -- 172.21.15.3:0/2109525197 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff068010dc0 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 10 client.18702 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5632b06c7d90 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff056ffd640 1 -- 172.21.15.3:0/2109525197 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5632b06efee0 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff056ffd640 1 -- 172.21.15.3:0/2109525197 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff06800f450 con 0x5632b07620b0 2024-02-09T22:59:49.769 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 10 client.18702 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 10 client.18702 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 20 client.18702 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b077ba10 0x5632b077de50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 4 client.18702 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632b03c9600 con 0x5632b077ba10 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.759+0000 7ff076ffd640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b077ba10 0x5632b077de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff056ffd640 10 client.18702 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff076ffd640 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b077ba10 0x5632b077de50 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2eb7640559326d4f server_cookie=844b1c8bad53f611 in_seq=0 out_seq=0 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff056ffd640 1 -- 172.21.15.3:0/2109525197 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5632b03c9600 con 0x5632b077ba10 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff056ffd640 10 client.18702 handle_command_reply: tid=0 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff080a59740 1 client.18702 shutdown 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 msgr2=0x5632b0768790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.770 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.760+0000 7ff080a59740 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0768790 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7ff068012480 tx=0x7ff068012940 comp rx=0 tx=0).stop 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 shutdown_connections 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b077ba10 0x5632b077de50 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0769210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0768cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 --2- 172.21.15.3:0/2109525197 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0768790 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff056ffd640 0 client.18702 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 >> 172.21.15.3:0/2109525197 conn(0x5632b075c9d0 msgr2=0x5632b075e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 shutdown_connections 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 1 -- 172.21.15.3:0/2109525197 wait complete. 2024-02-09T22:59:49.771 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.761+0000 7ff080a59740 20 client.18702 trim_cache size 0 max 16384 2024-02-09T22:59:49.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 Processor -- start 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 -- start start 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07620b0 0x5632b076aec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07629d0 0x5632b0768d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 0x5632b0769240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b075c860 con 0x5632b076b7a0 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b075d7d0 con 0x5632b07629d0 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff080a59740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b075d9d0 con 0x5632b07620b0 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff077fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 0x5632b0769240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff077fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 0x5632b0769240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33878/0 (socket says 172.21.15.3:33878) 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.774+0000 7ff077fff640 1 -- 172.21.15.3:0/2425324382 learned_addr learned my addr 172.21.15.3:0/2425324382 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff076ffd640 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07629d0 0x5632b0768d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff0777fe640 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07620b0 0x5632b076aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff077fff640 1 -- 172.21.15.3:0/2425324382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07620b0 msgr2=0x5632b076aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff077fff640 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07620b0 0x5632b076aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff077fff640 1 -- 172.21.15.3:0/2425324382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07629d0 msgr2=0x5632b0768d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff077fff640 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07629d0 0x5632b0768d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff077fff640 1 -- 172.21.15.3:0/2425324382 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632b0664820 con 0x5632b076b7a0 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff0777fe640 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07620b0 0x5632b076aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff077fff640 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 0x5632b0769240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7ff07009aa30 tx=0x7ff07009a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d310444a55afaf4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff05d7fa640 1 -- 172.21.15.3:0/2425324382 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff07009ebf0 con 0x5632b076b7a0 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff05d7fa640 1 -- 172.21.15.3:0/2425324382 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0700a3240 con 0x5632b076b7a0 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff05d7fa640 1 -- 172.21.15.3:0/2425324382 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff070099ef0 con 0x5632b076b7a0 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff080a59740 1 -- 172.21.15.3:0/2425324382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 msgr2=0x5632b0769240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.775+0000 7ff080a59740 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 0x5632b0769240 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7ff07009aa30 tx=0x7ff07009a400 comp rx=0 tx=0).stop 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- 172.21.15.3:0/2425324382 shutdown_connections 2024-02-09T22:59:49.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b07629d0 0x5632b0768d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b076b7a0 0x5632b0769240 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 --2- 172.21.15.3:0/2425324382 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07620b0 0x5632b076aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- 172.21.15.3:0/2425324382 >> 172.21.15.3:0/2425324382 conn(0x5632b075c9d0 msgr2=0x5632b0767080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- 172.21.15.3:0/2425324382 shutdown_connections 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- 172.21.15.3:0/2425324382 wait complete. 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- start start 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 Processor -- start 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0771030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0771570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0771ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b07d2020 con 0x5632b07620b0 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b07d2190 con 0x5632b076b7a0 2024-02-09T22:59:49.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b07d2350 con 0x5632b07629d0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff080a59740 10 client.0 _ll_register_callbacks cb 0x7ff06800b8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff0777fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0771030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.776+0000 7ff0777fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0771030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:33888/0 (socket says 172.21.15.3:33888) 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 -- 172.21.15.3:0/208619306 learned_addr learned my addr 172.21.15.3:0/208619306 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff076ffd640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0771570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff077fff640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0771ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 -- 172.21.15.3:0/208619306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 msgr2=0x5632b0771570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0771570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 -- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 msgr2=0x5632b0771ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0771ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 -- 172.21.15.3:0/208619306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0781a6c10 con 0x5632b07620b0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff0777fe640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0771030 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff068012450 tx=0x7ff068012e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c59a523d16b457d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff056ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T22:59:49.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff056ffd640 1 -- 172.21.15.3:0/208619306 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff068010aa0 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff056ffd640 1 -- 172.21.15.3:0/208619306 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 8469+0+0 (secure 0 0 0) 0x7ff068000e40 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff06c005430 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff056ffd640 1 -- 172.21.15.3:0/208619306 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff06800f5d0 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff080a59740 10 client.18708 fetch_fsmap learned FSMap version 47 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.777+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5632b068a6f0 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff056ffd640 1 -- 172.21.15.3:0/208619306 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff06c005430 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff056ffd640 1 -- 172.21.15.3:0/208619306 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff0680047a0 con 0x5632b07620b0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff080a59740 10 client.18708 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff080a59740 10 client.18708 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff080a59740 20 client.18708 populate_metadata read hostname 'smithi003' 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff080a59740 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b07d3af0 0x5632b07d5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff080a59740 4 client.18708 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5632b041a1b0 con 0x5632b07d3af0 2024-02-09T22:59:49.786 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff076ffd640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b07d3af0 0x5632b07d5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T22:59:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff076ffd640 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b07d3af0 0x5632b07d5f10 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d54c66f7c10f55 server_cookie=c90a74a476b943c2 in_seq=0 out_seq=0 2024-02-09T22:59:49.787 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.778+0000 7ff056ffd640 10 client.18708 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff056ffd640 1 -- 172.21.15.3:0/208619306 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5632b041a1b0 con 0x5632b07d3af0 2024-02-09T22:59:49.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff056ffd640 10 client.18708 handle_command_reply: tid=0 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 client.18708 shutdown 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 msgr2=0x5632b0771030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0771030 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7ff068012450 tx=0x7ff068012e40 comp rx=0 tx=0).stop 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 shutdown_connections 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5632b07d3af0 0x5632b07d5f10 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632b076b7a0 0x5632b0771ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632b07629d0 0x5632b0771570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 --2- 172.21.15.3:0/208619306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632b07620b0 0x5632b0771030 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff056ffd640 0 client.18708 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 >> 172.21.15.3:0/208619306 conn(0x5632b075c9d0 msgr2=0x5632b075b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:49.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 shutdown_connections 2024-02-09T22:59:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 1 -- 172.21.15.3:0/208619306 wait complete. 2024-02-09T22:59:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.812+0000 7ff080a59740 20 client.18708 trim_cache size 0 max 16384 2024-02-09T22:59:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T22:59:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T22:59:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T22:59:49.817 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0600a22a0 msgr2=0x7ff0600a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0600a22a0 0x7ff0600a4760 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ff06c000c00 tx=0x7ff06c004bd0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 msgr2=0x7ff0781a5bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 0x7ff0781a5bb0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7ff07000cad0 tx=0x7ff07000cf90 comp rx=0 tx=0).stop 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 shutdown_connections 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0600a22a0 0x7ff0600a4760 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.818 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff078073fb0 0x7ff0781a5bb0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff078106520 0x7ff078073a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 --2- 172.21.15.3:0/984433200 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff078101450 0x7ff078073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T22:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 >> 172.21.15.3:0/984433200 conn(0x7ff0780fccb0 msgr2=0x7ff0780fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T22:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 shutdown_connections 2024-02-09T22:59:49.819 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T22:59:49.813+0000 7ff07de8d640 1 -- 172.21.15.3:0/984433200 wait complete. 2024-02-09T22:59:50.627 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:50 smithi089 ceph-mon[36665]: pgmap v266: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 711 KiB/s wr, 59 op/s 2024-02-09T22:59:50.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:50 smithi003 ceph-mon[32678]: pgmap v266: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 711 KiB/s wr, 59 op/s 2024-02-09T22:59:50.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 248570 2.67 MB/sec execute 344 sec latency 1213.265 ms 2024-02-09T22:59:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:50 smithi064 ceph-mon[36638]: pgmap v266: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 711 KiB/s wr, 59 op/s 2024-02-09T22:59:51.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:51 smithi003 ceph-mon[32678]: pgmap v267: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-02-09T22:59:51.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 248996 2.66 MB/sec execute 345 sec latency 523.645 ms 2024-02-09T22:59:51.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:51 smithi064 ceph-mon[36638]: pgmap v267: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-02-09T22:59:51.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:51 smithi089 ceph-mon[36665]: pgmap v267: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-02-09T22:59:52.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 248996 2.65 MB/sec execute 346 sec latency 1523.748 ms 2024-02-09T22:59:53.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249052 2.65 MB/sec execute 347 sec latency 1923.592 ms 2024-02-09T22:59:53.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:53 smithi089 ceph-mon[36665]: pgmap v268: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 999 KiB/s wr, 113 op/s 2024-02-09T22:59:53.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:53 smithi003 ceph-mon[32678]: pgmap v268: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 999 KiB/s wr, 113 op/s 2024-02-09T22:59:53.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:53 smithi064 ceph-mon[36638]: pgmap v268: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 999 KiB/s wr, 113 op/s 2024-02-09T22:59:54.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249052 2.64 MB/sec execute 348 sec latency 1563.481 ms 2024-02-09T22:59:55.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249052 2.63 MB/sec execute 349 sec latency 2563.600 ms 2024-02-09T22:59:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:56 smithi003 ceph-mon[32678]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 999 KiB/s wr, 113 op/s 2024-02-09T22:59:56.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249052 2.62 MB/sec execute 350 sec latency 3563.667 ms 2024-02-09T22:59:56.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:56 smithi064 ceph-mon[36638]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 999 KiB/s wr, 113 op/s 2024-02-09T22:59:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:56 smithi089 ceph-mon[36665]: pgmap v269: 97 pgs: 97 active+clean; 97 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 736 KiB/s rd, 999 KiB/s wr, 113 op/s 2024-02-09T22:59:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:57.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:57 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:57.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249052 2.61 MB/sec execute 351 sec latency 4563.810 ms 2024-02-09T22:59:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:57.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:57 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:57 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:58.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:58 smithi064 ceph-mon[36638]: pgmap v270: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.4 MiB/s wr, 146 op/s 2024-02-09T22:59:58.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 22:59:58 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T22:59:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:58 smithi003 ceph-mon[32678]: pgmap v270: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.4 MiB/s wr, 146 op/s 2024-02-09T22:59:58.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 22:59:58 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T22:59:58.654 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249120 2.61 MB/sec execute 352 sec latency 4963.298 ms 2024-02-09T22:59:58.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:58 smithi089 ceph-mon[36665]: pgmap v270: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 924 KiB/s rd, 1.4 MiB/s wr, 146 op/s 2024-02-09T22:59:58.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 22:59:58 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T22:59:59.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249120 2.60 MB/sec execute 353 sec latency 1555.366 ms 2024-02-09T23:00:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:00 smithi003 ceph-mon[32678]: pgmap v271: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.0 MiB/s wr, 115 op/s 2024-02-09T23:00:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:00 smithi003 ceph-mon[32678]: overall HEALTH_OK 2024-02-09T23:00:00.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 249120 2.59 MB/sec execute 354 sec latency 2555.470 ms 2024-02-09T23:00:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:00 smithi064 ceph-mon[36638]: pgmap v271: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.0 MiB/s wr, 115 op/s 2024-02-09T23:00:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:00 smithi064 ceph-mon[36638]: overall HEALTH_OK 2024-02-09T23:00:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:00 smithi089 ceph-mon[36665]: pgmap v271: 97 pgs: 97 active+clean; 100 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.0 MiB/s wr, 115 op/s 2024-02-09T23:00:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:00 smithi089 ceph-mon[36665]: overall HEALTH_OK 2024-02-09T23:00:01.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 250021 2.60 MB/sec execute 355 sec latency 2607.195 ms 2024-02-09T23:00:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:02 smithi003 ceph-mon[32678]: pgmap v272: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-02-09T23:00:02.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 252170 2.63 MB/sec execute 356 sec latency 168.827 ms 2024-02-09T23:00:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:02 smithi064 ceph-mon[36638]: pgmap v272: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-02-09T23:00:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:02 smithi089 ceph-mon[36665]: pgmap v272: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 1.5 MiB/s wr, 138 op/s 2024-02-09T23:00:02.924 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.905+0000 7f67e8f9a640 1 -- 172.21.15.3:0/3931992374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e4107420 msgr2=0x7f67e4107800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.905+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/3931992374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e4107420 0x7f67e4107800 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f67d4009cd0 tx=0x7f67d402c180 comp rx=0 tx=0).stop 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 -- 172.21.15.3:0/3931992374 shutdown_connections 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/3931992374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e41004a0 0x7f67e410e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/3931992374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e40ffae0 0x7f67e40fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/3931992374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e4107420 0x7f67e4107800 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 -- 172.21.15.3:0/3931992374 >> 172.21.15.3:0/3931992374 conn(0x7f67e40753d0 msgr2=0x7f67e40757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 -- 172.21.15.3:0/3931992374 shutdown_connections 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 -- 172.21.15.3:0/3931992374 wait complete. 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 Processor -- start 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.906+0000 7f67e8f9a640 1 -- start start 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e8f9a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 0x7f67e419d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e8f9a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 0x7f67e419db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.925 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 0x7f67e419db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 0x7f67e419d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 0x7f67e419db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34530/0 (socket says 172.21.15.3:34530) 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 0x7f67e419d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:50744/0 (socket says 172.21.15.3:50744) 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e8f9a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e4107420 0x7f67e41976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e1d9b640 1 -- 172.21.15.3:0/1245199103 learned_addr learned my addr 172.21.15.3:0/1245199103 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e4197e50 con 0x7f67e41004a0 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e4197fc0 con 0x7f67e40ffae0 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e4198180 con 0x7f67e4107420 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e4107420 0x7f67e41976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 -- 172.21.15.3:0/1245199103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e4107420 msgr2=0x7f67e41976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e4107420 0x7f67e41976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 -- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 msgr2=0x7f67e419db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 0x7f67e419db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.926 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e259c640 1 -- 172.21.15.3:0/1245199103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d4009900 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 0x7f67e419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.907+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e4107420 0x7f67e41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.908+0000 7f67e259c640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 0x7f67e419d600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67d402c790 tx=0x7f67d4034760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.908+0000 7f67db7fe640 1 -- 172.21.15.3:0/1245199103 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d4003d80 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.908+0000 7f67db7fe640 1 -- 172.21.15.3:0/1245199103 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67d4032c30 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.908+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e4198400 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.908+0000 7f67db7fe640 1 -- 172.21.15.3:0/1245199103 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d40047f0 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.908+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e41a4580 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.910+0000 7f67db7fe640 1 -- 172.21.15.3:0/1245199103 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f67d400a040 con 0x7f67e40ffae0 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.910+0000 7f67ebb72740 1 Processor -- start 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 -- start start 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0be490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0c7260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.927 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c9b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a02ca00 con 0x55869a0be0b0 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a029f30 con 0x55869a0c77a0 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67ebb72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a025710 con 0x55869a0be9d0 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e2d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0be490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e2d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0be490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34542/0 (socket says 172.21.15.3:34542) 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e2d9d640 1 -- 172.21.15.3:0/4275151115 learned_addr learned my addr 172.21.15.3:0/4275151115 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e259c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:43492/0 (socket says 172.21.15.3:43492) 2024-02-09T23:00:02.928 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c9b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67db7fe640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67d00a2220 0x7f67d00a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e1d9b640 1 -- 172.21.15.3:0/4275151115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 msgr2=0x55869a0c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0c7260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e1d9b640 1 -- 172.21.15.3:0/4275151115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 msgr2=0x55869a0be490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0be490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e1d9b640 1 -- 172.21.15.3:0/4275151115 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558699fea3b0 con 0x55869a0c77a0 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e259c640 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67db7fe640 1 -- 172.21.15.3:0/1245199103 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f67d40f0c20 con 0x7f67e40ffae0 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.911+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67d00a2220 0x7f67d00a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c9b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f67cc010450 tx=0x7f67cc010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e7e45196e17e54 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67d8ff9640 1 -- 172.21.15.3:0/4275151115 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc015c30 con 0x55869a0c77a0 2024-02-09T23:00:02.929 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67d00a2220 0x7f67d00a46e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00b4e0 tx=0x7f67dc004730 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67d8ff9640 1 -- 172.21.15.3:0/4275151115 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67cc019070 con 0x55869a0c77a0 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67d8ff9640 1 -- 172.21.15.3:0/4275151115 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67cc01e5e0 con 0x55869a0c77a0 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67ebb72740 1 -- 172.21.15.3:0/4275151115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 msgr2=0x55869a0c9b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.912+0000 7f67ebb72740 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c9b90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f67cc010450 tx=0x7f67cc010910 comp rx=0 tx=0).stop 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- 172.21.15.3:0/4275151115 shutdown_connections 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c9b90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0c7260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 --2- 172.21.15.3:0/4275151115 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0be490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- 172.21.15.3:0/4275151115 >> 172.21.15.3:0/4275151115 conn(0x55869a0b89d0 msgr2=0x55869a0b8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- 172.21.15.3:0/4275151115 shutdown_connections 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- 172.21.15.3:0/4275151115 wait complete. 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- start start 2024-02-09T23:00:02.930 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 Processor -- start 2024-02-09T23:00:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0c4750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0be9d0 0x55869a0c4c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 0x55869a0c51d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558699fcab90 con 0x55869a0be0b0 2024-02-09T23:00:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558699cf9250 con 0x55869a0be9d0 2024-02-09T23:00:02.931 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558699fab200 con 0x55869a0c77a0 2024-02-09T23:00:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67ebb72740 10 client.0 _ll_register_callbacks cb 0x558699fe15d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67e259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67e2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 0x55869a0c51d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0be9d0 0x55869a0c4c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.932 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67e2d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 0x55869a0c51d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:43496/0 (socket says 172.21.15.3:43496) 2024-02-09T23:00:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67e259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34548/0 (socket says 172.21.15.3:34548) 2024-02-09T23:00:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.913+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2991000825 learned_addr learned my addr 172.21.15.3:0/2991000825 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0be9d0 msgr2=0x55869a0c4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0be9d0 0x55869a0c4c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2991000825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 msgr2=0x55869a0c4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.933 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0c4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2991000825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67cc00bcd0 con 0x55869a0c77a0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 0x55869a0c51d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f67dc012c10 tx=0x7f67dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa5c619e04aacd9c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2991000825 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc012e40 con 0x55869a0c77a0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2991000825 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f67dc00f450 con 0x55869a0c77a0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55869a04bee0 con 0x55869a0c77a0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2991000825 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67dc013e70 con 0x55869a0c77a0 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67ebb72740 10 client.27766 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:02.934 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55869a023d90 con 0x55869a0c77a0 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2991000825 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55869a04bee0 con 0x55869a0c77a0 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.914+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2991000825 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67dc0216e0 con 0x55869a0c77a0 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67ebb72740 10 client.27766 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67ebb72740 10 client.27766 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67ebb72740 20 client.27766 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a0d7ab0 0x55869a0d9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67ebb72740 4 client.27766 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558699d25600 con 0x55869a0d7ab0 2024-02-09T23:00:02.935 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67e259c640 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a0d7ab0 0x55869a0d9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67c9ffb640 10 client.27766 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.915+0000 7f67e259c640 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a0d7ab0 0x55869a0d9ed0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e859a79b0ab763a server_cookie=7d519997fb30f6ba in_seq=0 out_seq=0 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2991000825 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558699d25600 con 0x55869a0d7ab0 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67c9ffb640 10 client.27766 handle_command_reply: tid=0 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 client.27766 shutdown 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 msgr2=0x55869a0c51d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 0x55869a0c51d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f67dc012c10 tx=0x7f67dc0046c0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 shutdown_connections 2024-02-09T23:00:02.936 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a0d7ab0 0x55869a0d9ed0 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0be9d0 0x55869a0c4c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0c77a0 0x55869a0c51d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2991000825 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0c4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67c9ffb640 0 client.27766 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 >> 172.21.15.3:0/2991000825 conn(0x55869a0b89d0 msgr2=0x55869a0ba720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 shutdown_connections 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 1 -- 172.21.15.3:0/2991000825 wait complete. 2024-02-09T23:00:02.937 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.916+0000 7f67ebb72740 20 client.27766 trim_cache size 0 max 16384 2024-02-09T23:00:02.942 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 Processor -- start 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 -- start start 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be0b0 0x55869a0c4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 0x55869a0c3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a0b8500 con 0x55869a0be9d0 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558699d25600 con 0x55869a0c77a0 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67ebb72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a0b99d0 con 0x55869a0be0b0 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 0x55869a0c3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67e1d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 0x55869a0c3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34554/0 (socket says 172.21.15.3:34554) 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67e1d9b640 1 -- 172.21.15.3:0/2227614657 learned_addr learned my addr 172.21.15.3:0/2227614657 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.930+0000 7f67e259c640 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be0b0 0x55869a0c4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.943 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2227614657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be0b0 msgr2=0x55869a0c4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be0b0 0x55869a0c4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2227614657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 msgr2=0x55869a0c3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 0x55869a0c3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e2d9d640 1 -- 172.21.15.3:0/2227614657 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558699fc0820 con 0x55869a0c77a0 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 0x55869a0c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c3560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f67dc021ac0 tx=0x7f67dc012810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa698f6ac6e9b046 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67cbfff640 1 -- 172.21.15.3:0/2227614657 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc012c40 con 0x55869a0c77a0 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67cbfff640 1 -- 172.21.15.3:0/2227614657 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67dc012da0 con 0x55869a0c77a0 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.931+0000 7f67cbfff640 1 -- 172.21.15.3:0/2227614657 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67dc00fbd0 con 0x55869a0c77a0 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- 172.21.15.3:0/2227614657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 msgr2=0x55869a0c3560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c3560 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f67dc021ac0 tx=0x7f67dc012810 comp rx=0 tx=0).stop 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- 172.21.15.3:0/2227614657 shutdown_connections 2024-02-09T23:00:02.944 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0c3560 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be9d0 0x55869a0c3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2227614657 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be0b0 0x55869a0c4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- 172.21.15.3:0/2227614657 >> 172.21.15.3:0/2227614657 conn(0x55869a0b89d0 msgr2=0x55869a0bfa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- 172.21.15.3:0/2227614657 shutdown_connections 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- 172.21.15.3:0/2227614657 wait complete. 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- start start 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 Processor -- start 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0ce460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0ce9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0cca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a0cd130 con 0x55869a0be0b0 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a0cd2a0 con 0x55869a0c77a0 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.932+0000 7f67ebb72740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55869a0cd460 con 0x55869a0be9d0 2024-02-09T23:00:02.945 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67ebb72740 10 client.0 _ll_register_callbacks cb 0x7f67dc009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0ce460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0ce460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34568/0 (socket says 172.21.15.3:34568) 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 -- 172.21.15.3:0/2452175058 learned_addr learned my addr 172.21.15.3:0/2452175058 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0ce9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 -- 172.21.15.3:0/2452175058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 msgr2=0x55869a0ce9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0ce9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 -- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 msgr2=0x55869a0cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 -- 172.21.15.3:0/2452175058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558699fea3b0 con 0x55869a0be0b0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e2d9d640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67e259c640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0ce460 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f67d40ba0d0 tx=0x7f67d4034a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2572c3d08d8b458 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:02.946 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2452175058 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d40bd060 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2452175058 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f67d40bde00 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67cc019070 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2452175058 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67d40c1850 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67ebb72740 10 client.18726 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.933+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55869a023d90 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2452175058 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67cc019070 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2452175058 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67d40f6070 con 0x55869a0be0b0 2024-02-09T23:00:02.947 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67ebb72740 10 client.18726 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67ebb72740 10 client.18726 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67ebb72740 20 client.18726 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a12f090 0x55869a131480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67ebb72740 4 client.18726 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558699d761b0 con 0x55869a12f090 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a12f090 0x55869a131480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67e1d9b640 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a12f090 0x55869a131480 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82d3c82830a8f43 server_cookie=6981890bc894344a in_seq=0 out_seq=0 2024-02-09T23:00:02.948 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.934+0000 7f67c9ffb640 10 client.18726 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67c9ffb640 1 -- 172.21.15.3:0/2452175058 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558699d761b0 con 0x55869a12f090 2024-02-09T23:00:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67c9ffb640 10 client.18726 handle_command_reply: tid=0 2024-02-09T23:00:02.967 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 client.18726 shutdown 2024-02-09T23:00:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 msgr2=0x55869a0ce460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0ce460 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f67d40ba0d0 tx=0x7f67d4034a10 comp rx=0 tx=0).stop 2024-02-09T23:00:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 shutdown_connections 2024-02-09T23:00:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55869a12f090 0x55869a131480 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55869a0c77a0 0x55869a0cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55869a0be9d0 0x55869a0ce9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67c9ffb640 0 client.18726 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 --2- 172.21.15.3:0/2452175058 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55869a0be0b0 0x55869a0ce460 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.963+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 >> 172.21.15.3:0/2452175058 conn(0x55869a0b89d0 msgr2=0x55869a0c2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 shutdown_connections 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67ebb72740 1 -- 172.21.15.3:0/2452175058 wait complete. 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67ebb72740 20 client.18726 trim_cache size 0 max 16384 2024-02-09T23:00:02.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:00:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:00:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:00:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:00:02.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67d00a2220 msgr2=0x7f67d00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67d00a2220 0x7f67d00a46e0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67dc00b4e0 tx=0x7f67dc004730 comp rx=0 tx=0).stop 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 msgr2=0x7f67e419d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 0x7f67e419d600 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f67d402c790 tx=0x7f67d4034760 comp rx=0 tx=0).stop 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 shutdown_connections 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67d00a2220 0x7f67d00a46e0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f67e4107420 0x7f67e41976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67e41004a0 0x7f67e419db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 --2- 172.21.15.3:0/1245199103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67e40ffae0 0x7f67e419d600 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 >> 172.21.15.3:0/1245199103 conn(0x7f67e40753d0 msgr2=0x7f67e40fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 shutdown_connections 2024-02-09T23:00:02.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:02.964+0000 7f67e8f9a640 1 -- 172.21.15.3:0/1245199103 wait complete. 2024-02-09T23:00:03.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:03 smithi003 ceph-mon[32678]: osdmap e96: 12 total, 12 up, 12 in 2024-02-09T23:00:03.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 253438 2.64 MB/sec execute 357 sec latency 21.740 ms 2024-02-09T23:00:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:03 smithi064 ceph-mon[36638]: osdmap e96: 12 total, 12 up, 12 in 2024-02-09T23:00:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:03 smithi089 ceph-mon[36665]: osdmap e96: 12 total, 12 up, 12 in 2024-02-09T23:00:04.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:04 smithi003 ceph-mon[32678]: pgmap v274: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1023 KiB/s wr, 67 op/s 2024-02-09T23:00:04.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 253572 2.63 MB/sec execute 358 sec latency 927.442 ms 2024-02-09T23:00:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:04 smithi064 ceph-mon[36638]: pgmap v274: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1023 KiB/s wr, 67 op/s 2024-02-09T23:00:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:04 smithi089 ceph-mon[36665]: pgmap v274: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1023 KiB/s wr, 67 op/s 2024-02-09T23:00:05.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:05 smithi003 ceph-mon[32678]: pgmap v275: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1023 KiB/s wr, 67 op/s 2024-02-09T23:00:05.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 253572 2.62 MB/sec execute 359 sec latency 1927.524 ms 2024-02-09T23:00:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:05 smithi064 ceph-mon[36638]: pgmap v275: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1023 KiB/s wr, 67 op/s 2024-02-09T23:00:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:05 smithi089 ceph-mon[36665]: pgmap v275: 97 pgs: 97 active+clean; 101 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 1023 KiB/s wr, 67 op/s 2024-02-09T23:00:06.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 253572 2.62 MB/sec execute 360 sec latency 2927.637 ms 2024-02-09T23:00:07.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 255724 2.65 MB/sec execute 361 sec latency 2962.825 ms 2024-02-09T23:00:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:07 smithi089 ceph-mon[36665]: pgmap v276: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T23:00:07.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:07 smithi003 ceph-mon[32678]: pgmap v276: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T23:00:07.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:07 smithi064 ceph-mon[36638]: pgmap v276: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T23:00:08.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 257030 2.65 MB/sec execute 362 sec latency 14.911 ms 2024-02-09T23:00:09.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 257113 2.65 MB/sec execute 363 sec latency 922.035 ms 2024-02-09T23:00:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:10 smithi003 ceph-mon[32678]: pgmap v277: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T23:00:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:10 smithi003 ceph-mon[32678]: osdmap e97: 12 total, 12 up, 12 in 2024-02-09T23:00:10.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 258188 2.66 MB/sec execute 364 sec latency 1366.095 ms 2024-02-09T23:00:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:10 smithi064 ceph-mon[36638]: pgmap v277: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T23:00:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:10 smithi064 ceph-mon[36638]: osdmap e97: 12 total, 12 up, 12 in 2024-02-09T23:00:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:10 smithi089 ceph-mon[36665]: pgmap v277: 97 pgs: 97 active+clean; 95 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 163 op/s 2024-02-09T23:00:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:10 smithi089 ceph-mon[36665]: osdmap e97: 12 total, 12 up, 12 in 2024-02-09T23:00:11.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 259721 2.67 MB/sec execute 365 sec latency 14.329 ms 2024-02-09T23:00:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:12 smithi003 ceph-mon[32678]: pgmap v279: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 233 op/s 2024-02-09T23:00:12.655 INFO:tasks.workunit.client.0.smithi003.stdout: 1 260661 2.68 MB/sec execute 366 sec latency 481.872 ms 2024-02-09T23:00:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:12 smithi064 ceph-mon[36638]: pgmap v279: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 233 op/s 2024-02-09T23:00:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:12 smithi089 ceph-mon[36665]: pgmap v279: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.4 MiB/s wr, 233 op/s 2024-02-09T23:00:13.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 262156 2.69 MB/sec execute 367 sec latency 880.846 ms 2024-02-09T23:00:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:14 smithi003 ceph-mon[32678]: pgmap v280: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-02-09T23:00:14.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 263491 2.70 MB/sec execute 368 sec latency 29.666 ms 2024-02-09T23:00:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:14 smithi064 ceph-mon[36638]: pgmap v280: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-02-09T23:00:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:14 smithi089 ceph-mon[36665]: pgmap v280: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-02-09T23:00:15.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 264204 2.71 MB/sec execute 369 sec latency 478.149 ms 2024-02-09T23:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 -- 172.21.15.3:0/1718177184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8103c20 msgr2=0x7f6da81040d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 --2- 172.21.15.3:0/1718177184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8103c20 0x7f6da81040d0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009ca0 tx=0x7f6da002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 -- 172.21.15.3:0/1718177184 shutdown_connections 2024-02-09T23:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 --2- 172.21.15.3:0/1718177184 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8103c20 0x7f6da81040d0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 --2- 172.21.15.3:0/1718177184 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da8069010 0x7f6da8069470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.066 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 --2- 172.21.15.3:0/1718177184 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8106120 0x7f6da8068ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 -- 172.21.15.3:0/1718177184 >> 172.21.15.3:0/1718177184 conn(0x7f6da8077220 msgr2=0x7f6da8077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 -- 172.21.15.3:0/1718177184 shutdown_connections 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.060+0000 7f6daf756640 1 -- 172.21.15.3:0/1718177184 wait complete. 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 Processor -- start 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 -- start start 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da8069010 0x7f6da819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8103c20 0x7f6da819e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 0x7f6da81a2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da81a2d00 con 0x7f6da8106120 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da81a2e70 con 0x7f6da8103c20 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daf756640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6da81a3030 con 0x7f6da8069010 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 0x7f6da81a2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.067 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 0x7f6da81a2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49018/0 (socket says 172.21.15.3:49018) 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 -- 172.21.15.3:0/3039199679 learned_addr learned my addr 172.21.15.3:0/3039199679 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da8069010 0x7f6da819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daccf1640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8103c20 0x7f6da819e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 -- 172.21.15.3:0/3039199679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da8069010 msgr2=0x7f6da819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da8069010 0x7f6da819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 -- 172.21.15.3:0/3039199679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8103c20 msgr2=0x7f6da819e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8103c20 0x7f6da819e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6dadcf3640 1 -- 172.21.15.3:0/3039199679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da0009900 con 0x7f6da8106120 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.061+0000 7f6daccf1640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8103c20 0x7f6da819e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.062+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 0x7f6da81a2700 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f6da002c700 tx=0x7f6da0034b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.062+0000 7f6da67fc640 1 -- 172.21.15.3:0/3039199679 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da00328b0 con 0x7f6da8106120 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.062+0000 7f6da67fc640 1 -- 172.21.15.3:0/3039199679 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6da0003e50 con 0x7f6da8106120 2024-02-09T23:00:16.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.062+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6da81a32b0 con 0x7f6da8106120 2024-02-09T23:00:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.062+0000 7f6da67fc640 1 -- 172.21.15.3:0/3039199679 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da00048a0 con 0x7f6da8106120 2024-02-09T23:00:16.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.062+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6da81aab90 con 0x7f6da8106120 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.064+0000 7f6da67fc640 1 -- 172.21.15.3:0/3039199679 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6da000a040 con 0x7f6da8106120 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 Processor -- start 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 -- start start 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6da67fc640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d900a22a0 0x7f6d900a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490819490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 0x55c490822260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908227a0 0x55c490824b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490787a00 con 0x55c4908199d0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490784f30 con 0x55c4908227a0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6db240a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490780710 con 0x55c4908190b0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6da67fc640 1 -- 172.21.15.3:0/3039199679 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6da00f0960 con 0x7f6da8106120 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6daccf1640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d900a22a0 0x7f6d900a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6daccf1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 0x55c490822260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6daccf1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 0x55c490822260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49022/0 (socket says 172.21.15.3:49022) 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6daccf1640 1 -- 172.21.15.3:0/3003861356 learned_addr learned my addr 172.21.15.3:0/3003861356 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490819490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.065+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908227a0 0x55c490824b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6dad4f2640 1 -- 172.21.15.3:0/3003861356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908227a0 msgr2=0x55c490824b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908227a0 0x55c490824b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6dad4f2640 1 -- 172.21.15.3:0/3003861356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 msgr2=0x55c490822260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 0x55c490822260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6dad4f2640 1 -- 172.21.15.3:0/3003861356 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4907453b0 con 0x55c4908190b0 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6daccf1640 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 0x55c490822260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6daccf1640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d900a22a0 0x7f6d900a4760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c014a10 tx=0x7f6d9c00bc90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490819490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009c20 tx=0x7f6d98011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91e8ca204cbb701b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6d8f7fe640 1 -- 172.21.15.3:0/3003861356 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98004050 con 0x55c4908190b0 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6d8f7fe640 1 -- 172.21.15.3:0/3003861356 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d98005500 con 0x55c4908190b0 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.066+0000 7f6d8f7fe640 1 -- 172.21.15.3:0/3003861356 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d98004b80 con 0x55c4908190b0 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 -- 172.21.15.3:0/3003861356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 msgr2=0x55c490819490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490819490 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6d98009c20 tx=0x7f6d98011810 comp rx=0 tx=0).stop 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 -- 172.21.15.3:0/3003861356 shutdown_connections 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908227a0 0x55c490824b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908199d0 0x55c490822260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.076 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 --2- 172.21.15.3:0/3003861356 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490819490 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 -- 172.21.15.3:0/3003861356 >> 172.21.15.3:0/3003861356 conn(0x55c4908139d0 msgr2=0x55c490813db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 -- 172.21.15.3:0/3003861356 shutdown_connections 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 -- 172.21.15.3:0/3003861356 wait complete. 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 -- start start 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.067+0000 7f6db240a740 1 Processor -- start 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908190b0 0x55c49081f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908199d0 0x55c49081fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c490820210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490725b90 con 0x55c4908227a0 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490454250 con 0x55c4908190b0 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490706200 con 0x55c4908199d0 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6db240a740 10 client.0 _ll_register_callbacks cb 0x55c49073c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6dadcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c490820210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.077 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908199d0 0x55c49081fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6dadcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c490820210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49032/0 (socket says 172.21.15.3:49032) 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6dadcf3640 1 -- 172.21.15.3:0/2606581316 learned_addr learned my addr 172.21.15.3:0/2606581316 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908190b0 0x55c49081f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 -- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908190b0 msgr2=0x55c49081f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908190b0 0x55c49081f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 -- 172.21.15.3:0/2606581316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 msgr2=0x55c490820210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c490820210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 -- 172.21.15.3:0/2606581316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d98009850 con 0x55c4908199d0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c490820210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6daccf1640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908199d0 0x55c49081fcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55c490818b60 tx=0x7f6d9c005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed0fef29ae0123cc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.068+0000 7f6d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6d8cff9640 1 -- 172.21.15.3:0/2606581316 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9c005020 con 0x55c4908199d0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6d8cff9640 1 -- 172.21.15.3:0/2606581316 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6d9c005180 con 0x55c4908199d0 2024-02-09T23:00:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c4907a6ee0 con 0x55c4908199d0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6d8cff9640 1 -- 172.21.15.3:0/2606581316 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6d9c005340 con 0x55c4908199d0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 10 client.27784 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55c49077ed90 con 0x55c4908199d0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6d8cff9640 1 -- 172.21.15.3:0/2606581316 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c4907a6ee0 con 0x55c4908199d0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6d8cff9640 1 -- 172.21.15.3:0/2606581316 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6d9c033510 con 0x55c4908199d0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 10 client.27784 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 10 client.27784 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 20 client.27784 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c4908329f0 0x55c490834e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 4 client.27784 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.069+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c490480600 con 0x55c4908329f0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.070+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c4908329f0 0x55c490834e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.070+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c4908329f0 0x55c490834e10 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46cb50a3810a48fa server_cookie=ff9d214c4b799df2 in_seq=0 out_seq=0 2024-02-09T23:00:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.070+0000 7f6d8cff9640 10 client.27784 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6d8cff9640 1 -- 172.21.15.3:0/2606581316 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c490480600 con 0x55c4908329f0 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6d8cff9640 10 client.27784 handle_command_reply: tid=0 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 client.27784 shutdown 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908199d0 msgr2=0x55c49081fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908199d0 0x55c49081fcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x55c490818b60 tx=0x7f6d9c005850 comp rx=0 tx=0).stop 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 shutdown_connections 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c4908329f0 0x55c490834e10 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c490820210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908199d0 0x55c49081fcd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 --2- 172.21.15.3:0/2606581316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908190b0 0x55c49081f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.081 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6d8cff9640 0 client.27784 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 >> 172.21.15.3:0/2606581316 conn(0x55c4908139d0 msgr2=0x55c490815720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 shutdown_connections 2024-02-09T23:00:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 1 -- 172.21.15.3:0/2606581316 wait complete. 2024-02-09T23:00:16.082 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.071+0000 7f6db240a740 20 client.27784 trim_cache size 0 max 16384 2024-02-09T23:00:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 Processor -- start 2024-02-09T23:00:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 -- start start 2024-02-09T23:00:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 0x55c490821ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.090 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490820030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908227a0 0x55c49081c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490480600 con 0x55c4908190b0 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4907a6ee0 con 0x55c4908199d0 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6db240a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c49082f460 con 0x55c4908227a0 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 0x55c490821ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 0x55c490821ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49048/0 (socket says 172.21.15.3:49048) 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 -- 172.21.15.3:0/623243914 learned_addr learned my addr 172.21.15.3:0/623243914 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908227a0 0x55c49081c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6daccf1640 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490820030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 -- 172.21.15.3:0/623243914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908227a0 msgr2=0x55c49081c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908227a0 0x55c49081c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 -- 172.21.15.3:0/623243914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 msgr2=0x55c490820030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490820030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.091 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 -- 172.21.15.3:0/623243914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c49071b820 con 0x55c4908190b0 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6daccf1640 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490820030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.085+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 0x55c490821ec0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f6d98000c00 tx=0x7f6d98009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ee53475781b14df server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6d8effd640 1 -- 172.21.15.3:0/623243914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9802a040 con 0x55c4908190b0 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6d8effd640 1 -- 172.21.15.3:0/623243914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d98011d10 con 0x55c4908190b0 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6d8effd640 1 -- 172.21.15.3:0/623243914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d9802d900 con 0x55c4908190b0 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 -- 172.21.15.3:0/623243914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 msgr2=0x55c490821ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 0x55c490821ec0 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7f6d98000c00 tx=0x7f6d98009680 comp rx=0 tx=0).stop 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 -- 172.21.15.3:0/623243914 shutdown_connections 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490820030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908227a0 0x55c49081c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 --2- 172.21.15.3:0/623243914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908190b0 0x55c490821ec0 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 -- 172.21.15.3:0/623243914 >> 172.21.15.3:0/623243914 conn(0x55c4908139d0 msgr2=0x55c49081e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 -- 172.21.15.3:0/623243914 shutdown_connections 2024-02-09T23:00:16.092 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 -- 172.21.15.3:0/623243914 wait complete. 2024-02-09T23:00:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.086+0000 7f6db240a740 1 -- start start 2024-02-09T23:00:16.093 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 Processor -- start 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490829970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490827d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c4908284a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490828b70 con 0x55c4908227a0 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490888f00 con 0x55c4908199d0 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c490889040 con 0x55c4908190b0 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6db240a740 10 client.0 _ll_register_callbacks cb 0x7f6d9c0119e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c4908284a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dad4f2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490829970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.096 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c4908284a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49060/0 (socket says 172.21.15.3:49060) 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 -- 172.21.15.3:0/1797414297 learned_addr learned my addr 172.21.15.3:0/1797414297 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6daccf1640 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490827d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 -- 172.21.15.3:0/1797414297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 msgr2=0x55c490829970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490829970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 -- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 msgr2=0x55c490827d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490827d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.087+0000 7f6dadcf3640 1 -- 172.21.15.3:0/1797414297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c4907453b0 con 0x55c4908227a0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6dadcf3640 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c4908284a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009430 tx=0x7f6da0032c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb62c29b14255563 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6d8cff9640 1 -- 172.21.15.3:0/1797414297 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6da00c15b0 con 0x55c4908227a0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6d8cff9640 1 -- 172.21.15.3:0/1797414297 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6da00c5c00 con 0x55c4908227a0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6d98005500 con 0x55c4908227a0 2024-02-09T23:00:16.097 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6d8cff9640 1 -- 172.21.15.3:0/1797414297 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6da00bd6c0 con 0x55c4908227a0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 10 client.18744 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55c4907416f0 con 0x55c4908227a0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6d8cff9640 1 -- 172.21.15.3:0/1797414297 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d98005500 con 0x55c4908227a0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6d8cff9640 1 -- 172.21.15.3:0/1797414297 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f6da0004b40 con 0x55c4908227a0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 10 client.18744 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 10 client.18744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 20 client.18744 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c49088a6b0 0x55c49088caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 4 client.18744 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.088+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c4904d11b0 con 0x55c49088a6b0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.089+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c49088a6b0 0x55c49088caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.089+0000 7f6dad4f2640 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c49088a6b0 0x55c49088caa0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bbf5646ce76582c server_cookie=5156fb5085c030cd in_seq=0 out_seq=0 2024-02-09T23:00:16.098 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.089+0000 7f6d8cff9640 10 client.18744 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.126+0000 7f6d8cff9640 1 -- 172.21.15.3:0/1797414297 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c4904d11b0 con 0x55c49088a6b0 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6d8cff9640 10 client.18744 handle_command_reply: tid=0 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 client.18744 shutdown 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 msgr2=0x55c4908284a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c4908284a0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f6da0009430 tx=0x7f6da0032c30 comp rx=0 tx=0).stop 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 shutdown_connections 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c49088a6b0 0x55c49088caa0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4908199d0 0x55c490827d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4908227a0 0x55c4908284a0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6d8cff9640 0 client.18744 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 --2- 172.21.15.3:0/1797414297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4908190b0 0x55c490829970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 >> 172.21.15.3:0/1797414297 conn(0x55c4908139d0 msgr2=0x55c4908121a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 shutdown_connections 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 1 -- 172.21.15.3:0/1797414297 wait complete. 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.127+0000 7f6db240a740 20 client.18744 trim_cache size 0 max 16384 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:00:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:00:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d900a22a0 msgr2=0x7f6d900a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d900a22a0 0x7f6d900a4760 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f6d9c014a10 tx=0x7f6d9c00bc90 comp rx=0 tx=0).stop 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 msgr2=0x7f6da81a2700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 0x7f6da81a2700 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f6da002c700 tx=0x7f6da0034b10 comp rx=0 tx=0).stop 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 shutdown_connections 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6d900a22a0 0x7f6d900a4760 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6da8103c20 0x7f6da819e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6da8106120 0x7f6da81a2700 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 --2- 172.21.15.3:0/3039199679 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6da8069010 0x7f6da819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 >> 172.21.15.3:0/3039199679 conn(0x7f6da8077220 msgr2=0x7f6da8102c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:16.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 shutdown_connections 2024-02-09T23:00:16.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:16.128+0000 7f6daf756640 1 -- 172.21.15.3:0/3039199679 wait complete. 2024-02-09T23:00:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:16 smithi003 ceph-mon[32678]: pgmap v281: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-02-09T23:00:16.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 265271 2.72 MB/sec execute 370 sec latency 948.343 ms 2024-02-09T23:00:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:16 smithi064 ceph-mon[36638]: pgmap v281: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-02-09T23:00:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:16 smithi089 ceph-mon[36665]: pgmap v281: 97 pgs: 97 active+clean; 99 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 1.1 MiB/s wr, 190 op/s 2024-02-09T23:00:17.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 266726 2.73 MB/sec execute 371 sec latency 20.929 ms 2024-02-09T23:00:18.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:00:18.101 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:00:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 -- 172.21.15.3:0/2223348144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7108100120 msgr2=0x7f710810e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 --2- 172.21.15.3:0/2223348144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7108100120 0x7f710810e080 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f7100009cd0 tx=0x7f710002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 -- 172.21.15.3:0/2223348144 shutdown_connections 2024-02-09T23:00:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 --2- 172.21.15.3:0/2223348144 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71080ff760 0x7f71080ffbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 --2- 172.21.15.3:0/2223348144 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7108100120 0x7f710810e080 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 --2- 172.21.15.3:0/2223348144 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71081061d0 0x7f71081065b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 -- 172.21.15.3:0/2223348144 >> 172.21.15.3:0/2223348144 conn(0x7f7108076630 msgr2=0x7f71080fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 -- 172.21.15.3:0/2223348144 shutdown_connections 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.209+0000 7f710f163640 1 -- 172.21.15.3:0/2223348144 wait complete. 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 Processor -- start 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 -- start start 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71080ff760 0x7f710819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7108100120 0x7f710819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 0x7f71081976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108197e50 con 0x7f71081061d0 2024-02-09T23:00:18.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108197fc0 con 0x7f71080ff760 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710f163640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7108198180 con 0x7f7108100120 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 0x7f71081976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 0x7f71081976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35356/0 (socket says 172.21.15.3:35356) 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710d700640 1 -- 172.21.15.3:0/927784167 learned_addr learned my addr 172.21.15.3:0/927784167 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f710ceff640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71080ff760 0x7f710819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.210+0000 7f7107fff640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7108100120 0x7f710819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710d700640 1 -- 172.21.15.3:0/927784167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7108100120 msgr2=0x7f710819db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710d700640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7108100120 0x7f710819db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710d700640 1 -- 172.21.15.3:0/927784167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71080ff760 msgr2=0x7f710819d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710d700640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71080ff760 0x7f710819d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710d700640 1 -- 172.21.15.3:0/927784167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7100009900 con 0x7f71081061d0 2024-02-09T23:00:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710d700640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 0x7f71081976c0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f7100004af0 tx=0x7f7100032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f7105ffb640 1 -- 172.21.15.3:0/927784167 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000328b0 con 0x7f71081061d0 2024-02-09T23:00:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f7105ffb640 1 -- 172.21.15.3:0/927784167 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7100003e50 con 0x7f71081061d0 2024-02-09T23:00:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7108198400 con 0x7f71081061d0 2024-02-09T23:00:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f7105ffb640 1 -- 172.21.15.3:0/927784167 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7100004740 con 0x7f71081061d0 2024-02-09T23:00:18.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.211+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71081a4580 con 0x7f71081061d0 2024-02-09T23:00:18.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.213+0000 7f7105ffb640 1 -- 172.21.15.3:0/927784167 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f710000a040 con 0x7f71081061d0 2024-02-09T23:00:18.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 Processor -- start 2024-02-09T23:00:18.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 -- start start 2024-02-09T23:00:18.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x562980386d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7105ffb640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f70f80a2230 0x7f70f80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x562980392300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562980341d00 con 0x56298038feb0 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629802f62f0 con 0x562980386980 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7111e0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629802f3820 con 0x562980387330 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x562980392300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x562980386d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f710ceff640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f70f80a2230 0x7f70f80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x562980386d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:60014/0 (socket says 172.21.15.3:60014) 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x562980392300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35364/0 (socket says 172.21.15.3:35364) 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 -- 172.21.15.3:0/2467338590 learned_addr learned my addr 172.21.15.3:0/2467338590 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7105ffb640 1 -- 172.21.15.3:0/927784167 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f71000f0b10 con 0x7f71081061d0 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 -- 172.21.15.3:0/2467338590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 msgr2=0x56298038f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 -- 172.21.15.3:0/2467338590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 msgr2=0x562980392300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x562980392300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.214+0000 7f7107fff640 1 -- 172.21.15.3:0/2467338590 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629802b0ff0 con 0x562980386980 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.215+0000 7f710ceff640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f70f80a2230 0x7f70f80a46f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009bd0 tx=0x7f70fc005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.215+0000 7f7107fff640 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x562980386d60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009fd0 tx=0x7f70f4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40e5e5946728cca server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.215+0000 7f70f2ffd640 1 -- 172.21.15.3:0/2467338590 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4004040 con 0x562980386980 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.215+0000 7f70f2ffd640 1 -- 172.21.15.3:0/2467338590 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70f4005020 con 0x562980386980 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.215+0000 7f70f2ffd640 1 -- 172.21.15.3:0/2467338590 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4004b40 con 0x562980386980 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 -- 172.21.15.3:0/2467338590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 msgr2=0x562980386d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x562980386d60 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f70f4009fd0 tx=0x7f70f4005850 comp rx=0 tx=0).stop 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 -- 172.21.15.3:0/2467338590 shutdown_connections 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x562980392300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 --2- 172.21.15.3:0/2467338590 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x562980386d60 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 -- 172.21.15.3:0/2467338590 >> 172.21.15.3:0/2467338590 conn(0x562980381050 msgr2=0x562980381430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 -- 172.21.15.3:0/2467338590 shutdown_connections 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 -- 172.21.15.3:0/2467338590 wait complete. 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 -- start start 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.216+0000 7f7111e0b740 1 Processor -- start 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x56298038cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x56298038d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629802ef000 con 0x56298038feb0 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297ffc2430 con 0x562980386980 2024-02-09T23:00:18.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297ff7ee00 con 0x562980387330 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 10 client.0 _ll_register_callbacks cb 0x5629802e37f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x56298038cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x56298038cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:60022/0 (socket says 172.21.15.3:60022) 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 -- 172.21.15.3:0/4186584754 learned_addr learned my addr 172.21.15.3:0/4186584754 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7107fff640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710d700640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x56298038d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 -- 172.21.15.3:0/4186584754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 msgr2=0x56298038d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 -- 172.21.15.3:0/4186584754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 msgr2=0x56298038d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x56298038d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 -- 172.21.15.3:0/4186584754 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f4019040 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f710ceff640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x56298038cdc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f70fc003940 tx=0x7f70fc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff040e4c4a811f55 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f70e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f70e7fff640 1 -- 172.21.15.3:0/4186584754 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70fc00ecc0 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f70e7fff640 1 -- 172.21.15.3:0/4186584754 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f70fc00ee20 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.217+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56297ffee7e0 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f70e7fff640 1 -- 172.21.15.3:0/4186584754 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f70fc005020 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 10 client.27692 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5629802ed680 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f70e7fff640 1 -- 172.21.15.3:0/4186584754 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56297ffee7e0 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f70e7fff640 1 -- 172.21.15.3:0/4186584754 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f70fc024030 con 0x562980386980 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 10 client.27692 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:18.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 10 client.27692 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 20 client.27692 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629803a03d0 0x5629803a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 4 client.27692 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56298002b720 con 0x5629803a03d0 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.218+0000 7f7107fff640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629803a03d0 0x5629803a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7107fff640 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629803a03d0 0x5629803a27f0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff723c514c0439 server_cookie=7905eb9b834871d5 in_seq=0 out_seq=0 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f70e7fff640 10 client.27692 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f70e7fff640 1 -- 172.21.15.3:0/4186584754 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56298002b720 con 0x5629803a03d0 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f70e7fff640 10 client.27692 handle_command_reply: tid=0 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 client.27692 shutdown 2024-02-09T23:00:18.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 msgr2=0x56298038cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x56298038cdc0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f70fc003940 tx=0x7f70fc004730 comp rx=0 tx=0).stop 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 shutdown_connections 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5629803a03d0 0x5629803a27f0 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56298038feb0 0x56298038d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f70e7fff640 0 client.27692 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980387330 0x56298038d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.219+0000 7f7111e0b740 1 --2- 172.21.15.3:0/4186584754 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x56298038cdc0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.220+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 >> 172.21.15.3:0/4186584754 conn(0x562980381050 msgr2=0x562980382da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.220+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 shutdown_connections 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.220+0000 7f7111e0b740 1 -- 172.21.15.3:0/4186584754 wait complete. 2024-02-09T23:00:18.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.220+0000 7f7111e0b740 20 client.27692 trim_cache size 0 max 16384 2024-02-09T23:00:18.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 Processor -- start 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 -- start start 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980381050 0x56298038f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980386980 0x56298038d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x56298038a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562980273fd0 con 0x562980387330 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562980388a10 con 0x562980381050 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f7111e0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562980381c20 con 0x562980386980 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x56298038a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x56298038a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35384/0 (socket says 172.21.15.3:35384) 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 -- 172.21.15.3:0/3551393940 learned_addr learned my addr 172.21.15.3:0/3551393940 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710ceff640 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980381050 0x56298038f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 -- 172.21.15.3:0/3551393940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980386980 msgr2=0x56298038d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980386980 0x56298038d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 -- 172.21.15.3:0/3551393940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980381050 msgr2=0x56298038f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980381050 0x56298038f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 -- 172.21.15.3:0/3551393940 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562980289b30 con 0x562980387330 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710ceff640 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980381050 0x56298038f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f710d700640 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x56298038a040 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f71000ee3d0 tx=0x7f71000b9200 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7e49b05036c1e72 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.235+0000 7f70f27fc640 1 -- 172.21.15.3:0/3551393940 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000cfc80 con 0x562980387330 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f70f27fc640 1 -- 172.21.15.3:0/3551393940 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7100034da0 con 0x562980387330 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f70f27fc640 1 -- 172.21.15.3:0/3551393940 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000c5db0 con 0x562980387330 2024-02-09T23:00:18.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 -- 172.21.15.3:0/3551393940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 msgr2=0x56298038a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x56298038a040 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f71000ee3d0 tx=0x7f71000b9200 comp rx=0 tx=0).stop 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 -- 172.21.15.3:0/3551393940 shutdown_connections 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x56298038a040 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980386980 0x56298038d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3551393940 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980381050 0x56298038f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 -- 172.21.15.3:0/3551393940 >> 172.21.15.3:0/3551393940 conn(0x562980387a80 msgr2=0x5629803a7190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 -- 172.21.15.3:0/3551393940 shutdown_connections 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 -- 172.21.15.3:0/3551393940 wait complete. 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 -- start start 2024-02-09T23:00:18.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.236+0000 7f7111e0b740 1 Processor -- start 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980381050 0x562980396060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x5629803965a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x5629803f6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629803f6e60 con 0x562980387330 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629803f6fa0 con 0x562980386980 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629803f7160 con 0x562980381050 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7111e0b740 10 client.0 _ll_register_callbacks cb 0x7f70f4009ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:18.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x5629803f6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f7107fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x5629803965a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x5629803f6920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35390/0 (socket says 172.21.15.3:35390) 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 -- 172.21.15.3:0/3643665550 learned_addr learned my addr 172.21.15.3:0/3643665550 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710ceff640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980381050 0x562980396060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 -- 172.21.15.3:0/3643665550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980381050 msgr2=0x562980396060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980381050 0x562980396060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 -- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 msgr2=0x5629803965a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x5629803965a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 -- 172.21.15.3:0/3643665550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71081a4580 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710ceff640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980381050 0x562980396060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f710d700640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x5629803f6920 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f71000bd5a0 tx=0x7f710002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33163c2da539655a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f70e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.237+0000 7f70e7fff640 1 -- 172.21.15.3:0/3643665550 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71000c5db0 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f70e7fff640 1 -- 172.21.15.3:0/3643665550 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f71000b9040 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f70f4005020 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f70e7fff640 1 -- 172.21.15.3:0/3643665550 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71000bb910 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 10 client.18762 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5629802af850 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f70e7fff640 1 -- 172.21.15.3:0/3643665550 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f70f4005020 con 0x562980387330 2024-02-09T23:00:18.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f70e7fff640 1 -- 172.21.15.3:0/3643665550 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7100097d80 con 0x562980387330 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 10 client.18762 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 10 client.18762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 20 client.18762 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56298038feb0 0x5629803fa4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 4 client.18762 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56298007a890 con 0x56298038feb0 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.238+0000 7f710ceff640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56298038feb0 0x5629803fa4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f70e7fff640 10 client.18762 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f710ceff640 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56298038feb0 0x5629803fa4a0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3db18619ad2bcf4 server_cookie=d9b6fb82919beb08 in_seq=0 out_seq=0 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f70e7fff640 1 -- 172.21.15.3:0/3643665550 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56298007a890 con 0x56298038feb0 2024-02-09T23:00:18.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f70e7fff640 10 client.18762 handle_command_reply: tid=0 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 client.18762 shutdown 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 msgr2=0x5629803f6920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x5629803f6920 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7f71000bd5a0 tx=0x7f710002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 shutdown_connections 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56298038feb0 0x5629803fa4a0 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562980386980 0x5629803965a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562980387330 0x5629803f6920 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 --2- 172.21.15.3:0/3643665550 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562980381050 0x562980396060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f70e7fff640 0 client.18762 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 >> 172.21.15.3:0/3643665550 conn(0x562980387a80 msgr2=0x56298037f960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 shutdown_connections 2024-02-09T23:00:18.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 1 -- 172.21.15.3:0/3643665550 wait complete. 2024-02-09T23:00:18.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.239+0000 7f7111e0b740 20 client.18762 trim_cache size 0 max 16384 2024-02-09T23:00:18.254 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:00:18.254 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:00:18.254 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:00:18.254 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f70f80a2230 msgr2=0x7f70f80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f70f80a2230 0x7f70f80a46f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f70fc009bd0 tx=0x7f70fc005f90 comp rx=0 tx=0).stop 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 msgr2=0x7f71081976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 0x7f71081976c0 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f7100004af0 tx=0x7f7100032570 comp rx=0 tx=0).stop 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 shutdown_connections 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f70f80a2230 0x7f70f80a46f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f71081061d0 0x7f71081976c0 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7108100120 0x7f710819db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 --2- 172.21.15.3:0/927784167 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71080ff760 0x7f710819d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 >> 172.21.15.3:0/927784167 conn(0x7f7108076630 msgr2=0x7f71081041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 shutdown_connections 2024-02-09T23:00:18.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.240+0000 7f710f163640 1 -- 172.21.15.3:0/927784167 wait complete. 2024-02-09T23:00:18.257 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:00:18.257 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:00:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 -- 172.21.15.3:0/3886747716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 msgr2=0x7f0700101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 --2- 172.21.15.3:0/3886747716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f0700101830 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009930 tx=0x7f06fc02bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 -- 172.21.15.3:0/3886747716 shutdown_connections 2024-02-09T23:00:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 --2- 172.21.15.3:0/3886747716 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0700101e00 0x7f0700105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 --2- 172.21.15.3:0/3886747716 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 0x7f070010e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 --2- 172.21.15.3:0/3886747716 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f0700101830 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 -- 172.21.15.3:0/3886747716 >> 172.21.15.3:0/3886747716 conn(0x7f07000fccf0 msgr2=0x7f07000ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 -- 172.21.15.3:0/3886747716 shutdown_connections 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.357+0000 7f0707ab5640 1 -- 172.21.15.3:0/3886747716 wait complete. 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 Processor -- start 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 -- start start 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f070019d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 0x7f070019db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f07001976f0 0x7f0700197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001981c0 con 0x7f0700101450 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0700198330 con 0x7f07001976f0 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0707ab5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07001984f0 con 0x7f0700106520 2024-02-09T23:00:18.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f070019d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f070019d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35412/0 (socket says 172.21.15.3:35412) 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 -- 172.21.15.3:0/416927351 learned_addr learned my addr 172.21.15.3:0/416927351 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705050640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 0x7f070019db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0706052640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f07001976f0 0x7f0700197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 -- 172.21.15.3:0/416927351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 msgr2=0x7f070019db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 0x7f070019db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 -- 172.21.15.3:0/416927351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f07001976f0 msgr2=0x7f0700197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f07001976f0 0x7f0700197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705851640 1 -- 172.21.15.3:0/416927351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f0009490 con 0x7f0700101450 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.358+0000 7f0705050640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 0x7f070019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.359+0000 7f0705851640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f070019d630 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009230 tx=0x7f06fc02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.359+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc004350 con 0x7f0700101450 2024-02-09T23:00:18.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.359+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06fc0044b0 con 0x7f0700101450 2024-02-09T23:00:18.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.359+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06fc009560 con 0x7f0700101450 2024-02-09T23:00:18.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.359+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06fc003d00 con 0x7f0700101450 2024-02-09T23:00:18.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.359+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07001a4780 con 0x7f0700101450 2024-02-09T23:00:18.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.360+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0700102920 con 0x7f0700101450 2024-02-09T23:00:18.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.361+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f06fc004df0 con 0x7f0700101450 2024-02-09T23:00:18.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.361+0000 7f06f6ffd640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06ec0a2230 0x7f06ec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.362+0000 7f0705050640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06ec0a2230 0x7f06ec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.362+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f06fc0f0860 con 0x7f0700101450 2024-02-09T23:00:18.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.362+0000 7f0705050640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06ec0a2230 0x7f06ec0a46f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0700198f40 tx=0x7f06f0005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.365+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f06fc0b95a0 con 0x7f0700101450 2024-02-09T23:00:18.381 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:18 smithi003 ceph-mon[32678]: pgmap v282: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T23:00:18.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.584+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0700109750 con 0x7f0700101450 2024-02-09T23:00:18.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.584+0000 7f06f6ffd640 1 -- 172.21.15.3:0/416927351 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+11361 (secure 0 0 0) 0x7f06fc003e60 con 0x7f0700101450 2024-02-09T23:00:18.589 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:00:18.589 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:59:11.242420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:00:18.589 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 47 2024-02-09T23:00:18.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06ec0a2230 msgr2=0x7f06ec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06ec0a2230 0x7f06ec0a46f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f0700198f40 tx=0x7f06f0005f90 comp rx=0 tx=0).stop 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 msgr2=0x7f070019d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f070019d630 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f06fc009230 tx=0x7f06fc02c650 comp rx=0 tx=0).stop 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 shutdown_connections 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f06ec0a2230 0x7f06ec0a46f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f07001976f0 0x7f0700197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0700106520 0x7f070019db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 --2- 172.21.15.3:0/416927351 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0700101450 0x7f070019d630 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 >> 172.21.15.3:0/416927351 conn(0x7f07000fccf0 msgr2=0x7f07000fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 shutdown_connections 2024-02-09T23:00:18.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.586+0000 7f0707ab5640 1 -- 172.21.15.3:0/416927351 wait complete. 2024-02-09T23:00:18.601 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:18.601 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:00:18.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 267752 2.74 MB/sec execute 372 sec latency 439.128 ms 2024-02-09T23:00:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:18 smithi064 ceph-mon[36638]: pgmap v282: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 -- 172.21.15.3:0/2897202781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0105410 msgr2=0x7effe010cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 --2- 172.21.15.3:0/2897202781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0105410 0x7effe010cfe0 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7effd8009cd0 tx=0x7effd802c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 -- 172.21.15.3:0/2897202781 shutdown_connections 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 --2- 172.21.15.3:0/2897202781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0100cf0 0x7effe0104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 --2- 172.21.15.3:0/2897202781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0105410 0x7effe010cfe0 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 --2- 172.21.15.3:0/2897202781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0100340 0x7effe0100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 -- 172.21.15.3:0/2897202781 >> 172.21.15.3:0/2897202781 conn(0x7effe00772e0 msgr2=0x7effe00776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 -- 172.21.15.3:0/2897202781 shutdown_connections 2024-02-09T23:00:18.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.702+0000 7effe508a640 1 -- 172.21.15.3:0/2897202781 wait complete. 2024-02-09T23:00:18.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 Processor -- start 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 -- start start 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0100340 0x7effe0196470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0105410 0x7effe01969b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 0x7effe01a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01a6030 con 0x7effe0196ef0 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01a6170 con 0x7effe0100340 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effe508a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe01a62e0 con 0x7effe0105410 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 0x7effe01a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 0x7effe01a5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35432/0 (socket says 172.21.15.3:35432) 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 -- 172.21.15.3:0/624146039 learned_addr learned my addr 172.21.15.3:0/624146039 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effd7fff640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0105410 0x7effe01969b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effded9d640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0100340 0x7effe0196470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 -- 172.21.15.3:0/624146039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0105410 msgr2=0x7effe01969b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0105410 0x7effe01969b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 -- 172.21.15.3:0/624146039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0100340 msgr2=0x7effe0196470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0100340 0x7effe0196470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 -- 172.21.15.3:0/624146039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effd8009900 con 0x7effe0196ef0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.703+0000 7effdf59e640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 0x7effe01a5af0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7effd8004af0 tx=0x7effd8032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.704+0000 7effdcd99640 1 -- 172.21.15.3:0/624146039 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd80328b0 con 0x7effe0196ef0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.704+0000 7effdcd99640 1 -- 172.21.15.3:0/624146039 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd8003e50 con 0x7effe0196ef0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.704+0000 7effdcd99640 1 -- 172.21.15.3:0/624146039 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd8004730 con 0x7effe0196ef0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.704+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe01a6560 con 0x7effe0196ef0 2024-02-09T23:00:18.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.704+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe01a6a10 con 0x7effe0196ef0 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effdcd99640 1 -- 172.21.15.3:0/624146039 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7effd800a040 con 0x7effe0196ef0 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 Processor -- start 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 -- start start 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766250a30 0x563766250e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766259990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effdcd99640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effcc0a2230 0x7effcc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x56376625c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376620c020 con 0x5637662513e0 2024-02-09T23:00:18.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637661bf320 con 0x563766250a30 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.706+0000 7effe7c56740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637661bc850 con 0x563766259f60 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effded9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766259990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effded9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766259990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35436/0 (socket says 172.21.15.3:35436) 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effd7fff640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effcc0a2230 0x7effcc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effded9d640 1 -- 172.21.15.3:0/1386923324 learned_addr learned my addr 172.21.15.3:0/1386923324 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x56376625c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdcd99640 1 -- 172.21.15.3:0/624146039 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7effd80f0a90 con 0x7effe0196ef0 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effd7fff640 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766250a30 0x563766250e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 -- 172.21.15.3:0/1386923324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766250a30 msgr2=0x563766250e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766250a30 0x563766250e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 -- 172.21.15.3:0/1386923324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 msgr2=0x563766259990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766259990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 -- 172.21.15.3:0/1386923324 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56376617a210 con 0x563766259f60 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effded9d640 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766259990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effd7fff640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effcc0a2230 0x7effcc0a46f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7effe01985d0 tx=0x7effc80127c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effdf59e640 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x56376625c3b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effd80ee2f0 tx=0x7effd802c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c241d7b0d86b0244 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effd57fa640 1 -- 172.21.15.3:0/1386923324 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd80bd0d0 con 0x563766259f60 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effd57fa640 1 -- 172.21.15.3:0/1386923324 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd80bcef0 con 0x563766259f60 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.707+0000 7effd57fa640 1 -- 172.21.15.3:0/1386923324 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd80c6c80 con 0x563766259f60 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 -- 172.21.15.3:0/1386923324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 msgr2=0x56376625c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x56376625c3b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effd80ee2f0 tx=0x7effd802c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 -- 172.21.15.3:0/1386923324 shutdown_connections 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x56376625c3b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766259990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 --2- 172.21.15.3:0/1386923324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766250a30 0x563766250e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 -- 172.21.15.3:0/1386923324 >> 172.21.15.3:0/1386923324 conn(0x56376624b100 msgr2=0x56376624b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 -- 172.21.15.3:0/1386923324 shutdown_connections 2024-02-09T23:00:18.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.708+0000 7effe7c56740 1 -- 172.21.15.3:0/1386923324 wait complete. 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 -- start start 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 Processor -- start 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766250a30 0x563766256f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766257440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766259f60 0x563766257980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637661b8030 con 0x5637662513e0 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563765e8b430 con 0x563766259f60 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563765e47e00 con 0x563766250a30 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effe7c56740 10 client.0 _ll_register_callbacks cb 0x5637661ad030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766257440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766257440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35440/0 (socket says 172.21.15.3:35440) 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 -- 172.21.15.3:0/2819216211 learned_addr learned my addr 172.21.15.3:0/2819216211 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effded9d640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766250a30 0x563766256f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effdf59e640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766259f60 0x563766257980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 -- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766250a30 msgr2=0x563766256f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766250a30 0x563766256f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 -- 172.21.15.3:0/2819216211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766259f60 msgr2=0x563766257980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766259f60 0x563766257980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 -- 172.21.15.3:0/2819216211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe01a6a10 con 0x5637662513e0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effd7fff640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766257440 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7effc8010850 tx=0x7effc800b410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f1214b3bbd9a7c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.709+0000 7effc67fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effc67fc640 1 -- 172.21.15.3:0/2819216211 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effc801cae0 con 0x5637662513e0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effc67fc640 1 -- 172.21.15.3:0/2819216211 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7effc801cc40 con 0x5637662513e0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563765eb77e0 con 0x5637662513e0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effc67fc640 1 -- 172.21.15.3:0/2819216211 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7effc801ce00 con 0x5637662513e0 2024-02-09T23:00:18.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 10 client.18780 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5637661b66b0 con 0x5637662513e0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effc67fc640 1 -- 172.21.15.3:0/2819216211 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563765eb77e0 con 0x5637662513e0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effc67fc640 1 -- 172.21.15.3:0/2819216211 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7effc801f530 con 0x5637662513e0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 10 client.18780 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 10 client.18780 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 20 client.18780 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56376626a2e0 0x56376626c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 4 client.18780 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563765ef4720 con 0x56376626a2e0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.710+0000 7effded9d640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56376626a2e0 0x56376626c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.711+0000 7effded9d640 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56376626a2e0 0x56376626c700 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9346e45d595c3e24 server_cookie=6188882e0f289396 in_seq=0 out_seq=0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.711+0000 7effc67fc640 10 client.18780 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.711+0000 7effc67fc640 1 -- 172.21.15.3:0/2819216211 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563765ef4720 con 0x56376626a2e0 2024-02-09T23:00:18.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.711+0000 7effc67fc640 10 client.18780 handle_command_reply: tid=0 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.711+0000 7effe7c56740 1 client.18780 shutdown 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 msgr2=0x563766257440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766257440 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7effc8010850 tx=0x7effc800b410 comp rx=0 tx=0).stop 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 shutdown_connections 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56376626a2e0 0x56376626c700 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563766259f60 0x563766257980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5637662513e0 0x563766257440 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effc67fc640 0 client.18780 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 --2- 172.21.15.3:0/2819216211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766250a30 0x563766256f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 >> 172.21.15.3:0/2819216211 conn(0x56376624b100 msgr2=0x56376624ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 shutdown_connections 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 1 -- 172.21.15.3:0/2819216211 wait complete. 2024-02-09T23:00:18.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.712+0000 7effe7c56740 20 client.18780 trim_cache size 0 max 16384 2024-02-09T23:00:18.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:18 smithi089 ceph-mon[36665]: pgmap v282: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T23:00:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.723+0000 7effe7c56740 1 Processor -- start 2024-02-09T23:00:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.723+0000 7effe7c56740 1 -- start start 2024-02-09T23:00:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effe7c56740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x5637662577a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effe7c56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x56376624a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effe7c56740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x563766260d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effe7c56740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56376613d130 con 0x563766250a30 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effe7c56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563766266db0 con 0x5637662513e0 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effe7c56740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563766271930 con 0x563766259f60 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x5637662577a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x5637662577a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35442/0 (socket says 172.21.15.3:35442) 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 -- 172.21.15.3:0/3186810501 learned_addr learned my addr 172.21.15.3:0/3186810501 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effdf59e640 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x563766260d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effd7fff640 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x56376624a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 -- 172.21.15.3:0/3186810501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 msgr2=0x563766260d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x563766260d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 -- 172.21.15.3:0/3186810501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 msgr2=0x56376624a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x56376624a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 -- 172.21.15.3:0/3186810501 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563766152c90 con 0x563766250a30 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effded9d640 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x5637662577a0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c00 tx=0x7effd0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4626acaa118c42f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effd4ff9640 1 -- 172.21.15.3:0/3186810501 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0017070 con 0x563766250a30 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effd4ff9640 1 -- 172.21.15.3:0/3186810501 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd0018040 con 0x563766250a30 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.724+0000 7effd4ff9640 1 -- 172.21.15.3:0/3186810501 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0003850 con 0x563766250a30 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 -- 172.21.15.3:0/3186810501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 msgr2=0x5637662577a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x5637662577a0 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7effd0000c00 tx=0x7effd0005f90 comp rx=0 tx=0).stop 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 -- 172.21.15.3:0/3186810501 shutdown_connections 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x56376624a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x563766260d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 --2- 172.21.15.3:0/3186810501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x5637662577a0 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 -- 172.21.15.3:0/3186810501 >> 172.21.15.3:0/3186810501 conn(0x56376624b100 msgr2=0x5637662499b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 -- 172.21.15.3:0/3186810501 shutdown_connections 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 -- 172.21.15.3:0/3186810501 wait complete. 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 -- start start 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.725+0000 7effe7c56740 1 Processor -- start 2024-02-09T23:00:18.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x563766260080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x5637662605c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x5637662ba850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637662bad90 con 0x563766250a30 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637662baed0 con 0x5637662513e0 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5637662bb090 con 0x563766259f60 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 10 client.0 _ll_register_callbacks cb 0x7effd8004040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x563766260080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x563766260080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35452/0 (socket says 172.21.15.3:35452) 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 -- 172.21.15.3:0/1726889846 learned_addr learned my addr 172.21.15.3:0/1726889846 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effdf59e640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x5637662ba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effd7fff640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x5637662605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 -- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 msgr2=0x5637662ba850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x5637662ba850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 -- 172.21.15.3:0/1726889846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 msgr2=0x5637662605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x5637662605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 -- 172.21.15.3:0/1726889846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effd0010430 con 0x563766250a30 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effd7fff640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x5637662605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effded9d640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x563766260080 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7effd0009710 tx=0x7effd0010d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf8109066311663 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effc67fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effc67fc640 1 -- 172.21.15.3:0/1726889846 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd0017040 con 0x563766250a30 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effc67fc640 1 -- 172.21.15.3:0/1726889846 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7effd0004250 con 0x563766250a30 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.726+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7effd80bcef0 con 0x563766250a30 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effc67fc640 1 -- 172.21.15.3:0/1726889846 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7effd0004410 con 0x563766250a30 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 10 client.18786 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x563766178a70 con 0x563766250a30 2024-02-09T23:00:18.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effc67fc640 1 -- 172.21.15.3:0/1726889846 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd80bcef0 con 0x563766250a30 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effc67fc640 1 -- 172.21.15.3:0/1726889846 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7effd002e030 con 0x563766250a30 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 10 client.18786 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 10 client.18786 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 20 client.18786 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5637662c2790 0x5637662c4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 4 client.18786 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563765f43890 con 0x5637662c2790 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effd7fff640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5637662c2790 0x5637662c4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effc67fc640 10 client.18786 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.727+0000 7effd7fff640 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5637662c2790 0x5637662c4bb0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=b9ac9ebe81242a36 server_cookie=fc99e2214336576d in_seq=0 out_seq=0 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effc67fc640 1 -- 172.21.15.3:0/1726889846 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563765f43890 con 0x5637662c2790 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effc67fc640 10 client.18786 handle_command_reply: tid=0 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 client.18786 shutdown 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 msgr2=0x563766260080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x563766260080 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7effd0009710 tx=0x7effd0010d10 comp rx=0 tx=0).stop 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 shutdown_connections 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5637662c2790 0x5637662c4bb0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5637662513e0 0x5637662605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effc67fc640 0 client.18786 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563766259f60 0x5637662ba850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 --2- 172.21.15.3:0/1726889846 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563766250a30 0x563766260080 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 >> 172.21.15.3:0/1726889846 conn(0x56376624b100 msgr2=0x56376624a240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 shutdown_connections 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 1 -- 172.21.15.3:0/1726889846 wait complete. 2024-02-09T23:00:18.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe7c56740 20 client.18786 trim_cache size 0 max 16384 2024-02-09T23:00:18.740 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:18.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effcc0a2230 msgr2=0x7effcc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effcc0a2230 0x7effcc0a46f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7effe01985d0 tx=0x7effc80127c0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 msgr2=0x7effe01a5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 0x7effe01a5af0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7effd8004af0 tx=0x7effd8032570 comp rx=0 tx=0).stop 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 shutdown_connections 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7effcc0a2230 0x7effcc0a46f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7effe0196ef0 0x7effe01a5af0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7effe0105410 0x7effe01969b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 --2- 172.21.15.3:0/624146039 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7effe0100340 0x7effe0196470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 >> 172.21.15.3:0/624146039 conn(0x7effe00772e0 msgr2=0x7effe00fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 shutdown_connections 2024-02-09T23:00:18.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.728+0000 7effe508a640 1 -- 172.21.15.3:0/624146039 wait complete. 2024-02-09T23:00:18.743 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:18.743 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/13253036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c4102dc0 msgr2=0x7ff3c410b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/13253036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c4102dc0 0x7ff3c410b2a0 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8009900 tx=0x7ff3b802bcd0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/13253036 shutdown_connections 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/13253036 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c410b910 0x7ff3c410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/13253036 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c4102dc0 0x7ff3c410b2a0 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/13253036 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3c41024a0 0x7ff3c4102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/13253036 >> 172.21.15.3:0/13253036 conn(0x7ff3c40772a0 msgr2=0x7ff3c40776b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/13253036 shutdown_connections 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.844+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/13253036 wait complete. 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 Processor -- start 2024-02-09T23:00:18.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 -- start start 2024-02-09T23:00:18.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 0x7ff3c419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c4102dc0 0x7ff3c419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3c410b910 0x7ff3c41977c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4197e60 con 0x7ff3c41024a0 2024-02-09T23:00:18.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4197fd0 con 0x7ff3c410b910 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3ca0c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4198190 con 0x7ff3c4102dc0 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.845+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 0x7ff3c419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 0x7ff3c419d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35478/0 (socket says 172.21.15.3:35478) 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 -- 172.21.15.3:0/6894379 learned_addr learned my addr 172.21.15.3:0/6894379 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c3fff640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3c410b910 0x7ff3c41977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c4102dc0 0x7ff3c419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 -- 172.21.15.3:0/6894379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c4102dc0 msgr2=0x7ff3c419db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c4102dc0 0x7ff3c419db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 -- 172.21.15.3:0/6894379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3c410b910 msgr2=0x7ff3c41977c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3c410b910 0x7ff3c41977c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 -- 172.21.15.3:0/6894379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3b8009560 con 0x7ff3c41024a0 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 0x7ff3c419d650 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c890 tx=0x7ff3b400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c0ff9640 1 -- 172.21.15.3:0/6894379 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4004220 con 0x7ff3c41024a0 2024-02-09T23:00:18.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c0ff9640 1 -- 172.21.15.3:0/6894379 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b4007e90 con 0x7ff3c41024a0 2024-02-09T23:00:18.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c4198470 con 0x7ff3c41024a0 2024-02-09T23:00:18.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3c0ff9640 1 -- 172.21.15.3:0/6894379 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4004a00 con 0x7ff3c41024a0 2024-02-09T23:00:18.853 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.846+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c41a4680 con 0x7ff3c41024a0 2024-02-09T23:00:18.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.848+0000 7ff3c0ff9640 1 -- 172.21.15.3:0/6894379 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff3b40079b0 con 0x7ff3c41024a0 2024-02-09T23:00:18.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 Processor -- start 2024-02-09T23:00:18.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 -- start start 2024-02-09T23:00:18.859 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c0ff9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3ac0a22b0 0x7ff3ac0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e1412a30 0x55f2e1412e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e141b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e141bf60 0x55f2e141e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e13ce020 con 0x55f2e141bf60 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e1381320 con 0x55f2e1412a30 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e137e850 con 0x55f2e14133e0 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c3fff640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3ac0a22b0 0x7ff3ac0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c0ff9640 1 -- 172.21.15.3:0/6894379 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff3b40cdda0 con 0x7ff3c41024a0 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e141bf60 0x55f2e141e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e141bf60 0x55f2e141e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35494/0 (socket says 172.21.15.3:35494) 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e1412a30 0x55f2e1412e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c3fff640 1 -- 172.21.15.3:0/674192666 learned_addr learned my addr 172.21.15.3:0/674192666 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e141b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c37fe640 1 -- 172.21.15.3:0/674192666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 msgr2=0x55f2e141b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.860 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e141b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c37fe640 1 -- 172.21.15.3:0/674192666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e141bf60 msgr2=0x55f2e141e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e141bf60 0x55f2e141e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.849+0000 7ff3c37fe640 1 -- 172.21.15.3:0/674192666 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e133c210 con 0x55f2e1412a30 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3c3fff640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3ac0a22b0 0x7ff3ac0a4770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc007740 tx=0x7ff3bc007e30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e1412a30 0x55f2e1412e10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40cb490 tx=0x7ff3b409a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fa7aef304ff2c896 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3a9ffb640 1 -- 172.21.15.3:0/674192666 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40b8ea0 con 0x55f2e1412a30 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3a9ffb640 1 -- 172.21.15.3:0/674192666 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b409ae30 con 0x55f2e1412a30 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3a9ffb640 1 -- 172.21.15.3:0/674192666 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b409e880 con 0x55f2e1412a30 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3ccd80740 1 -- 172.21.15.3:0/674192666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e1412a30 msgr2=0x55f2e1412e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.850+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e1412a30 0x55f2e1412e10 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7ff3b40cb490 tx=0x7ff3b409a270 comp rx=0 tx=0).stop 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- 172.21.15.3:0/674192666 shutdown_connections 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e141bf60 0x55f2e141e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e141b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.861 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/674192666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e1412a30 0x55f2e1412e10 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- 172.21.15.3:0/674192666 >> 172.21.15.3:0/674192666 conn(0x55f2e140d100 msgr2=0x55f2e140d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- 172.21.15.3:0/674192666 shutdown_connections 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- 172.21.15.3:0/674192666 wait complete. 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- start start 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 Processor -- start 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 0x55f2e1418f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e1419460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14199a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e137a030 con 0x55f2e1412a30 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e104d430 con 0x55f2e141bf60 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e1009e00 con 0x55f2e14133e0 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3ccd80740 10 client.0 _ll_register_callbacks cb 0x55f2e136f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.851+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 0x55f2e1418f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 0x55f2e1418f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35508/0 (socket says 172.21.15.3:35508) 2024-02-09T23:00:18.862 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 -- 172.21.15.3:0/3943707297 learned_addr learned my addr 172.21.15.3:0/3943707297 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e1419460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c3fff640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14199a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 -- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 msgr2=0x55f2e1419460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e1419460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 -- 172.21.15.3:0/3943707297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 msgr2=0x55f2e14199a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14199a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 -- 172.21.15.3:0/3943707297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c41a4680 con 0x55f2e1412a30 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 0x55f2e1418f20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005fd0 tx=0x7ff3b406eb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bb09ca7613d849 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/3943707297 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b40b8ea0 con 0x55f2e1412a30 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/3943707297 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff3b409e880 con 0x55f2e1412a30 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2e10797e0 con 0x55f2e1412a30 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/3943707297 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3b4098a30 con 0x55f2e1412a30 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3ccd80740 10 client.18798 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:18.863 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55f2e13786b0 con 0x55f2e1412a30 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.852+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/3943707297 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b406c080 con 0x55f2e1412a30 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/3943707297 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff3b4066d20 con 0x55f2e1412a30 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3ccd80740 10 client.18798 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3ccd80740 10 client.18798 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3ccd80740 20 client.18798 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e142bfb0 0x55f2e142e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3ccd80740 4 client.18798 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2e10b6720 con 0x55f2e142bfb0 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e142bfb0 0x55f2e142e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3a2ffd640 10 client.18798 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.853+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e142bfb0 0x55f2e142e3a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=cef4f6fcad35ff43 server_cookie=55697333266fc3f3 in_seq=0 out_seq=0 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/3943707297 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f2e10b6720 con 0x55f2e142bfb0 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3a2ffd640 10 client.18798 handle_command_reply: tid=0 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 client.18798 shutdown 2024-02-09T23:00:18.864 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 msgr2=0x55f2e1418f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 0x55f2e1418f20 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4005fd0 tx=0x7ff3b406eb60 comp rx=0 tx=0).stop 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 shutdown_connections 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e142bfb0 0x55f2e142e3a0 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14199a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e14133e0 0x55f2e1419460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3a2ffd640 0 client.18798 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3943707297 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e1412a30 0x55f2e1418f20 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 >> 172.21.15.3:0/3943707297 conn(0x55f2e140d100 msgr2=0x55f2e140ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 shutdown_connections 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3943707297 wait complete. 2024-02-09T23:00:18.865 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.854+0000 7ff3ccd80740 20 client.18798 trim_cache size 0 max 16384 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 Processor -- start 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 -- start start 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e141b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e14197c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e1414250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e14088b0 con 0x55f2e14133e0 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e12ff130 con 0x55f2e141bf60 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e1417470 con 0x55f2e1412a30 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e14197c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e14197c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35514/0 (socket says 172.21.15.3:35514) 2024-02-09T23:00:18.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/3180273083 learned_addr learned my addr 172.21.15.3:0/3180273083 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e141b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/3180273083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 msgr2=0x55f2e141b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e141b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/3180273083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 msgr2=0x55f2e1414250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e1414250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.866+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/3180273083 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2e1314c90 con 0x55f2e14133e0 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e14197c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8005ab0 tx=0x7ff3b80058f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b11eb699ea6bf11 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3a97fa640 1 -- 172.21.15.3:0/3180273083 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8043040 con 0x55f2e14133e0 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3a97fa640 1 -- 172.21.15.3:0/3180273083 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b8045040 con 0x55f2e14133e0 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3a97fa640 1 -- 172.21.15.3:0/3180273083 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b802ce10 con 0x55f2e14133e0 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3180273083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 msgr2=0x55f2e14197c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e14197c0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8005ab0 tx=0x7ff3b80058f0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3180273083 shutdown_connections 2024-02-09T23:00:18.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e1414250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e14197c0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/3180273083 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e141b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3180273083 >> 172.21.15.3:0/3180273083 conn(0x55f2e140d100 msgr2=0x55f2e140d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3180273083 shutdown_connections 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.867+0000 7ff3ccd80740 1 -- 172.21.15.3:0/3180273083 wait complete. 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 -- start start 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 Processor -- start 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e1421ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e1422530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14823f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e1482930 con 0x55f2e14133e0 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e1482a70 con 0x55f2e141bf60 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2e1482bb0 con 0x55f2e1412a30 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3ccd80740 10 client.0 _ll_register_callbacks cb 0x7ff3b4096310 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:18.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e1422530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e1422530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35530/0 (socket says 172.21.15.3:35530) 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/261813811 learned_addr learned my addr 172.21.15.3:0/261813811 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e1421ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 msgr2=0x55f2e1421ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e1421ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/261813811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 msgr2=0x55f2e14823f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14823f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.868+0000 7ff3c2ffd640 1 -- 172.21.15.3:0/261813811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c410ea00 con 0x55f2e14133e0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3c2ffd640 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e1422530 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8036040 tx=0x7ff3b802cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55a01b99ae9c70b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/261813811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b8043040 con 0x55f2e14133e0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/261813811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff3b802ce20 con 0x55f2e14133e0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3b409ae30 con 0x55f2e14133e0 2024-02-09T23:00:18.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/261813811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3b8004510 con 0x55f2e14133e0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 10 client.18804 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55f2e133aa70 con 0x55f2e14133e0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/261813811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b409ae30 con 0x55f2e14133e0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/261813811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7ff3b8004df0 con 0x55f2e14133e0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 10 client.18804 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 10 client.18804 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 20 client.18804 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e1483fb0 0x55f2e14863d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 4 client.18804 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f2e1105890 con 0x55f2e1483fb0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.869+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e1483fb0 0x55f2e14863d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3a2ffd640 10 client.18804 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3c37fe640 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e1483fb0 0x55f2e14863d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d69b3d21792b3cce server_cookie=4f8d95ed72a7375a in_seq=0 out_seq=0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3a2ffd640 1 -- 172.21.15.3:0/261813811 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f2e1105890 con 0x55f2e1483fb0 2024-02-09T23:00:18.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3a2ffd640 10 client.18804 handle_command_reply: tid=0 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 client.18804 shutdown 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 msgr2=0x55f2e1422530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e1422530 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7ff3b8036040 tx=0x7ff3b802cd30 comp rx=0 tx=0).stop 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 shutdown_connections 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55f2e1483fb0 0x55f2e14863d0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2e141bf60 0x55f2e14823f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2e14133e0 0x55f2e1422530 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 --2- 172.21.15.3:0/261813811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2e1412a30 0x55f2e1421ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3a2ffd640 0 client.18804 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 >> 172.21.15.3:0/261813811 conn(0x55f2e140d100 msgr2=0x55f2e1414250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 shutdown_connections 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 1 -- 172.21.15.3:0/261813811 wait complete. 2024-02-09T23:00:18.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.870+0000 7ff3ccd80740 20 client.18804 trim_cache size 0 max 16384 2024-02-09T23:00:18.881 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3ac0a22b0 msgr2=0x7ff3ac0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3ac0a22b0 0x7ff3ac0a4770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc007740 tx=0x7ff3bc007e30 comp rx=0 tx=0).stop 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 msgr2=0x7ff3c419d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 0x7ff3c419d650 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400c890 tx=0x7ff3b400cd50 comp rx=0 tx=0).stop 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 shutdown_connections 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff3ac0a22b0 0x7ff3ac0a4770 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff3c410b910 0x7ff3c41977c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff3c4102dc0 0x7ff3c419db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 --2- 172.21.15.3:0/6894379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff3c41024a0 0x7ff3c419d650 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 >> 172.21.15.3:0/6894379 conn(0x7ff3c40772a0 msgr2=0x7ff3c40fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 shutdown_connections 2024-02-09T23:00:18.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.871+0000 7ff3ca0c9640 1 -- 172.21.15.3:0/6894379 wait complete. 2024-02-09T23:00:18.885 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:18.885 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:00:18.999 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 -- 172.21.15.3:0/3130614709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c8810b960 msgr2=0x7f1c88108230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 --2- 172.21.15.3:0/3130614709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c8810b960 0x7f1c88108230 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009900 tx=0x7f1c8402bcd0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 -- 172.21.15.3:0/3130614709 shutdown_connections 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 --2- 172.21.15.3:0/3130614709 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c88108770 0x7f1c88108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 --2- 172.21.15.3:0/3130614709 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c881162c0 0x7f1c881186d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 --2- 172.21.15.3:0/3130614709 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c8810b960 0x7f1c88108230 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 -- 172.21.15.3:0/3130614709 >> 172.21.15.3:0/3130614709 conn(0x7f1c8806c2a0 msgr2=0x7f1c8806c6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 -- 172.21.15.3:0/3130614709 shutdown_connections 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.993+0000 7f1c8f298640 1 -- 172.21.15.3:0/3130614709 wait complete. 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 Processor -- start 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 -- start start 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c88108770 0x7f1c881a2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8810b960 0x7f1c881a2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 0x7f1c881a6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c881a71e0 con 0x7f1c881162c0 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c881a7350 con 0x7f1c88108770 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8f298640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c881a7510 con 0x7f1c8810b960 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8d835640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 0x7f1c881a6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8d835640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 0x7f1c881a6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35556/0 (socket says 172.21.15.3:35556) 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8d835640 1 -- 172.21.15.3:0/840860666 learned_addr learned my addr 172.21.15.3:0/840860666 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8c833640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8810b960 0x7f1c881a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.994+0000 7f1c8d034640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c88108770 0x7f1c881a2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8d835640 1 -- 172.21.15.3:0/840860666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8810b960 msgr2=0x7f1c881a2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8d835640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8810b960 0x7f1c881a2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8d835640 1 -- 172.21.15.3:0/840860666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c88108770 msgr2=0x7f1c881a2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8d835640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c88108770 0x7f1c881a2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8d835640 1 -- 172.21.15.3:0/840860666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c84009560 con 0x7f1c881162c0 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8d835640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 0x7f1c881a6be0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000eb40 tx=0x7f1c8000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c7e7fc640 1 -- 172.21.15.3:0/840860666 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80004b20 con 0x7f1c881162c0 2024-02-09T23:00:19.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c881a77f0 con 0x7f1c881162c0 2024-02-09T23:00:19.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c7e7fc640 1 -- 172.21.15.3:0/840860666 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c80004c80 con 0x7f1c881162c0 2024-02-09T23:00:19.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c7e7fc640 1 -- 172.21.15.3:0/840860666 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c800107b0 con 0x7f1c881162c0 2024-02-09T23:00:19.002 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.995+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c881af070 con 0x7f1c881162c0 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.998+0000 7f1c7e7fc640 1 -- 172.21.15.3:0/840860666 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1c80005020 con 0x7f1c881162c0 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.998+0000 7f1c91e60740 1 Processor -- start 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.998+0000 7f1c91e60740 1 -- start start 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64008ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640095990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e6400983b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c91e60740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e640048020 con 0x55e64008d3e0 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c91e60740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e63fffb320 con 0x55e64008ca30 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c91e60740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e63fff8850 con 0x55e640095f60 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640095990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.010 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8c833640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64008ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640095990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35572/0 (socket says 172.21.15.3:35572) 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 -- 172.21.15.3:0/2784796809 learned_addr learned my addr 172.21.15.3:0/2784796809 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8c833640 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e6400983b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c7e7fc640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1c740a2230 0x7f1c740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 -- 172.21.15.3:0/2784796809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 msgr2=0x55e6400983b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e6400983b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 -- 172.21.15.3:0/2784796809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 msgr2=0x55e64008ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64008ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 -- 172.21.15.3:0/2784796809 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e63ffb6210 con 0x55e64008d3e0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d835640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1c740a2230 0x7f1c740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c7e7fc640 1 -- 172.21.15.3:0/840860666 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1c80014070 con 0x7f1c881162c0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c8d034640 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640095990 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009900 tx=0x7f1c8402c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108cd3dcd920efd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:18.999+0000 7f1c737fe640 1 -- 172.21.15.3:0/2784796809 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840046c0 con 0x55e64008d3e0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c8d835640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1c740a2230 0x7f1c740a46f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c8009b120 tx=0x7f1c80005960 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.011 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c737fe640 1 -- 172.21.15.3:0/2784796809 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c84004820 con 0x55e64008d3e0 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c737fe640 1 -- 172.21.15.3:0/2784796809 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84004df0 con 0x55e64008d3e0 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 -- 172.21.15.3:0/2784796809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 msgr2=0x55e640095990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640095990 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009900 tx=0x7f1c8402c6f0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 -- 172.21.15.3:0/2784796809 shutdown_connections 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e6400983b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640095990 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2784796809 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64008ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 -- 172.21.15.3:0/2784796809 >> 172.21.15.3:0/2784796809 conn(0x55e640087100 msgr2=0x55e6400874e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.000+0000 7f1c91e60740 1 -- 172.21.15.3:0/2784796809 shutdown_connections 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 -- 172.21.15.3:0/2784796809 wait complete. 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 -- start start 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 Processor -- start 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e640092ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.012 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640093430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e640093970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e63fff4030 con 0x55e64008d3e0 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e63fcc7430 con 0x55e64008ca30 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e63fc83e00 con 0x55e640095f60 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640093430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640093430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35580/0 (socket says 172.21.15.3:35580) 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c8d034640 1 -- 172.21.15.3:0/1670883097 learned_addr learned my addr 172.21.15.3:0/1670883097 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.001+0000 7f1c91e60740 10 client.0 _ll_register_callbacks cb 0x55e63ffe9030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8c833640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e640093970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8c833640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e640092ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8d034640 1 -- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 msgr2=0x55e640093970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8d034640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e640093970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8d034640 1 -- 172.21.15.3:0/1670883097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 msgr2=0x55e640092ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.013 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8d034640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e640092ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8d034640 1 -- 172.21.15.3:0/1670883097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c88119350 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c8d034640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640093430 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f1c8402c210 tx=0x7f1c8402bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65551f80d9bb8cd4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c70ff9640 1 -- 172.21.15.3:0/1670883097 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c840046c0 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c70ff9640 1 -- 172.21.15.3:0/1670883097 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1c8404d070 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e63fcf37e0 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c70ff9640 1 -- 172.21.15.3:0/1670883097 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c84004990 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c91e60740 10 client.18816 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55e63fff26b0 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.002+0000 7f1c70ff9640 1 -- 172.21.15.3:0/1670883097 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e63fcf37e0 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c70ff9640 1 -- 172.21.15.3:0/1670883097 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1c840344d0 con 0x55e64008d3e0 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c91e60740 10 client.18816 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c91e60740 10 client.18816 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c91e60740 20 client.18816 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c91e60740 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400a8400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c91e60740 4 client.18816 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e63fd30720 con 0x55e6400a6010 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c8c833640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400a8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c70ff9640 10 client.18816 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.003+0000 7f1c8c833640 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400a8400 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=178edd9a6ab277fa server_cookie=a41a34d3589df382 in_seq=0 out_seq=0 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c70ff9640 1 -- 172.21.15.3:0/1670883097 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e63fd30720 con 0x55e6400a6010 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c70ff9640 10 client.18816 handle_command_reply: tid=0 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 client.18816 shutdown 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 msgr2=0x55e640093430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640093430 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f1c8402c210 tx=0x7f1c8402bcd0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 shutdown_connections 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400a8400 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e640095f60 0x55e640093970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c70ff9640 0 client.18816 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e640093430 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 --2- 172.21.15.3:0/1670883097 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e640092ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 >> 172.21.15.3:0/1670883097 conn(0x55e640087100 msgr2=0x55e640088e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 shutdown_connections 2024-02-09T23:00:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 1 -- 172.21.15.3:0/1670883097 wait complete. 2024-02-09T23:00:19.016 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.004+0000 7f1c91e60740 20 client.18816 trim_cache size 0 max 16384 2024-02-09T23:00:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 Processor -- start 2024-02-09T23:00:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 -- start start 2024-02-09T23:00:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008ca30 0x55e640095620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e6400acd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.027 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e640095f60 0x55e640090170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6400828b0 con 0x55e64008d3e0 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e63ff79130 con 0x55e640095f60 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.016+0000 7f1c91e60740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e640086c30 con 0x55e64008ca30 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e6400acd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e6400acd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35592/0 (socket says 172.21.15.3:35592) 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 -- 172.21.15.3:0/2350662105 learned_addr learned my addr 172.21.15.3:0/2350662105 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8c833640 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008ca30 0x55e640095620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 -- 172.21.15.3:0/2350662105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008ca30 msgr2=0x55e640095620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008ca30 0x55e640095620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 -- 172.21.15.3:0/2350662105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e640095f60 msgr2=0x55e640090170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e640095f60 0x55e640090170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 -- 172.21.15.3:0/2350662105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e63ff8ec90 con 0x55e64008d3e0 2024-02-09T23:00:19.028 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8c833640 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008ca30 0x55e640095620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c8d034640 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e6400acd80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f1c84034800 tx=0x7f1c840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e98386b1ce06725 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c72ffd640 1 -- 172.21.15.3:0/2350662105 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84034440 con 0x55e64008d3e0 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c72ffd640 1 -- 172.21.15.3:0/2350662105 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c8404b070 con 0x55e64008d3e0 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.017+0000 7f1c72ffd640 1 -- 172.21.15.3:0/2350662105 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c84034e60 con 0x55e64008d3e0 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- 172.21.15.3:0/2350662105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 msgr2=0x55e6400acd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e6400acd80 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f1c84034800 tx=0x7f1c840046c0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- 172.21.15.3:0/2350662105 shutdown_connections 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e640095f60 0x55e640090170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e64008d3e0 0x55e6400acd80 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- 172.21.15.3:0/2350662105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008ca30 0x55e640095620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- 172.21.15.3:0/2350662105 >> 172.21.15.3:0/2350662105 conn(0x55e640087100 msgr2=0x55e6400874e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- 172.21.15.3:0/2350662105 shutdown_connections 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- 172.21.15.3:0/2350662105 wait complete. 2024-02-09T23:00:19.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- start start 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 Processor -- start 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64009cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008d3e0 0x55e64009d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 0x55e64009b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e64009bdf0 con 0x55e640095f60 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e64009bf60 con 0x55e64008ca30 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e64009c120 con 0x55e64008d3e0 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.018+0000 7f1c91e60740 10 client.0 _ll_register_callbacks cb 0x7f1c78005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 0x55e64009b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8d034640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008d3e0 0x55e64009d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 0x55e64009b6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35604/0 (socket says 172.21.15.3:35604) 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 -- 172.21.15.3:0/4050369337 learned_addr learned my addr 172.21.15.3:0/4050369337 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64009cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 -- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008d3e0 msgr2=0x55e64009d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008d3e0 0x55e64009d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 -- 172.21.15.3:0/4050369337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 msgr2=0x55e64009cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64009cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 -- 172.21.15.3:0/4050369337 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e63fff26b0 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c8c833640 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 0x55e64009b6f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f1c780058c0 tx=0x7f1c78009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc2757a90570a82 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c70ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c70ff9640 1 -- 172.21.15.3:0/4050369337 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c78016040 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c70ff9640 1 -- 172.21.15.3:0/4050369337 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1c780237d0 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1c84004820 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c70ff9640 1 -- 172.21.15.3:0/4050369337 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1c780035f0 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c91e60740 10 client.18822 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.019+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55e63ffb4a70 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c70ff9640 1 -- 172.21.15.3:0/4050369337 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c7803d020 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c70ff9640 1 -- 172.21.15.3:0/4050369337 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1c78021600 con 0x55e640095f60 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 10 client.18822 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:19.031 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 10 client.18822 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 20 client.18822 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400ff970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 4 client.18822 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e63fd7f890 con 0x55e6400a6010 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c8d034640 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400ff970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c70ff9640 10 client.18822 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c8d034640 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400ff970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1dfb980469bd8d3 server_cookie=976760ed2a7df400 in_seq=0 out_seq=0 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c70ff9640 1 -- 172.21.15.3:0/4050369337 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e63fd7f890 con 0x55e6400a6010 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c70ff9640 10 client.18822 handle_command_reply: tid=0 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 client.18822 shutdown 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 msgr2=0x55e64009b6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 0x55e64009b6f0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f1c780058c0 tx=0x7f1c78009ac0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 shutdown_connections 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e6400a6010 0x55e6400ff970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c70ff9640 0 client.18822 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.020+0000 7f1c91e60740 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e640095f60 0x55e64009b6f0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c91e60740 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e64008d3e0 0x55e64009d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c91e60740 1 --2- 172.21.15.3:0/4050369337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e64008ca30 0x55e64009cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 >> 172.21.15.3:0/4050369337 conn(0x55e640087100 msgr2=0x55e6400acd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 shutdown_connections 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c91e60740 1 -- 172.21.15.3:0/4050369337 wait complete. 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c91e60740 20 client.18822 trim_cache size 0 max 16384 2024-02-09T23:00:19.033 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:19.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1c740a2230 msgr2=0x7f1c740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1c740a2230 0x7f1c740a46f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f1c8009b120 tx=0x7f1c80005960 comp rx=0 tx=0).stop 2024-02-09T23:00:19.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 msgr2=0x7f1c881a6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 0x7f1c881a6be0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f1c8000eb40 tx=0x7f1c8000c2d0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 shutdown_connections 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1c740a2230 0x7f1c740a46f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1c881162c0 0x7f1c881a6be0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1c8810b960 0x7f1c881a2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 --2- 172.21.15.3:0/840860666 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1c88108770 0x7f1c881a2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 >> 172.21.15.3:0/840860666 conn(0x7f1c8806c2a0 msgr2=0x7f1c881166c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 shutdown_connections 2024-02-09T23:00:19.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.021+0000 7f1c8f298640 1 -- 172.21.15.3:0/840860666 wait complete. 2024-02-09T23:00:19.036 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:19.036 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:00:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 -- 172.21.15.3:0/1707897473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac100330 msgr2=0x7f19ac100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 --2- 172.21.15.3:0/1707897473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac100330 0x7f19ac100710 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f19a4009930 tx=0x7f19a402bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 -- 172.21.15.3:0/1707897473 shutdown_connections 2024-02-09T23:00:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 --2- 172.21.15.3:0/1707897473 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19ac105400 0x7f19ac10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 --2- 172.21.15.3:0/1707897473 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100ce0 0x7f19ac104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 --2- 172.21.15.3:0/1707897473 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac100330 0x7f19ac100710 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 -- 172.21.15.3:0/1707897473 >> 172.21.15.3:0/1707897473 conn(0x7f19ac077250 msgr2=0x7f19ac077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 -- 172.21.15.3:0/1707897473 shutdown_connections 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.138+0000 7f19b35bc640 1 -- 172.21.15.3:0/1707897473 wait complete. 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 Processor -- start 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 -- start start 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100330 0x7f19ac101dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19ac105400 0x7f19ac102300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 0x7f19ac1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac1a6030 con 0x7f19ac102840 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac1a61a0 con 0x7f19ac105400 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b35bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac1a6360 con 0x7f19ac100330 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 0x7f19ac1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 0x7f19ac1a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35630/0 (socket says 172.21.15.3:35630) 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 -- 172.21.15.3:0/2256123820 learned_addr learned my addr 172.21.15.3:0/2256123820 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1358640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100330 0x7f19ac101dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b0b57640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19ac105400 0x7f19ac102300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 -- 172.21.15.3:0/2256123820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100330 msgr2=0x7f19ac101dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100330 0x7f19ac101dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 -- 172.21.15.3:0/2256123820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19ac105400 msgr2=0x7f19ac102300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19ac105400 0x7f19ac102300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 -- 172.21.15.3:0/2256123820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a4009560 con 0x7f19ac102840 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1358640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100330 0x7f19ac101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.139+0000 7f19b1b59640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 0x7f19ac1a5a60 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f199c00eaa0 tx=0x7f199c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.140+0000 7f19aa7fc640 1 -- 172.21.15.3:0/2256123820 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c00cdb0 con 0x7f19ac102840 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.140+0000 7f19aa7fc640 1 -- 172.21.15.3:0/2256123820 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f199c005020 con 0x7f19ac102840 2024-02-09T23:00:19.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.140+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19ac1a6640 con 0x7f19ac102840 2024-02-09T23:00:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.140+0000 7f19aa7fc640 1 -- 172.21.15.3:0/2256123820 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199c0116b0 con 0x7f19ac102840 2024-02-09T23:00:19.147 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.140+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19ac1a6b00 con 0x7f19ac102840 2024-02-09T23:00:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19aa7fc640 1 -- 172.21.15.3:0/2256123820 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f199c005520 con 0x7f19ac102840 2024-02-09T23:00:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 Processor -- start 2024-02-09T23:00:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 -- start start 2024-02-09T23:00:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fdd5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fdde990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fddef60 0x555c1fde13b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fd91020 con 0x555c1fdd5a30 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fd44320 con 0x555c1fdd63e0 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b6191740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fd41850 con 0x555c1fddef60 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19aa7fc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19940a2260 0x7f19940a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.142+0000 7f19b0b57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fdd5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fdd5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35642/0 (socket says 172.21.15.3:35642) 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 -- 172.21.15.3:0/787349900 learned_addr learned my addr 172.21.15.3:0/787349900 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b1b59640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19940a2260 0x7f19940a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fddef60 0x555c1fde13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b1358640 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fdde990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19aa7fc640 1 -- 172.21.15.3:0/2256123820 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f199c0ce330 con 0x7f19ac102840 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 -- 172.21.15.3:0/787349900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fddef60 msgr2=0x555c1fde13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fddef60 0x555c1fde13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 -- 172.21.15.3:0/787349900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 msgr2=0x555c1fdde990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fdde990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 -- 172.21.15.3:0/787349900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c1fcff210 con 0x555c1fdd5a30 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b1358640 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fdde990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b0b57640 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fdd5e10 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f19a000bf70 tx=0x7f19a000b440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7ef4089d96a9b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19b1b59640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19940a2260 0x7f19940a4720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f199c0092f0 tx=0x7f199c06ab80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19937fe640 1 -- 172.21.15.3:0/787349900 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0012860 con 0x555c1fdd5a30 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19937fe640 1 -- 172.21.15.3:0/787349900 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19a0018070 con 0x555c1fdd5a30 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.143+0000 7f19937fe640 1 -- 172.21.15.3:0/787349900 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0011c20 con 0x555c1fdd5a30 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 -- 172.21.15.3:0/787349900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 msgr2=0x555c1fdd5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.156 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fdd5e10 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f19a000bf70 tx=0x7f19a000b440 comp rx=0 tx=0).stop 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 -- 172.21.15.3:0/787349900 shutdown_connections 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fdde990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fddef60 0x555c1fde13b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 --2- 172.21.15.3:0/787349900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fdd5e10 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 -- 172.21.15.3:0/787349900 >> 172.21.15.3:0/787349900 conn(0x555c1fdd0100 msgr2=0x555c1fdd04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 -- 172.21.15.3:0/787349900 shutdown_connections 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 -- 172.21.15.3:0/787349900 wait complete. 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 -- start start 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.144+0000 7f19b6191740 1 Processor -- start 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fddbf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 0x555c1fddc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fddef60 0x555c1fddc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.157 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fd3d030 con 0x555c1fdd63e0 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fa10430 con 0x555c1fddef60 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1f9cce00 con 0x555c1fdd5a30 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b6191740 10 client.0 _ll_register_callbacks cb 0x555c1fd32030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 0x555c1fddc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 0x555c1fddc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35658/0 (socket says 172.21.15.3:35658) 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 -- 172.21.15.3:0/1658019139 learned_addr learned my addr 172.21.15.3:0/1658019139 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b0b57640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fddbf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 -- 172.21.15.3:0/1658019139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 msgr2=0x555c1fddbf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fddbf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 -- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fddef60 msgr2=0x555c1fddc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fddef60 0x555c1fddc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.158 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 -- 172.21.15.3:0/1658019139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a000bb70 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b0b57640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fddbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f19b1358640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 0x555c1fddc440 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005ab0 tx=0x7f19a402ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95099f0c012d81e3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f1990ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.145+0000 7f1990ff9640 1 -- 172.21.15.3:0/1658019139 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a4004450 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f1990ff9640 1 -- 172.21.15.3:0/1658019139 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f19a4004df0 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555c1fa3c7e0 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f1990ff9640 1 -- 172.21.15.3:0/1658019139 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19a402c640 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 10 client.18834 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555c1fd3b6b0 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f1990ff9640 1 -- 172.21.15.3:0/1658019139 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555c1fa3c7e0 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f1990ff9640 1 -- 172.21.15.3:0/1658019139 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f19a403c040 con 0x555c1fdd63e0 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 10 client.18834 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 10 client.18834 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:00:19.159 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 20 client.18834 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fdef4e0 0x555c1fdf1900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 4 client.18834 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555c1fa79720 con 0x555c1fdef4e0 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.146+0000 7f19b0b57640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fdef4e0 0x555c1fdf1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.147+0000 7f1990ff9640 10 client.18834 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.147+0000 7f19b0b57640 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fdef4e0 0x555c1fdf1900 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=860c4b1b89ff1d7e server_cookie=53dddaf4452dc4cd in_seq=0 out_seq=0 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.147+0000 7f1990ff9640 1 -- 172.21.15.3:0/1658019139 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555c1fa79720 con 0x555c1fdef4e0 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.147+0000 7f1990ff9640 10 client.18834 handle_command_reply: tid=0 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.147+0000 7f19b6191740 1 client.18834 shutdown 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 msgr2=0x555c1fddc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 0x555c1fddc440 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f19a4005ab0 tx=0x7f19a402ca90 comp rx=0 tx=0).stop 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 shutdown_connections 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fdef4e0 0x555c1fdf1900 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fddef60 0x555c1fddc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.160 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f1990ff9640 0 client.18834 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd63e0 0x555c1fddc440 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 --2- 172.21.15.3:0/1658019139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fddbf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 >> 172.21.15.3:0/1658019139 conn(0x555c1fdd0100 msgr2=0x555c1fdd1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 shutdown_connections 2024-02-09T23:00:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 1 -- 172.21.15.3:0/1658019139 wait complete. 2024-02-09T23:00:19.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.148+0000 7f19b6191740 20 client.18834 trim_cache size 0 max 16384 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 Processor -- start 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 -- start start 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 0x555c1fdd7210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fdd7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fddc450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fcc2130 con 0x555c1fdd0100 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fdef2e0 con 0x555c1fdd63e0 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b6191740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fdcba40 con 0x555c1fdd5a30 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 0x555c1fdd7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.171 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 0x555c1fdd7210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35666/0 (socket says 172.21.15.3:35666) 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 -- 172.21.15.3:0/324596614 learned_addr learned my addr 172.21.15.3:0/324596614 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b1358640 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fdd7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fddc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 -- 172.21.15.3:0/324596614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 msgr2=0x555c1fdd7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fdd7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 -- 172.21.15.3:0/324596614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 msgr2=0x555c1fddc450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fddc450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 -- 172.21.15.3:0/324596614 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555c1fcd7c90 con 0x555c1fdd0100 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b1358640 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fdd7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f19b0b57640 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 0x555c1fdd7210 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f19a00058c0 tx=0x7f19a0011800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a6f457714c2220 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f1992ffd640 1 -- 172.21.15.3:0/324596614 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0022040 con 0x555c1fdd0100 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.160+0000 7f1992ffd640 1 -- 172.21.15.3:0/324596614 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19a000b4b0 con 0x555c1fdd0100 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f1992ffd640 1 -- 172.21.15.3:0/324596614 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a0012660 con 0x555c1fdd0100 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 -- 172.21.15.3:0/324596614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 msgr2=0x555c1fdd7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.172 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 0x555c1fdd7210 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f19a00058c0 tx=0x7f19a0011800 comp rx=0 tx=0).stop 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 -- 172.21.15.3:0/324596614 shutdown_connections 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fddc450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd5a30 0x555c1fdd7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 --2- 172.21.15.3:0/324596614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd0100 0x555c1fdd7210 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 -- 172.21.15.3:0/324596614 >> 172.21.15.3:0/324596614 conn(0x555c1fdd6b30 msgr2=0x555c1fdd04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 -- 172.21.15.3:0/324596614 shutdown_connections 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 -- 172.21.15.3:0/324596614 wait complete. 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 -- start start 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.161+0000 7f19b6191740 1 Processor -- start 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd0100 0x555c1fde6430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fde47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fde4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fde54d0 con 0x555c1fdd5a30 2024-02-09T23:00:19.173 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fde5640 con 0x555c1fdd63e0 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555c1fe45a20 con 0x555c1fdd0100 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b6191740 10 client.0 _ll_register_callbacks cb 0x7f19a000b2c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fde47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fde47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35670/0 (socket says 172.21.15.3:35670) 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 -- 172.21.15.3:0/3277945515 learned_addr learned my addr 172.21.15.3:0/3277945515 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b0b57640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fde4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b0b57640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd0100 0x555c1fde6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 -- 172.21.15.3:0/3277945515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd0100 msgr2=0x555c1fde6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd0100 0x555c1fde6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 -- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 msgr2=0x555c1fde4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fde4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 -- 172.21.15.3:0/3277945515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555c1fcff210 con 0x555c1fdd5a30 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b0b57640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd0100 0x555c1fde6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:19.174 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f19b1358640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fde47d0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x555c1fdcbbf0 tx=0x7f19a4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592e029652a12514 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.162+0000 7f1990ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f1990ff9640 1 -- 172.21.15.3:0/3277945515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19a4004450 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f1990ff9640 1 -- 172.21.15.3:0/3277945515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f19a4000e40 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f19a0018070 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f1990ff9640 1 -- 172.21.15.3:0/3277945515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19a40035f0 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 10 client.18840 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555c1fcfda70 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f1990ff9640 1 -- 172.21.15.3:0/3277945515 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19a405f080 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f1990ff9640 1 -- 172.21.15.3:0/3277945515 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f19a4059030 con 0x555c1fdd5a30 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 10 client.18840 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 10 client.18840 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:00:19.175 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 20 client.18840 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fddef60 0x555c1fe48ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 4 client.18840 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.163+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555c1fac8890 con 0x555c1fddef60 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b0b57640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fddef60 0x555c1fe48ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f1990ff9640 10 client.18840 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b0b57640 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fddef60 0x555c1fe48ee0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1d60d77e8c44a780 server_cookie=a468057cc0e30151 in_seq=0 out_seq=0 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f1990ff9640 1 -- 172.21.15.3:0/3277945515 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555c1fac8890 con 0x555c1fddef60 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f1990ff9640 10 client.18840 handle_command_reply: tid=0 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 client.18840 shutdown 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 msgr2=0x555c1fde47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fde47d0 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x555c1fdcbbf0 tx=0x7f19a4005dc0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.176 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 shutdown_connections 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555c1fddef60 0x555c1fe48ee0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555c1fdd63e0 0x555c1fde4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555c1fdd5a30 0x555c1fde47d0 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 --2- 172.21.15.3:0/3277945515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555c1fdd0100 0x555c1fde6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 >> 172.21.15.3:0/3277945515 conn(0x555c1fdd6b30 msgr2=0x555c1fdd7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f1990ff9640 0 client.18840 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 shutdown_connections 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 1 -- 172.21.15.3:0/3277945515 wait complete. 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.164+0000 7f19b6191740 20 client.18840 trim_cache size 0 max 16384 2024-02-09T23:00:19.177 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19940a2260 msgr2=0x7f19940a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19940a2260 0x7f19940a4720 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f199c0092f0 tx=0x7f199c06ab80 comp rx=0 tx=0).stop 2024-02-09T23:00:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 msgr2=0x7f19ac1a5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 0x7f19ac1a5a60 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 crypto rx=0x7f199c00eaa0 tx=0x7f199c00ef60 comp rx=0 tx=0).stop 2024-02-09T23:00:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 shutdown_connections 2024-02-09T23:00:19.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19940a2260 0x7f19940a4720 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f19ac105400 0x7f19ac102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f19ac102840 0x7f19ac1a5a60 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 --2- 172.21.15.3:0/2256123820 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f19ac100330 0x7f19ac101dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 >> 172.21.15.3:0/2256123820 conn(0x7f19ac077250 msgr2=0x7f19ac1040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 shutdown_connections 2024-02-09T23:00:19.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.165+0000 7f19b35bc640 1 -- 172.21.15.3:0/2256123820 wait complete. 2024-02-09T23:00:19.180 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:19.180 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:00:19.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 -- 172.21.15.3:0/1993556374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 msgr2=0x7f28ec100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1993556374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec100720 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009930 tx=0x7f28e802bc20 comp rx=0 tx=0).stop 2024-02-09T23:00:19.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 -- 172.21.15.3:0/1993556374 shutdown_connections 2024-02-09T23:00:19.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1993556374 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28ec100cf0 0x7f28ec104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.290 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1993556374 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28ec105410 0x7f28ec10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1993556374 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec100720 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 -- 172.21.15.3:0/1993556374 >> 172.21.15.3:0/1993556374 conn(0x7f28ec0772e0 msgr2=0x7f28ec0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 -- 172.21.15.3:0/1993556374 shutdown_connections 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.284+0000 7f28f47a1640 1 -- 172.21.15.3:0/1993556374 wait complete. 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 Processor -- start 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 -- start start 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28ec105410 0x7f28ec102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28ec102850 0x7f28ec1a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec1a5ff0 con 0x7f28ec100340 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec1a6160 con 0x7f28ec102850 2024-02-09T23:00:19.291 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f47a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec1a6320 con 0x7f28ec105410 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec101dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35680/0 (socket says 172.21.15.3:35680) 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 -- 172.21.15.3:0/1672935974 learned_addr learned my addr 172.21.15.3:0/1672935974 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f2d3e640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28ec102850 0x7f28ec1a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28ec105410 0x7f28ec102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 -- 172.21.15.3:0/1672935974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28ec105410 msgr2=0x7f28ec102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28ec105410 0x7f28ec102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 -- 172.21.15.3:0/1672935974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28ec102850 msgr2=0x7f28ec1a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28ec102850 0x7f28ec1a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.292 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 -- 172.21.15.3:0/1672935974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e8009560 con 0x7f28ec100340 2024-02-09T23:00:19.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28f253d640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec101dd0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009230 tx=0x7f28e802c6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.285+0000 7f28e37fe640 1 -- 172.21.15.3:0/1672935974 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8004350 con 0x7f28ec100340 2024-02-09T23:00:19.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.286+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec1a65a0 con 0x7f28ec100340 2024-02-09T23:00:19.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.286+0000 7f28e37fe640 1 -- 172.21.15.3:0/1672935974 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28e80044b0 con 0x7f28ec100340 2024-02-09T23:00:19.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.286+0000 7f28e37fe640 1 -- 172.21.15.3:0/1672935974 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e8003d70 con 0x7f28ec100340 2024-02-09T23:00:19.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.286+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec1a6ab0 con 0x7f28ec100340 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.288+0000 7f28e37fe640 1 -- 172.21.15.3:0/1672935974 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f28e8004df0 con 0x7f28ec100340 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.288+0000 7f28f7379740 1 Processor -- start 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.288+0000 7f28f7379740 1 -- start start 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f7379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f7379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f7379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 0x55d73b1d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f7379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b184020 con 0x55d73b1d1f60 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f7379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b137320 con 0x55d73b1c93e0 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28e37fe640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f28d80a2260 0x7f28d80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f7379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b134850 con 0x55d73b1c8a30 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f2d3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 0x55d73b1d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f2d3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54096/0 (socket says 172.21.15.3:54096) 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f2d3e640 1 -- 172.21.15.3:0/2785705751 learned_addr learned my addr 172.21.15.3:0/2785705751 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 0x55d73b1d43b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35696/0 (socket says 172.21.15.3:35696) 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f253d640 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f2d3e640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f28d80a2260 0x7f28d80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28e37fe640 1 -- 172.21.15.3:0/1672935974 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f28e80f0130 con 0x7f28ec100340 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 -- 172.21.15.3:0/2785705751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 msgr2=0x55d73b1c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.302 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 -- 172.21.15.3:0/2785705751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 msgr2=0x55d73b1d1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 -- 172.21.15.3:0/2785705751 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d73b0f2210 con 0x55d73b1d1f60 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f2d3e640 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 0x55d73b1d43b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f28ec06a920 tx=0x7f28dc00e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25ed71c6f7dc62ba server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28f2d3e640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f28d80a2260 0x7f28d80a4720 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f28e4005c40 tx=0x7f28e4009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.289+0000 7f28e0ff9640 1 -- 172.21.15.3:0/2785705751 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc004020 con 0x55d73b1d1f60 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28e0ff9640 1 -- 172.21.15.3:0/2785705751 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28dc004180 con 0x55d73b1d1f60 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28e0ff9640 1 -- 172.21.15.3:0/2785705751 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc0107b0 con 0x55d73b1d1f60 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 -- 172.21.15.3:0/2785705751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 msgr2=0x55d73b1d43b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 0x55d73b1d43b0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x7f28ec06a920 tx=0x7f28dc00e350 comp rx=0 tx=0).stop 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 -- 172.21.15.3:0/2785705751 shutdown_connections 2024-02-09T23:00:19.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1d1f60 0x55d73b1d43b0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 --2- 172.21.15.3:0/2785705751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 -- 172.21.15.3:0/2785705751 >> 172.21.15.3:0/2785705751 conn(0x55d73b1c3100 msgr2=0x55d73b1c34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 -- 172.21.15.3:0/2785705751 shutdown_connections 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.290+0000 7f28f7379740 1 -- 172.21.15.3:0/2785705751 wait complete. 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 -- start start 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 Processor -- start 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 0x55d73b1ceee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1d1f60 0x55d73b1cf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b130030 con 0x55d73b1c8a30 2024-02-09T23:00:19.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73ae03430 con 0x55d73b1d1f60 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73adbfe00 con 0x55d73b1c93e0 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f7379740 10 client.0 _ll_register_callbacks cb 0x55d73b125030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 0x55d73b1ceee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 0x55d73b1ceee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35708/0 (socket says 172.21.15.3:35708) 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 -- 172.21.15.3:0/3887493671 learned_addr learned my addr 172.21.15.3:0/3887493671 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f2d3e640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1d1f60 0x55d73b1cf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 -- 172.21.15.3:0/3887493671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c93e0 msgr2=0x55d73b1cf420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 -- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1d1f60 msgr2=0x55d73b1cf960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1d1f60 0x55d73b1cf960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f253d640 1 -- 172.21.15.3:0/3887493671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28dc00c7d0 con 0x55d73b1c8a30 2024-02-09T23:00:19.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.291+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f253d640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 0x55d73b1ceee0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009430 tx=0x7f28e8005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b95aafb0fc448b0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28d5ffb640 1 -- 172.21.15.3:0/3887493671 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80bcbd0 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28d5ffb640 1 -- 172.21.15.3:0/3887493671 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f28e80c0d20 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d73ae2f7e0 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28d5ffb640 1 -- 172.21.15.3:0/3887493671 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28e80c5370 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 10 client.18852 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55d73b12e6b0 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28d5ffb640 1 -- 172.21.15.3:0/3887493671 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28e8086020 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28d5ffb640 1 -- 172.21.15.3:0/3887493671 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f28e803c070 con 0x55d73b1c8a30 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 10 client.18852 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 10 client.18852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 20 client.18852 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:19.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1e21c0 0x55d73b1e45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 4 client.18852 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.292+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d73ae6c720 con 0x55d73b1e21c0 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.293+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1e21c0 0x55d73b1e45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.293+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1e21c0 0x55d73b1e45d0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a158548956cd3d4a server_cookie=3f78661b750e5ada in_seq=0 out_seq=0 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.293+0000 7f28d5ffb640 10 client.18852 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28d5ffb640 1 -- 172.21.15.3:0/3887493671 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d73ae6c720 con 0x55d73b1e21c0 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28d5ffb640 10 client.18852 handle_command_reply: tid=0 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 client.18852 shutdown 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 msgr2=0x55d73b1ceee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 0x55d73b1ceee0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009430 tx=0x7f28e8005fd0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 shutdown_connections 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1e21c0 0x55d73b1e45d0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1d1f60 0x55d73b1cf960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.307 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28d5ffb640 0 client.18852 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 --2- 172.21.15.3:0/3887493671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c8a30 0x55d73b1ceee0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 >> 172.21.15.3:0/3887493671 conn(0x55d73b1c3100 msgr2=0x55d73b1c4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 shutdown_connections 2024-02-09T23:00:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 1 -- 172.21.15.3:0/3887493671 wait complete. 2024-02-09T23:00:19.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.294+0000 7f28f7379740 20 client.18852 trim_cache size 0 max 16384 2024-02-09T23:00:19.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 Processor -- start 2024-02-09T23:00:19.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 -- start start 2024-02-09T23:00:19.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1ca210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1ca750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.316 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b0b5130 con 0x55d73b1c3100 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b1e9820 con 0x55d73b1c93e0 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f7379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b1cd430 con 0x55d73b1c8a30 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35720/0 (socket says 172.21.15.3:35720) 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.306+0000 7f28f253d640 1 -- 172.21.15.3:0/3832584466 learned_addr learned my addr 172.21.15.3:0/3832584466 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f253d640 1 -- 172.21.15.3:0/3832584466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 msgr2=0x55d73b1ca750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1ca750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f253d640 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1ca750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f2d3e640 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f253d640 1 -- 172.21.15.3:0/3832584466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 msgr2=0x55d73b1cf430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f253d640 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f253d640 1 -- 172.21.15.3:0/3832584466 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d73b0cac90 con 0x55d73b1c3100 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:19.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f253d640 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1ca210 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f28e8003fd0 tx=0x7f28e8003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa926226568de5ca server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28d7fff640 1 -- 172.21.15.3:0/3832584466 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80bc9a0 con 0x55d73b1c3100 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28d7fff640 1 -- 172.21.15.3:0/3832584466 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28e8086040 con 0x55d73b1c3100 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28d7fff640 1 -- 172.21.15.3:0/3832584466 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80eda10 con 0x55d73b1c3100 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 -- 172.21.15.3:0/3832584466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 msgr2=0x55d73b1ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1ca210 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f28e8003fd0 tx=0x7f28e8003840 comp rx=0 tx=0).stop 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 -- 172.21.15.3:0/3832584466 shutdown_connections 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1cf430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 --2- 172.21.15.3:0/3832584466 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1ca210 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 -- 172.21.15.3:0/3832584466 >> 172.21.15.3:0/3832584466 conn(0x55d73b1c9b30 msgr2=0x55d73b1c34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 -- 172.21.15.3:0/3832584466 shutdown_connections 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.307+0000 7f28f7379740 1 -- 172.21.15.3:0/3832584466 wait complete. 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 -- start start 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 Processor -- start 2024-02-09T23:00:19.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1d9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1d7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b1d84e0 con 0x55d73b1c3100 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b1d8650 con 0x55d73b1c93e0 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d73b238700 con 0x55d73b1c8a30 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f7379740 10 client.0 _ll_register_callbacks cb 0x7f28dc00c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1d9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1d9420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35736/0 (socket says 172.21.15.3:35736) 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 -- 172.21.15.3:0/2538162924 learned_addr learned my addr 172.21.15.3:0/2538162924 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1d7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 -- 172.21.15.3:0/2538162924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 msgr2=0x55d73b1d7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1d7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 -- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 msgr2=0x55d73b1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f253d640 1 -- 172.21.15.3:0/2538162924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d73b12e6b0 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.308+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1d7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f253d640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1d9420 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f28e80bcb60 tx=0x7f28e8005cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8627ab0d9401af8e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28d5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28d5ffb640 1 -- 172.21.15.3:0/2538162924 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28e80bc9a0 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28d5ffb640 1 -- 172.21.15.3:0/2538162924 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f28e80c0d20 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28dc004180 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28d5ffb640 1 -- 172.21.15.3:0/2538162924 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28e80bcbd0 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 10 client.18858 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55d73b0f0a70 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28d5ffb640 1 -- 172.21.15.3:0/2538162924 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28dc004180 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28d5ffb640 1 -- 172.21.15.3:0/2538162924 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f28e803c070 con 0x55d73b1c3100 2024-02-09T23:00:19.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 10 client.18858 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 10 client.18858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 20 client.18858 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1d1f60 0x55d73b23b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 4 client.18858 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d73aebb890 con 0x55d73b1d1f60 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.309+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1d1f60 0x55d73b23b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f1d3c640 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1d1f60 0x55d73b23b970 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c3e52a16e4ee295 server_cookie=4b67574bad8c3244 in_seq=0 out_seq=0 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28d5ffb640 10 client.18858 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28d5ffb640 1 -- 172.21.15.3:0/2538162924 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d73aebb890 con 0x55d73b1d1f60 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28d5ffb640 10 client.18858 handle_command_reply: tid=0 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 client.18858 shutdown 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 msgr2=0x55d73b1d9420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1d9420 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 crypto rx=0x7f28e80bcb60 tx=0x7f28e8005cb0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 shutdown_connections 2024-02-09T23:00:19.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55d73b1d1f60 0x55d73b23b970 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d73b1c93e0 0x55d73b1d7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d73b1c8a30 0x55d73b1d7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28d5ffb640 0 client.18858 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 --2- 172.21.15.3:0/2538162924 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d73b1c3100 0x55d73b1d9420 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 >> 172.21.15.3:0/2538162924 conn(0x55d73b1c9b30 msgr2=0x55d73b1ca210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 shutdown_connections 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 1 -- 172.21.15.3:0/2538162924 wait complete. 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.310+0000 7f28f7379740 20 client.18858 trim_cache size 0 max 16384 2024-02-09T23:00:19.322 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:19.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f28d80a2260 msgr2=0x7f28d80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f28d80a2260 0x7f28d80a4720 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f28e4005c40 tx=0x7f28e4009500 comp rx=0 tx=0).stop 2024-02-09T23:00:19.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 msgr2=0x7f28ec101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:19.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec101dd0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f28e8009230 tx=0x7f28e802c6b0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 shutdown_connections 2024-02-09T23:00:19.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f28d80a2260 0x7f28d80a4720 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f28ec102850 0x7f28ec1a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f28ec105410 0x7f28ec102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 --2- 172.21.15.3:0/1672935974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f28ec100340 0x7f28ec101dd0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 >> 172.21.15.3:0/1672935974 conn(0x7f28ec0772e0 msgr2=0x7f28ec0687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 shutdown_connections 2024-02-09T23:00:19.324 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:19.311+0000 7f28f47a1640 1 -- 172.21.15.3:0/1672935974 wait complete. 2024-02-09T23:00:19.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:19 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/416927351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:00:19.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 267752 2.73 MB/sec execute 373 sec latency 1439.218 ms 2024-02-09T23:00:19.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:19 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/416927351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:00:19.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:19 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/416927351' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:00:20.325 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:00:20.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:00:20.326 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:00:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 -- 172.21.15.3:0/1910516760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87801064e0 msgr2=0x7f878010e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 --2- 172.21.15.3:0/1910516760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87801064e0 0x7f878010e0b0 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f8778009cd0 tx=0x7f877802c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 -- 172.21.15.3:0/1910516760 shutdown_connections 2024-02-09T23:00:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 --2- 172.21.15.3:0/1910516760 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87800691a0 0x7f8780105e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 --2- 172.21.15.3:0/1910516760 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f87801064e0 0x7f878010e0b0 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 --2- 172.21.15.3:0/1910516760 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 0x7f8780068bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 -- 172.21.15.3:0/1910516760 >> 172.21.15.3:0/1910516760 conn(0x7f8780077230 msgr2=0x7f8780077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 -- 172.21.15.3:0/1910516760 shutdown_connections 2024-02-09T23:00:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 -- 172.21.15.3:0/1910516760 wait complete. 2024-02-09T23:00:20.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.437+0000 7f8785e76640 1 Processor -- start 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 -- start start 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 0x7f87801000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87801064e0 0x7f8780100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 0x7f87801a59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87801a5fb0 con 0x7f8780100b70 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87801a6120 con 0x7f87801064e0 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f8785e76640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87801a62e0 con 0x7f87800687f0 2024-02-09T23:00:20.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 0x7f87801a59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 0x7f87801a59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35750/0 (socket says 172.21.15.3:35750) 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 -- 172.21.15.3:0/1047113739 learned_addr learned my addr 172.21.15.3:0/1047113739 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 0x7f87801000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877effd640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87801064e0 0x7f8780100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 -- 172.21.15.3:0/1047113739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 msgr2=0x7f87801000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 0x7f87801000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 -- 172.21.15.3:0/1047113739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87801064e0 msgr2=0x7f8780100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87801064e0 0x7f8780100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 -- 172.21.15.3:0/1047113739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8778009900 con 0x7f8780100b70 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877f7fe640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 0x7f87801000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.438+0000 7f877ffff640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 0x7f87801a59e0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f8778004af0 tx=0x7f8778034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.439+0000 7f877cff9640 1 -- 172.21.15.3:0/1047113739 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780328c0 con 0x7f8780100b70 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.439+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87801a6560 con 0x7f8780100b70 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.439+0000 7f877cff9640 1 -- 172.21.15.3:0/1047113739 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8778003e50 con 0x7f8780100b70 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.439+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87801a6a60 con 0x7f8780100b70 2024-02-09T23:00:20.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.439+0000 7f877cff9640 1 -- 172.21.15.3:0/1047113739 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8778004860 con 0x7f8780100b70 2024-02-09T23:00:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.441+0000 7f877cff9640 1 -- 172.21.15.3:0/1047113739 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f877800a040 con 0x7f8780100b70 2024-02-09T23:00:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.441+0000 7f8788a48740 1 Processor -- start 2024-02-09T23:00:20.453 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.441+0000 7f8788a48740 1 -- start start 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.441+0000 7f877cff9640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f87680a2260 0x7f87680a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f8788a48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x5559347fd570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f8788a48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934806150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f8788a48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934808a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f8788a48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55593476b000 con 0x555934806690 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f8788a48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555934768530 con 0x5559347fd190 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f8788a48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555934763d10 con 0x5559347fdab0 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877effd640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f87680a2260 0x7f87680a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877cff9640 1 -- 172.21.15.3:0/1047113739 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f87780f0850 con 0x7f8780100b70 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x5559347fd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934806150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.454 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x5559347fd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:60380/0 (socket says 172.21.15.3:60380) 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 -- 172.21.15.3:0/3813280660 learned_addr learned my addr 172.21.15.3:0/3813280660 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877ffff640 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934808a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 -- 172.21.15.3:0/3813280660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 msgr2=0x555934806150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934806150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 -- 172.21.15.3:0/3813280660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 msgr2=0x555934808a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934808a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 -- 172.21.15.3:0/3813280660 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55593470a210 con 0x5559347fd190 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877ffff640 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934808a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877effd640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f87680a2260 0x7f87680a4720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8780101b80 tx=0x7f877400bac0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.442+0000 7f877f7fe640 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x5559347fd570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f87700115c0 tx=0x7f8770011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67671187a1c519fa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8765ffb640 1 -- 172.21.15.3:0/3813280660 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877000f640 con 0x5559347fd190 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8765ffb640 1 -- 172.21.15.3:0/3813280660 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f877000fc80 con 0x5559347fd190 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8765ffb640 1 -- 172.21.15.3:0/3813280660 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770019d60 con 0x5559347fd190 2024-02-09T23:00:20.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 -- 172.21.15.3:0/3813280660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 msgr2=0x5559347fd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x5559347fd570 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f87700115c0 tx=0x7f8770011a80 comp rx=0 tx=0).stop 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 -- 172.21.15.3:0/3813280660 shutdown_connections 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934808a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934806150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 --2- 172.21.15.3:0/3813280660 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x5559347fd570 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 -- 172.21.15.3:0/3813280660 >> 172.21.15.3:0/3813280660 conn(0x5559347f78c0 msgr2=0x5559347f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 -- 172.21.15.3:0/3813280660 shutdown_connections 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.443+0000 7f8788a48740 1 -- 172.21.15.3:0/3813280660 wait complete. 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 -- start start 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 Processor -- start 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fd190 0x5559348035c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x555934803b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934804040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555934708e60 con 0x555934806690 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559344374b0 con 0x5559347fdab0 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559346e9670 con 0x5559347fd190 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f8788a48740 10 client.0 _ll_register_callbacks cb 0x555934701190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f877ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934804040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fd190 0x5559348035c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f877ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934804040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35762/0 (socket says 172.21.15.3:35762) 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f877ffff640 1 -- 172.21.15.3:0/1400427966 learned_addr learned my addr 172.21.15.3:0/1400427966 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.444+0000 7f877effd640 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x555934803b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f877ffff640 1 -- 172.21.15.3:0/1400427966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fd190 msgr2=0x5559348035c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f877ffff640 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fd190 0x5559348035c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f877ffff640 1 -- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 msgr2=0x555934803b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f877ffff640 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x555934803b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f877ffff640 1 -- 172.21.15.3:0/1400427966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8770009d40 con 0x555934806690 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f877ffff640 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934804040 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f877804e000 tx=0x7f87780bcd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8bd3650397faf75 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f875effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f875effd640 1 -- 172.21.15.3:0/1400427966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780c5af0 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f875effd640 1 -- 172.21.15.3:0/1400427966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f87780b99c0 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555934463860 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f875effd640 1 -- 172.21.15.3:0/1400427966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87780b9b80 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 10 client.18870 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x555934762390 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f875effd640 1 -- 172.21.15.3:0/1400427966 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555934463860 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f875effd640 1 -- 172.21.15.3:0/1400427966 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8778050020 con 0x555934806690 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 10 client.18870 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 10 client.18870 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 20 client.18870 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559348163a0 0x5559348187c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 4 client.18870 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.445+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559344b4410 con 0x5559348163a0 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.446+0000 7f877f7fe640 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559348163a0 0x5559348187c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.446+0000 7f877f7fe640 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559348163a0 0x5559348187c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5554d30f12c208a server_cookie=aa147bd1a54c030 in_seq=0 out_seq=0 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.446+0000 7f875effd640 10 client.18870 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.446+0000 7f875effd640 1 -- 172.21.15.3:0/1400427966 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5559344b4410 con 0x5559348163a0 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.446+0000 7f875effd640 10 client.18870 handle_command_reply: tid=0 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.446+0000 7f8788a48740 1 client.18870 shutdown 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 msgr2=0x555934804040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934804040 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f877804e000 tx=0x7f87780bcd70 comp rx=0 tx=0).stop 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 shutdown_connections 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5559348163a0 0x5559348187c0 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x555934803b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x555934804040 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f875effd640 0 client.18870 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 --2- 172.21.15.3:0/1400427966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fd190 0x5559348035c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 >> 172.21.15.3:0/1400427966 conn(0x5559347f78c0 msgr2=0x5559347f9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 shutdown_connections 2024-02-09T23:00:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 1 -- 172.21.15.3:0/1400427966 wait complete. 2024-02-09T23:00:20.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.447+0000 7f8788a48740 20 client.18870 trim_cache size 0 max 16384 2024-02-09T23:00:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 Processor -- start 2024-02-09T23:00:20.470 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 -- start start 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 0x555934805cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x5559348025a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555934806690 0x555934800840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559347fcdf0 con 0x5559347fd190 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559347f30a0 con 0x5559347fdab0 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.466+0000 7f8788a48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559347ff1e0 con 0x555934806690 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 0x555934805cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877effd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x5559348025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 0x555934805cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35770/0 (socket says 172.21.15.3:35770) 2024-02-09T23:00:20.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 -- 172.21.15.3:0/4137627845 learned_addr learned my addr 172.21.15.3:0/4137627845 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 -- 172.21.15.3:0/4137627845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555934806690 msgr2=0x555934800840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555934806690 0x555934800840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 -- 172.21.15.3:0/4137627845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 msgr2=0x5559348025a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x5559348025a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877f7fe640 1 -- 172.21.15.3:0/4137627845 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559347f2b30 con 0x5559347fd190 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.467+0000 7f877effd640 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x5559348025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.468+0000 7f877f7fe640 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 0x555934805cf0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f87700115c0 tx=0x7f8770019820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e2ac6d31c4f9fa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.468+0000 7f87657fa640 1 -- 172.21.15.3:0/4137627845 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770009be0 con 0x5559347fd190 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.468+0000 7f87657fa640 1 -- 172.21.15.3:0/4137627845 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87700195b0 con 0x5559347fd190 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.468+0000 7f87657fa640 1 -- 172.21.15.3:0/4137627845 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8770005040 con 0x5559347fd190 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 -- 172.21.15.3:0/4137627845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 msgr2=0x555934805cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 0x555934805cf0 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f87700115c0 tx=0x7f8770019820 comp rx=0 tx=0).stop 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 -- 172.21.15.3:0/4137627845 shutdown_connections 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fdab0 0x5559348025a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555934806690 0x555934800840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 --2- 172.21.15.3:0/4137627845 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5559347fd190 0x555934805cf0 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 -- 172.21.15.3:0/4137627845 >> 172.21.15.3:0/4137627845 conn(0x5559347f78c0 msgr2=0x555934801e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 -- 172.21.15.3:0/4137627845 shutdown_connections 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 -- 172.21.15.3:0/4137627845 wait complete. 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.469+0000 7f8788a48740 1 -- start start 2024-02-09T23:00:20.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 Processor -- start 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x555934867000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934867540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x55593480caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55593480d1c0 con 0x555934806690 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55593480d330 con 0x5559347fd190 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55593480d4f0 con 0x5559347fdab0 2024-02-09T23:00:20.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f8788a48740 10 client.0 _ll_register_callbacks cb 0x7f8770009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f877ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x55593480caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f877ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x55593480caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35786/0 (socket says 172.21.15.3:35786) 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f877ffff640 1 -- 172.21.15.3:0/1049916588 learned_addr learned my addr 172.21.15.3:0/1049916588 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f877effd640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934867540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.470+0000 7f877f7fe640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x555934867000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877ffff640 1 -- 172.21.15.3:0/1049916588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 msgr2=0x555934867540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877ffff640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934867540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877ffff640 1 -- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 msgr2=0x555934867000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877ffff640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x555934867000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877ffff640 1 -- 172.21.15.3:0/1049916588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8774012e70 con 0x555934806690 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877effd640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934867540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:20.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f877ffff640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x55593480caf0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f87780bd2e0 tx=0x7f8778009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2084af874c2d76ad server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f875effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f875effd640 1 -- 172.21.15.3:0/1049916588 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87780c5af0 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f875effd640 1 -- 172.21.15.3:0/1049916588 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8778004040 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f877000fc80 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f875effd640 1 -- 172.21.15.3:0/1049916588 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f87780c57a0 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f8788a48740 10 client.18876 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.471+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55593470a210 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f875effd640 1 -- 172.21.15.3:0/1049916588 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87780c9020 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f875effd640 1 -- 172.21.15.3:0/1049916588 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8778050070 con 0x555934806690 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f8788a48740 10 client.18876 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f8788a48740 10 client.18876 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f8788a48740 20 client.18876 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f8788a48740 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55593486e9a0 0x555934870d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f8788a48740 4 client.18876 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:00:20.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5559344a07b0 con 0x55593486e9a0 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.472+0000 7f877f7fe640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55593486e9a0 0x555934870d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.473+0000 7f877f7fe640 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55593486e9a0 0x555934870d90 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23477df3c81eaa5f server_cookie=bf67d85977bd27e5 in_seq=0 out_seq=0 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.473+0000 7f875effd640 10 client.18876 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f875effd640 1 -- 172.21.15.3:0/1049916588 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5559344a07b0 con 0x55593486e9a0 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f875effd640 10 client.18876 handle_command_reply: tid=0 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 client.18876 shutdown 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 msgr2=0x55593480caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x55593480caf0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f87780bd2e0 tx=0x7f8778009c20 comp rx=0 tx=0).stop 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 shutdown_connections 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55593486e9a0 0x555934870d90 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555934806690 0x55593480caf0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5559347fdab0 0x555934867540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f875effd640 0 client.18876 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 --2- 172.21.15.3:0/1049916588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5559347fd190 0x555934867000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 >> 172.21.15.3:0/1049916588 conn(0x5559347f78c0 msgr2=0x555934803560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 shutdown_connections 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 1 -- 172.21.15.3:0/1049916588 wait complete. 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.474+0000 7f8788a48740 20 client.18876 trim_cache size 0 max 16384 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "a915e17f-9b3f-4121-aadf-dd7b055cf54f", 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:00:20.483 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f87680a2260 msgr2=0x7f87680a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f87680a2260 0x7f87680a4720 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8780101b80 tx=0x7f877400bac0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 msgr2=0x7f87801a59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 0x7f87801a59e0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f8778004af0 tx=0x7f8778034cc0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 shutdown_connections 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f87680a2260 0x7f87680a4720 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f87801064e0 0x7f8780100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8780100b70 0x7f87801a59e0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 --2- 172.21.15.3:0/1047113739 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f87800687f0 0x7f87801000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 >> 172.21.15.3:0/1047113739 conn(0x7f8780077230 msgr2=0x7f87800fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 shutdown_connections 2024-02-09T23:00:20.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.475+0000 7f8785e76640 1 -- 172.21.15.3:0/1047113739 wait complete. 2024-02-09T23:00:20.490 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:00:20.490 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:00:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 -- 172.21.15.3:0/206435037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c106510 msgr2=0x7f1b4c10e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 --2- 172.21.15.3:0/206435037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c106510 0x7f1b4c10e0e0 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f1b44009cd0 tx=0x7f1b4402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 -- 172.21.15.3:0/206435037 shutdown_connections 2024-02-09T23:00:20.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 --2- 172.21.15.3:0/206435037 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c106510 0x7f1b4c10e0e0 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 --2- 172.21.15.3:0/206435037 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b4c101df0 0x7f1b4c105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 --2- 172.21.15.3:0/206435037 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b4c101440 0x7f1b4c101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 -- 172.21.15.3:0/206435037 >> 172.21.15.3:0/206435037 conn(0x7f1b4c0fccc0 msgr2=0x7f1b4c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 -- 172.21.15.3:0/206435037 shutdown_connections 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.600+0000 7f1b51c35640 1 -- 172.21.15.3:0/206435037 wait complete. 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 Processor -- start 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 -- start start 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 0x7f1b4c19d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b4c106510 0x7f1b4c19db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b4c1976a0 0x7f1b4c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c198170 con 0x7f1b4c101440 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c1982e0 con 0x7f1b4c1976a0 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b51c35640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b4c1984a0 con 0x7f1b4c106510 2024-02-09T23:00:20.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 0x7f1b4c19d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 0x7f1b4c19d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35798/0 (socket says 172.21.15.3:35798) 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.601+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/2434211299 learned_addr learned my addr 172.21.15.3:0/2434211299 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4affd640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b4c106510 0x7f1b4c19db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b4c1976a0 0x7f1b4c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/2434211299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b4c106510 msgr2=0x7f1b4c19db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b4c106510 0x7f1b4c19db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/2434211299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b4c1976a0 msgr2=0x7f1b4c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b4c1976a0 0x7f1b4c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/2434211299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b44009900 con 0x7f1b4c101440 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 0x7f1b4c19d5e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9f0 tx=0x7f1b3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b48ff9640 1 -- 172.21.15.3:0/2434211299 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c009700 con 0x7f1b4c101440 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b48ff9640 1 -- 172.21.15.3:0/2434211299 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b3c00cd80 con 0x7f1b4c101440 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4c1a4440 con 0x7f1b4c101440 2024-02-09T23:00:20.608 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b48ff9640 1 -- 172.21.15.3:0/2434211299 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c018920 con 0x7f1b4c101440 2024-02-09T23:00:20.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.602+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c1a4920 con 0x7f1b4c101440 2024-02-09T23:00:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b48ff9640 1 -- 172.21.15.3:0/2434211299 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1b3c018a80 con 0x7f1b4c101440 2024-02-09T23:00:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 Processor -- start 2024-02-09T23:00:20.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 -- start start 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 0x564bbb1d7d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e08e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e3300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb192d00 con 0x564bbb1d8330 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb1472f0 con 0x564bbb1e0eb0 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.605+0000 7f1b54809740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb144820 con 0x564bbb1d7980 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 0x564bbb1d7d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e08e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 0x564bbb1d7d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54210/0 (socket says 172.21.15.3:54210) 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 -- 172.21.15.3:0/822942876 learned_addr learned my addr 172.21.15.3:0/822942876 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4affd640 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.612 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b48ff9640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b340a2230 0x7f1b340a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 -- 172.21.15.3:0/822942876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 msgr2=0x564bbb1e3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 -- 172.21.15.3:0/822942876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 msgr2=0x564bbb1e08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e08e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 -- 172.21.15.3:0/822942876 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bbb101ff0 con 0x564bbb1d7980 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b48ff9640 1 -- 172.21.15.3:0/2434211299 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1b3c014070 con 0x7f1b4c101440 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b340a2230 0x7f1b340a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 0x564bbb1d7d60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b4402c700 tx=0x7f1b44032d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1906bc11ae6a24c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.606+0000 7f1b31ffb640 1 -- 172.21.15.3:0/822942876 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44034640 con 0x564bbb1d7980 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b31ffb640 1 -- 172.21.15.3:0/822942876 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b44003970 con 0x564bbb1d7980 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b31ffb640 1 -- 172.21.15.3:0/822942876 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b44004340 con 0x564bbb1d7980 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 -- 172.21.15.3:0/822942876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 msgr2=0x564bbb1d7d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b340a2230 0x7f1b340a46f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b4404b000 tx=0x7f1b4404e000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 0x564bbb1d7d60 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f1b4402c700 tx=0x7f1b44032d70 comp rx=0 tx=0).stop 2024-02-09T23:00:20.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 -- 172.21.15.3:0/822942876 shutdown_connections 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e08e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 --2- 172.21.15.3:0/822942876 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d7980 0x564bbb1d7d60 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 -- 172.21.15.3:0/822942876 >> 172.21.15.3:0/822942876 conn(0x564bbb1d2050 msgr2=0x564bbb1d2430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 -- 172.21.15.3:0/822942876 shutdown_connections 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.607+0000 7f1b54809740 1 -- 172.21.15.3:0/822942876 wait complete. 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 -- start start 2024-02-09T23:00:20.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 Processor -- start 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1ddda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1de820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb140000 con 0x564bbb1d7980 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbae13430 con 0x564bbb1e0eb0 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbadcfe00 con 0x564bbb1d8330 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b54809740 10 client.0 _ll_register_callbacks cb 0x564bbb1347f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1ddda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1ddda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35824/0 (socket says 172.21.15.3:35824) 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/836687047 learned_addr learned my addr 172.21.15.3:0/836687047 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4affd640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.618 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1de820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/836687047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 msgr2=0x564bbb1de2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 msgr2=0x564bbb1de820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.608+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1de820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/836687047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b4c10e8f0 con 0x564bbb1d7980 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1ddda0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9c0 tx=0x7f1b3c0d1950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=170811f2c984e285 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b2affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b2affd640 1 -- 172.21.15.3:0/836687047 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c09d070 con 0x564bbb1d7980 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b2affd640 1 -- 172.21.15.3:0/836687047 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1b3c09aeb0 con 0x564bbb1d7980 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564bbae3f7e0 con 0x564bbb1d7980 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b2affd640 1 -- 172.21.15.3:0/836687047 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b3c09a9f0 con 0x564bbb1d7980 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 10 client.18888 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:20.619 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x564bbb13e680 con 0x564bbb1d7980 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b2affd640 1 -- 172.21.15.3:0/836687047 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564bbae3f7e0 con 0x564bbb1d7980 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b2affd640 1 -- 172.21.15.3:0/836687047 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1b3c0cd070 con 0x564bbb1d7980 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 10 client.18888 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 10 client.18888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 20 client.18888 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb1f13d0 0x564bbb1f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 4 client.18888 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.609+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564bbae7c720 con 0x564bbb1f13d0 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.610+0000 7f1b4affd640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb1f13d0 0x564bbb1f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.610+0000 7f1b4affd640 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb1f13d0 0x564bbb1f37e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2690c4893a6b9575 server_cookie=fdc44774cb542f7 in_seq=0 out_seq=0 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.610+0000 7f1b2affd640 10 client.18888 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b2affd640 1 -- 172.21.15.3:0/836687047 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564bbae7c720 con 0x564bbb1f13d0 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b2affd640 10 client.18888 handle_command_reply: tid=0 2024-02-09T23:00:20.620 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 client.18888 shutdown 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 msgr2=0x564bbb1ddda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1ddda0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9c0 tx=0x7f1b3c0d1950 comp rx=0 tx=0).stop 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 shutdown_connections 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb1f13d0 0x564bbb1f37e0 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1de820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 --2- 172.21.15.3:0/836687047 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1ddda0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 >> 172.21.15.3:0/836687047 conn(0x564bbb1d2050 msgr2=0x564bbb1d3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b2affd640 0 client.18888 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 shutdown_connections 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 1 -- 172.21.15.3:0/836687047 wait complete. 2024-02-09T23:00:20.621 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.611+0000 7f1b54809740 20 client.18888 trim_cache size 0 max 16384 2024-02-09T23:00:20.627 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:20 smithi089 ceph-mon[36665]: pgmap v283: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T23:00:20.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.626+0000 7f1b54809740 1 Processor -- start 2024-02-09T23:00:20.631 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.626+0000 7f1b54809740 1 -- start start 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b54809740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1e04d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b54809740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b54809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1d9060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b54809740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb1cd780 con 0x564bbb1d7980 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b54809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb0c4fd0 con 0x564bbb1e0eb0 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b54809740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb1deb80 con 0x564bbb1d8330 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1e04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1e04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35830/0 (socket says 172.21.15.3:35830) 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/22407006 learned_addr learned my addr 172.21.15.3:0/22407006 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4affd640 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1d9060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.632 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/22407006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 msgr2=0x564bbb1de640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/22407006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 msgr2=0x564bbb1d9060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1d9060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 -- 172.21.15.3:0/22407006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564bbb0dab30 con 0x564bbb1d7980 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1d9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1e04d0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c09ae40 tx=0x7f1b3c0b3d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ba38047e81e878 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b317fa640 1 -- 172.21.15.3:0/22407006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c09d070 con 0x564bbb1d7980 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b317fa640 1 -- 172.21.15.3:0/22407006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b3c09a810 con 0x564bbb1d7980 2024-02-09T23:00:20.633 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.627+0000 7f1b317fa640 1 -- 172.21.15.3:0/22407006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c09ef50 con 0x564bbb1d7980 2024-02-09T23:00:20.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:20 smithi003 ceph-mon[32678]: pgmap v283: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T23:00:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 -- 172.21.15.3:0/22407006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 msgr2=0x564bbb1e04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1e04d0 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c09ae40 tx=0x7f1b3c0b3d60 comp rx=0 tx=0).stop 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 -- 172.21.15.3:0/22407006 shutdown_connections 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1e0eb0 0x564bbb1d9060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1d8330 0x564bbb1de640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 --2- 172.21.15.3:0/22407006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d7980 0x564bbb1e04d0 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 -- 172.21.15.3:0/22407006 >> 172.21.15.3:0/22407006 conn(0x564bbb1d2050 msgr2=0x564bbb1f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 -- 172.21.15.3:0/22407006 shutdown_connections 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.628+0000 7f1b54809740 1 -- 172.21.15.3:0/22407006 wait complete. 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 -- start start 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 Processor -- start 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1d7980 0x564bbb1e6830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e72b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb1e8780 con 0x564bbb1d8330 2024-02-09T23:00:20.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb2478e0 con 0x564bbb1d7980 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bbb247a20 con 0x564bbb1e0eb0 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b54809740 10 client.0 _ll_register_callbacks cb 0x7f1b3c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e6d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35832/0 (socket says 172.21.15.3:35832) 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 -- 172.21.15.3:0/327557114 learned_addr learned my addr 172.21.15.3:0/327557114 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1d7980 0x564bbb1e6830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4bfff640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 -- 172.21.15.3:0/327557114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1e0eb0 msgr2=0x564bbb1e72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 -- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1d7980 msgr2=0x564bbb1e6830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1d7980 0x564bbb1e6830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.629+0000 7f1b4affd640 1 -- 172.21.15.3:0/327557114 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bbb13e680 con 0x564bbb1d8330 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b4affd640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e6d70 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005e00 tx=0x7f1b4001f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1294f37d97d511a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:20.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b2affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b2affd640 1 -- 172.21.15.3:0/327557114 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b40017040 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b2affd640 1 -- 172.21.15.3:0/327557114 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1b40010940 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1b44003970 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b2affd640 1 -- 172.21.15.3:0/327557114 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b40004520 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 10 client.18894 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x564bbb100850 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b2affd640 1 -- 172.21.15.3:0/327557114 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b44003970 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b2affd640 1 -- 172.21.15.3:0/327557114 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f1b40028440 con 0x564bbb1d8330 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 10 client.18894 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 10 client.18894 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 20 client.18894 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb2492c0 0x564bbb24b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 4 client.18894 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:00:20.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.630+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564bbaecb890 con 0x564bbb2492c0 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb2492c0 0x564bbb24b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b2affd640 10 client.18894 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b4b7fe640 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb2492c0 0x564bbb24b6e0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45c8f14a7f902d37 server_cookie=6d7a9f636503b5ce in_seq=0 out_seq=0 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b2affd640 1 -- 172.21.15.3:0/327557114 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x564bbaecb890 con 0x564bbb2492c0 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b2affd640 10 client.18894 handle_command_reply: tid=0 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 client.18894 shutdown 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 msgr2=0x564bbb1e6d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e6d70 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7f1b40005e00 tx=0x7f1b4001f890 comp rx=0 tx=0).stop 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 shutdown_connections 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564bbb2492c0 0x564bbb24b6e0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564bbb1e0eb0 0x564bbb1e72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564bbb1d8330 0x564bbb1e6d70 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 --2- 172.21.15.3:0/327557114 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564bbb1d7980 0x564bbb1e6830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b2affd640 0 client.18894 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 >> 172.21.15.3:0/327557114 conn(0x564bbb1d2050 msgr2=0x564bbb1d9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 shutdown_connections 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.631+0000 7f1b54809740 1 -- 172.21.15.3:0/327557114 wait complete. 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b54809740 20 client.18894 trim_cache size 0 max 16384 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout: "a915e17f-9b3f-4121-aadf-dd7b055cf54f": { 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "a915e17f-9b3f-4121-aadf-dd7b055cf54f", 2024-02-09T23:00:20.642 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:00:20.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:00:20.643 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:00:20.643 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:00:20.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b340a2230 msgr2=0x7f1b340a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b340a2230 0x7f1b340a46f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f1b4404b000 tx=0x7f1b4404e000 comp rx=0 tx=0).stop 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 msgr2=0x7f1b4c19d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 0x7f1b4c19d5e0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c00e9f0 tx=0x7f1b3c00eeb0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 shutdown_connections 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1b340a2230 0x7f1b340a46f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1b4c1976a0 0x7f1b4c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1b4c106510 0x7f1b4c19db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 --2- 172.21.15.3:0/2434211299 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1b4c101440 0x7f1b4c19d5e0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 >> 172.21.15.3:0/2434211299 conn(0x7f1b4c0fccc0 msgr2=0x7f1b4c0ff040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 shutdown_connections 2024-02-09T23:00:20.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:20.632+0000 7f1b51c35640 1 -- 172.21.15.3:0/2434211299 wait complete. 2024-02-09T23:00:20.647 INFO:tasks.cephfs.filesystem:scrub status for tag:a915e17f-9b3f-4121-aadf-dd7b055cf54f - {'path': '/', 'tag': 'a915e17f-9b3f-4121-aadf-dd7b055cf54f', 'options': 'recursive,force'} 2024-02-09T23:00:20.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 269054 2.74 MB/sec execute 374 sec latency 1886.438 ms 2024-02-09T23:00:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:20 smithi064 ceph-mon[36638]: pgmap v283: 97 pgs: 97 active+clean; 105 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 161 op/s 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:21 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 270592 2.76 MB/sec execute 375 sec latency 11.493 ms 2024-02-09T23:00:21.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:21 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:00:21.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:21.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:21.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:21 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:22.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:22 smithi003 ceph-mon[32678]: pgmap v284: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-02-09T23:00:22.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 271294 2.76 MB/sec execute 376 sec latency 477.337 ms 2024-02-09T23:00:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:22 smithi064 ceph-mon[36638]: pgmap v284: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-02-09T23:00:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:22 smithi089 ceph-mon[36665]: pgmap v284: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-02-09T23:00:23.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:23 smithi003 ceph-mon[32678]: pgmap v285: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-02-09T23:00:23.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 271294 2.75 MB/sec execute 377 sec latency 1477.446 ms 2024-02-09T23:00:23.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:23 smithi064 ceph-mon[36638]: pgmap v285: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-02-09T23:00:23.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:23 smithi089 ceph-mon[36665]: pgmap v285: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-02-09T23:00:24.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 271294 2.74 MB/sec execute 378 sec latency 2477.558 ms 2024-02-09T23:00:25.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 271294 2.74 MB/sec execute 379 sec latency 3477.621 ms 2024-02-09T23:00:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:26 smithi003 ceph-mon[32678]: pgmap v286: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-02-09T23:00:26.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 271294 2.73 MB/sec execute 380 sec latency 4477.739 ms 2024-02-09T23:00:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:26 smithi064 ceph-mon[36638]: pgmap v286: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-02-09T23:00:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:26 smithi089 ceph-mon[36665]: pgmap v286: 97 pgs: 97 active+clean; 108 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.3 MiB/s wr, 155 op/s 2024-02-09T23:00:27.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:00:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:00:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:00:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:27 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:00:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:27 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:00:27.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 273411 2.76 MB/sec execute 381 sec latency 4519.291 ms 2024-02-09T23:00:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:00:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:00:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:00:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:27 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:00:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:27 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:00:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:00:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:00:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:00:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:27 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:00:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:27 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:00:28.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:28 smithi064 ceph-mon[36638]: pgmap v287: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-02-09T23:00:28.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:28 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:00:28.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:28 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:00:28.586 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:28 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:00:28.587 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:28 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:00:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:28 smithi003 ceph-mon[32678]: pgmap v287: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-02-09T23:00:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:28 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:00:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:28 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:00:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:28 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:00:28.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:28 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:00:28.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 274843 2.77 MB/sec execute 382 sec latency 15.346 ms 2024-02-09T23:00:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:28 smithi089 ceph-mon[36665]: pgmap v287: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 217 op/s 2024-02-09T23:00:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:28 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:00:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:28 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:00:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:28 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:00:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:28 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.221+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/2758445853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e81003c0 msgr2=0x7fa1e81007a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.221+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/2758445853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e81003c0 0x7fa1e81007a0 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8009930 tx=0x7fa1d802bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/2758445853 shutdown_connections 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/2758445853 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1e8105440 0x7fa1e810d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/2758445853 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1e8100d70 0x7fa1e8104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/2758445853 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e81003c0 0x7fa1e81007a0 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.228 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/2758445853 >> 172.21.15.3:0/2758445853 conn(0x7fa1e8077340 msgr2=0x7fa1e8077750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/2758445853 shutdown_connections 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/2758445853 wait complete. 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 Processor -- start 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.222+0000 7fa1ee6c0640 1 -- start start 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1e81003c0 0x7fa1e81986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 0x7fa1e8198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1e81991b0 0x7fa1e81a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81a3e00 con 0x7fa1e8105440 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81a3f70 con 0x7fa1e81003c0 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e81a4130 con 0x7fa1e81991b0 2024-02-09T23:00:29.229 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 0x7fa1e8198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 0x7fa1e8198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52930/0 (socket says 172.21.15.3:52930) 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 -- 172.21.15.3:0/385555848 learned_addr learned my addr 172.21.15.3:0/385555848 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1e81003c0 0x7fa1e81986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1e81991b0 0x7fa1e81a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 -- 172.21.15.3:0/385555848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1e81991b0 msgr2=0x7fa1e81a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1e81991b0 0x7fa1e81a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 -- 172.21.15.3:0/385555848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1e81003c0 msgr2=0x7fa1e81986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1e81003c0 0x7fa1e81986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 -- 172.21.15.3:0/385555848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d8009560 con 0x7fa1e8105440 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 0x7fa1e8198c20 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00e9c0 tx=0x7fa1dc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e57fa640 1 -- 172.21.15.3:0/385555848 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc009700 con 0x7fa1e8105440 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e57fa640 1 -- 172.21.15.3:0/385555848 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1dc00cd80 con 0x7fa1e8105440 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e81a4410 con 0x7fa1e8105440 2024-02-09T23:00:29.230 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.223+0000 7fa1e57fa640 1 -- 172.21.15.3:0/385555848 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc010430 con 0x7fa1e8105440 2024-02-09T23:00:29.231 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.224+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e81a4970 con 0x7fa1e8105440 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1e57fa640 1 -- 172.21.15.3:0/385555848 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa1dc005000 con 0x7fa1e8105440 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 Processor -- start 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 -- start start 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 0x55c626f5c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1e57fa640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d00a22a0 0x7fa1d00a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c9d0 0x55c626f65260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f67b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626ecaa00 con 0x55c626f5c0b0 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626ec7f30 con 0x55c626f657a0 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.226+0000 7fa1f136a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626ec3710 con 0x55c626f5c9d0 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c9d0 0x55c626f65260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 0x55c626f5c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d00a22a0 0x7fa1d00a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 0x55c626f5c490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52934/0 (socket says 172.21.15.3:52934) 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c9d0 0x55c626f65260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:50100/0 (socket says 172.21.15.3:50100) 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/4203834250 learned_addr learned my addr 172.21.15.3:0/4203834250 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1e57fa640 1 -- 172.21.15.3:0/385555848 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa1dc014070 con 0x7fa1e8105440 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/4203834250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c9d0 msgr2=0x55c626f65260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c9d0 0x55c626f65260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/4203834250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 msgr2=0x55c626f67b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f67b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/4203834250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c626e883b0 con 0x55c626f5c0b0 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 0x55c626f5c490 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009ef0 tx=0x7fa1e0012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e01bdd2b0a54f6e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d00a22a0 0x7fa1d00a4760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8005ab0 tx=0x7fa1d8005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ce7fc640 1 -- 172.21.15.3:0/4203834250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0004d20 con 0x55c626f5c0b0 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ce7fc640 1 -- 172.21.15.3:0/4203834250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e0004e80 con 0x55c626f5c0b0 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.227+0000 7fa1ce7fc640 1 -- 172.21.15.3:0/4203834250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0013880 con 0x55c626f5c0b0 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 -- 172.21.15.3:0/4203834250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 msgr2=0x55c626f5c490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 0x55c626f5c490 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0009ef0 tx=0x7fa1e0012760 comp rx=0 tx=0).stop 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 -- 172.21.15.3:0/4203834250 shutdown_connections 2024-02-09T23:00:29.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f67b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c9d0 0x55c626f65260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4203834250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c0b0 0x55c626f5c490 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 -- 172.21.15.3:0/4203834250 >> 172.21.15.3:0/4203834250 conn(0x55c626f569d0 msgr2=0x55c626f56db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 -- 172.21.15.3:0/4203834250 shutdown_connections 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 -- 172.21.15.3:0/4203834250 wait complete. 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.228+0000 7fa1f136a740 1 -- start start 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 Processor -- start 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f62730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f62c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f631b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626e68b90 con 0x55c626f5c9d0 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626b97250 con 0x55c626f657a0 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626e49200 con 0x55c626f5c0b0 2024-02-09T23:00:29.240 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1f136a740 10 client.0 _ll_register_callbacks cb 0x55c626e7f5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f62c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f62c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52938/0 (socket says 172.21.15.3:52938) 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 -- 172.21.15.3:0/4039252002 learned_addr learned my addr 172.21.15.3:0/4039252002 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f62730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f631b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 -- 172.21.15.3:0/4039252002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 msgr2=0x55c626f62730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f62730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 -- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 msgr2=0x55c626f631b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f631b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 -- 172.21.15.3:0/4039252002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e0009bd0 con 0x55c626f5c9d0 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:29.241 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f62c70 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc09abb0 tx=0x7fa1dc09aab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5945ed8516c3199d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.229+0000 7fa1c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1c77fe640 1 -- 172.21.15.3:0/4039252002 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0a38a0 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1c77fe640 1 -- 172.21.15.3:0/4039252002 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa1dc066410 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c626bc3600 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1c77fe640 1 -- 172.21.15.3:0/4039252002 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1dc09aeb0 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 10 client.18906 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55c626ec1d90 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1c77fe640 1 -- 172.21.15.3:0/4039252002 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c626bc3600 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1c77fe640 1 -- 172.21.15.3:0/4039252002 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa1dc01f060 con 0x55c626f5c9d0 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 10 client.18906 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 10 client.18906 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 20 client.18906 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626f758c0 0x55c626f77ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.242 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 4 client.18906 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c626c141b0 con 0x55c626f758c0 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.230+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626f758c0 0x55c626f77ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.231+0000 7fa1c77fe640 10 client.18906 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.231+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626f758c0 0x55c626f77ce0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba3fde8005549d3c server_cookie=34acb6281b23d8e4 in_seq=0 out_seq=0 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.231+0000 7fa1c77fe640 1 -- 172.21.15.3:0/4039252002 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c626c141b0 con 0x55c626f758c0 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.231+0000 7fa1c77fe640 10 client.18906 handle_command_reply: tid=0 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.231+0000 7fa1f136a740 1 client.18906 shutdown 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 msgr2=0x55c626f62c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f62c70 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc09abb0 tx=0x7fa1dc09aab0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 shutdown_connections 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626f758c0 0x55c626f77ce0 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f657a0 0x55c626f631b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.243 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f62c70 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 --2- 172.21.15.3:0/4039252002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f62730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1c77fe640 0 client.18906 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 >> 172.21.15.3:0/4039252002 conn(0x55c626f569d0 msgr2=0x55c626f58720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 shutdown_connections 2024-02-09T23:00:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 1 -- 172.21.15.3:0/4039252002 wait complete. 2024-02-09T23:00:29.244 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.232+0000 7fa1f136a740 20 client.18906 trim_cache size 0 max 16384 2024-02-09T23:00:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 Processor -- start 2024-02-09T23:00:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 -- start start 2024-02-09T23:00:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f64e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c9d0 0x55c626f62fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 0x55c626f5f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626f5b4f0 con 0x55c626f657a0 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626f756c0 con 0x55c626f5c9d0 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1f136a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626f56500 con 0x55c626f5c0b0 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1ecc5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 0x55c626f5f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1ecc5d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 0x55c626f5f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52954/0 (socket says 172.21.15.3:52954) 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/185723962 learned_addr learned my addr 172.21.15.3:0/185723962 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.245+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f64e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/185723962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 msgr2=0x55c626f64e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f64e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/185723962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c9d0 msgr2=0x55c626f62fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c9d0 0x55c626f62fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1ecc5d640 1 -- 172.21.15.3:0/185723962 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c626e5e820 con 0x55c626f657a0 2024-02-09T23:00:29.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 0x55c626f5f990 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0013010 tx=0x7fa1e0009af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9dd994d5f8841a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1cdffb640 1 -- 172.21.15.3:0/185723962 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0004be0 con 0x55c626f657a0 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1cdffb640 1 -- 172.21.15.3:0/185723962 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e0021040 con 0x55c626f657a0 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1cdffb640 1 -- 172.21.15.3:0/185723962 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e001b7b0 con 0x55c626f657a0 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1f136a740 1 -- 172.21.15.3:0/185723962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 msgr2=0x55c626f5f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1f136a740 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 0x55c626f5f990 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa1e0013010 tx=0x7fa1e0009af0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1f136a740 1 -- 172.21.15.3:0/185723962 shutdown_connections 2024-02-09T23:00:29.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1f136a740 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c9d0 0x55c626f62fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.246+0000 7fa1f136a740 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f657a0 0x55c626f5f990 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 --2- 172.21.15.3:0/185723962 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f5c0b0 0x55c626f64e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- 172.21.15.3:0/185723962 >> 172.21.15.3:0/185723962 conn(0x55c626f569d0 msgr2=0x55c626f61000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- 172.21.15.3:0/185723962 shutdown_connections 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- 172.21.15.3:0/185723962 wait complete. 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- start start 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 Processor -- start 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c0b0 0x55c626f6b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f6b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f657a0 0x55c626f6bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626fcbf10 con 0x55c626f5c9d0 2024-02-09T23:00:29.256 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626fcc080 con 0x55c626f5c0b0 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c626fcc240 con 0x55c626f657a0 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1f136a740 10 client.0 _ll_register_callbacks cb 0x7fa1dc097200 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f6b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f6b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52964/0 (socket says 172.21.15.3:52964) 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1e77fe640 1 -- 172.21.15.3:0/694665691 learned_addr learned my addr 172.21.15.3:0/694665691 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c0b0 0x55c626f6b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.247+0000 7fa1ecc5d640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f657a0 0x55c626f6bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1e77fe640 1 -- 172.21.15.3:0/694665691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f657a0 msgr2=0x55c626f6bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f657a0 0x55c626f6bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1e77fe640 1 -- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c0b0 msgr2=0x55c626f6b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c0b0 0x55c626f6b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.257 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1e77fe640 1 -- 172.21.15.3:0/694665691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c626e883b0 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1e77fe640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f6b5c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c6a0 tx=0x7fa1dc09ab20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=739c9b480ebc79a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1c77fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1c77fe640 1 -- 172.21.15.3:0/694665691 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc0a38a0 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1c77fe640 1 -- 172.21.15.3:0/694665691 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa1dc09aeb0 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c626ee9ee0 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1c77fe640 1 -- 172.21.15.3:0/694665691 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1dc09f000 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 10 client.18912 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55c626e846f0 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1c77fe640 1 -- 172.21.15.3:0/694665691 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c626ee9ee0 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1c77fe640 1 -- 172.21.15.3:0/694665691 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa1dc066250 con 0x55c626f5c9d0 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 10 client.18912 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 10 client.18912 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:29.258 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 20 client.18912 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626fcd720 0x55c626fcfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 4 client.18912 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:00:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.248+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c626c00550 con 0x55c626fcd720 2024-02-09T23:00:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.249+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626fcd720 0x55c626fcfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.249+0000 7fa1c77fe640 10 client.18912 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:29.259 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.249+0000 7fa1e7fff640 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626fcd720 0x55c626fcfb40 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35423121c44753c2 server_cookie=6fe6136a9472d74f in_seq=0 out_seq=0 2024-02-09T23:00:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1c77fe640 1 -- 172.21.15.3:0/694665691 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c626c00550 con 0x55c626fcd720 2024-02-09T23:00:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1c77fe640 10 client.18912 handle_command_reply: tid=0 2024-02-09T23:00:29.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 client.18912 shutdown 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 msgr2=0x55c626f6b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f6b5c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c6a0 tx=0x7fa1dc09ab20 comp rx=0 tx=0).stop 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 shutdown_connections 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c626fcd720 0x55c626fcfb40 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c626f657a0 0x55c626f6bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c626f5c9d0 0x55c626f6b5c0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1c77fe640 0 client.18912 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 --2- 172.21.15.3:0/694665691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c626f5c0b0 0x55c626f6b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 >> 172.21.15.3:0/694665691 conn(0x55c626f569d0 msgr2=0x55c626f551a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 shutdown_connections 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 1 -- 172.21.15.3:0/694665691 wait complete. 2024-02-09T23:00:29.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.293+0000 7fa1f136a740 20 client.18912 trim_cache size 0 max 16384 2024-02-09T23:00:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:00:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:00:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:00:29.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d00a22a0 msgr2=0x7fa1d00a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d00a22a0 0x7fa1d00a4760 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa1d8005ab0 tx=0x7fa1d8005fd0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 msgr2=0x7fa1e8198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 0x7fa1e8198c20 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00e9c0 tx=0x7fa1dc00ee80 comp rx=0 tx=0).stop 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 shutdown_connections 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d00a22a0 0x7fa1d00a4760 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1e81991b0 0x7fa1e81a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1e8105440 0x7fa1e8198c20 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 --2- 172.21.15.3:0/385555848 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1e81003c0 0x7fa1e81986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:29.300 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 >> 172.21.15.3:0/385555848 conn(0x7fa1e8077340 msgr2=0x7fa1e80fe2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 shutdown_connections 2024-02-09T23:00:29.301 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:29.294+0000 7fa1ee6c0640 1 -- 172.21.15.3:0/385555848 wait complete. 2024-02-09T23:00:29.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 275869 2.77 MB/sec execute 383 sec latency 635.144 ms 2024-02-09T23:00:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:30 smithi003 ceph-mon[32678]: pgmap v288: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-02-09T23:00:30.656 INFO:tasks.workunit.client.0.smithi003.stdout: 1 277448 2.79 MB/sec execute 384 sec latency 12.478 ms 2024-02-09T23:00:30.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:30 smithi064 ceph-mon[36638]: pgmap v288: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-02-09T23:00:30.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:30 smithi089 ceph-mon[36665]: pgmap v288: 97 pgs: 97 active+clean; 110 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.2 MiB/s wr, 128 op/s 2024-02-09T23:00:31.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 278385 2.79 MB/sec execute 385 sec latency 364.451 ms 2024-02-09T23:00:32.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:32 smithi003 ceph-mon[32678]: pgmap v289: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-02-09T23:00:32.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 280450 2.82 MB/sec execute 386 sec latency 393.844 ms 2024-02-09T23:00:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:32 smithi064 ceph-mon[36638]: pgmap v289: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-02-09T23:00:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:32 smithi089 ceph-mon[36665]: pgmap v289: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.7 MiB/s wr, 210 op/s 2024-02-09T23:00:33.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 281772 2.83 MB/sec execute 387 sec latency 14.303 ms 2024-02-09T23:00:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:34 smithi003 ceph-mon[32678]: pgmap v290: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-02-09T23:00:34.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 281934 2.82 MB/sec execute 388 sec latency 920.032 ms 2024-02-09T23:00:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:34 smithi064 ceph-mon[36638]: pgmap v290: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-02-09T23:00:34.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:34 smithi089 ceph-mon[36665]: pgmap v290: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-02-09T23:00:35.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 283428 2.84 MB/sec execute 389 sec latency 1360.737 ms 2024-02-09T23:00:36.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:36 smithi003 ceph-mon[32678]: pgmap v291: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-02-09T23:00:36.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 284888 2.85 MB/sec execute 390 sec latency 12.394 ms 2024-02-09T23:00:36.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:36 smithi064 ceph-mon[36638]: pgmap v291: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-02-09T23:00:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:36 smithi089 ceph-mon[36665]: pgmap v291: 97 pgs: 97 active+clean; 109 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.1 MiB/s wr, 143 op/s 2024-02-09T23:00:37.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 285476 2.85 MB/sec execute 391 sec latency 512.971 ms 2024-02-09T23:00:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:38 smithi003 ceph-mon[32678]: pgmap v292: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 219 op/s 2024-02-09T23:00:38.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 287327 2.87 MB/sec execute 392 sec latency 637.777 ms 2024-02-09T23:00:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:38 smithi064 ceph-mon[36638]: pgmap v292: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 219 op/s 2024-02-09T23:00:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:38 smithi089 ceph-mon[36665]: pgmap v292: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 219 op/s 2024-02-09T23:00:39.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 288475 2.88 MB/sec execute 393 sec latency 19.881 ms 2024-02-09T23:00:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:40 smithi003 ceph-mon[32678]: pgmap v293: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-02-09T23:00:40.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 290502 2.90 MB/sec execute 394 sec latency 164.338 ms 2024-02-09T23:00:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:40 smithi064 ceph-mon[36638]: pgmap v293: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-02-09T23:00:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:40 smithi089 ceph-mon[36665]: pgmap v293: 97 pgs: 97 active+clean; 116 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.3 MiB/s wr, 157 op/s 2024-02-09T23:00:41.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:41 smithi003 ceph-mon[32678]: pgmap v294: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 256 op/s 2024-02-09T23:00:41.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 291849 2.91 MB/sec execute 395 sec latency 10.728 ms 2024-02-09T23:00:41.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:41 smithi064 ceph-mon[36638]: pgmap v294: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 256 op/s 2024-02-09T23:00:41.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:41 smithi089 ceph-mon[36665]: pgmap v294: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 256 op/s 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 -- 172.21.15.3:0/3808559887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c105430 msgr2=0x7f655c10d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3808559887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c105430 0x7f655c10d000 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7f6554009cd0 tx=0x7f655402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 -- 172.21.15.3:0/3808559887 shutdown_connections 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3808559887 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c100d10 0x7f655c104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3808559887 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c105430 0x7f655c10d000 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3808559887 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c100360 0x7f655c100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 -- 172.21.15.3:0/3808559887 >> 172.21.15.3:0/3808559887 conn(0x7f655c077250 msgr2=0x7f655c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:42.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 -- 172.21.15.3:0/3808559887 shutdown_connections 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 -- 172.21.15.3:0/3808559887 wait complete. 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.383+0000 7f6561e8d640 1 Processor -- start 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 -- start start 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 0x7f655c19d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c105430 0x7f655c19dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c197650 0x7f655c197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c198120 con 0x7f655c100360 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c198290 con 0x7f655c105430 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6561e8d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655c198450 con 0x7f655c197650 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 0x7f655c19d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 0x7f655c19d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34052/0 (socket says 172.21.15.3:34052) 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 -- 172.21.15.3:0/3062374365 learned_addr learned my addr 172.21.15.3:0/3062374365 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655affd640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c105430 0x7f655c19dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655bfff640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c197650 0x7f655c197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 -- 172.21.15.3:0/3062374365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c197650 msgr2=0x7f655c197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c197650 0x7f655c197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 -- 172.21.15.3:0/3062374365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c105430 msgr2=0x7f655c19dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c105430 0x7f655c19dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 -- 172.21.15.3:0/3062374365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554009900 con 0x7f655c100360 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655affd640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c105430 0x7f655c19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f655b7fe640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 0x7f655c19d590 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f654c00d770 tx=0x7f654c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.384+0000 7f6558ff9640 1 -- 172.21.15.3:0/3062374365 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c009c30 con 0x7f655c100360 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.385+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f655c1a43f0 con 0x7f655c100360 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.385+0000 7f6558ff9640 1 -- 172.21.15.3:0/3062374365 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f654c00de80 con 0x7f655c100360 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.385+0000 7f6558ff9640 1 -- 172.21.15.3:0/3062374365 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c004e80 con 0x7f655c100360 2024-02-09T23:00:42.390 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.385+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655c1a48d0 con 0x7f655c100360 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6558ff9640 1 -- 172.21.15.3:0/3062374365 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f654c005020 con 0x7f655c100360 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 Processor -- start 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 -- start start 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df40b0 0x562627df4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df49d0 0x562627dfd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 0x562627dffb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627d62a00 con 0x562627dfd7a0 2024-02-09T23:00:42.397 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627d5ff30 con 0x562627df49d0 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6564a5b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627d5b710 con 0x562627df40b0 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f6558ff9640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f65440a2220 0x7f65440a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f655affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 0x562627dffb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f655affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 0x562627dffb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34068/0 (socket says 172.21.15.3:34068) 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f655affd640 1 -- 172.21.15.3:0/1525453418 learned_addr learned my addr 172.21.15.3:0/1525453418 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f655bfff640 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df40b0 0x562627df4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df49d0 0x562627dfd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.387+0000 7f655bfff640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f65440a2220 0x7f65440a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f6558ff9640 1 -- 172.21.15.3:0/3062374365 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f654c0cdba0 con 0x7f655c100360 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655affd640 1 -- 172.21.15.3:0/1525453418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df40b0 msgr2=0x562627df4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655affd640 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df40b0 0x562627df4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655affd640 1 -- 172.21.15.3:0/1525453418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df49d0 msgr2=0x562627dfd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655affd640 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df49d0 0x562627dfd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655affd640 1 -- 172.21.15.3:0/1525453418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562627d203b0 con 0x562627dfd7a0 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655affd640 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 0x562627dffb90 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f655000cbc0 tx=0x7f655000e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114fda105dabcdae server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f6541ffb640 1 -- 172.21.15.3:0/1525453418 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550005220 con 0x562627dfd7a0 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f655bfff640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f65440a2220 0x7f65440a46e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6554006010 tx=0x7f65540095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f6541ffb640 1 -- 172.21.15.3:0/1525453418 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6550004c80 con 0x562627dfd7a0 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.388+0000 7f6541ffb640 1 -- 172.21.15.3:0/1525453418 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550010710 con 0x562627dfd7a0 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 -- 172.21.15.3:0/1525453418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 msgr2=0x562627dffb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 0x562627dffb90 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7f655000cbc0 tx=0x7f655000e720 comp rx=0 tx=0).stop 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 -- 172.21.15.3:0/1525453418 shutdown_connections 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df49d0 0x562627dfd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627dfd7a0 0x562627dffb90 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1525453418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df40b0 0x562627df4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 -- 172.21.15.3:0/1525453418 >> 172.21.15.3:0/1525453418 conn(0x562627dee9d0 msgr2=0x562627deedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 -- 172.21.15.3:0/1525453418 shutdown_connections 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 -- 172.21.15.3:0/1525453418 wait complete. 2024-02-09T23:00:42.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 -- start start 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 Processor -- start 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.389+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfa7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627dfb230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627d00b90 con 0x562627df49d0 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627a2f250 con 0x562627df40b0 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627ce1200 con 0x562627dfd7a0 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 10 client.0 _ll_register_callbacks cb 0x562627d175d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34080/0 (socket says 172.21.15.3:34080) 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 -- 172.21.15.3:0/1911677177 learned_addr learned my addr 172.21.15.3:0/1911677177 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655bfff640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627dfb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfa7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 -- 172.21.15.3:0/1911677177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 msgr2=0x562627dfb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627dfb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 -- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 msgr2=0x562627dfa7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfa7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 -- 172.21.15.3:0/1911677177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f655000c7f0 con 0x562627df49d0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655bfff640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627dfb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f655affd640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfacf0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f655000ec20 tx=0x7f6550005210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2453c4a95e0e4cc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f653affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f653affd640 1 -- 172.21.15.3:0/1911677177 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550020070 con 0x562627df49d0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f653affd640 1 -- 172.21.15.3:0/1911677177 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6550004df0 con 0x562627df49d0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562627d81ee0 con 0x562627df49d0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f653affd640 1 -- 172.21.15.3:0/1911677177 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f655000cec0 con 0x562627df49d0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.390+0000 7f6564a5b740 10 client.18924 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x562627d59d90 con 0x562627df49d0 2024-02-09T23:00:42.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f653affd640 1 -- 172.21.15.3:0/1911677177 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6550031020 con 0x562627df49d0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f653affd640 1 -- 172.21.15.3:0/1911677177 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f655001a5a0 con 0x562627df49d0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 10 client.18924 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 10 client.18924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 20 client.18924 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e0d8e0 0x562627e0fd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 4 client.18924 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562627a5b600 con 0x562627e0d8e0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e0d8e0 0x562627e0fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f653affd640 10 client.18924 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.391+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e0d8e0 0x562627e0fd00 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e41820ee47d70ae6 server_cookie=468bd3ff71e85551 in_seq=0 out_seq=0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f653affd640 1 -- 172.21.15.3:0/1911677177 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562627a5b600 con 0x562627e0d8e0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f653affd640 10 client.18924 handle_command_reply: tid=0 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 client.18924 shutdown 2024-02-09T23:00:42.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 msgr2=0x562627dfacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfacf0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 crypto rx=0x7f655000ec20 tx=0x7f6550005210 comp rx=0 tx=0).stop 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 shutdown_connections 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e0d8e0 0x562627e0fd00 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627dfb230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfacf0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f653affd640 0 client.18924 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1911677177 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfa7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 >> 172.21.15.3:0/1911677177 conn(0x562627dee9d0 msgr2=0x562627df0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 shutdown_connections 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 1 -- 172.21.15.3:0/1911677177 wait complete. 2024-02-09T23:00:42.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.392+0000 7f6564a5b740 20 client.18924 trim_cache size 0 max 16384 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.405+0000 7f6564a5b740 1 Processor -- start 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.405+0000 7f6564a5b740 1 -- start start 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfcee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfb050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627df9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f6564a5b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627df0d60 con 0x562627df49d0 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f6564a5b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627e14f00 con 0x562627df40b0 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f6564a5b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627de9f30 con 0x562627dfd7a0 2024-02-09T23:00:42.413 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfb050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfb050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34092/0 (socket says 172.21.15.3:34092) 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 -- 172.21.15.3:0/3755446221 learned_addr learned my addr 172.21.15.3:0/3755446221 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655bfff640 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627df9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655b7fe640 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfcee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 -- 172.21.15.3:0/3755446221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 msgr2=0x562627df9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627df9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 -- 172.21.15.3:0/3755446221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 msgr2=0x562627dfcee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfcee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 -- 172.21.15.3:0/3755446221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562627cf6820 con 0x562627df49d0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f655affd640 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfb050 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f655001aa00 tx=0x7f655000ef50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bd045ea9dfde24d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f65417fa640 1 -- 172.21.15.3:0/3755446221 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6550020070 con 0x562627df49d0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f65417fa640 1 -- 172.21.15.3:0/3755446221 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6550031070 con 0x562627df49d0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.406+0000 7f65417fa640 1 -- 172.21.15.3:0/3755446221 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655001a410 con 0x562627df49d0 2024-02-09T23:00:42.414 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 -- 172.21.15.3:0/3755446221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 msgr2=0x562627dfb050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfb050 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 crypto rx=0x7f655001aa00 tx=0x7f655000ef50 comp rx=0 tx=0).stop 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 -- 172.21.15.3:0/3755446221 shutdown_connections 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627dfd7a0 0x562627df9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df49d0 0x562627dfb050 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 --2- 172.21.15.3:0/3755446221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627df40b0 0x562627dfcee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 -- 172.21.15.3:0/3755446221 >> 172.21.15.3:0/3755446221 conn(0x562627dee9d0 msgr2=0x562627df5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 -- 172.21.15.3:0/3755446221 shutdown_connections 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 -- 172.21.15.3:0/3755446221 wait complete. 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 -- start start 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.407+0000 7f6564a5b740 1 Processor -- start 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 0x562627e04420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df49d0 0x562627e04960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.415 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627dfd7a0 0x562627e02a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627e03150 con 0x562627df40b0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627e032c0 con 0x562627dfd7a0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562627e03480 con 0x562627df49d0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 10 client.0 _ll_register_callbacks cb 0x7f6550014070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 0x562627e04420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 0x562627e04420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:34104/0 (socket says 172.21.15.3:34104) 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 -- 172.21.15.3:0/1728145207 learned_addr learned my addr 172.21.15.3:0/1728145207 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655affd640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df49d0 0x562627e04960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655bfff640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627dfd7a0 0x562627e02a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 -- 172.21.15.3:0/1728145207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df49d0 msgr2=0x562627e04960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df49d0 0x562627e04960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 -- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627dfd7a0 msgr2=0x562627e02a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:42.416 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627dfd7a0 0x562627e02a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 -- 172.21.15.3:0/1728145207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562627d59d90 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655bfff640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627dfd7a0 0x562627e02a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f655b7fe640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 0x562627e04420 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f654c09a290 tx=0x7f654c065a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188f977014007138 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f653affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f653affd640 1 -- 172.21.15.3:0/1728145207 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c09a590 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f653affd640 1 -- 172.21.15.3:0/1728145207 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f654c09e6e0 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6550004c80 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f653affd640 1 -- 172.21.15.3:0/1728145207 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f654c0a2d30 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 10 client.18930 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.408+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x562627d1c6f0 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f653affd640 1 -- 172.21.15.3:0/1728145207 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6550004c80 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f653affd640 1 -- 172.21.15.3:0/1728145207 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f654c096c00 con 0x562627df40b0 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f6564a5b740 10 client.18930 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:42.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f6564a5b740 10 client.18930 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f6564a5b740 20 client.18930 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e65120 0x562627e67540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f6564a5b740 4 client.18930 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562627aac1b0 con 0x562627e65120 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f655affd640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e65120 0x562627e67540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f655affd640 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e65120 0x562627e67540 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71f8a2b2ad221c3f server_cookie=d158516d3535b7bb in_seq=0 out_seq=0 2024-02-09T23:00:42.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.409+0000 7f653affd640 10 client.18930 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.443+0000 7f653affd640 1 -- 172.21.15.3:0/1728145207 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562627aac1b0 con 0x562627e65120 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.443+0000 7f653affd640 10 client.18930 handle_command_reply: tid=0 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.443+0000 7f6564a5b740 1 client.18930 shutdown 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.443+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 msgr2=0x562627e04420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 0x562627e04420 secure :-1 s=READY pgs=1442 cs=0 l=1 rev1=1 crypto rx=0x7f654c09a290 tx=0x7f654c065a80 comp rx=0 tx=0).stop 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 shutdown_connections 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562627e65120 0x562627e67540 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562627dfd7a0 0x562627e02a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562627df49d0 0x562627e04960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 --2- 172.21.15.3:0/1728145207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562627df40b0 0x562627e04420 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.447 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 >> 172.21.15.3:0/1728145207 conn(0x562627dee9d0 msgr2=0x562627df9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f653affd640 0 client.18930 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 shutdown_connections 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 1 -- 172.21.15.3:0/1728145207 wait complete. 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6564a5b740 20 client.18930 trim_cache size 0 max 16384 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:00:42.448 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:00:42.449 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f65440a2220 msgr2=0x7f65440a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f65440a2220 0x7f65440a46e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f6554006010 tx=0x7f65540095c0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 msgr2=0x7f655c19d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 0x7f655c19d590 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7f654c00d770 tx=0x7f654c00dc30 comp rx=0 tx=0).stop 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 shutdown_connections 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f65440a2220 0x7f65440a46e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f655c105430 0x7f655c19dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.450 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f655c197650 0x7f655c197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 --2- 172.21.15.3:0/3062374365 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f655c100360 0x7f655c19d590 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.444+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 >> 172.21.15.3:0/3062374365 conn(0x7f655c077250 msgr2=0x7f655c0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.445+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 shutdown_connections 2024-02-09T23:00:42.451 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:42.445+0000 7f6561e8d640 1 -- 172.21.15.3:0/3062374365 wait complete. 2024-02-09T23:00:42.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 292869 2.91 MB/sec execute 396 sec latency 206.438 ms 2024-02-09T23:00:43.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 294807 2.94 MB/sec execute 397 sec latency 11.603 ms 2024-02-09T23:00:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:43 smithi089 ceph-mon[36665]: pgmap v295: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-02-09T23:00:43.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:43 smithi003 ceph-mon[32678]: pgmap v295: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-02-09T23:00:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:43 smithi064 ceph-mon[36638]: pgmap v295: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-02-09T23:00:44.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 296065 2.94 MB/sec execute 398 sec latency 42.956 ms 2024-02-09T23:00:45.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 296117 2.94 MB/sec execute 399 sec latency 938.812 ms 2024-02-09T23:00:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:46 smithi003 ceph-mon[32678]: pgmap v296: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-02-09T23:00:46.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 296117 2.93 MB/sec execute 400 sec latency 1938.902 ms 2024-02-09T23:00:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:46 smithi064 ceph-mon[36638]: pgmap v296: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-02-09T23:00:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:46 smithi089 ceph-mon[36665]: pgmap v296: 97 pgs: 97 active+clean; 122 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 174 op/s 2024-02-09T23:00:47.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 298269 2.95 MB/sec execute 401 sec latency 1969.378 ms 2024-02-09T23:00:48.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:48 smithi003 ceph-mon[32678]: pgmap v297: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 278 op/s 2024-02-09T23:00:48.657 INFO:tasks.workunit.client.0.smithi003.stdout: 1 299507 2.96 MB/sec execute 402 sec latency 14.146 ms 2024-02-09T23:00:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:48 smithi064 ceph-mon[36638]: pgmap v297: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 278 op/s 2024-02-09T23:00:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:48 smithi089 ceph-mon[36665]: pgmap v297: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.7 MiB/s wr, 278 op/s 2024-02-09T23:00:49.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 299665 2.95 MB/sec execute 403 sec latency 869.848 ms 2024-02-09T23:00:50.529 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:50 smithi089 ceph-mon[36665]: pgmap v298: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-02-09T23:00:50.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:50 smithi003 ceph-mon[32678]: pgmap v298: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-02-09T23:00:50.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:00:50.647 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:00:50.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 300997 2.96 MB/sec execute 404 sec latency 1315.750 ms 2024-02-09T23:00:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:50 smithi064 ceph-mon[36638]: pgmap v298: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.0 MiB/s wr, 202 op/s 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 -- 172.21.15.3:0/3463017880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 msgr2=0x7f16bc100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/3463017880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc100720 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009930 tx=0x7f16b802bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 -- 172.21.15.3:0/3463017880 shutdown_connections 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/3463017880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16bc105410 0x7f16bc10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/3463017880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc100cf0 0x7f16bc104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/3463017880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc100720 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 -- 172.21.15.3:0/3463017880 >> 172.21.15.3:0/3463017880 conn(0x7f16bc0772e0 msgr2=0x7f16bc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 -- 172.21.15.3:0/3463017880 shutdown_connections 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 -- 172.21.15.3:0/3463017880 wait complete. 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 Processor -- start 2024-02-09T23:00:50.761 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.756+0000 7f16c3cc9640 1 -- start start 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc19d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc105410 0x7f16bc19db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16bc1976e0 0x7f16bc197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc1981b0 con 0x7f16bc100340 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc198320 con 0x7f16bc1976e0 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16bc1984e0 con 0x7f16bc105410 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc19d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc19d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58160/0 (socket says 172.21.15.3:58160) 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 -- 172.21.15.3:0/1100093909 learned_addr learned my addr 172.21.15.3:0/1100093909 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1264640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc105410 0x7f16bc19db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c2266640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16bc1976e0 0x7f16bc197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 -- 172.21.15.3:0/1100093909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc105410 msgr2=0x7f16bc19db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc105410 0x7f16bc19db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 -- 172.21.15.3:0/1100093909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16bc1976e0 msgr2=0x7f16bc197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16bc1976e0 0x7f16bc197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.762 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 -- 172.21.15.3:0/1100093909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8009560 con 0x7f16bc100340 2024-02-09T23:00:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c1a65640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc19d620 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009230 tx=0x7f16b802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16b2ffd640 1 -- 172.21.15.3:0/1100093909 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8004350 con 0x7f16bc100340 2024-02-09T23:00:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16b2ffd640 1 -- 172.21.15.3:0/1100093909 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16b80044b0 con 0x7f16bc100340 2024-02-09T23:00:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16bc1a4420 con 0x7f16bc100340 2024-02-09T23:00:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16b2ffd640 1 -- 172.21.15.3:0/1100093909 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b8003d00 con 0x7f16bc100340 2024-02-09T23:00:50.763 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.757+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16bc1a4950 con 0x7f16bc100340 2024-02-09T23:00:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.760+0000 7f16b2ffd640 1 -- 172.21.15.3:0/1100093909 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f16b8004df0 con 0x7f16bc100340 2024-02-09T23:00:50.764 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.760+0000 7f16c6979740 1 Processor -- start 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 -- start start 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 0x55e48e018c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e022310 0x55e48e022770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48dfd3d00 con 0x55e48e018860 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48df882f0 con 0x55e48e022310 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c6979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48df85820 con 0x55e48e019210 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 0x55e48e018c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c1a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 0x55e48e018c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58170/0 (socket says 172.21.15.3:58170) 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c1a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:40980/0 (socket says 172.21.15.3:40980) 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c1264640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e022310 0x55e48e022770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c2266640 1 -- 172.21.15.3:0/1333945668 learned_addr learned my addr 172.21.15.3:0/1333945668 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16b2ffd640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f16a80a22b0 0x7f16a80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c2266640 1 -- 172.21.15.3:0/1333945668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 msgr2=0x55e48e021d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.765 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.761+0000 7f16c2266640 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c2266640 1 -- 172.21.15.3:0/1333945668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e022310 msgr2=0x55e48e022770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c2266640 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e022310 0x55e48e022770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c2266640 1 -- 172.21.15.3:0/1333945668 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e48df42ff0 con 0x55e48e018860 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c1264640 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e022310 0x55e48e022770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16b2ffd640 1 -- 172.21.15.3:0/1100093909 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f16b80eff00 con 0x7f16bc100340 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c2266640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f16a80a22b0 0x7f16a80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c2266640 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 0x55e48e018c40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f16b4011590 tx=0x7f16b4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408905144535fc93 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16a7fff640 1 -- 172.21.15.3:0/1333945668 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b401bc10 con 0x55e48e018860 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16a7fff640 1 -- 172.21.15.3:0/1333945668 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16b401f070 con 0x55e48e018860 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16a7fff640 1 -- 172.21.15.3:0/1333945668 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400e3b0 con 0x55e48e018860 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.762+0000 7f16c2266640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f16a80a22b0 0x7f16a80a4770 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f16b4022000 tx=0x7f16b4023000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 -- 172.21.15.3:0/1333945668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 msgr2=0x55e48e018c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 0x55e48e018c40 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f16b4011590 tx=0x7f16b4011a50 comp rx=0 tx=0).stop 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 -- 172.21.15.3:0/1333945668 shutdown_connections 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e022310 0x55e48e022770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.771 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 --2- 172.21.15.3:0/1333945668 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e018860 0x55e48e018c40 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 -- 172.21.15.3:0/1333945668 >> 172.21.15.3:0/1333945668 conn(0x55e48e013050 msgr2=0x55e48e013430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 -- 172.21.15.3:0/1333945668 shutdown_connections 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 -- 172.21.15.3:0/1333945668 wait complete. 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 -- start start 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.763+0000 7f16c6979740 1 Processor -- start 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e025c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e0261c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e026700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48df81000 con 0x55e48e022310 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48dc54430 con 0x55e48e019210 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48dc10e00 con 0x55e48e018860 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c6979740 10 client.0 _ll_register_callbacks cb 0x55e48df757f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:50.772 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e026700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e026700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58178/0 (socket says 172.21.15.3:58178) 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 -- 172.21.15.3:0/1987987874 learned_addr learned my addr 172.21.15.3:0/1987987874 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c1a65640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e025c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c1264640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e0261c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 -- 172.21.15.3:0/1987987874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 msgr2=0x55e48e025c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e025c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 -- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 msgr2=0x55e48e0261c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e0261c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 -- 172.21.15.3:0/1987987874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b4009c40 con 0x55e48e022310 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c1264640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e0261c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16c2266640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e026700 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f16b4025d20 tx=0x7f16b4025930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5b4f40c3f340ac1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.764+0000 7f16a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16a57fa640 1 -- 172.21.15.3:0/1987987874 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400e3b0 con 0x55e48e022310 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16a57fa640 1 -- 172.21.15.3:0/1987987874 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f16b400f700 con 0x55e48e022310 2024-02-09T23:00:50.773 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e48dc807e0 con 0x55e48e022310 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16a57fa640 1 -- 172.21.15.3:0/1987987874 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16b401bc10 con 0x55e48e022310 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 10 client.18942 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55e48df7f680 con 0x55e48e022310 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16a57fa640 1 -- 172.21.15.3:0/1987987874 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16b403b020 con 0x55e48e022310 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16a57fa640 1 -- 172.21.15.3:0/1987987874 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f16b402c070 con 0x55e48e022310 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 10 client.18942 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 10 client.18942 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 20 client.18942 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e036770 0x55e48e038b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 4 client.18942 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e48dcbd720 con 0x55e48e036770 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.765+0000 7f16c1a65640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e036770 0x55e48e038b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16a57fa640 10 client.18942 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16c1a65640 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e036770 0x55e48e038b90 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d767d46665b74692 server_cookie=e14245a786d0921d in_seq=0 out_seq=0 2024-02-09T23:00:50.774 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16a57fa640 1 -- 172.21.15.3:0/1987987874 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e48dcbd720 con 0x55e48e036770 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16a57fa640 10 client.18942 handle_command_reply: tid=0 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16c6979740 1 client.18942 shutdown 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 msgr2=0x55e48e026700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.766+0000 7f16c6979740 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e026700 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f16b4025d20 tx=0x7f16b4025930 comp rx=0 tx=0).stop 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 shutdown_connections 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e036770 0x55e48e038b90 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e0261c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e026700 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 --2- 172.21.15.3:0/1987987874 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e025c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16a57fa640 0 client.18942 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 >> 172.21.15.3:0/1987987874 conn(0x55e48e013050 msgr2=0x55e48e014da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 shutdown_connections 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 1 -- 172.21.15.3:0/1987987874 wait complete. 2024-02-09T23:00:50.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.767+0000 7f16c6979740 20 client.18942 trim_cache size 0 max 16384 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 Processor -- start 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 -- start start 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e018860 0x55e48e0283b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e024c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48df05fd0 con 0x55e48e022310 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48e012ec0 con 0x55e48e018860 2024-02-09T23:00:50.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c6979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48e03dda0 con 0x55e48e019210 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e024c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e024c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58182/0 (socket says 172.21.15.3:58182) 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c2266640 1 -- 172.21.15.3:0/3837066132 learned_addr learned my addr 172.21.15.3:0/3837066132 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c1264640 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.783+0000 7f16c1a65640 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e018860 0x55e48e0283b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16c2266640 1 -- 172.21.15.3:0/3837066132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 msgr2=0x55e48e021ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16c2266640 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16c2266640 1 -- 172.21.15.3:0/3837066132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e018860 msgr2=0x55e48e0283b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16c2266640 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e018860 0x55e48e0283b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16c2266640 1 -- 172.21.15.3:0/3837066132 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e48df1bb30 con 0x55e48e022310 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16c2266640 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e024c60 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f16b4011fd0 tx=0x7f16b4011a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce5531013cd8eb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16a77fe640 1 -- 172.21.15.3:0/3837066132 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400e3b0 con 0x55e48e022310 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16a77fe640 1 -- 172.21.15.3:0/3837066132 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16b403b070 con 0x55e48e022310 2024-02-09T23:00:50.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.784+0000 7f16a77fe640 1 -- 172.21.15.3:0/3837066132 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b401ba10 con 0x55e48e022310 2024-02-09T23:00:50.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 -- 172.21.15.3:0/3837066132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 msgr2=0x55e48e024c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e024c60 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f16b4011fd0 tx=0x7f16b4011a50 comp rx=0 tx=0).stop 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 -- 172.21.15.3:0/3837066132 shutdown_connections 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e024c60 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e019210 0x55e48e021ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 --2- 172.21.15.3:0/3837066132 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e018860 0x55e48e0283b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 -- 172.21.15.3:0/3837066132 >> 172.21.15.3:0/3837066132 conn(0x55e48e013050 msgr2=0x55e48e024550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 -- 172.21.15.3:0/3837066132 shutdown_connections 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 -- 172.21.15.3:0/3837066132 wait complete. 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 -- start start 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.785+0000 7f16c6979740 1 Processor -- start 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e02d680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e02ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e02bf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48e02c660 con 0x55e48e022310 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48e02c7d0 con 0x55e48e019210 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e48e08cc80 con 0x55e48e018860 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 10 client.0 _ll_register_callbacks cb 0x7f16b4009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e02bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c1a65640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e02d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e02bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58196/0 (socket says 172.21.15.3:58196) 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 -- 172.21.15.3:0/3573998288 learned_addr learned my addr 172.21.15.3:0/3573998288 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c1264640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e02ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 -- 172.21.15.3:0/3573998288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 msgr2=0x55e48e02d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e02d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 -- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 msgr2=0x55e48e02ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e02ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 -- 172.21.15.3:0/3573998288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e48df7f680 con 0x55e48e022310 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c1a65640 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e02d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:50.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c2266640 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e02bf60 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000f80 tx=0x7f16b400e750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f3637f6a451af32 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16a57fa640 1 -- 172.21.15.3:0/3573998288 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16b400e3b0 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16a57fa640 1 -- 172.21.15.3:0/3573998288 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f16b403c080 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f16b401f070 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16a57fa640 1 -- 172.21.15.3:0/3573998288 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16b401b7d0 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 10 client.18948 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.786+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55e48df41850 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16a57fa640 1 -- 172.21.15.3:0/3573998288 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16b401f070 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16a57fa640 1 -- 172.21.15.3:0/3573998288 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f16b402c070 con 0x55e48e022310 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c6979740 10 client.18948 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c6979740 10 client.18948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c6979740 20 client.18948 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c6979740 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e08df70 0x55e48e090390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c6979740 4 client.18948 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e48dd0c890 con 0x55e48e08df70 2024-02-09T23:00:50.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c1a65640 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e08df70 0x55e48e090390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16a57fa640 10 client.18948 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16c1a65640 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e08df70 0x55e48e090390 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=331eb9c7fff3c38c server_cookie=c595c028bea1a05e in_seq=0 out_seq=0 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.787+0000 7f16a57fa640 1 -- 172.21.15.3:0/3573998288 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e48dd0c890 con 0x55e48e08df70 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16a57fa640 10 client.18948 handle_command_reply: tid=0 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 client.18948 shutdown 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 msgr2=0x55e48e02bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e02bf60 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f16b4000f80 tx=0x7f16b400e750 comp rx=0 tx=0).stop 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 shutdown_connections 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e48e08df70 0x55e48e090390 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e48e019210 0x55e48e02ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16a57fa640 0 client.18948 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e48e022310 0x55e48e02bf60 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 --2- 172.21.15.3:0/3573998288 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e48e018860 0x55e48e02d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 >> 172.21.15.3:0/3573998288 conn(0x55e48e013050 msgr2=0x55e48e011960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 shutdown_connections 2024-02-09T23:00:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 1 -- 172.21.15.3:0/3573998288 wait complete. 2024-02-09T23:00:50.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c6979740 20 client.18948 trim_cache size 0 max 16384 2024-02-09T23:00:50.797 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:00:50.797 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:00:50.797 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:00:50.797 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f16a80a22b0 msgr2=0x7f16a80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f16a80a22b0 0x7f16a80a4770 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f16b4022000 tx=0x7f16b4023000 comp rx=0 tx=0).stop 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 msgr2=0x7f16bc19d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc19d620 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f16b8009230 tx=0x7f16b802c650 comp rx=0 tx=0).stop 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 shutdown_connections 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f16a80a22b0 0x7f16a80a4770 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f16bc1976e0 0x7f16bc197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.788+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f16bc105410 0x7f16bc19db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.789+0000 7f16c3cc9640 1 --2- 172.21.15.3:0/1100093909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f16bc100340 0x7f16bc19d620 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.789+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 >> 172.21.15.3:0/1100093909 conn(0x7f16bc0772e0 msgr2=0x7f16bc0688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.789+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 shutdown_connections 2024-02-09T23:00:50.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.789+0000 7f16c3cc9640 1 -- 172.21.15.3:0/1100093909 wait complete. 2024-02-09T23:00:50.803 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:00:50.803 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 -- 172.21.15.3:0/3531728773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10105420 msgr2=0x7fed1010cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 --2- 172.21.15.3:0/3531728773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10105420 0x7fed1010cff0 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7fed08009cd0 tx=0x7fed0802c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 -- 172.21.15.3:0/3531728773 shutdown_connections 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 --2- 172.21.15.3:0/3531728773 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10105420 0x7fed1010cff0 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 --2- 172.21.15.3:0/3531728773 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed10100d00 0x7fed10104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 --2- 172.21.15.3:0/3531728773 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed10100350 0x7fed10100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 -- 172.21.15.3:0/3531728773 >> 172.21.15.3:0/3531728773 conn(0x7fed100772b0 msgr2=0x7fed100776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:50.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 -- 172.21.15.3:0/3531728773 shutdown_connections 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 -- 172.21.15.3:0/3531728773 wait complete. 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 Processor -- start 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.948+0000 7fed1738c640 1 -- start start 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed10100350 0x7fed1019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed10105420 0x7fed1019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 0x7fed10197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed10198130 con 0x7fed10197660 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed101982a0 con 0x7fed10105420 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed10198460 con 0x7fed10100350 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 0x7fed10197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 0x7fed10197ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58212/0 (socket says 172.21.15.3:58212) 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 -- 172.21.15.3:0/346428627 learned_addr learned my addr 172.21.15.3:0/346428627 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15128640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed10100350 0x7fed1019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed14927640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed10105420 0x7fed1019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 -- 172.21.15.3:0/346428627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed10100350 msgr2=0x7fed1019d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed10100350 0x7fed1019d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 -- 172.21.15.3:0/346428627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed10105420 msgr2=0x7fed1019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed10105420 0x7fed1019dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 -- 172.21.15.3:0/346428627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c009490 con 0x7fed10197660 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed15929640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 0x7fed10197ae0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fed08004af0 tx=0x7fed08032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed080328b0 con 0x7fed10197660 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed08003e50 con 0x7fed10197660 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed08009900 con 0x7fed10197660 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.949+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed08004740 con 0x7fed10197660 2024-02-09T23:00:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.950+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed101a46e0 con 0x7fed10197660 2024-02-09T23:00:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.951+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed10101820 con 0x7fed10197660 2024-02-09T23:00:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.952+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fed0800a040 con 0x7fed10197660 2024-02-09T23:00:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.952+0000 7fed067fc640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecfc0a22b0 0x7fecfc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.952+0000 7fed15128640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecfc0a22b0 0x7fecfc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.952+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fed080f0700 con 0x7fed10197660 2024-02-09T23:00:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.953+0000 7fed15128640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecfc0a22b0 0x7fecfc0a4770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009d60 tx=0x7fed0c005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:50.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:50.957+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fed080b94a0 con 0x7fed10197660 2024-02-09T23:00:51.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.197+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed10109700 con 0x7fed10197660 2024-02-09T23:00:51.201 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.198+0000 7fed067fc640 1 -- 172.21.15.3:0/346428627 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+11361 (secure 0 0 0) 0x7fed080bcb60 con 0x7fed10197660 2024-02-09T23:00:51.201 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:00:51.201 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:59:11.242420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:00:51.202 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 47 2024-02-09T23:00:51.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecfc0a22b0 msgr2=0x7fecfc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecfc0a22b0 0x7fecfc0a4770 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009d60 tx=0x7fed0c005f90 comp rx=0 tx=0).stop 2024-02-09T23:00:51.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 msgr2=0x7fed10197ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.203 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 0x7fed10197ae0 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fed08004af0 tx=0x7fed08032570 comp rx=0 tx=0).stop 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 shutdown_connections 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fecfc0a22b0 0x7fecfc0a4770 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed10105420 0x7fed1019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed10197660 0x7fed10197ae0 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 --2- 172.21.15.3:0/346428627 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed10100350 0x7fed1019d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 >> 172.21.15.3:0/346428627 conn(0x7fed100772b0 msgr2=0x7fed10068880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 shutdown_connections 2024-02-09T23:00:51.204 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.200+0000 7fed1738c640 1 -- 172.21.15.3:0/346428627 wait complete. 2024-02-09T23:00:51.214 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:51.214 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:00:51.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 -- 172.21.15.3:0/2145073937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd20100340 msgr2=0x7efd20100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 --2- 172.21.15.3:0/2145073937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd20100340 0x7efd20100720 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7efd18009900 tx=0x7efd1802bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 -- 172.21.15.3:0/2145073937 shutdown_connections 2024-02-09T23:00:51.326 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 --2- 172.21.15.3:0/2145073937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20105410 0x7efd2010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 --2- 172.21.15.3:0/2145073937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20100cf0 0x7efd20104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 --2- 172.21.15.3:0/2145073937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd20100340 0x7efd20100720 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 -- 172.21.15.3:0/2145073937 >> 172.21.15.3:0/2145073937 conn(0x7efd200772b0 msgr2=0x7efd200776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 -- 172.21.15.3:0/2145073937 shutdown_connections 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.323+0000 7efd27a17640 1 -- 172.21.15.3:0/2145073937 wait complete. 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 Processor -- start 2024-02-09T23:00:51.327 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 -- start start 2024-02-09T23:00:51.328 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 0x7efd2019d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20105410 0x7efd2019db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd201976a0 0x7efd20197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd20198170 con 0x7efd201976a0 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd201982e0 con 0x7efd20100340 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd27a17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd201984a0 con 0x7efd20105410 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 0x7efd2019d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd25fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd201976a0 0x7efd20197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd24fb2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20105410 0x7efd2019db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 0x7efd2019d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:50880/0 (socket says 172.21.15.3:50880) 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 -- 172.21.15.3:0/21080030 learned_addr learned my addr 172.21.15.3:0/21080030 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 -- 172.21.15.3:0/21080030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20105410 msgr2=0x7efd2019db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20105410 0x7efd2019db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 -- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd201976a0 msgr2=0x7efd20197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.329 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd201976a0 0x7efd20197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd257b3640 1 -- 172.21.15.3:0/21080030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd18009560 con 0x7efd20100340 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.324+0000 7efd24fb2640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20105410 0x7efd2019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.325+0000 7efd257b3640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 0x7efd2019d5e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd18005940 tx=0x7efd1802c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.325+0000 7efd1e7fc640 1 -- 172.21.15.3:0/21080030 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd180043d0 con 0x7efd20100340 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.325+0000 7efd1e7fc640 1 -- 172.21.15.3:0/21080030 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd180052d0 con 0x7efd20100340 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.325+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd2010d520 con 0x7efd20100340 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.325+0000 7efd1e7fc640 1 -- 172.21.15.3:0/21080030 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd18003d30 con 0x7efd20100340 2024-02-09T23:00:51.330 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.325+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2010da30 con 0x7efd20100340 2024-02-09T23:00:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.327+0000 7efd1e7fc640 1 -- 172.21.15.3:0/21080030 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efd18005440 con 0x7efd20100340 2024-02-09T23:00:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.327+0000 7efd2a5df740 1 Processor -- start 2024-02-09T23:00:51.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 -- start start 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db62832e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6283b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db6283e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd1e7fc640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd080a2230 0x7efd080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db627ee020 con 0x55db62832a30 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db627a1320 con 0x55db6283bf60 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd2a5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db6279e850 con 0x55db628333e0 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db62832e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db62832e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58228/0 (socket says 172.21.15.3:58228) 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 -- 172.21.15.3:0/281593138 learned_addr learned my addr 172.21.15.3:0/281593138 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd257b3640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6283b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd24fb2640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db6283e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd080a2230 0x7efd080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd1e7fc640 1 -- 172.21.15.3:0/21080030 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7efd180f0130 con 0x7efd20100340 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 -- 172.21.15.3:0/281593138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 msgr2=0x55db6283b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6283b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 -- 172.21.15.3:0/281593138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 msgr2=0x55db6283e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db6283e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 -- 172.21.15.3:0/281593138 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db6275c210 con 0x55db62832a30 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd257b3640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6283b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd24fb2640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db6283e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.328+0000 7efd25fb4640 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db62832e10 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7efd10012cf0 tx=0x7efd10010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95211d9b8dbb85f6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd25fb4640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd080a2230 0x7efd080a46f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efd10010c40 tx=0x7efd10017000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd077fe640 1 -- 172.21.15.3:0/281593138 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10012de0 con 0x55db62832a30 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd077fe640 1 -- 172.21.15.3:0/281593138 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd10009cf0 con 0x55db62832a30 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd077fe640 1 -- 172.21.15.3:0/281593138 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd10005020 con 0x55db62832a30 2024-02-09T23:00:51.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 -- 172.21.15.3:0/281593138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 msgr2=0x55db62832e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db62832e10 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7efd10012cf0 tx=0x7efd10010740 comp rx=0 tx=0).stop 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 -- 172.21.15.3:0/281593138 shutdown_connections 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db6283e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6283b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 --2- 172.21.15.3:0/281593138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db62832e10 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 -- 172.21.15.3:0/281593138 >> 172.21.15.3:0/281593138 conn(0x55db6282d100 msgr2=0x55db6282d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 -- 172.21.15.3:0/281593138 shutdown_connections 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.329+0000 7efd2a5df740 1 -- 172.21.15.3:0/281593138 wait complete. 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 -- start start 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 Processor -- start 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62838f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db628333e0 0x55db628394c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 0x55db62839a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db6279a030 con 0x55db6283bf60 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db6246d430 con 0x55db628333e0 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db62429e00 con 0x55db62832a30 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd2a5df740 10 client.0 _ll_register_callbacks cb 0x55db6278f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd25fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 0x55db62839a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd25fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 0x55db62839a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58230/0 (socket says 172.21.15.3:58230) 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd25fb4640 1 -- 172.21.15.3:0/403567906 learned_addr learned my addr 172.21.15.3:0/403567906 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.330+0000 7efd24fb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db628333e0 0x55db628394c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd257b3640 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62838f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd25fb4640 1 -- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 msgr2=0x55db62838f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd25fb4640 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62838f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd25fb4640 1 -- 172.21.15.3:0/403567906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db628333e0 msgr2=0x55db628394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd25fb4640 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db628333e0 0x55db628394c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd25fb4640 1 -- 172.21.15.3:0/403567906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1001a040 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd25fb4640 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 0x55db62839a00 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7efd10009a30 tx=0x7efd10012900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab79c520611559fa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd04ff9640 1 -- 172.21.15.3:0/403567906 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd1000dc50 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd04ff9640 1 -- 172.21.15.3:0/403567906 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7efd1000ddb0 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55db624997e0 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd04ff9640 1 -- 172.21.15.3:0/403567906 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd10004a00 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd2a5df740 10 client.18966 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55db627986b0 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd04ff9640 1 -- 172.21.15.3:0/403567906 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd1003c080 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.331+0000 7efd04ff9640 1 -- 172.21.15.3:0/403567906 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efd10036030 con 0x55db6283bf60 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd2a5df740 10 client.18966 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd2a5df740 10 client.18966 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:00:51.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd2a5df740 20 client.18966 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd2a5df740 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db6284c550 0x55db6284e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd2a5df740 4 client.18966 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55db624d6720 con 0x55db6284c550 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd257b3640 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db6284c550 0x55db6284e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd257b3640 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db6284c550 0x55db6284e970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=252608c2b37ab920 server_cookie=b21c9c98206bdd1 in_seq=0 out_seq=0 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.332+0000 7efd04ff9640 10 client.18966 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd04ff9640 1 -- 172.21.15.3:0/403567906 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55db624d6720 con 0x55db6284c550 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd04ff9640 10 client.18966 handle_command_reply: tid=0 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 client.18966 shutdown 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 msgr2=0x55db62839a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 0x55db62839a00 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7efd10009a30 tx=0x7efd10012900 comp rx=0 tx=0).stop 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 shutdown_connections 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db6284c550 0x55db6284e970 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db628333e0 0x55db628394c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd04ff9640 0 client.18966 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db6283bf60 0x55db62839a00 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 --2- 172.21.15.3:0/403567906 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62838f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 >> 172.21.15.3:0/403567906 conn(0x55db6282d100 msgr2=0x55db6282ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 shutdown_connections 2024-02-09T23:00:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 1 -- 172.21.15.3:0/403567906 wait complete. 2024-02-09T23:00:51.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.333+0000 7efd2a5df740 20 client.18966 trim_cache size 0 max 16384 2024-02-09T23:00:51.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 Processor -- start 2024-02-09T23:00:51.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 -- start start 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db6283b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6282d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62839820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db6271f130 con 0x55db62832a30 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db62828a70 con 0x55db6283bf60 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd2a5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db628374d0 con 0x55db628333e0 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db6283b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db6283b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58246/0 (socket says 172.21.15.3:58246) 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 -- 172.21.15.3:0/658472292 learned_addr learned my addr 172.21.15.3:0/658472292 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd25fb4640 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62839820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd24fb2640 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6282d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 -- 172.21.15.3:0/658472292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 msgr2=0x55db6282d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6282d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 -- 172.21.15.3:0/658472292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 msgr2=0x55db62839820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62839820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.346+0000 7efd257b3640 1 -- 172.21.15.3:0/658472292 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55db62734c90 con 0x55db62832a30 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd257b3640 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db6283b6b0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7efd18006010 tx=0x7efd1808ca30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=250aa6ed288030cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd06ffd640 1 -- 172.21.15.3:0/658472292 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd180035f0 con 0x55db62832a30 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd06ffd640 1 -- 172.21.15.3:0/658472292 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd180bd2d0 con 0x55db62832a30 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd06ffd640 1 -- 172.21.15.3:0/658472292 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd180b9240 con 0x55db62832a30 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 -- 172.21.15.3:0/658472292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 msgr2=0x55db6283b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db6283b6b0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7efd18006010 tx=0x7efd1808ca30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 -- 172.21.15.3:0/658472292 shutdown_connections 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62839820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db628333e0 0x55db6282d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 --2- 172.21.15.3:0/658472292 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db62832a30 0x55db6283b6b0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 -- 172.21.15.3:0/658472292 >> 172.21.15.3:0/658472292 conn(0x55db6282d100 msgr2=0x55db628337c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 -- 172.21.15.3:0/658472292 shutdown_connections 2024-02-09T23:00:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.347+0000 7efd2a5df740 1 -- 172.21.15.3:0/658472292 wait complete. 2024-02-09T23:00:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 -- start start 2024-02-09T23:00:51.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 Processor -- start 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62843350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 0x55db628416f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62841c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db62842370 con 0x55db628333e0 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db628424e0 con 0x55db6283bf60 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55db628a2a60 con 0x55db62832a30 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd2a5df740 10 client.0 _ll_register_callbacks cb 0x7efd10009f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 0x55db628416f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 0x55db628416f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58250/0 (socket says 172.21.15.3:58250) 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 -- 172.21.15.3:0/89128002 learned_addr learned my addr 172.21.15.3:0/89128002 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd257b3640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62843350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd25fb4640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62841c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 -- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 msgr2=0x55db62843350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62843350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 -- 172.21.15.3:0/89128002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 msgr2=0x55db62841c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.348+0000 7efd24fb2640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62841c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd24fb2640 1 -- 172.21.15.3:0/89128002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd2010d520 con 0x55db628333e0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd24fb2640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 0x55db628416f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x55db628387e0 tx=0x7efd14009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519b2820fc63d6d8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd04ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd04ff9640 1 -- 172.21.15.3:0/89128002 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd14015040 con 0x55db628333e0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd04ff9640 1 -- 172.21.15.3:0/89128002 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7efd14005020 con 0x55db628333e0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efd10009cf0 con 0x55db628333e0 2024-02-09T23:00:51.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd04ff9640 1 -- 172.21.15.3:0/89128002 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7efd140051e0 con 0x55db628333e0 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 10 client.18972 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55db6275aa70 con 0x55db628333e0 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd04ff9640 1 -- 172.21.15.3:0/89128002 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efd14030080 con 0x55db628333e0 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd04ff9640 1 -- 172.21.15.3:0/89128002 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7efd1402a030 con 0x55db628333e0 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 10 client.18972 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 10 client.18972 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 20 client.18972 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db628a4360 0x55db628a6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 4 client.18972 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.349+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55db62525890 con 0x55db628a4360 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd257b3640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db628a4360 0x55db628a6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd04ff9640 10 client.18972 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd257b3640 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db628a4360 0x55db628a6780 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7b6e3ecd181bb6 server_cookie=eb0714e10fbeef41 in_seq=0 out_seq=0 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd04ff9640 1 -- 172.21.15.3:0/89128002 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55db62525890 con 0x55db628a4360 2024-02-09T23:00:51.359 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd04ff9640 10 client.18972 handle_command_reply: tid=0 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 client.18972 shutdown 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 msgr2=0x55db628416f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 0x55db628416f0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x55db628387e0 tx=0x7efd14009d30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 shutdown_connections 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55db628a4360 0x55db628a6780 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55db6283bf60 0x55db62841c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55db628333e0 0x55db628416f0 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd04ff9640 0 client.18972 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 --2- 172.21.15.3:0/89128002 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55db62832a30 0x55db62843350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 >> 172.21.15.3:0/89128002 conn(0x55db6282d100 msgr2=0x55db62834270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 shutdown_connections 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 1 -- 172.21.15.3:0/89128002 wait complete. 2024-02-09T23:00:51.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.350+0000 7efd2a5df740 20 client.18972 trim_cache size 0 max 16384 2024-02-09T23:00:51.361 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd080a2230 msgr2=0x7efd080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd080a2230 0x7efd080a46f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7efd10010c40 tx=0x7efd10017000 comp rx=0 tx=0).stop 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 msgr2=0x7efd2019d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 0x7efd2019d5e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7efd18005940 tx=0x7efd1802c6e0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 shutdown_connections 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efd080a2230 0x7efd080a46f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efd201976a0 0x7efd20197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efd20105410 0x7efd2019db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 --2- 172.21.15.3:0/21080030 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efd20100340 0x7efd2019d5e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 >> 172.21.15.3:0/21080030 conn(0x7efd200772b0 msgr2=0x7efd200fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 shutdown_connections 2024-02-09T23:00:51.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.351+0000 7efd27a17640 1 -- 172.21.15.3:0/21080030 wait complete. 2024-02-09T23:00:51.365 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:51.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:00:51.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 -- 172.21.15.3:0/521782138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc100340 msgr2=0x7f0fcc100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/521782138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc100340 0x7f0fcc100720 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc009930 tx=0x7f0fbc02bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 -- 172.21.15.3:0/521782138 shutdown_connections 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/521782138 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fcc105410 0x7f0fcc10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/521782138 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100cf0 0x7f0fcc104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/521782138 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc100340 0x7f0fcc100720 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 -- 172.21.15.3:0/521782138 >> 172.21.15.3:0/521782138 conn(0x7f0fcc0772e0 msgr2=0x7f0fcc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 -- 172.21.15.3:0/521782138 shutdown_connections 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.471+0000 7f0fd0be9640 1 -- 172.21.15.3:0/521782138 wait complete. 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 Processor -- start 2024-02-09T23:00:51.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 -- start start 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100340 0x7f0fcc1986d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 0x7f0fcc198c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fcc1991a0 0x7f0fcc1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc1a3e20 con 0x7f0fcc105410 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc1a3f90 con 0x7f0fcc1991a0 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fd0be9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fcc1a4150 con 0x7f0fcc100340 2024-02-09T23:00:51.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 0x7f0fcc198c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 0x7f0fcc198c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58268/0 (socket says 172.21.15.3:58268) 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fca59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100340 0x7f0fcc1986d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3599685334 learned_addr learned my addr 172.21.15.3:0/3599685334 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fcad9d640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fcc1991a0 0x7f0fcc1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3599685334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100340 msgr2=0x7f0fcc1986d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100340 0x7f0fcc1986d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3599685334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fcc1991a0 msgr2=0x7f0fcc1a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fcc1991a0 0x7f0fcc1a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3599685334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fbc009560 con 0x7f0fcc105410 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.472+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100340 0x7f0fcc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.473+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 0x7f0fcc198c10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000e9c0 tx=0x7f0fc000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.473+0000 7f0fbb7fe640 1 -- 172.21.15.3:0/3599685334 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0009700 con 0x7f0fcc105410 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.473+0000 7f0fbb7fe640 1 -- 172.21.15.3:0/3599685334 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc000cd80 con 0x7f0fcc105410 2024-02-09T23:00:51.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.473+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fcc1a4430 con 0x7f0fcc105410 2024-02-09T23:00:51.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.473+0000 7f0fbb7fe640 1 -- 172.21.15.3:0/3599685334 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0010430 con 0x7f0fcc105410 2024-02-09T23:00:51.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.473+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc1a4940 con 0x7f0fcc105410 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.475+0000 7f0fbb7fe640 1 -- 172.21.15.3:0/3599685334 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0fc0005000 con 0x7f0fcc105410 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fbb7fe640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0fb40a2230 0x7f0fb40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 Processor -- start 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0fb40a2230 0x7f0fb40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fbb7fe640 1 -- 172.21.15.3:0/3599685334 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0fc0014070 con 0x7f0fcc105410 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 -- start start 2024-02-09T23:00:51.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b72910 0x55fab9b72cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 0x55fab9b7bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b7c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9b2e020 con 0x55fab9b72910 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9ae1320 con 0x55fab9b732c0 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9ade850 con 0x55fab9b7c3c0 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 0x55fab9b7bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 0x55fab9b7bdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:50936/0 (socket says 172.21.15.3:50936) 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3425430844 learned_addr learned my addr 172.21.15.3:0/3425430844 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fcad9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b7c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b72910 0x55fab9b72cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.476+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0fb40a2230 0x7f0fb40a46f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc005ab0 tx=0x7f0fbc005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3425430844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 msgr2=0x55fab9b7c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b7c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3425430844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b72910 msgr2=0x55fab9b72cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b72910 0x55fab9b72cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3425430844 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fab9a9c210 con 0x55fab9b732c0 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 0x55fab9b7bdf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00972e0 tx=0x7f0fc009ad80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a5036e06746841a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fb8ff9640 1 -- 172.21.15.3:0/3425430844 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00a3620 con 0x55fab9b732c0 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fb8ff9640 1 -- 172.21.15.3:0/3425430844 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0099180 con 0x55fab9b732c0 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.477+0000 7f0fb8ff9640 1 -- 172.21.15.3:0/3425430844 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0005510 con 0x55fab9b732c0 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3425430844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 msgr2=0x55fab9b7bdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 0x55fab9b7bdf0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00972e0 tx=0x7f0fc009ad80 comp rx=0 tx=0).stop 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3425430844 shutdown_connections 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b732c0 0x55fab9b7bdf0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b7c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3425430844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b72910 0x55fab9b72cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3425430844 >> 172.21.15.3:0/3425430844 conn(0x55fab9b6d100 msgr2=0x55fab9b6d4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3425430844 shutdown_connections 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3425430844 wait complete. 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- start start 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 Processor -- start 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b7fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b803c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b80900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.478+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9ada030 con 0x55fab9b732c0 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab97ad430 con 0x55fab9b72910 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9769e00 con 0x55fab9b7c3c0 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fd37c4740 10 client.0 _ll_register_callbacks cb 0x55fab9acf030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b803c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.488 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b803c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58284/0 (socket says 172.21.15.3:58284) 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3108996794 learned_addr learned my addr 172.21.15.3:0/3108996794 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b7fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fcad9d640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b80900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 msgr2=0x55fab9b80900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b80900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3108996794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 msgr2=0x55fab9b7fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b7fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/3108996794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fcc1a4940 con 0x55fab9b732c0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b803c0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00cc1e0 tx=0x7f0fc00056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9c3224e8d1d8db server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/3108996794 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00a3620 con 0x55fab9b732c0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/3108996794 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0fc00b92f0 con 0x55fab9b732c0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fab97d97e0 con 0x55fab9b732c0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/3108996794 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc0084020 con 0x55fab9b732c0 2024-02-09T23:00:51.489 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fd37c4740 10 client.18984 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.479+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55fab9ad86b0 con 0x55fab9b732c0 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/3108996794 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fab97d97e0 con 0x55fab9b732c0 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/3108996794 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0fc001f060 con 0x55fab9b732c0 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fd37c4740 10 client.18984 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fd37c4740 10 client.18984 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fd37c4740 20 client.18984 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9b90560 0x55fab9b92980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fd37c4740 4 client.18984 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fab9816720 con 0x55fab9b90560 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9b90560 0x55fab9b92980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.480+0000 7f0fca59c640 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9b90560 0x55fab9b92980 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=efdbd24bd22b7624 server_cookie=4a11da4e8be098c4 in_seq=0 out_seq=0 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fb1ffb640 10 client.18984 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/3108996794 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fab9816720 con 0x55fab9b90560 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fb1ffb640 10 client.18984 handle_command_reply: tid=0 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 client.18984 shutdown 2024-02-09T23:00:51.490 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 msgr2=0x55fab9b803c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b803c0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00cc1e0 tx=0x7f0fc00056f0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 shutdown_connections 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9b90560 0x55fab9b92980 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b80900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b803c0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fb1ffb640 0 client.18984 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/3108996794 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b7fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 >> 172.21.15.3:0/3108996794 conn(0x55fab9b6d100 msgr2=0x55fab9b6ee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 shutdown_connections 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.481+0000 7f0fd37c4740 1 -- 172.21.15.3:0/3108996794 wait complete. 2024-02-09T23:00:51.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.482+0000 7f0fd37c4740 20 client.18984 trim_cache size 0 max 16384 2024-02-09T23:00:51.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 Processor -- start 2024-02-09T23:00:51.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 -- start start 2024-02-09T23:00:51.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b825b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b80720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b972e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9b7a2f0 con 0x55fab9b732c0 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9a5f130 con 0x55fab9b72910 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.495+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9b71d70 con 0x55fab9b7c3c0 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b80720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b80720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58300/0 (socket says 172.21.15.3:58300) 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/4127542617 learned_addr learned my addr 172.21.15.3:0/4127542617 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fcad9d640 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b972e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fca59c640 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b825b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/4127542617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 msgr2=0x55fab9b972e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b972e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/4127542617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 msgr2=0x55fab9b825b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b825b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/4127542617 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fab9a74c90 con 0x55fab9b732c0 2024-02-09T23:00:51.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b80720 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f0fc009ae10 tx=0x7f0fc00ce440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91c32ce4026606f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fb3fff640 1 -- 172.21.15.3:0/4127542617 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00a3620 con 0x55fab9b732c0 2024-02-09T23:00:51.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fb3fff640 1 -- 172.21.15.3:0/4127542617 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc009efd0 con 0x55fab9b732c0 2024-02-09T23:00:51.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.496+0000 7f0fb3fff640 1 -- 172.21.15.3:0/4127542617 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00840f0 con 0x55fab9b732c0 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 -- 172.21.15.3:0/4127542617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 msgr2=0x55fab9b80720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b80720 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f0fc009ae10 tx=0x7f0fc00ce440 comp rx=0 tx=0).stop 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 -- 172.21.15.3:0/4127542617 shutdown_connections 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b7c3c0 0x55fab9b972e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b80720 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/4127542617 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b72910 0x55fab9b825b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 -- 172.21.15.3:0/4127542617 >> 172.21.15.3:0/4127542617 conn(0x55fab9b6d100 msgr2=0x55fab9b68180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 -- 172.21.15.3:0/4127542617 shutdown_connections 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.497+0000 7f0fd37c4740 1 -- 172.21.15.3:0/4127542617 wait complete. 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 -- start start 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 Processor -- start 2024-02-09T23:00:51.505 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b72910 0x55fab9b873a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b878e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b7c3c0 0x55fab9b859d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9b860a0 con 0x55fab9b732c0 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9b86210 con 0x55fab9b7c3c0 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fab9b863d0 con 0x55fab9b72910 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fd37c4740 10 client.0 _ll_register_callbacks cb 0x7f0fc000cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b878e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fca59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b72910 0x55fab9b873a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b878e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58316/0 (socket says 172.21.15.3:58316) 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/1937739948 learned_addr learned my addr 172.21.15.3:0/1937739948 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fcad9d640 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b7c3c0 0x55fab9b859d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b72910 msgr2=0x55fab9b873a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b72910 0x55fab9b873a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/1937739948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b7c3c0 msgr2=0x55fab9b859d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b7c3c0 0x55fab9b859d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.506 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.498+0000 7f0fc9d9b640 1 -- 172.21.15.3:0/1937739948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fab9ad86b0 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fc9d9b640 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b878e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00048f0 tx=0x7f0fc00cbc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d40fb7f257afad9c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fb1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/1937739948 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00b3bb0 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/1937739948 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0fc00a3620 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0fc0099180 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/1937739948 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0fc0085b40 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 10 client.18990 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55fab9a9aa70 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/1937739948 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0fc0099180 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/1937739948 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0fc009a3f0 con 0x55fab9b732c0 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 10 client.18990 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 10 client.18990 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 20 client.18990 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9be7870 0x55fab9be9c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 4 client.18990 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:51.507 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.499+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fab9865890 con 0x55fab9be7870 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fca59c640 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9be7870 0x55fab9be9c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fb1ffb640 10 client.18990 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fca59c640 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9be7870 0x55fab9be9c60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8970e70180dae375 server_cookie=7f53518170cc9ec8 in_seq=0 out_seq=0 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fb1ffb640 1 -- 172.21.15.3:0/1937739948 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fab9865890 con 0x55fab9be7870 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fb1ffb640 10 client.18990 handle_command_reply: tid=0 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 client.18990 shutdown 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 msgr2=0x55fab9b878e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b878e0 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f0fc00048f0 tx=0x7f0fc00cbc90 comp rx=0 tx=0).stop 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 shutdown_connections 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55fab9be7870 0x55fab9be9c60 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fab9b7c3c0 0x55fab9b859d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fb1ffb640 0 client.18990 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:00:51.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fab9b732c0 0x55fab9b878e0 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 --2- 172.21.15.3:0/1937739948 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fab9b72910 0x55fab9b873a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 >> 172.21.15.3:0/1937739948 conn(0x55fab9b6d100 msgr2=0x55fab9b972e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 shutdown_connections 2024-02-09T23:00:51.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 1 -- 172.21.15.3:0/1937739948 wait complete. 2024-02-09T23:00:51.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.500+0000 7f0fd37c4740 20 client.18990 trim_cache size 0 max 16384 2024-02-09T23:00:51.509 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0fb40a2230 msgr2=0x7f0fb40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0fb40a2230 0x7f0fb40a46f0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc005ab0 tx=0x7f0fbc005fd0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 msgr2=0x7f0fcc198c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 0x7f0fcc198c10 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000e9c0 tx=0x7f0fc000ee80 comp rx=0 tx=0).stop 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 shutdown_connections 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0fb40a2230 0x7f0fb40a46f0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0fcc1991a0 0x7f0fcc1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0fcc105410 0x7f0fcc198c10 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 --2- 172.21.15.3:0/3599685334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0fcc100340 0x7f0fcc1986d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 >> 172.21.15.3:0/3599685334 conn(0x7f0fcc0772e0 msgr2=0x7f0fcc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 shutdown_connections 2024-02-09T23:00:51.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.501+0000 7f0fd0be9640 1 -- 172.21.15.3:0/3599685334 wait complete. 2024-02-09T23:00:51.514 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/346428627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:00:51.515 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:51.515 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:00:51.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 -- 172.21.15.3:0/834984278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb240106200 msgr2=0x7fb2401065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 --2- 172.21.15.3:0/834984278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb240106200 0x7fb2401065e0 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7fb230009930 tx=0x7fb23002bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 -- 172.21.15.3:0/834984278 shutdown_connections 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 --2- 172.21.15.3:0/834984278 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb2400ff6a0 0x7fb2400ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 --2- 172.21.15.3:0/834984278 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb240100060 0x7fb24010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 --2- 172.21.15.3:0/834984278 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb240106200 0x7fb2401065e0 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 -- 172.21.15.3:0/834984278 >> 172.21.15.3:0/834984278 conn(0x7fb240076630 msgr2=0x7fb2400fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 -- 172.21.15.3:0/834984278 shutdown_connections 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.620+0000 7fb2467c2640 1 -- 172.21.15.3:0/834984278 wait complete. 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 Processor -- start 2024-02-09T23:00:51.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 -- start start 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 0x7fb240100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb240100060 0x7fb240100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb240106200 0x7fb240101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401a5cb0 con 0x7fb2400ff6a0 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401a5e20 con 0x7fb240106200 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb2467c2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2401a5fe0 con 0x7fb240100060 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 0x7fb240100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 0x7fb240100880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58322/0 (socket says 172.21.15.3:58322) 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb23f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb240100060 0x7fb240100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb23ffff640 1 -- 172.21.15.3:0/2446330111 learned_addr learned my addr 172.21.15.3:0/2446330111 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.621+0000 7fb244d5f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb240106200 0x7fb240101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23ffff640 1 -- 172.21.15.3:0/2446330111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb240100060 msgr2=0x7fb240100dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb240100060 0x7fb240100dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23ffff640 1 -- 172.21.15.3:0/2446330111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb240106200 msgr2=0x7fb240101300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb240106200 0x7fb240101300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23ffff640 1 -- 172.21.15.3:0/2446330111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230009560 con 0x7fb2400ff6a0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 0x7fb240100880 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb230009230 tx=0x7fb230004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23d7fa640 1 -- 172.21.15.3:0/2446330111 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300042d0 con 0x7fb2400ff6a0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23d7fa640 1 -- 172.21.15.3:0/2446330111 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb230004430 con 0x7fb2400ff6a0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2401a6260 con 0x7fb2400ff6a0 2024-02-09T23:00:51.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb23d7fa640 1 -- 172.21.15.3:0/2446330111 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230003c90 con 0x7fb2400ff6a0 2024-02-09T23:00:51.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.622+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2401a6770 con 0x7fb2400ff6a0 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.624+0000 7fb23d7fa640 1 -- 172.21.15.3:0/2446330111 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb230005560 con 0x7fb2400ff6a0 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 Processor -- start 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 -- start start 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbba30 0x558f3ffbbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 0x558f3ffc73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ff77020 con 0x558f3ffc4f60 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ff2a320 con 0x558f3ffbba30 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb24938a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ff27850 con 0x558f3ffbc3e0 2024-02-09T23:00:51.634 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23d7fa640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2280a2230 0x7fb2280a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 0x558f3ffc73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 0x558f3ffc73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58326/0 (socket says 172.21.15.3:58326) 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23f7fe640 1 -- 172.21.15.3:0/2479891872 learned_addr learned my addr 172.21.15.3:0/2479891872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbba30 0x558f3ffbbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2280a2230 0x7fb2280a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 -- 172.21.15.3:0/2479891872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 msgr2=0x558f3ffc4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23d7fa640 1 -- 172.21.15.3:0/2446330111 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb2300effd0 con 0x7fb2400ff6a0 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 -- 172.21.15.3:0/2479891872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 msgr2=0x558f3ffc73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 0x558f3ffc73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb244d5f640 1 -- 172.21.15.3:0/2479891872 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f3fee5210 con 0x558f3ffbba30 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.625+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 0x558f3ffc73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.635 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.626+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbba30 0x558f3ffbbe10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb238011cc0 tx=0x7fb23800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cd404af93b02c92 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.626+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2280a2230 0x7fb2280a46f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb23800fc90 tx=0x7fb238018000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.626+0000 7fb2267fc640 1 -- 172.21.15.3:0/2479891872 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238004b20 con 0x558f3ffbba30 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.626+0000 7fb2267fc640 1 -- 172.21.15.3:0/2479891872 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb238009d20 con 0x558f3ffbba30 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.626+0000 7fb2267fc640 1 -- 172.21.15.3:0/2479891872 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb238005420 con 0x558f3ffbba30 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 -- 172.21.15.3:0/2479891872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbba30 msgr2=0x558f3ffbbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbba30 0x558f3ffbbe10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb238011cc0 tx=0x7fb23800f790 comp rx=0 tx=0).stop 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 -- 172.21.15.3:0/2479891872 shutdown_connections 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffc4f60 0x558f3ffc73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 --2- 172.21.15.3:0/2479891872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbba30 0x558f3ffbbe10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 -- 172.21.15.3:0/2479891872 >> 172.21.15.3:0/2479891872 conn(0x558f3ffb6100 msgr2=0x558f3ffb64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 -- 172.21.15.3:0/2479891872 shutdown_connections 2024-02-09T23:00:51.636 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 -- 172.21.15.3:0/2479891872 wait complete. 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 -- start start 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.627+0000 7fb24938a740 1 Processor -- start 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffc4f60 0x558f3ffc29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ff23030 con 0x558f3ffbba30 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3fbf6430 con 0x558f3ffbc3e0 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3fbb2e00 con 0x558f3ffc4f60 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 10 client.0 _ll_register_callbacks cb 0x558f3ff18030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc1f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58338/0 (socket says 172.21.15.3:58338) 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 -- 172.21.15.3:0/2310244420 learned_addr learned my addr 172.21.15.3:0/2310244420 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.637 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb244d5f640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffc4f60 0x558f3ffc29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 -- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffc4f60 msgr2=0x558f3ffc29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffc4f60 0x558f3ffc29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 -- 172.21.15.3:0/2310244420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbc3e0 msgr2=0x558f3ffc2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 -- 172.21.15.3:0/2310244420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb238019040 con 0x558f3ffbba30 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc1f50 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fb2300bca60 tx=0x7fb23002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6db6c7603f17b0c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2310244420 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300bcad0 con 0x558f3ffbba30 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2310244420 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb2300c0c20 con 0x558f3ffbba30 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.628+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f3fc227e0 con 0x558f3ffbba30 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2310244420 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2300c5270 con 0x558f3ffbba30 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 10 client.19002 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x558f3ff216b0 con 0x558f3ffbba30 2024-02-09T23:00:51.638 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2310244420 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558f3fc227e0 con 0x558f3ffbba30 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2310244420 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb23003c070 con 0x558f3ffbba30 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 10 client.19002 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 10 client.19002 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 20 client.19002 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f3ffd5080 0x558f3ffd74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 4 client.19002 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f3fc5f720 con 0x558f3ffd5080 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.629+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f3ffd5080 0x558f3ffd74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f3ffd5080 0x558f3ffd74a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f013b72f7c740aa0 server_cookie=aa70e33988ead724 in_seq=0 out_seq=0 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb21f7fe640 10 client.19002 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2310244420 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558f3fc5f720 con 0x558f3ffd5080 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb21f7fe640 10 client.19002 handle_command_reply: tid=0 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb24938a740 1 client.19002 shutdown 2024-02-09T23:00:51.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 msgr2=0x558f3ffc1f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb24938a740 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc1f50 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fb2300bca60 tx=0x7fb23002bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.630+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 shutdown_connections 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f3ffd5080 0x558f3ffd74a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffc4f60 0x558f3ffc29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 --2- 172.21.15.3:0/2310244420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc1f50 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 >> 172.21.15.3:0/2310244420 conn(0x558f3ffb6100 msgr2=0x558f3ffb7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb21f7fe640 0 client.19002 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 shutdown_connections 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 1 -- 172.21.15.3:0/2310244420 wait complete. 2024-02-09T23:00:51.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.631+0000 7fb24938a740 20 client.19002 trim_cache size 0 max 16384 2024-02-09T23:00:51.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 Processor -- start 2024-02-09T23:00:51.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 -- start start 2024-02-09T23:00:51.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc27f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f3ffdbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3fea8130 con 0x558f3ffbba30 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3fc227e0 con 0x558f3ffc4f60 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb24938a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ffb1870 con 0x558f3ffbc3e0 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58348/0 (socket says 172.21.15.3:58348) 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb23ffff640 1 -- 172.21.15.3:0/4271984578 learned_addr learned my addr 172.21.15.3:0/4271984578 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.643+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb23ffff640 1 -- 172.21.15.3:0/4271984578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 msgr2=0x558f3ffc27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb23ffff640 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb23ffff640 1 -- 172.21.15.3:0/4271984578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 msgr2=0x558f3ffdbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb244d5f640 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f3ffdbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb23ffff640 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f3ffdbde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb23ffff640 1 -- 172.21.15.3:0/4271984578 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f3febdc90 con 0x558f3ffbba30 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb23ffff640 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc4680 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fb2300d4040 tx=0x7fb2300bafe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b53f4811462e90aa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb225ffb640 1 -- 172.21.15.3:0/4271984578 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300bc430 con 0x558f3ffbba30 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb225ffb640 1 -- 172.21.15.3:0/4271984578 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2300c0c20 con 0x558f3ffbba30 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.644+0000 7fb225ffb640 1 -- 172.21.15.3:0/4271984578 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300bbf20 con 0x558f3ffbba30 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- 172.21.15.3:0/4271984578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 msgr2=0x558f3ffc4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc4680 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fb2300d4040 tx=0x7fb2300bafe0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- 172.21.15.3:0/4271984578 shutdown_connections 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f3ffdbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffc27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- 172.21.15.3:0/4271984578 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc4680 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- 172.21.15.3:0/4271984578 >> 172.21.15.3:0/4271984578 conn(0x558f3ffb6100 msgr2=0x558f3ffbd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- 172.21.15.3:0/4271984578 shutdown_connections 2024-02-09T23:00:51.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- 172.21.15.3:0/4271984578 wait complete. 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- start start 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 Processor -- start 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffca450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f4002a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ffcabb0 con 0x558f3ffbba30 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f3ffcad20 con 0x558f3ffc4f60 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.645+0000 7fb24938a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f4002aaa0 con 0x558f3ffbc3e0 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb24938a740 10 client.0 _ll_register_callbacks cb 0x7fb238009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb244d5f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f4002a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc9f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58360/0 (socket says 172.21.15.3:58360) 2024-02-09T23:00:51.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23f7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffca450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 -- 172.21.15.3:0/2443456768 learned_addr learned my addr 172.21.15.3:0/2443456768 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 -- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 msgr2=0x558f3ffca450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffca450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 -- 172.21.15.3:0/2443456768 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 msgr2=0x558f4002a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f4002a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 -- 172.21.15.3:0/2443456768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb238019040 con 0x558f3ffbba30 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb23ffff640 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc9f10 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb230009320 tx=0x7fb230009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef032c4c36997945 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2443456768 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2300bc430 con 0x558f3ffbba30 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2443456768 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb2300bcad0 con 0x558f3ffbba30 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb238009d20 con 0x558f3ffbba30 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2443456768 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2300c4fe0 con 0x558f3ffbba30 2024-02-09T23:00:51.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb24938a740 10 client.19008 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.646+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x558f3fee3a70 con 0x558f3ffbba30 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2443456768 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb238009d20 con 0x558f3ffbba30 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2443456768 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb23003c070 con 0x558f3ffbba30 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 10 client.19008 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 10 client.19008 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 20 client.19008 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f4002bd60 0x558f4002e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 4 client.19008 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558f3fcae890 con 0x558f4002bd60 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f4002bd60 0x558f4002e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb21f7fe640 10 client.19008 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb23f7fe640 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f4002bd60 0x558f4002e150 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=801c2d2ef34629e4 server_cookie=d41daf1781310d5c in_seq=0 out_seq=0 2024-02-09T23:00:51.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb21f7fe640 1 -- 172.21.15.3:0/2443456768 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558f3fcae890 con 0x558f4002bd60 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb21f7fe640 10 client.19008 handle_command_reply: tid=0 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.647+0000 7fb24938a740 1 client.19008 shutdown 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 msgr2=0x558f3ffc9f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc9f10 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7fb230009320 tx=0x7fb230009260 comp rx=0 tx=0).stop 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 shutdown_connections 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x558f4002bd60 0x558f4002e150 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f3ffc4f60 0x558f4002a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f3ffbc3e0 0x558f3ffca450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb21f7fe640 0 client.19008 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 --2- 172.21.15.3:0/2443456768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f3ffbba30 0x558f3ffc9f10 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 >> 172.21.15.3:0/2443456768 conn(0x558f3ffb6100 msgr2=0x558f3ffdbde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 shutdown_connections 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 1 -- 172.21.15.3:0/2443456768 wait complete. 2024-02-09T23:00:51.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb24938a740 20 client.19008 trim_cache size 0 max 16384 2024-02-09T23:00:51.658 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2280a2230 msgr2=0x7fb2280a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2280a2230 0x7fb2280a46f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb23800fc90 tx=0x7fb238018000 comp rx=0 tx=0).stop 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 msgr2=0x7fb240100880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 0x7fb240100880 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fb230009230 tx=0x7fb230004ab0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 shutdown_connections 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb2280a2230 0x7fb2280a46f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb240106200 0x7fb240101300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb240100060 0x7fb240100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 --2- 172.21.15.3:0/2446330111 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb2400ff6a0 0x7fb240100880 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 >> 172.21.15.3:0/2446330111 conn(0x7fb240076630 msgr2=0x7fb2401042d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 shutdown_connections 2024-02-09T23:00:51.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.648+0000 7fb2467c2640 1 -- 172.21.15.3:0/2446330111 wait complete. 2024-02-09T23:00:51.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 302512 2.98 MB/sec execute 405 sec latency 12.189 ms 2024-02-09T23:00:51.662 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:51.662 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:00:51.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/346428627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:00:51.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/346428627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 -- 172.21.15.3:0/746854699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac105410 msgr2=0x7f0eac10cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/746854699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac105410 0x7f0eac10cfe0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4009cd0 tx=0x7f0ea402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 -- 172.21.15.3:0/746854699 shutdown_connections 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/746854699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac105410 0x7f0eac10cfe0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/746854699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100cf0 0x7f0eac104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/746854699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0eac100340 0x7f0eac100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 -- 172.21.15.3:0/746854699 >> 172.21.15.3:0/746854699 conn(0x7f0eac0772e0 msgr2=0x7f0eac0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 -- 172.21.15.3:0/746854699 shutdown_connections 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 -- 172.21.15.3:0/746854699 wait complete. 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.773+0000 7f0eb21c7640 1 Processor -- start 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 -- start start 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100340 0x7f0eac198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0eac105410 0x7f0eac198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 0x7f0eac1a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1a3df0 con 0x7f0eac1991a0 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1a3f60 con 0x7f0eac105410 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eb21c7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eac1a4120 con 0x7f0eac100340 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 0x7f0eac1a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 0x7f0eac1a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58380/0 (socket says 172.21.15.3:58380) 2024-02-09T23:00:51.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 -- 172.21.15.3:0/4063672681 learned_addr learned my addr 172.21.15.3:0/4063672681 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eab7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100340 0x7f0eac198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0eac105410 0x7f0eac198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 -- 172.21.15.3:0/4063672681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100340 msgr2=0x7f0eac198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100340 0x7f0eac198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 -- 172.21.15.3:0/4063672681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0eac105410 msgr2=0x7f0eac198c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0eac105410 0x7f0eac198c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 -- 172.21.15.3:0/4063672681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea4009900 con 0x7f0eac1991a0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100340 0x7f0eac198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.774+0000 7f0eabfff640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 0x7f0eac1a37a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4004af0 tx=0x7f0ea4032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.775+0000 7f0ea8ff9640 1 -- 172.21.15.3:0/4063672681 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40328b0 con 0x7f0eac1991a0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.775+0000 7f0ea8ff9640 1 -- 172.21.15.3:0/4063672681 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea4003e50 con 0x7f0eac1991a0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.775+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eac1a43a0 con 0x7f0eac1991a0 2024-02-09T23:00:51.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.775+0000 7f0ea8ff9640 1 -- 172.21.15.3:0/4063672681 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea4004740 con 0x7f0eac1991a0 2024-02-09T23:00:51.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.775+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac1a4850 con 0x7f0eac1991a0 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.777+0000 7f0ea8ff9640 1 -- 172.21.15.3:0/4063672681 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0ea400a040 con 0x7f0eac1991a0 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.777+0000 7f0eb4d97740 1 Processor -- start 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.777+0000 7f0ea8ff9640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e940a2230 0x7f0e940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.777+0000 7f0eb4d97740 1 -- start start 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.777+0000 7f0ea8ff9640 1 -- 172.21.15.3:0/4063672681 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0ea40f0aa0 con 0x7f0eac1991a0 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.777+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e940a2230 0x7f0e940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa0fce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa105990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1083b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa0b8020 con 0x5565fa105f60 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa06b320 con 0x5565fa0fca30 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa068850 con 0x5565fa0fd3e0 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eab7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa0fce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa105990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1083b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa105990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41236/0 (socket says 172.21.15.3:41236) 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eab7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa0fce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:51064/0 (socket says 172.21.15.3:51064) 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eaaffd640 1 -- 172.21.15.3:0/2016189474 learned_addr learned my addr 172.21.15.3:0/2016189474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 -- 172.21.15.3:0/2016189474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 msgr2=0x5565fa105990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa105990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 -- 172.21.15.3:0/2016189474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 msgr2=0x5565fa0fce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa0fce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e940a2230 0x7f0e940a46f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0011550 tx=0x7f0ea0011950 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 -- 172.21.15.3:0/2016189474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565fa026210 con 0x5565fa105f60 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0eabfff640 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1083b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40ee300 tx=0x7f0ea40bd250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7660640f8b47f39 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0e91ffb640 1 -- 172.21.15.3:0/2016189474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40bd650 con 0x5565fa105f60 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0e91ffb640 1 -- 172.21.15.3:0/2016189474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea40c16f0 con 0x5565fa105f60 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.778+0000 7f0e91ffb640 1 -- 172.21.15.3:0/2016189474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40b9cc0 con 0x5565fa105f60 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2016189474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 msgr2=0x5565fa1083b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1083b0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40ee300 tx=0x7f0ea40bd250 comp rx=0 tx=0).stop 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2016189474 shutdown_connections 2024-02-09T23:00:51.789 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1083b0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa105990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2016189474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa0fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2016189474 >> 172.21.15.3:0/2016189474 conn(0x5565fa0f7100 msgr2=0x5565fa0f74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2016189474 shutdown_connections 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2016189474 wait complete. 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.779+0000 7f0eb4d97740 1 -- start start 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 Processor -- start 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 0x5565fa102f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa1034d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa105f60 0x5565fa103a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa064030 con 0x5565fa0fca30 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9d37430 con 0x5565fa105f60 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9cf3e00 con 0x5565fa0fd3e0 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eb4d97740 10 client.0 _ll_register_callbacks cb 0x5565fa059030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 0x5565fa102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 0x5565fa102f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58392/0 (socket says 172.21.15.3:58392) 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 -- 172.21.15.3:0/1233455854 learned_addr learned my addr 172.21.15.3:0/1233455854 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa1034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eabfff640 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa105f60 0x5565fa103a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 -- 172.21.15.3:0/1233455854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 msgr2=0x5565fa1034d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa1034d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 -- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa105f60 msgr2=0x5565fa103a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa105f60 0x5565fa103a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 -- 172.21.15.3:0/1233455854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eac1a4850 con 0x5565fa0fca30 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 0x5565fa102f90 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c012d20 tx=0x7f0e9c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78eb3517505431f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.780+0000 7f0e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0e8affd640 1 -- 172.21.15.3:0/1233455854 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e9c017070 con 0x5565fa0fca30 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0e8affd640 1 -- 172.21.15.3:0/1233455854 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0e9c0120d0 con 0x5565fa0fca30 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5565f9d637e0 con 0x5565fa0fca30 2024-02-09T23:00:51.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0e8affd640 1 -- 172.21.15.3:0/1233455854 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e9c012290 con 0x5565fa0fca30 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 10 client.19020 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5565fa0626b0 con 0x5565fa0fca30 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0e8affd640 1 -- 172.21.15.3:0/1233455854 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e9c028080 con 0x5565fa0fca30 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0e8affd640 1 -- 172.21.15.3:0/1233455854 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0e9c022030 con 0x5565fa0fca30 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 10 client.19020 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 10 client.19020 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 20 client.19020 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa118800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 4 client.19020 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5565f9da0720 con 0x5565fa1163e0 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.781+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa118800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.782+0000 7f0e8affd640 10 client.19020 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:51.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.782+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa118800 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bfe6b18b858cc881 server_cookie=5af46ec55f58797c in_seq=0 out_seq=0 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.782+0000 7f0e8affd640 1 -- 172.21.15.3:0/1233455854 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5565f9da0720 con 0x5565fa1163e0 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.782+0000 7f0e8affd640 10 client.19020 handle_command_reply: tid=0 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.782+0000 7f0eb4d97740 1 client.19020 shutdown 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 msgr2=0x5565fa102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 0x5565fa102f90 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f0e9c012d20 tx=0x7f0e9c00f790 comp rx=0 tx=0).stop 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 shutdown_connections 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa118800 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa105f60 0x5565fa103a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0e8affd640 0 client.19020 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa1034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/1233455854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fca30 0x5565fa102f90 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 >> 172.21.15.3:0/1233455854 conn(0x5565fa0f7100 msgr2=0x5565fa0f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 shutdown_connections 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 1 -- 172.21.15.3:0/1233455854 wait complete. 2024-02-09T23:00:51.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.783+0000 7f0eb4d97740 20 client.19020 trim_cache size 0 max 16384 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 Processor -- start 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 -- start start 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa1056c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa103830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1001f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565f9fe9130 con 0x5565fa105f60 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa11da10 con 0x5565fa0fca30 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa1014e0 con 0x5565fa0fd3e0 2024-02-09T23:00:51.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1001f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1001f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58396/0 (socket says 172.21.15.3:58396) 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 -- 172.21.15.3:0/851574662 learned_addr learned my addr 172.21.15.3:0/851574662 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa1056c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 -- 172.21.15.3:0/851574662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 msgr2=0x5565fa103830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa103830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 -- 172.21.15.3:0/851574662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 msgr2=0x5565fa1056c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa1056c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 -- 172.21.15.3:0/851574662 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5565f9ffec90 con 0x5565fa105f60 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0eabfff640 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1001f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40ee7f0 tx=0x7f0ea40bd2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=259e45c15d7296ac server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0e917fa640 1 -- 172.21.15.3:0/851574662 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40bd650 con 0x5565fa105f60 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0e917fa640 1 -- 172.21.15.3:0/851574662 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea40c5d40 con 0x5565fa105f60 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.796+0000 7f0e917fa640 1 -- 172.21.15.3:0/851574662 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea40ee3e0 con 0x5565fa105f60 2024-02-09T23:00:51.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 -- 172.21.15.3:0/851574662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 msgr2=0x5565fa1001f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1001f0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f0ea40ee7f0 tx=0x7f0ea40bd2c0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 -- 172.21.15.3:0/851574662 shutdown_connections 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa105f60 0x5565fa1001f0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa0fd3e0 0x5565fa103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/851574662 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa1056c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 -- 172.21.15.3:0/851574662 >> 172.21.15.3:0/851574662 conn(0x5565fa0f7100 msgr2=0x5565fa0fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 -- 172.21.15.3:0/851574662 shutdown_connections 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.797+0000 7f0eb4d97740 1 -- 172.21.15.3:0/851574662 wait complete. 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 -- start start 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 Processor -- start 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa10bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 0x5565fa10c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa105f60 0x5565fa170d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa10c980 con 0x5565fa0fd3e0 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa171240 con 0x5565fa0fca30 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5565fa171380 con 0x5565fa105f60 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eb4d97740 10 client.0 _ll_register_callbacks cb 0x7f0ea40048a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 0x5565fa10c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 0x5565fa10c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58410/0 (socket says 172.21.15.3:58410) 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 -- 172.21.15.3:0/2358004040 learned_addr learned my addr 172.21.15.3:0/2358004040 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eabfff640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa105f60 0x5565fa170d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa10bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 -- 172.21.15.3:0/2358004040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa105f60 msgr2=0x5565fa170d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa105f60 0x5565fa170d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 -- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 msgr2=0x5565fa10bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa10bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.798+0000 7f0eaaffd640 1 -- 172.21.15.3:0/2358004040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5565fa026210 con 0x5565fa0fd3e0 2024-02-09T23:00:51.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eaaffd640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 0x5565fa10c080 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x5565fa0fc500 tx=0x7f0ea0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b6bcf34b833791b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0e8affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0e8affd640 1 -- 172.21.15.3:0/2358004040 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ea0016070 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0e8affd640 1 -- 172.21.15.3:0/2358004040 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0ea000fc90 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ea40c16f0 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0e8affd640 1 -- 172.21.15.3:0/2358004040 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ea000fe50 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 10 client.19026 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5565fa024a70 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0e8affd640 1 -- 172.21.15.3:0/2358004040 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ea40c16f0 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0e8affd640 1 -- 172.21.15.3:0/2358004040 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f0ea00047a0 con 0x5565fa0fd3e0 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 10 client.19026 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 10 client.19026 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 20 client.19026 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa174c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 4 client.19026 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.799+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5565f9def890 con 0x5565fa1163e0 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa174c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0e8affd640 10 client.19026 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eab7fe640 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa174c40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=be5073acae75c76a server_cookie=98f64875be47bc8e in_seq=0 out_seq=0 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0e8affd640 1 -- 172.21.15.3:0/2358004040 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5565f9def890 con 0x5565fa1163e0 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0e8affd640 10 client.19026 handle_command_reply: tid=0 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 client.19026 shutdown 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 msgr2=0x5565fa10c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 0x5565fa10c080 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x5565fa0fc500 tx=0x7f0ea0004730 comp rx=0 tx=0).stop 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 shutdown_connections 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5565fa1163e0 0x5565fa174c40 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5565fa105f60 0x5565fa170d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.800+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5565fa0fd3e0 0x5565fa10c080 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb4d97740 1 --2- 172.21.15.3:0/2358004040 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5565fa0fca30 0x5565fa10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0e8affd640 0 client.19026 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 >> 172.21.15.3:0/2358004040 conn(0x5565fa0f7100 msgr2=0x5565fa0f59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 shutdown_connections 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb4d97740 1 -- 172.21.15.3:0/2358004040 wait complete. 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb4d97740 20 client.19026 trim_cache size 0 max 16384 2024-02-09T23:00:51.810 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e940a2230 msgr2=0x7f0e940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e940a2230 0x7f0e940a46f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f0ea0011550 tx=0x7f0ea0011950 comp rx=0 tx=0).stop 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 msgr2=0x7f0eac1a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 0x7f0eac1a37a0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7f0ea4004af0 tx=0x7f0ea4032570 comp rx=0 tx=0).stop 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 shutdown_connections 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e940a2230 0x7f0e940a46f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0eac105410 0x7f0eac198c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0eac1991a0 0x7f0eac1a37a0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 --2- 172.21.15.3:0/4063672681 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0eac100340 0x7f0eac198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 >> 172.21.15.3:0/4063672681 conn(0x7f0eac0772e0 msgr2=0x7f0eac0fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 shutdown_connections 2024-02-09T23:00:51.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.801+0000 7f0eb21c7640 1 -- 172.21.15.3:0/4063672681 wait complete. 2024-02-09T23:00:51.815 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:00:51.815 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:00:51.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 -- 172.21.15.3:0/2088914509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100ce0 msgr2=0x7fb574104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/2088914509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100ce0 0x7fb574104d20 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fb564009930 tx=0x7fb56402bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:51.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 -- 172.21.15.3:0/2088914509 shutdown_connections 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/2088914509 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb574105400 0x7fb57410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/2088914509 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100ce0 0x7fb574104d20 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/2088914509 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574100330 0x7fb574100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 -- 172.21.15.3:0/2088914509 >> 172.21.15.3:0/2088914509 conn(0x7fb574077250 msgr2=0x7fb574077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 -- 172.21.15.3:0/2088914509 shutdown_connections 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.920+0000 7fb57c4b4640 1 -- 172.21.15.3:0/2088914509 wait complete. 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 Processor -- start 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 -- start start 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 0x7fb574101dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574105400 0x7fb574102300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb574102840 0x7fb5741a5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5741a6040 con 0x7fb574100330 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5741a61b0 con 0x7fb574105400 2024-02-09T23:00:51.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57c4b4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5741a6370 con 0x7fb574102840 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 0x7fb574101dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 0x7fb574101dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58420/0 (socket says 172.21.15.3:58420) 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 -- 172.21.15.3:0/69471379 learned_addr learned my addr 172.21.15.3:0/69471379 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57aa51640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb574102840 0x7fb5741a5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb579a4f640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574105400 0x7fb574102300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 -- 172.21.15.3:0/69471379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb574102840 msgr2=0x7fb5741a5ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb574102840 0x7fb5741a5ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 -- 172.21.15.3:0/69471379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574105400 msgr2=0x7fb574102300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574105400 0x7fb574102300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb57a250640 1 -- 172.21.15.3:0/69471379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564009560 con 0x7fb574100330 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.921+0000 7fb579a4f640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574105400 0x7fb574102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.922+0000 7fb57a250640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 0x7fb574101dc0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e9f0 tx=0x7fb57000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.922+0000 7fb56b7fe640 1 -- 172.21.15.3:0/69471379 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570009700 con 0x7fb574100330 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.922+0000 7fb56b7fe640 1 -- 172.21.15.3:0/69471379 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb57000cd80 con 0x7fb574100330 2024-02-09T23:00:51.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.922+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5741a6650 con 0x7fb574100330 2024-02-09T23:00:51.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.922+0000 7fb56b7fe640 1 -- 172.21.15.3:0/69471379 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570018920 con 0x7fb574100330 2024-02-09T23:00:51.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.922+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5741a6b30 con 0x7fb574100330 2024-02-09T23:00:51.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.924+0000 7fb56b7fe640 1 -- 172.21.15.3:0/69471379 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb570018a80 con 0x7fb574100330 2024-02-09T23:00:51.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.924+0000 7fb57f085740 1 Processor -- start 2024-02-09T23:00:51.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 -- start start 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb56b7fe640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5600a2230 0x7fb5600a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d8ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d93990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d963b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0d46020 con 0x55a9a0d8aa30 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0cf9320 con 0x55a9a0d93f60 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57f085740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0cf6850 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d8ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d8ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58430/0 (socket says 172.21.15.3:58430) 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 -- 172.21.15.3:0/3460648262 learned_addr learned my addr 172.21.15.3:0/3460648262 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb579a4f640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d93990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57aa51640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5600a2230 0x7fb5600a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57aa51640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb56b7fe640 1 -- 172.21.15.3:0/69471379 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb570014070 con 0x7fb574100330 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 -- 172.21.15.3:0/3460648262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 msgr2=0x55a9a0d93990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d93990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 -- 172.21.15.3:0/3460648262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 msgr2=0x55a9a0d963b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.925+0000 7fb57a250640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d963b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb57a250640 1 -- 172.21.15.3:0/3460648262 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9a0cb4210 con 0x55a9a0d8aa30 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb57aa51640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb579a4f640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d93990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb57a250640 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d8ae10 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e9c0 tx=0x7fb57001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a256847132ed2c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb57aa51640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5600a2230 0x7fb5600a46f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb56c005fd0 tx=0x7fb56c009b70 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb568bfe640 1 -- 172.21.15.3:0/3460648262 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700ad740 con 0x55a9a0d8aa30 2024-02-09T23:00:51.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb568bfe640 1 -- 172.21.15.3:0/3460648262 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5700ad560 con 0x55a9a0d8aa30 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.926+0000 7fb568bfe640 1 -- 172.21.15.3:0/3460648262 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb570010b80 con 0x55a9a0d8aa30 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 -- 172.21.15.3:0/3460648262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 msgr2=0x55a9a0d8ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d8ae10 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e9c0 tx=0x7fb57001f000 comp rx=0 tx=0).stop 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 -- 172.21.15.3:0/3460648262 shutdown_connections 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d963b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d93990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- 172.21.15.3:0/3460648262 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d8ae10 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 -- 172.21.15.3:0/3460648262 >> 172.21.15.3:0/3460648262 conn(0x55a9a0d85100 msgr2=0x55a9a0d854e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 -- 172.21.15.3:0/3460648262 shutdown_connections 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 -- 172.21.15.3:0/3460648262 wait complete. 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 -- start start 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 Processor -- start 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d90eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d913f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.927+0000 7fb57f085740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d93f60 0x55a9a0d91930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0cf2030 con 0x55a9a0d8aa30 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a09c5430 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0981e00 con 0x55a9a0d93f60 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d90eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 10 client.0 _ll_register_callbacks cb 0x55a9a0ce7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d90eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58434/0 (socket says 172.21.15.3:58434) 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 -- 172.21.15.3:0/2029217220 learned_addr learned my addr 172.21.15.3:0/2029217220 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb579a4f640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d913f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57aa51640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d93f60 0x55a9a0d91930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 -- 172.21.15.3:0/2029217220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d93f60 msgr2=0x55a9a0d91930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d93f60 0x55a9a0d91930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 -- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8b3e0 msgr2=0x55a9a0d913f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d913f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 -- 172.21.15.3:0/2029217220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5741a6b30 con 0x55a9a0d8aa30 2024-02-09T23:00:51.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57a250640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d90eb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb5700978c0 tx=0x7fb57009aef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe077fdd449d253c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb55dffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb55dffb640 1 -- 172.21.15.3:0/2029217220 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5700ad700 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb55dffb640 1 -- 172.21.15.3:0/2029217220 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb57009b5f0 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a9a09f17e0 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb55dffb640 1 -- 172.21.15.3:0/2029217220 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5700aef40 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 10 client.19038 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.928+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55a9a0cf06b0 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb55dffb640 1 -- 172.21.15.3:0/2029217220 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5700b7020 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb55dffb640 1 -- 172.21.15.3:0/2029217220 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5700ce200 con 0x55a9a0d8aa30 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb57f085740 10 client.19038 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb57f085740 10 client.19038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb57f085740 20 client.19038 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb57f085740 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0da6480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb57f085740 4 client.19038 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:51.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a9a0a2e720 con 0x55a9a0da4090 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb579a4f640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0da6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb579a4f640 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0da6480 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e637a12e83932139 server_cookie=1ee0cf9cf739bbb5 in_seq=0 out_seq=0 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.929+0000 7fb55dffb640 10 client.19038 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb55dffb640 1 -- 172.21.15.3:0/2029217220 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a9a0a2e720 con 0x55a9a0da4090 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb55dffb640 10 client.19038 handle_command_reply: tid=0 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 client.19038 shutdown 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 msgr2=0x55a9a0d90eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d90eb0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fb5700978c0 tx=0x7fb57009aef0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 shutdown_connections 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0da6480 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d913f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb55dffb640 0 client.19038 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:51.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d93f60 0x55a9a0d91930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 --2- 172.21.15.3:0/2029217220 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d90eb0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 >> 172.21.15.3:0/2029217220 conn(0x55a9a0d85100 msgr2=0x55a9a0d86e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 shutdown_connections 2024-02-09T23:00:51.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.930+0000 7fb57f085740 1 -- 172.21.15.3:0/2029217220 wait complete. 2024-02-09T23:00:51.941 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.931+0000 7fb57f085740 20 client.19038 trim_cache size 0 max 16384 2024-02-09T23:00:51.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.943+0000 7fb57f085740 1 Processor -- start 2024-02-09T23:00:51.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.943+0000 7fb57f085740 1 -- start start 2024-02-09T23:00:51.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57f085740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d93570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57f085740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d91750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57f085740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d8f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57f085740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0c77130 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57f085740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0d8caa0 con 0x55a9a0d93f60 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57f085740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0dab4d0 con 0x55a9a0d8aa30 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d91750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d91750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58448/0 (socket says 172.21.15.3:58448) 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 -- 172.21.15.3:0/4268617227 learned_addr learned my addr 172.21.15.3:0/4268617227 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57a250640 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d93570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb57aa51640 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d8f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 -- 172.21.15.3:0/4268617227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8aa30 msgr2=0x55a9a0d93570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d93570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 -- 172.21.15.3:0/4268617227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 msgr2=0x55a9a0d8f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d8f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 -- 172.21.15.3:0/4268617227 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a9a0c8cc90 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb579a4f640 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d91750 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb564005940 tx=0x7fb564009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27aa17660124a178 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb55ffff640 1 -- 172.21.15.3:0/4268617227 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb564045040 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb55ffff640 1 -- 172.21.15.3:0/4268617227 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb564048be0 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.944+0000 7fb55ffff640 1 -- 172.21.15.3:0/4268617227 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56402cce0 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 -- 172.21.15.3:0/4268617227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 msgr2=0x55a9a0d91750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d91750 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fb564005940 tx=0x7fb564009260 comp rx=0 tx=0).stop 2024-02-09T23:00:51.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 -- 172.21.15.3:0/4268617227 shutdown_connections 2024-02-09T23:00:51.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d93f60 0x55a9a0d8f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d91750 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 --2- 172.21.15.3:0/4268617227 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d93570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 -- 172.21.15.3:0/4268617227 >> 172.21.15.3:0/4268617227 conn(0x55a9a0d85100 msgr2=0x55a9a0daadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 -- 172.21.15.3:0/4268617227 shutdown_connections 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.945+0000 7fb57f085740 1 -- 172.21.15.3:0/4268617227 wait complete. 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 -- start start 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 Processor -- start 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d99b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d9a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d93f60 0x55a9a0d9a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0dfa6c0 con 0x55a9a0d93f60 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0dfa830 con 0x55a9a0d8aa30 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a9a0dfa9f0 con 0x55a9a0d8b3e0 2024-02-09T23:00:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57f085740 10 client.0 _ll_register_callbacks cb 0x7fb57000af50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb579a4f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d9a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57aa51640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d93f60 0x55a9a0d9a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d99b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb579a4f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d9a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41308/0 (socket says 172.21.15.3:41308) 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d99b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:51132/0 (socket says 172.21.15.3:51132) 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb579a4f640 1 -- 172.21.15.3:0/3939348460 learned_addr learned my addr 172.21.15.3:0/3939348460 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 -- 172.21.15.3:0/3939348460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 msgr2=0x55a9a0d9a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d9a0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 -- 172.21.15.3:0/3939348460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d93f60 msgr2=0x55a9a0d9a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d93f60 0x55a9a0d9a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57a250640 1 -- 172.21.15.3:0/3939348460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb564030070 con 0x55a9a0d8aa30 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb57aa51640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d93f60 0x55a9a0d9a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.946+0000 7fb579a4f640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d9a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57a250640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d99b60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e9c0 tx=0x7fb570010e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af79cf5a0e53bf33 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb55dffb640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb55dffb640 1 -- 172.21.15.3:0/3939348460 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57009a850 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb55dffb640 1 -- 172.21.15.3:0/3939348460 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb5700b8080 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb5700ad560 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb55dffb640 1 -- 172.21.15.3:0/3939348460 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5700af140 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 10 client.27926 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55a9a0cb2a70 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb55dffb640 1 -- 172.21.15.3:0/3939348460 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb570089020 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb55dffb640 1 -- 172.21.15.3:0/3939348460 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb5700cbf60 con 0x55a9a0d8aa30 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 10 client.27926 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 10 client.27926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 20 client.27926 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0dfdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 4 client.27926 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a9a0a7d890 con 0x55a9a0da4090 2024-02-09T23:00:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.947+0000 7fb579a4f640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0dfdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb579a4f640 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0dfdd00 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ede953542b19d874 server_cookie=c8f9394ca0601440 in_seq=0 out_seq=0 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb55dffb640 10 client.27926 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb55dffb640 1 -- 172.21.15.3:0/3939348460 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a9a0a7d890 con 0x55a9a0da4090 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb55dffb640 10 client.27926 handle_command_reply: tid=0 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 client.27926 shutdown 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 msgr2=0x55a9a0d99b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d99b60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e9c0 tx=0x7fb570010e50 comp rx=0 tx=0).stop 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 shutdown_connections 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a9a0da4090 0x55a9a0dfdd00 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a9a0d93f60 0x55a9a0d9a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a9a0d8b3e0 0x55a9a0d9a0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb55dffb640 0 client.27926 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 --2- 172.21.15.3:0/3939348460 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a9a0d8aa30 0x55a9a0d99b60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 >> 172.21.15.3:0/3939348460 conn(0x55a9a0d85100 msgr2=0x55a9a0d8f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 shutdown_connections 2024-02-09T23:00:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 1 -- 172.21.15.3:0/3939348460 wait complete. 2024-02-09T23:00:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.948+0000 7fb57f085740 20 client.27926 trim_cache size 0 max 16384 2024-02-09T23:00:51.958 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5600a2230 msgr2=0x7fb5600a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5600a2230 0x7fb5600a46f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb56c005fd0 tx=0x7fb56c009b70 comp rx=0 tx=0).stop 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 msgr2=0x7fb574101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 0x7fb574101dc0 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7fb57000e9f0 tx=0x7fb57000eeb0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 shutdown_connections 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5600a2230 0x7fb5600a46f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb574105400 0x7fb574102300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb574102840 0x7fb5741a5ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 --2- 172.21.15.3:0/69471379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb574100330 0x7fb574101dc0 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 >> 172.21.15.3:0/69471379 conn(0x7fb574077250 msgr2=0x7fb574069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 shutdown_connections 2024-02-09T23:00:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:51.949+0000 7fb57c4b4640 1 -- 172.21.15.3:0/69471379 wait complete. 2024-02-09T23:00:52.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:52 smithi003 ceph-mon[32678]: pgmap v299: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 267 op/s 2024-02-09T23:00:52.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 303214 2.98 MB/sec execute 406 sec latency 510.138 ms 2024-02-09T23:00:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:52 smithi064 ceph-mon[36638]: pgmap v299: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 267 op/s 2024-02-09T23:00:52.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:52 smithi089 ceph-mon[36665]: pgmap v299: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.6 MiB/s wr, 267 op/s 2024-02-09T23:00:52.963 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:00:52.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:00:52.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 -- 172.21.15.3:0/2837666719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c100310 msgr2=0x7fd76c1006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 --2- 172.21.15.3:0/2837666719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c100310 0x7fd76c1006f0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009930 tx=0x7fd75c02bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 -- 172.21.15.3:0/2837666719 shutdown_connections 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 --2- 172.21.15.3:0/2837666719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd76c1053e0 0x7fd76c10cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 --2- 172.21.15.3:0/2837666719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76c100cc0 0x7fd76c104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 --2- 172.21.15.3:0/2837666719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c100310 0x7fd76c1006f0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 -- 172.21.15.3:0/2837666719 >> 172.21.15.3:0/2837666719 conn(0x7fd76c077230 msgr2=0x7fd76c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 -- 172.21.15.3:0/2837666719 shutdown_connections 2024-02-09T23:00:53.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 -- 172.21.15.3:0/2837666719 wait complete. 2024-02-09T23:00:53.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 Processor -- start 2024-02-09T23:00:53.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.078+0000 7fd772563640 1 -- start start 2024-02-09T23:00:53.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd772563640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76c100310 0x7fd76c10d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd772563640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd76c1053e0 0x7fd76c10ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd772563640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 0x7fd76c1a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd772563640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1a6000 con 0x7fd76c10e310 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd772563640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1a6170 con 0x7fd76c1053e0 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd772563640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76c1a6330 con 0x7fd76c100310 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 0x7fd76c1a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 0x7fd76c1a5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58468/0 (socket says 172.21.15.3:58468) 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 -- 172.21.15.3:0/387533344 learned_addr learned my addr 172.21.15.3:0/387533344 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd76c1053e0 0x7fd76c10ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd76bfff640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76c100310 0x7fd76c10d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 -- 172.21.15.3:0/387533344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76c100310 msgr2=0x7fd76c10d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76c100310 0x7fd76c10d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 -- 172.21.15.3:0/387533344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd76c1053e0 msgr2=0x7fd76c10ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd76c1053e0 0x7fd76c10ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.079+0000 7fd770b00640 1 -- 172.21.15.3:0/387533344 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c009560 con 0x7fd76c10e310 2024-02-09T23:00:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.080+0000 7fd770b00640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 0x7fd76c1a5a30 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fd76400eb10 tx=0x7fd76400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.080+0000 7fd7697fa640 1 -- 172.21.15.3:0/387533344 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764004b20 con 0x7fd76c10e310 2024-02-09T23:00:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.080+0000 7fd7697fa640 1 -- 172.21.15.3:0/387533344 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd764004c80 con 0x7fd76c10e310 2024-02-09T23:00:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.080+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76c1a6610 con 0x7fd76c10e310 2024-02-09T23:00:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.080+0000 7fd7697fa640 1 -- 172.21.15.3:0/387533344 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd764010650 con 0x7fd76c10e310 2024-02-09T23:00:53.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.080+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76c1a6ad0 con 0x7fd76c10e310 2024-02-09T23:00:53.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.082+0000 7fd7697fa640 1 -- 172.21.15.3:0/387533344 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd764005020 con 0x7fd76c10e310 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.082+0000 7fd77520c740 1 Processor -- start 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 -- start start 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787190 0x55a353787570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787ab0 0x55a353790150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353792a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3536f5000 con 0x55a353790690 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3536f2530 con 0x55a353787ab0 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd77520c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3536edd10 con 0x55a353787190 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd7697fa640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7540a2260 0x7fd7540a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353792a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787ab0 0x55a353790150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353792a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58478/0 (socket says 172.21.15.3:58478) 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 -- 172.21.15.3:0/2068886520 learned_addr learned my addr 172.21.15.3:0/2068886520 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787190 0x55a353787570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd770b00640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7540a2260 0x7fd7540a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd7697fa640 1 -- 172.21.15.3:0/387533344 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd764014070 con 0x7fd76c10e310 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 -- 172.21.15.3:0/2068886520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787190 msgr2=0x55a353787570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787190 0x55a353787570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 -- 172.21.15.3:0/2068886520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787ab0 msgr2=0x55a353790150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787ab0 0x55a353790150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.083+0000 7fd76b7fe640 1 -- 172.21.15.3:0/2068886520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a353694210 con 0x55a353790690 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353792a80 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fd7600058c0 tx=0x7fd760005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78c3f338ae1c3d99 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd770b00640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7540a2260 0x7fd7540a4720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd7640cc310 tx=0x7fd76401bb30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd7527fc640 1 -- 172.21.15.3:0/2068886520 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760004040 con 0x55a353790690 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd7527fc640 1 -- 172.21.15.3:0/2068886520 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd760005500 con 0x55a353790690 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd7527fc640 1 -- 172.21.15.3:0/2068886520 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760004c60 con 0x55a353790690 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd77520c740 1 -- 172.21.15.3:0/2068886520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 msgr2=0x55a353792a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.084+0000 7fd77520c740 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353792a80 secure :-1 s=READY pgs=1479 cs=0 l=1 rev1=1 crypto rx=0x7fd7600058c0 tx=0x7fd760005850 comp rx=0 tx=0).stop 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- 172.21.15.3:0/2068886520 shutdown_connections 2024-02-09T23:00:53.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787ab0 0x55a353790150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353792a80 unknown :-1 s=CLOSED pgs=1479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 --2- 172.21.15.3:0/2068886520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787190 0x55a353787570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- 172.21.15.3:0/2068886520 >> 172.21.15.3:0/2068886520 conn(0x55a3537818c0 msgr2=0x55a353781ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- 172.21.15.3:0/2068886520 shutdown_connections 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- 172.21.15.3:0/2068886520 wait complete. 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- start start 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 Processor -- start 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a35378dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a35378e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a353692e60 con 0x55a353790690 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3533c14b0 con 0x55a353787190 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a353673670 con 0x55a353787ab0 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.085+0000 7fd77520c740 10 client.0 _ll_register_callbacks cb 0x55a35368b190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a35378e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a35378dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a35378dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41336/0 (socket says 172.21.15.3:41336) 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 -- 172.21.15.3:0/2186890868 learned_addr learned my addr 172.21.15.3:0/2186890868 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 -- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 msgr2=0x55a35378d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 -- 172.21.15.3:0/2186890868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 msgr2=0x55a35378e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a35378e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 -- 172.21.15.3:0/2186890868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd760019040 con 0x55a353787ab0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd76bfff640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a35378dc00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd75c036040 tx=0x7fd75c02ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec18cebb70f97903 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd74b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd74b7fe640 1 -- 172.21.15.3:0/2186890868 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c02c640 con 0x55a353787ab0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a3533ed860 con 0x55a353787ab0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.086+0000 7fd74b7fe640 1 -- 172.21.15.3:0/2186890868 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd75c004050 con 0x55a353787ab0 2024-02-09T23:00:53.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd74b7fe640 1 -- 172.21.15.3:0/2186890868 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd75c004250 con 0x55a353787ab0 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 10 client.28075 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55a3536ec390 con 0x55a353787ab0 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd74b7fe640 1 -- 172.21.15.3:0/2186890868 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a3533ed860 con 0x55a353787ab0 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd74b7fe640 1 -- 172.21.15.3:0/2186890868 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd75c04f020 con 0x55a353787ab0 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 10 client.28075 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 10 client.28075 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 20 client.28075 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537a0710 0x55a3537a2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 4 client.28075 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a35343e410 con 0x55a3537a0710 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.087+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537a0710 0x55a3537a2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.088+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537a0710 0x55a3537a2b30 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60d064a69fc38e5 server_cookie=2ee226c5c7373aa0 in_seq=0 out_seq=0 2024-02-09T23:00:53.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.088+0000 7fd74b7fe640 10 client.28075 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.088+0000 7fd74b7fe640 1 -- 172.21.15.3:0/2186890868 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a35343e410 con 0x55a3537a0710 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.088+0000 7fd74b7fe640 10 client.28075 handle_command_reply: tid=0 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.088+0000 7fd77520c740 1 client.28075 shutdown 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 msgr2=0x55a35378dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a35378dc00 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd75c036040 tx=0x7fd75c02ca10 comp rx=0 tx=0).stop 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 shutdown_connections 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537a0710 0x55a3537a2b30 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a35378e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd74b7fe640 0 client.28075 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a35378dc00 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 --2- 172.21.15.3:0/2186890868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 >> 172.21.15.3:0/2186890868 conn(0x55a3537818c0 msgr2=0x55a353783610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 shutdown_connections 2024-02-09T23:00:53.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 1 -- 172.21.15.3:0/2186890868 wait complete. 2024-02-09T23:00:53.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.089+0000 7fd77520c740 20 client.28075 trim_cache size 0 max 16384 2024-02-09T23:00:53.109 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.104+0000 7fd77520c740 1 Processor -- start 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.104+0000 7fd77520c740 1 -- start start 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd77520c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd77520c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a353797540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd77520c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353797a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd77520c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a353786280 con 0x55a353790690 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd77520c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a35379d210 con 0x55a353787190 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd77520c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a3537828c0 con 0x55a353787ab0 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353797a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a353797540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353797a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58502/0 (socket says 172.21.15.3:58502) 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76b7fe640 1 -- 172.21.15.3:0/2737509642 learned_addr learned my addr 172.21.15.3:0/2737509642 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.110 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 -- 172.21.15.3:0/2737509642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 msgr2=0x55a35378df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 -- 172.21.15.3:0/2737509642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 msgr2=0x55a353797a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353797a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 -- 172.21.15.3:0/2737509642 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a353688c70 con 0x55a353787ab0 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353797a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.105+0000 7fd76bfff640 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a353797540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55a35378ec90 tx=0x7fd75c02bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ce0e367cd75088 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd751ffb640 1 -- 172.21.15.3:0/2737509642 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c02c5d0 con 0x55a353787ab0 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd751ffb640 1 -- 172.21.15.3:0/2737509642 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd75c003af0 con 0x55a353787ab0 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd751ffb640 1 -- 172.21.15.3:0/2737509642 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c004520 con 0x55a353787ab0 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 -- 172.21.15.3:0/2737509642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 msgr2=0x55a353797540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a353797540 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x55a35378ec90 tx=0x7fd75c02bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 -- 172.21.15.3:0/2737509642 shutdown_connections 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353790690 0x55a353797a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a353797540 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.111 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 --2- 172.21.15.3:0/2737509642 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353787190 0x55a35378df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 -- 172.21.15.3:0/2737509642 >> 172.21.15.3:0/2737509642 conn(0x55a3537818c0 msgr2=0x55a353780090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 -- 172.21.15.3:0/2737509642 shutdown_connections 2024-02-09T23:00:53.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.106+0000 7fd77520c740 1 -- 172.21.15.3:0/2737509642 wait complete. 2024-02-09T23:00:53.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 -- start start 2024-02-09T23:00:53.112 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 Processor -- start 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 0x55a3537f0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a3537f1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353790690 0x55a3537966a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a353796d40 con 0x55a353787190 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a353796eb0 con 0x55a353790690 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a353797070 con 0x55a353787ab0 2024-02-09T23:00:53.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd77520c740 10 client.0 _ll_register_callbacks cb 0x7fd760009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 0x55a3537f0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 0x55a3537f0fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58510/0 (socket says 172.21.15.3:58510) 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 -- 172.21.15.3:0/4238820331 learned_addr learned my addr 172.21.15.3:0/4238820331 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76bfff640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a3537f1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353790690 0x55a3537966a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 -- 172.21.15.3:0/4238820331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 msgr2=0x55a3537f1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a3537f1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 -- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353790690 msgr2=0x55a3537966a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353790690 0x55a3537966a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.107+0000 7fd76b7fe640 1 -- 172.21.15.3:0/4238820331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a353694210 con 0x55a353787190 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd76b7fe640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 0x55a3537f0fc0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fd760009250 tx=0x7fd760005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fbb44c7e0559232 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd74b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd74b7fe640 1 -- 172.21.15.3:0/4238820331 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76002f3d0 con 0x55a353787190 2024-02-09T23:00:53.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd74b7fe640 1 -- 172.21.15.3:0/4238820331 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd76002f530 con 0x55a353787190 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd760005500 con 0x55a353787190 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd74b7fe640 1 -- 172.21.15.3:0/4238820331 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd76002f6f0 con 0x55a353787190 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 10 client.19062 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55a3536ec390 con 0x55a353787190 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd74b7fe640 1 -- 172.21.15.3:0/4238820331 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd760005500 con 0x55a353787190 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd74b7fe640 1 -- 172.21.15.3:0/4238820331 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fd76001d070 con 0x55a353787190 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 10 client.19062 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 10 client.19062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 20 client.19062 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537f8e60 0x55a3537fb280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 4 client.19062 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.108+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55a35342a7b0 con 0x55a3537f8e60 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.109+0000 7fd76bfff640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537f8e60 0x55a3537fb280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.109+0000 7fd74b7fe640 10 client.19062 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.109+0000 7fd76bfff640 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537f8e60 0x55a3537fb280 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec5b32527d2087ed server_cookie=2b6e766614851b55 in_seq=0 out_seq=0 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd74b7fe640 1 -- 172.21.15.3:0/4238820331 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55a35342a7b0 con 0x55a3537f8e60 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd74b7fe640 10 client.19062 handle_command_reply: tid=0 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 client.19062 shutdown 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 msgr2=0x55a3537f0fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 0x55a3537f0fc0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7fd760009250 tx=0x7fd760005850 comp rx=0 tx=0).stop 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 shutdown_connections 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a3537f8e60 0x55a3537fb280 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a353790690 0x55a3537966a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd74b7fe640 0 client.19062 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a353787ab0 0x55a3537f1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 --2- 172.21.15.3:0/4238820331 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a353787190 0x55a3537f0fc0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.110+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 >> 172.21.15.3:0/4238820331 conn(0x55a3537818c0 msgr2=0x55a353780920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 shutdown_connections 2024-02-09T23:00:53.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd77520c740 1 -- 172.21.15.3:0/4238820331 wait complete. 2024-02-09T23:00:53.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd77520c740 20 client.19062 trim_cache size 0 max 16384 2024-02-09T23:00:53.120 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:00:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:00:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "ae3cb810-f182-4f85-b683-364e1a0126c0", 2024-02-09T23:00:53.120 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:00:53.120 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7540a2260 msgr2=0x7fd7540a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7540a2260 0x7fd7540a4720 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd7640cc310 tx=0x7fd76401bb30 comp rx=0 tx=0).stop 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 msgr2=0x7fd76c1a5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 0x7fd76c1a5a30 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fd76400eb10 tx=0x7fd76400c2d0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 shutdown_connections 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd7540a2260 0x7fd7540a4720 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd76c1053e0 0x7fd76c10ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd76c10e310 0x7fd76c1a5a30 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 --2- 172.21.15.3:0/387533344 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd76c100310 0x7fd76c10d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 >> 172.21.15.3:0/387533344 conn(0x7fd76c077230 msgr2=0x7fd76c1065b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 shutdown_connections 2024-02-09T23:00:53.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.111+0000 7fd772563640 1 -- 172.21.15.3:0/387533344 wait complete. 2024-02-09T23:00:53.126 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:00:53.126 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:00:53.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 -- 172.21.15.3:0/271255098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f718410b910 msgr2=0x7f718410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 --2- 172.21.15.3:0/271255098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f718410b910 0x7f718410dd80 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f717c009ca0 tx=0x7f717c02c180 comp rx=0 tx=0).stop 2024-02-09T23:00:53.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 -- 172.21.15.3:0/271255098 shutdown_connections 2024-02-09T23:00:53.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 --2- 172.21.15.3:0/271255098 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f718410b910 0x7f718410dd80 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 --2- 172.21.15.3:0/271255098 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7184102dc0 0x7f718410b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 --2- 172.21.15.3:0/271255098 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f71841024a0 0x7f7184102880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 -- 172.21.15.3:0/271255098 >> 172.21.15.3:0/271255098 conn(0x7f71840772a0 msgr2=0x7f71840776b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 -- 172.21.15.3:0/271255098 shutdown_connections 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.234+0000 7f7189e2a640 1 -- 172.21.15.3:0/271255098 wait complete. 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 Processor -- start 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 -- start start 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71841024a0 0x7f71841000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 0x7f7184100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f718410b910 0x7f7184100b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71841a5cf0 con 0x7f7184102dc0 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71841a5e60 con 0x7f718410b910 2024-02-09T23:00:53.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.235+0000 7f7189e2a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71841a6020 con 0x7f71841024a0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 0x7f7184100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 0x7f7184100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58530/0 (socket says 172.21.15.3:58530) 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f71837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71841024a0 0x7f71841000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 -- 172.21.15.3:0/306791250 learned_addr learned my addr 172.21.15.3:0/306791250 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f7183fff640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f718410b910 0x7f7184100b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 -- 172.21.15.3:0/306791250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71841024a0 msgr2=0x7f71841000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71841024a0 0x7f71841000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 -- 172.21.15.3:0/306791250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f718410b910 msgr2=0x7f7184100b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f718410b910 0x7f7184100b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 -- 172.21.15.3:0/306791250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f717c009900 con 0x7f7184102dc0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f717affd640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 0x7f7184100630 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f716c00e9f0 tx=0x7f716c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f71817fa640 1 -- 172.21.15.3:0/306791250 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c009720 con 0x7f7184102dc0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f71817fa640 1 -- 172.21.15.3:0/306791250 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f716c00cd80 con 0x7f7184102dc0 2024-02-09T23:00:53.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71841a6300 con 0x7f7184102dc0 2024-02-09T23:00:53.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f71817fa640 1 -- 172.21.15.3:0/306791250 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c010690 con 0x7f7184102dc0 2024-02-09T23:00:53.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.236+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71841a67c0 con 0x7f7184102dc0 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.239+0000 7f71817fa640 1 -- 172.21.15.3:0/306791250 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f716c0046c0 con 0x7f7184102dc0 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.239+0000 7f718ca00740 1 Processor -- start 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.239+0000 7f718ca00740 1 -- start start 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.239+0000 7f71817fa640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71700a2230 0x7f71700a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f718ca00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f33d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f718ca00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f3c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f718ca00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f3ceb0 0x563663f3f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f718ca00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663eeed00 con 0x563663f3ceb0 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f718ca00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663ea32f0 con 0x563663f34330 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f718ca00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663ea0820 con 0x563663f33980 2024-02-09T23:00:53.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71817fa640 1 -- 172.21.15.3:0/306791250 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f716c014070 con 0x7f7184102dc0 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f33d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f33d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41380/0 (socket says 172.21.15.3:41380) 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 -- 172.21.15.3:0/2277713430 learned_addr learned my addr 172.21.15.3:0/2277713430 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f7183fff640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71700a2230 0x7f71700a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f7183fff640 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f3ceb0 0x563663f3f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 -- 172.21.15.3:0/2277713430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 msgr2=0x563663f3c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f3c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 -- 172.21.15.3:0/2277713430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f3ceb0 msgr2=0x563663f3f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f3ceb0 0x563663f3f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 -- 172.21.15.3:0/2277713430 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563663e5dff0 con 0x563663f33980 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f7183fff640 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f3ceb0 0x563663f3f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f71837fe640 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f33d60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f717400bd10 tx=0x7f71740107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac922040a8c2ddf server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f7183fff640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71700a2230 0x7f71700a46f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f717c004af0 tx=0x7f717c009820 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f7179ffb640 1 -- 172.21.15.3:0/2277713430 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174004030 con 0x563663f33980 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.240+0000 7f7179ffb640 1 -- 172.21.15.3:0/2277713430 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7174015070 con 0x563663f33980 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f7179ffb640 1 -- 172.21.15.3:0/2277713430 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71740116a0 con 0x563663f33980 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 -- 172.21.15.3:0/2277713430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 msgr2=0x563663f33d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f33d60 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f717400bd10 tx=0x7f71740107a0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 -- 172.21.15.3:0/2277713430 shutdown_connections 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f3c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f3ceb0 0x563663f3f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 --2- 172.21.15.3:0/2277713430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f33d60 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 -- 172.21.15.3:0/2277713430 >> 172.21.15.3:0/2277713430 conn(0x563663f2e050 msgr2=0x563663f2e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 -- 172.21.15.3:0/2277713430 shutdown_connections 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.241+0000 7f718ca00740 1 -- 172.21.15.3:0/2277713430 wait complete. 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 -- start start 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 Processor -- start 2024-02-09T23:00:53.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f39de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f34330 0x563663f3a320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f3ceb0 0x563663f3a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663e9c000 con 0x563663f34330 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663b6f430 con 0x563663f3ceb0 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663b2be00 con 0x563663f33980 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f718ca00740 10 client.0 _ll_register_callbacks cb 0x563663e907f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f717affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f34330 0x563663f3a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f71837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f39de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f71837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f39de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:41392/0 (socket says 172.21.15.3:41392) 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f71837fe640 1 -- 172.21.15.3:0/2122413103 learned_addr learned my addr 172.21.15.3:0/2122413103 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.242+0000 7f7183fff640 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f3ceb0 0x563663f3a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f717affd640 1 -- 172.21.15.3:0/2122413103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 msgr2=0x563663f39de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f717affd640 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f39de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f717affd640 1 -- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f3ceb0 msgr2=0x563663f3a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f717affd640 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f3ceb0 0x563663f3a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f717affd640 1 -- 172.21.15.3:0/2122413103 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f717400b940 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f717affd640 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f34330 0x563663f3a320 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f716c0cbb20 tx=0x7f716c09b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c9704f34b6f5c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f7162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f7162ffd640 1 -- 172.21.15.3:0/2122413103 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f716c0a32d0 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f7162ffd640 1 -- 172.21.15.3:0/2122413103 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f716c09e070 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563663b9b7e0 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f7162ffd640 1 -- 172.21.15.3:0/2122413103 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f716c0a3560 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 10 client.19074 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x563663e9a680 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f7162ffd640 1 -- 172.21.15.3:0/2122413103 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f716c0a7020 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f7162ffd640 1 -- 172.21.15.3:0/2122413103 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f716c01f060 con 0x563663f34330 2024-02-09T23:00:53.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 10 client.19074 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 10 client.19074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 20 client.19074 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f4d3d0 0x563663f4f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 4 client.19074 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.243+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563663bd8720 con 0x563663f4d3d0 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.244+0000 7f71837fe640 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f4d3d0 0x563663f4f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.244+0000 7f71837fe640 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f4d3d0 0x563663f4f7f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe1ad878b08c54c3 server_cookie=aca1bcaaaa4e870d in_seq=0 out_seq=0 2024-02-09T23:00:53.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.244+0000 7f7162ffd640 10 client.19074 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f7162ffd640 1 -- 172.21.15.3:0/2122413103 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563663bd8720 con 0x563663f4d3d0 2024-02-09T23:00:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f7162ffd640 10 client.19074 handle_command_reply: tid=0 2024-02-09T23:00:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 client.19074 shutdown 2024-02-09T23:00:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f34330 msgr2=0x563663f3a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.253 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f34330 0x563663f3a320 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f716c0cbb20 tx=0x7f716c09b4c0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 shutdown_connections 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f4d3d0 0x563663f4f7f0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f3ceb0 0x563663f3a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f34330 0x563663f3a320 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 --2- 172.21.15.3:0/2122413103 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f39de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 >> 172.21.15.3:0/2122413103 conn(0x563663f2e050 msgr2=0x563663f2fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f7162ffd640 0 client.19074 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 shutdown_connections 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 1 -- 172.21.15.3:0/2122413103 wait complete. 2024-02-09T23:00:53.254 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.245+0000 7f718ca00740 20 client.19074 trim_cache size 0 max 16384 2024-02-09T23:00:53.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 Processor -- start 2024-02-09T23:00:53.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 -- start start 2024-02-09T23:00:53.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f3c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f54170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f37040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663e20fd0 con 0x563663f2e050 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663f29980 con 0x563663f34330 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f718ca00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663f3adb0 con 0x563663f33980 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f3c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f3c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58556/0 (socket says 172.21.15.3:58556) 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 -- 172.21.15.3:0/2598108500 learned_addr learned my addr 172.21.15.3:0/2598108500 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f7183fff640 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f37040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f717affd640 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f54170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 -- 172.21.15.3:0/2598108500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 msgr2=0x563663f54170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f54170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 -- 172.21.15.3:0/2598108500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 msgr2=0x563663f37040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f37040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 -- 172.21.15.3:0/2598108500 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563663e36b30 con 0x563663f2e050 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.262+0000 7f71837fe640 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f3c510 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f7174003e50 tx=0x7f7174010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ef56c48895e5f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f71797fa640 1 -- 172.21.15.3:0/2598108500 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174020070 con 0x563663f2e050 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f71797fa640 1 -- 172.21.15.3:0/2598108500 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7174021040 con 0x563663f2e050 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f71797fa640 1 -- 172.21.15.3:0/2598108500 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174028370 con 0x563663f2e050 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 -- 172.21.15.3:0/2598108500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 msgr2=0x563663f3c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f3c510 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f7174003e50 tx=0x7f7174010d50 comp rx=0 tx=0).stop 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 -- 172.21.15.3:0/2598108500 shutdown_connections 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f37040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f54170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 --2- 172.21.15.3:0/2598108500 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f3c510 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.263+0000 7f718ca00740 1 -- 172.21.15.3:0/2598108500 >> 172.21.15.3:0/2598108500 conn(0x563663f34a80 msgr2=0x563663f35160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 -- 172.21.15.3:0/2598108500 shutdown_connections 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 -- 172.21.15.3:0/2598108500 wait complete. 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 -- start start 2024-02-09T23:00:53.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 Processor -- start 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f43100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f43640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f9d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663f9de10 con 0x563663f2e050 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663f9df80 con 0x563663f34330 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563663f9e140 con 0x563663f33980 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f718ca00740 10 client.0 _ll_register_callbacks cb 0x7f717400b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f71837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f43100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f71837fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f43100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58560/0 (socket says 172.21.15.3:58560) 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f71837fe640 1 -- 172.21.15.3:0/2042079211 learned_addr learned my addr 172.21.15.3:0/2042079211 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.264+0000 7f717affd640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f43640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7183fff640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f9d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f71837fe640 1 -- 172.21.15.3:0/2042079211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 msgr2=0x563663f43640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f71837fe640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f43640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f71837fe640 1 -- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 msgr2=0x563663f9d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f71837fe640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f9d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f71837fe640 1 -- 172.21.15.3:0/2042079211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563663e5dff0 con 0x563663f2e050 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7183fff640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f9d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f71837fe640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f43100 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f71740107e0 tx=0x7f7174010dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27ea0168c85d8df2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7162ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7162ffd640 1 -- 172.21.15.3:0/2042079211 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7174020040 con 0x563663f2e050 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7162ffd640 1 -- 172.21.15.3:0/2042079211 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7174017070 con 0x563663f2e050 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563663f39c10 con 0x563663f2e050 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7162ffd640 1 -- 172.21.15.3:0/2042079211 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7174005380 con 0x563663f2e050 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f718ca00740 10 client.19080 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x563663e5c850 con 0x563663f2e050 2024-02-09T23:00:53.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.265+0000 7f7162ffd640 1 -- 172.21.15.3:0/2042079211 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563663f39c10 con 0x563663f2e050 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f7162ffd640 1 -- 172.21.15.3:0/2042079211 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7174028690 con 0x563663f2e050 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 10 client.19080 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 10 client.19080 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 20 client.19080 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f3ceb0 0x563663fa7690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 4 client.19080 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f7174015070 con 0x563663f3ceb0 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f717affd640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f3ceb0 0x563663fa7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f717affd640 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f3ceb0 0x563663fa7690 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ad765c33b583b43 server_cookie=930cb3c381ab6364 in_seq=0 out_seq=0 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f7162ffd640 10 client.19080 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f7162ffd640 1 -- 172.21.15.3:0/2042079211 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x7f7174015070 con 0x563663f3ceb0 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f7162ffd640 10 client.19080 handle_command_reply: tid=0 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.266+0000 7f718ca00740 1 client.19080 shutdown 2024-02-09T23:00:53.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 msgr2=0x563663f43100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f43100 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f71740107e0 tx=0x7f7174010dc0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 shutdown_connections 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563663f3ceb0 0x563663fa7690 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7162ffd640 0 client.19080 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563663f34330 0x563663f9d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563663f33980 0x563663f43640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 --2- 172.21.15.3:0/2042079211 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563663f2e050 0x563663f43100 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 >> 172.21.15.3:0/2042079211 conn(0x563663f34a80 msgr2=0x563663f54170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 shutdown_connections 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 1 -- 172.21.15.3:0/2042079211 wait complete. 2024-02-09T23:00:53.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f718ca00740 20 client.19080 trim_cache size 0 max 16384 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "ae3cb810-f182-4f85-b683-364e1a0126c0": { 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "ae3cb810-f182-4f85-b683-364e1a0126c0", 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:00:53.278 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:00:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71700a2230 msgr2=0x7f71700a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71700a2230 0x7f71700a46f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f717c004af0 tx=0x7f717c009820 comp rx=0 tx=0).stop 2024-02-09T23:00:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 msgr2=0x7f7184100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 0x7f7184100630 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 crypto rx=0x7f716c00e9f0 tx=0x7f716c00edf0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 shutdown_connections 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f71700a2230 0x7f71700a46f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f718410b910 0x7f7184100b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7184102dc0 0x7f7184100630 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.267+0000 7f7189e2a640 1 --2- 172.21.15.3:0/306791250 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f71841024a0 0x7f71841000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.268+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 >> 172.21.15.3:0/306791250 conn(0x7f71840772a0 msgr2=0x7f71840fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.268+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 shutdown_connections 2024-02-09T23:00:53.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:00:53.268+0000 7f7189e2a640 1 -- 172.21.15.3:0/306791250 wait complete. 2024-02-09T23:00:53.282 INFO:tasks.cephfs.filesystem:scrub status for tag:ae3cb810-f182-4f85-b683-364e1a0126c0 - {'path': '/', 'tag': 'ae3cb810-f182-4f85-b683-364e1a0126c0', 'options': 'recursive,force'} 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:53.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:53 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 304436 2.99 MB/sec execute 407 sec latency 906.470 ms 2024-02-09T23:00:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:53.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:53.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:53 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:53 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: pgmap v300: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:00:54.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:54 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 305789 3.00 MB/sec execute 408 sec latency 13.950 ms 2024-02-09T23:00:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: pgmap v300: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-02-09T23:00:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:00:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:54 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: pgmap v300: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:00:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:54 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:00:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 -- 172.21.15.3:0/1702309523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58500ffb10 msgr2=0x7f58500fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 --2- 172.21.15.3:0/1702309523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58500ffb10 0x7f58500fff90 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f5844009930 tx=0x7f584402bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 -- 172.21.15.3:0/1702309523 shutdown_connections 2024-02-09T23:00:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 --2- 172.21.15.3:0/1702309523 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 0x7f58501102a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.554 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 --2- 172.21.15.3:0/1702309523 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f58500ffb10 0x7f58500fff90 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 --2- 172.21.15.3:0/1702309523 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5850102040 0x7f5850102420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 -- 172.21.15.3:0/1702309523 >> 172.21.15.3:0/1702309523 conn(0x7f5850075410 msgr2=0x7f5850075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 -- 172.21.15.3:0/1702309523 shutdown_connections 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 -- 172.21.15.3:0/1702309523 wait complete. 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.549+0000 7f58557e2640 1 Processor -- start 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 -- start start 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58500ffb10 0x7f585019f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 0x7f585019fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 0x7f5850199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5850199f50 con 0x7f5850102040 2024-02-09T23:00:55.555 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585019a0c0 con 0x7f58500ffb10 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f58557e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f585019a280 con 0x7f58501004d0 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 0x7f5850199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 0x7f5850199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58578/0 (socket says 172.21.15.3:58578) 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 -- 172.21.15.3:0/1553925752 learned_addr learned my addr 172.21.15.3:0/1553925752 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584e7fc640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 0x7f585019fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584effd640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58500ffb10 0x7f585019f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 -- 172.21.15.3:0/1553925752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 msgr2=0x7f585019fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 0x7f585019fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 -- 172.21.15.3:0/1553925752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58500ffb10 msgr2=0x7f585019f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58500ffb10 0x7f585019f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 -- 172.21.15.3:0/1553925752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5844009560 con 0x7f5850102040 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584effd640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58500ffb10 0x7f585019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584e7fc640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 0x7f585019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:55.556 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.550+0000 7f584f7fe640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 0x7f5850199900 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f5848009fd0 tx=0x7f584800b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.551+0000 7f5837fff640 1 -- 172.21.15.3:0/1553925752 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800bea0 con 0x7f5850102040 2024-02-09T23:00:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.551+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f585019a560 con 0x7f5850102040 2024-02-09T23:00:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.551+0000 7f5837fff640 1 -- 172.21.15.3:0/1553925752 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5848009940 con 0x7f5850102040 2024-02-09T23:00:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.551+0000 7f5837fff640 1 -- 172.21.15.3:0/1553925752 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58480106c0 con 0x7f5850102040 2024-02-09T23:00:55.557 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.551+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58501a67f0 con 0x7f5850102040 2024-02-09T23:00:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f5837fff640 1 -- 172.21.15.3:0/1553925752 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5848004240 con 0x7f5850102040 2024-02-09T23:00:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 Processor -- start 2024-02-09T23:00:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 -- start start 2024-02-09T23:00:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078bb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556607829a00 con 0x5566078bb9d0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556607826f30 con 0x5566078c47a0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f58583b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556607822710 con 0x5566078bb0b0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f584effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f5837fff640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f58380a2220 0x7f58380a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f584effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58580/0 (socket says 172.21.15.3:58580) 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.553+0000 7f584effd640 1 -- 172.21.15.3:0/1274152541 learned_addr learned my addr 172.21.15.3:0/1274152541 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584f7fe640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f58380a2220 0x7f58380a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584e7fc640 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078bb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584e7fc640 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.564 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f5837fff640 1 -- 172.21.15.3:0/1553925752 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f58480cde90 con 0x7f5850102040 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584effd640 1 -- 172.21.15.3:0/1274152541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 msgr2=0x5566078bb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584effd640 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078bb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584effd640 1 -- 172.21.15.3:0/1274152541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 msgr2=0x5566078c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584effd640 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584effd640 1 -- 172.21.15.3:0/1274152541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566077e73b0 con 0x5566078bb9d0 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584e7fc640 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584effd640 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c4260 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f584000bae0 tx=0x7f584000d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73aa14f477e4eb10 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f584f7fe640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f58380a2220 0x7f58380a46e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5848060c30 tx=0x7f5848085040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f58357fa640 1 -- 172.21.15.3:0/1274152541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840004020 con 0x5566078bb9d0 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f58357fa640 1 -- 172.21.15.3:0/1274152541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5840003610 con 0x5566078bb9d0 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.554+0000 7f58357fa640 1 -- 172.21.15.3:0/1274152541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584000f680 con 0x5566078bb9d0 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 -- 172.21.15.3:0/1274152541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 msgr2=0x5566078c4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.565 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c4260 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f584000bae0 tx=0x7f584000d340 comp rx=0 tx=0).stop 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 -- 172.21.15.3:0/1274152541 shutdown_connections 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c4260 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 --2- 172.21.15.3:0/1274152541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078bb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 -- 172.21.15.3:0/1274152541 >> 172.21.15.3:0/1274152541 conn(0x5566078b59d0 msgr2=0x5566078b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 -- 172.21.15.3:0/1274152541 shutdown_connections 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 -- 172.21.15.3:0/1274152541 wait complete. 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 -- start start 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.555+0000 7f58583b2740 1 Processor -- start 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078c1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566077c7b90 con 0x5566078bb9d0 2024-02-09T23:00:55.566 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566074f6250 con 0x5566078c47a0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566077a8200 con 0x5566078bb0b0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 10 client.0 _ll_register_callbacks cb 0x5566077de5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58596/0 (socket says 172.21.15.3:58596) 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 -- 172.21.15.3:0/3604502534 learned_addr learned my addr 172.21.15.3:0/3604502534 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078c1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 -- 172.21.15.3:0/3604502534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 msgr2=0x5566078c1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078c1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 -- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 msgr2=0x5566078c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 -- 172.21.15.3:0/3604502534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584000b710 con 0x5566078bb9d0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f584effd640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c1ca0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f5840003f20 tx=0x7f5840003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd81d33176ced60 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:55.567 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f5832ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f5832ff5640 1 -- 172.21.15.3:0/3604502534 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58400041b0 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f5832ff5640 1 -- 172.21.15.3:0/3604502534 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5840013070 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.556+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556607848ee0 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f5832ff5640 1 -- 172.21.15.3:0/3604502534 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5840004310 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 10 client.19092 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x556607820d90 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f5832ff5640 1 -- 172.21.15.3:0/3604502534 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556607848ee0 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f5832ff5640 1 -- 172.21.15.3:0/3604502534 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5840028060 con 0x5566078bb9d0 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 10 client.19092 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 10 client.19092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 20 client.19092 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566078d4990 0x5566078d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.568 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 4 client.19092 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556607522600 con 0x5566078d4990 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566078d4990 0x5566078d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566078d4990 0x5566078d6db0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1af1ceb906aec9c4 server_cookie=80eba13c57a09f20 in_seq=0 out_seq=0 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.557+0000 7f5832ff5640 10 client.19092 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f5832ff5640 1 -- 172.21.15.3:0/3604502534 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556607522600 con 0x5566078d4990 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f5832ff5640 10 client.19092 handle_command_reply: tid=0 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 client.19092 shutdown 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 msgr2=0x5566078c1ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c1ca0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f5840003f20 tx=0x7f5840003aa0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 shutdown_connections 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5566078d4990 0x5566078d6db0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078c47a0 0x5566078c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c1ca0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f5832ff5640 0 client.19092 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 --2- 172.21.15.3:0/3604502534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb0b0 0x5566078c1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.569 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 >> 172.21.15.3:0/3604502534 conn(0x5566078b59d0 msgr2=0x5566078b7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 shutdown_connections 2024-02-09T23:00:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 1 -- 172.21.15.3:0/3604502534 wait complete. 2024-02-09T23:00:55.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.558+0000 7f58583b2740 20 client.19092 trim_cache size 0 max 16384 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.571+0000 7f58583b2740 1 Processor -- start 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.571+0000 7f58583b2740 1 -- start start 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f58583b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f58583b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f58583b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078c47a0 0x5566078db6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f58583b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566078ba850 con 0x5566078bb9d0 2024-02-09T23:00:55.578 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f58583b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566078bfcb0 con 0x5566078bb0b0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f58583b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566078c2540 con 0x5566078c47a0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58602/0 (socket says 172.21.15.3:58602) 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 -- 172.21.15.3:0/572618911 learned_addr learned my addr 172.21.15.3:0/572618911 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584e7fc640 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078c47a0 0x5566078db6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 -- 172.21.15.3:0/572618911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078c47a0 msgr2=0x5566078db6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078c47a0 0x5566078db6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 -- 172.21.15.3:0/572618911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 msgr2=0x5566078c3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 -- 172.21.15.3:0/572618911 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5566077bd820 con 0x5566078bb9d0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584e7fc640 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078c47a0 0x5566078db6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f584effd640 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c2000 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f584000b630 tx=0x7f584000ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61ad0151c3e512d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f5834ff9640 1 -- 172.21.15.3:0/572618911 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840005d70 con 0x5566078bb9d0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f5834ff9640 1 -- 172.21.15.3:0/572618911 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5840017430 con 0x5566078bb9d0 2024-02-09T23:00:55.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.572+0000 7f5834ff9640 1 -- 172.21.15.3:0/572618911 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5840017ac0 con 0x5566078bb9d0 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 -- 172.21.15.3:0/572618911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 msgr2=0x5566078c2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c2000 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 crypto rx=0x7f584000b630 tx=0x7f584000ba60 comp rx=0 tx=0).stop 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 -- 172.21.15.3:0/572618911 shutdown_connections 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078c47a0 0x5566078db6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078bb9d0 0x5566078c2000 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 --2- 172.21.15.3:0/572618911 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 -- 172.21.15.3:0/572618911 >> 172.21.15.3:0/572618911 conn(0x5566078b59d0 msgr2=0x5566078b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 -- 172.21.15.3:0/572618911 shutdown_connections 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 -- 172.21.15.3:0/572618911 wait complete. 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 -- start start 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.573+0000 7f58583b2740 1 Processor -- start 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb9d0 0x5566078c9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 0x556607929e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5566078ca030 con 0x5566078c47a0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660792a3b0 con 0x5566078bb0b0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55660792a4f0 con 0x5566078bb9d0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 10 client.0 _ll_register_callbacks cb 0x7f584000b930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 0x556607929e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584effd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb9d0 0x5566078c9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 0x556607929e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:58608/0 (socket says 172.21.15.3:58608) 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 -- 172.21.15.3:0/3098808579 learned_addr learned my addr 172.21.15.3:0/3098808579 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 -- 172.21.15.3:0/3098808579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb9d0 msgr2=0x5566078c9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb9d0 0x5566078c9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 -- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 msgr2=0x5566078c9220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c9220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 -- 172.21.15.3:0/3098808579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556607820d90 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f584e7fc640 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 0x556607929e70 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f5844009850 tx=0x7f584402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df32ce81045f6b5f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f5832ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f5832ff5640 1 -- 172.21.15.3:0/3098808579 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5844047040 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f5832ff5640 1 -- 172.21.15.3:0/3098808579 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f584402c9e0 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.574+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5840003610 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f5832ff5640 1 -- 172.21.15.3:0/3098808579 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f58440044f0 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 10 client.19098 fetch_fsmap learned FSMap version 47 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5566077e36f0 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f5832ff5640 1 -- 172.21.15.3:0/3098808579 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5840003610 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f5832ff5640 1 -- 172.21.15.3:0/3098808579 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f584403c040 con 0x5566078c47a0 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 10 client.19098 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 10 client.19098 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 20 client.19098 populate_metadata read hostname 'smithi003' 2024-02-09T23:00:55.582 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55660792bba0 0x55660792df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:00:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 4 client.19098 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:00:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5566075731b0 con 0x55660792bba0 2024-02-09T23:00:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.575+0000 7f584effd640 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55660792bba0 0x55660792df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:00:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.576+0000 7f5832ff5640 10 client.19098 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:55.583 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.576+0000 7f584effd640 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55660792bba0 0x55660792df90 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41a60d09f2aceb2e server_cookie=4de090b8ecf9d6f5 in_seq=0 out_seq=0 2024-02-09T23:00:55.615 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f5832ff5640 1 -- 172.21.15.3:0/3098808579 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5566075731b0 con 0x55660792bba0 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f5832ff5640 10 client.19098 handle_command_reply: tid=0 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 client.19098 shutdown 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 msgr2=0x556607929e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 0x556607929e70 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f5844009850 tx=0x7f584402bc30 comp rx=0 tx=0).stop 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 shutdown_connections 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55660792bba0 0x55660792df90 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5566078c47a0 0x556607929e70 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5566078bb9d0 0x5566078c9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 --2- 172.21.15.3:0/3098808579 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5566078bb0b0 0x5566078c9220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f5832ff5640 0 client.19098 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.612+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 >> 172.21.15.3:0/3098808579 conn(0x5566078b59d0 msgr2=0x5566078db6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:55.616 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 shutdown_connections 2024-02-09T23:00:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58583b2740 1 -- 172.21.15.3:0/3098808579 wait complete. 2024-02-09T23:00:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58583b2740 20 client.19098 trim_cache size 0 max 16384 2024-02-09T23:00:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:00:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:00:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:00:55.617 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f58380a2220 msgr2=0x7f58380a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f58380a2220 0x7f58380a46e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5848060c30 tx=0x7f5848085040 comp rx=0 tx=0).stop 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 msgr2=0x7f5850199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 0x7f5850199900 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f5848009fd0 tx=0x7f584800b260 comp rx=0 tx=0).stop 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 shutdown_connections 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f58380a2220 0x7f58380a46e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.613+0000 7f58557e2640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5850102040 0x7f5850199900 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.614+0000 7f58557e2640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f58501004d0 0x7f585019fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.614+0000 7f58557e2640 1 --2- 172.21.15.3:0/1553925752 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f58500ffb10 0x7f585019f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:00:55.618 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.614+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 >> 172.21.15.3:0/1553925752 conn(0x7f5850075410 msgr2=0x7f58500fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:00:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.614+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 shutdown_connections 2024-02-09T23:00:55.619 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:00:55.614+0000 7f58557e2640 1 -- 172.21.15.3:0/1553925752 wait complete. 2024-02-09T23:00:55.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:55 smithi003 ceph-mon[32678]: pgmap v301: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-02-09T23:00:55.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 306891 3.00 MB/sec execute 409 sec latency 423.602 ms 2024-02-09T23:00:55.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:55 smithi064 ceph-mon[36638]: pgmap v301: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-02-09T23:00:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:55 smithi089 ceph-mon[36665]: pgmap v301: 97 pgs: 97 active+clean; 123 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 168 op/s 2024-02-09T23:00:56.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 308981 3.03 MB/sec execute 410 sec latency 12.738 ms 2024-02-09T23:00:57.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 310222 3.04 MB/sec execute 411 sec latency 14.739 ms 2024-02-09T23:00:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:00:57 smithi089 ceph-mon[36665]: pgmap v302: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-02-09T23:00:57.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:00:57 smithi003 ceph-mon[32678]: pgmap v302: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-02-09T23:00:57.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:00:57 smithi064 ceph-mon[36638]: pgmap v302: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.5 MiB/s wr, 269 op/s 2024-02-09T23:00:58.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 310305 3.03 MB/sec execute 412 sec latency 918.783 ms 2024-02-09T23:00:59.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 310305 3.02 MB/sec execute 413 sec latency 1918.871 ms 2024-02-09T23:01:00.585 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:00 smithi003 ceph-mon[32678]: pgmap v303: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-02-09T23:01:00.638 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:00 smithi064 ceph-mon[36638]: pgmap v303: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-02-09T23:01:00.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 311626 3.03 MB/sec execute 414 sec latency 2365.869 ms 2024-02-09T23:01:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:00 smithi089 ceph-mon[36665]: pgmap v303: 97 pgs: 97 active+clean; 128 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.4 MiB/s wr, 165 op/s 2024-02-09T23:01:01.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 312598 3.04 MB/sec execute 415 sec latency 25.540 ms 2024-02-09T23:01:02.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:02 smithi003 ceph-mon[32678]: pgmap v304: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 238 op/s 2024-02-09T23:01:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:02.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:02 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:02.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 313847 3.05 MB/sec execute 416 sec latency 189.168 ms 2024-02-09T23:01:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:02 smithi064 ceph-mon[36638]: pgmap v304: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 238 op/s 2024-02-09T23:01:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:02 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:02 smithi089 ceph-mon[36665]: pgmap v304: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.2 MiB/s wr, 238 op/s 2024-02-09T23:01:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:02 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:03.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:03 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:03.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:03 smithi003 ceph-mon[32678]: osdmap e98: 12 total, 12 up, 12 in 2024-02-09T23:01:03.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 315924 3.07 MB/sec execute 417 sec latency 311.996 ms 2024-02-09T23:01:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:03 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:03 smithi064 ceph-mon[36638]: osdmap e98: 12 total, 12 up, 12 in 2024-02-09T23:01:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:03 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:03 smithi089 ceph-mon[36665]: osdmap e98: 12 total, 12 up, 12 in 2024-02-09T23:01:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:04 smithi003 ceph-mon[32678]: pgmap v306: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:01:04.658 INFO:tasks.workunit.client.0.smithi003.stdout: 1 317129 3.08 MB/sec execute 418 sec latency 12.143 ms 2024-02-09T23:01:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:04 smithi064 ceph-mon[36638]: pgmap v306: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:01:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:04 smithi089 ceph-mon[36665]: pgmap v306: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:01:05.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 317396 3.07 MB/sec execute 419 sec latency 682.366 ms 2024-02-09T23:01:06.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:06 smithi003 ceph-mon[32678]: pgmap v307: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:01:06.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 317396 3.06 MB/sec execute 420 sec latency 1682.454 ms 2024-02-09T23:01:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:06 smithi064 ceph-mon[36638]: pgmap v307: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:01:06.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:06 smithi089 ceph-mon[36665]: pgmap v307: 97 pgs: 97 active+clean; 126 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:01:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:07 smithi003 ceph-mon[32678]: pgmap v308: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-02-09T23:01:07.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 319482 3.09 MB/sec execute 421 sec latency 1738.571 ms 2024-02-09T23:01:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:07 smithi064 ceph-mon[36638]: pgmap v308: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-02-09T23:01:07.726 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:07 smithi089 ceph-mon[36665]: pgmap v308: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-02-09T23:01:08.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 320742 3.09 MB/sec execute 422 sec latency 12.316 ms 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 -- 172.21.15.3:0/3827428498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce8100cf0 msgr2=0x7f5ce8104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 --2- 172.21.15.3:0/3827428498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce8100cf0 0x7f5ce8104d30 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009930 tx=0x7f5cd402bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 -- 172.21.15.3:0/3827428498 shutdown_connections 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 --2- 172.21.15.3:0/3827428498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce8105410 0x7f5ce810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 --2- 172.21.15.3:0/3827428498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce8100cf0 0x7f5ce8104d30 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 --2- 172.21.15.3:0/3827428498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ce8100340 0x7f5ce8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 -- 172.21.15.3:0/3827428498 >> 172.21.15.3:0/3827428498 conn(0x7f5ce8077220 msgr2=0x7f5ce8077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 -- 172.21.15.3:0/3827428498 shutdown_connections 2024-02-09T23:01:08.709 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 -- 172.21.15.3:0/3827428498 wait complete. 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 Processor -- start 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.703+0000 7f5cef076640 1 -- start start 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce8100340 0x7f5ce819c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ce8105410 0x7f5ce819c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 0x7f5ce8196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8196fc0 con 0x7f5ce81965d0 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce8197130 con 0x7f5ce8100340 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ce81972f0 con 0x7f5ce8105410 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 0x7f5ce8196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.710 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cece12640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce8100340 0x7f5ce819c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 0x7f5ce8196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35898/0 (socket says 172.21.15.3:35898) 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 -- 172.21.15.3:0/210632811 learned_addr learned my addr 172.21.15.3:0/210632811 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ce8105410 0x7f5ce819c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 -- 172.21.15.3:0/210632811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ce8105410 msgr2=0x7f5ce819c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ce8105410 0x7f5ce819c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 -- 172.21.15.3:0/210632811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce8100340 msgr2=0x7f5ce819c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce8100340 0x7f5ce819c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 -- 172.21.15.3:0/210632811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cd4009560 con 0x7f5ce81965d0 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ced613640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 0x7f5ce8196a80 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000eaa0 tx=0x7f5ce000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ce5ffb640 1 -- 172.21.15.3:0/210632811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce000cdb0 con 0x7f5ce81965d0 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ce5ffb640 1 -- 172.21.15.3:0/210632811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ce0005020 con 0x7f5ce81965d0 2024-02-09T23:01:08.711 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ce81a32e0 con 0x7f5ce81965d0 2024-02-09T23:01:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.704+0000 7f5ce5ffb640 1 -- 172.21.15.3:0/210632811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ce00116b0 con 0x7f5ce81965d0 2024-02-09T23:01:08.712 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.705+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ce81a37c0 con 0x7f5ce81965d0 2024-02-09T23:01:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.707+0000 7f5ce5ffb640 1 -- 172.21.15.3:0/210632811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5ce0005520 con 0x7f5ce81965d0 2024-02-09T23:01:08.718 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.707+0000 7f5ce5ffb640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5cd80a66b0 0x7f5cd80a8b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.707+0000 7f5cf1c4a740 1 Processor -- start 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.707+0000 7f5ce5ffb640 1 -- 172.21.15.3:0/210632811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f5ce00ce130 con 0x7f5ce81965d0 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.707+0000 7f5cf1c4a740 1 -- start start 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.707+0000 7f5cece12640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5cd80a66b0 0x7f5cd80a8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce2c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce26a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce32d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2cd95a00 con 0x55bd2ce26600 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2cd92f30 con 0x55bd2ce26fa0 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2cd8e710 con 0x55bd2ce2bf40 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5ce7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce26a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce2c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.719 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5ce7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce26a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35910/0 (socket says 172.21.15.3:35910) 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5ce7fff640 1 -- 172.21.15.3:0/4064745196 learned_addr learned my addr 172.21.15.3:0/4064745196 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5ced613640 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce32d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5cd80a66b0 0x7f5cd80a8b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009c20 tx=0x7f5cdc005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 -- 172.21.15.3:0/4064745196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 msgr2=0x55bd2ce32d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce32d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 -- 172.21.15.3:0/4064745196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 msgr2=0x55bd2ce26a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce26a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 -- 172.21.15.3:0/4064745196 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd2cd533b0 con 0x55bd2ce2bf40 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce26a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cece12640 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce2c320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc011ca0 tx=0x7f5cdc01c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1dfa7a87d750790 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cd2ffd640 1 -- 172.21.15.3:0/4064745196 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc005220 con 0x55bd2ce2bf40 2024-02-09T23:01:08.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cd2ffd640 1 -- 172.21.15.3:0/4064745196 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc01ce10 con 0x55bd2ce2bf40 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.708+0000 7f5cd2ffd640 1 -- 172.21.15.3:0/4064745196 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc024740 con 0x55bd2ce2bf40 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/4064745196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 msgr2=0x55bd2ce2c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce2c320 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc011ca0 tx=0x7f5cdc01c000 comp rx=0 tx=0).stop 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/4064745196 shutdown_connections 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce32d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce26a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/4064745196 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce2c320 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/4064745196 >> 172.21.15.3:0/4064745196 conn(0x55bd2ce21480 msgr2=0x55bd2ce21860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/4064745196 shutdown_connections 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/4064745196 wait complete. 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.709+0000 7f5cf1c4a740 1 -- start start 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 Processor -- start 2024-02-09T23:01:08.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26600 0x55bd2ce301d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce30710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2cd33b90 con 0x55bd2ce26fa0 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ca62250 con 0x55bd2ce26600 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2cd14200 con 0x55bd2ce2bf40 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cf1c4a740 10 client.0 _ll_register_callbacks cb 0x55bd2cd4a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce30710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce30710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35920/0 (socket says 172.21.15.3:35920) 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3000098268 learned_addr learned my addr 172.21.15.3:0/3000098268 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ced613640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cece12640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26600 0x55bd2ce301d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3000098268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 msgr2=0x55bd2ce30c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26600 msgr2=0x55bd2ce301d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26600 0x55bd2ce301d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3000098268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5cdc017050 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.710+0000 7f5cece12640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26600 0x55bd2ce301d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce30710 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4036040 tx=0x7f5cd402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae4ab7431d630256 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3000098268 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4004040 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3000098268 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f5cd40041a0 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bd2cdb4ee0 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3000098268 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cd4004360 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 10 client.19110 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55bd2cd8cd90 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3000098268 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bd2cdb4ee0 con 0x55bd2ce26fa0 2024-02-09T23:01:08.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3000098268 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5cd403c040 con 0x55bd2ce26fa0 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 10 client.19110 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 10 client.19110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 20 client.19110 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce408e0 0x55bd2ce42d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 4 client.19110 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bd2ca8e600 con 0x55bd2ce408e0 2024-02-09T23:01:08.724 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.711+0000 7f5cece12640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce408e0 0x55bd2ce42d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.712+0000 7f5cece12640 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce408e0 0x55bd2ce42d20 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ac225629540d4b8 server_cookie=f51d7f40f2a8d95a in_seq=0 out_seq=0 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.712+0000 7f5cc7fff640 10 client.19110 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3000098268 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bd2ca8e600 con 0x55bd2ce408e0 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cc7fff640 10 client.19110 handle_command_reply: tid=0 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 client.19110 shutdown 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 msgr2=0x55bd2ce30710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.725 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce30710 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4036040 tx=0x7f5cd402ca10 comp rx=0 tx=0).stop 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 shutdown_connections 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce408e0 0x55bd2ce42d20 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce30710 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cc7fff640 0 client.19110 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3000098268 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26600 0x55bd2ce301d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 >> 172.21.15.3:0/3000098268 conn(0x55bd2ce21480 msgr2=0x55bd2ce20a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:08.726 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 shutdown_connections 2024-02-09T23:01:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3000098268 wait complete. 2024-02-09T23:01:08.727 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.713+0000 7f5cf1c4a740 20 client.19110 trim_cache size 0 max 16384 2024-02-09T23:01:08.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 Processor -- start 2024-02-09T23:01:08.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 -- start start 2024-02-09T23:01:08.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce25ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce2dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ce27530 con 0x55bd2ce26600 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ce1be10 con 0x55bd2ce26fa0 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.727+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ce29f10 con 0x55bd2ce2bf40 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce25ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce25ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35934/0 (socket says 172.21.15.3:35934) 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 -- 172.21.15.3:0/1026546630 learned_addr learned my addr 172.21.15.3:0/1026546630 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5ced613640 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce2dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 -- 172.21.15.3:0/1026546630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 msgr2=0x55bd2ce30a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 -- 172.21.15.3:0/1026546630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 msgr2=0x55bd2ce2dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce2dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 -- 172.21.15.3:0/1026546630 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bd2cd29820 con 0x55bd2ce26600 2024-02-09T23:01:08.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cece12640 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce25ef0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc011ca0 tx=0x7f5cdc009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4d7bf9cada14f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cd27fc640 1 -- 172.21.15.3:0/1026546630 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc005380 con 0x55bd2ce26600 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cd27fc640 1 -- 172.21.15.3:0/1026546630 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc0346c0 con 0x55bd2ce26600 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.728+0000 7f5cd27fc640 1 -- 172.21.15.3:0/1026546630 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cdc003bc0 con 0x55bd2ce26600 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/1026546630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 msgr2=0x55bd2ce25ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce25ef0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc011ca0 tx=0x7f5cdc009f10 comp rx=0 tx=0).stop 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/1026546630 shutdown_connections 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce2dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce30a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/1026546630 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26600 0x55bd2ce25ef0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/1026546630 >> 172.21.15.3:0/1026546630 conn(0x55bd2ce21480 msgr2=0x55bd2ce2b330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/1026546630 shutdown_connections 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/1026546630 wait complete. 2024-02-09T23:01:08.738 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 -- start start 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.729+0000 7f5cf1c4a740 1 Processor -- start 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 0x55bd2ce35a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce35f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce8fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ce90350 con 0x55bd2ce26fa0 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ce904c0 con 0x55bd2ce2bf40 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bd2ce90680 con 0x55bd2ce26600 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 10 client.0 _ll_register_callbacks cb 0x7f5cd402c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce35f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cece12640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 0x55bd2ce35a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ced613640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce8fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cece12640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 0x55bd2ce35a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:34116/0 (socket says 172.21.15.3:34116) 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce35f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35938/0 (socket says 172.21.15.3:35938) 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cece12640 1 -- 172.21.15.3:0/3287584989 learned_addr learned my addr 172.21.15.3:0/3287584989 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:08.739 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3287584989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 msgr2=0x55bd2ce35a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 0x55bd2ce35a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce2bf40 msgr2=0x55bd2ce8fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce8fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 -- 172.21.15.3:0/3287584989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bd2cd533b0 con 0x55bd2ce26fa0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cece12640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 0x55bd2ce35a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ced613640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce8fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5ce7fff640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce35f60 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009850 tx=0x7f5cd4004560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32058eb54dcfb81 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cc7fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3287584989 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5cd4005850 con 0x55bd2ce26fa0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3287584989 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f5cd4000e40 con 0x55bd2ce26fa0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.730+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5cdc01ce10 con 0x55bd2ce26fa0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3287584989 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5cd4003a50 con 0x55bd2ce26fa0 2024-02-09T23:01:08.740 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 10 client.19116 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55bd2cd4f6f0 con 0x55bd2ce26fa0 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3287584989 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5cdc01ce10 con 0x55bd2ce26fa0 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3287584989 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f5cd403c040 con 0x55bd2ce26fa0 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 10 client.19116 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 10 client.19116 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 20 client.19116 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce97db0 0x55bd2ce9a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 4 client.19116 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bd2cadf1b0 con 0x55bd2ce97db0 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cece12640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce97db0 0x55bd2ce9a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cc7fff640 10 client.19116 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:08.741 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.731+0000 7f5cece12640 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce97db0 0x55bd2ce9a1d0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca47f342ca4ab8b3 server_cookie=bc1f9b5e8da712f9 in_seq=0 out_seq=0 2024-02-09T23:01:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cc7fff640 1 -- 172.21.15.3:0/3287584989 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bd2cadf1b0 con 0x55bd2ce97db0 2024-02-09T23:01:08.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cc7fff640 10 client.19116 handle_command_reply: tid=0 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 client.19116 shutdown 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 msgr2=0x55bd2ce35f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce35f60 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f5cd4009850 tx=0x7f5cd4004560 comp rx=0 tx=0).stop 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 shutdown_connections 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55bd2ce97db0 0x55bd2ce9a1d0 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55bd2ce2bf40 0x55bd2ce8fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55bd2ce26fa0 0x55bd2ce35f60 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cc7fff640 0 client.19116 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 --2- 172.21.15.3:0/3287584989 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55bd2ce26600 0x55bd2ce35a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 >> 172.21.15.3:0/3287584989 conn(0x55bd2ce21480 msgr2=0x55bd2ce2a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 shutdown_connections 2024-02-09T23:01:08.782 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 1 -- 172.21.15.3:0/3287584989 wait complete. 2024-02-09T23:01:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.778+0000 7f5cf1c4a740 20 client.19116 trim_cache size 0 max 16384 2024-02-09T23:01:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:01:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:01:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:01:08.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5cd80a66b0 msgr2=0x7f5cd80a8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5cd80a66b0 0x7f5cd80a8b70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc009c20 tx=0x7f5cdc005f90 comp rx=0 tx=0).stop 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 msgr2=0x7f5ce8196a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 0x7f5ce8196a80 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f5ce000eaa0 tx=0x7f5ce000ef60 comp rx=0 tx=0).stop 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 shutdown_connections 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5cd80a66b0 0x7f5cd80a8b70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ce81965d0 0x7f5ce8196a80 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ce8105410 0x7f5ce819c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 --2- 172.21.15.3:0/210632811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ce8100340 0x7f5ce819c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 >> 172.21.15.3:0/210632811 conn(0x7f5ce8077220 msgr2=0x7f5ce80fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 shutdown_connections 2024-02-09T23:01:08.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:08.779+0000 7f5cef076640 1 -- 172.21.15.3:0/210632811 wait complete. 2024-02-09T23:01:09.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 320938 3.09 MB/sec execute 423 sec latency 782.458 ms 2024-02-09T23:01:10.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:10 smithi003 ceph-mon[32678]: pgmap v309: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-02-09T23:01:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:10 smithi003 ceph-mon[32678]: osdmap e99: 12 total, 12 up, 12 in 2024-02-09T23:01:10.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 322015 3.09 MB/sec execute 424 sec latency 1227.858 ms 2024-02-09T23:01:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:10 smithi064 ceph-mon[36638]: pgmap v309: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-02-09T23:01:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:10 smithi064 ceph-mon[36638]: osdmap e99: 12 total, 12 up, 12 in 2024-02-09T23:01:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:10 smithi089 ceph-mon[36665]: pgmap v309: 97 pgs: 97 active+clean; 129 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.2 MiB/s wr, 246 op/s 2024-02-09T23:01:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:10 smithi089 ceph-mon[36665]: osdmap e99: 12 total, 12 up, 12 in 2024-02-09T23:01:11.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 323400 3.10 MB/sec execute 425 sec latency 45.429 ms 2024-02-09T23:01:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:12 smithi003 ceph-mon[32678]: pgmap v311: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 312 op/s 2024-02-09T23:01:12.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 324487 3.11 MB/sec execute 426 sec latency 415.239 ms 2024-02-09T23:01:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:12 smithi064 ceph-mon[36638]: pgmap v311: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 312 op/s 2024-02-09T23:01:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:12 smithi089 ceph-mon[36665]: pgmap v311: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.9 MiB/s wr, 312 op/s 2024-02-09T23:01:13.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 326572 3.13 MB/sec execute 427 sec latency 541.551 ms 2024-02-09T23:01:14.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:14 smithi003 ceph-mon[32678]: pgmap v312: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-02-09T23:01:14.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 327831 3.14 MB/sec execute 428 sec latency 13.951 ms 2024-02-09T23:01:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:14 smithi064 ceph-mon[36638]: pgmap v312: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-02-09T23:01:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:14 smithi089 ceph-mon[36665]: pgmap v312: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-02-09T23:01:15.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 328942 3.14 MB/sec execute 429 sec latency 209.292 ms 2024-02-09T23:01:16.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:16 smithi003 ceph-mon[32678]: pgmap v313: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-02-09T23:01:16.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 330547 3.16 MB/sec execute 430 sec latency 41.315 ms 2024-02-09T23:01:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:16 smithi064 ceph-mon[36638]: pgmap v313: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-02-09T23:01:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:16 smithi089 ceph-mon[36665]: pgmap v313: 97 pgs: 97 active+clean; 127 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-02-09T23:01:17.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 331577 3.16 MB/sec execute 431 sec latency 375.876 ms 2024-02-09T23:01:18.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:18 smithi003 ceph-mon[32678]: pgmap v314: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-02-09T23:01:18.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 331577 3.15 MB/sec execute 432 sec latency 1375.924 ms 2024-02-09T23:01:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:18 smithi064 ceph-mon[36638]: pgmap v314: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-02-09T23:01:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:18 smithi089 ceph-mon[36665]: pgmap v314: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-02-09T23:01:19.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 331577 3.15 MB/sec execute 433 sec latency 2376.018 ms 2024-02-09T23:01:20.529 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:20 smithi089 ceph-mon[36665]: pgmap v315: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-02-09T23:01:20.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:20 smithi003 ceph-mon[32678]: pgmap v315: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-02-09T23:01:20.659 INFO:tasks.workunit.client.0.smithi003.stdout: 1 332895 3.16 MB/sec execute 434 sec latency 2817.699 ms 2024-02-09T23:01:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:20 smithi064 ceph-mon[36638]: pgmap v315: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.7 MiB/s wr, 221 op/s 2024-02-09T23:01:21.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 334341 3.17 MB/sec execute 435 sec latency 14.629 ms 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.868+0000 7f3cf08d4640 1 -- 172.21.15.3:0/2144987687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec0696f0 msgr2=0x7f3cec105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/2144987687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec0696f0 0x7f3cec105e30 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009930 tx=0x7f3cd402bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 -- 172.21.15.3:0/2144987687 shutdown_connections 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/2144987687 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cec106510 0x7f3cec10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/2144987687 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec0696f0 0x7f3cec105e30 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/2144987687 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3cec068d40 0x7f3cec069120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 -- 172.21.15.3:0/2144987687 >> 172.21.15.3:0/2144987687 conn(0x7f3cec077220 msgr2=0x7f3cec077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 -- 172.21.15.3:0/2144987687 shutdown_connections 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 -- 172.21.15.3:0/2144987687 wait complete. 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 Processor -- start 2024-02-09T23:01:21.874 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.869+0000 7f3cf08d4640 1 -- start start 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3cec068d40 0x7f3cec19d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 0x7f3cec19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cec1976f0 0x7f3cec197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec1981c0 con 0x7f3cec106510 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec198330 con 0x7f3cec068d40 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cec1984f0 con 0x7f3cec1976f0 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 0x7f3cec19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 0x7f3cec19db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44480/0 (socket says 172.21.15.3:44480) 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 -- 172.21.15.3:0/1748146291 learned_addr learned my addr 172.21.15.3:0/1748146291 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cead9d640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cec1976f0 0x7f3cec197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cea59c640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3cec068d40 0x7f3cec19d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 -- 172.21.15.3:0/1748146291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cec1976f0 msgr2=0x7f3cec197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cec1976f0 0x7f3cec197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 -- 172.21.15.3:0/1748146291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3cec068d40 msgr2=0x7f3cec19d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3cec068d40 0x7f3cec19d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 -- 172.21.15.3:0/1748146291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cd4009560 con 0x7f3cec106510 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 0x7f3cec19db70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009230 tx=0x7f3cd402c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce37fe640 1 -- 172.21.15.3:0/1748146291 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4004350 con 0x7f3cec106510 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce37fe640 1 -- 172.21.15.3:0/1748146291 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cd40044b0 con 0x7f3cec106510 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cec1a4430 con 0x7f3cec106510 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.870+0000 7f3ce37fe640 1 -- 172.21.15.3:0/1748146291 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd4003da0 con 0x7f3cec106510 2024-02-09T23:01:21.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.871+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cec1a4910 con 0x7f3cec106510 2024-02-09T23:01:21.881 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3ce37fe640 1 -- 172.21.15.3:0/1748146291 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3cd4004df0 con 0x7f3cec106510 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 Processor -- start 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 -- start start 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33ba260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33bcb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc331fa00 con 0x562bc33b19d0 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc331cf30 con 0x562bc33b10b0 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3cf3583740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc3318710 con 0x562bc33ba7a0 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.873+0000 7f3ce37fe640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3cd80a2220 0x7f3cd80a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33ba260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cead9d640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3cd80a2220 0x7f3cd80a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33ba260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44482/0 (socket says 172.21.15.3:44482) 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 -- 172.21.15.3:0/4060921329 learned_addr learned my addr 172.21.15.3:0/4060921329 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:21.882 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33bcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cead9d640 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3ce37fe640 1 -- 172.21.15.3:0/1748146291 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3cd40f0100 con 0x7f3cec106510 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 -- 172.21.15.3:0/4060921329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 msgr2=0x562bc33bcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33bcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 -- 172.21.15.3:0/4060921329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 msgr2=0x562bc33b1490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b1490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 -- 172.21.15.3:0/4060921329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bc32dd3b0 con 0x562bc33b19d0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cea59c640 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33ba260 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009c80 tx=0x7f3cdc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a7008ab98d3f97f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3cead9d640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3cd80a2220 0x7f3cd80a46e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4005fd0 tx=0x7f3ce4009b60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3ce0ff9640 1 -- 172.21.15.3:0/4060921329 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc009750 con 0x562bc33b19d0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3ce0ff9640 1 -- 172.21.15.3:0/4060921329 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc012930 con 0x562bc33b19d0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.874+0000 7f3ce0ff9640 1 -- 172.21.15.3:0/4060921329 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00fac0 con 0x562bc33b19d0 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 -- 172.21.15.3:0/4060921329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 msgr2=0x562bc33ba260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.883 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33ba260 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009c80 tx=0x7f3cdc0117b0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 -- 172.21.15.3:0/4060921329 shutdown_connections 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33bcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33ba260 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 --2- 172.21.15.3:0/4060921329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 -- 172.21.15.3:0/4060921329 >> 172.21.15.3:0/4060921329 conn(0x562bc33ab9d0 msgr2=0x562bc33abdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 -- 172.21.15.3:0/4060921329 shutdown_connections 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 -- 172.21.15.3:0/4060921329 wait complete. 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.875+0000 7f3cf3583740 1 -- start start 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 Processor -- start 2024-02-09T23:01:21.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33b81c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc32bdb90 con 0x562bc33b19d0 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc2fec250 con 0x562bc33b10b0 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc329e200 con 0x562bc33ba7a0 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cf3583740 10 client.0 _ll_register_callbacks cb 0x562bc32d45d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33b7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44494/0 (socket says 172.21.15.3:44494) 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 -- 172.21.15.3:0/2449127075 learned_addr learned my addr 172.21.15.3:0/2449127075 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cea59c640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cead9d640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33b81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 -- 172.21.15.3:0/2449127075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 msgr2=0x562bc33b81c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33b81c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 -- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 msgr2=0x562bc33b7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 -- 172.21.15.3:0/2449127075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cdc0098b0 con 0x562bc33b19d0 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cea59c640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cead9d640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:21.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33b7c80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40ed9c0 tx=0x7f3cd4047f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ede4daaae0ad2c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.876+0000 7f3cd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/2449127075 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40c55f0 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/2449127075 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3cd40bd300 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562bc333eee0 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/2449127075 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cd40bcc00 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 10 client.19128 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x562bc3316d90 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/2449127075 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562bc333eee0 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/2449127075 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3cd403c070 con 0x562bc33b19d0 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 10 client.19128 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 10 client.19128 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 20 client.19128 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc33ca7a0 0x562bc33ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 4 client.19128 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562bc3018600 con 0x562bc33ca7a0 2024-02-09T23:01:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.877+0000 7f3cea59c640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc33ca7a0 0x562bc33ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.878+0000 7f3cd1ffb640 10 client.19128 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:21.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.878+0000 7f3cea59c640 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc33ca7a0 0x562bc33ccb90 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c9df8277bccb7d server_cookie=c4c00c3ca5c9d94b in_seq=0 out_seq=0 2024-02-09T23:01:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/2449127075 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562bc3018600 con 0x562bc33ca7a0 2024-02-09T23:01:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cd1ffb640 10 client.19128 handle_command_reply: tid=0 2024-02-09T23:01:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 client.19128 shutdown 2024-02-09T23:01:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 msgr2=0x562bc33b7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33b7c80 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40ed9c0 tx=0x7f3cd4047f90 comp rx=0 tx=0).stop 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 shutdown_connections 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc33ca7a0 0x562bc33ccb90 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33ba7a0 0x562bc33b81c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33b7c80 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 --2- 172.21.15.3:0/2449127075 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33b10b0 0x562bc33b7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 >> 172.21.15.3:0/2449127075 conn(0x562bc33ab9d0 msgr2=0x562bc33ad720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cd1ffb640 0 client.19128 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 shutdown_connections 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 1 -- 172.21.15.3:0/2449127075 wait complete. 2024-02-09T23:01:21.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.880+0000 7f3cf3583740 20 client.19128 trim_cache size 0 max 16384 2024-02-09T23:01:21.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 Processor -- start 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 -- start start 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 0x562bc33b9e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b19d0 0x562bc33d14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33b6010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc33ab1b0 con 0x562bc33b10b0 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc33ab500 con 0x562bc33ba7a0 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cf3583740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc33b31f0 con 0x562bc33b19d0 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 0x562bc33b9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 0x562bc33b9e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44502/0 (socket says 172.21.15.3:44502) 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 -- 172.21.15.3:0/671554769 learned_addr learned my addr 172.21.15.3:0/671554769 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:21.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b19d0 0x562bc33d14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cead9d640 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33b6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 -- 172.21.15.3:0/671554769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b19d0 msgr2=0x562bc33d14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b19d0 0x562bc33d14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 -- 172.21.15.3:0/671554769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 msgr2=0x562bc33b6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33b6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 -- 172.21.15.3:0/671554769 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562bc32b3820 con 0x562bc33b10b0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.894+0000 7f3cea59c640 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 0x562bc33b9e70 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009c50 tx=0x7f3cdc00f660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ca7143027a1bb19 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cd3fff640 1 -- 172.21.15.3:0/671554769 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc012650 con 0x562bc33b10b0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cd3fff640 1 -- 172.21.15.3:0/671554769 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc00f440 con 0x562bc33b10b0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cd3fff640 1 -- 172.21.15.3:0/671554769 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cdc00fad0 con 0x562bc33b10b0 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 -- 172.21.15.3:0/671554769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 msgr2=0x562bc33b9e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 0x562bc33b9e70 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f3cdc009c50 tx=0x7f3cdc00f660 comp rx=0 tx=0).stop 2024-02-09T23:01:21.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 -- 172.21.15.3:0/671554769 shutdown_connections 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33b6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b19d0 0x562bc33d14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 --2- 172.21.15.3:0/671554769 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b10b0 0x562bc33b9e70 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 -- 172.21.15.3:0/671554769 >> 172.21.15.3:0/671554769 conn(0x562bc33ab9d0 msgr2=0x562bc33b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 -- 172.21.15.3:0/671554769 shutdown_connections 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.895+0000 7f3cf3583740 1 -- 172.21.15.3:0/671554769 wait complete. 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 -- start start 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 Processor -- start 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b10b0 0x562bc33c1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33c1660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33bf650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc33bfc10 con 0x562bc33b19d0 2024-02-09T23:01:21.904 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc33bfd80 con 0x562bc33ba7a0 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562bc33bff40 con 0x562bc33b10b0 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cf3583740 10 client.0 _ll_register_callbacks cb 0x7f3cd4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33c1660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33c1660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44518/0 (socket says 172.21.15.3:44518) 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 -- 172.21.15.3:0/50211615 learned_addr learned my addr 172.21.15.3:0/50211615 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cea59c640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b10b0 0x562bc33c1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.905 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3cead9d640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33bf650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 -- 172.21.15.3:0/50211615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b10b0 msgr2=0x562bc33c1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b10b0 0x562bc33c1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 -- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 msgr2=0x562bc33bf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33bf650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.896+0000 7f3ce3fff640 1 -- 172.21.15.3:0/50211615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562bc32dd3b0 con 0x562bc33b19d0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3ce3fff640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33c1660 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40b95d0 tx=0x7f3cd40bb110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b86b669ec491d152 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cd1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/50211615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cd40b8ea0 con 0x562bc33b19d0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/50211615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3cd40bd300 con 0x562bc33b19d0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3cdc012930 con 0x562bc33b19d0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/50211615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3cd40bcc00 con 0x562bc33b19d0 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 10 client.19134 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:21.906 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x562bc32d96f0 con 0x562bc33b19d0 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/50211615 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cdc012930 con 0x562bc33b19d0 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/50211615 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f3cd403c040 con 0x562bc33b19d0 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 10 client.19134 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 10 client.19134 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 20 client.19134 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc3421910 0x562bc3423d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 4 client.19134 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562bc30691b0 con 0x562bc3421910 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.897+0000 7f3cea59c640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc3421910 0x562bc3423d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.898+0000 7f3cea59c640 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc3421910 0x562bc3423d00 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ebaf024d2ec7d525 server_cookie=f68342f16d36bd4e in_seq=0 out_seq=0 2024-02-09T23:01:21.907 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.898+0000 7f3cd1ffb640 10 client.19134 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cd1ffb640 1 -- 172.21.15.3:0/50211615 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562bc30691b0 con 0x562bc3421910 2024-02-09T23:01:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cd1ffb640 10 client.19134 handle_command_reply: tid=0 2024-02-09T23:01:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 client.19134 shutdown 2024-02-09T23:01:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 msgr2=0x562bc33c1660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33c1660 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f3cd40b95d0 tx=0x7f3cd40bb110 comp rx=0 tx=0).stop 2024-02-09T23:01:21.938 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 shutdown_connections 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562bc3421910 0x562bc3423d00 crc :-1 s=READY pgs=282 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562bc33ba7a0 0x562bc33bf650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562bc33b19d0 0x562bc33c1660 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cd1ffb640 0 client.19134 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 --2- 172.21.15.3:0/50211615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562bc33b10b0 0x562bc33c1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 >> 172.21.15.3:0/50211615 conn(0x562bc33ab9d0 msgr2=0x562bc33d14f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 shutdown_connections 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 1 -- 172.21.15.3:0/50211615 wait complete. 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.935+0000 7f3cf3583740 20 client.19134 trim_cache size 0 max 16384 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:01:21.939 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:01:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:01:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:01:21.940 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3cd80a2220 msgr2=0x7f3cd80a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3cd80a2220 0x7f3cd80a46e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f3ce4005fd0 tx=0x7f3ce4009b60 comp rx=0 tx=0).stop 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 msgr2=0x7f3cec19db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 0x7f3cec19db70 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f3cd4009230 tx=0x7f3cd402c680 comp rx=0 tx=0).stop 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 shutdown_connections 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3cd80a2220 0x7f3cd80a46e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3cec1976f0 0x7f3cec197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3cec106510 0x7f3cec19db70 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 --2- 172.21.15.3:0/1748146291 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3cec068d40 0x7f3cec19d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 >> 172.21.15.3:0/1748146291 conn(0x7f3cec077220 msgr2=0x7f3cec0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 shutdown_connections 2024-02-09T23:01:21.941 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:21.936+0000 7f3cf08d4640 1 -- 172.21.15.3:0/1748146291 wait complete. 2024-02-09T23:01:22.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:22 smithi003 ceph-mon[32678]: pgmap v316: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-02-09T23:01:22.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 335120 3.17 MB/sec execute 436 sec latency 561.991 ms 2024-02-09T23:01:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:22 smithi064 ceph-mon[36638]: pgmap v316: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-02-09T23:01:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:22 smithi089 ceph-mon[36665]: pgmap v316: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.8 MiB/s wr, 263 op/s 2024-02-09T23:01:23.283 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:01:23.283 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:01:23.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 -- 172.21.15.3:0/1193681598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 msgr2=0x7f77dc10d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/1193681598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc10d640 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7f77c8009900 tx=0x7f77c802bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 -- 172.21.15.3:0/1193681598 shutdown_connections 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/1193681598 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc10db80 0x7f77dc10ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/1193681598 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc10d640 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/1193681598 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77dc0fffd0 0x7f77dc1003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.392+0000 7f77e2c1e640 1 -- 172.21.15.3:0/1193681598 >> 172.21.15.3:0/1193681598 conn(0x7f77dc0772e0 msgr2=0x7f77dc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 -- 172.21.15.3:0/1193681598 shutdown_connections 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 -- 172.21.15.3:0/1193681598 wait complete. 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 Processor -- start 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 -- start start 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77dc0fffd0 0x7f77dc19f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc19fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc10db80 0x7f77dc1998e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc199f30 con 0x7f77dc1008f0 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc19a0a0 con 0x7f77dc0fffd0 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77e2c1e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77dc19a260 con 0x7f77dc10db80 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.393+0000 7f77dbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc19fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77dc0fffd0 0x7f77dc19f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc19fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44548/0 (socket says 172.21.15.3:44548) 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 -- 172.21.15.3:0/3988869742 learned_addr learned my addr 172.21.15.3:0/3988869742 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77e11bb640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc10db80 0x7f77dc1998e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 -- 172.21.15.3:0/3988869742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc10db80 msgr2=0x7f77dc1998e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc10db80 0x7f77dc1998e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 -- 172.21.15.3:0/3988869742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77dc0fffd0 msgr2=0x7f77dc19f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77dc0fffd0 0x7f77dc19f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 -- 172.21.15.3:0/3988869742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c8009560 con 0x7f77dc1008f0 2024-02-09T23:01:23.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77dbfff640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc19fc70 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f77c8005940 tx=0x7f77c802c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77d9ffb640 1 -- 172.21.15.3:0/3988869742 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8004350 con 0x7f77dc1008f0 2024-02-09T23:01:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77d9ffb640 1 -- 172.21.15.3:0/3988869742 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77c80044b0 con 0x7f77dc1008f0 2024-02-09T23:01:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77dc19a4e0 con 0x7f77dc1008f0 2024-02-09T23:01:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77d9ffb640 1 -- 172.21.15.3:0/3988869742 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77c8003d30 con 0x7f77dc1008f0 2024-02-09T23:01:23.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.394+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77dc1a67d0 con 0x7f77dc1008f0 2024-02-09T23:01:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.396+0000 7f77d9ffb640 1 -- 172.21.15.3:0/3988869742 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f77c8004df0 con 0x7f77dc1008f0 2024-02-09T23:01:23.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 Processor -- start 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 -- start start 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77d9ffb640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f77cc0a22b0 0x7f77cc0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba073d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581ba02ed00 con 0x5581ba073980 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b9fe32f0 con 0x5581ba074330 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e57f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b9fe0820 con 0x5581ba07ceb0 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e11bb640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f77cc0a22b0 0x7f77cc0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba073d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba073d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44550/0 (socket says 172.21.15.3:44550) 2024-02-09T23:01:23.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e09ba640 1 -- 172.21.15.3:0/1378043343 learned_addr learned my addr 172.21.15.3:0/1378043343 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77d9ffb640 1 -- 172.21.15.3:0/3988869742 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f77c80f0590 con 0x7f77dc1008f0 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77e11bb640 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.397+0000 7f77dbfff640 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e09ba640 1 -- 172.21.15.3:0/1378043343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 msgr2=0x5581ba07f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e09ba640 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e09ba640 1 -- 172.21.15.3:0/1378043343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 msgr2=0x5581ba07c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e09ba640 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e09ba640 1 -- 172.21.15.3:0/1378043343 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581b9f9dff0 con 0x5581ba073980 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77dbfff640 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e11bb640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f77cc0a22b0 0x7f77cc0a4770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f77dc06a920 tx=0x7f77d4009dd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77e09ba640 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba073d60 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009c90 tx=0x7f77d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2267a180f99a58e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77c6ffd640 1 -- 172.21.15.3:0/1378043343 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0019620 con 0x5581ba073980 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77c6ffd640 1 -- 172.21.15.3:0/1378043343 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77d0019c60 con 0x5581ba073980 2024-02-09T23:01:23.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.398+0000 7f77c6ffd640 1 -- 172.21.15.3:0/1378043343 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0018aa0 con 0x5581ba073980 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- 172.21.15.3:0/1378043343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 msgr2=0x5581ba073d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba073d60 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7f77d0009c90 tx=0x7f77d0011810 comp rx=0 tx=0).stop 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- 172.21.15.3:0/1378043343 shutdown_connections 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1378043343 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba073d60 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- 172.21.15.3:0/1378043343 >> 172.21.15.3:0/1378043343 conn(0x5581ba06e050 msgr2=0x5581ba06e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- 172.21.15.3:0/1378043343 shutdown_connections 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- 172.21.15.3:0/1378043343 wait complete. 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- start start 2024-02-09T23:01:23.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 Processor -- start 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba079db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b9fdc000 con 0x5581ba073980 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b9caf430 con 0x5581ba074330 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.399+0000 7f77e57f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b9c6be00 con 0x5581ba07ceb0 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e57f1740 10 client.0 _ll_register_callbacks cb 0x5581b9fd07f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba079db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba079db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44558/0 (socket says 172.21.15.3:44558) 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 -- 172.21.15.3:0/2564107712 learned_addr learned my addr 172.21.15.3:0/2564107712 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77dbfff640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e11bb640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 -- 172.21.15.3:0/2564107712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 msgr2=0x5581ba07a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 -- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 msgr2=0x5581ba07a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 -- 172.21.15.3:0/2564107712 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d00098f0 con 0x5581ba073980 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e09ba640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba079db0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f77d00186a0 tx=0x7f77d0005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6913648b4c0fe534 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77bbfff640 1 -- 172.21.15.3:0/2564107712 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0018800 con 0x5581ba073980 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77bbfff640 1 -- 172.21.15.3:0/2564107712 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f77d0018960 con 0x5581ba073980 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5581b9cdb7e0 con 0x5581ba073980 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77bbfff640 1 -- 172.21.15.3:0/2564107712 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77d0018b20 con 0x5581ba073980 2024-02-09T23:01:23.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e57f1740 10 client.19146 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.400+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5581b9fda680 con 0x5581ba073980 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77bbfff640 1 -- 172.21.15.3:0/2564107712 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5581b9cdb7e0 con 0x5581ba073980 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77bbfff640 1 -- 172.21.15.3:0/2564107712 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f77d0021040 con 0x5581ba073980 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77e57f1740 10 client.19146 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77e57f1740 10 client.19146 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77e57f1740 20 client.19146 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77e57f1740 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba08d3f0 0x5581ba08f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77e57f1740 4 client.19146 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581b9d18720 con 0x5581ba08d3f0 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77dbfff640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba08d3f0 0x5581ba08f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77dbfff640 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba08d3f0 0x5581ba08f800 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d36854637f6ae08 server_cookie=9f16e620b463d745 in_seq=0 out_seq=0 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.401+0000 7f77bbfff640 10 client.19146 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:23.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77bbfff640 1 -- 172.21.15.3:0/2564107712 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5581b9d18720 con 0x5581ba08d3f0 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77bbfff640 10 client.19146 handle_command_reply: tid=0 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 client.19146 shutdown 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 msgr2=0x5581ba079db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba079db0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f77d00186a0 tx=0x7f77d0005720 comp rx=0 tx=0).stop 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 shutdown_connections 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba08d3f0 0x5581ba08f800 crc :-1 s=READY pgs=283 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba074330 0x5581ba07a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba07ceb0 0x5581ba07a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 --2- 172.21.15.3:0/2564107712 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba079db0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77bbfff640 0 client.19146 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 >> 172.21.15.3:0/2564107712 conn(0x5581ba06e050 msgr2=0x5581ba06fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 shutdown_connections 2024-02-09T23:01:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 1 -- 172.21.15.3:0/2564107712 wait complete. 2024-02-09T23:01:23.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.402+0000 7f77e57f1740 20 client.19146 trim_cache size 0 max 16384 2024-02-09T23:01:23.422 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 Processor -- start 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 -- start start 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba07c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba074330 0x5581ba07a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba077010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581b9f60fd0 con 0x5581ba073980 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581ba043ab0 con 0x5581ba07ceb0 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e57f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581ba094a50 con 0x5581ba074330 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba07c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba07c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44566/0 (socket says 172.21.15.3:44566) 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 -- 172.21.15.3:0/1244020556 learned_addr learned my addr 172.21.15.3:0/1244020556 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77dbfff640 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba074330 0x5581ba07a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e11bb640 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba077010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 -- 172.21.15.3:0/1244020556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba074330 msgr2=0x5581ba07a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba074330 0x5581ba07a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 -- 172.21.15.3:0/1244020556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 msgr2=0x5581ba077010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba077010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 -- 172.21.15.3:0/1244020556 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581b9f76b30 con 0x5581ba073980 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.418+0000 7f77e09ba640 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba07c4e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f77d00098c0 tx=0x7f77d0005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=719ec0cd148ea0c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77c67fc640 1 -- 172.21.15.3:0/1244020556 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0004bb0 con 0x5581ba073980 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77c67fc640 1 -- 172.21.15.3:0/1244020556 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77d0004620 con 0x5581ba073980 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77c67fc640 1 -- 172.21.15.3:0/1244020556 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d0018930 con 0x5581ba073980 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 -- 172.21.15.3:0/1244020556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 msgr2=0x5581ba07c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba07c4e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f77d00098c0 tx=0x7f77d0005b90 comp rx=0 tx=0).stop 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 -- 172.21.15.3:0/1244020556 shutdown_connections 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba077010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.424 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba074330 0x5581ba07a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 --2- 172.21.15.3:0/1244020556 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba073980 0x5581ba07c4e0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 -- 172.21.15.3:0/1244020556 >> 172.21.15.3:0/1244020556 conn(0x5581ba06e050 msgr2=0x5581ba0750c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 -- 172.21.15.3:0/1244020556 shutdown_connections 2024-02-09T23:01:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.419+0000 7f77e57f1740 1 -- 172.21.15.3:0/1244020556 wait complete. 2024-02-09T23:01:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 -- start start 2024-02-09T23:01:23.425 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 Processor -- start 2024-02-09T23:01:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba073980 0x5581ba082a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.428 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 0x5581ba082f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba083480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581ba0e3930 con 0x5581ba074330 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581ba0e3a70 con 0x5581ba07ceb0 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581ba0e3c30 con 0x5581ba073980 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e57f1740 10 client.0 _ll_register_callbacks cb 0x7f77d00097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77dbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 0x5581ba082f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e11bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba083480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e09ba640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba073980 0x5581ba082a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e11bb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba083480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52980/0 (socket says 172.21.15.3:52980) 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77dbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 0x5581ba082f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44574/0 (socket says 172.21.15.3:44574) 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e11bb640 1 -- 172.21.15.3:0/3746042822 learned_addr learned my addr 172.21.15.3:0/3746042822 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.420+0000 7f77e09ba640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba073980 0x5581ba082a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57380/0 (socket says 172.21.15.3:57380) 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e11bb640 1 -- 172.21.15.3:0/3746042822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba073980 msgr2=0x5581ba082a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e11bb640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba073980 0x5581ba082a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.429 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e11bb640 1 -- 172.21.15.3:0/3746042822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 msgr2=0x5581ba082f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e11bb640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 0x5581ba082f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e11bb640 1 -- 172.21.15.3:0/3746042822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581b9fda680 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77dbfff640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 0x5581ba082f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e11bb640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba083480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f77d4011e60 tx=0x7f77d401d000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e53707476ab8687 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77bbfff640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77bbfff640 1 -- 172.21.15.3:0/3746042822 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77d4026040 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77bbfff640 1 -- 172.21.15.3:0/3746042822 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f77d4003e00 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f77d0019c60 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77bbfff640 1 -- 172.21.15.3:0/3746042822 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f77d40047c0 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e57f1740 10 client.28016 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.421+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5581b9f9c850 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77bbfff640 1 -- 172.21.15.3:0/3746042822 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77d0019c60 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77bbfff640 1 -- 172.21.15.3:0/3746042822 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f77d4012c30 con 0x5581ba07ceb0 2024-02-09T23:01:23.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 10 client.28016 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 10 client.28016 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 20 client.28016 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba0e54f0 0x5581ba0e7910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 4 client.28016 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5581b9d67890 con 0x5581ba0e54f0 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e09ba640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba0e54f0 0x5581ba0e7910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e09ba640 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba0e54f0 0x5581ba0e7910 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=151f89243e4ba95f server_cookie=fbe36b1daf91252f in_seq=0 out_seq=0 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77bbfff640 10 client.28016 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77bbfff640 1 -- 172.21.15.3:0/3746042822 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5581b9d67890 con 0x5581ba0e54f0 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77bbfff640 10 client.28016 handle_command_reply: tid=0 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.422+0000 7f77e57f1740 1 client.28016 shutdown 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 msgr2=0x5581ba083480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba083480 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f77d4011e60 tx=0x7f77d401d000 comp rx=0 tx=0).stop 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 shutdown_connections 2024-02-09T23:01:23.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5581ba0e54f0 0x5581ba0e7910 crc :-1 s=READY pgs=284 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5581ba07ceb0 0x5581ba083480 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5581ba074330 0x5581ba082f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 --2- 172.21.15.3:0/3746042822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5581ba073980 0x5581ba082a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77bbfff640 0 client.28016 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 >> 172.21.15.3:0/3746042822 conn(0x5581ba06e050 msgr2=0x5581ba06c960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 shutdown_connections 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 1 -- 172.21.15.3:0/3746042822 wait complete. 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e57f1740 20 client.28016 trim_cache size 0 max 16384 2024-02-09T23:01:23.432 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:01:23.433 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:01:23.433 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:01:23.433 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:01:23.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f77cc0a22b0 msgr2=0x7f77cc0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f77cc0a22b0 0x7f77cc0a4770 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f77dc06a920 tx=0x7f77d4009dd0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 msgr2=0x7f77dc19fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc19fc70 secure :-1 s=READY pgs=1504 cs=0 l=1 rev1=1 crypto rx=0x7f77c8005940 tx=0x7f77c802c710 comp rx=0 tx=0).stop 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 shutdown_connections 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f77cc0a22b0 0x7f77cc0a4770 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f77dc10db80 0x7f77dc1998e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f77dc1008f0 0x7f77dc19fc70 unknown :-1 s=CLOSED pgs=1504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 --2- 172.21.15.3:0/3988869742 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f77dc0fffd0 0x7f77dc19f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 >> 172.21.15.3:0/3988869742 conn(0x7f77dc0772e0 msgr2=0x7f77dc10f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.423+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 shutdown_connections 2024-02-09T23:01:23.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.424+0000 7f77e2c1e640 1 -- 172.21.15.3:0/3988869742 wait complete. 2024-02-09T23:01:23.437 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:01:23.438 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:01:23.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 -- 172.21.15.3:0/3119227068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c0ffc90 msgr2=0x7eff3c100070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 --2- 172.21.15.3:0/3119227068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c0ffc90 0x7eff3c100070 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7eff30009930 tx=0x7eff3002bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:23.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 -- 172.21.15.3:0/3119227068 shutdown_connections 2024-02-09T23:01:23.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 --2- 172.21.15.3:0/3119227068 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c1065b0 0x7eff3c10e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 --2- 172.21.15.3:0/3119227068 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c100640 0x7eff3c105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 --2- 172.21.15.3:0/3119227068 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c0ffc90 0x7eff3c100070 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 -- 172.21.15.3:0/3119227068 >> 172.21.15.3:0/3119227068 conn(0x7eff3c076580 msgr2=0x7eff3c076990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 -- 172.21.15.3:0/3119227068 shutdown_connections 2024-02-09T23:01:23.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.547+0000 7eff4200e640 1 -- 172.21.15.3:0/3119227068 wait complete. 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 Processor -- start 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 -- start start 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 0x7eff3c19d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c1065b0 0x7eff3c19dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c197740 0x7eff3c197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c198210 con 0x7eff3c1065b0 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c198380 con 0x7eff3c197740 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.548+0000 7eff4200e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3c198540 con 0x7eff3c0ffc90 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 0x7eff3c19d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c1065b0 0x7eff3c19dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 0x7eff3c19d680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57402/0 (socket says 172.21.15.3:57402) 2024-02-09T23:01:23.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3bfff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c197740 0x7eff3c197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 -- 172.21.15.3:0/963775318 learned_addr learned my addr 172.21.15.3:0/963775318 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 -- 172.21.15.3:0/963775318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c197740 msgr2=0x7eff3c197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c197740 0x7eff3c197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 -- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c1065b0 msgr2=0x7eff3c19dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c1065b0 0x7eff3c19dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 -- 172.21.15.3:0/963775318 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30009560 con 0x7eff3c0ffc90 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3bfff640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c197740 0x7eff3c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3affd640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c1065b0 0x7eff3c19dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff3b7fe640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 0x7eff3c19d680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7eff30009230 tx=0x7eff3002c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30004350 con 0x7eff3c0ffc90 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff3c100b30 con 0x7eff3c0ffc90 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff3c101090 con 0x7eff3c0ffc90 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff300044b0 con 0x7eff3c0ffc90 2024-02-09T23:01:23.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.549+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30003d00 con 0x7eff3c0ffc90 2024-02-09T23:01:23.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.551+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff3c069530 con 0x7eff3c0ffc90 2024-02-09T23:01:23.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.552+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7eff30004df0 con 0x7eff3c0ffc90 2024-02-09T23:01:23.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.552+0000 7eff38ff9640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7eff240a2230 0x7eff240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.553+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7eff300efe50 con 0x7eff3c0ffc90 2024-02-09T23:01:23.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.553+0000 7eff3affd640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7eff240a2230 0x7eff240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.553+0000 7eff3affd640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7eff240a2230 0x7eff240a46f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff3c198f90 tx=0x7eff2c00f820 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.559+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7eff300b8b90 con 0x7eff3c0ffc90 2024-02-09T23:01:23.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 337217 3.19 MB/sec execute 437 sec latency 686.655 ms 2024-02-09T23:01:23.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.793+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7eff3c109760 con 0x7eff3c0ffc90 2024-02-09T23:01:23.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.794+0000 7eff38ff9640 1 -- 172.21.15.3:0/963775318 <== mon.1 v2:172.21.15.89:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+11361 (secure 0 0 0) 0x7eff30003e60 con 0x7eff3c0ffc90 2024-02-09T23:01:23.797 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:01:23.797 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":47,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":47,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T22:59:11.242420+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:01:23.798 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 47 2024-02-09T23:01:23.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7eff240a2230 msgr2=0x7eff240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7eff240a2230 0x7eff240a46f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7eff3c198f90 tx=0x7eff2c00f820 comp rx=0 tx=0).stop 2024-02-09T23:01:23.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 msgr2=0x7eff3c19d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 0x7eff3c19d680 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7eff30009230 tx=0x7eff3002c6c0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 shutdown_connections 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7eff240a2230 0x7eff240a46f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7eff3c197740 0x7eff3c197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7eff3c1065b0 0x7eff3c19dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 --2- 172.21.15.3:0/963775318 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7eff3c0ffc90 0x7eff3c19d680 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 >> 172.21.15.3:0/963775318 conn(0x7eff3c076580 msgr2=0x7eff3c0fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 shutdown_connections 2024-02-09T23:01:23.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.796+0000 7eff4200e640 1 -- 172.21.15.3:0/963775318 wait complete. 2024-02-09T23:01:23.810 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:23.811 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:01:23.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 -- 172.21.15.3:0/1030780205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c10b820 msgr2=0x7fbb2c10dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 --2- 172.21.15.3:0/1030780205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c10b820 0x7fbb2c10dc50 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009ca0 tx=0x7fbb2402c180 comp rx=0 tx=0).stop 2024-02-09T23:01:23.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 -- 172.21.15.3:0/1030780205 shutdown_connections 2024-02-09T23:01:23.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 --2- 172.21.15.3:0/1030780205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c10b820 0x7fbb2c10dc50 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 --2- 172.21.15.3:0/1030780205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 0x7fbb2c10b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 --2- 172.21.15.3:0/1030780205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c102450 0x7fbb2c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 -- 172.21.15.3:0/1030780205 >> 172.21.15.3:0/1030780205 conn(0x7fbb2c0772e0 msgr2=0x7fbb2c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 -- 172.21.15.3:0/1030780205 shutdown_connections 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.917+0000 7fbb30b77640 1 -- 172.21.15.3:0/1030780205 wait complete. 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 Processor -- start 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 -- start start 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c102450 0x7fbb2c19f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 0x7fbb2c19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 0x7fbb2c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c199eb0 con 0x7fbb2c102450 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c19a020 con 0x7fbb2c10b820 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb30b77640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2c19a1e0 con 0x7fbb2c102d70 2024-02-09T23:01:23.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 0x7fbb2c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 0x7fbb2c199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53006/0 (socket says 172.21.15.3:53006) 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 0x7fbb2c19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb2ad9d640 1 -- 172.21.15.3:0/2481808378 learned_addr learned my addr 172.21.15.3:0/2481808378 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.918+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c102450 0x7fbb2c19f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb23fff640 1 -- 172.21.15.3:0/2481808378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 msgr2=0x7fbb2c199810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 0x7fbb2c199810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb23fff640 1 -- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c102450 msgr2=0x7fbb2c19f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c102450 0x7fbb2c19f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb23fff640 1 -- 172.21.15.3:0/2481808378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24009900 con 0x7fbb2c102d70 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 0x7fbb2c199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c102450 0x7fbb2c19f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 0x7fbb2c19fbe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400e9f0 tx=0x7fbb1400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb237fe640 1 -- 172.21.15.3:0/2481808378 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1400ccc0 con 0x7fbb2c102d70 2024-02-09T23:01:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2c19a4c0 con 0x7fbb2c102d70 2024-02-09T23:01:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb237fe640 1 -- 172.21.15.3:0/2481808378 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb1400ce20 con 0x7fbb2c102d70 2024-02-09T23:01:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb237fe640 1 -- 172.21.15.3:0/2481808378 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb14004a00 con 0x7fbb2c102d70 2024-02-09T23:01:23.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.919+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2c1a66d0 con 0x7fbb2c102d70 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.921+0000 7fbb237fe640 1 -- 172.21.15.3:0/2481808378 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fbb140050b0 con 0x7fbb2c102d70 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.921+0000 7fbb33828740 1 Processor -- start 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 -- start start 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3fece10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3fed3e0 0x5582e3ff5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3ff5f60 0x5582e3ff83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3fa8020 con 0x5582e3feca30 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3f5b320 con 0x5582e3fed3e0 2024-02-09T23:01:23.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb33828740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3f58850 con 0x5582e3ff5f60 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3fece10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3fece10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44618/0 (socket says 172.21.15.3:44618) 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2ad9d640 1 -- 172.21.15.3:0/3259989926 learned_addr learned my addr 172.21.15.3:0/3259989926 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3fed3e0 0x5582e3ff5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb237fe640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb180a2230 0x7fbb180a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb23fff640 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3ff5f60 0x5582e3ff83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 -- 172.21.15.3:0/3259989926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3ff5f60 msgr2=0x5582e3ff83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3ff5f60 0x5582e3ff83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 -- 172.21.15.3:0/3259989926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 msgr2=0x5582e3fece10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3fece10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 -- 172.21.15.3:0/3259989926 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582e3f16210 con 0x5582e3fed3e0 2024-02-09T23:01:23.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb180a2230 0x7fbb180a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb237fe640 1 -- 172.21.15.3:0/2481808378 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fbb14014070 con 0x7fbb2c102d70 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.922+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3fed3e0 0x5582e3ff5990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009c40 tx=0x7fbb1c010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d554a45afa957e9e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb20ff9640 1 -- 172.21.15.3:0/3259989926 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c004030 con 0x5582e3fed3e0 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb20ff9640 1 -- 172.21.15.3:0/3259989926 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb1c0035f0 con 0x5582e3fed3e0 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb20ff9640 1 -- 172.21.15.3:0/3259989926 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c0126b0 con 0x5582e3fed3e0 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 -- 172.21.15.3:0/3259989926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3fed3e0 msgr2=0x5582e3ff5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb180a2230 0x7fbb180a46f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbb24032730 tx=0x7fbb2402c180 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3fed3e0 0x5582e3ff5990 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009c40 tx=0x7fbb1c010740 comp rx=0 tx=0).stop 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 -- 172.21.15.3:0/3259989926 shutdown_connections 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3fed3e0 0x5582e3ff5990 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3ff5f60 0x5582e3ff83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 --2- 172.21.15.3:0/3259989926 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3fece10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 -- 172.21.15.3:0/3259989926 >> 172.21.15.3:0/3259989926 conn(0x5582e3fe7100 msgr2=0x5582e3fe74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 -- 172.21.15.3:0/3259989926 shutdown_connections 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.923+0000 7fbb33828740 1 -- 172.21.15.3:0/3259989926 wait complete. 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 -- start start 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 Processor -- start 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3feca30 0x5582e3ff2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 0x5582e3ff3940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3f54030 con 0x5582e3ff5f60 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3c27430 con 0x5582e3feca30 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3be3e00 con 0x5582e3fed3e0 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb33828740 10 client.0 _ll_register_callbacks cb 0x5582e3f49030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:23.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 0x5582e3ff3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 0x5582e3ff3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44624/0 (socket says 172.21.15.3:44624) 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 -- 172.21.15.3:0/2407278838 learned_addr learned my addr 172.21.15.3:0/2407278838 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3feca30 0x5582e3ff2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 -- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 msgr2=0x5582e3ff3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 -- 172.21.15.3:0/2407278838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3feca30 msgr2=0x5582e3ff2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3feca30 0x5582e3ff2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.924+0000 7fbb2ad9d640 1 -- 172.21.15.3:0/2407278838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c009870 con 0x5582e3ff5f60 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 0x5582e3ff3940 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fbb24005e30 tx=0x7fbb240095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e928356e78fabebd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:23.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2407278838 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24044bc0 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5582e3c537e0 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2407278838 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fbb24043460 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2407278838 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb24043620 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 10 client.19167 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5582e3f526b0 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2407278838 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5582e3c537e0 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2407278838 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbb24044d20 con 0x5582e3ff5f60 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 10 client.19167 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 10 client.19167 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 20 client.19167 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e4008820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 4 client.19167 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.925+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582e3c90720 con 0x5582e4006430 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.926+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e4008820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.926+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e4008820 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=fe750fbf74e96300 server_cookie=8d67a68bea32250f in_seq=0 out_seq=0 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.926+0000 7fbb11ffb640 10 client.19167 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2407278838 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5582e3c90720 con 0x5582e4006430 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb11ffb640 10 client.19167 handle_command_reply: tid=0 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 client.19167 shutdown 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 msgr2=0x5582e3ff3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 0x5582e3ff3940 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7fbb24005e30 tx=0x7fbb240095c0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 shutdown_connections 2024-02-09T23:01:23.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e4008820 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3ff5f60 0x5582e3ff3940 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb11ffb640 0 client.19167 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 --2- 172.21.15.3:0/2407278838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3feca30 0x5582e3ff2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 >> 172.21.15.3:0/2407278838 conn(0x5582e3fe7100 msgr2=0x5582e3fe8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 shutdown_connections 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 1 -- 172.21.15.3:0/2407278838 wait complete. 2024-02-09T23:01:23.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.927+0000 7fbb33828740 20 client.19167 trim_cache size 0 max 16384 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 Processor -- start 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 -- start start 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3feca30 0x5582e3ff55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e3ff0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3ed9130 con 0x5582e3fed3e0 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3c90720 con 0x5582e3ff5f60 2024-02-09T23:01:23.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.939+0000 7fbb33828740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3fe7cd0 con 0x5582e3feca30 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb2a59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3feca30 0x5582e3ff55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44638/0 (socket says 172.21.15.3:44638) 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 -- 172.21.15.3:0/2099925282 learned_addr learned my addr 172.21.15.3:0/2099925282 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e3ff0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 -- 172.21.15.3:0/2099925282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3feca30 msgr2=0x5582e3ff55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3feca30 0x5582e3ff55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 -- 172.21.15.3:0/2099925282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 msgr2=0x5582e3ff0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e3ff0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 -- 172.21.15.3:0/2099925282 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582e3eeec90 con 0x5582e3fed3e0 2024-02-09T23:01:23.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3760 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c6a0 tx=0x7fbb14005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc13fa41d71176b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb13fff640 1 -- 172.21.15.3:0/2099925282 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1409b170 con 0x5582e3fed3e0 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb13fff640 1 -- 172.21.15.3:0/2099925282 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb1409ebc0 con 0x5582e3fed3e0 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.940+0000 7fbb13fff640 1 -- 172.21.15.3:0/2099925282 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb140a2d80 con 0x5582e3fed3e0 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 -- 172.21.15.3:0/2099925282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 msgr2=0x5582e3ff3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3760 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400c6a0 tx=0x7fbb14005020 comp rx=0 tx=0).stop 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 -- 172.21.15.3:0/2099925282 shutdown_connections 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e3ff0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3fed3e0 0x5582e3ff3760 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 --2- 172.21.15.3:0/2099925282 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3feca30 0x5582e3ff55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 -- 172.21.15.3:0/2099925282 >> 172.21.15.3:0/2099925282 conn(0x5582e3fe7100 msgr2=0x5582e400d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 -- 172.21.15.3:0/2099925282 shutdown_connections 2024-02-09T23:01:23.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 -- 172.21.15.3:0/2099925282 wait complete. 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 -- start start 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.941+0000 7fbb33828740 1 Processor -- start 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3ffc000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ffc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e405c930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e3ffcc60 con 0x5582e3feca30 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e405ce70 con 0x5582e3ff5f60 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582e405cfe0 con 0x5582e3fed3e0 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 10 client.0 _ll_register_callbacks cb 0x7fbb1c0095f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3ffc000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2ad9d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e405c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb23fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ffc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3ffc000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44654/0 (socket says 172.21.15.3:44654) 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 -- 172.21.15.3:0/2290306298 learned_addr learned my addr 172.21.15.3:0/2290306298 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 -- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 msgr2=0x5582e3ffc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ffc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 -- 172.21.15.3:0/2290306298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 msgr2=0x5582e405c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e405c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 -- 172.21.15.3:0/2290306298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2c1a66d0 con 0x5582e3feca30 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2ad9d640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e405c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ffc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb2a59c640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3ffc000 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009790 tx=0x7fbb1c009d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29616257cb79e189 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:23.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb11ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2290306298 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb1c021040 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2290306298 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fbb1c016070 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.942+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb1c0035f0 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2290306298 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb1c003b70 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 10 client.19173 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5582e3f14a70 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2290306298 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb1c034080 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2290306298 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fbb1c02e030 con 0x5582e3feca30 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 10 client.19173 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 10 client.19173 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 20 client.19173 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e40607e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:23.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 4 client.19173 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5582e3cdf890 con 0x5582e4006430 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.943+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e40607e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb23fff640 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e40607e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4b3814c670d8e8a2 server_cookie=8c31bfae7b7a0f4a in_seq=0 out_seq=0 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb11ffb640 10 client.19173 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb11ffb640 1 -- 172.21.15.3:0/2290306298 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5582e3cdf890 con 0x5582e4006430 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb11ffb640 10 client.19173 handle_command_reply: tid=0 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 client.19173 shutdown 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 msgr2=0x5582e3ffc000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3ffc000 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7fbb1c009790 tx=0x7fbb1c009d30 comp rx=0 tx=0).stop 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 shutdown_connections 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5582e4006430 0x5582e40607e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5582e3ff5f60 0x5582e405c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5582e3fed3e0 0x5582e3ffc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb11ffb640 0 client.19173 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 --2- 172.21.15.3:0/2290306298 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5582e3feca30 0x5582e3ffc000 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 >> 172.21.15.3:0/2290306298 conn(0x5582e3fe7100 msgr2=0x5582e3fe59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 shutdown_connections 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 1 -- 172.21.15.3:0/2290306298 wait complete. 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.944+0000 7fbb33828740 20 client.19173 trim_cache size 0 max 16384 2024-02-09T23:01:23.955 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:23.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb180a2230 msgr2=0x7fbb180a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb180a2230 0x7fbb180a46f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbb24032730 tx=0x7fbb2402c180 comp rx=0 tx=0).stop 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 msgr2=0x7fbb2c19fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 0x7fbb2c19fbe0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbb1400e9f0 tx=0x7fbb1400eeb0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 shutdown_connections 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fbb180a2230 0x7fbb180a46f0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fbb2c10b820 0x7fbb2c199810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fbb2c102d70 0x7fbb2c19fbe0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 --2- 172.21.15.3:0/2481808378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fbb2c102450 0x7fbb2c19f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 >> 172.21.15.3:0/2481808378 conn(0x7fbb2c0772e0 msgr2=0x7fbb2c1076a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 shutdown_connections 2024-02-09T23:01:23.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:23.945+0000 7fbb30b77640 1 -- 172.21.15.3:0/2481808378 wait complete. 2024-02-09T23:01:23.959 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:23.959 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:01:24.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 -- 172.21.15.3:0/3460285483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86580fffd0 msgr2=0x7f86581003b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 --2- 172.21.15.3:0/3460285483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86580fffd0 0x7f86581003b0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7f8650009900 tx=0x7f865002bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 -- 172.21.15.3:0/3460285483 shutdown_connections 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 --2- 172.21.15.3:0/3460285483 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86581008f0 0x7f865810d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 --2- 172.21.15.3:0/3460285483 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f865810db80 0x7f865810ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 --2- 172.21.15.3:0/3460285483 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f86580fffd0 0x7f86581003b0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 -- 172.21.15.3:0/3460285483 >> 172.21.15.3:0/3460285483 conn(0x7f86580772e0 msgr2=0x7f86580776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 -- 172.21.15.3:0/3460285483 shutdown_connections 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.063+0000 7f865d69d640 1 -- 172.21.15.3:0/3460285483 wait complete. 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 Processor -- start 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 -- start start 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580fffd0 0x7f865819f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.069 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86581008f0 0x7f865819fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 0x7f8658199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8658199f50 con 0x7f865810db80 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865819a0c0 con 0x7f86580fffd0 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f865d69d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f865819a280 con 0x7f86581008f0 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f86577fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 0x7f8658199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f86577fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 0x7f8658199900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44680/0 (socket says 172.21.15.3:44680) 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.064+0000 7f86577fe640 1 -- 172.21.15.3:0/2580732686 learned_addr learned my addr 172.21.15.3:0/2580732686 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580fffd0 0x7f865819f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86567fc640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86581008f0 0x7f865819fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86577fe640 1 -- 172.21.15.3:0/2580732686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86581008f0 msgr2=0x7f865819fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86577fe640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86581008f0 0x7f865819fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86577fe640 1 -- 172.21.15.3:0/2580732686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580fffd0 msgr2=0x7f865819f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86577fe640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580fffd0 0x7f865819f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86577fe640 1 -- 172.21.15.3:0/2580732686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8650009560 con 0x7f865810db80 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f86577fe640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 0x7f8658199900 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f864800ea80 tx=0x7f864800ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f864ffff640 1 -- 172.21.15.3:0/2580732686 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f864800ce20 con 0x7f865810db80 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f864ffff640 1 -- 172.21.15.3:0/2580732686 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8648005020 con 0x7f865810db80 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f864ffff640 1 -- 172.21.15.3:0/2580732686 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8648010690 con 0x7f865810db80 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f865819a560 con 0x7f865810db80 2024-02-09T23:01:24.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.065+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86581a67f0 con 0x7f865810db80 2024-02-09T23:01:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.067+0000 7f864ffff640 1 -- 172.21.15.3:0/2580732686 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8648005500 con 0x7f865810db80 2024-02-09T23:01:24.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 Processor -- start 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 -- start start 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e704ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 0x5602e7057990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e705a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e700a020 con 0x5602e7057f60 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e6fbd320 con 0x5602e704ea30 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8660279740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e6fba850 con 0x5602e704f3e0 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f864ffff640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86440a2230 0x7f86440a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f86567fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e704ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 0x5602e7057990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 0x5602e7057990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57470/0 (socket says 172.21.15.3:57470) 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f8656ffd640 1 -- 172.21.15.3:0/2563206474 learned_addr learned my addr 172.21.15.3:0/2563206474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f86567fc640 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e705a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f86577fe640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86440a2230 0x7f86440a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.068+0000 7f864ffff640 1 -- 172.21.15.3:0/2580732686 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8648014070 con 0x7f865810db80 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f8656ffd640 1 -- 172.21.15.3:0/2563206474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 msgr2=0x5602e704ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e704ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f8656ffd640 1 -- 172.21.15.3:0/2563206474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 msgr2=0x5602e705a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e705a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f8656ffd640 1 -- 172.21.15.3:0/2563206474 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602e6f78210 con 0x5602e704f3e0 2024-02-09T23:01:24.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f86567fc640 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e704ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 0x5602e7057990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8650009900 tx=0x7f8650004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d4c6913f1e2d7a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f86577fe640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86440a2230 0x7f86440a46f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f864809b060 tx=0x7f8648058040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f864d7fa640 1 -- 172.21.15.3:0/2563206474 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8650004a20 con 0x5602e704f3e0 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f864d7fa640 1 -- 172.21.15.3:0/2563206474 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8650004040 con 0x5602e704f3e0 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.069+0000 7f864d7fa640 1 -- 172.21.15.3:0/2563206474 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500051b0 con 0x5602e704f3e0 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 -- 172.21.15.3:0/2563206474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 msgr2=0x5602e7057990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 0x5602e7057990 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f8650009900 tx=0x7f8650004770 comp rx=0 tx=0).stop 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 -- 172.21.15.3:0/2563206474 shutdown_connections 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e705a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704f3e0 0x5602e7057990 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 --2- 172.21.15.3:0/2563206474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e704ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 -- 172.21.15.3:0/2563206474 >> 172.21.15.3:0/2563206474 conn(0x5602e7049100 msgr2=0x5602e70494e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 -- 172.21.15.3:0/2563206474 shutdown_connections 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 -- 172.21.15.3:0/2563206474 wait complete. 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 -- start start 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.070+0000 7f8660279740 1 Processor -- start 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7054f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e7055440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e7057f60 0x5602e7055980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e6fb6030 con 0x5602e704f3e0 2024-02-09T23:01:24.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e6c89430 con 0x5602e7057f60 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e6c45e00 con 0x5602e704ea30 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 10 client.0 _ll_register_callbacks cb 0x5602e6fab030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e7055440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7054f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e7055440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44704/0 (socket says 172.21.15.3:44704) 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8656ffd640 1 -- 172.21.15.3:0/2422263665 learned_addr learned my addr 172.21.15.3:0/2422263665 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e7057f60 0x5602e7055980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 -- 172.21.15.3:0/2422263665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e7057f60 msgr2=0x5602e7055980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e7057f60 0x5602e7055980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 -- 172.21.15.3:0/2422263665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 msgr2=0x5602e7055440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e7055440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 -- 172.21.15.3:0/2422263665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8658110b00 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e7055440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f86567fc640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7054f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f865819b0f0 tx=0x7f8640004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39dd823f93aee382 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f863e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f863e7fc640 1 -- 172.21.15.3:0/2422263665 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640004020 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.071+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5602e6cb57e0 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f863e7fc640 1 -- 172.21.15.3:0/2422263665 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8640004180 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f863e7fc640 1 -- 172.21.15.3:0/2422263665 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8640004340 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 10 client.28201 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5602e6fb46b0 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f863e7fc640 1 -- 172.21.15.3:0/2422263665 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5602e6cb57e0 con 0x5602e704ea30 2024-02-09T23:01:24.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f863e7fc640 1 -- 172.21.15.3:0/2422263665 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f8640029510 con 0x5602e704ea30 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 10 client.28201 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 10 client.28201 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 20 client.28201 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e7068360 0x5602e706a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 4 client.28201 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5602e6cf2720 con 0x5602e7068360 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.072+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e7068360 0x5602e706a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8656ffd640 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e7068360 0x5602e706a780 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=261f3a407fd6828c server_cookie=cdf5fb0f6896e4bc in_seq=0 out_seq=0 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f863e7fc640 10 client.28201 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:24.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f863e7fc640 1 -- 172.21.15.3:0/2422263665 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5602e6cf2720 con 0x5602e7068360 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f863e7fc640 10 client.28201 handle_command_reply: tid=0 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 client.28201 shutdown 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 msgr2=0x5602e7054f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7054f00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f865819b0f0 tx=0x7f8640004730 comp rx=0 tx=0).stop 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 shutdown_connections 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e7068360 0x5602e706a780 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e7057f60 0x5602e7055980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e7055440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f863e7fc640 0 client.28201 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 --2- 172.21.15.3:0/2422263665 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7054f00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 >> 172.21.15.3:0/2422263665 conn(0x5602e7049100 msgr2=0x5602e704ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 shutdown_connections 2024-02-09T23:01:24.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.073+0000 7f8660279740 1 -- 172.21.15.3:0/2422263665 wait complete. 2024-02-09T23:01:24.087 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.074+0000 7f8660279740 20 client.28201 trim_cache size 0 max 16384 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.085+0000 7f8660279740 1 Processor -- start 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.085+0000 7f8660279740 1 -- start start 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.085+0000 7f8660279740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7057630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.085+0000 7f8660279740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704f3e0 0x5602e7050250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.085+0000 7f8660279740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e7050790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.085+0000 7f8660279740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e6f3b130 con 0x5602e7057f60 2024-02-09T23:01:24.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f8660279740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e7064e70 con 0x5602e704f3e0 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f8660279740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e7054d40 con 0x5602e704ea30 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7057630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e7050790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e7050790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44706/0 (socket says 172.21.15.3:44706) 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 -- 172.21.15.3:0/1073774244 learned_addr learned my addr 172.21.15.3:0/1073774244 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f8656ffd640 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704f3e0 0x5602e7050250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 -- 172.21.15.3:0/1073774244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 msgr2=0x5602e7057630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7057630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 -- 172.21.15.3:0/1073774244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704f3e0 msgr2=0x5602e7050250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704f3e0 0x5602e7050250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 -- 172.21.15.3:0/1073774244 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5602e6f50c90 con 0x5602e7057f60 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f86567fc640 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e7050790 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8640010010 tx=0x7f8640000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b31c76b75c161 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f864cff9640 1 -- 172.21.15.3:0/1073774244 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640029b40 con 0x5602e7057f60 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f864cff9640 1 -- 172.21.15.3:0/1073774244 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f864000ae00 con 0x5602e7057f60 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.086+0000 7f864cff9640 1 -- 172.21.15.3:0/1073774244 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8640038c50 con 0x5602e7057f60 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 -- 172.21.15.3:0/1073774244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 msgr2=0x5602e7050790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e7050790 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 crypto rx=0x7f8640010010 tx=0x7f8640000bc0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 -- 172.21.15.3:0/1073774244 shutdown_connections 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704f3e0 0x5602e7050250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e7057f60 0x5602e7050790 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 --2- 172.21.15.3:0/1073774244 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e704ea30 0x5602e7057630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 -- 172.21.15.3:0/1073774244 >> 172.21.15.3:0/1073774244 conn(0x5602e7049100 msgr2=0x5602e70494e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 -- 172.21.15.3:0/1073774244 shutdown_connections 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 -- 172.21.15.3:0/1073774244 wait complete. 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 -- start start 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.087+0000 7f8660279740 1 Processor -- start 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e705f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e705d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e7057f60 0x5602e705ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e705e480 con 0x5602e704f3e0 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e70be850 con 0x5602e704ea30 2024-02-09T23:01:24.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602e70be990 con 0x5602e7057f60 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 10 client.0 _ll_register_callbacks cb 0x7f8650009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e705d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e705d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44720/0 (socket says 172.21.15.3:44720) 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 -- 172.21.15.3:0/3965850194 learned_addr learned my addr 172.21.15.3:0/3965850194 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f86567fc640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e705f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f86567fc640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e7057f60 0x5602e705ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 -- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e7057f60 msgr2=0x5602e705ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e7057f60 0x5602e705ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 -- 172.21.15.3:0/3965850194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 msgr2=0x5602e705f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e705f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 -- 172.21.15.3:0/3965850194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5602e6fb46b0 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f86567fc640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e705f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8656ffd640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e705d670 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8650047000 tx=0x7f865002c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cf02e1fc50a21c6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f863e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f863e7fc640 1 -- 172.21.15.3:0/3965850194 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86500048e0 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f863e7fc640 1 -- 172.21.15.3:0/3965850194 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8650050070 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8650004040 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.088+0000 7f863e7fc640 1 -- 172.21.15.3:0/3965850194 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8650053df0 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 10 client.19191 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5602e6f76a70 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f863e7fc640 1 -- 172.21.15.3:0/3965850194 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8650004040 con 0x5602e704f3e0 2024-02-09T23:01:24.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f863e7fc640 1 -- 172.21.15.3:0/3965850194 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f865003c050 con 0x5602e704f3e0 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 10 client.19191 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 10 client.19191 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 20 client.19191 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e70bfdd0 0x5602e70c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 4 client.19191 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5602e6d41890 con 0x5602e70bfdd0 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f86567fc640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e70bfdd0 0x5602e70c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f863e7fc640 10 client.19191 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.089+0000 7f86567fc640 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e70bfdd0 0x5602e70c21f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=375fbe6f0020952f server_cookie=a17b2278b03e5329 in_seq=0 out_seq=0 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f863e7fc640 1 -- 172.21.15.3:0/3965850194 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5602e6d41890 con 0x5602e70bfdd0 2024-02-09T23:01:24.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f863e7fc640 10 client.19191 handle_command_reply: tid=0 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 client.19191 shutdown 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 msgr2=0x5602e705d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e705d670 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f8650047000 tx=0x7f865002c3d0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 shutdown_connections 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5602e70bfdd0 0x5602e70c21f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5602e7057f60 0x5602e705ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f863e7fc640 0 client.19191 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5602e704f3e0 0x5602e705d670 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 --2- 172.21.15.3:0/3965850194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5602e704ea30 0x5602e705f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 >> 172.21.15.3:0/3965850194 conn(0x5602e7049100 msgr2=0x5602e7050250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 shutdown_connections 2024-02-09T23:01:24.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 1 -- 172.21.15.3:0/3965850194 wait complete. 2024-02-09T23:01:24.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f8660279740 20 client.19191 trim_cache size 0 max 16384 2024-02-09T23:01:24.102 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86440a2230 msgr2=0x7f86440a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86440a2230 0x7f86440a46f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f864809b060 tx=0x7f8648058040 comp rx=0 tx=0).stop 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 msgr2=0x7f8658199900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 0x7f8658199900 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7f864800ea80 tx=0x7f864800ef40 comp rx=0 tx=0).stop 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 shutdown_connections 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f86440a2230 0x7f86440a46f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f865810db80 0x7f8658199900 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f86581008f0 0x7f865819fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 --2- 172.21.15.3:0/2580732686 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f86580fffd0 0x7f865819f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.090+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 >> 172.21.15.3:0/2580732686 conn(0x7f86580772e0 msgr2=0x7f865810f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.091+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 shutdown_connections 2024-02-09T23:01:24.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.091+0000 7f865d69d640 1 -- 172.21.15.3:0/2580732686 wait complete. 2024-02-09T23:01:24.105 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:24.105 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 -- 172.21.15.3:0/741695814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 msgr2=0x7f7aec10b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 --2- 172.21.15.3:0/741695814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec10b250 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4009900 tx=0x7f7ad402bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 -- 172.21.15.3:0/741695814 shutdown_connections 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 --2- 172.21.15.3:0/741695814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aec10b8c0 0x7f7aec10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 --2- 172.21.15.3:0/741695814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec10b250 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 --2- 172.21.15.3:0/741695814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7aec102450 0x7f7aec102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 -- 172.21.15.3:0/741695814 >> 172.21.15.3:0/741695814 conn(0x7f7aec077250 msgr2=0x7f7aec077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 -- 172.21.15.3:0/741695814 shutdown_connections 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.211+0000 7f7af189d640 1 -- 172.21.15.3:0/741695814 wait complete. 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 Processor -- start 2024-02-09T23:01:24.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 -- start start 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aec102450 0x7f7aec1000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec1005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7aec10b8c0 0x7f7aec100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec1a5dd0 con 0x7f7aec102d70 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec1a5f40 con 0x7f7aec102450 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7af189d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec1a6100 con 0x7f7aec10b8c0 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec1005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec1005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44736/0 (socket says 172.21.15.3:44736) 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 -- 172.21.15.3:0/3778316306 learned_addr learned my addr 172.21.15.3:0/3778316306 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aec102450 0x7f7aec1000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7aec10b8c0 0x7f7aec100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 -- 172.21.15.3:0/3778316306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7aec10b8c0 msgr2=0x7f7aec100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7aec10b8c0 0x7f7aec100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 -- 172.21.15.3:0/3778316306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aec102450 msgr2=0x7f7aec1000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aec102450 0x7f7aec1000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.212+0000 7f7aea7fc640 1 -- 172.21.15.3:0/3778316306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4009560 con 0x7f7aec102d70 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.213+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec1005e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4005940 tx=0x7f7ad402c710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.213+0000 7f7af089b640 1 -- 172.21.15.3:0/3778316306 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4004350 con 0x7f7aec102d70 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.213+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec1a6380 con 0x7f7aec102d70 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.213+0000 7f7af089b640 1 -- 172.21.15.3:0/3778316306 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ad40044b0 con 0x7f7aec102d70 2024-02-09T23:01:24.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.213+0000 7f7af089b640 1 -- 172.21.15.3:0/3778316306 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad4003d30 con 0x7f7aec102d70 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.213+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec1a6890 con 0x7f7aec102d70 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af089b640 1 -- 172.21.15.3:0/3778316306 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7ad4004df0 con 0x7f7aec102d70 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 Processor -- start 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 -- start start 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c64a30 0x563c65c64e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c653e0 0x563c65c6d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.221 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c703b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65c20020 con 0x563c65c64a30 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65bd3320 con 0x563c65c6df60 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af4477740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65bd0850 con 0x563c65c653e0 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c64a30 0x563c65c64e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c653e0 0x563c65c6d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c64a30 0x563c65c64e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44740/0 (socket says 172.21.15.3:44740) 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c703b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeb7fe640 1 -- 172.21.15.3:0/798348088 learned_addr learned my addr 172.21.15.3:0/798348088 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7af089b640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7ad80a2230 0x7f7ad80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7ad80a2230 0x7f7ad80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeaffd640 1 -- 172.21.15.3:0/798348088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 msgr2=0x563c65c703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c703b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeaffd640 1 -- 172.21.15.3:0/798348088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c64a30 msgr2=0x563c65c64e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c64a30 0x563c65c64e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.222 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.216+0000 7f7aeaffd640 1 -- 172.21.15.3:0/798348088 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c65b8e210 con 0x563c65c653e0 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.217+0000 7f7af089b640 1 -- 172.21.15.3:0/3778316306 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7ad40f0650 con 0x7f7aec102d70 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.217+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c653e0 0x563c65c6d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009fa0 tx=0x7f7ae40118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c3cfa013d8f1997a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.217+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7ad80a2230 0x7f7ad80a46f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0058c0 tx=0x7f7adc004730 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.217+0000 7f7ae1ffb640 1 -- 172.21.15.3:0/798348088 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae4003c30 con 0x563c65c653e0 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.217+0000 7f7ae1ffb640 1 -- 172.21.15.3:0/798348088 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ae4003d90 con 0x563c65c653e0 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.217+0000 7f7ae1ffb640 1 -- 172.21.15.3:0/798348088 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae400faa0 con 0x563c65c653e0 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 -- 172.21.15.3:0/798348088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c653e0 msgr2=0x563c65c6d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c653e0 0x563c65c6d990 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7ae4009fa0 tx=0x7f7ae40118d0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 -- 172.21.15.3:0/798348088 shutdown_connections 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c703b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c653e0 0x563c65c6d990 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 --2- 172.21.15.3:0/798348088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c64a30 0x563c65c64e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 -- 172.21.15.3:0/798348088 >> 172.21.15.3:0/798348088 conn(0x563c65c5f100 msgr2=0x563c65c5f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 -- 172.21.15.3:0/798348088 shutdown_connections 2024-02-09T23:01:24.223 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 -- 172.21.15.3:0/798348088 wait complete. 2024-02-09T23:01:24.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 -- start start 2024-02-09T23:01:24.224 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.218+0000 7f7af4477740 1 Processor -- start 2024-02-09T23:01:24.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c6b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65bcc030 con 0x563c65c653e0 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c6589f430 con 0x563c65c6df60 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c6585be00 con 0x563c65c64a30 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 10 client.0 _ll_register_callbacks cb 0x563c65bc1030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44756/0 (socket says 172.21.15.3:44756) 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57570/0 (socket says 172.21.15.3:57570) 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 -- 172.21.15.3:0/4225800049 learned_addr learned my addr 172.21.15.3:0/4225800049 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c6b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 -- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 msgr2=0x563c65c6aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 -- 172.21.15.3:0/4225800049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 msgr2=0x563c65c6b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c6b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 -- 172.21.15.3:0/4225800049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ae4009c00 con 0x563c65c653e0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b3e0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40bd0e0 tx=0x7f7ad40bd000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c872f6aac593de9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7acaffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7acaffd640 1 -- 172.21.15.3:0/4225800049 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40c10d0 con 0x563c65c653e0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7acaffd640 1 -- 172.21.15.3:0/4225800049 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7ad40bd850 con 0x563c65c653e0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.219+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563c658cb7e0 con 0x563c65c653e0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7acaffd640 1 -- 172.21.15.3:0/4225800049 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7ad40bd150 con 0x563c65c653e0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 10 client.19203 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x563c65bca6b0 con 0x563c65c653e0 2024-02-09T23:01:24.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7acaffd640 1 -- 172.21.15.3:0/4225800049 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563c658cb7e0 con 0x563c65c653e0 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7acaffd640 1 -- 172.21.15.3:0/4225800049 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7ad403c070 con 0x563c65c653e0 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 10 client.19203 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 10 client.19203 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 20 client.19203 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65c7e120 0x563c65c80530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 4 client.19203 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c65908720 con 0x563c65c7e120 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.220+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65c7e120 0x563c65c80530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.221+0000 7f7acaffd640 10 client.19203 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.221+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65c7e120 0x563c65c80530 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e1fcd78db5677158 server_cookie=fc76b9573b2d2ee5 in_seq=0 out_seq=0 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.221+0000 7f7acaffd640 1 -- 172.21.15.3:0/4225800049 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563c65908720 con 0x563c65c7e120 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.221+0000 7f7acaffd640 10 client.19203 handle_command_reply: tid=0 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.221+0000 7f7af4477740 1 client.19203 shutdown 2024-02-09T23:01:24.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 msgr2=0x563c65c6b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b3e0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40bd0e0 tx=0x7f7ad40bd000 comp rx=0 tx=0).stop 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 shutdown_connections 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65c7e120 0x563c65c80530 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c6b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b3e0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 --2- 172.21.15.3:0/4225800049 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 >> 172.21.15.3:0/4225800049 conn(0x563c65c5f100 msgr2=0x563c65c60e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7acaffd640 0 client.19203 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 shutdown_connections 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 1 -- 172.21.15.3:0/4225800049 wait complete. 2024-02-09T23:01:24.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.222+0000 7f7af4477740 20 client.19203 trim_cache size 0 max 16384 2024-02-09T23:01:24.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.234+0000 7f7af4477740 1 Processor -- start 2024-02-09T23:01:24.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.234+0000 7f7af4477740 1 -- start start 2024-02-09T23:01:24.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7af4477740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.239 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7af4477740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7af4477740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c661d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7af4477740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65b51130 con 0x563c65c653e0 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7af4477740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65908750 con 0x563c65c6df60 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7af4477740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65c64380 con 0x563c65c64a30 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44758/0 (socket says 172.21.15.3:44758) 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 -- 172.21.15.3:0/477498080 learned_addr learned my addr 172.21.15.3:0/477498080 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 -- 172.21.15.3:0/477498080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 msgr2=0x563c65c6d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 -- 172.21.15.3:0/477498080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 msgr2=0x563c65c661d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.240 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c661d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 -- 172.21.15.3:0/477498080 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c65b66c90 con 0x563c65c653e0 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b740 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40ee3a0 tx=0x7f7ad40057e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d58a1a3d65d750 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.235+0000 7f7ae17fa640 1 -- 172.21.15.3:0/477498080 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40c10d0 con 0x563c65c653e0 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7ae17fa640 1 -- 172.21.15.3:0/477498080 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ad40d5e80 con 0x563c65c653e0 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7ae17fa640 1 -- 172.21.15.3:0/477498080 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad40c58f0 con 0x563c65c653e0 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 -- 172.21.15.3:0/477498080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 msgr2=0x563c65c6b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b740 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f7ad40ee3a0 tx=0x7f7ad40057e0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 -- 172.21.15.3:0/477498080 shutdown_connections 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c6df60 0x563c65c661d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c653e0 0x563c65c6b740 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 --2- 172.21.15.3:0/477498080 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c6d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 -- 172.21.15.3:0/477498080 >> 172.21.15.3:0/477498080 conn(0x563c65c5f100 msgr2=0x563c65c5a180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 -- 172.21.15.3:0/477498080 shutdown_connections 2024-02-09T23:01:24.241 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.236+0000 7f7af4477740 1 -- 172.21.15.3:0/477498080 wait complete. 2024-02-09T23:01:24.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 -- start start 2024-02-09T23:01:24.242 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 Processor -- start 2024-02-09T23:01:24.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c72fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c653e0 0x563c65c734e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 0x563c65cd3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65c73de0 con 0x563c65c6df60 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65cd3b90 con 0x563c65c653e0 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c65cd3cd0 con 0x563c65c64a30 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7af4477740 10 client.0 _ll_register_callbacks cb 0x7f7ae4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 0x563c65cd3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeaffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c72fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 0x563c65cd3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44768/0 (socket says 172.21.15.3:44768) 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 -- 172.21.15.3:0/64116422 learned_addr learned my addr 172.21.15.3:0/64116422 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c653e0 0x563c65c734e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 -- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 msgr2=0x563c65c72fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c72fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 -- 172.21.15.3:0/64116422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c653e0 msgr2=0x563c65c734e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c653e0 0x563c65c734e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeb7fe640 1 -- 172.21.15.3:0/64116422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c65bca6b0 con 0x563c65c6df60 2024-02-09T23:01:24.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c72fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.237+0000 7f7aea7fc640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c653e0 0x563c65c734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7aeb7fe640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 0x563c65cd3650 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7adc012420 tx=0x7f7adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ad8ac10e2af2f93 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7acaffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7acaffd640 1 -- 172.21.15.3:0/64116422 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7adc027070 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7acaffd640 1 -- 172.21.15.3:0/64116422 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7adc022a00 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563c65957890 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7acaffd640 1 -- 172.21.15.3:0/64116422 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7adc022bc0 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 10 client.19209 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x563c65b8ca70 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7acaffd640 1 -- 172.21.15.3:0/64116422 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563c65957890 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7acaffd640 1 -- 172.21.15.3:0/64116422 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f7adc010430 con 0x563c65c6df60 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 10 client.19209 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 10 client.19209 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:01:24.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 20 client.19209 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65cd54d0 0x563c65cd78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 4 client.19209 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.238+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563c65877700 con 0x563c65cd54d0 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.239+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65cd54d0 0x563c65cd78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.239+0000 7f7aeaffd640 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65cd54d0 0x563c65cd78f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=eccca2bc2a710360 server_cookie=4819be1703b80834 in_seq=0 out_seq=0 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.239+0000 7f7acaffd640 10 client.19209 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7acaffd640 1 -- 172.21.15.3:0/64116422 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563c65877700 con 0x563c65cd54d0 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7acaffd640 10 client.19209 handle_command_reply: tid=0 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 client.19209 shutdown 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 msgr2=0x563c65cd3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 0x563c65cd3650 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f7adc012420 tx=0x7f7adc0046c0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 shutdown_connections 2024-02-09T23:01:24.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x563c65cd54d0 0x563c65cd78f0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563c65c653e0 0x563c65c734e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563c65c6df60 0x563c65cd3650 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 --2- 172.21.15.3:0/64116422 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563c65c64a30 0x563c65c72fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7acaffd640 0 client.19209 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.240+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 >> 172.21.15.3:0/64116422 conn(0x563c65c5f100 msgr2=0x563c65c661d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 shutdown_connections 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af4477740 1 -- 172.21.15.3:0/64116422 wait complete. 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af4477740 20 client.19209 trim_cache size 0 max 16384 2024-02-09T23:01:24.249 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:24.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7ad80a2230 msgr2=0x7f7ad80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7ad80a2230 0x7f7ad80a46f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7adc0058c0 tx=0x7f7adc004730 comp rx=0 tx=0).stop 2024-02-09T23:01:24.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 msgr2=0x7f7aec1005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec1005e0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f7ad4005940 tx=0x7f7ad402c710 comp rx=0 tx=0).stop 2024-02-09T23:01:24.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 shutdown_connections 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7ad80a2230 0x7f7ad80a46f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7aec10b8c0 0x7f7aec100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7aec102d70 0x7f7aec1005e0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 --2- 172.21.15.3:0/3778316306 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7aec102450 0x7f7aec1000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 >> 172.21.15.3:0/3778316306 conn(0x7f7aec077250 msgr2=0x7f7aec0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 shutdown_connections 2024-02-09T23:01:24.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.241+0000 7f7af189d640 1 -- 172.21.15.3:0/3778316306 wait complete. 2024-02-09T23:01:24.255 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:24.255 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 -- 172.21.15.3:0/1142323949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2880102470 msgr2=0x7f2880102850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 --2- 172.21.15.3:0/1142323949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2880102470 0x7f2880102850 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f287c009860 tx=0x7f287c02bc00 comp rx=0 tx=0).stop 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 -- 172.21.15.3:0/1142323949 shutdown_connections 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 --2- 172.21.15.3:0/1142323949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102d90 0x7f288010b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 --2- 172.21.15.3:0/1142323949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f288010b7b0 0x7f288010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 --2- 172.21.15.3:0/1142323949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2880102470 0x7f2880102850 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 -- 172.21.15.3:0/1142323949 >> 172.21.15.3:0/1142323949 conn(0x7f28800772e0 msgr2=0x7f28800776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 -- 172.21.15.3:0/1142323949 shutdown_connections 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 -- 172.21.15.3:0/1142323949 wait complete. 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 Processor -- start 2024-02-09T23:01:24.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 -- start start 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.366+0000 7f288844d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102470 0x7f28801000c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2880102d90 0x7f2880100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 0x7f2880100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28801a5c00 con 0x7f288010b7b0 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28801a5d70 con 0x7f2880102470 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28801a5f30 con 0x7f2880102d90 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 0x7f2880100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 0x7f2880100b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44786/0 (socket says 172.21.15.3:44786) 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 -- 172.21.15.3:0/2579488364 learned_addr learned my addr 172.21.15.3:0/2579488364 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28859e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2880102d90 0x7f2880100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28861e9640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102470 0x7f28801000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 -- 172.21.15.3:0/2579488364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2880102d90 msgr2=0x7f2880100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2880102d90 0x7f2880100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 -- 172.21.15.3:0/2579488364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102470 msgr2=0x7f28801000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102470 0x7f28801000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 -- 172.21.15.3:0/2579488364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2870009560 con 0x7f288010b7b0 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28859e8640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2880102d90 0x7f2880100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28861e9640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102470 0x7f28801000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28869ea640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 0x7f2880100b40 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f287800ea60 tx=0x7f287800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28777fe640 1 -- 172.21.15.3:0/2579488364 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287800ce40 con 0x7f288010b7b0 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f287c009490 con 0x7f288010b7b0 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28777fe640 1 -- 172.21.15.3:0/2579488364 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2878005020 con 0x7f288010b7b0 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f28777fe640 1 -- 172.21.15.3:0/2579488364 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2878010690 con 0x7f288010b7b0 2024-02-09T23:01:24.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.367+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28801a6590 con 0x7f288010b7b0 2024-02-09T23:01:24.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28777fe640 1 -- 172.21.15.3:0/2579488364 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f2878005500 con 0x7f288010b7b0 2024-02-09T23:01:24.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 Processor -- start 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 -- start start 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c457472020 con 0x55c4574bff60 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c457425320 con 0x55c4574b6a30 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f288b01c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c457422850 con 0x55c4574b73e0 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28859e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28861e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28861e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57606/0 (socket says 172.21.15.3:57606) 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28861e9640 1 -- 172.21.15.3:0/3809015840 learned_addr learned my addr 172.21.15.3:0/3809015840 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28859e8640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574b6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:53216/0 (socket says 172.21.15.3:53216) 2024-02-09T23:01:24.380 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28859e8640 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28777fe640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f286c0a2260 0x7f286c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.371+0000 7f28869ea640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f286c0a2260 0x7f286c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28859e8640 1 -- 172.21.15.3:0/3809015840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 msgr2=0x55c4574bf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28859e8640 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28859e8640 1 -- 172.21.15.3:0/3809015840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 msgr2=0x55c4574c23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28859e8640 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28859e8640 1 -- 172.21.15.3:0/3809015840 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4573e0210 con 0x55c4574b6a30 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28861e9640 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28777fe640 1 -- 172.21.15.3:0/2579488364 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2878014070 con 0x7f288010b7b0 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28859e8640 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574b6e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2870009530 tx=0x7f2870009c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c23f90fd2d94b7df server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f28869ea640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f286c0a2260 0x7f286c0a4720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2878097330 tx=0x7f287807b040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f2874bfe640 1 -- 172.21.15.3:0/3809015840 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000fc20 con 0x55c4574b6a30 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f2874bfe640 1 -- 172.21.15.3:0/3809015840 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28700115e0 con 0x55c4574b6a30 2024-02-09T23:01:24.381 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.372+0000 7f2874bfe640 1 -- 172.21.15.3:0/3809015840 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870004de0 con 0x55c4574b6a30 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 -- 172.21.15.3:0/3809015840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 msgr2=0x55c4574b6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574b6e10 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2870009530 tx=0x7f2870009c90 comp rx=0 tx=0).stop 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 -- 172.21.15.3:0/3809015840 shutdown_connections 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 --2- 172.21.15.3:0/3809015840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574b6e10 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 -- 172.21.15.3:0/3809015840 >> 172.21.15.3:0/3809015840 conn(0x55c4574b1100 msgr2=0x55c4574b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 -- 172.21.15.3:0/3809015840 shutdown_connections 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 -- 172.21.15.3:0/3809015840 wait complete. 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 -- start start 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.373+0000 7f288b01c740 1 Processor -- start 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574bcef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.382 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574bd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45741e030 con 0x55c4574bff60 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4570f1430 con 0x55c4574b6a30 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4570ade00 con 0x55c4574b73e0 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f288b01c740 10 client.0 _ll_register_callbacks cb 0x55c457413030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574bd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28861e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574bd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44802/0 (socket says 172.21.15.3:44802) 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 -- 172.21.15.3:0/3736509615 learned_addr learned my addr 172.21.15.3:0/3736509615 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28861e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57614/0 (socket says 172.21.15.3:57614) 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574bcef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 -- 172.21.15.3:0/3736509615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 msgr2=0x55c4574bd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bd430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 -- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 msgr2=0x55c4574bcef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574bcef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 -- 172.21.15.3:0/3736509615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2870019040 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f28859e8640 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574bd970 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f2870009c60 tx=0x7f28700049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16195382d90e8530 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f2869ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.374+0000 7f2869ffb640 1 -- 172.21.15.3:0/3736509615 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2870005200 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c45711d7e0 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f2869ffb640 1 -- 172.21.15.3:0/3736509615 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2870005360 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f2869ffb640 1 -- 172.21.15.3:0/3736509615 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2870005520 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 10 client.19221 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55c45741c6b0 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f2869ffb640 1 -- 172.21.15.3:0/3736509615 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c45711d7e0 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f2869ffb640 1 -- 172.21.15.3:0/3736509615 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f287000e3e0 con 0x55c4574bff60 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 10 client.19221 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 10 client.19221 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:01:24.384 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 20 client.19221 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c4574d0310 0x55c4574d2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 4 client.19221 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c45715a720 con 0x55c4574d0310 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.375+0000 7f28861e9640 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c4574d0310 0x55c4574d2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f2869ffb640 10 client.19221 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f28861e9640 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c4574d0310 0x55c4574d2700 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41e47bb2f7596348 server_cookie=73e3c6000e2afb1e in_seq=0 out_seq=0 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f2869ffb640 1 -- 172.21.15.3:0/3736509615 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c45715a720 con 0x55c4574d0310 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f2869ffb640 10 client.19221 handle_command_reply: tid=0 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f288b01c740 1 client.19221 shutdown 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 msgr2=0x55c4574bd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.376+0000 7f288b01c740 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574bd970 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7f2870009c60 tx=0x7f28700049c0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 shutdown_connections 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c4574d0310 0x55c4574d2700 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574bd970 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f2869ffb640 0 client.19221 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574bd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 --2- 172.21.15.3:0/3736509615 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574bcef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 >> 172.21.15.3:0/3736509615 conn(0x55c4574b1100 msgr2=0x55c4574b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 shutdown_connections 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 1 -- 172.21.15.3:0/3736509615 wait complete. 2024-02-09T23:01:24.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.377+0000 7f288b01c740 20 client.19221 trim_cache size 0 max 16384 2024-02-09T23:01:24.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:24 smithi003 ceph-mon[32678]: pgmap v317: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-02-09T23:01:24.386 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:24 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/963775318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 Processor -- start 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 -- start start 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b6a30 0x55c4574bf620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 0x55c4574ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574bff60 0x55c4574ba710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4573a3130 con 0x55c4574b73e0 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c45711d7e0 con 0x55c4574bff60 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f288b01c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4574b6380 con 0x55c4574b6a30 2024-02-09T23:01:24.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28861e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 0x55c4574ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28859e8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b6a30 0x55c4574bf620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28861e9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 0x55c4574ba1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44812/0 (socket says 172.21.15.3:44812) 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28861e9640 1 -- 172.21.15.3:0/1083391062 learned_addr learned my addr 172.21.15.3:0/1083391062 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28859e8640 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574bff60 0x55c4574ba710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28861e9640 1 -- 172.21.15.3:0/1083391062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b6a30 msgr2=0x55c4574bf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.392+0000 7f28861e9640 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b6a30 0x55c4574bf620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f28861e9640 1 -- 172.21.15.3:0/1083391062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574bff60 msgr2=0x55c4574ba710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f28861e9640 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574bff60 0x55c4574ba710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f28861e9640 1 -- 172.21.15.3:0/1083391062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c4573b8c90 con 0x55c4574b73e0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f28859e8640 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b6a30 0x55c4574bf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f28861e9640 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 0x55c4574ba1d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f287c005940 tx=0x7f287c02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e80dc660209b17d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f286bfff640 1 -- 172.21.15.3:0/1083391062 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c048600 con 0x55c4574b73e0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f286bfff640 1 -- 172.21.15.3:0/1083391062 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f287c0041c0 con 0x55c4574b73e0 2024-02-09T23:01:24.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.393+0000 7f286bfff640 1 -- 172.21.15.3:0/1083391062 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287c02c990 con 0x55c4574b73e0 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- 172.21.15.3:0/1083391062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 msgr2=0x55c4574ba1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 0x55c4574ba1d0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f287c005940 tx=0x7f287c02e040 comp rx=0 tx=0).stop 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- 172.21.15.3:0/1083391062 shutdown_connections 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574bff60 0x55c4574ba710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574b73e0 0x55c4574ba1d0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- 172.21.15.3:0/1083391062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b6a30 0x55c4574bf620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- 172.21.15.3:0/1083391062 >> 172.21.15.3:0/1083391062 conn(0x55c4574b1100 msgr2=0x55c4574ac180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- 172.21.15.3:0/1083391062 shutdown_connections 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- 172.21.15.3:0/1083391062 wait complete. 2024-02-09T23:01:24.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- start start 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 Processor -- start 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c6500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c4575267b0 con 0x55c4574bff60 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c457526920 con 0x55c4574b6a30 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c457526ae0 con 0x55c4574b73e0 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.394+0000 7f288b01c740 10 client.0 _ll_register_callbacks cb 0x7f2870011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c6500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28861e9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c6500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44822/0 (socket says 172.21.15.3:44822) 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 -- 172.21.15.3:0/282714905 learned_addr learned my addr 172.21.15.3:0/282714905 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28861e9640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57628/0 (socket says 172.21.15.3:57628) 2024-02-09T23:01:24.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 -- 172.21.15.3:0/282714905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 msgr2=0x55c4574c5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574c5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 -- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 msgr2=0x55c4574c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 -- 172.21.15.3:0/282714905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c45741c6b0 con 0x55c4574bff60 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28861e9640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f28859e8640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c6500 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2870009340 tx=0x7f28700049c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75fb18219823293 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f2869ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f2869ffb640 1 -- 172.21.15.3:0/282714905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f287000e3e0 con 0x55c4574bff60 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f28700115e0 con 0x55c4574bff60 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f2869ffb640 1 -- 172.21.15.3:0/282714905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2870000e40 con 0x55c4574bff60 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f2869ffb640 1 -- 172.21.15.3:0/282714905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f287000e540 con 0x55c4574bff60 2024-02-09T23:01:24.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f288b01c740 10 client.19227 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.395+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x7f287c030070 con 0x55c4574bff60 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f2869ffb640 1 -- 172.21.15.3:0/282714905 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28700115e0 con 0x55c4574bff60 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f2869ffb640 1 -- 172.21.15.3:0/282714905 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f28700054f0 con 0x55c4574bff60 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 10 client.19227 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 10 client.19227 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 20 client.19227 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c457528000 0x55c45752a420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 4 client.19227 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c4571a9890 con 0x55c457528000 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f28861e9640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c457528000 0x55c45752a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f28861e9640 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c457528000 0x55c45752a420 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4483251c69e75af7 server_cookie=d87726be0b47db73 in_seq=0 out_seq=0 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f2869ffb640 10 client.19227 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f2869ffb640 1 -- 172.21.15.3:0/282714905 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c4571a9890 con 0x55c457528000 2024-02-09T23:01:24.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f2869ffb640 10 client.19227 handle_command_reply: tid=0 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.396+0000 7f288b01c740 1 client.19227 shutdown 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 msgr2=0x55c4574c6500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c6500 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f2870009340 tx=0x7f28700049c0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 shutdown_connections 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x55c457528000 0x55c45752a420 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c4574bff60 0x55c4574c6500 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c4574b73e0 0x55c4574c5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 --2- 172.21.15.3:0/282714905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c4574b6a30 0x55c4574c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f2869ffb640 0 client.19227 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 >> 172.21.15.3:0/282714905 conn(0x55c4574b1100 msgr2=0x55c4574bb820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 shutdown_connections 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 1 -- 172.21.15.3:0/282714905 wait complete. 2024-02-09T23:01:24.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288b01c740 20 client.19227 trim_cache size 0 max 16384 2024-02-09T23:01:24.407 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f286c0a2260 msgr2=0x7f286c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f286c0a2260 0x7f286c0a4720 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f2878097330 tx=0x7f287807b040 comp rx=0 tx=0).stop 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 msgr2=0x7f2880100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 0x7f2880100b40 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f287800ea60 tx=0x7f287800ef20 comp rx=0 tx=0).stop 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 shutdown_connections 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f286c0a2260 0x7f286c0a4720 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f288010b7b0 0x7f2880100b40 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2880102d90 0x7f2880100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 --2- 172.21.15.3:0/2579488364 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2880102470 0x7f28801000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 >> 172.21.15.3:0/2579488364 conn(0x7f28800772e0 msgr2=0x7f28800fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 shutdown_connections 2024-02-09T23:01:24.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.397+0000 7f288844d640 1 -- 172.21.15.3:0/2579488364 wait complete. 2024-02-09T23:01:24.413 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:24.413 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 -- 172.21.15.3:0/56773600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 msgr2=0x7f997c109a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 --2- 172.21.15.3:0/56773600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c109a20 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f9974009cd0 tx=0x7f997402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 -- 172.21.15.3:0/56773600 shutdown_connections 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 --2- 172.21.15.3:0/56773600 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c1082f0 0x7f997c1147c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 --2- 172.21.15.3:0/56773600 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c107db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 --2- 172.21.15.3:0/56773600 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c109a20 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.519+0000 7f9983464640 1 -- 172.21.15.3:0/56773600 >> 172.21.15.3:0/56773600 conn(0x7f997c0770d0 msgr2=0x7f997c106050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 -- 172.21.15.3:0/56773600 shutdown_connections 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 -- 172.21.15.3:0/56773600 wait complete. 2024-02-09T23:01:24.525 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 Processor -- start 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 -- start start 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c19def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c1082f0 0x7f997c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c1a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c1a2db0 con 0x7f997c109640 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c1a2f20 con 0x7f997c1082f0 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9983464640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f997c1a30e0 con 0x7f997c107950 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c1a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9981200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c19def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c1a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44842/0 (socket says 172.21.15.3:44842) 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9981a01640 1 -- 172.21.15.3:0/114581082 learned_addr learned my addr 172.21.15.3:0/114581082 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f9981200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c19def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57644/0 (socket says 172.21.15.3:57644) 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.520+0000 7f99809ff640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c1082f0 0x7f997c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.526 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981200640 1 -- 172.21.15.3:0/114581082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c1082f0 msgr2=0x7f997c19e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981200640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c1082f0 0x7f997c19e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981200640 1 -- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 msgr2=0x7f997c1a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981200640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c1a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981200640 1 -- 172.21.15.3:0/114581082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974009900 con 0x7f997c107950 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981a01640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c1a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9981200640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c19def0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f997402c700 tx=0x7f99740325f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f997a7fc640 1 -- 172.21.15.3:0/114581082 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740328a0 con 0x7f997c107950 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997c1a3360 con 0x7f997c107950 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f997a7fc640 1 -- 172.21.15.3:0/114581082 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9974003e50 con 0x7f997c107950 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f997a7fc640 1 -- 172.21.15.3:0/114581082 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9974004830 con 0x7f997c107950 2024-02-09T23:01:24.527 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.521+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c1a3600 con 0x7f997c107950 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.523+0000 7f997a7fc640 1 -- 172.21.15.3:0/114581082 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f997403b040 con 0x7f997c107950 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 Processor -- start 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 -- start start 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab15ea30 0x5585ab15ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab167990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab167f60 0x5585ab16a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab11a020 con 0x5585ab15ea30 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab0cd320 con 0x5585ab15f3e0 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f998603f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab0ca850 con 0x5585ab167f60 2024-02-09T23:01:24.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f997a7fc640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f99640a2230 0x7f99640a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981200640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab167990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab15ea30 0x5585ab15ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab15ea30 0x5585ab15ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44852/0 (socket says 172.21.15.3:44852) 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981a01640 1 -- 172.21.15.3:0/4004653943 learned_addr learned my addr 172.21.15.3:0/4004653943 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f99809ff640 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab167f60 0x5585ab16a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f997a7fc640 1 -- 172.21.15.3:0/114581082 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f99740f0940 con 0x7f997c107950 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981a01640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f99640a2230 0x7f99640a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981200640 1 -- 172.21.15.3:0/4004653943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab167f60 msgr2=0x5585ab16a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981200640 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab167f60 0x5585ab16a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981200640 1 -- 172.21.15.3:0/4004653943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab15ea30 msgr2=0x5585ab15ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981200640 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab15ea30 0x5585ab15ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.524+0000 7f9981200640 1 -- 172.21.15.3:0/4004653943 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585ab088210 con 0x5585ab15f3e0 2024-02-09T23:01:24.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.525+0000 7f9981a01640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f99640a2230 0x7f99640a46f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f996c012e30 tx=0x7f996c00b2e0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.525+0000 7f9981200640 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab167990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f99740ee1a0 tx=0x7f99740889e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c3c5570c92a46c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.525+0000 7f99637fe640 1 -- 172.21.15.3:0/4004653943 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740bd210 con 0x5585ab15f3e0 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.525+0000 7f99637fe640 1 -- 172.21.15.3:0/4004653943 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99740bcf80 con 0x5585ab15f3e0 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.525+0000 7f99637fe640 1 -- 172.21.15.3:0/4004653943 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740c1590 con 0x5585ab15f3e0 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 -- 172.21.15.3:0/4004653943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 msgr2=0x5585ab167990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab167990 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f99740ee1a0 tx=0x7f99740889e0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 -- 172.21.15.3:0/4004653943 shutdown_connections 2024-02-09T23:01:24.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab167990 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab167f60 0x5585ab16a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 --2- 172.21.15.3:0/4004653943 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab15ea30 0x5585ab15ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 -- 172.21.15.3:0/4004653943 >> 172.21.15.3:0/4004653943 conn(0x5585ab159100 msgr2=0x5585ab1594e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 -- 172.21.15.3:0/4004653943 shutdown_connections 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 -- 172.21.15.3:0/4004653943 wait complete. 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 -- start start 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.526+0000 7f998603f740 1 Processor -- start 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab164e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab165350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab165890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab0c6030 con 0x5585ab167f60 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585aad99430 con 0x5585ab15f3e0 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585aad55e00 con 0x5585ab15ea30 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 10 client.0 _ll_register_callbacks cb 0x5585ab0bb030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab165890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab164e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab165890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44868/0 (socket says 172.21.15.3:44868) 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981a01640 1 -- 172.21.15.3:0/3844940788 learned_addr learned my addr 172.21.15.3:0/3844940788 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f99809ff640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab165350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 -- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 msgr2=0x5585ab165350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab165350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 -- 172.21.15.3:0/3844940788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 msgr2=0x5585ab165890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.539 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab165890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 -- 172.21.15.3:0/3844940788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997c1a3600 con 0x5585ab15ea30 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981a01640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab165890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9981200640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab164e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x5585ab15e500 tx=0x7f997402c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86bc1b2534461d71 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9960ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9960ff9640 1 -- 172.21.15.3:0/3844940788 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740bd210 con 0x5585ab15ea30 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9960ff9640 1 -- 172.21.15.3:0/3844940788 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f99740bdb40 con 0x5585ab15ea30 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5585aadc57e0 con 0x5585ab15ea30 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.527+0000 7f9960ff9640 1 -- 172.21.15.3:0/3844940788 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99740bcda0 con 0x5585ab15ea30 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 10 client.28255 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5585ab0c46b0 con 0x5585ab15ea30 2024-02-09T23:01:24.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f9960ff9640 1 -- 172.21.15.3:0/3844940788 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5585aadc57e0 con 0x5585ab15ea30 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f9960ff9640 1 -- 172.21.15.3:0/3844940788 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f99740bd6a0 con 0x5585ab15ea30 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 10 client.28255 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 10 client.28255 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 20 client.28255 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab177d20 0x5585ab17a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 4 client.28255 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5585aae02720 con 0x5585ab177d20 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f99809ff640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab177d20 0x5585ab17a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f9960ff9640 10 client.28255 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.528+0000 7f99809ff640 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab177d20 0x5585ab17a110 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ae12b9fe6699bed0 server_cookie=74762eb8241368e4 in_seq=0 out_seq=0 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f9960ff9640 1 -- 172.21.15.3:0/3844940788 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5585aae02720 con 0x5585ab177d20 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f9960ff9640 10 client.28255 handle_command_reply: tid=0 2024-02-09T23:01:24.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 client.28255 shutdown 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 msgr2=0x5585ab164e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab164e10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x5585ab15e500 tx=0x7f997402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 shutdown_connections 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab177d20 0x5585ab17a110 crc :-1 s=READY pgs=285 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab165350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab165890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f9960ff9640 0 client.28255 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 --2- 172.21.15.3:0/3844940788 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab164e10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 >> 172.21.15.3:0/3844940788 conn(0x5585ab159100 msgr2=0x5585ab15ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 shutdown_connections 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 1 -- 172.21.15.3:0/3844940788 wait complete. 2024-02-09T23:01:24.542 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.529+0000 7f998603f740 20 client.28255 trim_cache size 0 max 16384 2024-02-09T23:01:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 Processor -- start 2024-02-09T23:01:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 -- start start 2024-02-09T23:01:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab167540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab1656b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.550 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab162090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab04b130 con 0x5585ab167f60 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab154a40 con 0x5585ab15f3e0 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f998603f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab17f290 con 0x5585ab15ea30 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab162090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab162090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44882/0 (socket says 172.21.15.3:44882) 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 -- 172.21.15.3:0/819290199 learned_addr learned my addr 172.21.15.3:0/819290199 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f99809ff640 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab1656b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981200640 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab167540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 -- 172.21.15.3:0/819290199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 msgr2=0x5585ab167540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab167540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.551 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 -- 172.21.15.3:0/819290199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 msgr2=0x5585ab1656b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab1656b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 -- 172.21.15.3:0/819290199 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5585ab060c90 con 0x5585ab167f60 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.542+0000 7f9981a01640 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab162090 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f996c00bb50 tx=0x7f996c00bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9e2fba78266d47e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f9962ffd640 1 -- 172.21.15.3:0/819290199 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c005020 con 0x5585ab167f60 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f9962ffd640 1 -- 172.21.15.3:0/819290199 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f996c005660 con 0x5585ab167f60 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f9962ffd640 1 -- 172.21.15.3:0/819290199 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f996c021850 con 0x5585ab167f60 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 -- 172.21.15.3:0/819290199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 msgr2=0x5585ab162090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab162090 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7f996c00bb50 tx=0x7f996c00bbb0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 -- 172.21.15.3:0/819290199 shutdown_connections 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab1656b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab162090 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 --2- 172.21.15.3:0/819290199 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab167540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 -- 172.21.15.3:0/819290199 >> 172.21.15.3:0/819290199 conn(0x5585ab159100 msgr2=0x5585ab160110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 -- 172.21.15.3:0/819290199 shutdown_connections 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.543+0000 7f998603f740 1 -- 172.21.15.3:0/819290199 wait complete. 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 -- start start 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 Processor -- start 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab16f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab16d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab16da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab16e150 con 0x5585ab167f60 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab16e2c0 con 0x5585ab15f3e0 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5585ab1ce190 con 0x5585ab15ea30 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f998603f740 10 client.0 _ll_register_callbacks cb 0x7f996c0124d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab16da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f9981200640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab16f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f9981a01640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab16da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44896/0 (socket says 172.21.15.3:44896) 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f99809ff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab16d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.544+0000 7f9981a01640 1 -- 172.21.15.3:0/495155334 learned_addr learned my addr 172.21.15.3:0/495155334 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981200640 1 -- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 msgr2=0x5585ab16d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981200640 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab16d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981200640 1 -- 172.21.15.3:0/495155334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 msgr2=0x5585ab16da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981200640 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab16da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981200640 1 -- 172.21.15.3:0/495155334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f996c01a040 con 0x5585ab15ea30 2024-02-09T23:01:24.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981a01640 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab16da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9981200640 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab16f110 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f99740bd140 tx=0x7f9974032c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4a2f683234070725 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9960ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9960ff9640 1 -- 172.21.15.3:0/495155334 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99740c1340 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f99740bcf80 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9960ff9640 1 -- 172.21.15.3:0/495155334 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f99740bdb40 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9960ff9640 1 -- 172.21.15.3:0/495155334 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f99740bcda0 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 10 client.28261 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x5585ab086a70 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9960ff9640 1 -- 172.21.15.3:0/495155334 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f99740bcf80 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f9960ff9640 1 -- 172.21.15.3:0/495155334 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f99740dbbb0 con 0x5585ab15ea30 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 10 client.28261 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 10 client.28261 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 20 client.28261 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab1cf450 0x5585ab1d1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:24.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 4 client.28261 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.545+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5585aae51890 con 0x5585ab1cf450 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f99809ff640 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab1cf450 0x5585ab1d1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f99809ff640 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab1cf450 0x5585ab1d1840 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cabf0f246012341 server_cookie=cee42b579025b280 in_seq=0 out_seq=0 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f9960ff9640 10 client.28261 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f9960ff9640 1 -- 172.21.15.3:0/495155334 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5585aae51890 con 0x5585ab1cf450 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f9960ff9640 10 client.28261 handle_command_reply: tid=0 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f998603f740 1 client.28261 shutdown 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.546+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 msgr2=0x5585ab16f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab16f110 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f99740bd140 tx=0x7f9974032c50 comp rx=0 tx=0).stop 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 shutdown_connections 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5585ab1cf450 0x5585ab1d1840 crc :-1 s=READY pgs=286 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5585ab15f3e0 0x5585ab16d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5585ab167f60 0x5585ab16da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.556 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9960ff9640 0 client.28261 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 --2- 172.21.15.3:0/495155334 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5585ab15ea30 0x5585ab16f110 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 >> 172.21.15.3:0/495155334 conn(0x5585ab159100 msgr2=0x5585ab1636e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 shutdown_connections 2024-02-09T23:01:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 1 -- 172.21.15.3:0/495155334 wait complete. 2024-02-09T23:01:24.557 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f998603f740 20 client.28261 trim_cache size 0 max 16384 2024-02-09T23:01:24.557 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f99640a2230 msgr2=0x7f99640a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f99640a2230 0x7f99640a46f0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f996c012e30 tx=0x7f996c00b2e0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 msgr2=0x7f997c19def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c19def0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f997402c700 tx=0x7f99740325f0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 shutdown_connections 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f99640a2230 0x7f99640a46f0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f997c1082f0 0x7f997c19e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f997c109640 0x7f997c1a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 --2- 172.21.15.3:0/114581082 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f997c107950 0x7f997c19def0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:24.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 >> 172.21.15.3:0/114581082 conn(0x7f997c0770d0 msgr2=0x7f997c10a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 shutdown_connections 2024-02-09T23:01:24.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:24.547+0000 7f9983464640 1 -- 172.21.15.3:0/114581082 wait complete. 2024-02-09T23:01:24.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 338618 3.20 MB/sec execute 438 sec latency 12.377 ms 2024-02-09T23:01:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:24 smithi064 ceph-mon[36638]: pgmap v317: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-02-09T23:01:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:24 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/963775318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:01:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:24 smithi089 ceph-mon[36665]: pgmap v317: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-02-09T23:01:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:24 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/963775318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:01:25.562 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:01:25.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:01:25.563 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:01:25.591 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:25 smithi003 ceph-mon[32678]: pgmap v318: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-02-09T23:01:25.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 338670 3.19 MB/sec execute 439 sec latency 831.541 ms 2024-02-09T23:01:25.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 -- 172.21.15.3:0/2574959857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441053e0 msgr2=0x7f674410cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 --2- 172.21.15.3:0/2574959857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441053e0 0x7f674410cfb0 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f6734009930 tx=0x7f673402bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:25.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 -- 172.21.15.3:0/2574959857 shutdown_connections 2024-02-09T23:01:25.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 --2- 172.21.15.3:0/2574959857 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441053e0 0x7f674410cfb0 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 --2- 172.21.15.3:0/2574959857 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6744100cc0 0x7f6744104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 --2- 172.21.15.3:0/2574959857 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6744100310 0x7f67441006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 -- 172.21.15.3:0/2574959857 >> 172.21.15.3:0/2574959857 conn(0x7f6744077230 msgr2=0x7f6744077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 -- 172.21.15.3:0/2574959857 shutdown_connections 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.675+0000 7f674b040640 1 -- 172.21.15.3:0/2574959857 wait complete. 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 Processor -- start 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 -- start start 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6744100310 0x7f674419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67441053e0 0x7f674419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 0x7f6744197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6744198170 con 0x7f67441976a0 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67441982e0 con 0x7f67441053e0 2024-02-09T23:01:25.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f674b040640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67441984a0 con 0x7f6744100310 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 0x7f6744197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 0x7f6744197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44920/0 (socket says 172.21.15.3:44920) 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 -- 172.21.15.3:0/1737930979 learned_addr learned my addr 172.21.15.3:0/1737930979 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f6743fff640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67441053e0 0x7f674419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f6748ddc640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6744100310 0x7f674419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 -- 172.21.15.3:0/1737930979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6744100310 msgr2=0x7f674419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6744100310 0x7f674419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 -- 172.21.15.3:0/1737930979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67441053e0 msgr2=0x7f674419db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67441053e0 0x7f674419db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.676+0000 7f67495dd640 1 -- 172.21.15.3:0/1737930979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6734009560 con 0x7f67441976a0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.677+0000 7f67495dd640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 0x7f6744197b20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f6734005940 tx=0x7f673402c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.677+0000 7f6741ffb640 1 -- 172.21.15.3:0/1737930979 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340043d0 con 0x7f67441976a0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.677+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67441a43e0 con 0x7f67441976a0 2024-02-09T23:01:25.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.677+0000 7f6741ffb640 1 -- 172.21.15.3:0/1737930979 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67340052d0 con 0x7f67441976a0 2024-02-09T23:01:25.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.677+0000 7f6741ffb640 1 -- 172.21.15.3:0/1737930979 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6734003d10 con 0x7f67441976a0 2024-02-09T23:01:25.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.677+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67441a4910 con 0x7f67441976a0 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.679+0000 7f6741ffb640 1 -- 172.21.15.3:0/1737930979 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6734005440 con 0x7f67441976a0 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 Processor -- start 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6741ffb640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67300a2230 0x7f67300a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 -- start start 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6743fff640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67300a2230 0x7f67300a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc6045fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6046a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.688 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc605a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6741ffb640 1 -- 172.21.15.3:0/1737930979 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f67340f00e0 con 0x7f67441976a0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc5fb4000 con 0x556cc6045c00 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc5fb1530 con 0x556cc6057dc0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f674dc15740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc5facd10 con 0x556cc60465b0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6743fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6046a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6743fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6046a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57720/0 (socket says 172.21.15.3:57720) 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6743fff640 1 -- 172.21.15.3:0/2482375694 learned_addr learned my addr 172.21.15.3:0/2482375694 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f67495dd640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc605a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.680+0000 7f6748ddc640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc6045fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67300a2230 0x7f67300a46f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f672c00bac0 tx=0x7f672c00bec0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 -- 172.21.15.3:0/2482375694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 msgr2=0x556cc605a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc605a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 -- 172.21.15.3:0/2482375694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 msgr2=0x556cc6045fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.689 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc6045fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 -- 172.21.15.3:0/2482375694 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cc5f53210 con 0x556cc60465b0 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f67495dd640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc605a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6748ddc640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc6045fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f6743fff640 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6046a30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f672c014520 tx=0x7f672c01e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80709a193f69a584 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f673affd640 1 -- 172.21.15.3:0/2482375694 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c014ea0 con 0x556cc60465b0 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f673affd640 1 -- 172.21.15.3:0/2482375694 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f672c01a070 con 0x556cc60465b0 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.681+0000 7f673affd640 1 -- 172.21.15.3:0/2482375694 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c004d70 con 0x556cc60465b0 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 -- 172.21.15.3:0/2482375694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 msgr2=0x556cc6046a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6046a30 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f672c014520 tx=0x7f672c01e000 comp rx=0 tx=0).stop 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 -- 172.21.15.3:0/2482375694 shutdown_connections 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc605a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6046a30 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 --2- 172.21.15.3:0/2482375694 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc6045fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 -- 172.21.15.3:0/2482375694 >> 172.21.15.3:0/2482375694 conn(0x556cc60408c0 msgr2=0x556cc6040ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 -- 172.21.15.3:0/2482375694 shutdown_connections 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 -- 172.21.15.3:0/2482375694 wait complete. 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 -- start start 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.682+0000 7f674dc15740 1 Processor -- start 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6045c00 0x556cc60480e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6048620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 0x556cc6048b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc5f51e60 con 0x556cc6057dc0 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc5c804b0 con 0x556cc6045c00 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc5f32670 con 0x556cc60465b0 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f674dc15740 10 client.0 _ll_register_callbacks cb 0x556cc5f4a190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 0x556cc6048b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 0x556cc6048b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44942/0 (socket says 172.21.15.3:44942) 2024-02-09T23:01:25.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 -- 172.21.15.3:0/3259742720 learned_addr learned my addr 172.21.15.3:0/3259742720 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f6748ddc640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6045c00 0x556cc60480e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f6743fff640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6048620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 -- 172.21.15.3:0/3259742720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 msgr2=0x556cc6048620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6048620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 -- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6045c00 msgr2=0x556cc60480e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6045c00 0x556cc60480e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 -- 172.21.15.3:0/3259742720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f672c003e50 con 0x556cc6057dc0 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f6748ddc640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6045c00 0x556cc60480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f6743fff640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6048620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f67495dd640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 0x556cc6048b60 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f673402c780 tx=0x7f67340bc6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b42f36b0423592 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.683+0000 7f6723fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f6723fff640 1 -- 172.21.15.3:0/3259742720 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67340c0dd0 con 0x556cc6057dc0 2024-02-09T23:01:25.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f6723fff640 1 -- 172.21.15.3:0/3259742720 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f67340c5420 con 0x556cc6057dc0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556cc5cac860 con 0x556cc6057dc0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f6723fff640 1 -- 172.21.15.3:0/3259742720 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67340b92f0 con 0x556cc6057dc0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 10 client.19257 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x556cc5fab390 con 0x556cc6057dc0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f6723fff640 1 -- 172.21.15.3:0/3259742720 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6734086080 con 0x556cc6057dc0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f6723fff640 1 -- 172.21.15.3:0/3259742720 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f67340db020 con 0x556cc6057dc0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 10 client.19257 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 10 client.19257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 20 client.19257 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc6067be0 0x556cc606a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 4 client.19257 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556cc5cfd410 con 0x556cc6067be0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.684+0000 7f6748ddc640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc6067be0 0x556cc606a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.685+0000 7f6723fff640 10 client.19257 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.685+0000 7f6748ddc640 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc6067be0 0x556cc606a000 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=379695121fd00985 server_cookie=77fe6982df8a4c32 in_seq=0 out_seq=0 2024-02-09T23:01:25.694 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:25 smithi064 ceph-mon[36638]: pgmap v318: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f6723fff640 1 -- 172.21.15.3:0/3259742720 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556cc5cfd410 con 0x556cc6067be0 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f6723fff640 10 client.19257 handle_command_reply: tid=0 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 client.19257 shutdown 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 msgr2=0x556cc6048b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 0x556cc6048b60 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7f673402c780 tx=0x7f67340bc6a0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 shutdown_connections 2024-02-09T23:01:25.695 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc6067be0 0x556cc606a000 crc :-1 s=READY pgs=287 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6057dc0 0x556cc6048b60 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc6048620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 --2- 172.21.15.3:0/3259742720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6045c00 0x556cc60480e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f6723fff640 0 client.19257 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 >> 172.21.15.3:0/3259742720 conn(0x556cc60408c0 msgr2=0x556cc6042610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 shutdown_connections 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 1 -- 172.21.15.3:0/3259742720 wait complete. 2024-02-09T23:01:25.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.686+0000 7f674dc15740 20 client.19257 trim_cache size 0 max 16384 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 Processor -- start 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 -- start start 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc604a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc60470d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc6055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc605a610 con 0x556cc6045c00 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc606f1f0 con 0x556cc6057dc0 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f674dc15740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc603aad0 con 0x556cc60465b0 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f6748ddc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc604a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f6748ddc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc604a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44952/0 (socket says 172.21.15.3:44952) 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.701+0000 7f6748ddc640 1 -- 172.21.15.3:0/1995606585 learned_addr learned my addr 172.21.15.3:0/1995606585 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f6748ddc640 1 -- 172.21.15.3:0/1995606585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 msgr2=0x556cc60470d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f6748ddc640 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc60470d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f6748ddc640 1 -- 172.21.15.3:0/1995606585 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 msgr2=0x556cc6055ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f6748ddc640 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc6055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f6748ddc640 1 -- 172.21.15.3:0/1995606585 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cc5f47c70 con 0x556cc6045c00 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f6748ddc640 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc604a810 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f673c0094f0 tx=0x7f673c013bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47348f83faa357cb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f673a7fc640 1 -- 172.21.15.3:0/1995606585 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c017040 con 0x556cc6045c00 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f673a7fc640 1 -- 172.21.15.3:0/1995606585 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f673c022070 con 0x556cc6045c00 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.702+0000 7f673a7fc640 1 -- 172.21.15.3:0/1995606585 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c010d40 con 0x556cc6045c00 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- 172.21.15.3:0/1995606585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 msgr2=0x556cc604a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc604a810 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f673c0094f0 tx=0x7f673c013bd0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- 172.21.15.3:0/1995606585 shutdown_connections 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc6057dc0 0x556cc6055ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc60465b0 0x556cc60470d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- 172.21.15.3:0/1995606585 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc604a810 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- 172.21.15.3:0/1995606585 >> 172.21.15.3:0/1995606585 conn(0x556cc60408c0 msgr2=0x556cc6040ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- 172.21.15.3:0/1995606585 shutdown_connections 2024-02-09T23:01:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- 172.21.15.3:0/1995606585 wait complete. 2024-02-09T23:01:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- start start 2024-02-09T23:01:25.708 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 Processor -- start 2024-02-09T23:01:25.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc60b8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc60465b0 0x556cc60b8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc6057dc0 0x556cc605e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc605e8a0 con 0x556cc6045c00 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc605ea10 con 0x556cc60465b0 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cc605ebd0 con 0x556cc6057dc0 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f674dc15740 10 client.0 _ll_register_callbacks cb 0x7f67340b8f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f6748ddc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc60b8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f6748ddc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc60b8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44960/0 (socket says 172.21.15.3:44960) 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.703+0000 7f6748ddc640 1 -- 172.21.15.3:0/166218985 learned_addr learned my addr 172.21.15.3:0/166218985 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6743fff640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc60465b0 0x556cc60b8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f67495dd640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc6057dc0 0x556cc605e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6748ddc640 1 -- 172.21.15.3:0/166218985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc6057dc0 msgr2=0x556cc605e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6748ddc640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc6057dc0 0x556cc605e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6748ddc640 1 -- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc60465b0 msgr2=0x556cc60b8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6748ddc640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc60465b0 0x556cc60b8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6748ddc640 1 -- 172.21.15.3:0/166218985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f673c009d40 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6748ddc640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc60b8750 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f673c010960 tx=0x7f673c004e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8038c3981cbc5a8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6723fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6723fff640 1 -- 172.21.15.3:0/166218985 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c017050 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6723fff640 1 -- 172.21.15.3:0/166218985 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f673c010cc0 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f672c01a070 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f6723fff640 1 -- 172.21.15.3:0/166218985 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7f673c010e80 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f674dc15740 10 client.19263 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.704+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x556cc5f6dac0 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f6723fff640 1 -- 172.21.15.3:0/166218985 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f672c01a070 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f6723fff640 1 -- 172.21.15.3:0/166218985 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7f673c004280 con 0x556cc6045c00 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f674dc15740 10 client.19263 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f674dc15740 10 client.19263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:25.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f674dc15740 20 client.19263 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f674dc15740 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc60c05e0 0x556cc60c2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f674dc15740 4 client.19263 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x556cc5ce97b0 con 0x556cc60c05e0 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f6743fff640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc60c05e0 0x556cc60c2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f6723fff640 10 client.19263 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.705+0000 7f6743fff640 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc60c05e0 0x556cc60c2a00 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51ff2dc74831ace0 server_cookie=f51b47becf9d47c4 in_seq=0 out_seq=0 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f6723fff640 1 -- 172.21.15.3:0/166218985 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x556cc5ce97b0 con 0x556cc60c05e0 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f6723fff640 10 client.19263 handle_command_reply: tid=0 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 client.19263 shutdown 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 msgr2=0x556cc60b8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc60b8750 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f673c010960 tx=0x7f673c004e40 comp rx=0 tx=0).stop 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 shutdown_connections 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556cc60c05e0 0x556cc60c2a00 crc :-1 s=READY pgs=288 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.714 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556cc60465b0 0x556cc60b8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f6723fff640 0 client.19263 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556cc6057dc0 0x556cc605e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 --2- 172.21.15.3:0/166218985 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556cc6045c00 0x556cc60b8750 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 >> 172.21.15.3:0/166218985 conn(0x556cc60408c0 msgr2=0x556cc603f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 shutdown_connections 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 1 -- 172.21.15.3:0/166218985 wait complete. 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674dc15740 20 client.19263 trim_cache size 0 max 16384 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "c20fc305-9378-4242-bab7-cb23284aa6b1", 2024-02-09T23:01:25.715 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:01:25.716 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:01:25.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67300a2230 msgr2=0x7f67300a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.716 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674b040640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67300a2230 0x7f67300a46f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f672c00bac0 tx=0x7f672c00bec0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 msgr2=0x7f6744197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.707+0000 7f674b040640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 0x7f6744197b20 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f6734005940 tx=0x7f673402c6f0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 shutdown_connections 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f67300a2230 0x7f67300a46f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f67441053e0 0x7f674419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f67441976a0 0x7f6744197b20 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 --2- 172.21.15.3:0/1737930979 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6744100310 0x7f674419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 >> 172.21.15.3:0/1737930979 conn(0x7f6744077230 msgr2=0x7f67440fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 shutdown_connections 2024-02-09T23:01:25.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.708+0000 7f674b040640 1 -- 172.21.15.3:0/1737930979 wait complete. 2024-02-09T23:01:25.722 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:01:25.722 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:01:25.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:25 smithi089 ceph-mon[36665]: pgmap v318: 97 pgs: 97 active+clean; 132 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.6 MiB/s wr, 166 op/s 2024-02-09T23:01:25.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 -- 172.21.15.3:0/2432723263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102450 msgr2=0x7fa5e8102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 --2- 172.21.15.3:0/2432723263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102450 0x7fa5e8102830 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009900 tx=0x7fa5e402bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 -- 172.21.15.3:0/2432723263 shutdown_connections 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 --2- 172.21.15.3:0/2432723263 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5e8102d70 0x7fa5e810b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 --2- 172.21.15.3:0/2432723263 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 0x7fa5e810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 --2- 172.21.15.3:0/2432723263 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102450 0x7fa5e8102830 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 -- 172.21.15.3:0/2432723263 >> 172.21.15.3:0/2432723263 conn(0x7fa5e80772e0 msgr2=0x7fa5e80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 -- 172.21.15.3:0/2432723263 shutdown_connections 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.871+0000 7fa5f057a640 1 -- 172.21.15.3:0/2432723263 wait complete. 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 Processor -- start 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 -- start start 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5e8102450 0x7fa5e81000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 0x7fa5e81005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 0x7fa5e8100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e81a5c50 con 0x7fa5e8102d70 2024-02-09T23:01:25.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e81a5dc0 con 0x7fa5e8102450 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5f057a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5e81a5f80 con 0x7fa5e810b820 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5edb15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 0x7fa5e81005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5eeb17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 0x7fa5e8100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5edb15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 0x7fa5e81005e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44972/0 (socket says 172.21.15.3:44972) 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5ee316640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5e8102450 0x7fa5e81000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.872+0000 7fa5edb15640 1 -- 172.21.15.3:0/1371831914 learned_addr learned my addr 172.21.15.3:0/1371831914 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5edb15640 1 -- 172.21.15.3:0/1371831914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 msgr2=0x7fa5e8100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 0x7fa5e8100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5edb15640 1 -- 172.21.15.3:0/1371831914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5e8102450 msgr2=0x7fa5e81000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5e8102450 0x7fa5e81000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5edb15640 1 -- 172.21.15.3:0/1371831914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e4009560 con 0x7fa5e8102d70 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 0x7fa5e8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 0x7fa5e81005e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c920 tx=0x7fa5d800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5df7fe640 1 -- 172.21.15.3:0/1371831914 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8007c70 con 0x7fa5e8102d70 2024-02-09T23:01:25.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5df7fe640 1 -- 172.21.15.3:0/1371831914 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5d8007dd0 con 0x7fa5e8102d70 2024-02-09T23:01:25.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5e81a6260 con 0x7fa5e8102d70 2024-02-09T23:01:25.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5df7fe640 1 -- 172.21.15.3:0/1371831914 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d8018860 con 0x7fa5e8102d70 2024-02-09T23:01:25.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.873+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e81a6770 con 0x7fa5e8102d70 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.875+0000 7fa5df7fe640 1 -- 172.21.15.3:0/1371831914 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa5d800fac0 con 0x7fa5e8102d70 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 Processor -- start 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 -- start start 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5df7fe640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa5d40a2230 0x7fa5d40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4ead60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d4a5d00 con 0x56076d4f3eb0 2024-02-09T23:01:25.885 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d45a2f0 con 0x56076d4eb330 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5f322c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d457820 con 0x56076d4ea980 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5ee316640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4ead60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.876+0000 7fa5df7fe640 1 -- 172.21.15.3:0/1371831914 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa5d80ce1d0 con 0x7fa5e8102d70 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4ead60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57778/0 (socket says 172.21.15.3:57778) 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa5d40a2230 0x7fa5d40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 -- 172.21.15.3:0/3400122614 learned_addr learned my addr 172.21.15.3:0/3400122614 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5edb15640 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 -- 172.21.15.3:0/3400122614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 msgr2=0x56076d4f38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 -- 172.21.15.3:0/3400122614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 msgr2=0x56076d4f6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 -- 172.21.15.3:0/3400122614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56076d414ff0 con 0x56076d4ea980 2024-02-09T23:01:25.886 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa5d40a2230 0x7fa5d40a46f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0005fd0 tx=0x7fa5e0009f50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5ee316640 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4ead60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009900 tx=0x7fa5e4004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b1f547baf49e30c9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5dcbfe640 1 -- 172.21.15.3:0/3400122614 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4004a20 con 0x56076d4ea980 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5dcbfe640 1 -- 172.21.15.3:0/3400122614 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4004040 con 0x56076d4ea980 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.877+0000 7fa5dcbfe640 1 -- 172.21.15.3:0/3400122614 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e4005330 con 0x56076d4ea980 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 -- 172.21.15.3:0/3400122614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 msgr2=0x56076d4ead60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4ead60 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4009900 tx=0x7fa5e4004770 comp rx=0 tx=0).stop 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 -- 172.21.15.3:0/3400122614 shutdown_connections 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 --2- 172.21.15.3:0/3400122614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4ead60 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 -- 172.21.15.3:0/3400122614 >> 172.21.15.3:0/3400122614 conn(0x56076d4e5050 msgr2=0x56076d4e5430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 -- 172.21.15.3:0/3400122614 shutdown_connections 2024-02-09T23:01:25.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 -- 172.21.15.3:0/3400122614 wait complete. 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.878+0000 7fa5f322c740 1 -- start start 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 Processor -- start 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4f0d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d453000 con 0x56076d4f3eb0 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d126430 con 0x56076d4eb330 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d0e2e00 con 0x56076d4ea980 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5eeb17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4f0d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5f322c740 10 client.0 _ll_register_callbacks cb 0x56076d4477f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4f0d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:57784/0 (socket says 172.21.15.3:57784) 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5edb15640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 -- 172.21.15.3:0/4123187420 learned_addr learned my addr 172.21.15.3:0/4123187420 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 -- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 msgr2=0x56076d4f12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 -- 172.21.15.3:0/4123187420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 msgr2=0x56076d4f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5ee316640 1 -- 172.21.15.3:0/4123187420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e810e8d0 con 0x56076d4ea980 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.879+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5ee316640 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4f0d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4006010 tx=0x7fa5e402bcd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2029e2943b0ddf12 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/4123187420 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e40050b0 con 0x56076d4ea980 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/4123187420 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa5e4005210 con 0x56076d4ea980 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56076d1527e0 con 0x56076d4ea980 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/4123187420 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e40053d0 con 0x56076d4ea980 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 10 client.28285 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x56076d451680 con 0x56076d4ea980 2024-02-09T23:01:25.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/4123187420 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56076d1527e0 con 0x56076d4ea980 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/4123187420 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa5e403c050 con 0x56076d4ea980 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 10 client.28285 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 10 client.28285 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 20 client.28285 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d503db0 0x56076d5061d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 4 client.28285 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56076d18f720 con 0x56076d503db0 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.880+0000 7fa5edb15640 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d503db0 0x56076d5061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.881+0000 7fa5d1ffb640 10 client.28285 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.881+0000 7fa5edb15640 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d503db0 0x56076d5061d0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d69425e5d64f13b server_cookie=4f03895fa652ebc0 in_seq=0 out_seq=0 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.881+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/4123187420 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56076d18f720 con 0x56076d503db0 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.881+0000 7fa5d1ffb640 10 client.28285 handle_command_reply: tid=0 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.881+0000 7fa5f322c740 1 client.28285 shutdown 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 msgr2=0x56076d4f0d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4f0d90 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa5e4006010 tx=0x7fa5e402bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 shutdown_connections 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d503db0 0x56076d5061d0 crc :-1 s=READY pgs=289 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4eb330 0x56076d4f12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5d1ffb640 0 client.28285 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d4f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 --2- 172.21.15.3:0/4123187420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4ea980 0x56076d4f0d90 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 >> 172.21.15.3:0/4123187420 conn(0x56076d4e5050 msgr2=0x56076d4e6da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 shutdown_connections 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 1 -- 172.21.15.3:0/4123187420 wait complete. 2024-02-09T23:01:25.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.882+0000 7fa5f322c740 20 client.28285 trim_cache size 0 max 16384 2024-02-09T23:01:25.905 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 Processor -- start 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 -- start start 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f12e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 0x56076d4f1820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4f3eb0 0x56076d4edff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d3d7fd0 con 0x56076d4eb330 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d4f1d60 con 0x56076d4ea980 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5f322c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d500cf0 con 0x56076d4f3eb0 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 0x56076d4f1820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 0x56076d4f1820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44996/0 (socket says 172.21.15.3:44996) 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 -- 172.21.15.3:0/1487759118 learned_addr learned my addr 172.21.15.3:0/1487759118 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4f3eb0 0x56076d4edff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5ee316640 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f12e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 -- 172.21.15.3:0/1487759118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4f3eb0 msgr2=0x56076d4edff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4f3eb0 0x56076d4edff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 -- 172.21.15.3:0/1487759118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 msgr2=0x56076d4f12e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f12e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 -- 172.21.15.3:0/1487759118 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56076d3edb30 con 0x56076d4eb330 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.898+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 0x56076d4f1820 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8007590 tx=0x7fa5d800a5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c819e72b18d158c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5d3fff640 1 -- 172.21.15.3:0/1487759118 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d809ecb0 con 0x56076d4eb330 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5d3fff640 1 -- 172.21.15.3:0/1487759118 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5d809be90 con 0x56076d4eb330 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5d3fff640 1 -- 172.21.15.3:0/1487759118 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d80a2e70 con 0x56076d4eb330 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 -- 172.21.15.3:0/1487759118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 msgr2=0x56076d4f1820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 0x56076d4f1820 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7fa5d8007590 tx=0x7fa5d800a5e0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 -- 172.21.15.3:0/1487759118 shutdown_connections 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4f3eb0 0x56076d4edff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4eb330 0x56076d4f1820 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1487759118 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f12e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 -- 172.21.15.3:0/1487759118 >> 172.21.15.3:0/1487759118 conn(0x56076d4e5050 msgr2=0x56076d4ec080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 -- 172.21.15.3:0/1487759118 shutdown_connections 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.899+0000 7fa5f322c740 1 -- 172.21.15.3:0/1487759118 wait complete. 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 -- start start 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 Processor -- start 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f9f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4eb330 0x56076d4fa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d55a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d55a7e0 con 0x56076d4f3eb0 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d55a920 con 0x56076d4ea980 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56076d55aab0 con 0x56076d4eb330 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5f322c740 10 client.0 _ll_register_callbacks cb 0x7fa5e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d55a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d55a2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45012/0 (socket says 172.21.15.3:45012) 2024-02-09T23:01:25.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 -- 172.21.15.3:0/1942486063 learned_addr learned my addr 172.21.15.3:0/1942486063 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4eb330 0x56076d4fa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5ee316640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f9f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 -- 172.21.15.3:0/1942486063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4eb330 msgr2=0x56076d4fa4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4eb330 0x56076d4fa4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 -- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 msgr2=0x56076d4f9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f9f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 -- 172.21.15.3:0/1942486063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5e81a6770 con 0x56076d4f3eb0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5edb15640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4eb330 0x56076d4fa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5eeb17640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d55a2a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fa5e001e9c0 tx=0x7fa5e0004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e475909c3334f7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.900+0000 7fa5d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/1942486063 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e002d070 con 0x56076d4f3eb0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/1942486063 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa5e0010c80 con 0x56076d4f3eb0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5e4004040 con 0x56076d4f3eb0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/1942486063 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e0023070 con 0x56076d4f3eb0 2024-02-09T23:01:25.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 10 client.19281 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x56076d413850 con 0x56076d4f3eb0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/1942486063 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5e4004040 con 0x56076d4f3eb0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/1942486063 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fa5e0030020 con 0x56076d4f3eb0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 10 client.19281 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 10 client.19281 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 20 client.19281 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d55c3c0 0x56076d55e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 4 client.19281 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56076d1de890 con 0x56076d55c3c0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.901+0000 7fa5ee316640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d55c3c0 0x56076d55e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5ee316640 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d55c3c0 0x56076d55e7e0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=818ab6bbed9d151e server_cookie=dab37e699195f7d1 in_seq=0 out_seq=0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5d1ffb640 10 client.19281 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5d1ffb640 1 -- 172.21.15.3:0/1942486063 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x56076d1de890 con 0x56076d55c3c0 2024-02-09T23:01:25.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5d1ffb640 10 client.19281 handle_command_reply: tid=0 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 client.19281 shutdown 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 msgr2=0x56076d55a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d55a2a0 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 crypto rx=0x7fa5e001e9c0 tx=0x7fa5e0004ab0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 shutdown_connections 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56076d55c3c0 0x56076d55e7e0 crc :-1 s=READY pgs=290 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56076d4f3eb0 0x56076d55a2a0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56076d4eb330 0x56076d4fa4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5d1ffb640 0 client.19281 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 --2- 172.21.15.3:0/1942486063 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56076d4ea980 0x56076d4f9f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 >> 172.21.15.3:0/1942486063 conn(0x56076d4e5050 msgr2=0x56076d4e3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 shutdown_connections 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 1 -- 172.21.15.3:0/1942486063 wait complete. 2024-02-09T23:01:25.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.902+0000 7fa5f322c740 20 client.19281 trim_cache size 0 max 16384 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: "c20fc305-9378-4242-bab7-cb23284aa6b1": { 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "c20fc305-9378-4242-bab7-cb23284aa6b1", 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:01:25.912 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:01:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa5d40a2230 msgr2=0x7fa5d40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa5d40a2230 0x7fa5d40a46f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa5e0005fd0 tx=0x7fa5e0009f50 comp rx=0 tx=0).stop 2024-02-09T23:01:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 msgr2=0x7fa5e81005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 0x7fa5e81005e0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7fa5d800c920 tx=0x7fa5d800cde0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 shutdown_connections 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa5d40a2230 0x7fa5d40a46f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa5e810b820 0x7fa5e8100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa5e8102d70 0x7fa5e81005e0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 --2- 172.21.15.3:0/1371831914 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa5e8102450 0x7fa5e81000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 >> 172.21.15.3:0/1371831914 conn(0x7fa5e80772e0 msgr2=0x7fa5e80fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 shutdown_connections 2024-02-09T23:01:25.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:25.903+0000 7fa5f057a640 1 -- 172.21.15.3:0/1371831914 wait complete. 2024-02-09T23:01:25.916 INFO:tasks.cephfs.filesystem:scrub status for tag:c20fc305-9378-4242-bab7-cb23284aa6b1 - {'path': '/', 'tag': 'c20fc305-9378-4242-bab7-cb23284aa6b1', 'options': 'recursive,force'} 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:26.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:26 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 340774 3.21 MB/sec execute 440 sec latency 912.923 ms 2024-02-09T23:01:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:01:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:26.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:26 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:26.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:26.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:26 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:27 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:27 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:27 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:27 smithi003 ceph-mon[32678]: pgmap v319: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 271 op/s 2024-02-09T23:01:27.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 342037 3.22 MB/sec execute 441 sec latency 14.112 ms 2024-02-09T23:01:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:27 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:27 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:27 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:27 smithi064 ceph-mon[36638]: pgmap v319: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 271 op/s 2024-02-09T23:01:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:27 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:27 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:27 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:27.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:27 smithi089 ceph-mon[36665]: pgmap v319: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 271 op/s 2024-02-09T23:01:28.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:28 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:01:28.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 342210 3.21 MB/sec execute 442 sec latency 824.148 ms 2024-02-09T23:01:28.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:28 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:01:28.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:28 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:01:29.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 342210 3.21 MB/sec execute 443 sec latency 1824.188 ms 2024-02-09T23:01:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: pgmap v320: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-02-09T23:01:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:01:30.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:01:30.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 342210 3.20 MB/sec execute 444 sec latency 2824.238 ms 2024-02-09T23:01:30.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: pgmap v320: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-02-09T23:01:30.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:01:30.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:01:30.688 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: pgmap v320: 97 pgs: 97 active+clean; 134 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.3 MiB/s wr, 166 op/s 2024-02-09T23:01:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-02-09T23:01:30.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:01:30.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:01:30.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:01:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:31 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-02-09T23:01:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:31 smithi003 ceph-mon[32678]: Unable to set osd_memory_target on smithi089 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-02-09T23:01:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:31 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi064 to 174.1M 2024-02-09T23:01:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:31 smithi003 ceph-mon[32678]: Unable to set osd_memory_target on smithi064 to 182641356: error parsing value: Value '182641356' is below minimum 939524096 2024-02-09T23:01:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:31 smithi003 ceph-mon[32678]: Adjusting osd_memory_target on smithi003 to 177.2M 2024-02-09T23:01:31.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:31 smithi003 ceph-mon[32678]: Unable to set osd_memory_target on smithi003 to 185898496: error parsing value: Value '185898496' is below minimum 939524096 2024-02-09T23:01:31.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 342210 3.19 MB/sec execute 445 sec latency 3824.282 ms 2024-02-09T23:01:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:31 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-02-09T23:01:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:31 smithi064 ceph-mon[36638]: Unable to set osd_memory_target on smithi089 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-02-09T23:01:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:31 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi064 to 174.1M 2024-02-09T23:01:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:31 smithi064 ceph-mon[36638]: Unable to set osd_memory_target on smithi064 to 182641356: error parsing value: Value '182641356' is below minimum 939524096 2024-02-09T23:01:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:31 smithi064 ceph-mon[36638]: Adjusting osd_memory_target on smithi003 to 177.2M 2024-02-09T23:01:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:31 smithi064 ceph-mon[36638]: Unable to set osd_memory_target on smithi003 to 185898496: error parsing value: Value '185898496' is below minimum 939524096 2024-02-09T23:01:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:31 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi089 to 174.1M 2024-02-09T23:01:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:31 smithi089 ceph-mon[36665]: Unable to set osd_memory_target on smithi089 to 182576128: error parsing value: Value '182576128' is below minimum 939524096 2024-02-09T23:01:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:31 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi064 to 174.1M 2024-02-09T23:01:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:31 smithi089 ceph-mon[36665]: Unable to set osd_memory_target on smithi064 to 182641356: error parsing value: Value '182641356' is below minimum 939524096 2024-02-09T23:01:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:31 smithi089 ceph-mon[36665]: Adjusting osd_memory_target on smithi003 to 177.2M 2024-02-09T23:01:31.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:31 smithi089 ceph-mon[36665]: Unable to set osd_memory_target on smithi003 to 185898496: error parsing value: Value '185898496' is below minimum 939524096 2024-02-09T23:01:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:32 smithi003 ceph-mon[32678]: pgmap v321: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 237 op/s 2024-02-09T23:01:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:32.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:32 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:32.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 344309 3.21 MB/sec execute 446 sec latency 3870.440 ms 2024-02-09T23:01:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:32 smithi064 ceph-mon[36638]: pgmap v321: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 237 op/s 2024-02-09T23:01:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:32.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:32 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:32 smithi089 ceph-mon[36665]: pgmap v321: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 237 op/s 2024-02-09T23:01:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:32.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:32 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:33.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:33 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:01:33.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 345562 3.22 MB/sec execute 447 sec latency 12.934 ms 2024-02-09T23:01:33.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:33 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:01:33.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:33 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:01:34.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:34 smithi003 ceph-mon[32678]: pgmap v322: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-02-09T23:01:34.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 345759 3.21 MB/sec execute 448 sec latency 758.877 ms 2024-02-09T23:01:34.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:34 smithi064 ceph-mon[36638]: pgmap v322: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-02-09T23:01:34.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:34 smithi089 ceph-mon[36665]: pgmap v322: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-02-09T23:01:35.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.033+0000 7fb3b306f640 1 -- 172.21.15.3:0/2579170539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0ffb30 msgr2=0x7fb3ac0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.033+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2579170539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0ffb30 0x7fb3ac0fffb0 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7fb398009900 tx=0x7fb39802bc80 comp rx=0 tx=0).stop 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 -- 172.21.15.3:0/2579170539 shutdown_connections 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2579170539 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2579170539 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac0ffb30 0x7fb3ac0fffb0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2579170539 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac1073c0 0x7fb3ac1077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 -- 172.21.15.3:0/2579170539 >> 172.21.15.3:0/2579170539 conn(0x7fb3ac075440 msgr2=0x7fb3ac075850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 -- 172.21.15.3:0/2579170539 shutdown_connections 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 -- 172.21.15.3:0/2579170539 wait complete. 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 Processor -- start 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 -- start start 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3ac0ffb30 0x7fb3ac100d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.039 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.034+0000 7fb3b306f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac101250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b306f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac1073c0 0x7fb3ac101790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b306f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac1a5c90 con 0x7fb3ac1004f0 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b306f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac1a5e00 con 0x7fb3ac0ffb30 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b306f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac1a5fc0 con 0x7fb3ac1073c0 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac101250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac101250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49688/0 (socket says 172.21.15.3:49688) 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 -- 172.21.15.3:0/2747885171 learned_addr learned my addr 172.21.15.3:0/2747885171 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3ac0ffb30 0x7fb3ac100d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b160c640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac1073c0 0x7fb3ac101790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 -- 172.21.15.3:0/2747885171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac1073c0 msgr2=0x7fb3ac101790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac1073c0 0x7fb3ac101790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 -- 172.21.15.3:0/2747885171 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3ac0ffb30 msgr2=0x7fb3ac100d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3ac0ffb30 0x7fb3ac100d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.040 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 -- 172.21.15.3:0/2747885171 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb398009560 con 0x7fb3ac1004f0 2024-02-09T23:01:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac101250 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb398005940 tx=0x7fb398004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3a9ffb640 1 -- 172.21.15.3:0/2747885171 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398004440 con 0x7fb3ac1004f0 2024-02-09T23:01:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3a9ffb640 1 -- 172.21.15.3:0/2747885171 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3980052d0 con 0x7fb3ac1004f0 2024-02-09T23:01:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac1a6240 con 0x7fb3ac1004f0 2024-02-09T23:01:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.035+0000 7fb3a9ffb640 1 -- 172.21.15.3:0/2747885171 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398003da0 con 0x7fb3ac1004f0 2024-02-09T23:01:35.041 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.036+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac1a6750 con 0x7fb3ac1004f0 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.038+0000 7fb3a9ffb640 1 -- 172.21.15.3:0/2747885171 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb398005440 con 0x7fb3ac1004f0 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.038+0000 7fb3b5c44740 1 Processor -- start 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 -- start start 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3a9ffb640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb39c0a2220 0x7fb39c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cbb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6c29a00 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6c26f30 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6c22710 con 0x55aeb6cc47a0 2024-02-09T23:01:35.048 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b160c640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb39c0a2220 0x7fb39c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc4260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49698/0 (socket says 172.21.15.3:49698) 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3a9ffb640 1 -- 172.21.15.3:0/2747885171 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb3980f01c0 con 0x7fb3ac1004f0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 -- 172.21.15.3:0/911518566 learned_addr learned my addr 172.21.15.3:0/911518566 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cbb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b160c640 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 -- 172.21.15.3:0/911518566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 msgr2=0x55aeb6cc6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 -- 172.21.15.3:0/911518566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 msgr2=0x55aeb6cbb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cbb490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 -- 172.21.15.3:0/911518566 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeb6be73b0 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3abfff640 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc4260 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb398045020 tx=0x7fb398003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1dd92130bd47b4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.039+0000 7fb3b160c640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb39c0a2220 0x7fb39c0a46e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4005c40 tx=0x7fb3a40129e0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:35.049 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb396ffd640 1 -- 172.21.15.3:0/911518566 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398097070 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb396ffd640 1 -- 172.21.15.3:0/911518566 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3980bc5c0 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb396ffd640 1 -- 172.21.15.3:0/911518566 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3980c5400 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 -- 172.21.15.3:0/911518566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 msgr2=0x55aeb6cc4260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc4260 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fb398045020 tx=0x7fb398003f90 comp rx=0 tx=0).stop 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 -- 172.21.15.3:0/911518566 shutdown_connections 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc4260 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/911518566 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cbb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 -- 172.21.15.3:0/911518566 >> 172.21.15.3:0/911518566 conn(0x55aeb6cb59d0 msgr2=0x55aeb6cb5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 -- 172.21.15.3:0/911518566 shutdown_connections 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.040+0000 7fb3b5c44740 1 -- 172.21.15.3:0/911518566 wait complete. 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 -- start start 2024-02-09T23:01:35.050 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 Processor -- start 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc1d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc2280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6bc7b90 con 0x55aeb6cc47a0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb68f6250 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6ba8200 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b5c44740 10 client.0 _ll_register_callbacks cb 0x55aeb6bde5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b160c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b160c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc2280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49702/0 (socket says 172.21.15.3:49702) 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.041+0000 7fb3b160c640 1 -- 172.21.15.3:0/987729531 learned_addr learned my addr 172.21.15.3:0/987729531 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3abfff640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.051 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b160c640 1 -- 172.21.15.3:0/987729531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cbb0b0 msgr2=0x55aeb6cc1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b160c640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b160c640 1 -- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 msgr2=0x55aeb6cc1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b160c640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc1d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b160c640 1 -- 172.21.15.3:0/987729531 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac1a6750 con 0x55aeb6cc47a0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3abfff640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b160c640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc2280 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4010d80 tx=0x7fb3a4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76ccd4d12857ec7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb38bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb38bfff640 1 -- 172.21.15.3:0/987729531 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a400de60 con 0x55aeb6cc47a0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb38bfff640 1 -- 172.21.15.3:0/987729531 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb3a4004b20 con 0x55aeb6cc47a0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55aeb6c48ee0 con 0x55aeb6cc47a0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb38bfff640 1 -- 172.21.15.3:0/987729531 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3a4004ce0 con 0x55aeb6cc47a0 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b5c44740 10 client.19293 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:35.052 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.042+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55aeb6c20d90 con 0x55aeb6cc47a0 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb38bfff640 1 -- 172.21.15.3:0/987729531 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55aeb6c48ee0 con 0x55aeb6cc47a0 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb38bfff640 1 -- 172.21.15.3:0/987729531 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3a40044d0 con 0x55aeb6cc47a0 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b5c44740 10 client.19293 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b5c44740 10 client.19293 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b5c44740 20 client.19293 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6cd4800 0x55aeb6cd6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b5c44740 4 client.19293 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aeb6922600 con 0x55aeb6cd4800 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6cd4800 0x55aeb6cd6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6cd4800 0x55aeb6cd6c20 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c48a45cd91b788c server_cookie=31565c6ff444be48 in_seq=0 out_seq=0 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.043+0000 7fb38bfff640 10 client.19293 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb38bfff640 1 -- 172.21.15.3:0/987729531 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aeb6922600 con 0x55aeb6cd4800 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb38bfff640 10 client.19293 handle_command_reply: tid=0 2024-02-09T23:01:35.053 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 client.19293 shutdown 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 msgr2=0x55aeb6cc2280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc2280 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4010d80 tx=0x7fb3a4004ab0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 shutdown_connections 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6cd4800 0x55aeb6cd6c20 crc :-1 s=READY pgs=291 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc1d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc2280 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/987729531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb38bfff640 0 client.19293 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 >> 172.21.15.3:0/987729531 conn(0x55aeb6cb59d0 msgr2=0x55aeb6cb7720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 shutdown_connections 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.044+0000 7fb3b5c44740 1 -- 172.21.15.3:0/987729531 wait complete. 2024-02-09T23:01:35.054 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.045+0000 7fb3b5c44740 20 client.19293 trim_cache size 0 max 16384 2024-02-09T23:01:35.062 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 Processor -- start 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 -- start start 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cbc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6cb5270 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6cb69d0 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6cb5500 con 0x55aeb6cc47a0 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cbc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cbc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49712/0 (socket says 172.21.15.3:49712) 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 -- 172.21.15.3:0/2150905492 learned_addr learned my addr 172.21.15.3:0/2150905492 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.063 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3b160c640 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 -- 172.21.15.3:0/2150905492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 msgr2=0x55aeb6cc20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 -- 172.21.15.3:0/2150905492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 msgr2=0x55aeb6cc3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.058+0000 7fb3abfff640 1 -- 172.21.15.3:0/2150905492 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aeb6bbd820 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.059+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cbc1c0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb398006010 tx=0x7fb3980bd410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f64ad3857bec028 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.059+0000 7fb3967fc640 1 -- 172.21.15.3:0/2150905492 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3980c5400 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.059+0000 7fb3967fc640 1 -- 172.21.15.3:0/2150905492 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3980bcc60 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.059+0000 7fb3967fc640 1 -- 172.21.15.3:0/2150905492 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3980c4f70 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.059+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2150905492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 msgr2=0x55aeb6cbc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.059+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cbc1c0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7fb398006010 tx=0x7fb3980bd410 comp rx=0 tx=0).stop 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2150905492 shutdown_connections 2024-02-09T23:01:35.064 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cc20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cbc1c0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2150905492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6cc3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2150905492 >> 172.21.15.3:0/2150905492 conn(0x55aeb6cb59d0 msgr2=0x55aeb6cc00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2150905492 shutdown_connections 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2150905492 wait complete. 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- start start 2024-02-09T23:01:35.065 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 Processor -- start 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6ccb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc9a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6cca810 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6d2ad10 con 0x55aeb6cbb9d0 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aeb6d2ae50 con 0x55aeb6cc47a0 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b5c44740 10 client.0 _ll_register_callbacks cb 0x7fb3a4010a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b0e0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6ccb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b0e0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6ccb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:49718/0 (socket says 172.21.15.3:49718) 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.060+0000 7fb3b0e0b640 1 -- 172.21.15.3:0/2108974808 learned_addr learned my addr 172.21.15.3:0/2108974808 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.068 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b160c640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b0e0b640 1 -- 172.21.15.3:0/2108974808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 msgr2=0x55aeb6cca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b0e0b640 1 -- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 msgr2=0x55aeb6cc9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b0e0b640 1 -- 172.21.15.3:0/2108974808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aeb6be73b0 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b160c640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b0e0b640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6ccb6f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0004700 tx=0x7fb3a001c6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be2739bb09e472f1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb38bfff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb38bfff640 1 -- 172.21.15.3:0/2108974808 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a0017040 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb38bfff640 1 -- 172.21.15.3:0/2108974808 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb3a0011d00 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3980bc5c0 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb38bfff640 1 -- 172.21.15.3:0/2108974808 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3a0005020 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b5c44740 10 client.19299 fetch_fsmap learned FSMap version 47 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) v3 -- 0x55aeb6be36f0 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.069 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.061+0000 7fb38bfff640 1 -- 172.21.15.3:0/2108974808 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3a0030080 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb38bfff640 1 -- 172.21.15.3:0/2108974808 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 47) v1 ==== 7882+0+0 (secure 0 0 0) 0x7fb3a002c410 con 0x55aeb6cbb0b0 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3b5c44740 10 client.19299 fetch_fsmap finished waiting for FSMap version 47 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3b5c44740 10 client.19299 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3b5c44740 20 client.19299 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6d2c640 0x55aeb6d2ea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3b5c44740 4 client.19299 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55aeb69731b0 con 0x55aeb6d2c640 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6d2c640 0x55aeb6d2ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb38bfff640 10 client.19299 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:35.070 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.062+0000 7fb3abfff640 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6d2c640 0x55aeb6d2ea60 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=607377fdf7c912f4 server_cookie=b25ed93098915dac in_seq=0 out_seq=0 2024-02-09T23:01:35.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb38bfff640 1 -- 172.21.15.3:0/2108974808 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55aeb69731b0 con 0x55aeb6d2c640 2024-02-09T23:01:35.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb38bfff640 10 client.19299 handle_command_reply: tid=0 2024-02-09T23:01:35.122 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 client.19299 shutdown 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 msgr2=0x55aeb6ccb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6ccb6f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7fb3a0004700 tx=0x7fb3a001c6e0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 shutdown_connections 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55aeb6d2c640 0x55aeb6d2ea60 crc :-1 s=READY pgs=292 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55aeb6cbb9d0 0x55aeb6cc9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55aeb6cc47a0 0x55aeb6cca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 --2- 172.21.15.3:0/2108974808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55aeb6cbb0b0 0x55aeb6ccb6f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb38bfff640 0 client.19299 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 >> 172.21.15.3:0/2108974808 conn(0x55aeb6cb59d0 msgr2=0x55aeb6cb41a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 shutdown_connections 2024-02-09T23:01:35.123 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 1 -- 172.21.15.3:0/2108974808 wait complete. 2024-02-09T23:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.119+0000 7fb3b5c44740 20 client.19299 trim_cache size 0 max 16384 2024-02-09T23:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:01:35.124 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb39c0a2220 msgr2=0x7fb39c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb39c0a2220 0x7fb39c0a46e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4005c40 tx=0x7fb3a40129e0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 msgr2=0x7fb3ac101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac101250 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7fb398005940 tx=0x7fb398004040 comp rx=0 tx=0).stop 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 shutdown_connections 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb39c0a2220 0x7fb39c0a46e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb3ac1073c0 0x7fb3ac101790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb3ac1004f0 0x7fb3ac101250 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 --2- 172.21.15.3:0/2747885171 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb3ac0ffb30 0x7fb3ac100d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 >> 172.21.15.3:0/2747885171 conn(0x7fb3ac075440 msgr2=0x7fb3ac0fe7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 shutdown_connections 2024-02-09T23:01:35.125 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:35.120+0000 7fb3b306f640 1 -- 172.21.15.3:0/2747885171 wait complete. 2024-02-09T23:01:35.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:35 smithi003 ceph-mon[32678]: pgmap v323: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-02-09T23:01:35.660 INFO:tasks.workunit.client.0.smithi003.stdout: 1 346826 3.22 MB/sec execute 449 sec latency 1203.509 ms 2024-02-09T23:01:35.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:35 smithi064 ceph-mon[36638]: pgmap v323: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-02-09T23:01:35.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:35 smithi089 ceph-mon[36665]: pgmap v323: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 176 op/s 2024-02-09T23:01:36.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:36 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:01:36.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347415 3.22 MB/sec execute 450 sec latency 744.037 ms 2024-02-09T23:01:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:36 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:01:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:36 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:01:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:37 smithi003 ceph-mon[32678]: pgmap v324: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 251 op/s 2024-02-09T23:01:37.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347415 3.21 MB/sec execute 451 sec latency 1744.119 ms 2024-02-09T23:01:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:37 smithi064 ceph-mon[36638]: pgmap v324: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 251 op/s 2024-02-09T23:01:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:37 smithi089 ceph-mon[36665]: pgmap v324: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 251 op/s 2024-02-09T23:01:38.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347415 3.20 MB/sec execute 452 sec latency 2744.240 ms 2024-02-09T23:01:39.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347415 3.20 MB/sec execute 453 sec latency 3744.321 ms 2024-02-09T23:01:40.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:40 smithi003 ceph-mon[32678]: pgmap v325: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-02-09T23:01:40.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347428 3.19 MB/sec execute 454 sec latency 4546.826 ms 2024-02-09T23:01:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:40 smithi064 ceph-mon[36638]: pgmap v325: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-02-09T23:01:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:40 smithi089 ceph-mon[36665]: pgmap v325: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 1.5 MiB/s wr, 145 op/s 2024-02-09T23:01:41.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:41 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:01:41.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347428 3.18 MB/sec execute 455 sec latency 1181.516 ms 2024-02-09T23:01:41.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:41 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:01:41.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:41 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:01:42.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:42 smithi003 ceph-mon[32678]: pgmap v326: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-02-09T23:01:42.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347428 3.18 MB/sec execute 456 sec latency 2181.616 ms 2024-02-09T23:01:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:42 smithi064 ceph-mon[36638]: pgmap v326: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-02-09T23:01:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:42 smithi089 ceph-mon[36665]: pgmap v326: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1022 KiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-02-09T23:01:43.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:43 smithi003 ceph-mon[32678]: pgmap v327: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-02-09T23:01:43.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347428 3.17 MB/sec execute 457 sec latency 3181.718 ms 2024-02-09T23:01:43.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:43 smithi064 ceph-mon[36638]: pgmap v327: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-02-09T23:01:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:43 smithi089 ceph-mon[36665]: pgmap v327: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-02-09T23:01:44.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347428 3.16 MB/sec execute 458 sec latency 4181.817 ms 2024-02-09T23:01:45.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 347826 3.16 MB/sec execute 459 sec latency 4983.482 ms 2024-02-09T23:01:46.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:46 smithi003 ceph-mon[32678]: pgmap v328: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-02-09T23:01:46.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 349116 3.17 MB/sec execute 460 sec latency 22.402 ms 2024-02-09T23:01:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:46 smithi064 ceph-mon[36638]: pgmap v328: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-02-09T23:01:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:46 smithi089 ceph-mon[36665]: pgmap v328: 97 pgs: 97 active+clean; 136 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 618 KiB/s rd, 1.1 MiB/s wr, 110 op/s 2024-02-09T23:01:47.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 349302 3.16 MB/sec execute 461 sec latency 889.027 ms 2024-02-09T23:01:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 -- 172.21.15.3:0/1254657974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4069840 msgr2=0x7fd5d4069ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- 172.21.15.3:0/1254657974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4069840 0x7fd5d4069ca0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009930 tx=0x7fd5c402bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 -- 172.21.15.3:0/1254657974 shutdown_connections 2024-02-09T23:01:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- 172.21.15.3:0/1254657974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d406a1e0 0x7fd5d410e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.223 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- 172.21.15.3:0/1254657974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4069840 0x7fd5d4069ca0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- 172.21.15.3:0/1254657974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d4107320 0x7fd5d4107700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 -- 172.21.15.3:0/1254657974 >> 172.21.15.3:0/1254657974 conn(0x7fd5d4076f80 msgr2=0x7fd5d4077390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 -- 172.21.15.3:0/1254657974 shutdown_connections 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 -- 172.21.15.3:0/1254657974 wait complete. 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 Processor -- start 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 -- start start 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d4069840 0x7fd5d419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d406a1e0 0x7fd5d419d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.217+0000 7fd5db017640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 0x7fd5d41975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5db017640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4197c00 con 0x7fd5d4107320 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5db017640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4197d70 con 0x7fd5d4069840 2024-02-09T23:01:48.224 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5db017640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d4197f30 con 0x7fd5d406a1e0 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 0x7fd5d41975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 0x7fd5d41975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44804/0 (socket says 172.21.15.3:44804) 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3679972390 learned_addr learned my addr 172.21.15.3:0/3679972390 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d4069840 0x7fd5d419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d3fff640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d406a1e0 0x7fd5d419d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3679972390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d406a1e0 msgr2=0x7fd5d419d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d406a1e0 0x7fd5d419d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3679972390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d4069840 msgr2=0x7fd5d419d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d4069840 0x7fd5d419d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3679972390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5c4009560 con 0x7fd5d4107320 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 0x7fd5d41975b0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00d680 tx=0x7fd5bc00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d1ffb640 1 -- 172.21.15.3:0/3679972390 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc005220 con 0x7fd5d4107320 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d1ffb640 1 -- 172.21.15.3:0/3679972390 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5bc009d00 con 0x7fd5d4107320 2024-02-09T23:01:48.225 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d4198210 con 0x7fd5d4107320 2024-02-09T23:01:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.218+0000 7fd5d1ffb640 1 -- 172.21.15.3:0/3679972390 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc004dd0 con 0x7fd5d4107320 2024-02-09T23:01:48.226 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.219+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d41a44a0 con 0x7fd5d4107320 2024-02-09T23:01:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.221+0000 7fd5d1ffb640 1 -- 172.21.15.3:0/3679972390 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd5bc009820 con 0x7fd5d4107320 2024-02-09T23:01:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d1ffb640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd5c00a22a0 0x7fd5c00a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 Processor -- start 2024-02-09T23:01:48.232 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 -- start start 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd5c00a22a0 0x7fd5c00a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d1ffb640 1 -- 172.21.15.3:0/3679972390 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd5bc0cd970 con 0x7fd5d4107320 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f2895be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f28969670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f2896bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f288caa00 con 0x558f28969bb0 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f288c7f30 con 0x558f2895ba70 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f288c3710 con 0x558f2895c390 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d95b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f2896bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d95b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f2896bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44818/0 (socket says 172.21.15.3:44818) 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3305905330 learned_addr learned my addr 172.21.15.3:0/3305905330 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.222+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd5c00a22a0 0x7fd5c00a4760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc007ec0 tx=0x7fd5cc0075b0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:48.233 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d3fff640 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f28969670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f2895be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3305905330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 msgr2=0x558f28969670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f28969670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3305905330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 msgr2=0x558f2895be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f2895be50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d95b4640 1 -- 172.21.15.3:0/3305905330 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f288883b0 con 0x558f28969bb0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f2896bfa0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc096d80 tx=0x7fd5bc099d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76c581fce69dcc35 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5caffd640 1 -- 172.21.15.3:0/3305905330 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc09e450 con 0x558f28969bb0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5caffd640 1 -- 172.21.15.3:0/3305905330 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5bc0a2aa0 con 0x558f28969bb0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.223+0000 7fd5caffd640 1 -- 172.21.15.3:0/3305905330 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc096a20 con 0x558f28969bb0 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/3305905330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 msgr2=0x558f2896bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f2896bfa0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc096d80 tx=0x7fd5bc099d10 comp rx=0 tx=0).stop 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/3305905330 shutdown_connections 2024-02-09T23:01:48.234 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f2896bfa0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f28969670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/3305905330 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f2895be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/3305905330 >> 172.21.15.3:0/3305905330 conn(0x558f289569d0 msgr2=0x558f28956db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/3305905330 shutdown_connections 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/3305905330 wait complete. 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 -- start start 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.224+0000 7fd5ddbe0740 1 Processor -- start 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f28966ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f289671e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f28967720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f28868b90 con 0x558f28969bb0 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f28597250 con 0x558f2895ba70 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f28849200 con 0x558f2895c390 2024-02-09T23:01:48.235 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 10 client.0 _ll_register_callbacks cb 0x558f2887f5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f28967720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f28967720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44830/0 (socket says 172.21.15.3:44830) 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 -- 172.21.15.3:0/1571293148 learned_addr learned my addr 172.21.15.3:0/1571293148 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f28966ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d3fff640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f289671e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 -- 172.21.15.3:0/1571293148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 msgr2=0x558f289671e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f289671e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 -- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 msgr2=0x558f28966ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f28966ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 -- 172.21.15.3:0/1571293148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d41a44a0 con 0x558f28969bb0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f28967720 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc004ff0 tx=0x7fd5bc00b350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b7b0f5e4a8cd2e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5b3fff640 1 -- 172.21.15.3:0/1571293148 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc014020 con 0x558f28969bb0 2024-02-09T23:01:48.236 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5b3fff640 1 -- 172.21.15.3:0/1571293148 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd5bc09e450 con 0x558f28969bb0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.225+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558f288e9ee0 con 0x558f28969bb0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5b3fff640 1 -- 172.21.15.3:0/1571293148 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5bc0b6020 con 0x558f28969bb0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 10 client.19311 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x558f288c1d90 con 0x558f28969bb0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5b3fff640 1 -- 172.21.15.3:0/1571293148 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558f288e9ee0 con 0x558f28969bb0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5b3fff640 1 -- 172.21.15.3:0/1571293148 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd5bc099890 con 0x558f28969bb0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 10 client.19311 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 10 client.19311 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 20 client.19311 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f28979ac0 0x558f2897beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 4 client.19311 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558f285c3600 con 0x558f28979ac0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.226+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f28979ac0 0x558f2897beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.237 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.227+0000 7fd5b3fff640 10 client.19311 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.227+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f28979ac0 0x558f2897beb0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e630c2187d6e62d6 server_cookie=a7a3fadc5ec7d0bd in_seq=0 out_seq=0 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5b3fff640 1 -- 172.21.15.3:0/1571293148 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558f285c3600 con 0x558f28979ac0 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5b3fff640 10 client.19311 handle_command_reply: tid=0 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 client.19311 shutdown 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 msgr2=0x558f28967720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f28967720 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc004ff0 tx=0x7fd5bc00b350 comp rx=0 tx=0).stop 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 shutdown_connections 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f28979ac0 0x558f2897beb0 crc :-1 s=READY pgs=293 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f28969bb0 0x558f28967720 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895c390 0x558f289671e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5b3fff640 0 client.19311 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1571293148 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895ba70 0x558f28966ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 >> 172.21.15.3:0/1571293148 conn(0x558f289569d0 msgr2=0x558f28958720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:48.238 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 shutdown_connections 2024-02-09T23:01:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1571293148 wait complete. 2024-02-09T23:01:48.239 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.228+0000 7fd5ddbe0740 20 client.19311 trim_cache size 0 max 16384 2024-02-09T23:01:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.241+0000 7fd5ddbe0740 1 Processor -- start 2024-02-09T23:01:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.241+0000 7fd5ddbe0740 1 -- start start 2024-02-09T23:01:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 0x558f289671f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895c390 0x558f28967730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f28969bb0 0x558f28963f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.250 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f28961bb0 con 0x558f2895ba70 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f285c3600 con 0x558f2895c390 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f28950e10 con 0x558f28969bb0 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 0x558f289671f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 0x558f289671f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44832/0 (socket says 172.21.15.3:44832) 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 -- 172.21.15.3:0/1292318525 learned_addr learned my addr 172.21.15.3:0/1292318525 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f28969bb0 0x558f28963f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 -- 172.21.15.3:0/1292318525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f28969bb0 msgr2=0x558f28963f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f28969bb0 0x558f28963f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 -- 172.21.15.3:0/1292318525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895c390 msgr2=0x558f28967730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895c390 0x558f28967730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 -- 172.21.15.3:0/1292318525 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558f2885e820 con 0x558f2895ba70 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 0x558f289671f0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x558f2897c760 tx=0x7fd5cc012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a280319d9274a3e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ca7fc640 1 -- 172.21.15.3:0/1292318525 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc022070 con 0x558f2895ba70 2024-02-09T23:01:48.251 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ca7fc640 1 -- 172.21.15.3:0/1292318525 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc014070 con 0x558f2895ba70 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.242+0000 7fd5ca7fc640 1 -- 172.21.15.3:0/1292318525 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc01c670 con 0x558f2895ba70 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1292318525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 msgr2=0x558f289671f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 0x558f289671f0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x558f2897c760 tx=0x7fd5cc012000 comp rx=0 tx=0).stop 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1292318525 shutdown_connections 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f2895c390 0x558f28967730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f28969bb0 0x558f28963f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/1292318525 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895ba70 0x558f289671f0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1292318525 >> 172.21.15.3:0/1292318525 conn(0x558f289569d0 msgr2=0x558f289807b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1292318525 shutdown_connections 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/1292318525 wait complete. 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 -- start start 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.243+0000 7fd5ddbe0740 1 Processor -- start 2024-02-09T23:01:48.252 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895ba70 0x558f2896f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 0x558f2896f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f28969bb0 0x558f2896fe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f289cffe0 con 0x558f2895c390 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f289d0150 con 0x558f28969bb0 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558f289d0310 con 0x558f2895ba70 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5ddbe0740 10 client.0 _ll_register_callbacks cb 0x7fd5cc007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 0x558f2896f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 0x558f2896f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44834/0 (socket says 172.21.15.3:44834) 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 -- 172.21.15.3:0/4169138851 learned_addr learned my addr 172.21.15.3:0/4169138851 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895ba70 0x558f2896f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d95b4640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f28969bb0 0x558f2896fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 -- 172.21.15.3:0/4169138851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895ba70 msgr2=0x558f2896f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895ba70 0x558f2896f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.253 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 -- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f28969bb0 msgr2=0x558f2896fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f28969bb0 0x558f2896fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.244+0000 7fd5d3fff640 1 -- 172.21.15.3:0/4169138851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc0071c0 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5d3fff640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 0x558f2896f910 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009900 tx=0x7fd5c402ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e583675595e05cc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5b3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5b3fff640 1 -- 172.21.15.3:0/4169138851 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c4004040 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5b3fff640 1 -- 172.21.15.3:0/4169138851 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd5c40041a0 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5bc0a2aa0 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5b3fff640 1 -- 172.21.15.3:0/4169138851 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5c4004360 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 10 client.19317 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x558f288846f0 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5b3fff640 1 -- 172.21.15.3:0/4169138851 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5bc0a2aa0 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5b3fff640 1 -- 172.21.15.3:0/4169138851 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd5c403c040 con 0x558f2895c390 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 10 client.19317 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:48.254 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 10 client.19317 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 20 client.19317 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f289d1ba0 0x558f289d3fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 4 client.19317 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.245+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x558f286141b0 con 0x558f289d1ba0 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.246+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f289d1ba0 0x558f289d3fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.246+0000 7fd5b3fff640 10 client.19317 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:48.255 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.246+0000 7fd5d8db3640 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f289d1ba0 0x558f289d3fc0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95c80bc529dbdcc6 server_cookie=1b59b2d94e735d14 in_seq=0 out_seq=0 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5b3fff640 1 -- 172.21.15.3:0/4169138851 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x558f286141b0 con 0x558f289d1ba0 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5b3fff640 10 client.19317 handle_command_reply: tid=0 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5ddbe0740 1 client.19317 shutdown 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 msgr2=0x558f2896f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 0x558f2896f910 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009900 tx=0x7fd5c402ca10 comp rx=0 tx=0).stop 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 shutdown_connections 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.278+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x558f289d1ba0 0x558f289d3fc0 crc :-1 s=READY pgs=294 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558f28969bb0 0x558f2896fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.283 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5b3fff640 0 client.19317 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558f2895c390 0x558f2896f910 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 1 --2- 172.21.15.3:0/4169138851 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558f2895ba70 0x558f2896f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 >> 172.21.15.3:0/4169138851 conn(0x558f289569d0 msgr2=0x558f28957d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 shutdown_connections 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 1 -- 172.21.15.3:0/4169138851 wait complete. 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.279+0000 7fd5ddbe0740 20 client.19317 trim_cache size 0 max 16384 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:01:48.284 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:01:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd5c00a22a0 msgr2=0x7fd5c00a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd5c00a22a0 0x7fd5c00a4760 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc007ec0 tx=0x7fd5cc0075b0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 msgr2=0x7fd5d41975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:48.285 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 0x7fd5d41975b0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc00d680 tx=0x7fd5bc00db40 comp rx=0 tx=0).stop 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 shutdown_connections 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd5c00a22a0 0x7fd5c00a4760 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd5d4107320 0x7fd5d41975b0 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd5d406a1e0 0x7fd5d419d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 --2- 172.21.15.3:0/3679972390 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd5d4069840 0x7fd5d419d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 >> 172.21.15.3:0/3679972390 conn(0x7fd5d4076f80 msgr2=0x7fd5d40fee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 shutdown_connections 2024-02-09T23:01:48.286 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:01:48.280+0000 7fd5db017640 1 -- 172.21.15.3:0/3679972390 wait complete. 2024-02-09T23:01:48.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:48 smithi003 ceph-mon[32678]: pgmap v329: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-02-09T23:01:48.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 350206 3.16 MB/sec execute 462 sec latency 1504.408 ms 2024-02-09T23:01:48.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:48 smithi064 ceph-mon[36638]: pgmap v329: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-02-09T23:01:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:48 smithi089 ceph-mon[36665]: pgmap v329: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 673 KiB/s rd, 1.6 MiB/s wr, 133 op/s 2024-02-09T23:01:49.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 351773 3.18 MB/sec execute 463 sec latency 18.007 ms 2024-02-09T23:01:50.544 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:50 smithi089 ceph-mon[36665]: pgmap v330: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 888 KiB/s wr, 58 op/s 2024-02-09T23:01:50.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:50 smithi003 ceph-mon[32678]: pgmap v330: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 888 KiB/s wr, 58 op/s 2024-02-09T23:01:50.661 INFO:tasks.workunit.client.0.smithi003.stdout: 1 352850 3.18 MB/sec execute 464 sec latency 252.069 ms 2024-02-09T23:01:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:50 smithi064 ceph-mon[36638]: pgmap v330: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 888 KiB/s wr, 58 op/s 2024-02-09T23:01:51.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 352850 3.18 MB/sec execute 465 sec latency 1252.166 ms 2024-02-09T23:01:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:52 smithi003 ceph-mon[32678]: pgmap v331: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-02-09T23:01:52.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 354968 3.20 MB/sec execute 466 sec latency 1287.894 ms 2024-02-09T23:01:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:52 smithi064 ceph-mon[36638]: pgmap v331: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-02-09T23:01:52.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:52 smithi089 ceph-mon[36665]: pgmap v331: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 872 KiB/s rd, 1.9 MiB/s wr, 155 op/s 2024-02-09T23:01:53.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 356231 3.20 MB/sec execute 467 sec latency 13.390 ms 2024-02-09T23:01:54.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:54 smithi003 ceph-mon[32678]: pgmap v332: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-02-09T23:01:54.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 356392 3.20 MB/sec execute 468 sec latency 863.488 ms 2024-02-09T23:01:54.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:54 smithi064 ceph-mon[36638]: pgmap v332: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-02-09T23:01:54.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:54 smithi089 ceph-mon[36665]: pgmap v332: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-02-09T23:01:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:55 smithi003 ceph-mon[32678]: pgmap v333: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-02-09T23:01:55.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 357469 3.20 MB/sec execute 469 sec latency 1303.650 ms 2024-02-09T23:01:55.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:55 smithi064 ceph-mon[36638]: pgmap v333: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-02-09T23:01:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:55 smithi089 ceph-mon[36665]: pgmap v333: 97 pgs: 97 active+clean; 137 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 1.5 MiB/s wr, 119 op/s 2024-02-09T23:01:55.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:01:55.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 -- 172.21.15.3:0/1347304818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 msgr2=0x7f5500100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1347304818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f5500100720 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009930 tx=0x7f54f002bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 -- 172.21.15.3:0/1347304818 shutdown_connections 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1347304818 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 0x7f550010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1347304818 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5500100cf0 0x7f5500104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1347304818 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f5500100720 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 -- 172.21.15.3:0/1347304818 >> 172.21.15.3:0/1347304818 conn(0x7f55000772e0 msgr2=0x7f55000776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 -- 172.21.15.3:0/1347304818 shutdown_connections 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.026+0000 7f5505cc0640 1 -- 172.21.15.3:0/1347304818 wait complete. 2024-02-09T23:01:56.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 Processor -- start 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 -- start start 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f55001986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 0x7f5500198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55001991c0 0x7f55001a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a3e10 con 0x7f5500100340 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a3f80 con 0x7f5500105410 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.027+0000 7f5505cc0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55001a4140 con 0x7f55001991c0 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f55001986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f55001986f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44856/0 (socket says 172.21.15.3:44856) 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54feffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 0x7f5500198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 -- 172.21.15.3:0/1119560745 learned_addr learned my addr 172.21.15.3:0/1119560745 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54fffff640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55001991c0 0x7f55001a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 -- 172.21.15.3:0/1119560745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55001991c0 msgr2=0x7f55001a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55001991c0 0x7f55001a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 -- 172.21.15.3:0/1119560745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 msgr2=0x7f5500198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 0x7f5500198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 -- 172.21.15.3:0/1119560745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f0009560 con 0x7f5500100340 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54feffd640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 0x7f5500198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54ff7fe640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f55001986f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009230 tx=0x7f54f002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54fcff9640 1 -- 172.21.15.3:0/1119560745 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0004350 con 0x7f5500100340 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54fcff9640 1 -- 172.21.15.3:0/1119560745 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54f00044b0 con 0x7f5500100340 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55001a43c0 con 0x7f5500100340 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f54fcff9640 1 -- 172.21.15.3:0/1119560745 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f0003d00 con 0x7f5500100340 2024-02-09T23:01:56.035 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.028+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55001a48d0 con 0x7f5500100340 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.030+0000 7f54fcff9640 1 -- 172.21.15.3:0/1119560745 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f54f0004df0 con 0x7f5500100340 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 Processor -- start 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 -- start start 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 0x556e4fe0dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fcff9640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f54e80a2230 0x7f54e80a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 0x556e4fe168e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe19300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fdc8d00 con 0x556e4fe0d980 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fd7d2f0 con 0x556e4fe0e330 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f550896a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fd7a820 con 0x556e4fe16eb0 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 0x556e4fe168e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 0x556e4fe0dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54ff7fe640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f54e80a2230 0x7f54e80a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 0x556e4fe0dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44860/0 (socket says 172.21.15.3:44860) 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 0x556e4fe168e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:47880/0 (socket says 172.21.15.3:47880) 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fffff640 1 -- 172.21.15.3:0/2000691913 learned_addr learned my addr 172.21.15.3:0/2000691913 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fffff640 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe19300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fcff9640 1 -- 172.21.15.3:0/1119560745 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f54f00eff70 con 0x7f5500100340 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 -- 172.21.15.3:0/2000691913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 msgr2=0x556e4fe19300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe19300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 -- 172.21.15.3:0/2000691913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 msgr2=0x556e4fe0dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 0x556e4fe0dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54feffd640 1 -- 172.21.15.3:0/2000691913 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e4fd37ff0 con 0x556e4fe0e330 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.031+0000 7f54fffff640 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 0x556e4fe0dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f54ff7fe640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f54e80a2230 0x7f54e80a46f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f54f00c4ce0 tx=0x7f54f00a7040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f54feffd640 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 0x556e4fe168e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f550006a920 tx=0x7f54f40118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b56d007f38c62aa5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f54e5ffb640 1 -- 172.21.15.3:0/2000691913 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4004030 con 0x556e4fe0e330 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f54e5ffb640 1 -- 172.21.15.3:0/2000691913 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54f4005500 con 0x556e4fe0e330 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f54e5ffb640 1 -- 172.21.15.3:0/2000691913 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4004c10 con 0x556e4fe0e330 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f550896a740 1 -- 172.21.15.3:0/2000691913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 msgr2=0x556e4fe168e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.032+0000 7f550896a740 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 0x556e4fe168e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f550006a920 tx=0x7f54f40118d0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 -- 172.21.15.3:0/2000691913 shutdown_connections 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0e330 0x556e4fe168e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe19300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 --2- 172.21.15.3:0/2000691913 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0d980 0x556e4fe0dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 -- 172.21.15.3:0/2000691913 >> 172.21.15.3:0/2000691913 conn(0x556e4fe08050 msgr2=0x556e4fe08430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 -- 172.21.15.3:0/2000691913 shutdown_connections 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 -- 172.21.15.3:0/2000691913 wait complete. 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 -- start start 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 Processor -- start 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe13d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.033+0000 7f550896a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe142d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe14810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fd76000 con 0x556e4fe0e330 2024-02-09T23:01:56.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fa49430 con 0x556e4fe16eb0 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fa05e00 con 0x556e4fe0d980 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 10 client.0 _ll_register_callbacks cb 0x556e4fd6a7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe142d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe142d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44868/0 (socket says 172.21.15.3:44868) 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 -- 172.21.15.3:0/4250534066 learned_addr learned my addr 172.21.15.3:0/4250534066 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54fffff640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe13d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54fffff640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe14810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 -- 172.21.15.3:0/4250534066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 msgr2=0x556e4fe13d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe13d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 -- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 msgr2=0x556e4fe14810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe14810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 -- 172.21.15.3:0/4250534066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f4009c00 con 0x556e4fe0e330 2024-02-09T23:01:56.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54fffff640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe13d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54feffd640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe142d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009730 tx=0x7f54f400fc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4f6817773856073 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54deffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54deffd640 1 -- 172.21.15.3:0/4250534066 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f40043f0 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54deffd640 1 -- 172.21.15.3:0/4250534066 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f54f40046c0 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556e4fa757e0 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f54deffd640 1 -- 172.21.15.3:0/4250534066 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54f4005670 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 10 client.19329 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.034+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x556e4fd74680 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f54deffd640 1 -- 172.21.15.3:0/4250534066 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556e4fa757e0 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f54deffd640 1 -- 172.21.15.3:0/4250534066 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f54f4021040 con 0x556e4fe0e330 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f550896a740 10 client.19329 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f550896a740 10 client.19329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f550896a740 20 client.19329 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f550896a740 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe27200 0x556e4fe29620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f550896a740 4 client.19329 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e4fab2720 con 0x556e4fe27200 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f54fffff640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe27200 0x556e4fe29620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f54deffd640 10 client.19329 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.035+0000 7f54fffff640 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe27200 0x556e4fe29620 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f450e43c042fb53 server_cookie=74b2ef40cc292bbd in_seq=0 out_seq=0 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f54deffd640 1 -- 172.21.15.3:0/4250534066 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556e4fab2720 con 0x556e4fe27200 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f54deffd640 10 client.19329 handle_command_reply: tid=0 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 client.19329 shutdown 2024-02-09T23:01:56.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 msgr2=0x556e4fe142d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe142d0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7f54f4009730 tx=0x7f54f400fc20 comp rx=0 tx=0).stop 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 shutdown_connections 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe27200 0x556e4fe29620 crc :-1 s=READY pgs=295 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe14810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe142d0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f54deffd640 0 client.19329 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 --2- 172.21.15.3:0/4250534066 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe13d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 >> 172.21.15.3:0/4250534066 conn(0x556e4fe08050 msgr2=0x556e4fe09da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 shutdown_connections 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 1 -- 172.21.15.3:0/4250534066 wait complete. 2024-02-09T23:01:56.049 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.036+0000 7f550896a740 20 client.19329 trim_cache size 0 max 16384 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 Processor -- start 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 -- start start 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe164c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe12d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe11010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fe03780 con 0x556e4fe0e330 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fcfafd0 con 0x556e4fe16eb0 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f550896a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fe27000 con 0x556e4fe0d980 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe12d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe12d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44876/0 (socket says 172.21.15.3:44876) 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 -- 172.21.15.3:0/2521314738 learned_addr learned my addr 172.21.15.3:0/2521314738 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54fffff640 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe164c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54fffff640 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe11010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 -- 172.21.15.3:0/2521314738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 msgr2=0x556e4fe164c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe164c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 -- 172.21.15.3:0/2521314738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 msgr2=0x556e4fe11010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe11010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54feffd640 1 -- 172.21.15.3:0/2521314738 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e4fd10b30 con 0x556e4fe0e330 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.052+0000 7f54fffff640 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe11010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.053+0000 7f54feffd640 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe12d70 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f54f4000c00 tx=0x7f54f4001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b56aa63703e4f82d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.053+0000 7f54e57fa640 1 -- 172.21.15.3:0/2521314738 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4005670 con 0x556e4fe0e330 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.053+0000 7f54e57fa640 1 -- 172.21.15.3:0/2521314738 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54f4019a10 con 0x556e4fe0e330 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.053+0000 7f54e57fa640 1 -- 172.21.15.3:0/2521314738 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f4003d30 con 0x556e4fe0e330 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.053+0000 7f550896a740 1 -- 172.21.15.3:0/2521314738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 msgr2=0x556e4fe12d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.059 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.053+0000 7f550896a740 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe12d70 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f54f4000c00 tx=0x7f54f4001e70 comp rx=0 tx=0).stop 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- 172.21.15.3:0/2521314738 shutdown_connections 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe16eb0 0x556e4fe11010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe12d70 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 --2- 172.21.15.3:0/2521314738 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe0d980 0x556e4fe164c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- 172.21.15.3:0/2521314738 >> 172.21.15.3:0/2521314738 conn(0x556e4fe08050 msgr2=0x556e4fe12660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- 172.21.15.3:0/2521314738 shutdown_connections 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- 172.21.15.3:0/2521314738 wait complete. 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- start start 2024-02-09T23:01:56.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 Processor -- start 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0d980 0x556e4fe1e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe1c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe1c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fe1cfa0 con 0x556e4fe0e330 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fe1d110 con 0x556e4fe0d980 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e4fe1d2d0 con 0x556e4fe16eb0 2024-02-09T23:01:56.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f550896a740 10 client.0 _ll_register_callbacks cb 0x7f54f4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f54feffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe1c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f54feffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe1c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44886/0 (socket says 172.21.15.3:44886) 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.054+0000 7f54feffd640 1 -- 172.21.15.3:0/2614862823 learned_addr learned my addr 172.21.15.3:0/2614862823 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54fffff640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe1c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54fffff640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0d980 0x556e4fe1e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54feffd640 1 -- 172.21.15.3:0/2614862823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 msgr2=0x556e4fe1c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54feffd640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe1c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54feffd640 1 -- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0d980 msgr2=0x556e4fe1e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54feffd640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0d980 0x556e4fe1e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54feffd640 1 -- 172.21.15.3:0/2614862823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e4fd74680 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54fffff640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe1c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54feffd640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe1c4a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f54f40059f0 tx=0x7f54f4003ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf98482bcef63270 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54deffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54deffd640 1 -- 172.21.15.3:0/2614862823 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54f403d070 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54deffd640 1 -- 172.21.15.3:0/2614862823 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f54f4019e40 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f54f4005500 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54deffd640 1 -- 172.21.15.3:0/2614862823 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f54f4005670 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f550896a740 10 client.19335 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x556e4fd36850 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.055+0000 7f54deffd640 1 -- 172.21.15.3:0/2614862823 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f54f404e080 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f54deffd640 1 -- 172.21.15.3:0/2614862823 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f54f4048030 con 0x556e4fe0e330 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 10 client.19335 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 10 client.19335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:56.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 20 client.19335 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe82ce0 0x556e4fe85100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 4 client.19335 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556e4fb01890 con 0x556e4fe82ce0 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f54fffff640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe82ce0 0x556e4fe85100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f54deffd640 10 client.19335 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f54fffff640 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe82ce0 0x556e4fe85100 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17434e3a2a7ce5ed server_cookie=d5dc530c9032d2f8 in_seq=0 out_seq=0 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f54deffd640 1 -- 172.21.15.3:0/2614862823 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556e4fb01890 con 0x556e4fe82ce0 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f54deffd640 10 client.19335 handle_command_reply: tid=0 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.056+0000 7f550896a740 1 client.19335 shutdown 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 msgr2=0x556e4fe1c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe1c4a0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f54f40059f0 tx=0x7f54f4003ae0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 shutdown_connections 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e4fe82ce0 0x556e4fe85100 crc :-1 s=READY pgs=296 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e4fe16eb0 0x556e4fe1c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f54deffd640 0 client.19335 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e4fe0e330 0x556e4fe1c4a0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 --2- 172.21.15.3:0/2614862823 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e4fe0d980 0x556e4fe1e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 >> 172.21.15.3:0/2614862823 conn(0x556e4fe08050 msgr2=0x556e4fe071f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 shutdown_connections 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 1 -- 172.21.15.3:0/2614862823 wait complete. 2024-02-09T23:01:56.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f550896a740 20 client.19335 trim_cache size 0 max 16384 2024-02-09T23:01:56.069 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:01:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:01:56.069 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:01:56.069 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f54e80a2230 msgr2=0x7f54e80a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f54e80a2230 0x7f54e80a46f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f54f00c4ce0 tx=0x7f54f00a7040 comp rx=0 tx=0).stop 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 msgr2=0x7f55001986f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f55001986f0 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7f54f0009230 tx=0x7f54f002c650 comp rx=0 tx=0).stop 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 shutdown_connections 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f54e80a2230 0x7f54e80a46f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5500105410 0x7f5500198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f55001991c0 0x7f55001a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 --2- 172.21.15.3:0/1119560745 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5500100340 0x7f55001986f0 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 >> 172.21.15.3:0/1119560745 conn(0x7f55000772e0 msgr2=0x7f55000fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 shutdown_connections 2024-02-09T23:01:56.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.057+0000 7f5505cc0640 1 -- 172.21.15.3:0/1119560745 wait complete. 2024-02-09T23:01:56.072 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:01:56.072 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 -- 172.21.15.3:0/2948674779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00105420 msgr2=0x7efe0010cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 --2- 172.21.15.3:0/2948674779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00105420 0x7efe0010cff0 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009cd0 tx=0x7efdf002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 -- 172.21.15.3:0/2948674779 shutdown_connections 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 --2- 172.21.15.3:0/2948674779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00105420 0x7efe0010cff0 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 --2- 172.21.15.3:0/2948674779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00100d00 0x7efe00104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 --2- 172.21.15.3:0/2948674779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe00100350 0x7efe00100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.175+0000 7efe05635640 1 -- 172.21.15.3:0/2948674779 >> 172.21.15.3:0/2948674779 conn(0x7efe000772b0 msgr2=0x7efe000776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.176+0000 7efe05635640 1 -- 172.21.15.3:0/2948674779 shutdown_connections 2024-02-09T23:01:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.176+0000 7efe05635640 1 -- 172.21.15.3:0/2948674779 wait complete. 2024-02-09T23:01:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.176+0000 7efe05635640 1 Processor -- start 2024-02-09T23:01:56.181 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.176+0000 7efe05635640 1 -- start start 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 0x7efe0019d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00105420 0x7efe0019dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe00197630 0x7efe00197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe00198100 con 0x7efe00100350 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe00198270 con 0x7efe00197630 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe00198430 con 0x7efe00105420 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 0x7efe0019d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 0x7efe0019d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:44906/0 (socket says 172.21.15.3:44906) 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 -- 172.21.15.3:0/3032766835 learned_addr learned my addr 172.21.15.3:0/3032766835 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfe7fc640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00105420 0x7efe0019dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdff7fe640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe00197630 0x7efe00197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 -- 172.21.15.3:0/3032766835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00105420 msgr2=0x7efe0019dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00105420 0x7efe0019dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 -- 172.21.15.3:0/3032766835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe00197630 msgr2=0x7efe00197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe00197630 0x7efe00197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 -- 172.21.15.3:0/3032766835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf0009900 con 0x7efe00100350 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfe7fc640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00105420 0x7efe0019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdfeffd640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 0x7efe0019d570 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7efdf800e9f0 tx=0x7efdf800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8009700 con 0x7efe00100350 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe001a43d0 con 0x7efe00100350 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf800cd80 con 0x7efe00100350 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8018920 con 0x7efe00100350 2024-02-09T23:01:56.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.177+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe001a4900 con 0x7efe00100350 2024-02-09T23:01:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.178+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe00101820 con 0x7efe00100350 2024-02-09T23:01:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.180+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7efdf8018a80 con 0x7efe00100350 2024-02-09T23:01:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.180+0000 7efdf7fff640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdec0a2230 0x7efdec0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.180+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7efdf8014070 con 0x7efe00100350 2024-02-09T23:01:56.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.180+0000 7efdfe7fc640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdec0a2230 0x7efdec0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.181+0000 7efdfe7fc640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdec0a2230 0x7efdec0a46f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efe00198e80 tx=0x7efde8005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.184+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7efdf80975e0 con 0x7efe00100350 2024-02-09T23:01:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.419+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe000680b0 con 0x7efe00100350 2024-02-09T23:01:56.423 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.420+0000 7efdf7fff640 1 -- 172.21.15.3:0/3032766835 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+11364 (secure 0 0 0) 0x7efdf809aca0 con 0x7efe00100350 2024-02-09T23:01:56.424 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:01:56.424 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:01:40.086647+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:01:56.424 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 49 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdec0a2230 msgr2=0x7efdec0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdec0a2230 0x7efdec0a46f0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7efe00198e80 tx=0x7efde8005f90 comp rx=0 tx=0).stop 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 msgr2=0x7efe0019d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 0x7efe0019d570 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7efdf800e9f0 tx=0x7efdf800eeb0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 shutdown_connections 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7efdec0a2230 0x7efdec0a46f0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7efe00197630 0x7efe00197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7efe00105420 0x7efe0019dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.426 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 --2- 172.21.15.3:0/3032766835 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7efe00100350 0x7efe0019d570 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 >> 172.21.15.3:0/3032766835 conn(0x7efe000772b0 msgr2=0x7efe00103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 shutdown_connections 2024-02-09T23:01:56.427 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.422+0000 7efe05635640 1 -- 172.21.15.3:0/3032766835 wait complete. 2024-02-09T23:01:56.437 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:56.437 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:01:56.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 -- 172.21.15.3:0/2000057534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 msgr2=0x7f4018102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.552 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 --2- 172.21.15.3:0/2000057534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f4018102830 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4008009900 tx=0x7f400802bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 -- 172.21.15.3:0/2000057534 shutdown_connections 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 --2- 172.21.15.3:0/2000057534 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4018102d70 0x7f401810b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 --2- 172.21.15.3:0/2000057534 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 0x7f401810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 --2- 172.21.15.3:0/2000057534 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f4018102830 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 -- 172.21.15.3:0/2000057534 >> 172.21.15.3:0/2000057534 conn(0x7f40180772e0 msgr2=0x7f40180776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 -- 172.21.15.3:0/2000057534 shutdown_connections 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.546+0000 7f4020148640 1 -- 172.21.15.3:0/2000057534 wait complete. 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 Processor -- start 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 -- start start 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f401819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4018102d70 0x7f401819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.553 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 0x7f40181976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4018197e70 con 0x7f401810b820 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4018197fe0 con 0x7f4018102d70 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f4020148640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40181981a0 con 0x7f4018102450 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.547+0000 7f401dee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f401819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401e6e5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 0x7f40181976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f401819d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53880/0 (socket says 172.21.15.3:53880) 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 -- 172.21.15.3:0/1237875639 learned_addr learned my addr 172.21.15.3:0/1237875639 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401d6e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4018102d70 0x7f401819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 -- 172.21.15.3:0/1237875639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4018102d70 msgr2=0x7f401819db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4018102d70 0x7f401819db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 -- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 msgr2=0x7f40181976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 0x7f40181976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 -- 172.21.15.3:0/1237875639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4008009560 con 0x7f4018102450 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401e6e5640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 0x7f40181976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.554 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f401dee4640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f401819d620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4008005940 tx=0x7f400802c710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f400effd640 1 -- 172.21.15.3:0/1237875639 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008004350 con 0x7f4018102450 2024-02-09T23:01:56.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f400effd640 1 -- 172.21.15.3:0/1237875639 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40080044b0 con 0x7f4018102450 2024-02-09T23:01:56.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4018198420 con 0x7f4018102450 2024-02-09T23:01:56.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f400effd640 1 -- 172.21.15.3:0/1237875639 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008003d30 con 0x7f4018102450 2024-02-09T23:01:56.555 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.548+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40181a45a0 con 0x7f4018102450 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f400effd640 1 -- 172.21.15.3:0/1237875639 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f4008004df0 con 0x7f4018102450 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 Processor -- start 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 -- start start 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 0x55d42591ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591b3e0 0x55d425923990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d425923f60 0x55d4259263b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4258d6020 con 0x55d42591b3e0 2024-02-09T23:01:56.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425889320 con 0x55d42591aa30 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.551+0000 7f4022df6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425886850 con 0x55d425923f60 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 0x55d42591ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401dee4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591b3e0 0x55d425923990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 0x55d42591ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37386/0 (socket says 172.21.15.3:37386) 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 -- 172.21.15.3:0/2772260810 learned_addr learned my addr 172.21.15.3:0/2772260810 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401d6e3640 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d425923f60 0x55d4259263b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f400effd640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f40040a2230 0x7f40040a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 -- 172.21.15.3:0/2772260810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d425923f60 msgr2=0x55d4259263b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d425923f60 0x55d4259263b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 -- 172.21.15.3:0/2772260810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591b3e0 msgr2=0x55d425923990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591b3e0 0x55d425923990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 -- 172.21.15.3:0/2772260810 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d425844210 con 0x55d42591aa30 2024-02-09T23:01:56.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401d6e3640 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d425923f60 0x55d4259263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f40040a2230 0x7f40040a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f400effd640 1 -- 172.21.15.3:0/1237875639 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f40080f0630 con 0x7f4018102450 2024-02-09T23:01:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.552+0000 7f401e6e5640 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 0x55d42591ae10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f40100118b0 tx=0x7f4010011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88d4ea1c015a57fe server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.553+0000 7f4003fff640 1 -- 172.21.15.3:0/2772260810 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401000fea0 con 0x55d42591aa30 2024-02-09T23:01:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.553+0000 7f4003fff640 1 -- 172.21.15.3:0/2772260810 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f401000fc20 con 0x55d42591aa30 2024-02-09T23:01:56.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.553+0000 7f4003fff640 1 -- 172.21.15.3:0/2772260810 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010024620 con 0x55d42591aa30 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f401e6e5640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f40040a2230 0x7f40040a46f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4010021000 tx=0x7f4010022000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 -- 172.21.15.3:0/2772260810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 msgr2=0x55d42591ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 0x55d42591ae10 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f40100118b0 tx=0x7f4010011d70 comp rx=0 tx=0).stop 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 -- 172.21.15.3:0/2772260810 shutdown_connections 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d425923f60 0x55d4259263b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591b3e0 0x55d425923990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 --2- 172.21.15.3:0/2772260810 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591aa30 0x55d42591ae10 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 -- 172.21.15.3:0/2772260810 >> 172.21.15.3:0/2772260810 conn(0x55d425915100 msgr2=0x55d4259154e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 -- 172.21.15.3:0/2772260810 shutdown_connections 2024-02-09T23:01:56.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 -- 172.21.15.3:0/2772260810 wait complete. 2024-02-09T23:01:56.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 -- start start 2024-02-09T23:01:56.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.554+0000 7f4022df6740 1 Processor -- start 2024-02-09T23:01:56.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591aa30 0x55d425920e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 0x55d4259213d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d425923f60 0x55d425921910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425882030 con 0x55d42591aa30 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425555430 con 0x55d425923f60 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425511e00 con 0x55d42591b3e0 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f4022df6740 10 client.0 _ll_register_callbacks cb 0x55d425877030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f401d6e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 0x55d4259213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f401d6e3640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 0x55d4259213d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53892/0 (socket says 172.21.15.3:53892) 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f401d6e3640 1 -- 172.21.15.3:0/4279206004 learned_addr learned my addr 172.21.15.3:0/4279206004 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f401e6e5640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d425923f60 0x55d425921910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.555+0000 7f401dee4640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591aa30 0x55d425920e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401d6e3640 1 -- 172.21.15.3:0/4279206004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d425923f60 msgr2=0x55d425921910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401d6e3640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d425923f60 0x55d425921910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401d6e3640 1 -- 172.21.15.3:0/4279206004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591aa30 msgr2=0x55d425920e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401d6e3640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591aa30 0x55d425920e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401d6e3640 1 -- 172.21.15.3:0/4279206004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4010011490 con 0x55d42591b3e0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401e6e5640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d425923f60 0x55d425921910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401dee4640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591aa30 0x55d425920e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f401d6e3640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 0x55d4259213d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4014018790 tx=0x7f4014018b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d12ea17ed42212c5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.556+0000 7f40017fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f40017fa640 1 -- 172.21.15.3:0/4279206004 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4014004220 con 0x55d42591b3e0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d4255817e0 con 0x55d42591b3e0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f40017fa640 1 -- 172.21.15.3:0/4279206004 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f4014004380 con 0x55d42591b3e0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f40017fa640 1 -- 172.21.15.3:0/4279206004 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40140129d0 con 0x55d42591b3e0 2024-02-09T23:01:56.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 10 client.28348 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55d4258806b0 con 0x55d42591b3e0 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f40017fa640 1 -- 172.21.15.3:0/4279206004 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55d4255817e0 con 0x55d42591b3e0 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f40017fa640 1 -- 172.21.15.3:0/4279206004 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f4014003cf0 con 0x55d42591b3e0 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 10 client.28348 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 10 client.28348 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 20 client.28348 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d425934680 0x55d425936aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 4 client.28348 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.557+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d4255be720 con 0x55d425934680 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.558+0000 7f401dee4640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d425934680 0x55d425936aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.559+0000 7f401dee4640 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d425934680 0x55d425936aa0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c0765399aeff0cc7 server_cookie=f5239211900afb25 in_seq=0 out_seq=0 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.559+0000 7f40017fa640 10 client.28348 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:56.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f40017fa640 1 -- 172.21.15.3:0/4279206004 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d4255be720 con 0x55d425934680 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f40017fa640 10 client.28348 handle_command_reply: tid=0 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 client.28348 shutdown 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 msgr2=0x55d4259213d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 0x55d4259213d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4014018790 tx=0x7f4014018b90 comp rx=0 tx=0).stop 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 shutdown_connections 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d425934680 0x55d425936aa0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d425923f60 0x55d425921910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591b3e0 0x55d4259213d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f40017fa640 0 client.28348 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 --2- 172.21.15.3:0/4279206004 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d42591aa30 0x55d425920e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 >> 172.21.15.3:0/4279206004 conn(0x55d425915100 msgr2=0x55d425916e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 shutdown_connections 2024-02-09T23:01:56.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 1 -- 172.21.15.3:0/4279206004 wait complete. 2024-02-09T23:01:56.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.560+0000 7f4022df6740 20 client.28348 trim_cache size 0 max 16384 2024-02-09T23:01:56.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 Processor -- start 2024-02-09T23:01:56.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 -- start start 2024-02-09T23:01:56.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d4259235c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425921730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d42591e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425807130 con 0x55d425923f60 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425919dd0 con 0x55d42591b3e0 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.572+0000 7f4022df6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425910ab0 con 0x55d42591aa30 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d4259235c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401e6e5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d42591e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d4259235c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53904/0 (socket says 172.21.15.3:53904) 2024-02-09T23:01:56.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 -- 172.21.15.3:0/2313139232 learned_addr learned my addr 172.21.15.3:0/2313139232 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401d6e3640 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425921730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 -- 172.21.15.3:0/2313139232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 msgr2=0x55d425921730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425921730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 -- 172.21.15.3:0/2313139232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 msgr2=0x55d42591e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d42591e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 -- 172.21.15.3:0/2313139232 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d42581cc90 con 0x55d42591aa30 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f401dee4640 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d4259235c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f40080edef0 tx=0x7f40080b9500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55817d9e7ab7c99f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f40037fe640 1 -- 172.21.15.3:0/2313139232 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080bd830 con 0x55d42591aa30 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f40037fe640 1 -- 172.21.15.3:0/2313139232 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40080bd130 con 0x55d42591aa30 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.573+0000 7f40037fe640 1 -- 172.21.15.3:0/2313139232 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080bb550 con 0x55d42591aa30 2024-02-09T23:01:56.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 -- 172.21.15.3:0/2313139232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 msgr2=0x55d4259235c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d4259235c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f40080edef0 tx=0x7f40080b9500 comp rx=0 tx=0).stop 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 -- 172.21.15.3:0/2313139232 shutdown_connections 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425921730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d42591e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 --2- 172.21.15.3:0/2313139232 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d4259235c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 -- 172.21.15.3:0/2313139232 >> 172.21.15.3:0/2313139232 conn(0x55d425915100 msgr2=0x55d42591c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 -- 172.21.15.3:0/2313139232 shutdown_connections 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.574+0000 7f4022df6740 1 -- 172.21.15.3:0/2313139232 wait complete. 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 -- start start 2024-02-09T23:01:56.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 Processor -- start 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d42592b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425929410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d425929950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d425929e90 con 0x55d425923f60 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d42592a000 con 0x55d42591b3e0 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d42592a1c0 con 0x55d42591aa30 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f4022df6740 10 client.0 _ll_register_callbacks cb 0x7f4010009ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f401d6e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425929410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f401dee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d42592b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f401d6e3640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425929410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37414/0 (socket says 172.21.15.3:37414) 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f401dee4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d42592b020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53906/0 (socket says 172.21.15.3:53906) 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f401d6e3640 1 -- 172.21.15.3:0/308592942 learned_addr learned my addr 172.21.15.3:0/308592942 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.575+0000 7f401e6e5640 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d425929950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f401dee4640 1 -- 172.21.15.3:0/308592942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 msgr2=0x55d425929410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f401dee4640 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425929410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f401dee4640 1 -- 172.21.15.3:0/308592942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 msgr2=0x55d425929950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f401dee4640 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d425929950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f401dee4640 1 -- 172.21.15.3:0/308592942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40181a45a0 con 0x55d42591aa30 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f401dee4640 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d42592b020 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4008009320 tx=0x7f40080f09a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a0ba1455afb53e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f40017fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:56.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f40017fa640 1 -- 172.21.15.3:0/308592942 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40080c1280 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f40017fa640 1 -- 172.21.15.3:0/308592942 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f40080c58d0 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f401000fc20 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f40017fa640 1 -- 172.21.15.3:0/308592942 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40080bd830 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f4022df6740 10 client.28354 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.576+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55d425842a70 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f40017fa640 1 -- 172.21.15.3:0/308592942 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f401000fc20 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f40017fa640 1 -- 172.21.15.3:0/308592942 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f400803c070 con 0x55d42591aa30 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 10 client.28354 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 10 client.28354 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 20 client.28354 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d42598bbc0 0x55d42598dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 4 client.28354 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d42560d890 con 0x55d42598bbc0 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f401d6e3640 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d42598bbc0 0x55d42598dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f401d6e3640 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d42598bbc0 0x55d42598dfe0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6fa2541fd5fa6c01 server_cookie=38f9baf242dadf57 in_seq=0 out_seq=0 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f40017fa640 10 client.28354 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f40017fa640 1 -- 172.21.15.3:0/308592942 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d42560d890 con 0x55d42598bbc0 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f40017fa640 10 client.28354 handle_command_reply: tid=0 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.577+0000 7f4022df6740 1 client.28354 shutdown 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 msgr2=0x55d42592b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d42592b020 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4008009320 tx=0x7f40080f09a0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 shutdown_connections 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x55d42598bbc0 0x55d42598dfe0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55d42591b3e0 0x55d425929410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55d425923f60 0x55d425929950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 --2- 172.21.15.3:0/308592942 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55d42591aa30 0x55d42592b020 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f40017fa640 0 client.28354 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 >> 172.21.15.3:0/308592942 conn(0x55d425915100 msgr2=0x55d4259139b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 shutdown_connections 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 1 -- 172.21.15.3:0/308592942 wait complete. 2024-02-09T23:01:56.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4022df6740 20 client.28354 trim_cache size 0 max 16384 2024-02-09T23:01:56.590 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:56.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f40040a2230 msgr2=0x7f40040a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f40040a2230 0x7f40040a46f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f4010021000 tx=0x7f4010022000 comp rx=0 tx=0).stop 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 msgr2=0x7f401819d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f401819d620 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f4008005940 tx=0x7f400802c710 comp rx=0 tx=0).stop 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 shutdown_connections 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f40040a2230 0x7f40040a46f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f4018102d70 0x7f401819db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f401810b820 0x7f40181976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 --2- 172.21.15.3:0/1237875639 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f4018102450 0x7f401819d620 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 >> 172.21.15.3:0/1237875639 conn(0x7f40180772e0 msgr2=0x7f40180fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 shutdown_connections 2024-02-09T23:01:56.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.578+0000 7f4020148640 1 -- 172.21.15.3:0/1237875639 wait complete. 2024-02-09T23:01:56.596 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:56.597 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:01:56.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 358858 3.21 MB/sec execute 470 sec latency 23.396 ms 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 -- 172.21.15.3:0/84209447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 msgr2=0x7f8f6410b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 --2- 172.21.15.3:0/84209447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6410b250 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c009900 tx=0x7f8f4c02bcd0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 -- 172.21.15.3:0/84209447 shutdown_connections 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 --2- 172.21.15.3:0/84209447 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 0x7f8f6410dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 --2- 172.21.15.3:0/84209447 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6410b250 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 --2- 172.21.15.3:0/84209447 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 0x7f8f64102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 -- 172.21.15.3:0/84209447 >> 172.21.15.3:0/84209447 conn(0x7f8f640772e0 msgr2=0x7f8f640776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 -- 172.21.15.3:0/84209447 shutdown_connections 2024-02-09T23:01:56.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 -- 172.21.15.3:0/84209447 wait complete. 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.711+0000 7f8f6997a640 1 Processor -- start 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 -- start start 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 0x7f8f6419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 0x7f8f641976f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64197e80 con 0x7f8f64102d70 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64197ff0 con 0x7f8f6410b820 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f6997a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f641981b0 con 0x7f8f64102450 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f62ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 0x7f8f6419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 0x7f8f641976f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6419db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36374/0 (socket says 172.21.15.3:36374) 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 -- 172.21.15.3:0/3862611497 learned_addr learned my addr 172.21.15.3:0/3862611497 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f62ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 0x7f8f6419d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53930/0 (socket says 172.21.15.3:53930) 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 -- 172.21.15.3:0/3862611497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 msgr2=0x7f8f6419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 0x7f8f6419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 -- 172.21.15.3:0/3862611497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 msgr2=0x7f8f641976f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 0x7f8f641976f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f627fc640 1 -- 172.21.15.3:0/3862611497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f4c009560 con 0x7f8f64102d70 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.712+0000 7f8f637fe640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 0x7f8f641976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.713+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6419db20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c000c00 tx=0x7f8f4c02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.713+0000 7f8f68978640 1 -- 172.21.15.3:0/3862611497 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c02dc60 con 0x7f8f64102d70 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.713+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64198430 con 0x7f8f64102d70 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.713+0000 7f8f68978640 1 -- 172.21.15.3:0/3862611497 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f4c02ddc0 con 0x7f8f64102d70 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.713+0000 7f8f68978640 1 -- 172.21.15.3:0/3862611497 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c003bf0 con 0x7f8f64102d70 2024-02-09T23:01:56.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.713+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f641a4560 con 0x7f8f64102d70 2024-02-09T23:01:56.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f68978640 1 -- 172.21.15.3:0/3862611497 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f8f4c003d50 con 0x7f8f64102d70 2024-02-09T23:01:56.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 Processor -- start 2024-02-09T23:01:56.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 -- start start 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b09ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b056020 con 0x559d4b09b3e0 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b009320 con 0x559d4b09aa30 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b006850 con 0x559d4b0a3f60 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f68978640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8f500a2230 0x7f8f500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.715+0000 7f8f62ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36376/0 (socket says 172.21.15.3:36376) 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 -- 172.21.15.3:0/960551638 learned_addr learned my addr 172.21.15.3:0/960551638 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f637fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b09ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f627fc640 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f637fe640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8f500a2230 0x7f8f500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f68978640 1 -- 172.21.15.3:0/3862611497 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8f4c0f0560 con 0x7f8f64102d70 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 -- 172.21.15.3:0/960551638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 msgr2=0x559d4b0a63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 -- 172.21.15.3:0/960551638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 msgr2=0x559d4b09ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b09ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 -- 172.21.15.3:0/960551638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d4afc4210 con 0x559d4b09b3e0 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f637fe640 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b09ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f637fe640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8f500a2230 0x7f8f500a46f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c005fd0 tx=0x7f8f5c00b260 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a3990 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8f540114f0 tx=0x7f8f540119b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e8f35ceddef3e4e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f59ffb640 1 -- 172.21.15.3:0/960551638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f54004d20 con 0x559d4b09b3e0 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f59ffb640 1 -- 172.21.15.3:0/960551638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f54004e80 con 0x559d4b09b3e0 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.716+0000 7f8f59ffb640 1 -- 172.21.15.3:0/960551638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5400f760 con 0x559d4b09b3e0 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 -- 172.21.15.3:0/960551638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 msgr2=0x559d4b0a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a3990 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7f8f540114f0 tx=0x7f8f540119b0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 -- 172.21.15.3:0/960551638 shutdown_connections 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a3990 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/960551638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b09ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 -- 172.21.15.3:0/960551638 >> 172.21.15.3:0/960551638 conn(0x559d4b095100 msgr2=0x559d4b0954e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 -- 172.21.15.3:0/960551638 shutdown_connections 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 -- 172.21.15.3:0/960551638 wait complete. 2024-02-09T23:01:56.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 -- start start 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.718+0000 7f8f6c62b740 1 Processor -- start 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b002030 con 0x559d4b09b3e0 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4acd5430 con 0x559d4b09aa30 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4ac91e00 con 0x559d4b0a3f60 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f6c62b740 10 client.0 _ll_register_callbacks cb 0x559d4aff7030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36378/0 (socket says 172.21.15.3:36378) 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 -- 172.21.15.3:0/3493353894 learned_addr learned my addr 172.21.15.3:0/3493353894 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f637fe640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 -- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 msgr2=0x559d4b0a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 -- 172.21.15.3:0/3493353894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 msgr2=0x559d4b0a0ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0a0ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 -- 172.21.15.3:0/3493353894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54009c80 con 0x559d4b09b3e0 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f637fe640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1400 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005ec0 tx=0x7f8f4c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e9fc861fdbc702b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:56.731 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.719+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3493353894 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0bd060 con 0x559d4b09b3e0 2024-02-09T23:01:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3493353894 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8f4c0c11b0 con 0x559d4b09b3e0 2024-02-09T23:01:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d4ad3e720 con 0x559d4b09b3e0 2024-02-09T23:01:56.732 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3493353894 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f4c0c5800 con 0x559d4b09b3e0 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f6c62b740 10 client.19371 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x559d4b0006b0 con 0x559d4b09b3e0 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3493353894 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559d4ad3e720 con 0x559d4b09b3e0 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3493353894 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8f4c03c070 con 0x559d4b09b3e0 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f6c62b740 10 client.19371 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.720+0000 7f8f6c62b740 10 client.19371 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:01:56.733 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f6c62b740 20 client.19371 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b0b40a0 0x559d4b0b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f6c62b740 4 client.19371 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d4ad8d890 con 0x559d4b0b40a0 2024-02-09T23:01:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b0b40a0 0x559d4b0b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b0b40a0 0x559d4b0b64d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a6818ae06b85be61 server_cookie=ad8f737c13929510 in_seq=0 out_seq=0 2024-02-09T23:01:56.734 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.721+0000 7f8f42ffd640 10 client.19371 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3493353894 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559d4ad8d890 con 0x559d4b0b40a0 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f42ffd640 10 client.19371 handle_command_reply: tid=0 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 client.19371 shutdown 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 msgr2=0x559d4b0a1400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1400 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005ec0 tx=0x7f8f4c0036f0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 shutdown_connections 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b0b40a0 0x559d4b0b64d0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.735 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1400 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3493353894 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0a0ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f42ffd640 0 client.19371 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 >> 172.21.15.3:0/3493353894 conn(0x559d4b095100 msgr2=0x559d4b096e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 shutdown_connections 2024-02-09T23:01:56.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3493353894 wait complete. 2024-02-09T23:01:56.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.722+0000 7f8f6c62b740 20 client.19371 trim_cache size 0 max 16384 2024-02-09T23:01:56.737 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:56 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3032766835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:01:56.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 Processor -- start 2024-02-09T23:01:56.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 -- start start 2024-02-09T23:01:56.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b09aa30 0x559d4b0a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.747 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b0a3f60 0x559d4b09e120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b0908b0 con 0x559d4b09b3e0 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4af87130 con 0x559d4b0a3f60 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4ad3e720 con 0x559d4b09aa30 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f627fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36388/0 (socket says 172.21.15.3:36388) 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f627fc640 1 -- 172.21.15.3:0/1180858492 learned_addr learned my addr 172.21.15.3:0/1180858492 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b09aa30 0x559d4b0a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.736+0000 7f8f637fe640 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b0a3f60 0x559d4b09e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f627fc640 1 -- 172.21.15.3:0/1180858492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b09aa30 msgr2=0x559d4b0a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f627fc640 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b09aa30 0x559d4b0a35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f627fc640 1 -- 172.21.15.3:0/1180858492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b0a3f60 msgr2=0x559d4b09e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f627fc640 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b0a3f60 0x559d4b09e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f627fc640 1 -- 172.21.15.3:0/1180858492 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d4af9cc90 con 0x559d4b09b3e0 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b09aa30 0x559d4b0a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f627fc640 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1760 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005ab0 tx=0x7f8f4c02c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=291695656efacd49 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f597fa640 1 -- 172.21.15.3:0/1180858492 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0c5610 con 0x559d4b09b3e0 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f597fa640 1 -- 172.21.15.3:0/1180858492 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f4c0f0380 con 0x559d4b09b3e0 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.737+0000 7f8f597fa640 1 -- 172.21.15.3:0/1180858492 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f4c0c5800 con 0x559d4b09b3e0 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 -- 172.21.15.3:0/1180858492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 msgr2=0x559d4b0a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1760 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c005ab0 tx=0x7f8f4c02c4f0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 -- 172.21.15.3:0/1180858492 shutdown_connections 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b0a3f60 0x559d4b09e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a1760 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/1180858492 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b09aa30 0x559d4b0a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 -- 172.21.15.3:0/1180858492 >> 172.21.15.3:0/1180858492 conn(0x559d4b095100 msgr2=0x559d4b09c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 -- 172.21.15.3:0/1180858492 shutdown_connections 2024-02-09T23:01:56.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 -- 172.21.15.3:0/1180858492 wait complete. 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.738+0000 7f8f6c62b740 1 -- start start 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 Processor -- start 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0ab1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b0aa030 con 0x559d4b09b3e0 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b0aa1a0 con 0x559d4b09aa30 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d4b0aa360 con 0x559d4b0a3f60 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f6c62b740 10 client.0 _ll_register_callbacks cb 0x7f8f54009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0ab1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f62ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0ab1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37448/0 (socket says 172.21.15.3:37448) 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f62ffd640 1 -- 172.21.15.3:0/3347488380 learned_addr learned my addr 172.21.15.3:0/3347488380 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f637fe640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.739+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f62ffd640 1 -- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 msgr2=0x559d4b0a9af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a9af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f62ffd640 1 -- 172.21.15.3:0/3347488380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 msgr2=0x559d4b0a95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f62ffd640 1 -- 172.21.15.3:0/3347488380 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d4afc4210 con 0x559d4b09aa30 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f62ffd640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0ab1c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009f70 tx=0x7f8f54004620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3601f9409f5a88f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f42ffd640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3347488380 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f5402c040 con 0x559d4b09aa30 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3347488380 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f8f54011250 con 0x559d4b09aa30 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d4ad017e0 con 0x559d4b09aa30 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3347488380 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8f540096c0 con 0x559d4b09aa30 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.740+0000 7f8f6c62b740 10 client.28220 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x559d4b0006b0 con 0x559d4b09aa30 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3347488380 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559d4ad017e0 con 0x559d4b09aa30 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3347488380 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8f54021040 con 0x559d4b09aa30 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 10 client.28220 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 10 client.28220 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 20 client.28220 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b10bae0 0x559d4b10df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 4 client.28220 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559d4acad700 con 0x559d4b10bae0 2024-02-09T23:01:56.752 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.741+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b10bae0 0x559d4b10df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f627fc640 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b10bae0 0x559d4b10df00 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d44425053aa7ee50 server_cookie=13d78b9688bcb302 in_seq=0 out_seq=0 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f42ffd640 10 client.28220 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f42ffd640 1 -- 172.21.15.3:0/3347488380 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559d4acad700 con 0x559d4b10bae0 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f42ffd640 10 client.28220 handle_command_reply: tid=0 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f6c62b740 1 client.28220 shutdown 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 msgr2=0x559d4b0ab1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0ab1c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8f54009f70 tx=0x7f8f54004620 comp rx=0 tx=0).stop 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 shutdown_connections 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.742+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559d4b10bae0 0x559d4b10df00 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559d4b0a3f60 0x559d4b0a9af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559d4b09b3e0 0x559d4b0a95b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.753 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 1 --2- 172.21.15.3:0/3347488380 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559d4b09aa30 0x559d4b0ab1c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 >> 172.21.15.3:0/3347488380 conn(0x559d4b095100 msgr2=0x559d4b0939b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f42ffd640 0 client.28220 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:01:56.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 shutdown_connections 2024-02-09T23:01:56.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 1 -- 172.21.15.3:0/3347488380 wait complete. 2024-02-09T23:01:56.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6c62b740 20 client.28220 trim_cache size 0 max 16384 2024-02-09T23:01:56.754 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:56.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8f500a2230 msgr2=0x7f8f500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6997a640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8f500a2230 0x7f8f500a46f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8f5c005fd0 tx=0x7f8f5c00b260 comp rx=0 tx=0).stop 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 msgr2=0x7f8f6419db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.743+0000 7f8f6997a640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6419db20 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f8f4c000c00 tx=0x7f8f4c02c740 comp rx=0 tx=0).stop 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 shutdown_connections 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f8f500a2230 0x7f8f500a46f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f8f6410b820 0x7f8f641976f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f8f64102d70 0x7f8f6419db20 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 --2- 172.21.15.3:0/3862611497 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f8f64102450 0x7f8f6419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 >> 172.21.15.3:0/3862611497 conn(0x7f8f640772e0 msgr2=0x7f8f640fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 shutdown_connections 2024-02-09T23:01:56.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.744+0000 7f8f6997a640 1 -- 172.21.15.3:0/3862611497 wait complete. 2024-02-09T23:01:56.758 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:56 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3032766835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:01:56.759 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:56.759 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:01:56.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 -- 172.21.15.3:0/1923413171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 msgr2=0x7f593010e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 --2- 172.21.15.3:0/1923413171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593010e090 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f5928009c10 tx=0x7f592802bef0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 -- 172.21.15.3:0/1923413171 shutdown_connections 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 --2- 172.21.15.3:0/1923413171 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59300ff6a0 0x7f59300ffb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 --2- 172.21.15.3:0/1923413171 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593010e090 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 --2- 172.21.15.3:0/1923413171 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5930106200 0x7f59301065e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.873+0000 7f593869d640 1 -- 172.21.15.3:0/1923413171 >> 172.21.15.3:0/1923413171 conn(0x7f5930076630 msgr2=0x7f59300fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 -- 172.21.15.3:0/1923413171 shutdown_connections 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 -- 172.21.15.3:0/1923413171 wait complete. 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 Processor -- start 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 -- start start 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59300ff6a0 0x7f593019d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593019db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5930106200 0x7f59301976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5930197e40 con 0x7f5930100060 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5930197fb0 con 0x7f59300ff6a0 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f593869d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5930198170 con 0x7f5930106200 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f5935c38640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593019db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f5935c38640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593019db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36416/0 (socket says 172.21.15.3:36416) 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f5935c38640 1 -- 172.21.15.3:0/1070280701 learned_addr learned my addr 172.21.15.3:0/1070280701 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f5936439640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59300ff6a0 0x7f593019d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.874+0000 7f5936c3a640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5930106200 0x7f59301976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f5935c38640 1 -- 172.21.15.3:0/1070280701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5930106200 msgr2=0x7f59301976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f5935c38640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5930106200 0x7f59301976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f5935c38640 1 -- 172.21.15.3:0/1070280701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59300ff6a0 msgr2=0x7f593019d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f5935c38640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59300ff6a0 0x7f593019d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f5935c38640 1 -- 172.21.15.3:0/1070280701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5928009840 con 0x7f5930100060 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f5935c38640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593019db30 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f592000e9f0 tx=0x7f592000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f59277fe640 1 -- 172.21.15.3:0/1070280701 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920009700 con 0x7f5930100060 2024-02-09T23:01:56.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f59277fe640 1 -- 172.21.15.3:0/1070280701 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f592000ccc0 con 0x7f5930100060 2024-02-09T23:01:56.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5930198450 con 0x7f5930100060 2024-02-09T23:01:56.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f59277fe640 1 -- 172.21.15.3:0/1070280701 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5920018820 con 0x7f5930100060 2024-02-09T23:01:56.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.875+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59301a4570 con 0x7f5930100060 2024-02-09T23:01:56.888 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.878+0000 7f59277fe640 1 -- 172.21.15.3:0/1070280701 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5920010430 con 0x7f5930100060 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.878+0000 7f593b27a740 1 Processor -- start 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.878+0000 7f593b27a740 1 -- start start 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f593b27a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682adce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f593b27a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ae5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f593b27a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682ae5f60 0x55c682ae83b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f593b27a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682a98020 con 0x55c682add3e0 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f593b27a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682a4b320 con 0x55c682ae5f60 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f593b27a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682a48850 con 0x55c682adca30 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ae5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936c3a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682adce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ae5990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36422/0 (socket says 172.21.15.3:36422) 2024-02-09T23:01:56.889 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 -- 172.21.15.3:0/2490373477 learned_addr learned my addr 172.21.15.3:0/2490373477 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5935c38640 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682ae5f60 0x55c682ae83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f59277fe640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f591c0a2230 0x7f591c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 -- 172.21.15.3:0/2490373477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 msgr2=0x55c682adce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682adce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 -- 172.21.15.3:0/2490373477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682ae5f60 msgr2=0x55c682ae83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682ae5f60 0x55c682ae83b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 -- 172.21.15.3:0/2490373477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c682a06210 con 0x55c682add3e0 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5935c38640 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682ae5f60 0x55c682ae83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936c3a640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f591c0a2230 0x7f591c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f59277fe640 1 -- 172.21.15.3:0/1070280701 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5920014070 con 0x7f5930100060 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5936439640 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ae5990 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f592c009cd0 tx=0x7f592c011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a50c786f174606 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.879+0000 7f5924ff9640 1 -- 172.21.15.3:0/2490373477 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c019630 con 0x55c682add3e0 2024-02-09T23:01:56.890 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f5924ff9640 1 -- 172.21.15.3:0/2490373477 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f592c019c70 con 0x55c682add3e0 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f5924ff9640 1 -- 172.21.15.3:0/2490373477 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c00ec20 con 0x55c682add3e0 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 -- 172.21.15.3:0/2490373477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 msgr2=0x55c682ae5990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f5936c3a640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f591c0a2230 0x7f591c0a46f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5928004af0 tx=0x7f592802e040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ae5990 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f592c009cd0 tx=0x7f592c011810 comp rx=0 tx=0).stop 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 -- 172.21.15.3:0/2490373477 shutdown_connections 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682ae5f60 0x55c682ae83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ae5990 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 --2- 172.21.15.3:0/2490373477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682adce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 -- 172.21.15.3:0/2490373477 >> 172.21.15.3:0/2490373477 conn(0x55c682ad7100 msgr2=0x55c682ad74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 -- 172.21.15.3:0/2490373477 shutdown_connections 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.880+0000 7f593b27a740 1 -- 172.21.15.3:0/2490373477 wait complete. 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 -- start start 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 Processor -- start 2024-02-09T23:01:56.891 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682ae2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682ae3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 0x55c682ae39c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682a44030 con 0x55c682ae5f60 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682717430 con 0x55c682add3e0 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6826d3e00 con 0x55c682adca30 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f593b27a740 10 client.0 _ll_register_callbacks cb 0x55c682a39030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f5936c3a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 0x55c682ae39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f5936439640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682ae2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f5935c38640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682ae3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f5936c3a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 0x55c682ae39c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36436/0 (socket says 172.21.15.3:36436) 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f5936439640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682ae2f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54004/0 (socket says 172.21.15.3:54004) 2024-02-09T23:01:56.892 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.881+0000 7f5936c3a640 1 -- 172.21.15.3:0/1611753378 learned_addr learned my addr 172.21.15.3:0/1611753378 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936c3a640 1 -- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 msgr2=0x55c682ae2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936c3a640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682ae2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936c3a640 1 -- 172.21.15.3:0/1611753378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 msgr2=0x55c682ae3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936c3a640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682ae3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936c3a640 1 -- 172.21.15.3:0/1611753378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f592c009930 con 0x55c682ae5f60 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936439640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682ae2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5935c38640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682ae3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5936c3a640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 0x55c682ae39c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f5928035000 tx=0x7f5928005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52bc8893e464923f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5919ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5919ffb640 1 -- 172.21.15.3:0/1611753378 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5928003b70 con 0x55c682ae5f60 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5919ffb640 1 -- 172.21.15.3:0/1611753378 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5928003cd0 con 0x55c682ae5f60 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c6827437e0 con 0x55c682ae5f60 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5919ffb640 1 -- 172.21.15.3:0/1611753378 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5928033870 con 0x55c682ae5f60 2024-02-09T23:01:56.893 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f593b27a740 10 client.19389 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55c682a426b0 con 0x55c682ae5f60 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.882+0000 7f5919ffb640 1 -- 172.21.15.3:0/1611753378 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c6827437e0 con 0x55c682ae5f60 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f5919ffb640 1 -- 172.21.15.3:0/1611753378 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5928048040 con 0x55c682ae5f60 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f593b27a740 10 client.19389 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f593b27a740 10 client.19389 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f593b27a740 20 client.19389 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f593b27a740 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682af65d0 0x55c682af89f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f593b27a740 4 client.19389 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c682780720 con 0x55c682af65d0 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f5936439640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682af65d0 0x55c682af89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f5936439640 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682af65d0 0x55c682af89f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=131af1d84f8c0755 server_cookie=2d0b997774e3348c in_seq=0 out_seq=0 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.883+0000 7f5919ffb640 10 client.19389 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:56.894 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f5919ffb640 1 -- 172.21.15.3:0/1611753378 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c682780720 con 0x55c682af65d0 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f5919ffb640 10 client.19389 handle_command_reply: tid=0 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 client.19389 shutdown 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 msgr2=0x55c682ae39c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 0x55c682ae39c0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7f5928035000 tx=0x7f5928005fd0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 shutdown_connections 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682af65d0 0x55c682af89f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682ae3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682ae5f60 0x55c682ae39c0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f5919ffb640 0 client.19389 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 --2- 172.21.15.3:0/1611753378 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682adca30 0x55c682ae2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 >> 172.21.15.3:0/1611753378 conn(0x55c682ad7100 msgr2=0x55c682ad8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 shutdown_connections 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 1 -- 172.21.15.3:0/1611753378 wait complete. 2024-02-09T23:01:56.895 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.884+0000 7f593b27a740 20 client.19389 trim_cache size 0 max 16384 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 Processor -- start 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 -- start start 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682adca30 0x55c682ae5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ad2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682ad74e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c6829c9130 con 0x55c682add3e0 2024-02-09T23:01:56.906 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682afdc20 con 0x55c682adca30 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f593b27a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682ae1490 con 0x55c682ae5f60 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5935c38640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ad2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5935c38640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ad2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36444/0 (socket says 172.21.15.3:36444) 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936439640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682adca30 0x55c682ae5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5935c38640 1 -- 172.21.15.3:0/2514627406 learned_addr learned my addr 172.21.15.3:0/2514627406 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936c3a640 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682ad74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936439640 1 -- 172.21.15.3:0/2514627406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 msgr2=0x55c682ad74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936439640 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682ad74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936439640 1 -- 172.21.15.3:0/2514627406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 msgr2=0x55c682ad2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936439640 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ad2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5936439640 1 -- 172.21.15.3:0/2514627406 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c6829dec90 con 0x55c682adca30 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.897+0000 7f5935c38640 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ad2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:56.907 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f5936439640 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682adca30 0x55c682ae5670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55c682add7e0 tx=0x7f592c00e7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca44a65418f8703f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f591bfff640 1 -- 172.21.15.3:0/2514627406 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c011dc0 con 0x55c682adca30 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f591bfff640 1 -- 172.21.15.3:0/2514627406 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f592c009e80 con 0x55c682adca30 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f591bfff640 1 -- 172.21.15.3:0/2514627406 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c02a2a0 con 0x55c682adca30 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 -- 172.21.15.3:0/2514627406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682adca30 msgr2=0x55c682ae5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682adca30 0x55c682ae5670 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55c682add7e0 tx=0x7f592c00e7e0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 -- 172.21.15.3:0/2514627406 shutdown_connections 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682ad74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682add3e0 0x55c682ad2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 --2- 172.21.15.3:0/2514627406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682adca30 0x55c682ae5670 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 -- 172.21.15.3:0/2514627406 >> 172.21.15.3:0/2514627406 conn(0x55c682ad7100 msgr2=0x55c682add810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.908 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 -- 172.21.15.3:0/2514627406 shutdown_connections 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.898+0000 7f593b27a740 1 -- 172.21.15.3:0/2514627406 wait complete. 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 -- start start 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 Processor -- start 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 0x55c682aecc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682aed150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682aeb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682aeb7a0 con 0x55c682adca30 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682aeb910 con 0x55c682add3e0 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c682aebad0 con 0x55c682ae5f60 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f593b27a740 10 client.0 _ll_register_callbacks cb 0x7f592c0097e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 0x55c682aecc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 0x55c682aecc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36456/0 (socket says 172.21.15.3:36456) 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 -- 172.21.15.3:0/3865212840 learned_addr learned my addr 172.21.15.3:0/3865212840 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:56.909 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936c3a640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682aeb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5935c38640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682aed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 -- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 msgr2=0x55c682aeb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682aeb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 -- 172.21.15.3:0/3865212840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 msgr2=0x55c682aed150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682aed150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5936439640 1 -- 172.21.15.3:0/3865212840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59301a4570 con 0x55c682adca30 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.899+0000 7f5935c38640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682aed150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5936439640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 0x55c682aecc10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f592c006010 tx=0x7f592c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a998d6143f0f34 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5919ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5919ffb640 1 -- 172.21.15.3:0/3865212840 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f592c005500 con 0x55c682adca30 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5919ffb640 1 -- 172.21.15.3:0/3865212840 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f592c00ec70 con 0x55c682adca30 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f592c019c70 con 0x55c682adca30 2024-02-09T23:01:56.910 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5919ffb640 1 -- 172.21.15.3:0/3865212840 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f592c005660 con 0x55c682adca30 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 10 client.19395 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55c682a06210 con 0x55c682adca30 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5919ffb640 1 -- 172.21.15.3:0/3865212840 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f592c040020 con 0x55c682adca30 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f5919ffb640 1 -- 172.21.15.3:0/3865212840 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f592c0039f0 con 0x55c682adca30 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 10 client.19395 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 10 client.19395 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 20 client.19395 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682b4d860 0x55c682b4fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 4 client.19395 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.900+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c6827cf890 con 0x55c682b4d860 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f5935c38640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682b4d860 0x55c682b4fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f5919ffb640 10 client.19395 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:56.911 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f5935c38640 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682b4d860 0x55c682b4fc80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=aa04f0212c510820 server_cookie=811ee3a4216d0965 in_seq=0 out_seq=0 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f5919ffb640 1 -- 172.21.15.3:0/3865212840 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c6827cf890 con 0x55c682b4d860 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f5919ffb640 10 client.19395 handle_command_reply: tid=0 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 client.19395 shutdown 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 msgr2=0x55c682aecc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 0x55c682aecc10 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f592c006010 tx=0x7f592c005d90 comp rx=0 tx=0).stop 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 shutdown_connections 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55c682b4d860 0x55c682b4fc80 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c682add3e0 0x55c682aed150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f5919ffb640 0 client.19395 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c682ae5f60 0x55c682aeb0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 --2- 172.21.15.3:0/3865212840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c682adca30 0x55c682aecc10 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 >> 172.21.15.3:0/3865212840 conn(0x55c682ad7100 msgr2=0x55c682ade250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.912 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 shutdown_connections 2024-02-09T23:01:56.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 1 -- 172.21.15.3:0/3865212840 wait complete. 2024-02-09T23:01:56.913 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.901+0000 7f593b27a740 20 client.19395 trim_cache size 0 max 16384 2024-02-09T23:01:56.913 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f591c0a2230 msgr2=0x7f591c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f591c0a2230 0x7f591c0a46f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f5928004af0 tx=0x7f592802e040 comp rx=0 tx=0).stop 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 msgr2=0x7f593019db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593019db30 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f592000e9f0 tx=0x7f592000edf0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 shutdown_connections 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f591c0a2230 0x7f591c0a46f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5930106200 0x7f59301976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5930100060 0x7f593019db30 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 --2- 172.21.15.3:0/1070280701 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f59300ff6a0 0x7f593019d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 >> 172.21.15.3:0/1070280701 conn(0x7f5930076630 msgr2=0x7f5930104030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 shutdown_connections 2024-02-09T23:01:56.914 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:56.902+0000 7f593869d640 1 -- 172.21.15.3:0/1070280701 wait complete. 2024-02-09T23:01:56.916 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:56.917 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:01:56.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:56 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3032766835' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:01:57.029 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 -- 172.21.15.3:0/4280403699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4105410 msgr2=0x7f82c410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 --2- 172.21.15.3:0/4280403699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4105410 0x7f82c410cfe0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f82bc009cd0 tx=0x7f82bc02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 -- 172.21.15.3:0/4280403699 shutdown_connections 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 --2- 172.21.15.3:0/4280403699 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100cf0 0x7f82c4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 --2- 172.21.15.3:0/4280403699 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4105410 0x7f82c410cfe0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 --2- 172.21.15.3:0/4280403699 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4100340 0x7f82c4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 -- 172.21.15.3:0/4280403699 >> 172.21.15.3:0/4280403699 conn(0x7f82c40772e0 msgr2=0x7f82c40776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 -- 172.21.15.3:0/4280403699 shutdown_connections 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.025+0000 7f82c9783640 1 -- 172.21.15.3:0/4280403699 wait complete. 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 Processor -- start 2024-02-09T23:01:57.030 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 -- start start 2024-02-09T23:01:57.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 0x7f82c419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.032 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4105410 0x7f82c419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 0x7f82c4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c4198140 con 0x7f82c4197670 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c41982b0 con 0x7f82c4100340 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c9783640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82c4198470 con 0x7f82c4105410 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 0x7f82c4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 0x7f82c419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 0x7f82c419d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37520/0 (socket says 172.21.15.3:37520) 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 0x7f82c4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36476/0 (socket says 172.21.15.3:36476) 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82c2ffd640 1 -- 172.21.15.3:0/1661351751 learned_addr learned my addr 172.21.15.3:0/1661351751 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.026+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4105410 0x7f82c419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c2ffd640 1 -- 172.21.15.3:0/1661351751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4105410 msgr2=0x7f82c419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4105410 0x7f82c419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c2ffd640 1 -- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 msgr2=0x7f82c4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.033 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 0x7f82c4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c2ffd640 1 -- 172.21.15.3:0/1661351751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82bc009900 con 0x7f82c4100340 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4105410 0x7f82c419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c37fe640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 0x7f82c4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 0x7f82c419d5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82b400e9f0 tx=0x7f82b400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c0ff9640 1 -- 172.21.15.3:0/1661351751 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4009720 con 0x7f82c4100340 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82c41a4410 con 0x7f82c4100340 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c0ff9640 1 -- 172.21.15.3:0/1661351751 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82b400cd80 con 0x7f82c4100340 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c0ff9640 1 -- 172.21.15.3:0/1661351751 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b4010690 con 0x7f82c4100340 2024-02-09T23:01:57.034 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.027+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c41a48f0 con 0x7f82c4100340 2024-02-09T23:01:57.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.029+0000 7f82c0ff9640 1 -- 172.21.15.3:0/1661351751 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f82b40046c0 con 0x7f82c4100340 2024-02-09T23:01:57.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.029+0000 7f82cc431740 1 Processor -- start 2024-02-09T23:01:57.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 -- start start 2024-02-09T23:01:57.041 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1f8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c201990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c2043b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c1b4020 con 0x555e4c1f93e0 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c167320 con 0x555e4c1f8a30 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82cc431740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c164850 con 0x555e4c201f60 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c201990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c0ff9640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82b00a2230 0x7f82b00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c201990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36478/0 (socket says 172.21.15.3:36478) 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1f8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3261506542 learned_addr learned my addr 172.21.15.3:0/3261506542 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c2043b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c37fe640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82b00a2230 0x7f82b00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.030+0000 7f82c0ff9640 1 -- 172.21.15.3:0/1661351751 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f82b4014070 con 0x7f82c4100340 2024-02-09T23:01:57.042 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3261506542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 msgr2=0x555e4c2043b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c2043b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3261506542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 msgr2=0x555e4c1f8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1f8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3261506542 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e4c122210 con 0x555e4c1f93e0 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c201990 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f82b409eaa0 tx=0x7f82b4004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ec0258276c5d95 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82c37fe640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82b00a2230 0x7f82b00a46f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f82bc005cc0 tx=0x7f82bc02c0f0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82b97fa640 1 -- 172.21.15.3:0/3261506542 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b409ef60 con 0x555e4c1f93e0 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82b97fa640 1 -- 172.21.15.3:0/3261506542 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82b409d070 con 0x555e4c1f93e0 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82b97fa640 1 -- 172.21.15.3:0/3261506542 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40a3120 con 0x555e4c1f93e0 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- 172.21.15.3:0/3261506542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 msgr2=0x555e4c201990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c201990 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7f82b409eaa0 tx=0x7f82b4004b30 comp rx=0 tx=0).stop 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- 172.21.15.3:0/3261506542 shutdown_connections 2024-02-09T23:01:57.043 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c2043b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c201990 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- 172.21.15.3:0/3261506542 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1f8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- 172.21.15.3:0/3261506542 >> 172.21.15.3:0/3261506542 conn(0x555e4c1f3100 msgr2=0x555e4c1f34e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- 172.21.15.3:0/3261506542 shutdown_connections 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- 172.21.15.3:0/3261506542 wait complete. 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- start start 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 Processor -- start 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1feec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c1ff400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c1ff940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.032+0000 7f82cc431740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c160030 con 0x555e4c1f93e0 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82cc431740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4be33430 con 0x555e4c1f8a30 2024-02-09T23:01:57.044 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82cc431740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4bdefe00 con 0x555e4c201f60 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82cc431740 10 client.0 _ll_register_callbacks cb 0x555e4c155030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c1ff400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1feec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c1ff400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36492/0 (socket says 172.21.15.3:36492) 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 -- 172.21.15.3:0/867049755 learned_addr learned my addr 172.21.15.3:0/867049755 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82c37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c1ff940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 -- 172.21.15.3:0/867049755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 msgr2=0x555e4c1ff940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c1ff940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 -- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 msgr2=0x555e4c1feec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1feec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 -- 172.21.15.3:0/867049755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82c41a48f0 con 0x555e4c1f93e0 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.045 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c1ff400 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f82ac012d20 tx=0x7f82ac00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f51e35fc505f9e0e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82aa7fc640 1 -- 172.21.15.3:0/867049755 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ac017070 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82aa7fc640 1 -- 172.21.15.3:0/867049755 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f82ac0120d0 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e4be5f7e0 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82aa7fc640 1 -- 172.21.15.3:0/867049755 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82ac012290 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82cc431740 10 client.19407 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.033+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x555e4c15e6b0 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82aa7fc640 1 -- 172.21.15.3:0/867049755 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82ac028080 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82aa7fc640 1 -- 172.21.15.3:0/867049755 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f82ac022030 con 0x555e4c1f93e0 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82cc431740 10 client.19407 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82cc431740 10 client.19407 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82cc431740 20 client.19407 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82cc431740 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c2124e0 0x555e4c214920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.046 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82cc431740 4 client.19407 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e4be9c720 con 0x555e4c2124e0 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.034+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c2124e0 0x555e4c214920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c2124e0 0x555e4c214920 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9c552c7cc6653f09 server_cookie=43dd748ef0aa2a8c in_seq=0 out_seq=0 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82aa7fc640 10 client.19407 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82aa7fc640 1 -- 172.21.15.3:0/867049755 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e4be9c720 con 0x555e4c2124e0 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82aa7fc640 10 client.19407 handle_command_reply: tid=0 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82cc431740 1 client.19407 shutdown 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 msgr2=0x555e4c1ff400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.035+0000 7f82cc431740 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c1ff400 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f82ac012d20 tx=0x7f82ac00f790 comp rx=0 tx=0).stop 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 shutdown_connections 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c2124e0 0x555e4c214920 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c201f60 0x555e4c1ff940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f93e0 0x555e4c1ff400 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.047 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 --2- 172.21.15.3:0/867049755 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1feec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 >> 172.21.15.3:0/867049755 conn(0x555e4c1f3100 msgr2=0x555e4c1f4e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82aa7fc640 0 client.19407 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:57.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 shutdown_connections 2024-02-09T23:01:57.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 1 -- 172.21.15.3:0/867049755 wait complete. 2024-02-09T23:01:57.048 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.036+0000 7f82cc431740 20 client.19407 trim_cache size 0 max 16384 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 Processor -- start 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 -- start start 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c2015f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1ff760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c1fa210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c0e5130 con 0x555e4c1f3100 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c20de70 con 0x555e4c1f8a30 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82cc431740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c1ffca0 con 0x555e4c1f93e0 2024-02-09T23:01:57.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c2015f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c2015f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36508/0 (socket says 172.21.15.3:36508) 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3715778720 learned_addr learned my addr 172.21.15.3:0/3715778720 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82c37fe640 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c1fa210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.048+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1ff760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3715778720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 msgr2=0x555e4c1fa210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c1fa210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3715778720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 msgr2=0x555e4c1ff760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1ff760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3715778720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e4c0fac90 con 0x555e4c1f3100 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c2015f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f82b4097790 tx=0x7f82b40cbae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f26227844ab6b00 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82b8ff9640 1 -- 172.21.15.3:0/3715778720 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40ce520 con 0x555e4c1f3100 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82b8ff9640 1 -- 172.21.15.3:0/3715778720 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82b409efe0 con 0x555e4c1f3100 2024-02-09T23:01:57.056 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.049+0000 7f82b8ff9640 1 -- 172.21.15.3:0/3715778720 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40a3120 con 0x555e4c1f3100 2024-02-09T23:01:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- 172.21.15.3:0/3715778720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 msgr2=0x555e4c2015f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.060 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c2015f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f82b4097790 tx=0x7f82b40cbae0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- 172.21.15.3:0/3715778720 shutdown_connections 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c1ff760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c1fa210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- 172.21.15.3:0/3715778720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c2015f0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- 172.21.15.3:0/3715778720 >> 172.21.15.3:0/3715778720 conn(0x555e4c1f9b30 msgr2=0x555e4c2192c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- 172.21.15.3:0/3715778720 shutdown_connections 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- 172.21.15.3:0/3715778720 wait complete. 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- start start 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 Processor -- start 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c206f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c207450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c267a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c207bf0 con 0x555e4c1f3100 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c207d60 con 0x555e4c1f8a30 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e4c267f80 con 0x555e4c1f93e0 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82cc431740 10 client.0 _ll_register_callbacks cb 0x7f82b409abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.050+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c206f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c206f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36516/0 (socket says 172.21.15.3:36516) 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3525762074 learned_addr learned my addr 172.21.15.3:0/3525762074 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82ba7fc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c207450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c37fe640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c267a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3525762074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 msgr2=0x555e4c267a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c267a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 msgr2=0x555e4c207450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c207450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 -- 172.21.15.3:0/3525762074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e4c122210 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c207450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82c2ffd640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c206f10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f82b4004ae0 tx=0x7f82b40ce3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ade0d4bacfdbee9a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82aa7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82aa7fc640 1 -- 172.21.15.3:0/3525762074 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82b40a35b0 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82aa7fc640 1 -- 172.21.15.3:0/3525762074 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f82b40b9270 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82b409d070 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82aa7fc640 1 -- 172.21.15.3:0/3525762074 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82b409b1c0 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82cc431740 10 client.19413 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x555e4c120a70 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.051+0000 7f82aa7fc640 1 -- 172.21.15.3:0/3525762074 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82b409d070 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82aa7fc640 1 -- 172.21.15.3:0/3525762074 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f82b401f060 con 0x555e4c1f3100 2024-02-09T23:01:57.063 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 10 client.19413 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 10 client.19413 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 20 client.19413 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c201f60 0x555e4c26b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 4 client.19413 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e4beeb890 con 0x555e4c201f60 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c201f60 0x555e4c26b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82ba7fc640 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c201f60 0x555e4c26b3e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aabaed458175e2e8 server_cookie=b7f7b491ef03b219 in_seq=0 out_seq=0 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82aa7fc640 10 client.19413 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82aa7fc640 1 -- 172.21.15.3:0/3525762074 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e4beeb890 con 0x555e4c201f60 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82aa7fc640 10 client.19413 handle_command_reply: tid=0 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 client.19413 shutdown 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 msgr2=0x555e4c206f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c206f10 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f82b4004ae0 tx=0x7f82b40ce3b0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 shutdown_connections 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e4c201f60 0x555e4c26b3e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e4c1f8a30 0x555e4c207450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e4c1f93e0 0x555e4c267a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 --2- 172.21.15.3:0/3525762074 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e4c1f3100 0x555e4c206f10 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82aa7fc640 0 client.19413 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.052+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 >> 172.21.15.3:0/3525762074 conn(0x555e4c1f9b30 msgr2=0x555e4c1fa210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 shutdown_connections 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82cc431740 1 -- 172.21.15.3:0/3525762074 wait complete. 2024-02-09T23:01:57.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82cc431740 20 client.19413 trim_cache size 0 max 16384 2024-02-09T23:01:57.066 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:57.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82b00a2230 msgr2=0x7f82b00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82b00a2230 0x7f82b00a46f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f82bc005cc0 tx=0x7f82bc02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 msgr2=0x7f82c419d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 0x7f82c419d5b0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f82b400e9f0 tx=0x7f82b400edf0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 shutdown_connections 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f82b00a2230 0x7f82b00a46f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f82c4197670 0x7f82c4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f82c4105410 0x7f82c419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 --2- 172.21.15.3:0/1661351751 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f82c4100340 0x7f82c419d5b0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 >> 172.21.15.3:0/1661351751 conn(0x7f82c40772e0 msgr2=0x7f82c4104010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 shutdown_connections 2024-02-09T23:01:57.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.053+0000 7f82c9783640 1 -- 172.21.15.3:0/1661351751 wait complete. 2024-02-09T23:01:57.068 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:01:57.069 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:01:57.182 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 -- 172.21.15.3:0/1172172088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 msgr2=0x7feeb810afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 --2- 172.21.15.3:0/1172172088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb810afa0 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7feea0009900 tx=0x7feea002bc30 comp rx=0 tx=0).stop 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 -- 172.21.15.3:0/1172172088 shutdown_connections 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 --2- 172.21.15.3:0/1172172088 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb810b610 0x7feeb810da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 --2- 172.21.15.3:0/1172172088 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb810afa0 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 --2- 172.21.15.3:0/1172172088 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb81021a0 0x7feeb8102580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 -- 172.21.15.3:0/1172172088 >> 172.21.15.3:0/1172172088 conn(0x7feeb80fba00 msgr2=0x7feeb80fde20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 -- 172.21.15.3:0/1172172088 shutdown_connections 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.178+0000 7feebc9a0640 1 -- 172.21.15.3:0/1172172088 wait complete. 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 Processor -- start 2024-02-09T23:01:57.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 -- start start 2024-02-09T23:01:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb81021a0 0x7feeb81976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb8197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb810b610 0x7feeb819bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb819c570 con 0x7feeb8102ac0 2024-02-09T23:01:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb819c6e0 con 0x7feeb810b610 2024-02-09T23:01:57.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feebc9a0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeb819c8a0 con 0x7feeb81021a0 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb8197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb8197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36538/0 (socket says 172.21.15.3:36538) 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feeaeffd640 1 -- 172.21.15.3:0/4148997654 learned_addr learned my addr 172.21.15.3:0/4148997654 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feeb77fe640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb81021a0 0x7feeb81976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.179+0000 7feeb7fff640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb810b610 0x7feeb819bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeaeffd640 1 -- 172.21.15.3:0/4148997654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb81021a0 msgr2=0x7feeb81976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeaeffd640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb81021a0 0x7feeb81976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeaeffd640 1 -- 172.21.15.3:0/4148997654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb810b610 msgr2=0x7feeb819bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeaeffd640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb810b610 0x7feeb819bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeaeffd640 1 -- 172.21.15.3:0/4148997654 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0009560 con 0x7feeb8102ac0 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeb7fff640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb810b610 0x7feeb819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeaeffd640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb8197bf0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7feea0036040 tx=0x7feea002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeb57fa640 1 -- 172.21.15.3:0/4148997654 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0004350 con 0x7feeb8102ac0 2024-02-09T23:01:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeb819cb20 con 0x7feeb8102ac0 2024-02-09T23:01:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeb57fa640 1 -- 172.21.15.3:0/4148997654 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea00044b0 con 0x7feeb8102ac0 2024-02-09T23:01:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feeb57fa640 1 -- 172.21.15.3:0/4148997654 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0003da0 con 0x7feeb8102ac0 2024-02-09T23:01:57.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.180+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb81a4400 con 0x7feeb8102ac0 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.182+0000 7feeb57fa640 1 -- 172.21.15.3:0/4148997654 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7feea0004df0 con 0x7feeb8102ac0 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 Processor -- start 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 -- start start 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 0x562586afefa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b07b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586b080f0 0x562586b0a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586aba020 con 0x562586afebc0 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586a6d320 con 0x562586b080f0 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feec07d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586a6a850 con 0x562586aff570 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 0x562586afefa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b07b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 0x562586afefa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36550/0 (socket says 172.21.15.3:36550) 2024-02-09T23:01:57.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb77fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b07b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54104/0 (socket says 172.21.15.3:54104) 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 -- 172.21.15.3:0/3188925595 learned_addr learned my addr 172.21.15.3:0/3188925595 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeaeffd640 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586b080f0 0x562586b0a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb57fa640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7feea40a2230 0x7feea40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 -- 172.21.15.3:0/3188925595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 msgr2=0x562586b07b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b07b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 -- 172.21.15.3:0/3188925595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586b080f0 msgr2=0x562586b0a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586b080f0 0x562586b0a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 -- 172.21.15.3:0/3188925595 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562586a28210 con 0x562586afebc0 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb57fa640 1 -- 172.21.15.3:0/4148997654 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feea00f0040 con 0x7feeb8102ac0 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb77fe640 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b07b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeaeffd640 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586b080f0 0x562586b0a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.183+0000 7feeb7fff640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7feea40a2230 0x7feea40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.184+0000 7feeb7fff640 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 0x562586afefa0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7feeb00118d0 tx=0x7feeb0011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3049bfbcb26f96d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.184+0000 7feeadffb640 1 -- 172.21.15.3:0/3188925595 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb001bb90 con 0x562586afebc0 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.184+0000 7feeadffb640 1 -- 172.21.15.3:0/3188925595 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feeb001bcf0 con 0x562586afebc0 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.184+0000 7feeadffb640 1 -- 172.21.15.3:0/3188925595 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feeb000e5b0 con 0x562586afebc0 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 -- 172.21.15.3:0/3188925595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 msgr2=0x562586afefa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feeb7fff640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7feea40a2230 0x7feea40a46f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feeb0023000 tx=0x7feeb0024000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 0x562586afefa0 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7feeb00118d0 tx=0x7feeb0011d90 comp rx=0 tx=0).stop 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 -- 172.21.15.3:0/3188925595 shutdown_connections 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586b080f0 0x562586b0a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b07b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 --2- 172.21.15.3:0/3188925595 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586afebc0 0x562586afefa0 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 -- 172.21.15.3:0/3188925595 >> 172.21.15.3:0/3188925595 conn(0x562586af9290 msgr2=0x562586af9670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 -- 172.21.15.3:0/3188925595 shutdown_connections 2024-02-09T23:01:57.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 -- 172.21.15.3:0/3188925595 wait complete. 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.185+0000 7feec07d3740 1 -- start start 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 Processor -- start 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b05070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b055b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b05af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586a66030 con 0x562586aff570 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586739430 con 0x562586afebc0 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625866f5e00 con 0x562586b080f0 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feec07d3740 10 client.0 _ll_register_callbacks cb 0x562586a5b030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b055b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b055b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36560/0 (socket says 172.21.15.3:36560) 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 -- 172.21.15.3:0/712713774 learned_addr learned my addr 172.21.15.3:0/712713774 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeb77fe640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b05070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeb7fff640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b05af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 -- 172.21.15.3:0/712713774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 msgr2=0x562586b05af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b05af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 -- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 msgr2=0x562586b05070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b05070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 -- 172.21.15.3:0/712713774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeb0011490 con 0x562586aff570 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeb77fe640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b05070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7feeaeffd640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b055b0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7feea0006010 tx=0x7feea0003840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9cb12aadf445652 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.186+0000 7fee96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7fee96ffd640 1 -- 172.21.15.3:0/712713774 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00bc710 con 0x562586aff570 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7fee96ffd640 1 -- 172.21.15.3:0/712713774 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feea00bd2d0 con 0x562586aff570 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5625867657e0 con 0x562586aff570 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7fee96ffd640 1 -- 172.21.15.3:0/712713774 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7feea00bc530 con 0x562586aff570 2024-02-09T23:01:57.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7feec07d3740 10 client.19425 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x562586a646b0 con 0x562586aff570 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.187+0000 7fee96ffd640 1 -- 172.21.15.3:0/712713774 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5625867657e0 con 0x562586aff570 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7fee96ffd640 1 -- 172.21.15.3:0/712713774 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7feea00dc040 con 0x562586aff570 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feec07d3740 10 client.19425 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feec07d3740 10 client.19425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feec07d3740 20 client.19425 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feec07d3740 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b183a0 0x562586b1a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feec07d3740 4 client.19425 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5625867a2720 con 0x562586b183a0 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feeb77fe640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b183a0 0x562586b1a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7feeb77fe640 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b183a0 0x562586b1a790 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fcb3b76f02ee0ba server_cookie=a2accbe8724d5d1d in_seq=0 out_seq=0 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.188+0000 7fee96ffd640 10 client.19425 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7fee96ffd640 1 -- 172.21.15.3:0/712713774 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5625867a2720 con 0x562586b183a0 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7fee96ffd640 10 client.19425 handle_command_reply: tid=0 2024-02-09T23:01:57.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 client.19425 shutdown 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 msgr2=0x562586b055b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b055b0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7feea0006010 tx=0x7feea0003840 comp rx=0 tx=0).stop 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 shutdown_connections 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b183a0 0x562586b1a790 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b05af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b055b0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7fee96ffd640 0 client.19425 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 --2- 172.21.15.3:0/712713774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b05070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 >> 172.21.15.3:0/712713774 conn(0x562586af9290 msgr2=0x562586af84f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 shutdown_connections 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 1 -- 172.21.15.3:0/712713774 wait complete. 2024-02-09T23:01:57.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.189+0000 7feec07d3740 20 client.19425 trim_cache size 0 max 16384 2024-02-09T23:01:57.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 Processor -- start 2024-02-09T23:01:57.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 -- start start 2024-02-09T23:01:57.206 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b077a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b05910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b003a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5625869eb130 con 0x562586aff570 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586afe510 con 0x562586afebc0 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feec07d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586b1fa60 con 0x562586b080f0 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b05910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b05910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36572/0 (socket says 172.21.15.3:36572) 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 -- 172.21.15.3:0/552859831 learned_addr learned my addr 172.21.15.3:0/552859831 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b003a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeb77fe640 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 -- 172.21.15.3:0/552859831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 msgr2=0x562586b003a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b003a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 -- 172.21.15.3:0/552859831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 msgr2=0x562586b077a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.207 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b077a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 -- 172.21.15.3:0/552859831 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562586a00c90 con 0x562586aff570 2024-02-09T23:01:57.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feeaeffd640 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b05910 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7feea0005ec0 tx=0x7feea00bb0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34608f5c8d467f43 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feead7fa640 1 -- 172.21.15.3:0/552859831 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00bc710 con 0x562586aff570 2024-02-09T23:01:57.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feead7fa640 1 -- 172.21.15.3:0/552859831 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea00f2110 con 0x562586aff570 2024-02-09T23:01:57.208 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.202+0000 7feead7fa640 1 -- 172.21.15.3:0/552859831 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00f3430 con 0x562586aff570 2024-02-09T23:01:57.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 -- 172.21.15.3:0/552859831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 msgr2=0x562586b05910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b05910 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7feea0005ec0 tx=0x7feea00bb0e0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 -- 172.21.15.3:0/552859831 shutdown_connections 2024-02-09T23:01:57.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586b080f0 0x562586b003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586aff570 0x562586b05910 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 --2- 172.21.15.3:0/552859831 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 -- 172.21.15.3:0/552859831 >> 172.21.15.3:0/552859831 conn(0x562586af9290 msgr2=0x562586af4380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 -- 172.21.15.3:0/552859831 shutdown_connections 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 -- 172.21.15.3:0/552859831 wait complete. 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.203+0000 7feec07d3740 1 -- start start 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 Processor -- start 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b0f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b0d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586b080f0 0x562586b0dc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586b0e160 con 0x562586b080f0 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586b0e2d0 con 0x562586afebc0 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562586b0e490 con 0x562586aff570 2024-02-09T23:01:57.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feec07d3740 10 client.0 _ll_register_callbacks cb 0x7feeb0009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b0d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeb77fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b0f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeb7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586b080f0 0x562586b0dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b0d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54126/0 (socket says 172.21.15.3:54126) 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 -- 172.21.15.3:0/2662293553 learned_addr learned my addr 172.21.15.3:0/2662293553 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 -- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 msgr2=0x562586b0f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b0f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 -- 172.21.15.3:0/2662293553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586b080f0 msgr2=0x562586b0dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586b080f0 0x562586b0dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeaeffd640 1 -- 172.21.15.3:0/2662293553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562586a646b0 con 0x562586aff570 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeb77fe640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b0f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.204+0000 7feeb7fff640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586b080f0 0x562586b0dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:57.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feeaeffd640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b0d6e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feea00edd90 tx=0x7feea0003840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72e8d74e5d80bd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7fee96ffd640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7fee96ffd640 1 -- 172.21.15.3:0/2662293553 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea00bc710 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7fee96ffd640 1 -- 172.21.15.3:0/2662293553 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feea00c0d20 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7feeb001bcf0 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7fee96ffd640 1 -- 172.21.15.3:0/2662293553 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7feea00bd2d0 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 10 client.28426 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x562586a26a70 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7fee96ffd640 1 -- 172.21.15.3:0/2662293553 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feeb001bcf0 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7fee96ffd640 1 -- 172.21.15.3:0/2662293553 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7feea00dc070 con 0x562586aff570 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 10 client.28426 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 10 client.28426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 20 client.28426 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:57.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b73b80 0x562586b75f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 4 client.28426 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5625867f1890 con 0x562586b73b80 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.205+0000 7feeb77fe640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b73b80 0x562586b75f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feeb77fe640 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b73b80 0x562586b75f70 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=daeed0297d90d98e server_cookie=7e1a72b1fc58070c in_seq=0 out_seq=0 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7fee96ffd640 10 client.28426 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7fee96ffd640 1 -- 172.21.15.3:0/2662293553 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5625867f1890 con 0x562586b73b80 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7fee96ffd640 10 client.28426 handle_command_reply: tid=0 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 client.28426 shutdown 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 msgr2=0x562586b0d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b0d6e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7feea00edd90 tx=0x7feea0003840 comp rx=0 tx=0).stop 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 shutdown_connections 2024-02-09T23:01:57.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562586b73b80 0x562586b75f70 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562586b080f0 0x562586b0dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562586aff570 0x562586b0d6e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 --2- 172.21.15.3:0/2662293553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562586afebc0 0x562586b0f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7fee96ffd640 0 client.28426 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 >> 172.21.15.3:0/2662293553 conn(0x562586af9290 msgr2=0x562586b003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 shutdown_connections 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 1 -- 172.21.15.3:0/2662293553 wait complete. 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.206+0000 7feec07d3740 20 client.28426 trim_cache size 0 max 16384 2024-02-09T23:01:57.217 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:01:57.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7feea40a2230 msgr2=0x7feea40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7feea40a2230 0x7feea40a46f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7feeb0023000 tx=0x7feeb0024000 comp rx=0 tx=0).stop 2024-02-09T23:01:57.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 msgr2=0x7feeb8197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:57.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb8197bf0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7feea0036040 tx=0x7feea002c680 comp rx=0 tx=0).stop 2024-02-09T23:01:57.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 shutdown_connections 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7feea40a2230 0x7feea40a46f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeb810b610 0x7feeb819bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeb8102ac0 0x7feeb8197bf0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 --2- 172.21.15.3:0/4148997654 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeb81021a0 0x7feeb81976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 >> 172.21.15.3:0/4148997654 conn(0x7feeb80fba00 msgr2=0x7feeb80fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 shutdown_connections 2024-02-09T23:01:57.219 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:57.207+0000 7feebc9a0640 1 -- 172.21.15.3:0/4148997654 wait complete. 2024-02-09T23:01:57.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 359941 3.22 MB/sec execute 471 sec latency 435.786 ms 2024-02-09T23:01:57.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:57 smithi003 ceph-mon[32678]: pgmap v334: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 182 op/s 2024-02-09T23:01:57.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:57 smithi064 ceph-mon[36638]: pgmap v334: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 182 op/s 2024-02-09T23:01:57.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:57 smithi089 ceph-mon[36665]: pgmap v334: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.1 MiB/s wr, 182 op/s 2024-02-09T23:01:58.222 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:01:58.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:01:58.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:01:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 -- 172.21.15.3:0/2592825342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 msgr2=0x7f62d010cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 --2- 172.21.15.3:0/2592825342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d010cfb0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f62c8009cd0 tx=0x7f62c802c0f0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 -- 172.21.15.3:0/2592825342 shutdown_connections 2024-02-09T23:01:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 --2- 172.21.15.3:0/2592825342 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100cc0 0x7f62d0104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 --2- 172.21.15.3:0/2592825342 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d010cfb0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 --2- 172.21.15.3:0/2592825342 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62d0100310 0x7f62d01006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 -- 172.21.15.3:0/2592825342 >> 172.21.15.3:0/2592825342 conn(0x7f62d0077230 msgr2=0x7f62d0077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 -- 172.21.15.3:0/2592825342 shutdown_connections 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.335+0000 7f62d486f640 1 -- 172.21.15.3:0/2592825342 wait complete. 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 Processor -- start 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 -- start start 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100310 0x7f62d0101da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d01022e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62d0102820 0x7f62d01a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01a5fa0 con 0x7f62d01053e0 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01a6110 con 0x7f62d0100310 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62d486f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d01a62d0 con 0x7f62d0102820 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62cdd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d01022e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62ce59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100310 0x7f62d0101da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62cdd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d01022e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36600/0 (socket says 172.21.15.3:36600) 2024-02-09T23:01:58.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2918178406 learned_addr learned my addr 172.21.15.3:0/2918178406 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.336+0000 7f62ced9d640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62d0102820 0x7f62d01a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2918178406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62d0102820 msgr2=0x7f62d01a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62d0102820 0x7f62d01a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2918178406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100310 msgr2=0x7f62d0101da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100310 0x7f62d0101da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2918178406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c8009900 con 0x7f62d01053e0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62ce59c640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100310 0x7f62d0101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d01022e0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9c0 tx=0x7f62c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62bf7fe640 1 -- 172.21.15.3:0/2918178406 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4009700 con 0x7f62d01053e0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62bf7fe640 1 -- 172.21.15.3:0/2918178406 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c400cd80 con 0x7f62d01053e0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d01a65b0 con 0x7f62d01053e0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62bf7fe640 1 -- 172.21.15.3:0/2918178406 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c4010430 con 0x7f62d01053e0 2024-02-09T23:01:58.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.337+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d01a6ac0 con 0x7f62d01053e0 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.339+0000 7f62bf7fe640 1 -- 172.21.15.3:0/2918178406 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f62c4005000 con 0x7f62d01053e0 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 Processor -- start 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 -- start start 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b652d2570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652db150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652db690 0x561b652dda80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b65240000 con 0x561b652d2ab0 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b6523d530 con 0x561b652db690 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62d7440740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b65238d10 con 0x561b652d2190 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62bf7fe640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62b80a2260 0x7f62b80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62ce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652db150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62ce59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652db150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36610/0 (socket says 172.21.15.3:36610) 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62ce59c640 1 -- 172.21.15.3:0/4230085183 learned_addr learned my addr 172.21.15.3:0/4230085183 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62ced9d640 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b652d2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652db690 0x561b652dda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.340+0000 7f62ced9d640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62b80a2260 0x7f62b80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62bf7fe640 1 -- 172.21.15.3:0/2918178406 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f62c4014070 con 0x7f62d01053e0 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ce59c640 1 -- 172.21.15.3:0/4230085183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 msgr2=0x561b652d2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ce59c640 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b652d2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ce59c640 1 -- 172.21.15.3:0/4230085183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652db690 msgr2=0x561b652dda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ce59c640 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652db690 0x561b652dda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ce59c640 1 -- 172.21.15.3:0/4230085183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b651df210 con 0x561b652d2ab0 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ce59c640 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652db150 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f62c00115c0 tx=0x7f62c0011a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe51db2f66f9e4c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62ced9d640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62b80a2260 0x7f62b80a4720 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005cc0 tx=0x7f62c8005c50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62bcff9640 1 -- 172.21.15.3:0/4230085183 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000f640 con 0x561b652d2ab0 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62bcff9640 1 -- 172.21.15.3:0/4230085183 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c000fc80 con 0x561b652d2ab0 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.341+0000 7f62bcff9640 1 -- 172.21.15.3:0/4230085183 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0019cb0 con 0x561b652d2ab0 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- 172.21.15.3:0/4230085183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 msgr2=0x561b652db150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652db150 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7f62c00115c0 tx=0x7f62c0011a80 comp rx=0 tx=0).stop 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- 172.21.15.3:0/4230085183 shutdown_connections 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652db690 0x561b652dda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652db150 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- 172.21.15.3:0/4230085183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b652d2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- 172.21.15.3:0/4230085183 >> 172.21.15.3:0/4230085183 conn(0x561b652cc8c0 msgr2=0x561b652ccca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- 172.21.15.3:0/4230085183 shutdown_connections 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- 172.21.15.3:0/4230085183 wait complete. 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- start start 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 Processor -- start 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652d8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2ab0 0x561b652d8bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652d90f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b651dde60 con 0x561b652db690 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b64f0c4b0 con 0x561b652d2190 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b651be670 con 0x561b652d2ab0 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.342+0000 7f62d7440740 10 client.0 _ll_register_callbacks cb 0x561b651d6190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652d90f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ce59c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652d8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652d90f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36612/0 (socket says 172.21.15.3:36612) 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62cdd9b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2ab0 0x561b652d8bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 -- 172.21.15.3:0/3319767546 learned_addr learned my addr 172.21.15.3:0/3319767546 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 -- 172.21.15.3:0/3319767546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2ab0 msgr2=0x561b652d8bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2ab0 0x561b652d8bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 -- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 msgr2=0x561b652d8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652d8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 -- 172.21.15.3:0/3319767546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c0009d40 con 0x561b652db690 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62ced9d640 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652d90f0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f62c8036000 tx=0x7f62c80095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701f79b884645416 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62b5ffb640 1 -- 172.21.15.3:0/3319767546 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c8034560 con 0x561b652db690 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62b5ffb640 1 -- 172.21.15.3:0/3319767546 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f62c8003b70 con 0x561b652db690 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x561b64f38860 con 0x561b652db690 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62b5ffb640 1 -- 172.21.15.3:0/3319767546 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62c8003d30 con 0x561b652db690 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62d7440740 10 client.19443 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:58.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.343+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x561b65237390 con 0x561b652db690 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62b5ffb640 1 -- 172.21.15.3:0/3319767546 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c805c020 con 0x561b652db690 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62b5ffb640 1 -- 172.21.15.3:0/3319767546 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f62c804c070 con 0x561b652db690 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62d7440740 10 client.19443 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62d7440740 10 client.19443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62d7440740 20 client.19443 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62d7440740 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b652eb700 0x561b652edb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62d7440740 4 client.19443 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561b64f89410 con 0x561b652eb700 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62ce59c640 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b652eb700 0x561b652edb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62b5ffb640 10 client.19443 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.344+0000 7f62ce59c640 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b652eb700 0x561b652edb20 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=832b71e8a5db38cc server_cookie=f7cccb1a6d25157a in_seq=0 out_seq=0 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62b5ffb640 1 -- 172.21.15.3:0/3319767546 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561b64f89410 con 0x561b652eb700 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62b5ffb640 10 client.19443 handle_command_reply: tid=0 2024-02-09T23:01:58.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 client.19443 shutdown 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 msgr2=0x561b652d90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652d90f0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7f62c8036000 tx=0x7f62c80095c0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 shutdown_connections 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b652eb700 0x561b652edb20 crc :-1 s=READY pgs=299 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652d90f0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2ab0 0x561b652d8bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62b5ffb640 0 client.19443 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 --2- 172.21.15.3:0/3319767546 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652d8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 >> 172.21.15.3:0/3319767546 conn(0x561b652cc8c0 msgr2=0x561b652ce610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 shutdown_connections 2024-02-09T23:01:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 1 -- 172.21.15.3:0/3319767546 wait complete. 2024-02-09T23:01:58.356 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.345+0000 7f62d7440740 20 client.19443 trim_cache size 0 max 16384 2024-02-09T23:01:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 Processor -- start 2024-02-09T23:01:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 -- start start 2024-02-09T23:01:58.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652dada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652d8f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652db690 0x561b652d5910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b652f2d30 con 0x561b652d2ab0 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b652d6be0 con 0x561b652d2190 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62d7440740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b652cd240 con 0x561b652db690 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652d8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652d8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36614/0 (socket says 172.21.15.3:36614) 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2449593644 learned_addr learned my addr 172.21.15.3:0/2449593644 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62ce59c640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652dada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62ced9d640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652db690 0x561b652d5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2449593644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652db690 msgr2=0x561b652d5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652db690 0x561b652d5910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2449593644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 msgr2=0x561b652dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652dada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62cdd9b640 1 -- 172.21.15.3:0/2449593644 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561b651d3c70 con 0x561b652d2ab0 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62ce59c640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652dada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.362+0000 7f62ced9d640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652db690 0x561b652d5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.363+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652d8f10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004a00 tx=0x7f62c406de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0555ee3a83fa0eb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.363+0000 7f62b7fff640 1 -- 172.21.15.3:0/2449593644 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c409ae10 con 0x561b652d2ab0 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.363+0000 7f62b7fff640 1 -- 172.21.15.3:0/2449593644 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c409d070 con 0x561b652d2ab0 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.363+0000 7f62b7fff640 1 -- 172.21.15.3:0/2449593644 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c40cbdf0 con 0x561b652d2ab0 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.363+0000 7f62d7440740 1 -- 172.21.15.3:0/2449593644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 msgr2=0x561b652d8f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.363+0000 7f62d7440740 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652d8f10 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7f62c4004a00 tx=0x7f62c406de80 comp rx=0 tx=0).stop 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- 172.21.15.3:0/2449593644 shutdown_connections 2024-02-09T23:01:58.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652db690 0x561b652d5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652d2ab0 0x561b652d8f10 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 --2- 172.21.15.3:0/2449593644 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2190 0x561b652dada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- 172.21.15.3:0/2449593644 >> 172.21.15.3:0/2449593644 conn(0x561b652cc8c0 msgr2=0x561b652d3990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- 172.21.15.3:0/2449593644 shutdown_connections 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- 172.21.15.3:0/2449593644 wait complete. 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- start start 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 Processor -- start 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b6533c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2ab0 0x561b6533c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652e1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b652e2130 con 0x561b652db690 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b652e22a0 con 0x561b652d2ab0 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x561b652e2460 con 0x561b652d2190 2024-02-09T23:01:58.372 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62d7440740 10 client.0 _ll_register_callbacks cb 0x7f62c0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652e1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62ce59c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b6533c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62ced9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652e1a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36616/0 (socket says 172.21.15.3:36616) 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62ced9d640 1 -- 172.21.15.3:0/92451348 learned_addr learned my addr 172.21.15.3:0/92451348 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.364+0000 7f62cdd9b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2ab0 0x561b6533c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ce59c640 1 -- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2ab0 msgr2=0x561b6533c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ce59c640 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2ab0 0x561b6533c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ce59c640 1 -- 172.21.15.3:0/92451348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 msgr2=0x561b652e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ce59c640 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652e1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ce59c640 1 -- 172.21.15.3:0/92451348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561b651df210 con 0x561b652d2190 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ced9d640 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652e1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62ce59c640 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b6533c270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62c0011f80 tx=0x7f62c0009310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16058d92b50d7dbc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.373 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62b5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62b5ffb640 1 -- 172.21.15.3:0/92451348 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000f640 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62c000fc80 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62b5ffb640 1 -- 172.21.15.3:0/92451348 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f62c000f7a0 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62b5ffb640 1 -- 172.21.15.3:0/92451348 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62c000f980 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62d7440740 10 client.28444 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x561b65237390 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.365+0000 7f62b5ffb640 1 -- 172.21.15.3:0/92451348 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62c0038080 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62b5ffb640 1 -- 172.21.15.3:0/92451348 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f62c0005490 con 0x561b652d2190 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62d7440740 10 client.28444 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62d7440740 10 client.28444 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62d7440740 20 client.28444 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62d7440740 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b65344110 0x561b65346530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62d7440740 4 client.28444 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x561b64f757b0 con 0x561b65344110 2024-02-09T23:01:58.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b65344110 0x561b65346530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62cdd9b640 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b65344110 0x561b65346530 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f9a1b8ffea916e1 server_cookie=942edbde5010cd5c in_seq=0 out_seq=0 2024-02-09T23:01:58.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.366+0000 7f62b5ffb640 10 client.28444 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62b5ffb640 1 -- 172.21.15.3:0/92451348 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x561b64f757b0 con 0x561b65344110 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62b5ffb640 10 client.28444 handle_command_reply: tid=0 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 client.28444 shutdown 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 msgr2=0x561b6533c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b6533c270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f62c0011f80 tx=0x7f62c0009310 comp rx=0 tx=0).stop 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 shutdown_connections 2024-02-09T23:01:58.376 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x561b65344110 0x561b65346530 crc :-1 s=READY pgs=300 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x561b652d2ab0 0x561b6533c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x561b652db690 0x561b652e1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62b5ffb640 0 client.28444 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 --2- 172.21.15.3:0/92451348 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x561b652d2190 0x561b6533c270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 >> 172.21.15.3:0/92451348 conn(0x561b652cc8c0 msgr2=0x561b652cb090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 shutdown_connections 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.367+0000 7f62d7440740 1 -- 172.21.15.3:0/92451348 wait complete. 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d7440740 20 client.28444 trim_cache size 0 max 16384 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:01:58.377 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:01:58.378 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "7ac5e943-e7c6-4457-a453-568062717648", 2024-02-09T23:01:58.378 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:01:58.378 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:01:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62b80a2260 msgr2=0x7f62b80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62b80a2260 0x7f62b80a4720 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f62c8005cc0 tx=0x7f62c8005c50 comp rx=0 tx=0).stop 2024-02-09T23:01:58.378 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 msgr2=0x7f62d01022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d01022e0 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f62c400e9c0 tx=0x7f62c400ee80 comp rx=0 tx=0).stop 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 shutdown_connections 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62b80a2260 0x7f62b80a4720 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62d0102820 0x7f62d01a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62d01053e0 0x7f62d01022e0 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 --2- 172.21.15.3:0/2918178406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62d0100310 0x7f62d0101da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 >> 172.21.15.3:0/2918178406 conn(0x7f62d0077230 msgr2=0x7f62d00687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 shutdown_connections 2024-02-09T23:01:58.379 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.368+0000 7f62d486f640 1 -- 172.21.15.3:0/2918178406 wait complete. 2024-02-09T23:01:58.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:01:58.382 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:01:58.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 -- 172.21.15.3:0/1558534205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 msgr2=0x7faf08100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 --2- 172.21.15.3:0/1558534205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08100720 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7faf0400dd40 tx=0x7faf0402b980 comp rx=0 tx=0).stop 2024-02-09T23:01:58.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 -- 172.21.15.3:0/1558534205 shutdown_connections 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 --2- 172.21.15.3:0/1558534205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf08105410 0x7faf0810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 --2- 172.21.15.3:0/1558534205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf08100cf0 0x7faf08104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 --2- 172.21.15.3:0/1558534205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08100720 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 -- 172.21.15.3:0/1558534205 >> 172.21.15.3:0/1558534205 conn(0x7faf080772e0 msgr2=0x7faf080776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 -- 172.21.15.3:0/1558534205 shutdown_connections 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 -- 172.21.15.3:0/1558534205 wait complete. 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 Processor -- start 2024-02-09T23:01:58.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.491+0000 7faf10583640 1 -- start start 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08195700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf08105410 0x7faf08195c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf08199fc0 0x7faf0819a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0819a9b0 con 0x7faf08100340 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0819ab20 con 0x7faf08105410 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf0819ac90 con 0x7faf08199fc0 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08195700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08195700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36634/0 (socket says 172.21.15.3:36634) 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 -- 172.21.15.3:0/3408857637 learned_addr learned my addr 172.21.15.3:0/3408857637 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0db1e640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf08105410 0x7faf08195c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0eb20640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf08199fc0 0x7faf0819a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 -- 172.21.15.3:0/3408857637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf08199fc0 msgr2=0x7faf0819a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf08199fc0 0x7faf0819a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 -- 172.21.15.3:0/3408857637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf08105410 msgr2=0x7faf08195c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf08105410 0x7faf08195c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 -- 172.21.15.3:0/3408857637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000130b0 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf0e31f640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08195700 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7faf04006010 tx=0x7faf0400ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faeff7fe640 1 -- 172.21.15.3:0/3408857637 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0400be30 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faeff7fe640 1 -- 172.21.15.3:0/3408857637 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf0400b5d0 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0400d970 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.492+0000 7faeff7fe640 1 -- 172.21.15.3:0/3408857637 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf04043920 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.493+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf081a2560 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.495+0000 7faeff7fe640 1 -- 172.21.15.3:0/3408857637 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7faf04005320 con 0x7faf08100340 2024-02-09T23:01:58.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.495+0000 7faf13231740 1 Processor -- start 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 -- start start 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823232d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x56082323b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 0x56082323e300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faeff7fe640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faef40a2230 0x7faef40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608231edd00 con 0x560823232980 2024-02-09T23:01:58.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608231a22f0 con 0x56082323beb0 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf13231740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082319f820 con 0x560823233330 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 0x56082323e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 0x56082323e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:37690/0 (socket says 172.21.15.3:37690) 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf0db1e640 1 -- 172.21.15.3:0/358914972 learned_addr learned my addr 172.21.15.3:0/358914972 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf0eb20640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faef40a2230 0x7faef40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faeff7fe640 1 -- 172.21.15.3:0/3408857637 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7faf040efd20 con 0x7faf08100340 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf0e31f640 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x56082323b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.496+0000 7faf0eb20640 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823232d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0db1e640 1 -- 172.21.15.3:0/358914972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 msgr2=0x56082323b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0db1e640 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x56082323b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0db1e640 1 -- 172.21.15.3:0/358914972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 msgr2=0x560823232d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0db1e640 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823232d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0db1e640 1 -- 172.21.15.3:0/358914972 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56082315cff0 con 0x56082323beb0 2024-02-09T23:01:58.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0eb20640 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823232d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0db1e640 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 0x56082323e300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faef8009c60 tx=0x7faef8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60f2f36458c83298 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faf0eb20640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faef40a2230 0x7faef40a46f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf00011fd0 tx=0x7faf00011d10 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faefcbfe640 1 -- 172.21.15.3:0/358914972 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8004230 con 0x56082323beb0 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faefcbfe640 1 -- 172.21.15.3:0/358914972 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef8004390 con 0x56082323beb0 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.497+0000 7faefcbfe640 1 -- 172.21.15.3:0/358914972 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8019780 con 0x56082323beb0 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 -- 172.21.15.3:0/358914972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 msgr2=0x56082323e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 0x56082323e300 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7faef8009c60 tx=0x7faef8011810 comp rx=0 tx=0).stop 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 -- 172.21.15.3:0/358914972 shutdown_connections 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082323beb0 0x56082323e300 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x56082323b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 --2- 172.21.15.3:0/358914972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823232d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 -- 172.21.15.3:0/358914972 >> 172.21.15.3:0/358914972 conn(0x56082322d050 msgr2=0x56082322d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 -- 172.21.15.3:0/358914972 shutdown_connections 2024-02-09T23:01:58.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 -- 172.21.15.3:0/358914972 wait complete. 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 -- start start 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.498+0000 7faf13231740 1 Processor -- start 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560823232980 0x560823238db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x5608232392f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56082323beb0 0x560823239830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082319b000 con 0x56082323beb0 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560822e6e430 con 0x560823232980 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560822e2ae00 con 0x560823233330 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 10 client.0 _ll_register_callbacks cb 0x56082318f7f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0eb20640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56082323beb0 0x560823239830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0e31f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560823232980 0x560823238db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x5608232392f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0eb20640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56082323beb0 0x560823239830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36652/0 (socket says 172.21.15.3:36652) 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0eb20640 1 -- 172.21.15.3:0/1175684353 learned_addr learned my addr 172.21.15.3:0/1175684353 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 -- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560823232980 msgr2=0x560823238db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560823232980 0x560823238db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 -- 172.21.15.3:0/1175684353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56082323beb0 msgr2=0x560823239830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56082323beb0 0x560823239830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 -- 172.21.15.3:0/1175684353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef8009890 con 0x560823233330 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x5608232392f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faef8003f80 tx=0x7faef8005270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=394b9b1ec03c7208 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faef1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faef1ffb640 1 -- 172.21.15.3:0/1175684353 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8005660 con 0x560823233330 2024-02-09T23:01:58.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faef1ffb640 1 -- 172.21.15.3:0/1175684353 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7faef80035f0 con 0x560823233330 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.499+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560822e9a7e0 con 0x560823233330 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faef1ffb640 1 -- 172.21.15.3:0/1175684353 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7faef8004500 con 0x560823233330 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 10 client.28456 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x560823199680 con 0x560823233330 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faef1ffb640 1 -- 172.21.15.3:0/1175684353 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560822e9a7e0 con 0x560823233330 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faef1ffb640 1 -- 172.21.15.3:0/1175684353 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faef8004d50 con 0x560823233330 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 10 client.28456 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 10 client.28456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 20 client.28456 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082324c340 0x56082324e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 4 client.28456 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560822ed7720 con 0x56082324c340 2024-02-09T23:01:58.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.500+0000 7faf0e31f640 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082324c340 0x56082324e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.501+0000 7faf0e31f640 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082324c340 0x56082324e760 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c593ec87cecf10d2 server_cookie=b5fa028f8996f3d5 in_seq=0 out_seq=0 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.501+0000 7faef1ffb640 10 client.28456 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.501+0000 7faef1ffb640 1 -- 172.21.15.3:0/1175684353 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560822ed7720 con 0x56082324c340 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.501+0000 7faef1ffb640 10 client.28456 handle_command_reply: tid=0 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.501+0000 7faf13231740 1 client.28456 shutdown 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 msgr2=0x5608232392f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x5608232392f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7faef8003f80 tx=0x7faef8005270 comp rx=0 tx=0).stop 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 shutdown_connections 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082324c340 0x56082324e760 crc :-1 s=READY pgs=301 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56082323beb0 0x560823239830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x5608232392f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 --2- 172.21.15.3:0/1175684353 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560823232980 0x560823238db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faef1ffb640 0 client.28456 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 >> 172.21.15.3:0/1175684353 conn(0x56082322d050 msgr2=0x56082322eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 shutdown_connections 2024-02-09T23:01:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 1 -- 172.21.15.3:0/1175684353 wait complete. 2024-02-09T23:01:58.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.502+0000 7faf13231740 20 client.28456 trim_cache size 0 max 16384 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 Processor -- start 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 -- start start 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x56082323b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823237d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823239650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56082311ffd0 con 0x560823232980 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.518+0000 7faf13231740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560823253950 con 0x56082322d050 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf13231740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560823228910 con 0x560823233330 2024-02-09T23:01:58.531 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823237d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0e31f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x56082323b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0eb20640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823239650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823237d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36668/0 (socket says 172.21.15.3:36668) 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 -- 172.21.15.3:0/1049799951 learned_addr learned my addr 172.21.15.3:0/1049799951 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 -- 172.21.15.3:0/1049799951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 msgr2=0x560823239650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823239650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 -- 172.21.15.3:0/1049799951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 msgr2=0x56082323b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x56082323b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 -- 172.21.15.3:0/1049799951 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560823135b30 con 0x560823232980 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0eb20640 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823239650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0e31f640 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x56082323b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:01:58.532 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823237d90 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7faef8019540 tx=0x7faef8019480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfd5bb2dd2ad1c2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faef3fff640 1 -- 172.21.15.3:0/1049799951 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8011dc0 con 0x560823232980 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faef3fff640 1 -- 172.21.15.3:0/1049799951 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef8009ed0 con 0x560823232980 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.519+0000 7faef3fff640 1 -- 172.21.15.3:0/1049799951 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8003b00 con 0x560823232980 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 -- 172.21.15.3:0/1049799951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 msgr2=0x560823237d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823237d90 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7faef8019540 tx=0x7faef8019480 comp rx=0 tx=0).stop 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 -- 172.21.15.3:0/1049799951 shutdown_connections 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823239650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x560823237d90 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 --2- 172.21.15.3:0/1049799951 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x56082323b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 -- 172.21.15.3:0/1049799951 >> 172.21.15.3:0/1049799951 conn(0x560823233a80 msgr2=0x560823237680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 -- 172.21.15.3:0/1049799951 shutdown_connections 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 -- 172.21.15.3:0/1049799951 wait complete. 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 -- start start 2024-02-09T23:01:58.533 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.520+0000 7faf13231740 1 Processor -- start 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x560823243240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x5608232415e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823241b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560823242220 con 0x560823232980 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560823242390 con 0x56082322d050 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5608232a2850 con 0x560823233330 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 10 client.0 _ll_register_callbacks cb 0x7faef8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x5608232415e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x5608232415e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36672/0 (socket says 172.21.15.3:36672) 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 -- 172.21.15.3:0/1368399319 learned_addr learned my addr 172.21.15.3:0/1368399319 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:01:58.534 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0e31f640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x560823243240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0eb20640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823241b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 -- 172.21.15.3:0/1368399319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 msgr2=0x560823241b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823241b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 -- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 msgr2=0x560823243240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x560823243240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 -- 172.21.15.3:0/1368399319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf081a2560 con 0x560823232980 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf0db1e640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x5608232415e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7faef8011d10 tx=0x7faef8019940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ee694916ce9880 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faef1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faef1ffb640 1 -- 172.21.15.3:0/1368399319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faef8005250 con 0x560823232980 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faef1ffb640 1 -- 172.21.15.3:0/1368399319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7faef8000e40 con 0x560823232980 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.521+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faef8004390 con 0x560823232980 2024-02-09T23:01:58.535 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faef1ffb640 1 -- 172.21.15.3:0/1368399319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7faef80053b0 con 0x560823232980 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 10 client.19467 fetch_fsmap learned FSMap version 49 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x7faf0001be30 con 0x560823232980 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faef1ffb640 1 -- 172.21.15.3:0/1368399319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faef8004390 con 0x560823232980 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faef1ffb640 1 -- 172.21.15.3:0/1368399319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faef8004760 con 0x560823232980 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 10 client.19467 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 10 client.19467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 20 client.19467 populate_metadata read hostname 'smithi003' 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082323beb0 0x5608232a5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 4 client.19467 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560822f26890 con 0x56082323beb0 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.522+0000 7faf0e31f640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082323beb0 0x5608232a5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:01:58.536 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf0e31f640 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082323beb0 0x5608232a5d00 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=60b8f6f89bda38ca server_cookie=eb986c1c196557e4 in_seq=0 out_seq=0 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faef1ffb640 10 client.19467 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faef1ffb640 1 -- 172.21.15.3:0/1368399319 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560822f26890 con 0x56082323beb0 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faef1ffb640 10 client.19467 handle_command_reply: tid=0 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 client.19467 shutdown 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 msgr2=0x5608232415e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x5608232415e0 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7faef8011d10 tx=0x7faef8019940 comp rx=0 tx=0).stop 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 shutdown_connections 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56082323beb0 0x5608232a5d00 crc :-1 s=READY pgs=302 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560823233330 0x560823241b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560823232980 0x5608232415e0 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faef1ffb640 0 client.19467 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 --2- 172.21.15.3:0/1368399319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56082322d050 0x560823243240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.537 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 >> 172.21.15.3:0/1368399319 conn(0x560823233a80 msgr2=0x56082322c1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 shutdown_connections 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 1 -- 172.21.15.3:0/1368399319 wait complete. 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.523+0000 7faf13231740 20 client.19467 trim_cache size 0 max 16384 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout: "7ac5e943-e7c6-4457-a453-568062717648": { 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "7ac5e943-e7c6-4457-a453-568062717648", 2024-02-09T23:01:58.538 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:01:58.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:01:58.539 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:01:58.539 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:01:58.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faef40a2230 msgr2=0x7faef40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faef40a2230 0x7faef40a46f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7faf00011fd0 tx=0x7faf00011d10 comp rx=0 tx=0).stop 2024-02-09T23:01:58.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 msgr2=0x7faf08195700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:01:58.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08195700 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7faf04006010 tx=0x7faf0400ba30 comp rx=0 tx=0).stop 2024-02-09T23:01:58.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 shutdown_connections 2024-02-09T23:01:58.540 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faef40a2230 0x7faef40a46f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faf08105410 0x7faf08195c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faf08199fc0 0x7faf0819a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 --2- 172.21.15.3:0/3408857637 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faf08100340 0x7faf08195700 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:01:58.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 >> 172.21.15.3:0/3408857637 conn(0x7faf080772e0 msgr2=0x7faf080fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:01:58.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 shutdown_connections 2024-02-09T23:01:58.541 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:01:58.524+0000 7faf10583640 1 -- 172.21.15.3:0/3408857637 wait complete. 2024-02-09T23:01:58.541 INFO:tasks.cephfs.filesystem:scrub status for tag:7ac5e943-e7c6-4457-a453-568062717648 - {'path': '/', 'tag': '7ac5e943-e7c6-4457-a453-568062717648', 'options': 'recursive,force'} 2024-02-09T23:01:58.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 361455 3.23 MB/sec execute 472 sec latency 832.055 ms 2024-02-09T23:01:59.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 362899 3.24 MB/sec execute 473 sec latency 13.504 ms 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:01:59 smithi003 ceph-mon[32678]: pgmap v335: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-02-09T23:01:59.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:01:59.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:01:59.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:01:59 smithi064 ceph-mon[36638]: pgmap v335: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:01:59.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:01:59 smithi089 ceph-mon[36665]: pgmap v335: 97 pgs: 97 active+clean; 140 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.7 MiB/s wr, 159 op/s 2024-02-09T23:02:00.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 363483 3.24 MB/sec execute 474 sec latency 521.549 ms 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.379+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1529458317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff8101460 msgr2=0x7f5ff8101840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.379+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1529458317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff8101460 0x7f5ff8101840 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0009930 tx=0x7f5ff002bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1529458317 shutdown_connections 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1529458317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ff8101e10 0x7f5ff8105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1529458317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ff8106530 0x7f5ff810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1529458317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff8101460 0x7f5ff8101840 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1529458317 >> 172.21.15.3:0/1529458317 conn(0x7f5ff80fccc0 msgr2=0x7f5ff80ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1529458317 shutdown_connections 2024-02-09T23:02:01.383 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1529458317 wait complete. 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.380+0000 7f5ffd1c7640 1 Processor -- start 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 -- start start 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ff8101460 0x7f5ff819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ff8106530 0x7f5ff819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 0x7f5ff8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff81981a0 con 0x7f5ff81976d0 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff8198310 con 0x7f5ff8106530 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ffd1c7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ff81984d0 con 0x7f5ff8101460 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 0x7f5ff8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 0x7f5ff8197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36686/0 (socket says 172.21.15.3:36686) 2024-02-09T23:02:01.386 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 -- 172.21.15.3:0/1861924872 learned_addr learned my addr 172.21.15.3:0/1861924872 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ff8101460 0x7f5ff819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ff8106530 0x7f5ff819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 -- 172.21.15.3:0/1861924872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ff8101460 msgr2=0x7f5ff819d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ff8101460 0x7f5ff819d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 -- 172.21.15.3:0/1861924872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ff8106530 msgr2=0x7f5ff819db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ff8106530 0x7f5ff819db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.381+0000 7f5ff759e640 1 -- 172.21.15.3:0/1861924872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff0009560 con 0x7f5ff81976d0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.382+0000 7f5ff759e640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 0x7f5ff8197b50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800eaa0 tx=0x7f5fe800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.382+0000 7f5feffff640 1 -- 172.21.15.3:0/1861924872 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe800cdb0 con 0x7f5ff81976d0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.382+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff81a4470 con 0x7f5ff81976d0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.382+0000 7f5feffff640 1 -- 172.21.15.3:0/1861924872 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe8005020 con 0x7f5ff81976d0 2024-02-09T23:02:01.387 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.382+0000 7f5feffff640 1 -- 172.21.15.3:0/1861924872 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe80116b0 con 0x7f5ff81976d0 2024-02-09T23:02:01.388 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.382+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff81a4950 con 0x7f5ff81976d0 2024-02-09T23:02:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.385+0000 7f5fffd99740 1 Processor -- start 2024-02-09T23:02:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.386+0000 7f5feffff640 1 -- 172.21.15.3:0/1861924872 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5fe8005520 con 0x7f5ff81976d0 2024-02-09T23:02:01.389 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 -- start start 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa80000b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7ffed480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa7ffedf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7ff61a00 con 0x55fa7ffedaa0 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7ff5ef30 con 0x55fa800010e0 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.387+0000 7f5fffd99740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7ff5a710 con 0x55fa80000730 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5ff659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa7ffedf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5ff659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa7ffedf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36692/0 (socket says 172.21.15.3:36692) 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5ff659c640 1 -- 172.21.15.3:0/3872684603 learned_addr learned my addr 172.21.15.3:0/3872684603 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7ffed480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.391 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5feffff640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5fe40a2220 0x7f5fe40a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.392 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5ff659c640 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa80000b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.388+0000 7f5feffff640 1 -- 172.21.15.3:0/1861924872 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5fe80ce110 con 0x7f5ff81976d0 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff659c640 1 -- 172.21.15.3:0/3872684603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 msgr2=0x55fa80000b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff659c640 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa80000b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff659c640 1 -- 172.21.15.3:0/3872684603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 msgr2=0x55fa7ffed480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff659c640 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7ffed480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff659c640 1 -- 172.21.15.3:0/3872684603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa7ff1f3b0 con 0x55fa7ffedaa0 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff759e640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5fe40a2220 0x7f5fe40a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.389+0000 7f5ff659c640 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa7ffedf00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0012530 tx=0x7f5fe000b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5c14060edd2d24 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:01.393 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.390+0000 7f5fed7fa640 1 -- 172.21.15.3:0/3872684603 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe001d5f0 con 0x55fa7ffedaa0 2024-02-09T23:02:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.390+0000 7f5fed7fa640 1 -- 172.21.15.3:0/3872684603 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe0010910 con 0x55fa7ffedaa0 2024-02-09T23:02:01.394 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.390+0000 7f5fed7fa640 1 -- 172.21.15.3:0/3872684603 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe001c890 con 0x55fa7ffedaa0 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 -- 172.21.15.3:0/3872684603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 msgr2=0x55fa7ffedf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5ff759e640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5fe40a2220 0x7f5fe40a46e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5fe80cb9b0 tx=0x7f5fe800cf70 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa7ffedf00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0012530 tx=0x7f5fe000b9b0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 -- 172.21.15.3:0/3872684603 shutdown_connections 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7ffed480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa7ffedf00 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 --2- 172.21.15.3:0/3872684603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa80000b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 -- 172.21.15.3:0/3872684603 >> 172.21.15.3:0/3872684603 conn(0x55fa7ffeff40 msgr2=0x55fa7fff0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 -- 172.21.15.3:0/3872684603 shutdown_connections 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 -- 172.21.15.3:0/3872684603 wait complete. 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 -- start start 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.391+0000 7f5fffd99740 1 Processor -- start 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa800047b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa80000730 0x55fa80004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa800010e0 0x55fa80005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7feffb90 con 0x55fa7ffedaa0 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7fc2e250 con 0x55fa80000730 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7fee0200 con 0x55fa800010e0 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa800047b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa800047b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36696/0 (socket says 172.21.15.3:36696) 2024-02-09T23:02:01.399 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 -- 172.21.15.3:0/4172009633 learned_addr learned my addr 172.21.15.3:0/4172009633 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5fffd99740 10 client.0 _ll_register_callbacks cb 0x55fa7ff165d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa80000730 0x55fa80004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa800010e0 0x55fa80005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 -- 172.21.15.3:0/4172009633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa800010e0 msgr2=0x55fa80005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa800010e0 0x55fa80005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 -- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa80000730 msgr2=0x55fa80004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.392+0000 7f5ff659c640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa80000730 0x55fa80004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5ff659c640 1 -- 172.21.15.3:0/4172009633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe000bd70 con 0x55fa7ffedaa0 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5ff659c640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa800047b0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5fe00057a0 tx=0x7f5fe00056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7076908b2e7c721 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fde7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fde7fc640 1 -- 172.21.15.3:0/4172009633 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0010dd0 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fde7fc640 1 -- 172.21.15.3:0/4172009633 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5fe0000e40 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fa7ff80ee0 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fde7fc640 1 -- 172.21.15.3:0/4172009633 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fe001c610 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fffd99740 10 client.19479 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55fa7ff58d90 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.393+0000 7f5fde7fc640 1 -- 172.21.15.3:0/4172009633 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe0038020 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fde7fc640 1 -- 172.21.15.3:0/4172009633 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5fe002b800 con 0x55fa7ffedaa0 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fffd99740 10 client.19479 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fffd99740 10 client.19479 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fffd99740 20 client.19479 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fffd99740 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa800150a0 0x55fa80017490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fffd99740 4 client.19479 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fa7fc5a600 con 0x55fa800150a0 2024-02-09T23:02:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa800150a0 0x55fa80017490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa800150a0 0x55fa80017490 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0f241757d89c557 server_cookie=ead10d875a5e02ab in_seq=0 out_seq=0 2024-02-09T23:02:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.394+0000 7f5fde7fc640 10 client.19479 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fde7fc640 1 -- 172.21.15.3:0/4172009633 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fa7fc5a600 con 0x55fa800150a0 2024-02-09T23:02:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fde7fc640 10 client.19479 handle_command_reply: tid=0 2024-02-09T23:02:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 client.19479 shutdown 2024-02-09T23:02:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 msgr2=0x55fa800047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.403 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa800047b0 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f5fe00057a0 tx=0x7f5fe00056d0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 shutdown_connections 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa800150a0 0x55fa80017490 crc :-1 s=READY pgs=303 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa80000730 0x55fa80004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa800010e0 0x55fa80005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fde7fc640 0 client.19479 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 --2- 172.21.15.3:0/4172009633 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa800047b0 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 >> 172.21.15.3:0/4172009633 conn(0x55fa7ffeff40 msgr2=0x55fa7ffef6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 shutdown_connections 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 1 -- 172.21.15.3:0/4172009633 wait complete. 2024-02-09T23:02:01.404 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.396+0000 7f5fffd99740 20 client.19479 trim_cache size 0 max 16384 2024-02-09T23:02:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 Processor -- start 2024-02-09T23:02:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 -- start start 2024-02-09T23:02:01.417 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa80006ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa8000bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa8000c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa80011de0 con 0x55fa7ffedaa0 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa800045a0 con 0x55fa800010e0 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5fffd99740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa7ffe8940 con 0x55fa80000730 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa80006ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa80006ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36712/0 (socket says 172.21.15.3:36712) 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 -- 172.21.15.3:0/1072236513 learned_addr learned my addr 172.21.15.3:0/1072236513 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 -- 172.21.15.3:0/1072236513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 msgr2=0x55fa8000bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa8000bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 -- 172.21.15.3:0/1072236513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 msgr2=0x55fa8000c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.418 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa8000c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.410+0000 7f5ff659c640 1 -- 172.21.15.3:0/1072236513 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa7fef5820 con 0x55fa7ffedaa0 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa80006ee0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f5fe002bd80 tx=0x7f5fe000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8e40d48c4bd9de server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fecff9640 1 -- 172.21.15.3:0/1072236513 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe00057d0 con 0x55fa7ffedaa0 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fecff9640 1 -- 172.21.15.3:0/1072236513 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe0038070 con 0x55fa7ffedaa0 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fecff9640 1 -- 172.21.15.3:0/1072236513 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe002b800 con 0x55fa7ffedaa0 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 -- 172.21.15.3:0/1072236513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 msgr2=0x55fa80006ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa80006ee0 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f5fe002bd80 tx=0x7f5fe000bf90 comp rx=0 tx=0).stop 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 -- 172.21.15.3:0/1072236513 shutdown_connections 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa8000c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa80000730 0x55fa8000bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1072236513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa7ffedaa0 0x55fa80006ee0 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 -- 172.21.15.3:0/1072236513 >> 172.21.15.3:0/1072236513 conn(0x55fa7ffeff40 msgr2=0x55fa7fffe880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 -- 172.21.15.3:0/1072236513 shutdown_connections 2024-02-09T23:02:01.419 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.411+0000 7f5fffd99740 1 -- 172.21.15.3:0/1072236513 wait complete. 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 -- start start 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 Processor -- start 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa7ffedaa0 0x55fa7fffc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 0x55fa7fffce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7fffd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa8006b610 con 0x55fa80000730 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa8006b780 con 0x55fa800010e0 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa8006b940 con 0x55fa7ffedaa0 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5fffd99740 10 client.0 _ll_register_callbacks cb 0x7f5ff0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 0x55fa7fffce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 0x55fa7fffce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36728/0 (socket says 172.21.15.3:36728) 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 -- 172.21.15.3:0/1209001920 learned_addr learned my addr 172.21.15.3:0/1209001920 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:01.420 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa7ffedaa0 0x55fa7fffc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7fffd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 -- 172.21.15.3:0/1209001920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa7ffedaa0 msgr2=0x55fa7fffc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa7ffedaa0 0x55fa7fffc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 -- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 msgr2=0x55fa7fffd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.412+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7fffd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5ff6d9d640 1 -- 172.21.15.3:0/1209001920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa7ff58d90 con 0x55fa80000730 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5ff6d9d640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 0x55fa7fffce30 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0036040 tx=0x7f5ff00048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7728f03ee4c771af server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fde7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fde7fc640 1 -- 172.21.15.3:0/1209001920 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ff0048df0 con 0x55fa80000730 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fde7fc640 1 -- 172.21.15.3:0/1209001920 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5ff0045040 con 0x55fa80000730 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fe0010910 con 0x55fa80000730 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fde7fc640 1 -- 172.21.15.3:0/1209001920 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ff0004040 con 0x55fa80000730 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 10 client.19485 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:01.421 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55fa7ff1b6f0 con 0x55fa80000730 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fde7fc640 1 -- 172.21.15.3:0/1209001920 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fe0010910 con 0x55fa80000730 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fde7fc640 1 -- 172.21.15.3:0/1209001920 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5ff003c040 con 0x55fa80000730 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 10 client.19485 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 10 client.19485 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 20 client.19485 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa8006d1a0 0x55fa8006f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 4 client.19485 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.413+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55fa7fcab1b0 con 0x55fa8006d1a0 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.414+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa8006d1a0 0x55fa8006f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.414+0000 7f5fde7fc640 10 client.19485 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:01.422 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.414+0000 7f5ff659c640 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa8006d1a0 0x55fa8006f5c0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35577c9e300fdb55 server_cookie=c187f6a2048587d5 in_seq=0 out_seq=0 2024-02-09T23:02:01.463 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.459+0000 7f5fde7fc640 1 -- 172.21.15.3:0/1209001920 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55fa7fcab1b0 con 0x55fa8006d1a0 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.459+0000 7f5fde7fc640 10 client.19485 handle_command_reply: tid=0 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 client.19485 shutdown 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 msgr2=0x55fa7fffce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 0x55fa7fffce30 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f5ff0036040 tx=0x7f5ff00048b0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 shutdown_connections 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55fa8006d1a0 0x55fa8006f5c0 crc :-1 s=READY pgs=304 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fde7fc640 0 client.19485 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55fa800010e0 0x55fa7fffd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55fa80000730 0x55fa7fffce30 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 --2- 172.21.15.3:0/1209001920 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55fa7ffedaa0 0x55fa7fffc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 >> 172.21.15.3:0/1209001920 conn(0x55fa7ffeff40 msgr2=0x55fa7ffff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 shutdown_connections 2024-02-09T23:02:01.464 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 1 -- 172.21.15.3:0/1209001920 wait complete. 2024-02-09T23:02:01.465 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.460+0000 7f5fffd99740 20 client.19485 trim_cache size 0 max 16384 2024-02-09T23:02:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:02:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:02:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:02:01.467 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5fe40a2220 msgr2=0x7f5fe40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5fe40a2220 0x7f5fe40a46e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5fe80cb9b0 tx=0x7f5fe800cf70 comp rx=0 tx=0).stop 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 msgr2=0x7f5ff8197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 0x7f5ff8197b50 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5fe800eaa0 tx=0x7f5fe800ef60 comp rx=0 tx=0).stop 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 shutdown_connections 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5fe40a2220 0x7f5fe40a46e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5ff8106530 0x7f5ff819db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.468 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5ff81976d0 0x7f5ff8197b50 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 --2- 172.21.15.3:0/1861924872 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5ff8101460 0x7f5ff819d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 >> 172.21.15.3:0/1861924872 conn(0x7f5ff80fccc0 msgr2=0x7f5ff80fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 shutdown_connections 2024-02-09T23:02:01.469 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:01.461+0000 7f5ffd1c7640 1 -- 172.21.15.3:0/1861924872 wait complete. 2024-02-09T23:02:01.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 364052 3.23 MB/sec execute 475 sec latency 1297.916 ms 2024-02-09T23:02:01.814 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:01 smithi089 ceph-mon[36665]: pgmap v336: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 266 op/s 2024-02-09T23:02:01.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:01 smithi003 ceph-mon[32678]: pgmap v336: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 266 op/s 2024-02-09T23:02:01.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:01 smithi064 ceph-mon[36638]: pgmap v336: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.7 MiB/s wr, 266 op/s 2024-02-09T23:02:02.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 365429 3.25 MB/sec execute 476 sec latency 31.770 ms 2024-02-09T23:02:03.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 366725 3.26 MB/sec execute 477 sec latency 18.451 ms 2024-02-09T23:02:03.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:03 smithi003 ceph-mon[32678]: pgmap v337: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-02-09T23:02:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:03 smithi003 ceph-mon[32678]: osdmap e100: 12 total, 12 up, 12 in 2024-02-09T23:02:03.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:03 smithi064 ceph-mon[36638]: pgmap v337: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-02-09T23:02:03.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:03 smithi064 ceph-mon[36638]: osdmap e100: 12 total, 12 up, 12 in 2024-02-09T23:02:03.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:03 smithi089 ceph-mon[36665]: pgmap v337: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 169 op/s 2024-02-09T23:02:03.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:03 smithi089 ceph-mon[36665]: osdmap e100: 12 total, 12 up, 12 in 2024-02-09T23:02:04.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 367032 3.25 MB/sec execute 478 sec latency 798.758 ms 2024-02-09T23:02:05.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 368326 3.26 MB/sec execute 479 sec latency 1239.878 ms 2024-02-09T23:02:06.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:06 smithi003 ceph-mon[32678]: pgmap v339: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 203 op/s 2024-02-09T23:02:06.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 369605 3.27 MB/sec execute 480 sec latency 42.969 ms 2024-02-09T23:02:06.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:06 smithi064 ceph-mon[36638]: pgmap v339: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 203 op/s 2024-02-09T23:02:06.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:06 smithi089 ceph-mon[36665]: pgmap v339: 97 pgs: 97 active+clean; 145 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 2.1 MiB/s wr, 203 op/s 2024-02-09T23:02:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:07.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:07 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:07.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 370575 3.27 MB/sec execute 481 sec latency 410.739 ms 2024-02-09T23:02:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:07 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:07.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:07 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:08.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:08 smithi003 ceph-mon[32678]: pgmap v340: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 258 op/s 2024-02-09T23:02:08.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:08 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:08.662 INFO:tasks.workunit.client.0.smithi003.stdout: 1 372574 3.29 MB/sec execute 482 sec latency 536.622 ms 2024-02-09T23:02:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:08 smithi064 ceph-mon[36638]: pgmap v340: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 258 op/s 2024-02-09T23:02:08.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:08 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:08 smithi089 ceph-mon[36665]: pgmap v340: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 258 op/s 2024-02-09T23:02:08.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:08 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:09.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 373834 3.30 MB/sec execute 483 sec latency 18.111 ms 2024-02-09T23:02:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:10 smithi003 ceph-mon[32678]: pgmap v341: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 258 op/s 2024-02-09T23:02:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:10 smithi003 ceph-mon[32678]: osdmap e101: 12 total, 12 up, 12 in 2024-02-09T23:02:10.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 374123 3.29 MB/sec execute 484 sec latency 676.165 ms 2024-02-09T23:02:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:10 smithi064 ceph-mon[36638]: pgmap v341: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 258 op/s 2024-02-09T23:02:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:10 smithi064 ceph-mon[36638]: osdmap e101: 12 total, 12 up, 12 in 2024-02-09T23:02:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:10 smithi089 ceph-mon[36665]: pgmap v341: 97 pgs: 97 active+clean; 144 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.0 MiB/s wr, 258 op/s 2024-02-09T23:02:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:10 smithi089 ceph-mon[36665]: osdmap e101: 12 total, 12 up, 12 in 2024-02-09T23:02:11.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 374123 3.28 MB/sec execute 485 sec latency 1676.255 ms 2024-02-09T23:02:12.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:12 smithi003 ceph-mon[32678]: pgmap v343: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-02-09T23:02:12.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 376054 3.30 MB/sec execute 486 sec latency 1703.123 ms 2024-02-09T23:02:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:12 smithi064 ceph-mon[36638]: pgmap v343: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-02-09T23:02:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:12 smithi089 ceph-mon[36665]: pgmap v343: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-02-09T23:02:13.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 377154 3.30 MB/sec execute 487 sec latency 53.406 ms 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 -- 172.21.15.3:0/1239388067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb00696f0 msgr2=0x7f5fb0105e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 --2- 172.21.15.3:0/1239388067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb00696f0 0x7f5fb0105e30 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009930 tx=0x7f5fa402bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 -- 172.21.15.3:0/1239388067 shutdown_connections 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 --2- 172.21.15.3:0/1239388067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0106510 0x7f5fb010e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 --2- 172.21.15.3:0/1239388067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb00696f0 0x7f5fb0105e30 unknown :-1 s=CLOSED pgs=1592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 --2- 172.21.15.3:0/1239388067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb0068d40 0x7f5fb0069120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 -- 172.21.15.3:0/1239388067 >> 172.21.15.3:0/1239388067 conn(0x7f5fb0077220 msgr2=0x7f5fb0077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 -- 172.21.15.3:0/1239388067 shutdown_connections 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.555+0000 7f5fb506f640 1 -- 172.21.15.3:0/1239388067 wait complete. 2024-02-09T23:02:14.559 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 Processor -- start 2024-02-09T23:02:14.560 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 -- start start 2024-02-09T23:02:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb0068d40 0x7f5fb01000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.561 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 0x7f5fb0100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0100b70 0x7f5fb01a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01a6070 con 0x7f5fb0106510 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01a61e0 con 0x7f5fb0100b70 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.556+0000 7f5fb506f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01a63a0 con 0x7f5fb0068d40 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 0x7f5fb0100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 0x7f5fb0100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45520/0 (socket says 172.21.15.3:45520) 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 -- 172.21.15.3:0/239959652 learned_addr learned my addr 172.21.15.3:0/239959652 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5faed9d640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb0068d40 0x7f5fb01000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5faf59e640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0100b70 0x7f5fb01a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 -- 172.21.15.3:0/239959652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb0068d40 msgr2=0x7f5fb01000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb0068d40 0x7f5fb01000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 -- 172.21.15.3:0/239959652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0100b70 msgr2=0x7f5fb01a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0100b70 0x7f5fb01a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 -- 172.21.15.3:0/239959652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8009840 con 0x7f5fb0106510 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5faf59e640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0100b70 0x7f5fb01a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fae59c640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 0x7f5fb0100630 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009230 tx=0x7f5fa402c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5f97fff640 1 -- 172.21.15.3:0/239959652 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4004350 con 0x7f5fb0106510 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.557+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fa4009560 con 0x7f5fb0106510 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.558+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb01a69f0 con 0x7f5fb0106510 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.558+0000 7f5f97fff640 1 -- 172.21.15.3:0/239959652 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa40044b0 con 0x7f5fb0106510 2024-02-09T23:02:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.558+0000 7f5f97fff640 1 -- 172.21.15.3:0/239959652 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa4003da0 con 0x7f5fb0106510 2024-02-09T23:02:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.560+0000 7f5f97fff640 1 -- 172.21.15.3:0/239959652 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5fa4004df0 con 0x7f5fb0106510 2024-02-09T23:02:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.560+0000 7f5fb7c42740 1 Processor -- start 2024-02-09T23:02:14.570 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.560+0000 7f5fb7c42740 1 -- start start 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.560+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b34490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.560+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b3d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901aa2a00 con 0x55c901b3d7a0 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901a9ff30 con 0x55c901b340b0 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901a9b710 con 0x55c901b349d0 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5f97fff640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f980a2250 0x7f5f980a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b3d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45528/0 (socket says 172.21.15.3:45528) 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fae59c640 1 -- 172.21.15.3:0/2151133488 learned_addr learned my addr 172.21.15.3:0/2151133488 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faf59e640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f980a2250 0x7f5f980a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.571 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faf59e640 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b34490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5f97fff640 1 -- 172.21.15.3:0/239959652 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5fa40f0060 con 0x7f5fb0106510 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 -- 172.21.15.3:0/2151133488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 msgr2=0x55c901b34490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b34490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 -- 172.21.15.3:0/2151133488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 msgr2=0x55c901b3fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 -- 172.21.15.3:0/2151133488 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c901a603b0 con 0x55c901b349d0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5fae59c640 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faf59e640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f980a2250 0x7f5f980a4710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8005fd0 tx=0x7f5fa8009d30 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.561+0000 7f5faed9d640 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b3d260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009c00 tx=0x7f5fa0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c74c395d9d8a6fa server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5f957fa640 1 -- 172.21.15.3:0/2151133488 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0004230 con 0x55c901b349d0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5f957fa640 1 -- 172.21.15.3:0/2151133488 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa0009e70 con 0x55c901b349d0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5f957fa640 1 -- 172.21.15.3:0/2151133488 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0004d00 con 0x55c901b349d0 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 -- 172.21.15.3:0/2151133488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 msgr2=0x55c901b3d260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.572 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b3d260 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5fa0009c00 tx=0x7f5fa0011810 comp rx=0 tx=0).stop 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 -- 172.21.15.3:0/2151133488 shutdown_connections 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b3d260 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/2151133488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b34490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 -- 172.21.15.3:0/2151133488 >> 172.21.15.3:0/2151133488 conn(0x55c901b2e9d0 msgr2=0x55c901b2edb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 -- 172.21.15.3:0/2151133488 shutdown_connections 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.562+0000 7f5fb7c42740 1 -- 172.21.15.3:0/2151133488 wait complete. 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 -- start start 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 Processor -- start 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3a770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.573 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b349d0 0x55c901b3acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901a40b90 con 0x55c901b3d7a0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c90176f250 con 0x55c901b349d0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901a21200 con 0x55c901b340b0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fb7c42740 10 client.0 _ll_register_callbacks cb 0x55c901a575d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45530/0 (socket says 172.21.15.3:45530) 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 -- 172.21.15.3:0/1661196722 learned_addr learned my addr 172.21.15.3:0/1661196722 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5fae59c640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b349d0 0x55c901b3acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faed9d640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 -- 172.21.15.3:0/1661196722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 msgr2=0x55c901b3a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.574 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 -- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b349d0 msgr2=0x55c901b3acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b349d0 0x55c901b3acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faf59e640 1 -- 172.21.15.3:0/1661196722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa0009830 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.563+0000 7f5faed9d640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5faf59e640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3b1f0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8003ba0 tx=0x7f5fa8009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52a425fc62e48bdb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5f92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5f92ff5640 1 -- 172.21.15.3:0/1661196722 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8017030 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5f92ff5640 1 -- 172.21.15.3:0/1661196722 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5fa80047d0 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c901ac1ee0 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5f92ff5640 1 -- 172.21.15.3:0/1661196722 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fa8029440 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 10 client.19497 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55c901a99d90 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5f92ff5640 1 -- 172.21.15.3:0/1661196722 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa8035080 con 0x55c901b3d7a0 2024-02-09T23:02:14.575 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5f92ff5640 1 -- 172.21.15.3:0/1661196722 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5fa802f030 con 0x55c901b3d7a0 2024-02-09T23:02:14.576 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 10 client.19497 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 10 client.19497 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 20 client.19497 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901b4d9d0 0x55c901b4fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 4 client.19497 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.564+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c90179b600 con 0x55c901b4d9d0 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.565+0000 7f5faed9d640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901b4d9d0 0x55c901b4fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.565+0000 7f5f92ff5640 10 client.19497 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:14.579 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.565+0000 7f5faed9d640 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901b4d9d0 0x55c901b4fdf0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f408e67718a7ee7 server_cookie=f16f08519b5de5d8 in_seq=0 out_seq=0 2024-02-09T23:02:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5f92ff5640 1 -- 172.21.15.3:0/1661196722 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c90179b600 con 0x55c901b4d9d0 2024-02-09T23:02:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5f92ff5640 10 client.19497 handle_command_reply: tid=0 2024-02-09T23:02:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 client.19497 shutdown 2024-02-09T23:02:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 msgr2=0x55c901b3b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3b1f0 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8003ba0 tx=0x7f5fa8009500 comp rx=0 tx=0).stop 2024-02-09T23:02:14.580 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 shutdown_connections 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901b4d9d0 0x55c901b4fdf0 crc :-1 s=READY pgs=305 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b349d0 0x55c901b3acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b3b1f0 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1661196722 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5f92ff5640 0 client.19497 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 >> 172.21.15.3:0/1661196722 conn(0x55c901b2e9d0 msgr2=0x55c901b30720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 shutdown_connections 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1661196722 wait complete. 2024-02-09T23:02:14.581 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.566+0000 7f5fb7c42740 20 client.19497 trim_cache size 0 max 16384 2024-02-09T23:02:14.586 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 Processor -- start 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 -- start start 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 0x55c901b3b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b3d7a0 0x55c901b37a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901b2f9d0 con 0x55c901b349d0 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901b4a450 con 0x55c901b3d7a0 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901b338a0 con 0x55c901b340b0 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 0x55c901b3b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 0x55c901b3b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45536/0 (socket says 172.21.15.3:45536) 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5fae59c640 1 -- 172.21.15.3:0/1670030458 learned_addr learned my addr 172.21.15.3:0/1670030458 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5faed9d640 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.579+0000 7f5faf59e640 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b3d7a0 0x55c901b37a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.587 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fae59c640 1 -- 172.21.15.3:0/1670030458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 msgr2=0x55c901b3cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fae59c640 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fae59c640 1 -- 172.21.15.3:0/1670030458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b3d7a0 msgr2=0x55c901b37a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fae59c640 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b3d7a0 0x55c901b37a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fae59c640 1 -- 172.21.15.3:0/1670030458 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c901a36820 con 0x55c901b349d0 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fae59c640 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 0x55c901b3b010 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006010 tx=0x7f5fa4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a46d527ff8d40eda server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5f94ff9640 1 -- 172.21.15.3:0/1670030458 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40bd260 con 0x55c901b349d0 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5f94ff9640 1 -- 172.21.15.3:0/1670030458 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa40bde90 con 0x55c901b349d0 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5f94ff9640 1 -- 172.21.15.3:0/1670030458 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa40baef0 con 0x55c901b349d0 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1670030458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 msgr2=0x55c901b3b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 0x55c901b3b010 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4006010 tx=0x7f5fa4000bc0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1670030458 shutdown_connections 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b3d7a0 0x55c901b37a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b349d0 0x55c901b3b010 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.588 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/1670030458 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b340b0 0x55c901b3cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1670030458 >> 172.21.15.3:0/1670030458 conn(0x55c901b2e9d0 msgr2=0x55c901b39080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1670030458 shutdown_connections 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.580+0000 7f5fb7c42740 1 -- 172.21.15.3:0/1670030458 wait complete. 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 -- start start 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 Processor -- start 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b42f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b43480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b439c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901bac710 con 0x55c901b3d7a0 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901bac850 con 0x55c901b340b0 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c901baca10 con 0x55c901b349d0 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5fb7c42740 10 client.0 _ll_register_callbacks cb 0x7f5fa8012a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:14.589 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b439c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b439c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45548/0 (socket says 172.21.15.3:45548) 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 -- 172.21.15.3:0/3252540714 learned_addr learned my addr 172.21.15.3:0/3252540714 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faed9d640 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b42f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 -- 172.21.15.3:0/3252540714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 msgr2=0x55c901b43480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b43480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 -- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 msgr2=0x55c901b42f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b42f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.581+0000 7f5faf59e640 1 -- 172.21.15.3:0/3252540714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fa4009560 con 0x55c901b3d7a0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5faf59e640 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b439c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5fa801e630 tx=0x7f5fa8010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4a7bd10ee3f1000 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5f92ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5f92ff5640 1 -- 172.21.15.3:0/3252540714 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8017070 con 0x55c901b3d7a0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5f92ff5640 1 -- 172.21.15.3:0/3252540714 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f5fa801e420 con 0x55c901b3d7a0 2024-02-09T23:02:14.590 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fa0009e70 con 0x55c901b3d7a0 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5f92ff5640 1 -- 172.21.15.3:0/3252540714 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fa801ebe0 con 0x55c901b3d7a0 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 10 client.19503 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55c901a5c6f0 con 0x55c901b3d7a0 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5f92ff5640 1 -- 172.21.15.3:0/3252540714 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa0009e70 con 0x55c901b3d7a0 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5f92ff5640 1 -- 172.21.15.3:0/3252540714 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5fa8011850 con 0x55c901b3d7a0 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 10 client.19503 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 10 client.19503 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 20 client.19503 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901bae210 0x55c901bb0630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 4 client.19503 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c9017ec1b0 con 0x55c901bae210 2024-02-09T23:02:14.591 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.582+0000 7f5faed9d640 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901bae210 0x55c901bb0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.583+0000 7f5f92ff5640 10 client.19503 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:14.592 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.583+0000 7f5faed9d640 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901bae210 0x55c901bb0630 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2823974050b024f8 server_cookie=aea0303af2fee25e in_seq=0 out_seq=0 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5f92ff5640 1 -- 172.21.15.3:0/3252540714 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c9017ec1b0 con 0x55c901bae210 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5f92ff5640 10 client.19503 handle_command_reply: tid=0 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 client.19503 shutdown 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 msgr2=0x55c901b439c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b439c0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f5fa801e630 tx=0x7f5fa8010d80 comp rx=0 tx=0).stop 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 shutdown_connections 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55c901bae210 0x55c901bb0630 crc :-1 s=READY pgs=306 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55c901b3d7a0 0x55c901b439c0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55c901b349d0 0x55c901b43480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 --2- 172.21.15.3:0/3252540714 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55c901b340b0 0x55c901b42f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5f92ff5640 0 client.19503 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 >> 172.21.15.3:0/3252540714 conn(0x55c901b2e9d0 msgr2=0x55c901b2d1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 shutdown_connections 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 1 -- 172.21.15.3:0/3252540714 wait complete. 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.623+0000 7f5fb7c42740 20 client.19503 trim_cache size 0 max 16384 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:02:14.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:02:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f980a2250 msgr2=0x7f5f980a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f980a2250 0x7f5f980a4710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8005fd0 tx=0x7f5fa8009d30 comp rx=0 tx=0).stop 2024-02-09T23:02:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 msgr2=0x7f5fb0100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 0x7f5fb0100630 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4009230 tx=0x7f5fa402c680 comp rx=0 tx=0).stop 2024-02-09T23:02:14.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 shutdown_connections 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5f980a2250 0x7f5f980a4710 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5fb0100b70 0x7f5fb01a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5fb0106510 0x7f5fb0100630 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 --2- 172.21.15.3:0/239959652 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5fb0068d40 0x7f5fb01000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 >> 172.21.15.3:0/239959652 conn(0x7f5fb0077220 msgr2=0x7f5fb00fe2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 shutdown_connections 2024-02-09T23:02:14.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:14.624+0000 7f5fb506f640 1 -- 172.21.15.3:0/239959652 wait complete. 2024-02-09T23:02:14.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:14 smithi003 ceph-mon[32678]: pgmap v344: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 251 op/s 2024-02-09T23:02:14.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 377785 3.30 MB/sec execute 488 sec latency 663.019 ms 2024-02-09T23:02:14.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:14 smithi064 ceph-mon[36638]: pgmap v344: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 251 op/s 2024-02-09T23:02:14.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:14 smithi089 ceph-mon[36665]: pgmap v344: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 251 op/s 2024-02-09T23:02:15.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 379929 3.32 MB/sec execute 489 sec latency 13.954 ms 2024-02-09T23:02:16.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:16 smithi003 ceph-mon[32678]: pgmap v345: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-02-09T23:02:16.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 381222 3.33 MB/sec execute 490 sec latency 13.013 ms 2024-02-09T23:02:16.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:16 smithi064 ceph-mon[36638]: pgmap v345: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-02-09T23:02:16.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:16 smithi089 ceph-mon[36665]: pgmap v345: 97 pgs: 97 active+clean; 150 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 2.0 MiB/s wr, 249 op/s 2024-02-09T23:02:17.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 381255 3.32 MB/sec execute 491 sec latency 987.126 ms 2024-02-09T23:02:18.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:18 smithi003 ceph-mon[32678]: pgmap v346: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:02:18.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 383349 3.34 MB/sec execute 492 sec latency 1110.329 ms 2024-02-09T23:02:18.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:18 smithi064 ceph-mon[36638]: pgmap v346: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:02:18.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:18 smithi089 ceph-mon[36665]: pgmap v346: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:02:19.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 384613 3.35 MB/sec execute 493 sec latency 15.324 ms 2024-02-09T23:02:20.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:20 smithi003 ceph-mon[32678]: pgmap v347: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:02:20.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 385687 3.34 MB/sec execute 494 sec latency 251.247 ms 2024-02-09T23:02:20.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:20 smithi064 ceph-mon[36638]: pgmap v347: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:02:20.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:20 smithi089 ceph-mon[36665]: pgmap v347: 97 pgs: 97 active+clean; 147 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:02:21.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 387270 3.36 MB/sec execute 495 sec latency 11.946 ms 2024-02-09T23:02:22.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:22 smithi003 ceph-mon[32678]: pgmap v348: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2024-02-09T23:02:22.663 INFO:tasks.workunit.client.0.smithi003.stdout: 1 388368 3.36 MB/sec execute 496 sec latency 392.260 ms 2024-02-09T23:02:22.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:22 smithi064 ceph-mon[36638]: pgmap v348: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2024-02-09T23:02:22.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:22 smithi089 ceph-mon[36665]: pgmap v348: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 2.3 MiB/s wr, 272 op/s 2024-02-09T23:02:23.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 388368 3.36 MB/sec execute 497 sec latency 1392.346 ms 2024-02-09T23:02:24.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:24 smithi003 ceph-mon[32678]: pgmap v349: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 154 op/s 2024-02-09T23:02:24.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 388368 3.35 MB/sec execute 498 sec latency 2392.464 ms 2024-02-09T23:02:24.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:24 smithi064 ceph-mon[36638]: pgmap v349: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 154 op/s 2024-02-09T23:02:24.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:24 smithi089 ceph-mon[36665]: pgmap v349: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 154 op/s 2024-02-09T23:02:25.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 389844 3.36 MB/sec execute 499 sec latency 2834.265 ms 2024-02-09T23:02:26.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:26 smithi003 ceph-mon[32678]: pgmap v350: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 154 op/s 2024-02-09T23:02:26.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 391258 3.37 MB/sec execute 500 sec latency 15.012 ms 2024-02-09T23:02:26.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:26 smithi064 ceph-mon[36638]: pgmap v350: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 154 op/s 2024-02-09T23:02:26.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:26 smithi089 ceph-mon[36665]: pgmap v350: 97 pgs: 97 active+clean; 153 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.1 MiB/s wr, 154 op/s 2024-02-09T23:02:27.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:27 smithi003 ceph-mon[32678]: pgmap v351: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-02-09T23:02:27.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 391911 3.37 MB/sec execute 501 sec latency 535.867 ms 2024-02-09T23:02:27.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:27 smithi064 ceph-mon[36638]: pgmap v351: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-02-09T23:02:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 -- 172.21.15.3:0/3826202352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 msgr2=0x7fed7c10e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 --2- 172.21.15.3:0/3826202352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c10e0f0 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fed74009cd0 tx=0x7fed7402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 -- 172.21.15.3:0/3826202352 shutdown_connections 2024-02-09T23:02:27.720 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 --2- 172.21.15.3:0/3826202352 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c10e0f0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 --2- 172.21.15.3:0/3826202352 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c101e00 0x7fed7c105e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 --2- 172.21.15.3:0/3826202352 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed7c101450 0x7fed7c101830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 -- 172.21.15.3:0/3826202352 >> 172.21.15.3:0/3826202352 conn(0x7fed7c0fccb0 msgr2=0x7fed7c0ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 -- 172.21.15.3:0/3826202352 shutdown_connections 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 -- 172.21.15.3:0/3826202352 wait complete. 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.715+0000 7fed8184e640 1 Processor -- start 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 -- start start 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c101450 0x7fed7c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed7c1976d0 0x7fed7c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c1981a0 con 0x7fed7c106520 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c198310 con 0x7fed7c1976d0 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7c1984d0 con 0x7fed7c101450 2024-02-09T23:02:27.721 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c19db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35266/0 (socket says 172.21.15.3:35266) 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 -- 172.21.15.3:0/4030115530 learned_addr learned my addr 172.21.15.3:0/4030115530 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed7affd640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c101450 0x7fed7c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed7c1976d0 0x7fed7c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 -- 172.21.15.3:0/4030115530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c101450 msgr2=0x7fed7c19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c101450 0x7fed7c19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 -- 172.21.15.3:0/4030115530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed7c1976d0 msgr2=0x7fed7c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed7c1976d0 0x7fed7c197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 -- 172.21.15.3:0/4030115530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed74009900 con 0x7fed7c106520 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed73fff640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c19db00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fed6400d740 tx=0x7fed6400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8084c640 1 -- 172.21.15.3:0/4030115530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64009c30 con 0x7fed7c106520 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8084c640 1 -- 172.21.15.3:0/4030115530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed6400de50 con 0x7fed7c106520 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7c1a4420 con 0x7fed7c106520 2024-02-09T23:02:27.722 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8084c640 1 -- 172.21.15.3:0/4030115530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64010460 con 0x7fed7c106520 2024-02-09T23:02:27.723 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.716+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c1a4900 con 0x7fed7c106520 2024-02-09T23:02:27.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:27 smithi089 ceph-mon[36665]: pgmap v351: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.8 MiB/s wr, 215 op/s 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.718+0000 7fed8084c640 1 -- 172.21.15.3:0/4030115530 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fed64004f10 con 0x7fed7c106520 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 Processor -- start 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 -- start start 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430558490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed8084c640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fed680a22a0 0x7fed680a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305589d0 0x559430561260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305617a0 0x559430563b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594304c6a00 con 0x5594305580b0 2024-02-09T23:02:27.730 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594304c3f30 con 0x5594305589d0 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed84427740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594304bf710 con 0x5594305617a0 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed7b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430558490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed7affd640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fed680a22a0 0x7fed680a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed7b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430558490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35268/0 (socket says 172.21.15.3:35268) 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed7b7fe640 1 -- 172.21.15.3:0/3313695110 learned_addr learned my addr 172.21.15.3:0/3313695110 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.719+0000 7fed8084c640 1 -- 172.21.15.3:0/4030115530 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fed640cda10 con 0x7fed7c106520 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed73fff640 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305589d0 0x559430561260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305617a0 0x559430563b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 -- 172.21.15.3:0/3313695110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305617a0 msgr2=0x559430563b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305617a0 0x559430563b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 -- 172.21.15.3:0/3313695110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305589d0 msgr2=0x559430561260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.731 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305589d0 0x559430561260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 -- 172.21.15.3:0/3313695110 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594304843b0 con 0x5594305580b0 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7affd640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fed680a22a0 0x7fed680a4760 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fed6c005fd0 tx=0x7fed6c00b590 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430558490 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fed74005e30 tx=0x7fed74004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24dfa06917bdd65 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed71ffb640 1 -- 172.21.15.3:0/3313695110 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74034440 con 0x5594305580b0 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed71ffb640 1 -- 172.21.15.3:0/3313695110 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed740345a0 con 0x5594305580b0 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.720+0000 7fed71ffb640 1 -- 172.21.15.3:0/3313695110 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74003bd0 con 0x5594305580b0 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 -- 172.21.15.3:0/3313695110 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 msgr2=0x559430558490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430558490 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7fed74005e30 tx=0x7fed74004a60 comp rx=0 tx=0).stop 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 -- 172.21.15.3:0/3313695110 shutdown_connections 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305589d0 0x559430561260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305617a0 0x559430563b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.732 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 --2- 172.21.15.3:0/3313695110 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430558490 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 -- 172.21.15.3:0/3313695110 >> 172.21.15.3:0/3313695110 conn(0x5594305529d0 msgr2=0x559430552db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 -- 172.21.15.3:0/3313695110 shutdown_connections 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 -- 172.21.15.3:0/3313695110 wait complete. 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 -- start start 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.721+0000 7fed84427740 1 Processor -- start 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x55943055e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055edc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055f300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559430464b90 con 0x5594305589d0 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559430193250 con 0x5594305617a0 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559430445200 con 0x5594305580b0 2024-02-09T23:02:27.733 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 10 client.0 _ll_register_callbacks cb 0x55943047b5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35272/0 (socket says 172.21.15.3:35272) 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 -- 172.21.15.3:0/339239966 learned_addr learned my addr 172.21.15.3:0/339239966 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed7affd640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x55943055e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 -- 172.21.15.3:0/339239966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 msgr2=0x55943055e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x55943055e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 -- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 msgr2=0x55943055f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 -- 172.21.15.3:0/339239966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7c10e900 con 0x5594305589d0 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed73fff640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055edc0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fed640cb1c0 tx=0x7fed6409abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df07e93184c2ca1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:27.734 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed5affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed5affd640 1 -- 172.21.15.3:0/339239966 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6409a400 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed5affd640 1 -- 172.21.15.3:0/339239966 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fed6409e550 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5594304e5ee0 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed5affd640 1 -- 172.21.15.3:0/339239966 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed640a2ba0 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 10 client.19515 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.722+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x5594304bdd90 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed5affd640 1 -- 172.21.15.3:0/339239966 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5594304e5ee0 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed5affd640 1 -- 172.21.15.3:0/339239966 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fed640a9da0 con 0x5594305589d0 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed84427740 10 client.19515 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed84427740 10 client.19515 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed84427740 20 client.19515 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed84427740 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x559430573e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed84427740 4 client.19515 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:27.735 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5594301bf600 con 0x559430571a40 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.723+0000 7fed7affd640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x559430573e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed7affd640 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x559430573e60 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd2a863bf0ff572a server_cookie=72ef26b199f1cf3 in_seq=0 out_seq=0 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed5affd640 10 client.19515 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed5affd640 1 -- 172.21.15.3:0/339239966 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5594301bf600 con 0x559430571a40 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed5affd640 10 client.19515 handle_command_reply: tid=0 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 client.19515 shutdown 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 msgr2=0x55943055edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055edc0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7fed640cb1c0 tx=0x7fed6409abc0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 shutdown_connections 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x559430573e60 crc :-1 s=READY pgs=307 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055edc0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.736 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed5affd640 0 client.19515 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 --2- 172.21.15.3:0/339239966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x55943055e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 >> 172.21.15.3:0/339239966 conn(0x5594305529d0 msgr2=0x559430554720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 shutdown_connections 2024-02-09T23:02:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 1 -- 172.21.15.3:0/339239966 wait complete. 2024-02-09T23:02:27.737 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.724+0000 7fed84427740 20 client.19515 trim_cache size 0 max 16384 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 Processor -- start 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 -- start start 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x559430560f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55943054ce10 con 0x5594305589d0 2024-02-09T23:02:27.745 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594305539d0 con 0x5594305617a0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed84427740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594305521b0 con 0x5594305580b0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35282/0 (socket says 172.21.15.3:35282) 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 -- 172.21.15.3:0/1120180862 learned_addr learned my addr 172.21.15.3:0/1120180862 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed7affd640 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x559430560f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 -- 172.21.15.3:0/1120180862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 msgr2=0x559430560f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x559430560f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 -- 172.21.15.3:0/1120180862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 msgr2=0x55943055bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 -- 172.21.15.3:0/1120180862 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55943045a820 con 0x5594305589d0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.738+0000 7fed73fff640 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055f120 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fed64096860 tx=0x7fed640049b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50cf43076b73504 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:27.746 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed717fa640 1 -- 172.21.15.3:0/1120180862 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed64099d60 con 0x5594305589d0 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed717fa640 1 -- 172.21.15.3:0/1120180862 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed6409e550 con 0x5594305589d0 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed717fa640 1 -- 172.21.15.3:0/1120180862 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed640cb3e0 con 0x5594305589d0 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 -- 172.21.15.3:0/1120180862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 msgr2=0x55943055f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055f120 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7fed64096860 tx=0x7fed640049b0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 -- 172.21.15.3:0/1120180862 shutdown_connections 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x55943055bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305589d0 0x55943055f120 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 --2- 172.21.15.3:0/1120180862 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305580b0 0x559430560f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 -- 172.21.15.3:0/1120180862 >> 172.21.15.3:0/1120180862 conn(0x5594305529d0 msgr2=0x55943055d150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 -- 172.21.15.3:0/1120180862 shutdown_connections 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 -- 172.21.15.3:0/1120180862 wait complete. 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.739+0000 7fed84427740 1 -- start start 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 Processor -- start 2024-02-09T23:02:27.747 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430568750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305589d0 0x559430566af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x559430567030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559430567700 con 0x5594305580b0 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559430567870 con 0x5594305617a0 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594305c7f70 con 0x5594305589d0 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed84427740 10 client.0 _ll_register_callbacks cb 0x7fed640ba0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430568750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430568750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35292/0 (socket says 172.21.15.3:35292) 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 -- 172.21.15.3:0/843133726 learned_addr learned my addr 172.21.15.3:0/843133726 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7b7fe640 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x559430567030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.748 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 -- 172.21.15.3:0/843133726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305589d0 msgr2=0x559430566af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305589d0 0x559430566af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 -- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 msgr2=0x559430567030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x559430567030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 -- 172.21.15.3:0/843133726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594304bdd90 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed7affd640 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430568750 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x559430574790 tx=0x7fed6c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4684863eb302187f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.740+0000 7fed5affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed5affd640 1 -- 172.21.15.3:0/843133726 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c024040 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed5affd640 1 -- 172.21.15.3:0/843133726 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fed6c027290 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed740345a0 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed5affd640 1 -- 172.21.15.3:0/843133726 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed6c027490 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 10 client.19521 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x5594304806f0 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed5affd640 1 -- 172.21.15.3:0/843133726 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fed740345a0 con 0x5594305580b0 2024-02-09T23:02:27.749 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed5affd640 1 -- 172.21.15.3:0/843133726 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fed6c02d020 con 0x5594305580b0 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 10 client.19521 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 10 client.19521 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 20 client.19521 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x5594305cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 4 client.19521 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.741+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5594302101b0 con 0x559430571a40 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.742+0000 7fed73fff640 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x5594305cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.742+0000 7fed73fff640 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x5594305cb790 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37d2a45c6cff94cc server_cookie=1f1bbc1c2f48e14 in_seq=0 out_seq=0 2024-02-09T23:02:27.750 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.742+0000 7fed5affd640 10 client.19521 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:27.777 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed5affd640 1 -- 172.21.15.3:0/843133726 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5594302101b0 con 0x559430571a40 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed5affd640 10 client.19521 handle_command_reply: tid=0 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed84427740 1 client.19521 shutdown 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 msgr2=0x559430568750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed84427740 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430568750 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x559430574790 tx=0x7fed6c004730 comp rx=0 tx=0).stop 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 shutdown_connections 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.774+0000 7fed84427740 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x559430571a40 0x5594305cb790 crc :-1 s=READY pgs=308 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5594305617a0 0x559430567030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5594305589d0 0x559430566af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 1 --2- 172.21.15.3:0/843133726 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5594305580b0 0x559430568750 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed5affd640 0 client.19521 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 >> 172.21.15.3:0/843133726 conn(0x5594305529d0 msgr2=0x559430551a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 shutdown_connections 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 1 -- 172.21.15.3:0/843133726 wait complete. 2024-02-09T23:02:27.778 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed84427740 20 client.19521 trim_cache size 0 max 16384 2024-02-09T23:02:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:02:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:02:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:02:27.779 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fed680a22a0 msgr2=0x7fed680a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fed680a22a0 0x7fed680a4760 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fed6c005fd0 tx=0x7fed6c00b590 comp rx=0 tx=0).stop 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 msgr2=0x7fed7c19db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c19db00 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fed6400d740 tx=0x7fed6400dc00 comp rx=0 tx=0).stop 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 shutdown_connections 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fed680a22a0 0x7fed680a4760 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fed7c1976d0 0x7fed7c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fed7c106520 0x7fed7c19db00 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 --2- 172.21.15.3:0/4030115530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fed7c101450 0x7fed7c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:27.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 >> 172.21.15.3:0/4030115530 conn(0x7fed7c0fccb0 msgr2=0x7fed7c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 shutdown_connections 2024-02-09T23:02:27.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:27.775+0000 7fed8184e640 1 -- 172.21.15.3:0/4030115530 wait complete. 2024-02-09T23:02:28.542 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:02:28.543 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 -- 172.21.15.3:0/3220022520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d501008f0 msgr2=0x7f9d5010d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/3220022520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d501008f0 0x7f9d5010d640 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009900 tx=0x7f9d3c02bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 -- 172.21.15.3:0/3220022520 shutdown_connections 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/3220022520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d5010db80 0x7f9d5010ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/3220022520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d501008f0 0x7f9d5010d640 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/3220022520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d500fffd0 0x7f9d501003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 -- 172.21.15.3:0/3220022520 >> 172.21.15.3:0/3220022520 conn(0x7f9d50077250 msgr2=0x7f9d50077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.664 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 -- 172.21.15.3:0/3220022520 shutdown_connections 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 -- 172.21.15.3:0/3220022520 wait complete. 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 Processor -- start 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 -- start start 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 0x7f9d5019f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d501008f0 0x7f9d5019fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.659+0000 7f9d56c0e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d5010db80 0x7f9d501998a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d56c0e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d50199ef0 con 0x7f9d500fffd0 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d56c0e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5019a060 con 0x7f9d501008f0 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d56c0e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5019a220 con 0x7f9d5010db80 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 0x7f9d5019f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 0x7f9d5019f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35316/0 (socket says 172.21.15.3:35316) 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 -- 172.21.15.3:0/929961059 learned_addr learned my addr 172.21.15.3:0/929961059 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:28.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d551ab640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d5010db80 0x7f9d501998a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d501008f0 0x7f9d5019fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 -- 172.21.15.3:0/929961059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d5010db80 msgr2=0x7f9d501998a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d5010db80 0x7f9d501998a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 -- 172.21.15.3:0/929961059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d501008f0 msgr2=0x7f9d5019fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d501008f0 0x7f9d5019fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 -- 172.21.15.3:0/929961059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d3c009560 con 0x7f9d500fffd0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d549aa640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 0x7f9d5019f6f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400e9f0 tx=0x7f9d4400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d4dffb640 1 -- 172.21.15.3:0/929961059 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44009700 con 0x7f9d500fffd0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d4dffb640 1 -- 172.21.15.3:0/929961059 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d4400cd80 con 0x7f9d500fffd0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5019a500 con 0x7f9d500fffd0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d4dffb640 1 -- 172.21.15.3:0/929961059 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44018920 con 0x7f9d500fffd0 2024-02-09T23:02:28.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.660+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d501a6790 con 0x7f9d500fffd0 2024-02-09T23:02:28.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 394018 3.39 MB/sec execute 502 sec latency 645.192 ms 2024-02-09T23:02:28.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.663+0000 7f9d4dffb640 1 -- 172.21.15.3:0/929961059 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f9d44018a80 con 0x7f9d500fffd0 2024-02-09T23:02:28.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 Processor -- start 2024-02-09T23:02:28.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 -- start start 2024-02-09T23:02:28.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d4dffb640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d400a22b0 0x7f9d400a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c615d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c621300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c5d0d00 con 0x55f09c616330 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c5852f0 con 0x55f09c61eeb0 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d597e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c582820 con 0x55f09c615980 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c615d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d551ab640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d400a22b0 0x7f9d400a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35328/0 (socket says 172.21.15.3:35328) 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c615d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:52938/0 (socket says 172.21.15.3:52938) 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d4ffff640 1 -- 172.21.15.3:0/1013960977 learned_addr learned my addr 172.21.15.3:0/1013960977 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d551ab640 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c621300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.664+0000 7f9d4dffb640 1 -- 172.21.15.3:0/929961059 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9d44014070 con 0x7f9d500fffd0 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d549aa640 1 -- 172.21.15.3:0/1013960977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 msgr2=0x55f09c621300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d549aa640 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c621300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d549aa640 1 -- 172.21.15.3:0/1013960977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 msgr2=0x55f09c61e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d549aa640 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d549aa640 1 -- 172.21.15.3:0/1013960977 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f09c53fff0 con 0x55f09c615980 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d551ab640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d400a22b0 0x7f9d400a4770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009fd0 tx=0x7f9d48009d80 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d549aa640 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c615d60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d440cbb00 tx=0x7f9d4409a7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=405aa8267e415597 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d3affd640 1 -- 172.21.15.3:0/1013960977 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b3ad0 con 0x55f09c615980 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d3affd640 1 -- 172.21.15.3:0/1013960977 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d440b1f00 con 0x55f09c615980 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.665+0000 7f9d3affd640 1 -- 172.21.15.3:0/1013960977 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d44010c50 con 0x55f09c615980 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 -- 172.21.15.3:0/1013960977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 msgr2=0x55f09c615d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c615d60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d440cbb00 tx=0x7f9d4409a7b0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 -- 172.21.15.3:0/1013960977 shutdown_connections 2024-02-09T23:02:28.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c621300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1013960977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c615d60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 -- 172.21.15.3:0/1013960977 >> 172.21.15.3:0/1013960977 conn(0x55f09c610050 msgr2=0x55f09c610430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 -- 172.21.15.3:0/1013960977 shutdown_connections 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.666+0000 7f9d597e8740 1 -- 172.21.15.3:0/1013960977 wait complete. 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 -- start start 2024-02-09T23:02:28.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 Processor -- start 2024-02-09T23:02:28.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c61bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c61c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c57e000 con 0x55f09c616330 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c251430 con 0x55f09c61eeb0 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c20de00 con 0x55f09c615980 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d597e8740 10 client.0 _ll_register_callbacks cb 0x55f09c5727f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c61bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35342/0 (socket says 172.21.15.3:35342) 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d4ffff640 1 -- 172.21.15.3:0/1925018283 learned_addr learned my addr 172.21.15.3:0/1925018283 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d551ab640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c61c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d4ffff640 1 -- 172.21.15.3:0/1925018283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 msgr2=0x55f09c61bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.667+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c61bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d4ffff640 1 -- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 msgr2=0x55f09c61c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c61c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d4ffff640 1 -- 172.21.15.3:0/1925018283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d501a6790 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d549aa640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c61bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61c300 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009230 tx=0x7f9d3c02ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1593cb9cde23368 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1925018283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c004040 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1925018283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9d3c0041a0 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f09c27d7e0 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1925018283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d3c004380 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d597e8740 10 client.19533 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55f09c57c680 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.668+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1925018283 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f09c27d7e0 con 0x55f09c616330 2024-02-09T23:02:28.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1925018283 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9d3c03c040 con 0x55f09c616330 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d597e8740 10 client.19533 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d597e8740 10 client.19533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d597e8740 20 client.19533 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c62f2b0 0x55f09c6316d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d597e8740 4 client.19533 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f09c2ba720 con 0x55f09c62f2b0 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d549aa640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c62f2b0 0x55f09c6316d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d2ffff640 10 client.19533 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.669+0000 7f9d549aa640 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c62f2b0 0x55f09c6316d0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51544fdfa73fcf20 server_cookie=c9349677923603eb in_seq=0 out_seq=0 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d2ffff640 1 -- 172.21.15.3:0/1925018283 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f09c2ba720 con 0x55f09c62f2b0 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d2ffff640 10 client.19533 handle_command_reply: tid=0 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 client.19533 shutdown 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 msgr2=0x55f09c61c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61c300 secure :-1 s=READY pgs=1605 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009230 tx=0x7f9d3c02ca10 comp rx=0 tx=0).stop 2024-02-09T23:02:28.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 shutdown_connections 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c62f2b0 0x55f09c6316d0 crc :-1 s=READY pgs=309 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c61c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c61c300 unknown :-1 s=CLOSED pgs=1605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d2ffff640 0 client.19533 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 --2- 172.21.15.3:0/1925018283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c615980 0x55f09c61bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 >> 172.21.15.3:0/1925018283 conn(0x55f09c610050 msgr2=0x55f09c611da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 shutdown_connections 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 1 -- 172.21.15.3:0/1925018283 wait complete. 2024-02-09T23:02:28.679 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.670+0000 7f9d597e8740 20 client.19533 trim_cache size 0 max 16384 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 Processor -- start 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 -- start start 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c615980 0x55f09c61e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c619020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c61eeb0 0x55f09c619560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c502fd0 con 0x55f09c616330 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c27d7e0 con 0x55f09c615980 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d597e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c5e5ab0 con 0x55f09c61eeb0 2024-02-09T23:02:28.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c619020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c615980 0x55f09c61e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c619020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35350/0 (socket says 172.21.15.3:35350) 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d4ffff640 1 -- 172.21.15.3:0/3534513512 learned_addr learned my addr 172.21.15.3:0/3534513512 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.686+0000 7f9d551ab640 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c61eeb0 0x55f09c619560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d549aa640 1 -- 172.21.15.3:0/3534513512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c61eeb0 msgr2=0x55f09c619560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d549aa640 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c61eeb0 0x55f09c619560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d549aa640 1 -- 172.21.15.3:0/3534513512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 msgr2=0x55f09c619020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d549aa640 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c619020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d549aa640 1 -- 172.21.15.3:0/3534513512 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f09c518b30 con 0x55f09c615980 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d551ab640 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c61eeb0 0x55f09c619560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c619020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d549aa640 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c615980 0x55f09c61e4f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x55f09c621af0 tx=0x7f9d44000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0fc3b94bb69ffc0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d3a7fc640 1 -- 172.21.15.3:0/3534513512 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b9070 con 0x55f09c615980 2024-02-09T23:02:28.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d3a7fc640 1 -- 172.21.15.3:0/3534513512 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d4409e040 con 0x55f09c615980 2024-02-09T23:02:28.692 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.687+0000 7f9d3a7fc640 1 -- 172.21.15.3:0/3534513512 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b91d0 con 0x55f09c615980 2024-02-09T23:02:28.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 -- 172.21.15.3:0/3534513512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c615980 msgr2=0x55f09c61e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c615980 0x55f09c61e4f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x55f09c621af0 tx=0x7f9d44000bc0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.696 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 -- 172.21.15.3:0/3534513512 shutdown_connections 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c61eeb0 0x55f09c619560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c616330 0x55f09c619020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3534513512 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c615980 0x55f09c61e4f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 -- 172.21.15.3:0/3534513512 >> 172.21.15.3:0/3534513512 conn(0x55f09c610050 msgr2=0x55f09c636070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 -- 172.21.15.3:0/3534513512 shutdown_connections 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 -- 172.21.15.3:0/3534513512 wait complete. 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 -- start start 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.688+0000 7f9d597e8740 1 Processor -- start 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 0x55f09c67ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 0x55f09c680490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c625430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c625b00 con 0x55f09c615980 2024-02-09T23:02:28.697 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c625c70 con 0x55f09c61eeb0 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f09c625e30 con 0x55f09c616330 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 10 client.0 _ll_register_callbacks cb 0x7f9d4400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 0x55f09c67ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 0x55f09c680490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 0x55f09c67ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35356/0 (socket says 172.21.15.3:35356) 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d4ffff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 0x55f09c680490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:52962/0 (socket says 172.21.15.3:52962) 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 -- 172.21.15.3:0/3229442840 learned_addr learned my addr 172.21.15.3:0/3229442840 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d551ab640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c625430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 -- 172.21.15.3:0/3229442840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 msgr2=0x55f09c680490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 0x55f09c680490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 -- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 msgr2=0x55f09c625430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c625430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.698 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 -- 172.21.15.3:0/3229442840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f09c57c680 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 0x55f09c680490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d549aa640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 0x55f09c67ff50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9d44097820 tx=0x7f9d44010e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b63ae804d372b7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d2ffff640 1 -- 172.21.15.3:0/3229442840 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d440b9040 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9d440b1f00 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.689+0000 7f9d2ffff640 1 -- 172.21.15.3:0/3229442840 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9d4409eef0 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d2ffff640 1 -- 172.21.15.3:0/3229442840 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9d440a3540 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 10 client.19539 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55f09c53fff0 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d2ffff640 1 -- 172.21.15.3:0/3229442840 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d440b1f00 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d2ffff640 1 -- 172.21.15.3:0/3229442840 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9d44096ba0 con 0x55f09c615980 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 10 client.19539 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:28.699 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 10 client.19539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 20 client.19539 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c6878b0 0x55f09c689ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 4 client.19539 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f09c309890 con 0x55f09c6878b0 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c6878b0 0x55f09c689ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d2ffff640 10 client.19539 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.690+0000 7f9d4ffff640 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c6878b0 0x55f09c689ca0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48e30e6a4203ba00 server_cookie=3db6df486b1f670f in_seq=0 out_seq=0 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d2ffff640 1 -- 172.21.15.3:0/3229442840 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f09c309890 con 0x55f09c6878b0 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d2ffff640 10 client.19539 handle_command_reply: tid=0 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 client.19539 shutdown 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 msgr2=0x55f09c67ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 0x55f09c67ff50 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9d44097820 tx=0x7f9d44010e50 comp rx=0 tx=0).stop 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 shutdown_connections 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f09c6878b0 0x55f09c689ca0 crc :-1 s=READY pgs=310 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.700 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f09c61eeb0 0x55f09c625430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f09c616330 0x55f09c680490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d2ffff640 0 client.19539 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 --2- 172.21.15.3:0/3229442840 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f09c615980 0x55f09c67ff50 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 >> 172.21.15.3:0/3229442840 conn(0x55f09c610050 msgr2=0x55f09c60e960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 shutdown_connections 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 1 -- 172.21.15.3:0/3229442840 wait complete. 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.691+0000 7f9d597e8740 20 client.19539 trim_cache size 0 max 16384 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:02:28.701 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:02:28.702 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:02:28.702 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d400a22b0 msgr2=0x7f9d400a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d400a22b0 0x7f9d400a4770 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9d48009fd0 tx=0x7f9d48009d80 comp rx=0 tx=0).stop 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 msgr2=0x7f9d5019f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 0x7f9d5019f6f0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f9d4400e9f0 tx=0x7f9d4400eeb0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 shutdown_connections 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f9d400a22b0 0x7f9d400a4770 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f9d501008f0 0x7f9d5019fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f9d5010db80 0x7f9d501998a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 --2- 172.21.15.3:0/929961059 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f9d500fffd0 0x7f9d5019f6f0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 >> 172.21.15.3:0/929961059 conn(0x7f9d50077250 msgr2=0x7f9d5010f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 shutdown_connections 2024-02-09T23:02:28.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.692+0000 7f9d56c0e640 1 -- 172.21.15.3:0/929961059 wait complete. 2024-02-09T23:02:28.706 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:02:28.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:02:28.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 -- 172.21.15.3:0/1941992159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100d00 msgr2=0x7f2794104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 --2- 172.21.15.3:0/1941992159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100d00 0x7f2794104d40 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f2784009930 tx=0x7f278402bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:28.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 -- 172.21.15.3:0/1941992159 shutdown_connections 2024-02-09T23:02:28.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 --2- 172.21.15.3:0/1941992159 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2794105420 0x7f279410cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 --2- 172.21.15.3:0/1941992159 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100d00 0x7f2794104d40 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 --2- 172.21.15.3:0/1941992159 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2794100350 0x7f2794100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 -- 172.21.15.3:0/1941992159 >> 172.21.15.3:0/1941992159 conn(0x7f27940772b0 msgr2=0x7f27940776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 -- 172.21.15.3:0/1941992159 shutdown_connections 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.812+0000 7f279bdbc640 1 -- 172.21.15.3:0/1941992159 wait complete. 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 Processor -- start 2024-02-09T23:02:28.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 -- start start 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 0x7f279419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2794105420 0x7f279419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2794197700 0x7f2794197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27941981d0 con 0x7f2794100350 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2794198340 con 0x7f2794197700 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279bdbc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2794198500 con 0x7f2794105420 2024-02-09T23:02:28.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 0x7f279419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 0x7f279419d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35368/0 (socket says 172.21.15.3:35368) 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 -- 172.21.15.3:0/3200207100 learned_addr learned my addr 172.21.15.3:0/3200207100 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799357640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2794105420 0x7f279419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f279a359640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2794197700 0x7f2794197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 -- 172.21.15.3:0/3200207100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2794105420 msgr2=0x7f279419db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2794105420 0x7f279419db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 -- 172.21.15.3:0/3200207100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2794197700 msgr2=0x7f2794197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2794197700 0x7f2794197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.813+0000 7f2799b58640 1 -- 172.21.15.3:0/3200207100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2784009560 con 0x7f2794100350 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.814+0000 7f2799b58640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 0x7f279419d5f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f278c00e9f0 tx=0x7f278c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.814+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c009700 con 0x7f2794100350 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.814+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f278c00cd80 con 0x7f2794100350 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.814+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27941a4450 con 0x7f2794100350 2024-02-09T23:02:28.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.814+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278c018920 con 0x7f2794100350 2024-02-09T23:02:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.814+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f279410da40 con 0x7f2794100350 2024-02-09T23:02:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.815+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2794101820 con 0x7f2794100350 2024-02-09T23:02:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.816+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f278c018a80 con 0x7f2794100350 2024-02-09T23:02:28.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.817+0000 7f278affd640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27800a2230 0x7f27800a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.817+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f278c014070 con 0x7f2794100350 2024-02-09T23:02:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.817+0000 7f2799357640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27800a2230 0x7f27800a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:28.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.817+0000 7f2799357640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27800a2230 0x7f27800a46f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2784000c00 tx=0x7f2784005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:28.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:28.821+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f278c097620 con 0x7f2794100350 2024-02-09T23:02:29.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.052+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27940680b0 con 0x7f2794100350 2024-02-09T23:02:29.055 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.052+0000 7f278affd640 1 -- 172.21.15.3:0/3200207100 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+11364 (secure 0 0 0) 0x7f278c09ace0 con 0x7f2794100350 2024-02-09T23:02:29.056 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:02:29.056 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:01:40.086647+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:02:29.056 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 49 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27800a2230 msgr2=0x7f27800a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27800a2230 0x7f27800a46f0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2784000c00 tx=0x7f2784005fd0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 msgr2=0x7f279419d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 0x7f279419d5f0 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f278c00e9f0 tx=0x7f278c00eeb0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 shutdown_connections 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f27800a2230 0x7f27800a46f0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2794197700 0x7f2794197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2794105420 0x7f279419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 --2- 172.21.15.3:0/3200207100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2794100350 0x7f279419d5f0 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 >> 172.21.15.3:0/3200207100 conn(0x7f27940772b0 msgr2=0x7f27940ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 shutdown_connections 2024-02-09T23:02:29.058 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.055+0000 7f279bdbc640 1 -- 172.21.15.3:0/3200207100 wait complete. 2024-02-09T23:02:29.068 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:02:29.068 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:02:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 -- 172.21.15.3:0/666947606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 msgr2=0x7f7c38101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.183 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 --2- 172.21.15.3:0/666947606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c38101810 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009930 tx=0x7f7c3402bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 -- 172.21.15.3:0/666947606 shutdown_connections 2024-02-09T23:02:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 --2- 172.21.15.3:0/666947606 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c38101de0 0x7f7c38105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 --2- 172.21.15.3:0/666947606 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c38106500 0x7f7c3810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 --2- 172.21.15.3:0/666947606 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c38101810 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 -- 172.21.15.3:0/666947606 >> 172.21.15.3:0/666947606 conn(0x7f7c380fccd0 msgr2=0x7f7c380ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 -- 172.21.15.3:0/666947606 shutdown_connections 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.178+0000 7f7c40617640 1 -- 172.21.15.3:0/666947606 wait complete. 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 Processor -- start 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 -- start start 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c3819d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c38106500 0x7f7c3819db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c381976c0 0x7f7c38197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38198190 con 0x7f7c38101430 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c38198300 con 0x7f7c381976c0 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c40617640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c381984c0 con 0x7f7c38106500 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c3819d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c3819d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35398/0 (socket says 172.21.15.3:35398) 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/397722870 learned_addr learned my addr 172.21.15.3:0/397722870 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c381976c0 0x7f7c38197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.179+0000 7f7c3dbb2640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c38106500 0x7f7c3819db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/397722870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c38106500 msgr2=0x7f7c3819db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c38106500 0x7f7c3819db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/397722870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c381976c0 msgr2=0x7f7c38197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c381976c0 0x7f7c38197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/397722870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c34009560 con 0x7f7c38101430 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c3819d600 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000c00 tx=0x7f7c3402cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c2f7fe640 1 -- 172.21.15.3:0/397722870 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34004350 con 0x7f7c38101430 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c2f7fe640 1 -- 172.21.15.3:0/397722870 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c340044b0 con 0x7f7c38101430 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c381a4400 con 0x7f7c38101430 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c2f7fe640 1 -- 172.21.15.3:0/397722870 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34003bf0 con 0x7f7c38101430 2024-02-09T23:02:29.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.180+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c381a48e0 con 0x7f7c38101430 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c2f7fe640 1 -- 172.21.15.3:0/397722870 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7c34003d50 con 0x7f7c38101430 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 Processor -- start 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 -- start start 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cba30 0x56031a1cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 0x56031a1d4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1d73b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a187020 con 0x56031a1cc3e0 2024-02-09T23:02:29.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a13a320 con 0x56031a1d4f60 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c431e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a137850 con 0x56031a1cba30 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.183+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 0x56031a1d4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cba30 0x56031a1cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 0x56031a1d4990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35408/0 (socket says 172.21.15.3:35408) 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3dbb2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1d73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/1438134194 learned_addr learned my addr 172.21.15.3:0/1438134194 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c2f7fe640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c240a2230 0x7f7c240a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/1438134194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 msgr2=0x56031a1d73b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1d73b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/1438134194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 msgr2=0x56031a1d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 0x56031a1d4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/1438134194 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56031a0f5210 con 0x56031a1cba30 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 0x56031a1d4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c2f7fe640 1 -- 172.21.15.3:0/397722870 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7c340f0680 con 0x7f7c38101430 2024-02-09T23:02:29.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c240a2230 0x7f7c240a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cba30 0x56031a1cbe10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009f00 tx=0x7f7c30011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94429d94f1e72fb1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c2cbfe640 1 -- 172.21.15.3:0/1438134194 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3001e070 con 0x56031a1cba30 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c2cbfe640 1 -- 172.21.15.3:0/1438134194 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c30004530 con 0x56031a1cba30 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.184+0000 7f7c2cbfe640 1 -- 172.21.15.3:0/1438134194 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3000e900 con 0x56031a1cba30 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 -- 172.21.15.3:0/1438134194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cba30 msgr2=0x56031a1cbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cba30 0x56031a1cbe10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009f00 tx=0x7f7c30011810 comp rx=0 tx=0).stop 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c240a2230 0x7f7c240a46f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7c3001d000 tx=0x7f7c30019b60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 -- 172.21.15.3:0/1438134194 shutdown_connections 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1d73b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cc3e0 0x56031a1d4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 --2- 172.21.15.3:0/1438134194 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cba30 0x56031a1cbe10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 -- 172.21.15.3:0/1438134194 >> 172.21.15.3:0/1438134194 conn(0x56031a1c6100 msgr2=0x56031a1c64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 -- 172.21.15.3:0/1438134194 shutdown_connections 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.185+0000 7f7c431e0740 1 -- 172.21.15.3:0/1438134194 wait complete. 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 -- start start 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 Processor -- start 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cc3e0 0x56031a1d23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1d4f60 0x56031a1d28f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a133030 con 0x56031a1cba30 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560319e06430 con 0x56031a1cc3e0 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560319dc2e00 con 0x56031a1d4f60 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c431e0740 10 client.0 _ll_register_callbacks cb 0x56031a128030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35418/0 (socket says 172.21.15.3:35418) 2024-02-09T23:02:29.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/3089348317 learned_addr learned my addr 172.21.15.3:0/3089348317 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3dbb2640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cc3e0 0x56031a1d23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1d4f60 0x56031a1d28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1d4f60 msgr2=0x56031a1d28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1d4f60 0x56031a1d28f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/3089348317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cc3e0 msgr2=0x56031a1d23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cc3e0 0x56031a1d23b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/3089348317 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c30009be0 con 0x56031a1cba30 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d1e70 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f7c34006010 tx=0x7f7c3402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5629a73a38d2b41 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.186+0000 7f7c21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3089348317 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c340b94b0 con 0x56031a1cba30 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3089348317 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7c340bd910 con 0x56031a1cba30 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560319e327e0 con 0x56031a1cba30 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3089348317 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c340bcfe0 con 0x56031a1cba30 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 10 client.19557 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.198 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56031a1316b0 con 0x56031a1cba30 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3089348317 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560319e327e0 con 0x56031a1cba30 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3089348317 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7c340f8070 con 0x56031a1cba30 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 10 client.19557 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 10 client.19557 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 20 client.19557 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a1e53c0 0x56031a1e7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 4 client.19557 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560319e6f720 con 0x56031a1e53c0 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.187+0000 7f7c3dbb2640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a1e53c0 0x56031a1e7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.188+0000 7f7c3dbb2640 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a1e53c0 0x56031a1e7800 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2d81c6140fe02fc2 server_cookie=86b4cc0d74afb6e in_seq=0 out_seq=0 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.188+0000 7f7c21ffb640 10 client.19557 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.188+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3089348317 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560319e6f720 con 0x56031a1e53c0 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.188+0000 7f7c21ffb640 10 client.19557 handle_command_reply: tid=0 2024-02-09T23:02:29.199 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.188+0000 7f7c431e0740 1 client.19557 shutdown 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 msgr2=0x56031a1d1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d1e70 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f7c34006010 tx=0x7f7c3402bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 shutdown_connections 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a1e53c0 0x56031a1e7800 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cc3e0 0x56031a1d23b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1d4f60 0x56031a1d28f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c21ffb640 0 client.19557 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3089348317 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d1e70 unknown :-1 s=CLOSED pgs=1611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 >> 172.21.15.3:0/3089348317 conn(0x56031a1c6100 msgr2=0x56031a1c7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 shutdown_connections 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 1 -- 172.21.15.3:0/3089348317 wait complete. 2024-02-09T23:02:29.200 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.189+0000 7f7c431e0740 20 client.19557 trim_cache size 0 max 16384 2024-02-09T23:02:29.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 Processor -- start 2024-02-09T23:02:29.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 -- start start 2024-02-09T23:02:29.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d45a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1d2710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.210 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a0b8130 con 0x56031a1cba30 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560319e6f720 con 0x56031a1d4f60 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c431e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a1d7db0 con 0x56031a1cc3e0 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35420/0 (socket says 172.21.15.3:35420) 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/4044603832 learned_addr learned my addr 172.21.15.3:0/4044603832 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3dbb2640 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/4044603832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 msgr2=0x56031a1d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1d2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/4044603832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 msgr2=0x56031a1cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1cf0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.202+0000 7f7c3e3b3640 1 -- 172.21.15.3:0/4044603832 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56031a0cdc90 con 0x56031a1cba30 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d45a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f7c340b9670 tx=0x7f7c340c1360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8ea12e49b571e01 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.211 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c23fff640 1 -- 172.21.15.3:0/4044603832 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c340b94b0 con 0x56031a1cba30 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c23fff640 1 -- 172.21.15.3:0/4044603832 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c340c59a0 con 0x56031a1cba30 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c23fff640 1 -- 172.21.15.3:0/4044603832 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c340f6240 con 0x56031a1cba30 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 -- 172.21.15.3:0/4044603832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 msgr2=0x56031a1d45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d45a0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f7c340b9670 tx=0x7f7c340c1360 comp rx=0 tx=0).stop 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 -- 172.21.15.3:0/4044603832 shutdown_connections 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1d4f60 0x56031a1cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 --2- 172.21.15.3:0/4044603832 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1cba30 0x56031a1d45a0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 -- 172.21.15.3:0/4044603832 >> 172.21.15.3:0/4044603832 conn(0x56031a1c6100 msgr2=0x56031a1cd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.203+0000 7f7c431e0740 1 -- 172.21.15.3:0/4044603832 shutdown_connections 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 -- 172.21.15.3:0/4044603832 wait complete. 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 -- start start 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 Processor -- start 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cba30 0x56031a1da9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.212 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1daf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 0x56031a1db440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a23b930 con 0x56031a1d4f60 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a23ba70 con 0x56031a1cba30 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56031a23bbb0 con 0x56031a1cc3e0 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c431e0740 10 client.0 _ll_register_callbacks cb 0x7f7c30009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c3ebb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 0x56031a1db440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c3ebb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 0x56031a1db440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35422/0 (socket says 172.21.15.3:35422) 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/3403296577 learned_addr learned my addr 172.21.15.3:0/3403296577 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cba30 0x56031a1da9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.204+0000 7f7c3dbb2640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1daf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 msgr2=0x56031a1daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1daf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/3403296577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cba30 msgr2=0x56031a1da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.213 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cba30 0x56031a1da9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3ebb4640 1 -- 172.21.15.3:0/3403296577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56031a1316b0 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cba30 0x56031a1da9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c3ebb4640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 0x56031a1db440 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009f00 tx=0x7f7c30017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee514f9c7f7c58 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c21ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3403296577 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3001e040 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c30004530 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3403296577 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7c30019610 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3403296577 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c30004250 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 10 client.19563 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56031a0f3a70 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3403296577 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c30004530 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3403296577 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7c3002d040 con 0x56031a1d4f60 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 10 client.19563 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 10 client.19563 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 20 client.19563 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a23d460 0x56031a23f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 4 client.19563 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.205+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560319ebe890 con 0x56031a23d460 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a23d460 0x56031a23f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c3e3b3640 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a23d460 0x56031a23f880 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c5076112bad1e170 server_cookie=f3c836e66bfb9e53 in_seq=0 out_seq=0 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c21ffb640 10 client.19563 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c21ffb640 1 -- 172.21.15.3:0/3403296577 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560319ebe890 con 0x56031a23d460 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c21ffb640 10 client.19563 handle_command_reply: tid=0 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 client.19563 shutdown 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 msgr2=0x56031a1db440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 0x56031a1db440 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7f7c30009f00 tx=0x7f7c30017000 comp rx=0 tx=0).stop 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 shutdown_connections 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x56031a23d460 0x56031a23f880 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56031a1d4f60 0x56031a1db440 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56031a1cc3e0 0x56031a1daf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c21ffb640 0 client.19563 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 --2- 172.21.15.3:0/3403296577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56031a1cba30 0x56031a1da9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 >> 172.21.15.3:0/3403296577 conn(0x56031a1c6100 msgr2=0x56031a1c49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 shutdown_connections 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.206+0000 7f7c431e0740 1 -- 172.21.15.3:0/3403296577 wait complete. 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c431e0740 20 client.19563 trim_cache size 0 max 16384 2024-02-09T23:02:29.216 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:02:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c240a2230 msgr2=0x7f7c240a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c240a2230 0x7f7c240a46f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7c3001d000 tx=0x7f7c30019b60 comp rx=0 tx=0).stop 2024-02-09T23:02:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 msgr2=0x7f7c3819d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c3819d600 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7f7c34000c00 tx=0x7f7c3402cd80 comp rx=0 tx=0).stop 2024-02-09T23:02:29.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 shutdown_connections 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7c240a2230 0x7f7c240a46f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7c381976c0 0x7f7c38197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7c38106500 0x7f7c3819db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 --2- 172.21.15.3:0/397722870 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7c38101430 0x7f7c3819d600 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 >> 172.21.15.3:0/397722870 conn(0x7f7c380fccd0 msgr2=0x7f7c380fefa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 shutdown_connections 2024-02-09T23:02:29.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.207+0000 7f7c40617640 1 -- 172.21.15.3:0/397722870 wait complete. 2024-02-09T23:02:29.221 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:02:29.221 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:02:29.335 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 -- 172.21.15.3:0/3093347718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984100cf0 msgr2=0x7f7984104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3093347718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984100cf0 0x7f7984104d30 secure :-1 s=READY pgs=1614 cs=0 l=1 rev1=1 crypto rx=0x7f7978009930 tx=0x7f797802bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 -- 172.21.15.3:0/3093347718 shutdown_connections 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3093347718 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7984105410 0x7f798410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3093347718 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984100cf0 0x7f7984104d30 unknown :-1 s=CLOSED pgs=1614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3093347718 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7984100340 0x7f7984100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 -- 172.21.15.3:0/3093347718 >> 172.21.15.3:0/3093347718 conn(0x7f79840772e0 msgr2=0x7f79840776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 -- 172.21.15.3:0/3093347718 shutdown_connections 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 -- 172.21.15.3:0/3093347718 wait complete. 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 Processor -- start 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.330+0000 7f7989f3e640 1 -- start start 2024-02-09T23:02:29.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7989f3e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7984100340 0x7f798419d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7989f3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 0x7f798419db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7989f3e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7984197760 0x7f7984197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7989f3e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984198230 con 0x7f7984105410 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7989f3e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79841983a0 con 0x7f7984100340 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7989f3e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984198560 con 0x7f7984197760 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 0x7f798419db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 0x7f798419db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35448/0 (socket says 172.21.15.3:35448) 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 -- 172.21.15.3:0/3408118517 learned_addr learned my addr 172.21.15.3:0/3408118517 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f79837fe640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7984100340 0x7f798419d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7983fff640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7984197760 0x7f7984197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 -- 172.21.15.3:0/3408118517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7984197760 msgr2=0x7f7984197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7984197760 0x7f7984197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 -- 172.21.15.3:0/3408118517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7984100340 msgr2=0x7f798419d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7984100340 0x7f798419d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 -- 172.21.15.3:0/3408118517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978009560 con 0x7f7984105410 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7982ffd640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 0x7f798419db90 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f7978009230 tx=0x7f797802c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.331+0000 7f7980ff9640 1 -- 172.21.15.3:0/3408118517 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978004350 con 0x7f7984105410 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.332+0000 7f7980ff9640 1 -- 172.21.15.3:0/3408118517 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79780044b0 con 0x7f7984105410 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.332+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79841a4450 con 0x7f7984105410 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.332+0000 7f7980ff9640 1 -- 172.21.15.3:0/3408118517 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978003da0 con 0x7f7984105410 2024-02-09T23:02:29.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.332+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841a4980 con 0x7f7984105410 2024-02-09T23:02:29.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.334+0000 7f7980ff9640 1 -- 172.21.15.3:0/3408118517 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7978004df0 con 0x7f7984105410 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.334+0000 7f798cbed740 1 Processor -- start 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.334+0000 7f798cbed740 1 -- start start 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.334+0000 7f798cbed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560114575a30 0x560114575e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f798cbed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f798cbed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56011457ef60 0x5601145813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f798cbed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560114531020 con 0x560114575a30 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7980ff9640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f796c0a22b0 0x7f796c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f798cbed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601144e4320 con 0x5601145763e0 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f798cbed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601144e1850 con 0x56011457ef60 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7983fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560114575a30 0x560114575e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f79837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56011457ef60 0x5601145813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7983fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560114575a30 0x560114575e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35454/0 (socket says 172.21.15.3:35454) 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f79837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:49472/0 (socket says 172.21.15.3:49472) 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7983fff640 1 -- 172.21.15.3:0/2960146137 learned_addr learned my addr 172.21.15.3:0/2960146137 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7983fff640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f796c0a22b0 0x7f796c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7980ff9640 1 -- 172.21.15.3:0/3408118517 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f79780effc0 con 0x7f7984105410 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 -- 172.21.15.3:0/2960146137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 msgr2=0x56011457e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.346 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 -- 172.21.15.3:0/2960146137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560114575a30 msgr2=0x560114575e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560114575a30 0x560114575e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 -- 172.21.15.3:0/2960146137 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56011449f210 con 0x56011457ef60 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7982ffd640 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56011457ef60 0x5601145813b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f79780ed820 tx=0x7f7978000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=117d709b3ee2730b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7983fff640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f796c0a22b0 0x7f796c0a4770 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f797c009fd0 tx=0x7f797c009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7969ffb640 1 -- 172.21.15.3:0/2960146137 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780bcac0 con 0x56011457ef60 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7969ffb640 1 -- 172.21.15.3:0/2960146137 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79780c0c10 con 0x56011457ef60 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.335+0000 7f7969ffb640 1 -- 172.21.15.3:0/2960146137 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780edaf0 con 0x56011457ef60 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 -- 172.21.15.3:0/2960146137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56011457ef60 msgr2=0x5601145813b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56011457ef60 0x5601145813b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f79780ed820 tx=0x7f7978000bc0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 -- 172.21.15.3:0/2960146137 shutdown_connections 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56011457ef60 0x5601145813b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.347 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 --2- 172.21.15.3:0/2960146137 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560114575a30 0x560114575e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 -- 172.21.15.3:0/2960146137 >> 172.21.15.3:0/2960146137 conn(0x560114570100 msgr2=0x5601145704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 -- 172.21.15.3:0/2960146137 shutdown_connections 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 -- 172.21.15.3:0/2960146137 wait complete. 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.336+0000 7f798cbed740 1 -- start start 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 Processor -- start 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x56011457bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601144dd030 con 0x56011457ef60 2024-02-09T23:02:29.348 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601141b0430 con 0x5601145763e0 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56011416ce00 con 0x560114575a30 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f798cbed740 10 client.0 _ll_register_callbacks cb 0x5601144d2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f79837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x56011457bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f79837fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x56011457bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53040/0 (socket says 172.21.15.3:53040) 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f79837fe640 1 -- 172.21.15.3:0/554454808 learned_addr learned my addr 172.21.15.3:0/554454808 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35462/0 (socket says 172.21.15.3:35462) 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7982ffd640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 -- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 msgr2=0x56011457bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x56011457bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 -- 172.21.15.3:0/554454808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 msgr2=0x56011457c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.349 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7983fff640 1 -- 172.21.15.3:0/554454808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841a4980 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f79837fe640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x56011457bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.337+0000 7f7982ffd640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7983fff640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c9f0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f797c009930 tx=0x7f797c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5293de04095b9475 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7962ffd640 1 -- 172.21.15.3:0/554454808 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797c004830 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7962ffd640 1 -- 172.21.15.3:0/554454808 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f797c004b20 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5601141dc7e0 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7962ffd640 1 -- 172.21.15.3:0/554454808 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f797c004ce0 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 10 client.19575 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x5601144db6b0 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7962ffd640 1 -- 172.21.15.3:0/554454808 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f797c035080 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f7962ffd640 1 -- 172.21.15.3:0/554454808 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f797c02f030 con 0x56011457ef60 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 10 client.19575 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.350 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 10 client.19575 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 20 client.19575 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56011458f380 0x560114591770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 4 client.19575 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.338+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560114219720 con 0x56011458f380 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.339+0000 7f79837fe640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56011458f380 0x560114591770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.339+0000 7f79837fe640 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56011458f380 0x560114591770 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=e0370a9efa5be570 server_cookie=9b6de8a02bd1cadf in_seq=0 out_seq=0 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.339+0000 7f7962ffd640 10 client.19575 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f7962ffd640 1 -- 172.21.15.3:0/554454808 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560114219720 con 0x56011458f380 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f7962ffd640 10 client.19575 handle_command_reply: tid=0 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 client.19575 shutdown 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 msgr2=0x56011457c9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c9f0 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7f797c009930 tx=0x7f797c004a40 comp rx=0 tx=0).stop 2024-02-09T23:02:29.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 shutdown_connections 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x56011458f380 0x560114591770 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5601145763e0 0x56011457c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c9f0 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f7962ffd640 0 client.19575 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 --2- 172.21.15.3:0/554454808 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x56011457bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 >> 172.21.15.3:0/554454808 conn(0x560114570100 msgr2=0x560114571e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 shutdown_connections 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 1 -- 172.21.15.3:0/554454808 wait complete. 2024-02-09T23:02:29.352 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.340+0000 7f798cbed740 20 client.19575 trim_cache size 0 max 16384 2024-02-09T23:02:29.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 Processor -- start 2024-02-09T23:02:29.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 -- start start 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560114575a30 0x56011457e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601145763e0 0x56011456b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560114462130 con 0x56011457ef60 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560114219720 con 0x560114575a30 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f798cbed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560114596980 con 0x5601145763e0 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7983fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f79837fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560114575a30 0x56011457e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7983fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35468/0 (socket says 172.21.15.3:35468) 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601145763e0 0x56011456b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7983fff640 1 -- 172.21.15.3:0/612831852 learned_addr learned my addr 172.21.15.3:0/612831852 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7982ffd640 1 -- 172.21.15.3:0/612831852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560114575a30 msgr2=0x56011457e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.362 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7982ffd640 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560114575a30 0x56011457e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7982ffd640 1 -- 172.21.15.3:0/612831852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 msgr2=0x56011457c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7982ffd640 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7982ffd640 1 -- 172.21.15.3:0/612831852 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560114477c90 con 0x5601145763e0 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.353+0000 7f7983fff640 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.354+0000 7f7982ffd640 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601145763e0 0x56011456b210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f79780b9490 tx=0x7f7978003840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3769f0e93b2318f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.354+0000 7f79697fa640 1 -- 172.21.15.3:0/612831852 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780bab60 con 0x5601145763e0 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.354+0000 7f79697fa640 1 -- 172.21.15.3:0/612831852 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79780bcac0 con 0x5601145763e0 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.354+0000 7f79697fa640 1 -- 172.21.15.3:0/612831852 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780c4dd0 con 0x5601145763e0 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.354+0000 7f798cbed740 1 -- 172.21.15.3:0/612831852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601145763e0 msgr2=0x56011456b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.354+0000 7f798cbed740 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601145763e0 0x56011456b210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f79780b9490 tx=0x7f7978003840 comp rx=0 tx=0).stop 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- 172.21.15.3:0/612831852 shutdown_connections 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56011457ef60 0x56011457c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5601145763e0 0x56011456b210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.363 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 --2- 172.21.15.3:0/612831852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560114575a30 0x56011457e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- 172.21.15.3:0/612831852 >> 172.21.15.3:0/612831852 conn(0x560114570100 msgr2=0x5601145704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- 172.21.15.3:0/612831852 shutdown_connections 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- 172.21.15.3:0/612831852 wait complete. 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- start start 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 Processor -- start 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x5601145859b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 0x560114585ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56011457ef60 0x560114583fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560114584530 con 0x5601145763e0 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601145846a0 con 0x56011457ef60 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560114584860 con 0x560114575a30 2024-02-09T23:02:29.364 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f798cbed740 10 client.0 _ll_register_callbacks cb 0x7f7978003f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 0x560114585ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f7982ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 0x560114585ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35480/0 (socket says 172.21.15.3:35480) 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f7982ffd640 1 -- 172.21.15.3:0/3061851476 learned_addr learned my addr 172.21.15.3:0/3061851476 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.355+0000 7f79837fe640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x5601145859b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7983fff640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56011457ef60 0x560114583fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7982ffd640 1 -- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 msgr2=0x5601145859b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7982ffd640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x5601145859b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7982ffd640 1 -- 172.21.15.3:0/3061851476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56011457ef60 msgr2=0x560114583fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7982ffd640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56011457ef60 0x560114583fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7982ffd640 1 -- 172.21.15.3:0/3061851476 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601144db6b0 con 0x5601145763e0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f79837fe640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x5601145859b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7982ffd640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 0x560114585ef0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f79780b9440 tx=0x7f79780edc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7ea4c17ceadb4e7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.365 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7962ffd640 1 -- 172.21.15.3:0/3061851476 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780bab60 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7962ffd640 1 -- 172.21.15.3:0/3061851476 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f79780c5260 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f79780c0c10 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f7962ffd640 1 -- 172.21.15.3:0/3061851476 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f79780c4fe0 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f798cbed740 10 client.19581 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.356+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56011449f210 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f7962ffd640 1 -- 172.21.15.3:0/3061851476 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79780c0c10 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f7962ffd640 1 -- 172.21.15.3:0/3061851476 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f79780a3070 con 0x5601145763e0 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f798cbed740 10 client.19581 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f798cbed740 10 client.19581 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f798cbed740 20 client.19581 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f798cbed740 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5601145e66a0 0x5601145e8a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f798cbed740 4 client.19581 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:02:29.366 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560114268890 con 0x5601145e66a0 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f79837fe640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5601145e66a0 0x5601145e8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f7962ffd640 10 client.19581 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.357+0000 7f79837fe640 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5601145e66a0 0x5601145e8a90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9356db56895b0d2e server_cookie=1b25da84b7c396f2 in_seq=0 out_seq=0 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f7962ffd640 1 -- 172.21.15.3:0/3061851476 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560114268890 con 0x5601145e66a0 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f7962ffd640 10 client.19581 handle_command_reply: tid=0 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 client.19581 shutdown 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 msgr2=0x560114585ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 0x560114585ef0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7f79780b9440 tx=0x7f79780edc40 comp rx=0 tx=0).stop 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 shutdown_connections 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x5601145e66a0 0x5601145e8a90 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56011457ef60 0x560114583fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5601145763e0 0x560114585ef0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.367 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f7962ffd640 0 client.19581 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:02:29.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 --2- 172.21.15.3:0/3061851476 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560114575a30 0x5601145859b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 >> 172.21.15.3:0/3061851476 conn(0x560114570100 msgr2=0x56011456b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 shutdown_connections 2024-02-09T23:02:29.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 1 -- 172.21.15.3:0/3061851476 wait complete. 2024-02-09T23:02:29.368 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.358+0000 7f798cbed740 20 client.19581 trim_cache size 0 max 16384 2024-02-09T23:02:29.368 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:02:29.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f796c0a22b0 msgr2=0x7f796c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f796c0a22b0 0x7f796c0a4770 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f797c009fd0 tx=0x7f797c009500 comp rx=0 tx=0).stop 2024-02-09T23:02:29.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 msgr2=0x7f798419db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.369 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 0x7f798419db90 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f7978009230 tx=0x7f797802c680 comp rx=0 tx=0).stop 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 shutdown_connections 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f796c0a22b0 0x7f796c0a4770 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7984197760 0x7f7984197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7984105410 0x7f798419db90 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 --2- 172.21.15.3:0/3408118517 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7984100340 0x7f798419d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 >> 172.21.15.3:0/3408118517 conn(0x7f79840772e0 msgr2=0x7f79841065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 shutdown_connections 2024-02-09T23:02:29.370 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.359+0000 7f7989f3e640 1 -- 172.21.15.3:0/3408118517 wait complete. 2024-02-09T23:02:29.373 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:02:29.373 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/1910901987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 msgr2=0x7f0e24101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/1910901987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e24101810 secure :-1 s=READY pgs=1618 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009930 tx=0x7f0e1402bc20 comp rx=0 tx=0).stop 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/1910901987 shutdown_connections 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/1910901987 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e24106500 0x7f0e2410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/1910901987 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24101de0 0x7f0e24105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/1910901987 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e24101810 unknown :-1 s=CLOSED pgs=1618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/1910901987 >> 172.21.15.3:0/1910901987 conn(0x7f0e240fccd0 msgr2=0x7f0e240ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/1910901987 shutdown_connections 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/1910901987 wait complete. 2024-02-09T23:02:29.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 Processor -- start 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.480+0000 7f0e2a4c9640 1 -- start start 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e2a4c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e2419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e2a4c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24106500 0x7f0e2419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e2a4c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e241976e0 0x7f0e24197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e2a4c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e241981b0 con 0x7f0e24101430 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e2a4c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e24198320 con 0x7f0e241976e0 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e2a4c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e241984e0 con 0x7f0e24106500 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e2419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e2419d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35490/0 (socket says 172.21.15.3:35490) 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 -- 172.21.15.3:0/2702835060 learned_addr learned my addr 172.21.15.3:0/2702835060 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e28a66640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e241976e0 0x7f0e24197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e237fe640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24106500 0x7f0e2419db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 -- 172.21.15.3:0/2702835060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24106500 msgr2=0x7f0e2419db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24106500 0x7f0e2419db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 -- 172.21.15.3:0/2702835060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e241976e0 msgr2=0x7f0e24197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e241976e0 0x7f0e24197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 -- 172.21.15.3:0/2702835060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e1c009840 con 0x7f0e24101430 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.481+0000 7f0e23fff640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e2419d620 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009230 tx=0x7f0e1402c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.482+0000 7f0e217fa640 1 -- 172.21.15.3:0/2702835060 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14004350 con 0x7f0e24101430 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.482+0000 7f0e217fa640 1 -- 172.21.15.3:0/2702835060 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e140044b0 con 0x7f0e24101430 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.482+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e14009560 con 0x7f0e24101430 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.482+0000 7f0e217fa640 1 -- 172.21.15.3:0/2702835060 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e14003bf0 con 0x7f0e24101430 2024-02-09T23:02:29.487 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.482+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e241a4770 con 0x7f0e24101430 2024-02-09T23:02:29.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e217fa640 1 -- 172.21.15.3:0/2702835060 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0e14003d50 con 0x7f0e24101430 2024-02-09T23:02:29.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 Processor -- start 2024-02-09T23:02:29.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 -- start start 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81dacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c8196020 con 0x55e3c81da910 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c8149320 con 0x55e3c81e43c0 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e2d170740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c8146850 con 0x55e3c81db2c0 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e28a66640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81dacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e28a66640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81dacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35502/0 (socket says 172.21.15.3:35502) 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e28a66640 1 -- 172.21.15.3:0/3772959929 learned_addr learned my addr 172.21.15.3:0/3772959929 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.484+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e217fa640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e0c0a2230 0x7f0e0c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e237fe640 1 -- 172.21.15.3:0/3772959929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 msgr2=0x55e3c81e3df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e3df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e237fe640 1 -- 172.21.15.3:0/3772959929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 msgr2=0x55e3c81dacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81dacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e237fe640 1 -- 172.21.15.3:0/3772959929 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3c8104210 con 0x55e3c81e43c0 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e28a66640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e0c0a2230 0x7f0e0c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e217fa640 1 -- 172.21.15.3:0/2702835060 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0e140eff70 con 0x7f0e24101430 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e4820 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e2406a990 tx=0x7f0e18011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1752b58690cca5f4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e0a7fc640 1 -- 172.21.15.3:0/3772959929 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18004230 con 0x55e3c81e43c0 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e0a7fc640 1 -- 172.21.15.3:0/3772959929 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e18009e10 con 0x55e3c81e43c0 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.485+0000 7f0e0a7fc640 1 -- 172.21.15.3:0/3772959929 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18004c30 con 0x55e3c81e43c0 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 -- 172.21.15.3:0/3772959929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 msgr2=0x55e3c81e4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e28a66640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e0c0a2230 0x7f0e0c0a46f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009fd0 tx=0x7f0e1c004a40 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e4820 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0e2406a990 tx=0x7f0e18011810 comp rx=0 tx=0).stop 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 -- 172.21.15.3:0/3772959929 shutdown_connections 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e4820 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e3df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3772959929 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81dacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 -- 172.21.15.3:0/3772959929 >> 172.21.15.3:0/3772959929 conn(0x55e3c81d5100 msgr2=0x55e3c81d54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 -- 172.21.15.3:0/3772959929 shutdown_connections 2024-02-09T23:02:29.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 -- 172.21.15.3:0/3772959929 wait complete. 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 -- start start 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.486+0000 7f0e2d170740 1 Processor -- start 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81e7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e88a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c8142030 con 0x55e3c81da910 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c7e15430 con 0x55e3c81e43c0 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c7dd1e00 con 0x55e3c81db2c0 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 10 client.0 _ll_register_callbacks cb 0x55e3c8137030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81e7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e237fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81e7e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35510/0 (socket says 172.21.15.3:35510) 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 -- 172.21.15.3:0/3525829750 learned_addr learned my addr 172.21.15.3:0/3525829750 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.501 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e28a66640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 -- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 msgr2=0x55e3c81e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 -- 172.21.15.3:0/3525829750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 msgr2=0x55e3c81e88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 -- 172.21.15.3:0/3525829750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e18009830 con 0x55e3c81da910 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81e7e20 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009430 tx=0x7f0e140f3670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387bfb04454a9a0a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e037fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e037fe640 1 -- 172.21.15.3:0/3525829750 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140bc5b0 con 0x55e3c81da910 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e037fe640 1 -- 172.21.15.3:0/3525829750 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0e140bd170 con 0x55e3c81da910 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.487+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e3c7e417e0 con 0x55e3c81da910 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e037fe640 1 -- 172.21.15.3:0/3525829750 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e140bc3d0 con 0x55e3c81da910 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 10 client.19593 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55e3c81406b0 con 0x55e3c81da910 2024-02-09T23:02:29.502 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e037fe640 1 -- 172.21.15.3:0/3525829750 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e3c7e417e0 con 0x55e3c81da910 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e037fe640 1 -- 172.21.15.3:0/3525829750 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0e1403c070 con 0x55e3c81da910 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 10 client.19593 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 10 client.19593 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 20 client.19593 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c81f86f0 0x55e3c81faae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 4 client.19593 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e3c7e7e720 con 0x55e3c81f86f0 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.488+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c81f86f0 0x55e3c81faae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.489+0000 7f0e237fe640 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c81f86f0 0x55e3c81faae0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1781ee5c48b63b7 server_cookie=a270763a8104d039 in_seq=0 out_seq=0 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.489+0000 7f0e037fe640 10 client.19593 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e037fe640 1 -- 172.21.15.3:0/3525829750 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e3c7e7e720 con 0x55e3c81f86f0 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e037fe640 10 client.19593 handle_command_reply: tid=0 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 client.19593 shutdown 2024-02-09T23:02:29.503 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 msgr2=0x55e3c81e7e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81e7e20 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009430 tx=0x7f0e140f3670 comp rx=0 tx=0).stop 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 shutdown_connections 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c81f86f0 0x55e3c81faae0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81e43c0 0x55e3c81e88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e037fe640 0 client.19593 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81db2c0 0x55e3c81e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3525829750 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81e7e20 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 >> 172.21.15.3:0/3525829750 conn(0x55e3c81d5100 msgr2=0x55e3c81d6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 shutdown_connections 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 1 -- 172.21.15.3:0/3525829750 wait complete. 2024-02-09T23:02:29.504 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.490+0000 7f0e2d170740 20 client.19593 trim_cache size 0 max 16384 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 Processor -- start 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 -- start start 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81ea490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81db2c0 0x55e3c81e57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c80c7130 con 0x55e3c81da910 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c81d5a40 con 0x55e3c81db2c0 2024-02-09T23:02:29.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e2d170740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c81e2130 con 0x55e3c81e43c0 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81ea490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e237fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81db2c0 0x55e3c81e57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e28a66640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81ea490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35514/0 (socket says 172.21.15.3:35514) 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 -- 172.21.15.3:0/3224727358 learned_addr learned my addr 172.21.15.3:0/3224727358 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 -- 172.21.15.3:0/3224727358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 msgr2=0x55e3c81e86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 -- 172.21.15.3:0/3224727358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81db2c0 msgr2=0x55e3c81e57b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81db2c0 0x55e3c81e57b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.503+0000 7f0e23fff640 1 -- 172.21.15.3:0/3224727358 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e3c80dcc90 con 0x55e3c81da910 2024-02-09T23:02:29.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e23fff640 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81ea490 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f0e140bc770 tx=0x7f0e14005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40083ac4387490ef server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e09ffb640 1 -- 172.21.15.3:0/3224727358 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140bc3d0 con 0x55e3c81da910 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e09ffb640 1 -- 172.21.15.3:0/3224727358 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e140bc5b0 con 0x55e3c81da910 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e09ffb640 1 -- 172.21.15.3:0/3224727358 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e140c0bc0 con 0x55e3c81da910 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 -- 172.21.15.3:0/3224727358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 msgr2=0x55e3c81ea490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81ea490 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f0e140bc770 tx=0x7f0e14005fd0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 -- 172.21.15.3:0/3224727358 shutdown_connections 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81db2c0 0x55e3c81e57b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 --2- 172.21.15.3:0/3224727358 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81da910 0x55e3c81ea490 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.504+0000 7f0e2d170740 1 -- 172.21.15.3:0/3224727358 >> 172.21.15.3:0/3224727358 conn(0x55e3c81d5100 msgr2=0x55e3c81e66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 -- 172.21.15.3:0/3224727358 shutdown_connections 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 -- 172.21.15.3:0/3224727358 wait complete. 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 -- start start 2024-02-09T23:02:29.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 Processor -- start 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81da910 0x55e3c81ef5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 0x55e3c81dfec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e0400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c81efd40 con 0x55e3c81db2c0 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c81e0970 con 0x55e3c81da910 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e3c81e0b30 con 0x55e3c81e43c0 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e2d170740 10 client.0 _ll_register_callbacks cb 0x7f0e18009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 0x55e3c81dfec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e23fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81da910 0x55e3c81ef5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e237fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 0x55e3c81dfec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35526/0 (socket says 172.21.15.3:35526) 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e237fe640 1 -- 172.21.15.3:0/1667896956 learned_addr learned my addr 172.21.15.3:0/1667896956 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.505+0000 7f0e28a66640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e237fe640 1 -- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 msgr2=0x55e3c81e0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e237fe640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e237fe640 1 -- 172.21.15.3:0/1667896956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81da910 msgr2=0x55e3c81ef5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e237fe640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81da910 0x55e3c81ef5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e237fe640 1 -- 172.21.15.3:0/1667896956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e3c81406b0 con 0x55e3c81db2c0 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e23fff640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81da910 0x55e3c81ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e237fe640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 0x55e3c81dfec0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009b20 tx=0x7f0e18005d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc8ed94c3a85ba9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e037fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e037fe640 1 -- 172.21.15.3:0/1667896956 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18019bb0 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0e18009e10 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e037fe640 1 -- 172.21.15.3:0/1667896956 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0e18019d10 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e037fe640 1 -- 172.21.15.3:0/1667896956 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0e18004040 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e2d170740 10 client.19599 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55e3c8102a70 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.506+0000 7f0e037fe640 1 -- 172.21.15.3:0/1667896956 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e1803b080 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e037fe640 1 -- 172.21.15.3:0/1667896956 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0e18035030 con 0x55e3c81db2c0 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 10 client.19599 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 10 client.19599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 20 client.19599 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c824fff0 0x55e3c8252410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 4 client.19599 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e3c7ecd890 con 0x55e3c824fff0 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e23fff640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c824fff0 0x55e3c8252410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e037fe640 10 client.19599 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e23fff640 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c824fff0 0x55e3c8252410 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f23c15b4c4a25bb server_cookie=fe5742eaac7b6ff5 in_seq=0 out_seq=0 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e037fe640 1 -- 172.21.15.3:0/1667896956 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e3c7ecd890 con 0x55e3c824fff0 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e037fe640 10 client.19599 handle_command_reply: tid=0 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 1 client.19599 shutdown 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 msgr2=0x55e3c81dfec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 0x55e3c81dfec0 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f0e18009b20 tx=0x7f0e18005d70 comp rx=0 tx=0).stop 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.507+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 shutdown_connections 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x55e3c824fff0 0x55e3c8252410 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e3c81e43c0 0x55e3c81e0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e3c81db2c0 0x55e3c81dfec0 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e037fe640 0 client.19599 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:02:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 --2- 172.21.15.3:0/1667896956 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e3c81da910 0x55e3c81ef5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 >> 172.21.15.3:0/1667896956 conn(0x55e3c81d5100 msgr2=0x55e3c81d39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 shutdown_connections 2024-02-09T23:02:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 1 -- 172.21.15.3:0/1667896956 wait complete. 2024-02-09T23:02:29.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2d170740 20 client.19599 trim_cache size 0 max 16384 2024-02-09T23:02:29.519 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:02:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e0c0a2230 msgr2=0x7f0e0c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e0c0a2230 0x7f0e0c0a46f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0e1c009fd0 tx=0x7f0e1c004a40 comp rx=0 tx=0).stop 2024-02-09T23:02:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 msgr2=0x7f0e2419d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e2419d620 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f0e14009230 tx=0x7f0e1402c640 comp rx=0 tx=0).stop 2024-02-09T23:02:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 shutdown_connections 2024-02-09T23:02:29.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0e0c0a2230 0x7f0e0c0a46f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0e241976e0 0x7f0e24197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0e24106500 0x7f0e2419db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 --2- 172.21.15.3:0/2702835060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0e24101430 0x7f0e2419d620 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 >> 172.21.15.3:0/2702835060 conn(0x7f0e240fccd0 msgr2=0x7f0e240fe110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 shutdown_connections 2024-02-09T23:02:29.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.508+0000 7f0e2a4c9640 1 -- 172.21.15.3:0/2702835060 wait complete. 2024-02-09T23:02:29.522 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:02:29.522 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:02:29.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:29 smithi003 ceph-mon[32678]: pgmap v352: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-02-09T23:02:29.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:29 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3200207100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:02:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.638+0000 7f2261ca4640 1 -- 172.21.15.3:0/1786476768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c100cf0 msgr2=0x7f225c104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.638+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1786476768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c100cf0 0x7f225c104d30 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7f2250009930 tx=0x7f225002bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 -- 172.21.15.3:0/1786476768 shutdown_connections 2024-02-09T23:02:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1786476768 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f225c105410 0x7f225c10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1786476768 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c100cf0 0x7f225c104d30 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1786476768 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f225c100340 0x7f225c100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 -- 172.21.15.3:0/1786476768 >> 172.21.15.3:0/1786476768 conn(0x7f225c0772e0 msgr2=0x7f225c0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 -- 172.21.15.3:0/1786476768 shutdown_connections 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 -- 172.21.15.3:0/1786476768 wait complete. 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 Processor -- start 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.639+0000 7f2261ca4640 1 -- start start 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f225c100340 0x7f225c101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 0x7f225c102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f225c102850 0x7f225c1a5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c1a6080 con 0x7f225c105410 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c1a61f0 con 0x7f225c102850 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c1a63b0 con 0x7f225c100340 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 0x7f225c102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f225c100340 0x7f225c101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 0x7f225c102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35550/0 (socket says 172.21.15.3:35550) 2024-02-09T23:02:29.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 -- 172.21.15.3:0/1415213400 learned_addr learned my addr 172.21.15.3:0/1415213400 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225bfff640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f225c102850 0x7f225c1a5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 -- 172.21.15.3:0/1415213400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f225c100340 msgr2=0x7f225c101dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f225c100340 0x7f225c101dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 -- 172.21.15.3:0/1415213400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f225c102850 msgr2=0x7f225c1a5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f225c102850 0x7f225c1a5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 -- 172.21.15.3:0/1415213400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2250009560 con 0x7f225c105410 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f225affd640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 0x7f225c102310 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f2250009230 tx=0x7f2250004c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2258ff9640 1 -- 172.21.15.3:0/1415213400 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500043d0 con 0x7f225c105410 2024-02-09T23:02:29.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2258ff9640 1 -- 172.21.15.3:0/1415213400 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22500052d0 con 0x7f225c105410 2024-02-09T23:02:29.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2258ff9640 1 -- 172.21.15.3:0/1415213400 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250003da0 con 0x7f225c105410 2024-02-09T23:02:29.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c1a6630 con 0x7f225c105410 2024-02-09T23:02:29.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.640+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c1a6b40 con 0x7f225c105410 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.643+0000 7f2258ff9640 1 -- 172.21.15.3:0/1415213400 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f2250005440 con 0x7f225c105410 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.643+0000 7f2258ff9640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22440a2260 0x7f22440a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.643+0000 7f2258ff9640 1 -- 172.21.15.3:0/1415213400 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f22500f0120 con 0x7f225c105410 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.643+0000 7f226487e740 1 Processor -- start 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.643+0000 7f225b7fe640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22440a2260 0x7f22440a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 -- start start 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe43ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 0x560ebe445990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.655 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe4483b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe3f8020 con 0x560ebe43d3e0 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe3ab320 con 0x560ebe445f60 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f226487e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe3a8850 con 0x560ebe43ca30 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 0x560ebe445990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 0x560ebe445990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35566/0 (socket says 172.21.15.3:35566) 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 -- 172.21.15.3:0/4158470255 learned_addr learned my addr 172.21.15.3:0/4158470255 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225bfff640 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe4483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225b7fe640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22440a2260 0x7f22440a4720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f224c009bd0 tx=0x7f224c005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225b7fe640 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe43ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 -- 172.21.15.3:0/4158470255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 msgr2=0x560ebe43ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe43ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 -- 172.21.15.3:0/4158470255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 msgr2=0x560ebe4483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe4483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.656 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 -- 172.21.15.3:0/4158470255 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ebe366210 con 0x560ebe43d3e0 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225bfff640 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe4483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.644+0000 7f225affd640 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 0x560ebe445990 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f22500c4e90 tx=0x7f2250009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5d1fb65e25a531e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f2241ffb640 1 -- 172.21.15.3:0/4158470255 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500bcc20 con 0x560ebe43d3e0 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f2241ffb640 1 -- 172.21.15.3:0/4158470255 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22500c0d70 con 0x560ebe43d3e0 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f2241ffb640 1 -- 172.21.15.3:0/4158470255 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22500b9290 con 0x560ebe43d3e0 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 -- 172.21.15.3:0/4158470255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 msgr2=0x560ebe445990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 0x560ebe445990 secure :-1 s=READY pgs=1625 cs=0 l=1 rev1=1 crypto rx=0x7f22500c4e90 tx=0x7f2250009260 comp rx=0 tx=0).stop 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 -- 172.21.15.3:0/4158470255 shutdown_connections 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe4483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43d3e0 0x560ebe445990 unknown :-1 s=CLOSED pgs=1625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 --2- 172.21.15.3:0/4158470255 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe43ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 -- 172.21.15.3:0/4158470255 >> 172.21.15.3:0/4158470255 conn(0x560ebe437100 msgr2=0x560ebe4374e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 -- 172.21.15.3:0/4158470255 shutdown_connections 2024-02-09T23:02:29.657 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.645+0000 7f226487e740 1 -- 172.21.15.3:0/4158470255 wait complete. 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 -- start start 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 Processor -- start 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe442f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43d3e0 0x560ebe443470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4439b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe3a4030 con 0x560ebe445f60 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe077430 con 0x560ebe43d3e0 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe033e00 con 0x560ebe43ca30 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f226487e740 10 client.0 _ll_register_callbacks cb 0x560ebe399030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f225bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f225b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe442f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f225affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43d3e0 0x560ebe443470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f225bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4439b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35572/0 (socket says 172.21.15.3:35572) 2024-02-09T23:02:29.658 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f225bfff640 1 -- 172.21.15.3:0/1815301638 learned_addr learned my addr 172.21.15.3:0/1815301638 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.646+0000 7f225b7fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe442f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53172/0 (socket says 172.21.15.3:53172) 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225bfff640 1 -- 172.21.15.3:0/1815301638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 msgr2=0x560ebe442f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225bfff640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe442f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225bfff640 1 -- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43d3e0 msgr2=0x560ebe443470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225bfff640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43d3e0 0x560ebe443470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225bfff640 1 -- 172.21.15.3:0/1815301638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22500a7040 con 0x560ebe445f60 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225b7fe640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe442f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225affd640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43d3e0 0x560ebe443470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f225bfff640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4439b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f2254011460 tx=0x7f2254011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7eb7a9ae9d22fc6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f223affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f223affd640 1 -- 172.21.15.3:0/1815301638 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254012980 con 0x560ebe445f60 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f223affd640 1 -- 172.21.15.3:0/1815301638 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2254012ae0 con 0x560ebe445f60 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560ebe0a37e0 con 0x560ebe445f60 2024-02-09T23:02:29.659 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f223affd640 1 -- 172.21.15.3:0/1815301638 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2254012ca0 con 0x560ebe445f60 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f226487e740 10 client.19611 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.647+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x560ebe3a26b0 con 0x560ebe445f60 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f223affd640 1 -- 172.21.15.3:0/1815301638 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560ebe0a37e0 con 0x560ebe445f60 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f223affd640 1 -- 172.21.15.3:0/1815301638 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f225400c070 con 0x560ebe445f60 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f226487e740 10 client.19611 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f226487e740 10 client.19611 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f226487e740 20 client.19611 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f226487e740 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe456350 0x560ebe458770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f226487e740 4 client.19611 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ebe0e0720 con 0x560ebe456350 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f225b7fe640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe456350 0x560ebe458770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f223affd640 10 client.19611 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:02:29.660 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.648+0000 7f225b7fe640 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe456350 0x560ebe458770 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=81941b3f3ec36dab server_cookie=899faae47c9cac5 in_seq=0 out_seq=0 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f223affd640 1 -- 172.21.15.3:0/1815301638 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ebe0e0720 con 0x560ebe456350 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f223affd640 10 client.19611 handle_command_reply: tid=0 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 client.19611 shutdown 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 msgr2=0x560ebe4439b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4439b0 secure :-1 s=READY pgs=1626 cs=0 l=1 rev1=1 crypto rx=0x7f2254011460 tx=0x7f2254011920 comp rx=0 tx=0).stop 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 shutdown_connections 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe456350 0x560ebe458770 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43d3e0 0x560ebe443470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4439b0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 --2- 172.21.15.3:0/1815301638 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43ca30 0x560ebe442f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f223affd640 0 client.19611 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 >> 172.21.15.3:0/1815301638 conn(0x560ebe437100 msgr2=0x560ebe438e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 shutdown_connections 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 1 -- 172.21.15.3:0/1815301638 wait complete. 2024-02-09T23:02:29.661 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.649+0000 7f226487e740 20 client.19611 trim_cache size 0 max 16384 2024-02-09T23:02:29.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 395276 3.39 MB/sec execute 503 sec latency 17.109 ms 2024-02-09T23:02:29.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 Processor -- start 2024-02-09T23:02:29.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 -- start start 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43ca30 0x560ebe445660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe4374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4437d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe329130 con 0x560ebe445f60 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe432a40 con 0x560ebe43ca30 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f226487e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe441480 con 0x560ebe43d3e0 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f225bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4437d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f225b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43ca30 0x560ebe445660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f225affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe4374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f225bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4437d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35586/0 (socket says 172.21.15.3:35586) 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.662+0000 7f225bfff640 1 -- 172.21.15.3:0/3282492408 learned_addr learned my addr 172.21.15.3:0/3282492408 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225bfff640 1 -- 172.21.15.3:0/3282492408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 msgr2=0x560ebe4374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225bfff640 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe4374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225bfff640 1 -- 172.21.15.3:0/3282492408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43ca30 msgr2=0x560ebe445660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225bfff640 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43ca30 0x560ebe445660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225bfff640 1 -- 172.21.15.3:0/3282492408 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ebe33ec90 con 0x560ebe445f60 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225b7fe640 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43ca30 0x560ebe445660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f225bfff640 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4437d0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f2254000f80 tx=0x7f2254005cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3794343e6793d1e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f22417fa640 1 -- 172.21.15.3:0/3282492408 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254019610 con 0x560ebe445f60 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f22417fa640 1 -- 172.21.15.3:0/3282492408 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2254019770 con 0x560ebe445f60 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f22417fa640 1 -- 172.21.15.3:0/3282492408 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540129e0 con 0x560ebe445f60 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f226487e740 1 -- 172.21.15.3:0/3282492408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 msgr2=0x560ebe4437d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.663+0000 7f226487e740 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4437d0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f2254000f80 tx=0x7f2254005cf0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- 172.21.15.3:0/3282492408 shutdown_connections 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe445f60 0x560ebe4437d0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe4374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 --2- 172.21.15.3:0/3282492408 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe43ca30 0x560ebe445660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- 172.21.15.3:0/3282492408 >> 172.21.15.3:0/3282492408 conn(0x560ebe437100 msgr2=0x560ebe43d810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- 172.21.15.3:0/3282492408 shutdown_connections 2024-02-09T23:02:29.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- 172.21.15.3:0/3282492408 wait complete. 2024-02-09T23:02:29.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- start start 2024-02-09T23:02:29.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 Processor -- start 2024-02-09T23:02:29.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 0x560ebe44d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe44b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe44b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe44c040 con 0x560ebe43ca30 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe44c1b0 con 0x560ebe445f60 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ebe4ac860 con 0x560ebe43d3e0 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f226487e740 10 client.0 _ll_register_callbacks cb 0x7f2250005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f225b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 0x560ebe44d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f225b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 0x560ebe44d090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35588/0 (socket says 172.21.15.3:35588) 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f225b7fe640 1 -- 172.21.15.3:0/2754737261 learned_addr learned my addr 172.21.15.3:0/2754737261 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f225affd640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe44b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.664+0000 7f225bfff640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe44b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f225b7fe640 1 -- 172.21.15.3:0/2754737261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 msgr2=0x560ebe44b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f225b7fe640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe44b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f225b7fe640 1 -- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 msgr2=0x560ebe44b970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.673 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f225b7fe640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe44b970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f225b7fe640 1 -- 172.21.15.3:0/2754737261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c017070 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f225b7fe640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 0x560ebe44d090 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f224c013010 tx=0x7f224c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69e989399ebbcc01 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f223affd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f223affd640 1 -- 172.21.15.3:0/2754737261 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224c024040 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f223affd640 1 -- 172.21.15.3:0/2754737261 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f224c003e60 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22500c0d70 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f223affd640 1 -- 172.21.15.3:0/2754737261 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f224c01f640 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f226487e740 10 client.19617 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x560ebe3a26b0 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.665+0000 7f223affd640 1 -- 172.21.15.3:0/2754737261 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22500c0d70 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f223affd640 1 -- 172.21.15.3:0/2754737261 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f224c033a40 con 0x560ebe43ca30 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 10 client.19617 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 10 client.19617 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:02:29.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 20 client.19617 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe4adf10 0x560ebe4b0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 4 client.19617 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560ebe12f890 con 0x560ebe4adf10 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f225affd640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe4adf10 0x560ebe4b0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f225affd640 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe4adf10 0x560ebe4b0330 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d5e06b37a9c01210 server_cookie=e8300fa835eb3edd in_seq=0 out_seq=0 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f223affd640 10 client.19617 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f223affd640 1 -- 172.21.15.3:0/2754737261 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560ebe12f890 con 0x560ebe4adf10 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f223affd640 10 client.19617 handle_command_reply: tid=0 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.666+0000 7f226487e740 1 client.19617 shutdown 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 msgr2=0x560ebe44d090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 0x560ebe44d090 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f224c013010 tx=0x7f224c004730 comp rx=0 tx=0).stop 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 shutdown_connections 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x560ebe4adf10 0x560ebe4b0330 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560ebe445f60 0x560ebe44b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.675 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560ebe43d3e0 0x560ebe44b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f223affd640 0 client.19617 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 --2- 172.21.15.3:0/2754737261 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560ebe43ca30 0x560ebe44d090 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 >> 172.21.15.3:0/2754737261 conn(0x560ebe437100 msgr2=0x560ebe4374e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 shutdown_connections 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 1 -- 172.21.15.3:0/2754737261 wait complete. 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f226487e740 20 client.19617 trim_cache size 0 max 16384 2024-02-09T23:02:29.676 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22440a2260 msgr2=0x7f22440a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22440a2260 0x7f22440a4720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f224c009bd0 tx=0x7f224c005f90 comp rx=0 tx=0).stop 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 msgr2=0x7f225c102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 0x7f225c102310 secure :-1 s=READY pgs=1624 cs=0 l=1 rev1=1 crypto rx=0x7f2250009230 tx=0x7f2250004c00 comp rx=0 tx=0).stop 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 shutdown_connections 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f22440a2260 0x7f22440a4720 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.677 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f225c102850 0x7f225c1a5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f225c105410 0x7f225c102310 unknown :-1 s=CLOSED pgs=1624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 --2- 172.21.15.3:0/1415213400 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f225c100340 0x7f225c101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 >> 172.21.15.3:0/1415213400 conn(0x7f225c0772e0 msgr2=0x7f225c0688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 shutdown_connections 2024-02-09T23:02:29.678 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.667+0000 7f2261ca4640 1 -- 172.21.15.3:0/1415213400 wait complete. 2024-02-09T23:02:29.681 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:02:29.681 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:02:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 -- 172.21.15.3:0/370955060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1960102450 msgr2=0x7f1960102830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 --2- 172.21.15.3:0/370955060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1960102450 0x7f1960102830 secure :-1 s=READY pgs=1629 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002bcd0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 -- 172.21.15.3:0/370955060 shutdown_connections 2024-02-09T23:02:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 --2- 172.21.15.3:0/370955060 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102d70 0x7f196010b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 --2- 172.21.15.3:0/370955060 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f196010b820 0x7f196010dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.794 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 --2- 172.21.15.3:0/370955060 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1960102450 0x7f1960102830 unknown :-1 s=CLOSED pgs=1629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 -- 172.21.15.3:0/370955060 >> 172.21.15.3:0/370955060 conn(0x7f19600772e0 msgr2=0x7f19600776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 -- 172.21.15.3:0/370955060 shutdown_connections 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 -- 172.21.15.3:0/370955060 wait complete. 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 Processor -- start 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.789+0000 7f1965a8f640 1 -- start start 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f1965a8f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102450 0x7f19601000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f1965a8f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1960102d70 0x7f19601005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f1965a8f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 0x7f1960100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f1965a8f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19601a5bf0 con 0x7f196010b820 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f1965a8f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19601a5d60 con 0x7f1960102450 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f1965a8f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19601a5f20 con 0x7f1960102d70 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 0x7f1960100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102450 0x7f19601000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.795 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 0x7f1960100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35618/0 (socket says 172.21.15.3:35618) 2024-02-09T23:02:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 -- 172.21.15.3:0/257627692 learned_addr learned my addr 172.21.15.3:0/257627692 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195effd640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1960102d70 0x7f19601005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 -- 172.21.15.3:0/257627692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1960102d70 msgr2=0x7f19601005e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1960102d70 0x7f19601005e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 -- 172.21.15.3:0/257627692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102450 msgr2=0x7f19601000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.796 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102450 0x7f19601000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 -- 172.21.15.3:0/257627692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1950009560 con 0x7f196010b820 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195f7fe640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102450 0x7f19601000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195ffff640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 0x7f1960100b20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f195800ea60 tx=0x7f195800ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.790+0000 7f195cff9640 1 -- 172.21.15.3:0/257627692 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195800ce40 con 0x7f196010b820 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.791+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19601a6200 con 0x7f196010b820 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.791+0000 7f195cff9640 1 -- 172.21.15.3:0/257627692 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1958005020 con 0x7f196010b820 2024-02-09T23:02:29.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.791+0000 7f195cff9640 1 -- 172.21.15.3:0/257627692 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1958010690 con 0x7f196010b820 2024-02-09T23:02:29.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.791+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19601a6710 con 0x7f196010b820 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f195cff9640 1 -- 172.21.15.3:0/257627692 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1958005500 con 0x7f196010b820 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 Processor -- start 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 -- start start 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c75e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 0x556e65c7e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c7ef60 0x556e65c813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65c31020 con 0x556e65c763e0 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65be4320 con 0x556e65c7ef60 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f196866e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65be1850 con 0x556e65c75a30 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f195f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 0x556e65c7e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f195f7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 0x556e65c7e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35630/0 (socket says 172.21.15.3:35630) 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f195f7fe640 1 -- 172.21.15.3:0/2237802552 learned_addr learned my addr 172.21.15.3:0/2237802552 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.793+0000 7f195effd640 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c75e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195effd640 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c7ef60 0x556e65c813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195cff9640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19480a2260 0x7f19480a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195f7fe640 1 -- 172.21.15.3:0/2237802552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 msgr2=0x556e65c75e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c75e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195f7fe640 1 -- 172.21.15.3:0/2237802552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c7ef60 msgr2=0x556e65c813b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c7ef60 0x556e65c813b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195f7fe640 1 -- 172.21.15.3:0/2237802552 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e65b9f210 con 0x556e65c763e0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195ffff640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19480a2260 0x7f19480a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195effd640 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c7ef60 0x556e65c813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195cff9640 1 -- 172.21.15.3:0/257627692 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f1958014070 con 0x7f196010b820 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 0x556e65c7e990 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13d36407e74b584d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f1945ffb640 1 -- 172.21.15.3:0/2237802552 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195002c880 con 0x556e65c763e0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f195ffff640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19480a2260 0x7f19480a4720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f19580972f0 tx=0x7f195806df00 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f1945ffb640 1 -- 172.21.15.3:0/2237802552 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f195002c9e0 con 0x556e65c763e0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.794+0000 7f1945ffb640 1 -- 172.21.15.3:0/2237802552 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1950005540 con 0x556e65c763e0 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 -- 172.21.15.3:0/2237802552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 msgr2=0x556e65c7e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 0x556e65c7e990 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7f1950009900 tx=0x7f195002c720 comp rx=0 tx=0).stop 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 -- 172.21.15.3:0/2237802552 shutdown_connections 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c7ef60 0x556e65c813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c763e0 0x556e65c7e990 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 --2- 172.21.15.3:0/2237802552 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c75e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 -- 172.21.15.3:0/2237802552 >> 172.21.15.3:0/2237802552 conn(0x556e65c70100 msgr2=0x556e65c704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 -- 172.21.15.3:0/2237802552 shutdown_connections 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 -- 172.21.15.3:0/2237802552 wait complete. 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.795+0000 7f196866e740 1 -- start start 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 Processor -- start 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 0x556e65c7be90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c7ef60 0x556e65c7c910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65bdd030 con 0x556e65c75a30 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e658b0430 con 0x556e65c763e0 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e6586ce00 con 0x556e65c7ef60 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f196866e740 10 client.0 _ll_register_callbacks cb 0x556e65bd2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f195effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 0x556e65c7be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f195f7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f195effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 0x556e65c7be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35642/0 (socket says 172.21.15.3:35642) 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f195effd640 1 -- 172.21.15.3:0/4249328482 learned_addr learned my addr 172.21.15.3:0/4249328482 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.796+0000 7f195effd640 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c7ef60 0x556e65c7c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f195effd640 1 -- 172.21.15.3:0/4249328482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c7ef60 msgr2=0x556e65c7c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f195effd640 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c7ef60 0x556e65c7c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f195effd640 1 -- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 msgr2=0x556e65c7c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f195effd640 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f195effd640 1 -- 172.21.15.3:0/4249328482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196010e8d0 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f195effd640 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 0x556e65c7be90 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f1954010f60 tx=0x7f19540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44069ec20d287674 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f193effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f193effd640 1 -- 172.21.15.3:0/4249328482 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954004260 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f193effd640 1 -- 172.21.15.3:0/4249328482 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f19540043c0 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556e658dc7e0 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f193effd640 1 -- 172.21.15.3:0/4249328482 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f19540035f0 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f196866e740 10 client.19629 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x556e65bdb6b0 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.797+0000 7f193effd640 1 -- 172.21.15.3:0/4249328482 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x556e658dc7e0 con 0x556e65c75a30 2024-02-09T23:02:29.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f193effd640 1 -- 172.21.15.3:0/4249328482 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f195400e740 con 0x556e65c75a30 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f196866e740 10 client.19629 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f196866e740 10 client.19629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f196866e740 20 client.19629 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f196866e740 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65c8f310 0x556e65c91700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f196866e740 4 client.19629 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556e65919720 con 0x556e65c8f310 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f195f7fe640 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65c8f310 0x556e65c91700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f195f7fe640 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65c8f310 0x556e65c91700 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be518ddc3f2da7a6 server_cookie=4a34dd8c00a8012f in_seq=0 out_seq=0 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.798+0000 7f193effd640 10 client.19629 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f193effd640 1 -- 172.21.15.3:0/4249328482 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556e65919720 con 0x556e65c8f310 2024-02-09T23:02:29.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f193effd640 10 client.19629 handle_command_reply: tid=0 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 client.19629 shutdown 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 msgr2=0x556e65c7be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 0x556e65c7be90 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7f1954010f60 tx=0x7f19540046c0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 shutdown_connections 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65c8f310 0x556e65c91700 crc :-1 s=READY pgs=311 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c7ef60 0x556e65c7c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 --2- 172.21.15.3:0/4249328482 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c75a30 0x556e65c7be90 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f193effd640 0 client.19629 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 >> 172.21.15.3:0/4249328482 conn(0x556e65c70100 msgr2=0x556e65c71e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 shutdown_connections 2024-02-09T23:02:29.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 1 -- 172.21.15.3:0/4249328482 wait complete. 2024-02-09T23:02:29.813 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.799+0000 7f196866e740 20 client.19629 trim_cache size 0 max 16384 2024-02-09T23:02:29.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 Processor -- start 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 -- start start 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c7e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c7aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65b62130 con 0x556e65c7ef60 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65c75380 con 0x556e65c763e0 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f196866e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65c6ff30 con 0x556e65c75a30 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f195effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c7aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f195effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c7aca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35646/0 (socket says 172.21.15.3:35646) 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f195effd640 1 -- 172.21.15.3:0/1476847801 learned_addr learned my addr 172.21.15.3:0/1476847801 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f195f7fe640 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.812+0000 7f195effd640 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c7e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f195effd640 1 -- 172.21.15.3:0/1476847801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 msgr2=0x556e65c7e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f195effd640 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c7e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f195effd640 1 -- 172.21.15.3:0/1476847801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 msgr2=0x556e65c7a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f195effd640 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f195effd640 1 -- 172.21.15.3:0/1476847801 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556e65b77c90 con 0x556e65c7ef60 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f195effd640 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c7aca0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1954001eb0 tx=0x7f1954005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2339a5c6af600c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f19457fa640 1 -- 172.21.15.3:0/1476847801 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400ecc0 con 0x556e65c7ef60 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f19457fa640 1 -- 172.21.15.3:0/1476847801 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1954026070 con 0x556e65c7ef60 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.813+0000 7f19457fa640 1 -- 172.21.15.3:0/1476847801 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1954005160 con 0x556e65c7ef60 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 -- 172.21.15.3:0/1476847801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 msgr2=0x556e65c7aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c7aca0 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7f1954001eb0 tx=0x7f1954005970 comp rx=0 tx=0).stop 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 -- 172.21.15.3:0/1476847801 shutdown_connections 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c763e0 0x556e65c7a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c7aca0 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 --2- 172.21.15.3:0/1476847801 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c75a30 0x556e65c7e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 -- 172.21.15.3:0/1476847801 >> 172.21.15.3:0/1476847801 conn(0x556e65c70100 msgr2=0x556e65c77170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 -- 172.21.15.3:0/1476847801 shutdown_connections 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 -- 172.21.15.3:0/1476847801 wait complete. 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 -- start start 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.814+0000 7f196866e740 1 Processor -- start 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c75a30 0x556e65c85f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c763e0 0x556e65c842b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c847f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65c84d30 con 0x556e65c7ef60 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65c84ea0 con 0x556e65c75a30 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e65c85060 con 0x556e65c763e0 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 10 client.0 _ll_register_callbacks cb 0x7f19580093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c847f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c847f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35652/0 (socket says 172.21.15.3:35652) 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 -- 172.21.15.3:0/2043498407 learned_addr learned my addr 172.21.15.3:0/2043498407 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c763e0 0x556e65c842b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c75a30 0x556e65c85f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 -- 172.21.15.3:0/2043498407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c763e0 msgr2=0x556e65c842b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c763e0 0x556e65c842b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 -- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c75a30 msgr2=0x556e65c85f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c75a30 0x556e65c85f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 -- 172.21.15.3:0/2043498407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e65bdb6b0 con 0x556e65c7ef60 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c763e0 0x556e65c842b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:29.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f195effd640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c847f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f195400ebf0 tx=0x7f1954010870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ab604897cf161f3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f193effd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f193effd640 1 -- 172.21.15.3:0/2043498407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195400ecc0 con 0x556e65c7ef60 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f193effd640 1 -- 172.21.15.3:0/2043498407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f195400ee20 con 0x556e65c7ef60 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.815+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f195002c9e0 con 0x556e65c7ef60 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f193effd640 1 -- 172.21.15.3:0/2043498407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f195400e3b0 con 0x556e65c7ef60 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 10 client.19635 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x556e65b9da70 con 0x556e65c7ef60 2024-02-09T23:02:29.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f193effd640 1 -- 172.21.15.3:0/2043498407 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f195002c9e0 con 0x556e65c7ef60 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f193effd640 1 -- 172.21.15.3:0/2043498407 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f19540046c0 con 0x556e65c7ef60 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 10 client.19635 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 10 client.19635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 20 client.19635 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65ce68d0 0x556e65ce8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 4 client.19635 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556e65968890 con 0x556e65ce68d0 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.816+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65ce68d0 0x556e65ce8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.817+0000 7f193effd640 10 client.19635 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.817+0000 7f195f7fe640 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65ce68d0 0x556e65ce8cf0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c639ad68d0d7ec server_cookie=80efec6b0b5767bb in_seq=0 out_seq=0 2024-02-09T23:02:29.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.818+0000 7f193effd640 1 -- 172.21.15.3:0/2043498407 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556e65968890 con 0x556e65ce68d0 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.818+0000 7f193effd640 10 client.19635 handle_command_reply: tid=0 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.818+0000 7f196866e740 1 client.19635 shutdown 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 msgr2=0x556e65c847f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c847f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7f195400ebf0 tx=0x7f1954010870 comp rx=0 tx=0).stop 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 shutdown_connections 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x556e65ce68d0 0x556e65ce8cf0 crc :-1 s=READY pgs=312 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x556e65c7ef60 0x556e65c847f0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x556e65c763e0 0x556e65c842b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 --2- 172.21.15.3:0/2043498407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x556e65c75a30 0x556e65c85f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f193effd640 0 client.19635 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:29.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 >> 172.21.15.3:0/2043498407 conn(0x556e65c70100 msgr2=0x556e65c7a320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 shutdown_connections 2024-02-09T23:02:29.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 1 -- 172.21.15.3:0/2043498407 wait complete. 2024-02-09T23:02:29.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.819+0000 7f196866e740 20 client.19635 trim_cache size 0 max 16384 2024-02-09T23:02:29.833 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19480a2260 msgr2=0x7f19480a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19480a2260 0x7f19480a4720 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f19580972f0 tx=0x7f195806df00 comp rx=0 tx=0).stop 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 msgr2=0x7f1960100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 0x7f1960100b20 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7f195800ea60 tx=0x7f195800ef20 comp rx=0 tx=0).stop 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 shutdown_connections 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f19480a2260 0x7f19480a4720 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f196010b820 0x7f1960100b20 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.835 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1960102d70 0x7f19601005e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 --2- 172.21.15.3:0/257627692 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1960102450 0x7f19601000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:29.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 >> 172.21.15.3:0/257627692 conn(0x7f19600772e0 msgr2=0x7f19600fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:29.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 shutdown_connections 2024-02-09T23:02:29.836 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:29.820+0000 7f1965a8f640 1 -- 172.21.15.3:0/257627692 wait complete. 2024-02-09T23:02:29.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:29 smithi064 ceph-mon[36638]: pgmap v352: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-02-09T23:02:29.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:29 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3200207100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:02:29.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:29 smithi089 ceph-mon[36665]: pgmap v352: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-02-09T23:02:29.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:29 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3200207100' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:02:30.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 395459 3.39 MB/sec execute 504 sec latency 823.924 ms 2024-02-09T23:02:30.837 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:02:30.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:02:30.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:02:30.862 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:30 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:02:30.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:30 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:02:30.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 -- 172.21.15.3:0/753691498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb624100080 msgr2=0x7fb62410e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 --2- 172.21.15.3:0/753691498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb624100080 0x7fb62410e050 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009cd0 tx=0x7fb61c02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 -- 172.21.15.3:0/753691498 shutdown_connections 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 --2- 172.21.15.3:0/753691498 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb6240ff6c0 0x7fb6240ffb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 --2- 172.21.15.3:0/753691498 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb624100080 0x7fb62410e050 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 --2- 172.21.15.3:0/753691498 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6241061a0 0x7fb624106580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 -- 172.21.15.3:0/753691498 >> 172.21.15.3:0/753691498 conn(0x7fb624077680 msgr2=0x7fb6240fd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 -- 172.21.15.3:0/753691498 shutdown_connections 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.944+0000 7fb62902a640 1 -- 172.21.15.3:0/753691498 wait complete. 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 Processor -- start 2024-02-09T23:02:30.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 -- start start 2024-02-09T23:02:30.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 0x7fb6241008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb624100080 0x7fb624100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6241061a0 0x7fb624101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6241a5c20 con 0x7fb6240ff6c0 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6241a5d90 con 0x7fb624100080 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb62902a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6241a5f50 con 0x7fb6241061a0 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 0x7fb6241008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 0x7fb6241008a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35670/0 (socket says 172.21.15.3:35670) 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.945+0000 7fb622d9d640 1 -- 172.21.15.3:0/3443431474 learned_addr learned my addr 172.21.15.3:0/3443431474 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb62259c640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb624100080 0x7fb624100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb62359e640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6241061a0 0x7fb624101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb622d9d640 1 -- 172.21.15.3:0/3443431474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6241061a0 msgr2=0x7fb624101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb622d9d640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6241061a0 0x7fb624101320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb622d9d640 1 -- 172.21.15.3:0/3443431474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb624100080 msgr2=0x7fb624100de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb622d9d640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb624100080 0x7fb624100de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb622d9d640 1 -- 172.21.15.3:0/3443431474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c009900 con 0x7fb6240ff6c0 2024-02-09T23:02:30.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb622d9d640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 0x7fb6241008a0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c920 tx=0x7fb61400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:30.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb60bfff640 1 -- 172.21.15.3:0/3443431474 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614007c70 con 0x7fb6240ff6c0 2024-02-09T23:02:30.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb60bfff640 1 -- 172.21.15.3:0/3443431474 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb614007dd0 con 0x7fb6240ff6c0 2024-02-09T23:02:30.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6241a6230 con 0x7fb6240ff6c0 2024-02-09T23:02:30.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb60bfff640 1 -- 172.21.15.3:0/3443431474 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614018860 con 0x7fb6240ff6c0 2024-02-09T23:02:30.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.946+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6241a6740 con 0x7fb6240ff6c0 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.948+0000 7fb60bfff640 1 -- 172.21.15.3:0/3443431474 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb6140189c0 con 0x7fb6240ff6c0 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 Processor -- start 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 -- start start 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71190 0x56378cd71570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd7a150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd7ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378ccdf000 con 0x56378cd71ab0 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378ccdc530 con 0x56378cd7a690 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62bc00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378ccd7d10 con 0x56378cd71190 2024-02-09T23:02:30.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd7a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd7a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35676/0 (socket says 172.21.15.3:35676) 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 -- 172.21.15.3:0/1799875100 learned_addr learned my addr 172.21.15.3:0/1799875100 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb60bfff640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2230 0x7fb60c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62259c640 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd7ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62359e640 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71190 0x56378cd71570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb62359e640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2230 0x7fb60c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb60bfff640 1 -- 172.21.15.3:0/3443431474 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb6140ce110 con 0x7fb6240ff6c0 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 -- 172.21.15.3:0/1799875100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71190 msgr2=0x56378cd71570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71190 0x56378cd71570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 -- 172.21.15.3:0/1799875100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 msgr2=0x56378cd7ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd7ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.949+0000 7fb622d9d640 1 -- 172.21.15.3:0/1799875100 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56378cc7e210 con 0x56378cd71ab0 2024-02-09T23:02:30.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.950+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd7a150 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fb61409ab30 tx=0x7fb61400fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c58a1be0c69814 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.950+0000 7fb62359e640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2230 0x7fb60c0a46f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb61c006010 tx=0x7fb61c0095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.950+0000 7fb6097fa640 1 -- 172.21.15.3:0/1799875100 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61409a500 con 0x56378cd71ab0 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.950+0000 7fb6097fa640 1 -- 172.21.15.3:0/1799875100 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb61409b2a0 con 0x56378cd71ab0 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.950+0000 7fb6097fa640 1 -- 172.21.15.3:0/1799875100 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6140cbb80 con 0x56378cd71ab0 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- 172.21.15.3:0/1799875100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 msgr2=0x56378cd7a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd7a150 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7fb61409ab30 tx=0x7fb61400fec0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- 172.21.15.3:0/1799875100 shutdown_connections 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd7ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd7a150 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1799875100 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71190 0x56378cd71570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- 172.21.15.3:0/1799875100 >> 172.21.15.3:0/1799875100 conn(0x56378cd6b8c0 msgr2=0x56378cd6bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- 172.21.15.3:0/1799875100 shutdown_connections 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- 172.21.15.3:0/1799875100 wait complete. 2024-02-09T23:02:30.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- start start 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 Processor -- start 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71ab0 0x56378cd77c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd781c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cc7ce60 con 0x56378cd71190 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c9ab4b0 con 0x56378cd7a690 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cc5d670 con 0x56378cd71ab0 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb62bc00740 10 client.0 _ll_register_callbacks cb 0x56378cc75190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35680/0 (socket says 172.21.15.3:35680) 2024-02-09T23:02:30.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.951+0000 7fb622d9d640 1 -- 172.21.15.3:0/1244470393 learned_addr learned my addr 172.21.15.3:0/1244470393 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb62259c640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71ab0 0x56378cd77c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb62359e640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb622d9d640 1 -- 172.21.15.3:0/1244470393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71ab0 msgr2=0x56378cd77c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71ab0 0x56378cd77c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb622d9d640 1 -- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 msgr2=0x56378cd781c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd781c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb622d9d640 1 -- 172.21.15.3:0/1244470393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6241a6740 con 0x56378cd71190 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb62359e640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd781c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77740 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fb614097570 tx=0x7fb6140990b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ae88c5e274a440 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb606ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb606ff5640 1 -- 172.21.15.3:0/1244470393 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61409ecf0 con 0x56378cd71190 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb606ff5640 1 -- 172.21.15.3:0/1244470393 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb614018c70 con 0x56378cd71190 2024-02-09T23:02:30.963 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56378c9d7860 con 0x56378cd71190 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb606ff5640 1 -- 172.21.15.3:0/1244470393 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb614018e70 con 0x56378cd71190 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb62bc00740 10 client.19647 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56378ccd6390 con 0x56378cd71190 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.952+0000 7fb606ff5640 1 -- 172.21.15.3:0/1244470393 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56378c9d7860 con 0x56378cd71190 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb606ff5640 1 -- 172.21.15.3:0/1244470393 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb614016020 con 0x56378cd71190 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62bc00740 10 client.19647 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62bc00740 10 client.19647 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62bc00740 20 client.19647 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cd8a2b0 0x56378cd8c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62bc00740 4 client.19647 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56378ca28410 con 0x56378cd8a2b0 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62259c640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cd8a2b0 0x56378cd8c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.964 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb606ff5640 10 client.19647 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.953+0000 7fb62259c640 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cd8a2b0 0x56378cd8c6a0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50eb7725e20a3c96 server_cookie=92a41e1a5edaa51f in_seq=0 out_seq=0 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb606ff5640 1 -- 172.21.15.3:0/1244470393 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56378ca28410 con 0x56378cd8a2b0 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb606ff5640 10 client.19647 handle_command_reply: tid=0 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 client.19647 shutdown 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 msgr2=0x56378cd77740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77740 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7fb614097570 tx=0x7fb6140990b0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 shutdown_connections 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cd8a2b0 0x56378cd8c6a0 crc :-1 s=READY pgs=313 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd7a690 0x56378cd781c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb606ff5640 0 client.19647 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd71ab0 0x56378cd77c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1244470393 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77740 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 >> 172.21.15.3:0/1244470393 conn(0x56378cd6b8c0 msgr2=0x56378cd6d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:30.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 shutdown_connections 2024-02-09T23:02:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 1 -- 172.21.15.3:0/1244470393 wait complete. 2024-02-09T23:02:30.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.954+0000 7fb62bc00740 20 client.19647 trim_cache size 0 max 16384 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 Processor -- start 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 -- start start 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71ab0 0x56378cd814e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cd81a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cd91880 con 0x56378cd71190 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cd76680 con 0x56378cd71ab0 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62bc00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cd732a0 con 0x56378cd7a690 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35690/0 (socket says 172.21.15.3:35690) 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 -- 172.21.15.3:0/1570482056 learned_addr learned my addr 172.21.15.3:0/1570482056 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:30.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62259c640 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71ab0 0x56378cd814e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62359e640 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cd81a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 -- 172.21.15.3:0/1570482056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 msgr2=0x56378cd81a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cd81a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 -- 172.21.15.3:0/1570482056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71ab0 msgr2=0x56378cd814e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71ab0 0x56378cd814e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb622d9d640 1 -- 172.21.15.3:0/1570482056 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56378cc72c70 con 0x56378cd71190 2024-02-09T23:02:30.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.970+0000 7fb62259c640 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71ab0 0x56378cd814e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:30.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.971+0000 7fb622d9d640 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77fe0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fb61400cfd0 tx=0x7fb61400fdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e6850590b472258 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:30.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.971+0000 7fb608ff9640 1 -- 172.21.15.3:0/1570482056 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614099b30 con 0x56378cd71190 2024-02-09T23:02:30.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.971+0000 7fb608ff9640 1 -- 172.21.15.3:0/1570482056 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb61409a500 con 0x56378cd71190 2024-02-09T23:02:30.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.971+0000 7fb608ff9640 1 -- 172.21.15.3:0/1570482056 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614018c70 con 0x56378cd71190 2024-02-09T23:02:30.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:30 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.971+0000 7fb62bc00740 1 -- 172.21.15.3:0/1570482056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 msgr2=0x56378cd77fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.971+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77fe0 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7fb61400cfd0 tx=0x7fb61400fdc0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- 172.21.15.3:0/1570482056 shutdown_connections 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71ab0 0x56378cd814e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cd81a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 --2- 172.21.15.3:0/1570482056 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71190 0x56378cd77fe0 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- 172.21.15.3:0/1570482056 >> 172.21.15.3:0/1570482056 conn(0x56378cd6b8c0 msgr2=0x56378cd6a090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- 172.21.15.3:0/1570482056 shutdown_connections 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- 172.21.15.3:0/1570482056 wait complete. 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- start start 2024-02-09T23:02:30.982 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 Processor -- start 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71190 0x56378cd80dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd81300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cdd9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cdda050 con 0x56378cd71ab0 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cdda1c0 con 0x56378cd71190 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378cdda380 con 0x56378cd7a690 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62bc00740 10 client.0 _ll_register_callbacks cb 0x7fb614096ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd81300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd81300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35700/0 (socket says 172.21.15.3:35700) 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.972+0000 7fb62259c640 1 -- 172.21.15.3:0/4126953153 learned_addr learned my addr 172.21.15.3:0/4126953153 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb622d9d640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71190 0x56378cd80dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62359e640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cdd9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.983 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62259c640 1 -- 172.21.15.3:0/4126953153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 msgr2=0x56378cdd9b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62259c640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cdd9b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62259c640 1 -- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71190 msgr2=0x56378cd80dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62259c640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71190 0x56378cd80dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62259c640 1 -- 172.21.15.3:0/4126953153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb618010430 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62259c640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd81300 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fb618009e50 tx=0x7fb618005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a70786952223ec6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb606ff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb606ff5640 1 -- 172.21.15.3:0/4126953153 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb618017070 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb606ff5640 1 -- 172.21.15.3:0/4126953153 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb618004230 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb61409b2a0 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb606ff5640 1 -- 172.21.15.3:0/4126953153 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6180043f0 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62bc00740 10 client.19653 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56378ccd6390 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.973+0000 7fb606ff5640 1 -- 172.21.15.3:0/4126953153 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb61409b2a0 con 0x56378cd71ab0 2024-02-09T23:02:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb606ff5640 1 -- 172.21.15.3:0/4126953153 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb618030750 con 0x56378cd71ab0 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb62bc00740 10 client.19653 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb62bc00740 10 client.19653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb62bc00740 20 client.19653 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb62bc00740 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cde2ef0 0x56378cde5310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb62bc00740 4 client.19653 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56378ca147b0 con 0x56378cde2ef0 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb622d9d640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cde2ef0 0x56378cde5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb606ff5640 10 client.19653 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.974+0000 7fb622d9d640 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cde2ef0 0x56378cde5310 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=21fe29426295593e server_cookie=879932a91737767c in_seq=0 out_seq=0 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.975+0000 7fb606ff5640 1 -- 172.21.15.3:0/4126953153 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56378ca147b0 con 0x56378cde2ef0 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.975+0000 7fb606ff5640 10 client.19653 handle_command_reply: tid=0 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.975+0000 7fb62bc00740 1 client.19653 shutdown 2024-02-09T23:02:30.985 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 msgr2=0x56378cd81300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd81300 secure :-1 s=READY pgs=1640 cs=0 l=1 rev1=1 crypto rx=0x7fb618009e50 tx=0x7fb618005850 comp rx=0 tx=0).stop 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 shutdown_connections 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56378cde2ef0 0x56378cde5310 crc :-1 s=READY pgs=314 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56378cd7a690 0x56378cdd9b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb606ff5640 0 client.19653 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56378cd71ab0 0x56378cd81300 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 --2- 172.21.15.3:0/4126953153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56378cd71190 0x56378cd80dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 >> 172.21.15.3:0/4126953153 conn(0x56378cd6b8c0 msgr2=0x56378cd6a920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 shutdown_connections 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 1 -- 172.21.15.3:0/4126953153 wait complete. 2024-02-09T23:02:30.986 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62bc00740 20 client.19653 trim_cache size 0 max 16384 2024-02-09T23:02:30.987 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:02:30.987 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:02:30.987 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "106a0732-9ed1-4f85-a0b2-46092fcbfe2a", 2024-02-09T23:02:30.987 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:02:30.987 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2230 msgr2=0x7fb60c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2230 0x7fb60c0a46f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb61c006010 tx=0x7fb61c0095c0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 msgr2=0x7fb6241008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 0x7fb6241008a0 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c920 tx=0x7fb61400cde0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 shutdown_connections 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb60c0a2230 0x7fb60c0a46f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb624100080 0x7fb624100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb6241061a0 0x7fb624101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.988 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 --2- 172.21.15.3:0/3443431474 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb6240ff6c0 0x7fb6241008a0 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:30.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.976+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 >> 172.21.15.3:0/3443431474 conn(0x7fb624077680 msgr2=0x7fb624104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:30.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.977+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 shutdown_connections 2024-02-09T23:02:30.989 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:30.977+0000 7fb62902a640 1 -- 172.21.15.3:0/3443431474 wait complete. 2024-02-09T23:02:30.993 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:02:30.994 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:02:31.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 -- 172.21.15.3:0/3512425513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4100340 msgr2=0x7fc5e4100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/3512425513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4100340 0x7fc5e4100720 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4009900 tx=0x7fc5d402bc30 comp rx=0 tx=0).stop 2024-02-09T23:02:31.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 -- 172.21.15.3:0/3512425513 shutdown_connections 2024-02-09T23:02:31.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/3512425513 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4105410 0x7fc5e410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/3512425513 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100cf0 0x7fc5e4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/3512425513 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4100340 0x7fc5e4100720 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 -- 172.21.15.3:0/3512425513 >> 172.21.15.3:0/3512425513 conn(0x7fc5e40772e0 msgr2=0x7fc5e40776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 -- 172.21.15.3:0/3512425513 shutdown_connections 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 -- 172.21.15.3:0/3512425513 wait complete. 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.146+0000 7fc5ec28d640 1 Processor -- start 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 -- start start 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 0x7fc5e419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4105410 0x7fc5e419dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4197730 0x7fc5e4197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4198200 con 0x7fc5e4100340 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4198370 con 0x7fc5e4197730 2024-02-09T23:02:31.152 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ec28d640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e4198530 con 0x7fc5e4105410 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 0x7fc5e419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5e9828640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4105410 0x7fc5e419dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 0x7fc5e419d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35708/0 (socket says 172.21.15.3:35708) 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 -- 172.21.15.3:0/4134355339 learned_addr learned my addr 172.21.15.3:0/4134355339 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4197730 0x7fc5e4197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 -- 172.21.15.3:0/4134355339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4105410 msgr2=0x7fc5e419dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4105410 0x7fc5e419dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 -- 172.21.15.3:0/4134355339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4197730 msgr2=0x7fc5e4197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4197730 0x7fc5e4197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 -- 172.21.15.3:0/4134355339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e0009490 con 0x7fc5e4100340 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5e9828640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4105410 0x7fc5e419dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.147+0000 7fc5ea029640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 0x7fc5e419d670 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4005940 tx=0x7fc5d402c670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:31.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.148+0000 7fc5db7fe640 1 -- 172.21.15.3:0/4134355339 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40042d0 con 0x7fc5e4100340 2024-02-09T23:02:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.148+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d4009560 con 0x7fc5e4100340 2024-02-09T23:02:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.148+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e41a4810 con 0x7fc5e4100340 2024-02-09T23:02:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.148+0000 7fc5db7fe640 1 -- 172.21.15.3:0/4134355339 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5d4004430 con 0x7fc5e4100340 2024-02-09T23:02:31.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.148+0000 7fc5db7fe640 1 -- 172.21.15.3:0/4134355339 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d4005350 con 0x7fc5e4100340 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.150+0000 7fc5db7fe640 1 -- 172.21.15.3:0/4134355339 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc5d40054b0 con 0x7fc5e4100340 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.150+0000 7fc5eee5c740 1 Processor -- start 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.150+0000 7fc5eee5c740 1 -- start start 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 0x55833aaf4d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab00300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833aaafd00 con 0x55833aafdeb0 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833aa642f0 con 0x55833aaf5330 2024-02-09T23:02:31.161 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833aa61820 con 0x55833aaf4980 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5db7fe640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5d00a2230 0x7fc5d00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5ea82a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 0x55833aaf4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5ea82a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 0x55833aaf4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53362/0 (socket says 172.21.15.3:53362) 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5e9828640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab00300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5ea82a640 1 -- 172.21.15.3:0/544597009 learned_addr learned my addr 172.21.15.3:0/544597009 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5d00a2230 0x7fc5d00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5db7fe640 1 -- 172.21.15.3:0/4134355339 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc5d40efcc0 con 0x7fc5e4100340 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5e9828640 1 -- 172.21.15.3:0/544597009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 msgr2=0x55833aaf4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5e9828640 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 0x55833aaf4d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5e9828640 1 -- 172.21.15.3:0/544597009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 msgr2=0x55833aafd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5e9828640 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5e9828640 1 -- 172.21.15.3:0/544597009 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55833aa1eff0 con 0x55833aafdeb0 2024-02-09T23:02:31.162 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.151+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 0x55833aaf4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.152+0000 7fc5e9828640 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab00300 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc5e406a920 tx=0x7fc5e0011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b71c5590edfcb7ee server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.152+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5d00a2230 0x7fc5d00a46f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009fd0 tx=0x7fc5dc004ab0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.152+0000 7fc5d8bfe640 1 -- 172.21.15.3:0/544597009 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0004020 con 0x55833aafdeb0 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.152+0000 7fc5d8bfe640 1 -- 172.21.15.3:0/544597009 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5e0005500 con 0x55833aafdeb0 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.152+0000 7fc5d8bfe640 1 -- 172.21.15.3:0/544597009 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5e0004ce0 con 0x55833aafdeb0 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- 172.21.15.3:0/544597009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 msgr2=0x55833ab00300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab00300 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7fc5e406a920 tx=0x7fc5e0011940 comp rx=0 tx=0).stop 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- 172.21.15.3:0/544597009 shutdown_connections 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab00300 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/544597009 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf4980 0x55833aaf4d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- 172.21.15.3:0/544597009 >> 172.21.15.3:0/544597009 conn(0x55833aaef050 msgr2=0x55833aaef430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- 172.21.15.3:0/544597009 shutdown_connections 2024-02-09T23:02:31.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- 172.21.15.3:0/544597009 wait complete. 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- start start 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 Processor -- start 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833aafd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833aafb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833aa5d000 con 0x55833aaf4980 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833a730430 con 0x55833aaf5330 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833a6ece00 con 0x55833aafdeb0 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5eee5c740 10 client.0 _ll_register_callbacks cb 0x55833aa517f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833aafd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.153+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833aafd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35718/0 (socket says 172.21.15.3:35718) 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 -- 172.21.15.3:0/1647744229 learned_addr learned my addr 172.21.15.3:0/1647744229 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5e9828640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.164 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833aafb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 -- 172.21.15.3:0/1647744229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 msgr2=0x55833aafb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833aafb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 -- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 msgr2=0x55833aafac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 -- 172.21.15.3:0/1647744229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e0009d90 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5ea029640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833aafd880 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4004f00 tx=0x7fc5d4005760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eda075134c4e9805 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5cdffb640 1 -- 172.21.15.3:0/1647744229 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40f1070 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5cdffb640 1 -- 172.21.15.3:0/1647744229 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc5d40c0910 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55833a75c7e0 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5cdffb640 1 -- 172.21.15.3:0/1647744229 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5d40c4f60 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5eee5c740 10 client.19665 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55833aa5b680 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.154+0000 7fc5cdffb640 1 -- 172.21.15.3:0/1647744229 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55833a75c7e0 con 0x55833aaf4980 2024-02-09T23:02:31.165 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5cdffb640 1 -- 172.21.15.3:0/1647744229 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc5d403c070 con 0x55833aaf4980 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5eee5c740 10 client.19665 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5eee5c740 10 client.19665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5eee5c740 20 client.19665 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab0dfb0 0x55833ab103c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5eee5c740 4 client.19665 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55833a799720 con 0x55833ab0dfb0 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5e9828640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab0dfb0 0x55833ab103c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5e9828640 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab0dfb0 0x55833ab103c0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6770bd14978b97a server_cookie=6eacd4596ea0e824 in_seq=0 out_seq=0 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.155+0000 7fc5cdffb640 10 client.19665 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5cdffb640 1 -- 172.21.15.3:0/1647744229 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55833a799720 con 0x55833ab0dfb0 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5cdffb640 10 client.19665 handle_command_reply: tid=0 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 client.19665 shutdown 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 msgr2=0x55833aafd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.166 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833aafd880 secure :-1 s=READY pgs=1643 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4004f00 tx=0x7fc5d4005760 comp rx=0 tx=0).stop 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 shutdown_connections 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab0dfb0 0x55833ab103c0 crc :-1 s=READY pgs=315 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833aafac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833aafb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/1647744229 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833aafd880 unknown :-1 s=CLOSED pgs=1643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 >> 172.21.15.3:0/1647744229 conn(0x55833aaef050 msgr2=0x55833aaf0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5cdffb640 0 client.19665 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 shutdown_connections 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 1 -- 172.21.15.3:0/1647744229 wait complete. 2024-02-09T23:02:31.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.156+0000 7fc5eee5c740 20 client.19665 trim_cache size 0 max 16384 2024-02-09T23:02:31.178 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.174+0000 7fc5eee5c740 1 Processor -- start 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.174+0000 7fc5eee5c740 1 -- start start 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf4980 0x55833aafd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 0x55833aafd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab04ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833a9e1fd0 con 0x55833aafdeb0 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833aaefc20 con 0x55833aaf4980 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ab0a650 con 0x55833aaf5330 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab04ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5e9828640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 0x55833aafd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab04ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35734/0 (socket says 172.21.15.3:35734) 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5e9828640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 0x55833aafd880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:53384/0 (socket says 172.21.15.3:53384) 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 -- 172.21.15.3:0/3301414604 learned_addr learned my addr 172.21.15.3:0/3301414604 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea029640 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf4980 0x55833aafd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.179 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 -- 172.21.15.3:0/3301414604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 msgr2=0x55833aafd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 0x55833aafd880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 -- 172.21.15.3:0/3301414604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf4980 msgr2=0x55833aafd340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf4980 0x55833aafd340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 -- 172.21.15.3:0/3301414604 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55833a9f7b30 con 0x55833aafdeb0 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5e9828640 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 0x55833aafd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea029640 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf4980 0x55833aafd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab04ee0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc013400 tx=0x7fc5dc005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22290cf1e17df8a5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.175+0000 7fc5cffff640 1 -- 172.21.15.3:0/3301414604 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc023b60 con 0x55833aafdeb0 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5cffff640 1 -- 172.21.15.3:0/3301414604 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5dc023cc0 con 0x55833aafdeb0 2024-02-09T23:02:31.180 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5cffff640 1 -- 172.21.15.3:0/3301414604 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc02b640 con 0x55833aafdeb0 2024-02-09T23:02:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 -- 172.21.15.3:0/3301414604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 msgr2=0x55833ab04ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab04ee0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc013400 tx=0x7fc5dc005bd0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 -- 172.21.15.3:0/3301414604 shutdown_connections 2024-02-09T23:02:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aafdeb0 0x55833ab04ee0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aaf5330 0x55833aafd880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.184 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/3301414604 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf4980 0x55833aafd340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 -- 172.21.15.3:0/3301414604 >> 172.21.15.3:0/3301414604 conn(0x55833aaef050 msgr2=0x55833aaf5ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 -- 172.21.15.3:0/3301414604 shutdown_connections 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.176+0000 7fc5eee5c740 1 -- 172.21.15.3:0/3301414604 wait complete. 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 -- start start 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 Processor -- start 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833ab05140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833ab034e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833ab03a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ab04000 con 0x55833aaf4980 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ab04170 con 0x55833aaf5330 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55833ab04330 con 0x55833aafdeb0 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5eee5c740 10 client.0 _ll_register_callbacks cb 0x7fc5e0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833ab05140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833ab05140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35740/0 (socket says 172.21.15.3:35740) 2024-02-09T23:02:31.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 -- 172.21.15.3:0/474010209 learned_addr learned my addr 172.21.15.3:0/474010209 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea82a640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833ab03a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5e9828640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833ab034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 -- 172.21.15.3:0/474010209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 msgr2=0x55833ab03a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833ab03a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 -- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 msgr2=0x55833ab034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833ab034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 -- 172.21.15.3:0/474010209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55833aa1eff0 con 0x55833aaf4980 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5ea029640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833ab05140 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40039c0 tx=0x7fc5d40bcc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b098cc7740a0d1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.177+0000 7fc5cdffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5cdffb640 1 -- 172.21.15.3:0/474010209 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5d40c4cd0 con 0x55833aaf4980 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5cdffb640 1 -- 172.21.15.3:0/474010209 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fc5d40c0910 con 0x55833aaf4980 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc5e0005500 con 0x55833aaf4980 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5cdffb640 1 -- 172.21.15.3:0/474010209 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5d40bc120 con 0x55833aaf4980 2024-02-09T23:02:31.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 10 client.19671 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x7fc5dc013170 con 0x55833aaf4980 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5cdffb640 1 -- 172.21.15.3:0/474010209 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5e0005500 con 0x55833aaf4980 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5cdffb640 1 -- 172.21.15.3:0/474010209 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc5d403c050 con 0x55833aaf4980 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 10 client.19671 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 10 client.19671 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 20 client.19671 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab65630 0x55833ab67a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 4 client.19671 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.178+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55833a7e8890 con 0x55833ab65630 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5e9828640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab65630 0x55833ab67a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5cdffb640 10 client.19671 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5e9828640 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab65630 0x55833ab67a20 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c78e268decd52c3b server_cookie=d09241c6a1f2fd4f in_seq=0 out_seq=0 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5cdffb640 1 -- 172.21.15.3:0/474010209 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55833a7e8890 con 0x55833ab65630 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5cdffb640 10 client.19671 handle_command_reply: tid=0 2024-02-09T23:02:31.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 client.19671 shutdown 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 msgr2=0x55833ab05140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833ab05140 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7fc5d40039c0 tx=0x7fc5d40bcc50 comp rx=0 tx=0).stop 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 shutdown_connections 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55833ab65630 0x55833ab67a20 crc :-1 s=READY pgs=316 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55833aaf5330 0x55833ab034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55833aafdeb0 0x55833ab03a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5cdffb640 0 client.19671 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 --2- 172.21.15.3:0/474010209 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55833aaf4980 0x55833ab05140 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 >> 172.21.15.3:0/474010209 conn(0x55833aaef050 msgr2=0x55833aaed960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 shutdown_connections 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 1 -- 172.21.15.3:0/474010209 wait complete. 2024-02-09T23:02:31.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.179+0000 7fc5eee5c740 20 client.19671 trim_cache size 0 max 16384 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "106a0732-9ed1-4f85-a0b2-46092fcbfe2a": { 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "106a0732-9ed1-4f85-a0b2-46092fcbfe2a", 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:02:31.189 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5d00a2230 msgr2=0x7fc5d00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5d00a2230 0x7fc5d00a46f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc009fd0 tx=0x7fc5dc004ab0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 msgr2=0x7fc5e419d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 0x7fc5e419d670 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7fc5d4005940 tx=0x7fc5d402c670 comp rx=0 tx=0).stop 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 shutdown_connections 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5d00a2230 0x7fc5d00a46f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc5e4197730 0x7fc5e4197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc5e4105410 0x7fc5e419dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 --2- 172.21.15.3:0/4134355339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc5e4100340 0x7fc5e419d670 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 >> 172.21.15.3:0/4134355339 conn(0x7fc5e40772e0 msgr2=0x7fc5e40688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 shutdown_connections 2024-02-09T23:02:31.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:02:31.180+0000 7fc5ec28d640 1 -- 172.21.15.3:0/4134355339 wait complete. 2024-02-09T23:02:31.194 INFO:tasks.cephfs.filesystem:scrub status for tag:106a0732-9ed1-4f85-a0b2-46092fcbfe2a - {'path': '/', 'tag': '106a0732-9ed1-4f85-a0b2-46092fcbfe2a', 'options': 'recursive,force'} 2024-02-09T23:02:31.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 396941 3.40 MB/sec execute 505 sec latency 1154.694 ms 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: pgmap v353: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 253 op/s 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:02:31.883 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:31 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: pgmap v353: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 253 op/s 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:02:32.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:31 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: pgmap v353: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 2.3 MiB/s wr, 253 op/s 2024-02-09T23:02:32.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.228 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.229 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:32.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:02:32.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:02:32.230 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:31 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:02:32.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 398420 3.40 MB/sec execute 506 sec latency 28.359 ms 2024-02-09T23:02:32.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:32 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:02:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:32 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:32 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:32 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:02:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:32 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:32 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:32 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:02:33.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:32 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:32 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:33.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:32 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:02:33.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:32 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.186 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:32 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:32 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:02:33.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:32 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:32 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:33.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:32 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:02:33.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:32 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:32 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:02:33.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 399001 3.40 MB/sec execute 507 sec latency 706.562 ms 2024-02-09T23:02:34.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:33 smithi003 ceph-mon[32678]: pgmap v354: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-02-09T23:02:34.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:33 smithi064 ceph-mon[36638]: pgmap v354: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-02-09T23:02:34.227 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:33 smithi089 ceph-mon[36665]: pgmap v354: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-02-09T23:02:34.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 399001 3.40 MB/sec execute 508 sec latency 1706.657 ms 2024-02-09T23:02:35.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 399001 3.39 MB/sec execute 509 sec latency 2706.778 ms 2024-02-09T23:02:36.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:36 smithi003 ceph-mon[32678]: pgmap v355: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-02-09T23:02:36.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 399001 3.38 MB/sec execute 510 sec latency 3706.837 ms 2024-02-09T23:02:36.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:36 smithi064 ceph-mon[36638]: pgmap v355: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-02-09T23:02:36.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:36 smithi089 ceph-mon[36665]: pgmap v355: 97 pgs: 97 active+clean; 154 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 172 op/s 2024-02-09T23:02:37.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:37 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:37.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 401110 3.40 MB/sec execute 511 sec latency 3737.261 ms 2024-02-09T23:02:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:37 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:37 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:38.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:38 smithi003 ceph-mon[32678]: pgmap v356: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-02-09T23:02:38.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:38 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:02:38.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 402527 3.41 MB/sec execute 512 sec latency 13.488 ms 2024-02-09T23:02:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:38 smithi064 ceph-mon[36638]: pgmap v356: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-02-09T23:02:38.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:38 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:02:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:38 smithi089 ceph-mon[36665]: pgmap v356: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-02-09T23:02:38.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:38 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:02:39.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 402551 3.40 MB/sec execute 513 sec latency 993.537 ms 2024-02-09T23:02:40.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:40 smithi003 ceph-mon[32678]: pgmap v357: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-02-09T23:02:40.664 INFO:tasks.workunit.client.0.smithi003.stdout: 1 404031 3.41 MB/sec execute 514 sec latency 1431.796 ms 2024-02-09T23:02:40.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:40 smithi064 ceph-mon[36638]: pgmap v357: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-02-09T23:02:40.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:40 smithi089 ceph-mon[36665]: pgmap v357: 97 pgs: 97 active+clean; 157 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.8 MiB/s wr, 177 op/s 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.869+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/1967662493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4105410 msgr2=0x7fa6a410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.869+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/1967662493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4105410 0x7fa6a410cfe0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009cd0 tx=0x7fa69c02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/1967662493 shutdown_connections 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/1967662493 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4105410 0x7fa6a410cfe0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/1967662493 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a4100cf0 0x7fa6a4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/1967662493 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6a4100340 0x7fa6a4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/1967662493 >> 172.21.15.3:0/1967662493 conn(0x7fa6a4077220 msgr2=0x7fa6a4077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/1967662493 shutdown_connections 2024-02-09T23:02:40.873 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/1967662493 wait complete. 2024-02-09T23:02:40.875 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.870+0000 7fa6ab7a1640 1 Processor -- start 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 -- start start 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 0x7fa6a4198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6a4105410 0x7fa6a4198bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a4199150 0x7fa6a41a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41a3da0 con 0x7fa6a4100340 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41a3f10 con 0x7fa6a4105410 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6ab7a1640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a41a40d0 con 0x7fa6a4199150 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 0x7fa6a4198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 0x7fa6a4198680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51392/0 (socket says 172.21.15.3:51392) 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 -- 172.21.15.3:0/3565397269 learned_addr learned my addr 172.21.15.3:0/3565397269 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6a4105410 0x7fa6a4198bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a4199150 0x7fa6a41a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 -- 172.21.15.3:0/3565397269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a4199150 msgr2=0x7fa6a41a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a4199150 0x7fa6a41a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 -- 172.21.15.3:0/3565397269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6a4105410 msgr2=0x7fa6a4198bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6a4105410 0x7fa6a4198bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 -- 172.21.15.3:0/3565397269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c009900 con 0x7fa6a4100340 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.871+0000 7fa6a953d640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 0x7fa6a4198680 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000e9f0 tx=0x7fa6a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.872+0000 7fa69a7fc640 1 -- 172.21.15.3:0/3565397269 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0009700 con 0x7fa6a4100340 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.872+0000 7fa69a7fc640 1 -- 172.21.15.3:0/3565397269 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6a000cd80 con 0x7fa6a4100340 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.872+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a41a43b0 con 0x7fa6a4100340 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.872+0000 7fa69a7fc640 1 -- 172.21.15.3:0/3565397269 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a0018920 con 0x7fa6a4100340 2024-02-09T23:02:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.872+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a41a4910 con 0x7fa6a4100340 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.874+0000 7fa69a7fc640 1 -- 172.21.15.3:0/3565397269 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa6a0018a80 con 0x7fa6a4100340 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.874+0000 7fa69a7fc640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa6900a22a0 0x7fa6900a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.874+0000 7fa69a7fc640 1 -- 172.21.15.3:0/3565397269 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa6a0014070 con 0x7fa6a4100340 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.874+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa6900a22a0 0x7fa6900a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 Processor -- start 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 -- start start 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa72490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa7b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa6900a22a0 0x7fa6900a4760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4069570 tx=0x7fa694005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa7db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaa9e0a00 con 0x55baaaa729d0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaa9ddf30 con 0x55baaaa720b0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6ae379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaa9d9710 con 0x55baaaa7b7a0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa7b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa7b260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51396/0 (socket says 172.21.15.3:51396) 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/479051235 learned_addr learned my addr 172.21.15.3:0/479051235 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a953d640 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa72490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa7db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/479051235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 msgr2=0x55baaaa7db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa7db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/479051235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 msgr2=0x55baaaa72490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa72490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/479051235 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baaa99e3b0 con 0x55baaaa729d0 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.875+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa7b260 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa694005f60 tx=0x7fa694014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3971070e714933e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa68f7fe640 1 -- 172.21.15.3:0/479051235 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694003d70 con 0x55baaaa729d0 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa68f7fe640 1 -- 172.21.15.3:0/479051235 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6940044d0 con 0x55baaaa729d0 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa68f7fe640 1 -- 172.21.15.3:0/479051235 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69401e420 con 0x55baaaa729d0 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 -- 172.21.15.3:0/479051235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 msgr2=0x55baaaa7b260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa7b260 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fa694005f60 tx=0x7fa694014000 comp rx=0 tx=0).stop 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 -- 172.21.15.3:0/479051235 shutdown_connections 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa7db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa7b260 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 --2- 172.21.15.3:0/479051235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa72490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.876+0000 7fa6ae379740 1 -- 172.21.15.3:0/479051235 >> 172.21.15.3:0/479051235 conn(0x55baaaa6c9d0 msgr2=0x55baaaa6cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.877+0000 7fa6ae379740 1 -- 172.21.15.3:0/479051235 shutdown_connections 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.877+0000 7fa6ae379740 1 -- 172.21.15.3:0/479051235 wait complete. 2024-02-09T23:02:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.877+0000 7fa6ae379740 1 -- start start 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.877+0000 7fa6ae379740 1 Processor -- start 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 0x55baaaa787f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa78d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa7b7a0 0x55baaaa79270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaa97eb90 con 0x55baaaa729d0 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaa6ad250 con 0x55baaaa7b7a0 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaa95f200 con 0x55baaaa720b0 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 10 client.0 _ll_register_callbacks cb 0x55baaa9955d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a8d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa78d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 0x55baaaa787f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 0x55baaaa787f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:55596/0 (socket says 172.21.15.3:55596) 2024-02-09T23:02:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 -- 172.21.15.3:0/4181654013 learned_addr learned my addr 172.21.15.3:0/4181654013 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa7b7a0 0x55baaaa79270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a8d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa78d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51408/0 (socket says 172.21.15.3:51408) 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 -- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa7b7a0 msgr2=0x55baaaa79270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa7b7a0 0x55baaaa79270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 -- 172.21.15.3:0/4181654013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 msgr2=0x55baaaa78d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa78d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 -- 172.21.15.3:0/4181654013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa694017070 con 0x55baaaa720b0 2024-02-09T23:02:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6a953d640 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 0x55baaaa787f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0097260 tx=0x7fa6a009af10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ca340b6f1501194 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa68cff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa68cff9640 1 -- 172.21.15.3:0/4181654013 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6a009e070 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa68cff9640 1 -- 172.21.15.3:0/4181654013 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa6a00974c0 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55baaa9ffee0 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa68cff9640 1 -- 172.21.15.3:0/4181654013 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a0097680 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 10 client.28663 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.878+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55baaa9d7d90 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa68cff9640 1 -- 172.21.15.3:0/4181654013 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55baaa9ffee0 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa68cff9640 1 -- 172.21.15.3:0/4181654013 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa6a00b3bf0 con 0x55baaaa720b0 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6ae379740 10 client.28663 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6ae379740 10 client.28663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6ae379740 20 client.28663 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6ae379740 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaa8b9d0 0x55baaaa8ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6ae379740 4 client.28663 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55baaa6d9600 con 0x55baaaa8b9d0 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.879+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaa8b9d0 0x55baaaa8ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa68cff9640 10 client.28663 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaa8b9d0 0x55baaaa8ddc0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4941841fb38da158 server_cookie=463ffb970c3e4d4e in_seq=0 out_seq=0 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa68cff9640 1 -- 172.21.15.3:0/4181654013 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55baaa6d9600 con 0x55baaaa8b9d0 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa68cff9640 10 client.28663 handle_command_reply: tid=0 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 client.28663 shutdown 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 msgr2=0x55baaaa787f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 0x55baaaa787f0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa6a0097260 tx=0x7fa6a009af10 comp rx=0 tx=0).stop 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 shutdown_connections 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaa8b9d0 0x55baaaa8ddc0 crc :-1 s=READY pgs=317 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa7b7a0 0x55baaaa79270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa78d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.890 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa68cff9640 0 client.28663 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 --2- 172.21.15.3:0/4181654013 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa720b0 0x55baaaa787f0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 >> 172.21.15.3:0/4181654013 conn(0x55baaaa6c9d0 msgr2=0x55baaaa6e720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.880+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 shutdown_connections 2024-02-09T23:02:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.881+0000 7fa6ae379740 1 -- 172.21.15.3:0/4181654013 wait complete. 2024-02-09T23:02:40.891 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.881+0000 7fa6ae379740 20 client.28663 trim_cache size 0 max 16384 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 Processor -- start 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 -- start start 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa7af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa729d0 0x55baaaa79090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 0x55baaaa770c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaaa6c1b0 con 0x55baaaa7b7a0 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaaa682a0 con 0x55baaaa720b0 2024-02-09T23:02:40.900 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6ae379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaaa76d40 con 0x55baaaa729d0 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 0x55baaaa770c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 0x55baaaa770c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51424/0 (socket says 172.21.15.3:51424) 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 -- 172.21.15.3:0/1672025131 learned_addr learned my addr 172.21.15.3:0/1672025131 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa729d0 0x55baaaa79090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a953d640 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa7af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 -- 172.21.15.3:0/1672025131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa729d0 msgr2=0x55baaaa79090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa729d0 0x55baaaa79090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.901 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 -- 172.21.15.3:0/1672025131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 msgr2=0x55baaaa7af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa7af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.895+0000 7fa6a9d3e640 1 -- 172.21.15.3:0/1672025131 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55baaa974820 con 0x55baaaa7b7a0 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.896+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 0x55baaaa770c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009cd0 tx=0x7fa69c032eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e12d2a1d33b2fec server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.896+0000 7fa68effd640 1 -- 172.21.15.3:0/1672025131 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c0323c0 con 0x55baaaa7b7a0 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.896+0000 7fa68effd640 1 -- 172.21.15.3:0/1672025131 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa69c034a90 con 0x55baaaa7b7a0 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.896+0000 7fa68effd640 1 -- 172.21.15.3:0/1672025131 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69c003bd0 con 0x55baaaa7b7a0 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 -- 172.21.15.3:0/1672025131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 msgr2=0x55baaaa770c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 0x55baaaa770c0 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fa69c009cd0 tx=0x7fa69c032eb0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 -- 172.21.15.3:0/1672025131 shutdown_connections 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa7b7a0 0x55baaaa770c0 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa729d0 0x55baaaa79090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 --2- 172.21.15.3:0/1672025131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa7af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.902 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 -- 172.21.15.3:0/1672025131 >> 172.21.15.3:0/1672025131 conn(0x55baaaa6c9d0 msgr2=0x55baaaa73b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 -- 172.21.15.3:0/1672025131 shutdown_connections 2024-02-09T23:02:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 -- 172.21.15.3:0/1672025131 wait complete. 2024-02-09T23:02:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 -- start start 2024-02-09T23:02:40.903 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.897+0000 7fa6ae379740 1 Processor -- start 2024-02-09T23:02:40.908 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa82600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa80a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa80fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaaa81510 con 0x55baaaa729d0 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaaa81680 con 0x55baaaa720b0 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55baaaa81840 con 0x55baaaa7b7a0 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 10 client.0 _ll_register_callbacks cb 0x7fa6a000e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:40.909 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa80a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa80a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:51438/0 (socket says 172.21.15.3:51438) 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a953d640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa82600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/2826597430 learned_addr learned my addr 172.21.15.3:0/2826597430 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a9d3e640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa80fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/2826597430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 msgr2=0x55baaaa80fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa80fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 msgr2=0x55baaaa82600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa82600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 -- 172.21.15.3:0/2826597430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a410d7f0 con 0x55baaaa729d0 2024-02-09T23:02:40.910 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a953d640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa82600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6a8d3c640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa80a90 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fa694004b00 tx=0x7fa694011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91f76d73ce8a60cd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa68cff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa68cff9640 1 -- 172.21.15.3:0/2826597430 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa694030040 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa68cff9640 1 -- 172.21.15.3:0/2826597430 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa694003d00 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.898+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6940044d0 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa68cff9640 1 -- 172.21.15.3:0/2826597430 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa694004700 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 10 client.19689 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55baaa99a6f0 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa68cff9640 1 -- 172.21.15.3:0/2826597430 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6940044d0 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa68cff9640 1 -- 172.21.15.3:0/2826597430 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa694024030 con 0x55baaaa729d0 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 10 client.19689 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 10 client.19689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 20 client.19689 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:40.911 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaae3480 0x55baaaae58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 4 client.19689 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:02:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55baaa72a1b0 con 0x55baaaae3480 2024-02-09T23:02:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.899+0000 7fa6a953d640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaae3480 0x55baaaae58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.900+0000 7fa68cff9640 10 client.19689 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:40.912 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:40.900+0000 7fa6a953d640 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaae3480 0x55baaaae58a0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=327c9491d80a5f4a server_cookie=38badbb984742d9c in_seq=0 out_seq=0 2024-02-09T23:02:41.010 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa68cff9640 1 -- 172.21.15.3:0/2826597430 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55baaa72a1b0 con 0x55baaaae3480 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa68cff9640 10 client.19689 handle_command_reply: tid=0 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 client.19689 shutdown 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 msgr2=0x55baaaa80a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa80a90 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fa694004b00 tx=0x7fa694011ca0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 shutdown_connections 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55baaaae3480 0x55baaaae58a0 crc :-1 s=READY pgs=318 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55baaaa7b7a0 0x55baaaa80fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa68cff9640 0 client.19689 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55baaaa729d0 0x55baaaa80a90 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 --2- 172.21.15.3:0/2826597430 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55baaaa720b0 0x55baaaa82600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 >> 172.21.15.3:0/2826597430 conn(0x55baaaa6c9d0 msgr2=0x55baaaa77060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:41.011 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 shutdown_connections 2024-02-09T23:02:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.007+0000 7fa6ae379740 1 -- 172.21.15.3:0/2826597430 wait complete. 2024-02-09T23:02:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ae379740 20 client.19689 trim_cache size 0 max 16384 2024-02-09T23:02:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:02:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:02:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:02:41.012 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa6900a22a0 msgr2=0x7fa6900a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa6900a22a0 0x7fa6900a4760 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fa6a4069570 tx=0x7fa694005f90 comp rx=0 tx=0).stop 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 msgr2=0x7fa6a4198680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 0x7fa6a4198680 secure :-1 s=READY pgs=1647 cs=0 l=1 rev1=1 crypto rx=0x7fa6a000e9f0 tx=0x7fa6a000eeb0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 shutdown_connections 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa6900a22a0 0x7fa6900a4760 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa6a4105410 0x7fa6a4198bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa6a4199150 0x7fa6a41a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 --2- 172.21.15.3:0/3565397269 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa6a4100340 0x7fa6a4198680 unknown :-1 s=CLOSED pgs=1647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:41.013 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 >> 172.21.15.3:0/3565397269 conn(0x7fa6a4077220 msgr2=0x7fa6a40fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 shutdown_connections 2024-02-09T23:02:41.014 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:41.008+0000 7fa6ab7a1640 1 -- 172.21.15.3:0/3565397269 wait complete. 2024-02-09T23:02:41.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 405201 3.42 MB/sec execute 515 sec latency 97.821 ms 2024-02-09T23:02:42.615 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:42 smithi003 ceph-mon[32678]: pgmap v358: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-02-09T23:02:42.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 406092 3.42 MB/sec execute 516 sec latency 370.738 ms 2024-02-09T23:02:42.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:42 smithi064 ceph-mon[36638]: pgmap v358: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-02-09T23:02:42.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:42 smithi089 ceph-mon[36665]: pgmap v358: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.6 MiB/s wr, 244 op/s 2024-02-09T23:02:43.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 407661 3.43 MB/sec execute 517 sec latency 767.512 ms 2024-02-09T23:02:44.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:44 smithi003 ceph-mon[32678]: pgmap v359: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-02-09T23:02:44.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 408927 3.43 MB/sec execute 518 sec latency 11.837 ms 2024-02-09T23:02:44.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:44 smithi064 ceph-mon[36638]: pgmap v359: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-02-09T23:02:44.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:44 smithi089 ceph-mon[36665]: pgmap v359: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-02-09T23:02:45.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:45 smithi003 ceph-mon[32678]: pgmap v360: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-02-09T23:02:45.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 409641 3.44 MB/sec execute 519 sec latency 468.703 ms 2024-02-09T23:02:45.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:45 smithi064 ceph-mon[36638]: pgmap v360: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-02-09T23:02:45.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:45 smithi089 ceph-mon[36665]: pgmap v360: 97 pgs: 97 active+clean; 160 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 752 KiB/s rd, 1.5 MiB/s wr, 134 op/s 2024-02-09T23:02:46.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 409641 3.43 MB/sec execute 520 sec latency 1468.787 ms 2024-02-09T23:02:47.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 411489 3.45 MB/sec execute 521 sec latency 1511.562 ms 2024-02-09T23:02:47.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:47 smithi003 ceph-mon[32678]: pgmap v361: 97 pgs: 97 active+clean; 161 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 241 op/s 2024-02-09T23:02:47.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:47 smithi064 ceph-mon[36638]: pgmap v361: 97 pgs: 97 active+clean; 161 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 241 op/s 2024-02-09T23:02:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:47 smithi089 ceph-mon[36665]: pgmap v361: 97 pgs: 97 active+clean; 161 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.5 MiB/s wr, 241 op/s 2024-02-09T23:02:48.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 412633 3.45 MB/sec execute 522 sec latency 13.031 ms 2024-02-09T23:02:49.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 413183 3.45 MB/sec execute 523 sec latency 716.211 ms 2024-02-09T23:02:50.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:50 smithi003 ceph-mon[32678]: pgmap v362: 97 pgs: 97 active+clean; 161 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-02-09T23:02:50.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 414505 3.45 MB/sec execute 524 sec latency 1155.429 ms 2024-02-09T23:02:50.679 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:50 smithi089 ceph-mon[36665]: pgmap v362: 97 pgs: 97 active+clean; 161 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-02-09T23:02:50.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:50 smithi064 ceph-mon[36638]: pgmap v362: 97 pgs: 97 active+clean; 161 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.8 MiB/s wr, 173 op/s 2024-02-09T23:02:51.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 415841 3.46 MB/sec execute 525 sec latency 16.847 ms 2024-02-09T23:02:52.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:52 smithi003 ceph-mon[32678]: pgmap v363: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-02-09T23:02:52.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 416732 3.46 MB/sec execute 526 sec latency 574.327 ms 2024-02-09T23:02:52.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:52 smithi064 ceph-mon[36638]: pgmap v363: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-02-09T23:02:52.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:52 smithi089 ceph-mon[36665]: pgmap v363: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.5 MiB/s wr, 242 op/s 2024-02-09T23:02:53.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:53 smithi003 ceph-mon[32678]: pgmap v364: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T23:02:53.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 418832 3.48 MB/sec execute 527 sec latency 692.712 ms 2024-02-09T23:02:53.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:53 smithi064 ceph-mon[36638]: pgmap v364: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T23:02:53.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:53 smithi089 ceph-mon[36665]: pgmap v364: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T23:02:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 -- 172.21.15.3:0/558687029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204105410 msgr2=0x7f220410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 --2- 172.21.15.3:0/558687029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204105410 0x7f220410cfe0 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7f21fc009cd0 tx=0x7f21fc02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 -- 172.21.15.3:0/558687029 shutdown_connections 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 --2- 172.21.15.3:0/558687029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204100cf0 0x7f2204104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 --2- 172.21.15.3:0/558687029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204105410 0x7f220410cfe0 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 --2- 172.21.15.3:0/558687029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204100340 0x7f2204100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 -- 172.21.15.3:0/558687029 >> 172.21.15.3:0/558687029 conn(0x7f2204077220 msgr2=0x7f2204077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 -- 172.21.15.3:0/558687029 shutdown_connections 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.097+0000 7f220bdeb640 1 -- 172.21.15.3:0/558687029 wait complete. 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 Processor -- start 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 -- start start 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 0x7f2204101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 0x7f2204102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204102850 0x7f22041a5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22041a6030 con 0x7f2204100340 2024-02-09T23:02:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22041a61a0 con 0x7f2204102850 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220bdeb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22041a6360 con 0x7f2204105410 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f2209b87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 0x7f2204101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f2209386640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 0x7f2204102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f2209386640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 0x7f2204102310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:47152/0 (socket says 172.21.15.3:47152) 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f2209b87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 0x7f2204101dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37392/0 (socket says 172.21.15.3:37392) 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f2209386640 1 -- 172.21.15.3:0/91311496 learned_addr learned my addr 172.21.15.3:0/91311496 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.098+0000 7f220a388640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204102850 0x7f22041a5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209b87640 1 -- 172.21.15.3:0/91311496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 msgr2=0x7f2204102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209b87640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 0x7f2204102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209b87640 1 -- 172.21.15.3:0/91311496 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204102850 msgr2=0x7f22041a5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209b87640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204102850 0x7f22041a5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209b87640 1 -- 172.21.15.3:0/91311496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc009900 con 0x7f2204100340 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209386640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 0x7f2204102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f220a388640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204102850 0x7f22041a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f2209b87640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 0x7f2204101dd0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f220000e9f0 tx=0x7f220000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f21faffd640 1 -- 172.21.15.3:0/91311496 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220000ccc0 con 0x7f2204100340 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22041a6640 con 0x7f2204100340 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f21faffd640 1 -- 172.21.15.3:0/91311496 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f220000ce20 con 0x7f2204100340 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f21faffd640 1 -- 172.21.15.3:0/91311496 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200004a00 con 0x7f2204100340 2024-02-09T23:02:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.099+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22041a6b00 con 0x7f2204100340 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.101+0000 7f21faffd640 1 -- 172.21.15.3:0/91311496 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f22000050b0 con 0x7f2204100340 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f21faffd640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21f00a2220 0x7f21f00a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 Processor -- start 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f2209386640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21f00a2220 0x7f21f00a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 -- start start 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f21faffd640 1 -- 172.21.15.3:0/91311496 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2200014070 con 0x7f2204100340 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a2347490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2350260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23507a0 0x55e7a2352b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a22b5a00 con 0x55e7a23470b0 2024-02-09T23:02:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a22b2f30 con 0x55e7a23507a0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220e9b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a22ae710 con 0x55e7a23479d0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f2209b87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a2347490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f2209b87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a2347490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37402/0 (socket says 172.21.15.3:37402) 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f2209b87640 1 -- 172.21.15.3:0/3669082183 learned_addr learned my addr 172.21.15.3:0/3669082183 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f220a388640 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23507a0 0x55e7a2352b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f2209386640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21f00a2220 0x7f21f00a46e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2204104e00 tx=0x7f21f4005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.102+0000 7f2209386640 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2350260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209b87640 1 -- 172.21.15.3:0/3669082183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 msgr2=0x55e7a2350260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209b87640 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2350260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209b87640 1 -- 172.21.15.3:0/3669082183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23507a0 msgr2=0x55e7a2352b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209b87640 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23507a0 0x55e7a2352b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209b87640 1 -- 172.21.15.3:0/3669082183 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7a22733b0 con 0x55e7a23470b0 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209386640 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2350260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:02:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f2209b87640 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a2347490 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2200097400 tx=0x7f220009a390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16eb6c0418539e51 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f21effff640 1 -- 172.21.15.3:0/3669082183 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220009eb80 con 0x55e7a23470b0 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f21effff640 1 -- 172.21.15.3:0/3669082183 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22000a31d0 con 0x55e7a23470b0 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.103+0000 7f21effff640 1 -- 172.21.15.3:0/3669082183 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000a2fa0 con 0x55e7a23470b0 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 -- 172.21.15.3:0/3669082183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 msgr2=0x55e7a2347490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a2347490 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f2200097400 tx=0x7f220009a390 comp rx=0 tx=0).stop 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 -- 172.21.15.3:0/3669082183 shutdown_connections 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23507a0 0x55e7a2352b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2350260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 --2- 172.21.15.3:0/3669082183 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a2347490 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 -- 172.21.15.3:0/3669082183 >> 172.21.15.3:0/3669082183 conn(0x55e7a23419d0 msgr2=0x55e7a2341db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 -- 172.21.15.3:0/3669082183 shutdown_connections 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 -- 172.21.15.3:0/3669082183 wait complete. 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 -- start start 2024-02-09T23:02:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.104+0000 7f220e9b6740 1 Processor -- start 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 0x55e7a234d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a234e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a2253b90 con 0x55e7a23507a0 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a1f82250 con 0x55e7a23479d0 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a2234200 con 0x55e7a23470b0 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220e9b6740 10 client.0 _ll_register_callbacks cb 0x55e7a226a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 0x55e7a234d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220a388640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a234e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 0x55e7a234d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:47160/0 (socket says 172.21.15.3:47160) 2024-02-09T23:02:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 -- 172.21.15.3:0/2149477140 learned_addr learned my addr 172.21.15.3:0/2149477140 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f220a388640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a234e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37412/0 (socket says 172.21.15.3:37412) 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209386640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 -- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 msgr2=0x55e7a234dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 -- 172.21.15.3:0/2149477140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 msgr2=0x55e7a234e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a234e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 -- 172.21.15.3:0/2149477140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22041a6b00 con 0x55e7a23470b0 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209386640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f2209b87640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 0x55e7a234d820 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f220009a9c0 tx=0x7f220009a4c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d07f25795508be0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f21ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.105+0000 7f21ed7fa640 1 -- 172.21.15.3:0/2149477140 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22000a2f50 con 0x55e7a23470b0 2024-02-09T23:02:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f21ed7fa640 1 -- 172.21.15.3:0/2149477140 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f2200004b60 con 0x55e7a23470b0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e7a22d4ee0 con 0x55e7a23470b0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f21ed7fa640 1 -- 172.21.15.3:0/2149477140 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2200004d20 con 0x55e7a23470b0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 10 client.28681 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55e7a22acd90 con 0x55e7a23470b0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f21ed7fa640 1 -- 172.21.15.3:0/2149477140 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55e7a22d4ee0 con 0x55e7a23470b0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f21ed7fa640 1 -- 172.21.15.3:0/2149477140 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f22000969b0 con 0x55e7a23470b0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 10 client.28681 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 10 client.28681 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 20 client.28681 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a2360570 0x55e7a2362990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 4 client.28681 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e7a1fae600 con 0x55e7a2360570 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.106+0000 7f2209386640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a2360570 0x55e7a2362990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.107+0000 7f21ed7fa640 10 client.28681 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.107+0000 7f2209386640 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a2360570 0x55e7a2362990 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=694ae624c36ce84c server_cookie=8f349b9f50d9f90b in_seq=0 out_seq=0 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f21ed7fa640 1 -- 172.21.15.3:0/2149477140 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e7a1fae600 con 0x55e7a2360570 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f21ed7fa640 10 client.28681 handle_command_reply: tid=0 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 client.28681 shutdown 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 msgr2=0x55e7a234d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 0x55e7a234d820 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f220009a9c0 tx=0x7f220009a4c0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 shutdown_connections 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a2360570 0x55e7a2362990 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a234e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 --2- 172.21.15.3:0/2149477140 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23470b0 0x55e7a234d820 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f21ed7fa640 0 client.28681 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.108+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 >> 172.21.15.3:0/2149477140 conn(0x55e7a23419d0 msgr2=0x55e7a2343720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.109+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 shutdown_connections 2024-02-09T23:02:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.109+0000 7f220e9b6740 1 -- 172.21.15.3:0/2149477140 wait complete. 2024-02-09T23:02:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.109+0000 7f220e9b6740 20 client.28681 trim_cache size 0 max 16384 2024-02-09T23:02:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 Processor -- start 2024-02-09T23:02:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 -- start start 2024-02-09T23:02:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a234ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.131 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23507a0 0x55e7a234aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a2346890 con 0x55e7a23470b0 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a2346520 con 0x55e7a23479d0 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.122+0000 7f220e9b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a233d2a0 con 0x55e7a23507a0 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a234ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a234ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37414/0 (socket says 172.21.15.3:37414) 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209386640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 -- 172.21.15.3:0/1307602793 learned_addr learned my addr 172.21.15.3:0/1307602793 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f220a388640 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23507a0 0x55e7a234aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 -- 172.21.15.3:0/1307602793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23507a0 msgr2=0x55e7a234aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23507a0 0x55e7a234aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 -- 172.21.15.3:0/1307602793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 msgr2=0x55e7a234e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.132 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 -- 172.21.15.3:0/1307602793 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e7a2249820 con 0x55e7a23470b0 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f2209b87640 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a234ff50 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f22000b90f0 tx=0x7f220000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ee0edb086aa40a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f21ef7fe640 1 -- 172.21.15.3:0/1307602793 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f220009a800 con 0x55e7a23470b0 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f21ef7fe640 1 -- 172.21.15.3:0/1307602793 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22000a2f50 con 0x55e7a23470b0 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.123+0000 7f21ef7fe640 1 -- 172.21.15.3:0/1307602793 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200099e80 con 0x55e7a23470b0 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 -- 172.21.15.3:0/1307602793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 msgr2=0x55e7a234ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a234ff50 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f22000b90f0 tx=0x7f220000cf90 comp rx=0 tx=0).stop 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 -- 172.21.15.3:0/1307602793 shutdown_connections 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23479d0 0x55e7a234e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23507a0 0x55e7a234aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 --2- 172.21.15.3:0/1307602793 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23470b0 0x55e7a234ff50 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 -- 172.21.15.3:0/1307602793 >> 172.21.15.3:0/1307602793 conn(0x55e7a23419d0 msgr2=0x55e7a234c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 -- 172.21.15.3:0/1307602793 shutdown_connections 2024-02-09T23:02:54.133 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 -- 172.21.15.3:0/1307602793 wait complete. 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 -- start start 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.124+0000 7f220e9b6740 1 Processor -- start 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23470b0 0x55e7a23575f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2355a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a2355f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a23564d0 con 0x55e7a23507a0 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a2356610 con 0x55e7a23470b0 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e7a23567d0 con 0x55e7a23479d0 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220e9b6740 10 client.0 _ll_register_callbacks cb 0x7f22000b1070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220a388640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a2355f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209b87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23470b0 0x55e7a23575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.134 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2355a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209b87640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23470b0 0x55e7a23575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:51232/0 (socket says 172.21.15.3:51232) 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209b87640 1 -- 172.21.15.3:0/995543174 learned_addr learned my addr 172.21.15.3:0/995543174 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220a388640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a2355f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:37418/0 (socket says 172.21.15.3:37418) 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 -- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23470b0 msgr2=0x55e7a23575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23470b0 0x55e7a23575f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 -- 172.21.15.3:0/995543174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 msgr2=0x55e7a2355f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a2355f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 -- 172.21.15.3:0/995543174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e7a22acd90 con 0x55e7a23479d0 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f220a388640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a2355f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f2209386640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2355a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55e7a23632a0 tx=0x7f21f4004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23db11a7775c78ff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.125+0000 7f21ed7fa640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f21ed7fa640 1 -- 172.21.15.3:0/995543174 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f4024040 con 0x55e7a23479d0 2024-02-09T23:02:54.135 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f21ed7fa640 1 -- 172.21.15.3:0/995543174 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f21f400f480 con 0x55e7a23479d0 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22000a31d0 con 0x55e7a23479d0 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f21ed7fa640 1 -- 172.21.15.3:0/995543174 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f21f400f640 con 0x55e7a23479d0 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 10 client.28687 fetch_fsmap learned FSMap version 49 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55e7a226f6f0 con 0x55e7a23479d0 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f21ed7fa640 1 -- 172.21.15.3:0/995543174 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22000a31d0 con 0x55e7a23479d0 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f21ed7fa640 1 -- 172.21.15.3:0/995543174 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f21f403c8c0 con 0x55e7a23479d0 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 10 client.28687 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 10 client.28687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 20 client.28687 populate_metadata read hostname 'smithi003' 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a23b7fc0 0x55e7a23ba3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 4 client.28687 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:02:54.136 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e7a1fff1b0 con 0x55e7a23b7fc0 2024-02-09T23:02:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.126+0000 7f2209b87640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a23b7fc0 0x55e7a23ba3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:02:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.127+0000 7f21ed7fa640 10 client.28687 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:54.137 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.127+0000 7f2209b87640 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a23b7fc0 0x55e7a23ba3e0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7713eb041d06bd8b server_cookie=49e3647bb8279096 in_seq=0 out_seq=0 2024-02-09T23:02:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f21ed7fa640 1 -- 172.21.15.3:0/995543174 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e7a1fff1b0 con 0x55e7a23b7fc0 2024-02-09T23:02:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f21ed7fa640 10 client.28687 handle_command_reply: tid=0 2024-02-09T23:02:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 client.28687 shutdown 2024-02-09T23:02:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 msgr2=0x55e7a2355a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2355a50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x55e7a23632a0 tx=0x7f21f4004730 comp rx=0 tx=0).stop 2024-02-09T23:02:54.164 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 shutdown_connections 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55e7a23b7fc0 0x55e7a23ba3e0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55e7a23507a0 0x55e7a2355f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55e7a23479d0 0x55e7a2355a50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 --2- 172.21.15.3:0/995543174 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55e7a23470b0 0x55e7a23575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 >> 172.21.15.3:0/995543174 conn(0x55e7a23419d0 msgr2=0x55e7a23401a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f21ed7fa640 0 client.28687 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 shutdown_connections 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 1 -- 172.21.15.3:0/995543174 wait complete. 2024-02-09T23:02:54.165 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.160+0000 7f220e9b6740 20 client.28687 trim_cache size 0 max 16384 2024-02-09T23:02:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:02:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:02:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:02:54.166 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21f00a2220 msgr2=0x7f21f00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21f00a2220 0x7f21f00a46e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2204104e00 tx=0x7f21f4005f90 comp rx=0 tx=0).stop 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 msgr2=0x7f2204101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 0x7f2204101dd0 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f220000e9f0 tx=0x7f220000eeb0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 shutdown_connections 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f21f00a2220 0x7f21f00a46e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f2204102850 0x7f22041a5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f2204105410 0x7f2204102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 --2- 172.21.15.3:0/91311496 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f2204100340 0x7f2204101dd0 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 >> 172.21.15.3:0/91311496 conn(0x7f2204077220 msgr2=0x7f2204068d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:02:54.167 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 shutdown_connections 2024-02-09T23:02:54.168 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:02:54.161+0000 7f220bdeb640 1 -- 172.21.15.3:0/91311496 wait complete. 2024-02-09T23:02:54.665 INFO:tasks.workunit.client.0.smithi003.stdout: 1 420089 3.49 MB/sec execute 528 sec latency 15.884 ms 2024-02-09T23:02:55.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 420275 3.48 MB/sec execute 529 sec latency 806.572 ms 2024-02-09T23:02:56.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:56 smithi003 ceph-mon[32678]: pgmap v365: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T23:02:56.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 420275 3.47 MB/sec execute 530 sec latency 1806.673 ms 2024-02-09T23:02:56.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:56 smithi064 ceph-mon[36638]: pgmap v365: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T23:02:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:56 smithi089 ceph-mon[36665]: pgmap v365: 97 pgs: 97 active+clean; 165 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.7 MiB/s wr, 175 op/s 2024-02-09T23:02:57.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:02:57 smithi003 ceph-mon[32678]: pgmap v366: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 287 op/s 2024-02-09T23:02:57.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 422391 3.49 MB/sec execute 531 sec latency 1843.717 ms 2024-02-09T23:02:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:02:57 smithi064 ceph-mon[36638]: pgmap v366: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 287 op/s 2024-02-09T23:02:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:02:57 smithi089 ceph-mon[36665]: pgmap v366: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 287 op/s 2024-02-09T23:02:58.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 423741 3.50 MB/sec execute 532 sec latency 24.150 ms 2024-02-09T23:02:59.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 423824 3.49 MB/sec execute 533 sec latency 955.709 ms 2024-02-09T23:03:00.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:00 smithi003 ceph-mon[32678]: pgmap v367: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-02-09T23:03:00.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 425154 3.50 MB/sec execute 534 sec latency 1393.710 ms 2024-02-09T23:03:00.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:00 smithi064 ceph-mon[36638]: pgmap v367: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-02-09T23:03:00.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:00 smithi089 ceph-mon[36665]: pgmap v367: 97 pgs: 97 active+clean; 162 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 1.9 MiB/s wr, 181 op/s 2024-02-09T23:03:01.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:03:01.195 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:03:01.305 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 -- 172.21.15.3:0/1332288131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc102e00 msgr2=0x7f11bc103260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1332288131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc102e00 0x7f11bc103260 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f11b0009930 tx=0x7f11b002bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 -- 172.21.15.3:0/1332288131 shutdown_connections 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1332288131 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11bc1037a0 0x7f11bc10e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1332288131 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc102e00 0x7f11bc103260 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1332288131 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11bc101520 0x7f11bc101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 -- 172.21.15.3:0/1332288131 >> 172.21.15.3:0/1332288131 conn(0x7f11bc0fccc0 msgr2=0x7f11bc0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 -- 172.21.15.3:0/1332288131 shutdown_connections 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.301+0000 7f11c28a7640 1 -- 172.21.15.3:0/1332288131 wait complete. 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 Processor -- start 2024-02-09T23:03:01.306 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 -- start start 2024-02-09T23:03:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 0x7f11bc19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.308 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11bc102e00 0x7f11bc19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11bc1037a0 0x7f11bc1977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc197f70 con 0x7f11bc101520 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc1980e0 con 0x7f11bc1037a0 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.302+0000 7f11c28a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11bc1982a0 con 0x7f11bc102e00 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 0x7f11bc19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 0x7f11bc19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35842/0 (socket says 172.21.15.3:35842) 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 -- 172.21.15.3:0/1985043130 learned_addr learned my addr 172.21.15.3:0/1985043130 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11c0e44640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11bc1037a0 0x7f11bc1977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11bc102e00 0x7f11bc19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 -- 172.21.15.3:0/1985043130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11bc102e00 msgr2=0x7f11bc19dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11bc102e00 0x7f11bc19dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 -- 172.21.15.3:0/1985043130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11bc1037a0 msgr2=0x7f11bc1977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.309 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11bc1037a0 0x7f11bc1977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 -- 172.21.15.3:0/1985043130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b0009560 con 0x7f11bc101520 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11bbfff640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 0x7f11bc19d690 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00e9f0 tx=0x7f11ac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11b97fa640 1 -- 172.21.15.3:0/1985043130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac009720 con 0x7f11bc101520 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc198580 con 0x7f11bc101520 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11b97fa640 1 -- 172.21.15.3:0/1985043130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11ac00cd80 con 0x7f11bc101520 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11b97fa640 1 -- 172.21.15.3:0/1985043130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac010690 con 0x7f11bc101520 2024-02-09T23:03:01.310 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.303+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc1a4680 con 0x7f11bc101520 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.305+0000 7f11b97fa640 1 -- 172.21.15.3:0/1985043130 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f11ac0046c0 con 0x7f11bc101520 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 Processor -- start 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 -- start start 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11b97fa640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11a40a2230 0x7f11a40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b0715bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b071ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b071f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b06d0d00 con 0x55f2b07157e0 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b06852f0 con 0x55f2b071f2f0 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c5482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0682820 con 0x55f2b0716190 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c0e44640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11a40a2230 0x7f11a40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.317 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11b97fa640 1 -- 172.21.15.3:0/1985043130 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f11ac014070 con 0x7f11bc101520 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b0715bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b0715bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35846/0 (socket says 172.21.15.3:35846) 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 -- 172.21.15.3:0/3733522561 learned_addr learned my addr 172.21.15.3:0/3733522561 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b071ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11c0e44640 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b071f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 -- 172.21.15.3:0/3733522561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 msgr2=0x55f2b071ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b071ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 -- 172.21.15.3:0/3733522561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 msgr2=0x55f2b071f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b071f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.306+0000 7f11bbfff640 1 -- 172.21.15.3:0/3733522561 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2b063fff0 con 0x55f2b07157e0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c0e44640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11a40a2230 0x7f11a40a46f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009fd0 tx=0x7f11b4013b00 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11bbfff640 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b0715bc0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f11ac097280 tx=0x7f11ac09ad20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922d976795aa1be7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11a27fc640 1 -- 172.21.15.3:0/3733522561 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac09b520 con 0x55f2b07157e0 2024-02-09T23:03:01.318 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11a27fc640 1 -- 172.21.15.3:0/3733522561 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11ac048120 con 0x55f2b07157e0 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11a27fc640 1 -- 172.21.15.3:0/3733522561 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac097490 con 0x55f2b07157e0 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 -- 172.21.15.3:0/3733522561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 msgr2=0x55f2b0715bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b0715bc0 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f11ac097280 tx=0x7f11ac09ad20 comp rx=0 tx=0).stop 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 -- 172.21.15.3:0/3733522561 shutdown_connections 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b071f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b071ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 --2- 172.21.15.3:0/3733522561 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b0715bc0 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 -- 172.21.15.3:0/3733522561 >> 172.21.15.3:0/3733522561 conn(0x55f2b0710050 msgr2=0x55f2b0710430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 -- 172.21.15.3:0/3733522561 shutdown_connections 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.307+0000 7f11c5482740 1 -- 172.21.15.3:0/3733522561 wait complete. 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 -- start start 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 Processor -- start 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0722d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.319 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0723290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07237d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b067e000 con 0x55f2b071f2f0 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0351430 con 0x55f2b07157e0 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b030de00 con 0x55f2b0716190 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c5482740 10 client.0 _ll_register_callbacks cb 0x55f2b06727f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c0e44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07237d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c0e44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07237d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35848/0 (socket says 172.21.15.3:35848) 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11c0e44640 1 -- 172.21.15.3:0/952137593 learned_addr learned my addr 172.21.15.3:0/952137593 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11bbfff640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0722d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.308+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0723290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c0e44640 1 -- 172.21.15.3:0/952137593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 msgr2=0x55f2b0723290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c0e44640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0723290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.320 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c0e44640 1 -- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 msgr2=0x55f2b0722d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c0e44640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0722d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c0e44640 1 -- 172.21.15.3:0/952137593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11bc1a4680 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c0e44640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07237d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f11b400fba0 tx=0x7f11b4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c396d1a8b6e77c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f119b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f119b7fe640 1 -- 172.21.15.3:0/952137593 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4010c40 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f119b7fe640 1 -- 172.21.15.3:0/952137593 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f11b4010da0 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f2b037d7e0 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f119b7fe640 1 -- 172.21.15.3:0/952137593 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11b4020400 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 10 client.19719 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55f2b067c680 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f119b7fe640 1 -- 172.21.15.3:0/952137593 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f2b037d7e0 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f119b7fe640 1 -- 172.21.15.3:0/952137593 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f11b400e570 con 0x55f2b071f2f0 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 10 client.19719 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:01.321 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 10 client.19719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 20 client.19719 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b0733810 0x55f2b0735c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 4 client.19719 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.309+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f2b03ba720 con 0x55f2b0733810 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.310+0000 7f11bbfff640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b0733810 0x55f2b0735c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.310+0000 7f11bbfff640 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b0733810 0x55f2b0735c30 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=271336b3a08c55c4 server_cookie=8bb1be93d5e92d9e in_seq=0 out_seq=0 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.310+0000 7f119b7fe640 10 client.19719 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f119b7fe640 1 -- 172.21.15.3:0/952137593 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f2b03ba720 con 0x55f2b0733810 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f119b7fe640 10 client.19719 handle_command_reply: tid=0 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 client.19719 shutdown 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 msgr2=0x55f2b07237d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07237d0 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7f11b400fba0 tx=0x7f11b4004ab0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 shutdown_connections 2024-02-09T23:03:01.322 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b0733810 0x55f2b0735c30 crc :-1 s=READY pgs=321 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07237d0 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0723290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f119b7fe640 0 client.19719 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 --2- 172.21.15.3:0/952137593 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0722d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 >> 172.21.15.3:0/952137593 conn(0x55f2b0710050 msgr2=0x55f2b0711da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 shutdown_connections 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 1 -- 172.21.15.3:0/952137593 wait complete. 2024-02-09T23:03:01.323 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.311+0000 7f11c5482740 20 client.19719 trim_cache size 0 max 16384 2024-02-09T23:03:01.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.326+0000 7f11c5482740 1 Processor -- start 2024-02-09T23:03:01.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.326+0000 7f11c5482740 1 -- start start 2024-02-09T23:03:01.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c5482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0725480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c5482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b07206e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c5482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07235f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.336 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c5482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0602fd0 con 0x55f2b071f2f0 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c5482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0733610 con 0x55f2b07157e0 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c5482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b073ae40 con 0x55f2b0716190 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07235f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07235f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35852/0 (socket says 172.21.15.3:35852) 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 -- 172.21.15.3:0/475791696 learned_addr learned my addr 172.21.15.3:0/475791696 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b07206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11bbfff640 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0725480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 -- 172.21.15.3:0/475791696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 msgr2=0x55f2b07206e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b07206e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 -- 172.21.15.3:0/475791696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 msgr2=0x55f2b0725480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0725480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 -- 172.21.15.3:0/475791696 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f2b0618b30 con 0x55f2b071f2f0 2024-02-09T23:03:01.337 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11bbfff640 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0725480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11c0e44640 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07235f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f11b4010840 tx=0x7f11b40091e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4204a0337384788 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11a1ffb640 1 -- 172.21.15.3:0/475791696 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b400f760 con 0x55f2b071f2f0 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11a1ffb640 1 -- 172.21.15.3:0/475791696 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11b4013460 con 0x55f2b071f2f0 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.327+0000 7f11a1ffb640 1 -- 172.21.15.3:0/475791696 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4004030 con 0x55f2b071f2f0 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 -- 172.21.15.3:0/475791696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 msgr2=0x55f2b07235f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07235f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f11b4010840 tx=0x7f11b40091e0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 -- 172.21.15.3:0/475791696 shutdown_connections 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b071f2f0 0x55f2b07235f0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b07206e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 --2- 172.21.15.3:0/475791696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b07157e0 0x55f2b0725480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 -- 172.21.15.3:0/475791696 >> 172.21.15.3:0/475791696 conn(0x55f2b0710050 msgr2=0x55f2b071daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 -- 172.21.15.3:0/475791696 shutdown_connections 2024-02-09T23:03:01.338 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 -- 172.21.15.3:0/475791696 wait complete. 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 -- start start 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.328+0000 7f11c5482740 1 Processor -- start 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b072a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0728b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b07291a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0729840 con 0x55f2b07157e0 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0729980 con 0x55f2b071f2f0 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f2b0789d20 con 0x55f2b0716190 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 10 client.0 _ll_register_callbacks cb 0x7f11ac00cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:01.339 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b072a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b072a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35858/0 (socket says 172.21.15.3:35858) 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 -- 172.21.15.3:0/424660774 learned_addr learned my addr 172.21.15.3:0/424660774 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0728b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c0e44640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b07291a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 -- 172.21.15.3:0/424660774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 msgr2=0x55f2b0728b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0728b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 -- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 msgr2=0x55f2b07291a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b07291a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 -- 172.21.15.3:0/424660774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f2b067c680 con 0x55f2b07157e0 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11bbfff640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b072a7d0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0977f0 tx=0x7f11ac0971f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7261bda8944fc9c7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f119b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:01.340 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f119b7fe640 1 -- 172.21.15.3:0/424660774 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11ac0b3040 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f119b7fe640 1 -- 172.21.15.3:0/424660774 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f11ac09ae20 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.329+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11ac048120 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f119b7fe640 1 -- 172.21.15.3:0/424660774 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11ac09ef70 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 10 client.19725 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55f2b063e850 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f119b7fe640 1 -- 172.21.15.3:0/424660774 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11ac048120 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f119b7fe640 1 -- 172.21.15.3:0/424660774 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f11ac01f060 con 0x55f2b07157e0 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 10 client.19725 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 10 client.19725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 20 client.19725 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b078b2c0 0x55f2b078d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 4 client.19725 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:03:01.341 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f2b0409890 con 0x55f2b078b2c0 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b078b2c0 0x55f2b078d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f119b7fe640 10 client.19725 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.330+0000 7f11bb7fe640 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b078b2c0 0x55f2b078d6e0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fcfc2a702ab63cc server_cookie=8d929c1b72458f36 in_seq=0 out_seq=0 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f119b7fe640 1 -- 172.21.15.3:0/424660774 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f2b0409890 con 0x55f2b078b2c0 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f119b7fe640 10 client.19725 handle_command_reply: tid=0 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 client.19725 shutdown 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 msgr2=0x55f2b072a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b072a7d0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0977f0 tx=0x7f11ac0971f0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 shutdown_connections 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f2b078b2c0 0x55f2b078d6e0 crc :-1 s=READY pgs=322 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f119b7fe640 0 client.19725 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f2b071f2f0 0x55f2b07291a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.342 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f2b0716190 0x55f2b0728b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 --2- 172.21.15.3:0/424660774 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f2b07157e0 0x55f2b072a7d0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 >> 172.21.15.3:0/424660774 conn(0x55f2b0710050 msgr2=0x55f2b071cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 shutdown_connections 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 1 -- 172.21.15.3:0/424660774 wait complete. 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c5482740 20 client.19725 trim_cache size 0 max 16384 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:03:01.343 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:03:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11a40a2230 msgr2=0x7f11a40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11a40a2230 0x7f11a40a46f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f11b4009fd0 tx=0x7f11b4013b00 comp rx=0 tx=0).stop 2024-02-09T23:03:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 msgr2=0x7f11bc19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.344 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.331+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 0x7f11bc19d690 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f11ac00e9f0 tx=0x7f11ac00edf0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 shutdown_connections 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11a40a2230 0x7f11a40a46f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f11bc1037a0 0x7f11bc1977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f11bc102e00 0x7f11bc19dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 --2- 172.21.15.3:0/1985043130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f11bc101520 0x7f11bc19d690 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 >> 172.21.15.3:0/1985043130 conn(0x7f11bc0fccc0 msgr2=0x7f11bc0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 shutdown_connections 2024-02-09T23:03:01.345 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.332+0000 7f11c28a7640 1 -- 172.21.15.3:0/1985043130 wait complete. 2024-02-09T23:03:01.346 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:03:01.346 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:03:01.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 -- 172.21.15.3:0/3923374830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c102d70 msgr2=0x7fb57c10b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 --2- 172.21.15.3:0/3923374830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c102d70 0x7fb57c10b250 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fb564009900 tx=0x7fb56402bcd0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 -- 172.21.15.3:0/3923374830 shutdown_connections 2024-02-09T23:03:01.455 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 --2- 172.21.15.3:0/3923374830 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb57c10b820 0x7fb57c10dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 --2- 172.21.15.3:0/3923374830 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c102d70 0x7fb57c10b250 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 --2- 172.21.15.3:0/3923374830 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 0x7fb57c102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 -- 172.21.15.3:0/3923374830 >> 172.21.15.3:0/3923374830 conn(0x7fb57c0772b0 msgr2=0x7fb57c0776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 -- 172.21.15.3:0/3923374830 shutdown_connections 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.449+0000 7fb5818a7640 1 -- 172.21.15.3:0/3923374830 wait complete. 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 Processor -- start 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 -- start start 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 0x7fb57c19d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb57c102d70 0x7fb57c19dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 0x7fb57c1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c197e40 con 0x7fb57c10b820 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c197fb0 con 0x7fb57c102d70 2024-02-09T23:03:01.456 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb5818a7640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c198170 con 0x7fb57c102450 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 0x7fb57c19d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 0x7fb57c1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 0x7fb57c1976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35870/0 (socket says 172.21.15.3:35870) 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb573fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb57c102d70 0x7fb57c19dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 0x7fb57c19d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54266/0 (socket says 172.21.15.3:54266) 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 -- 172.21.15.3:0/959430949 learned_addr learned my addr 172.21.15.3:0/959430949 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 -- 172.21.15.3:0/959430949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 msgr2=0x7fb57c19d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 0x7fb57c19d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 -- 172.21.15.3:0/959430949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb57c102d70 msgr2=0x7fb57c19dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb57c102d70 0x7fb57c19dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.450+0000 7fb57b7fe640 1 -- 172.21.15.3:0/959430949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb564009560 con 0x7fb57c10b820 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.451+0000 7fb57b7fe640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 0x7fb57c1976b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fb57400eb10 tx=0x7fb57400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.457 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.451+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574004b20 con 0x7fb57c10b820 2024-02-09T23:03:01.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.451+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb574004c80 con 0x7fb57c10b820 2024-02-09T23:03:01.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.451+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c198450 con 0x7fb57c10b820 2024-02-09T23:03:01.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.451+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb574010650 con 0x7fb57c10b820 2024-02-09T23:03:01.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.451+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c1a4520 con 0x7fb57c10b820 2024-02-09T23:03:01.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.452+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c068d40 con 0x7fb57c10b820 2024-02-09T23:03:01.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.453+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb574005020 con 0x7fb57c10b820 2024-02-09T23:03:01.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.454+0000 7fb5808a5640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5680a2230 0x7fb5680a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.454+0000 7fb57affd640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5680a2230 0x7fb5680a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.454+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb574014070 con 0x7fb57c10b820 2024-02-09T23:03:01.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.454+0000 7fb57affd640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5680a2230 0x7fb5680a46f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009bd0 tx=0x7fb56c005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.458+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fb574097270 con 0x7fb57c10b820 2024-02-09T23:03:01.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 426100 3.50 MB/sec execute 535 sec latency 26.958 ms 2024-02-09T23:03:01.690 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.687+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb57c198b90 con 0x7fb57c10b820 2024-02-09T23:03:01.691 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.687+0000 7fb5808a5640 1 -- 172.21.15.3:0/959430949 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+11364 (secure 0 0 0) 0x7fb57409a930 con 0x7fb57c10b820 2024-02-09T23:03:01.691 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:03:01.692 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":49,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":49,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:01:40.086647+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27395":{"gid":27395,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/3325656265","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":3325656265},{"type":"v1","addr":"172.21.15.64:6835","nonce":3325656265}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:03:01.692 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 49 2024-02-09T23:03:01.693 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.689+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5680a2230 msgr2=0x7fb5680a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.689+0000 7fb5818a7640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5680a2230 0x7fb5680a46f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009bd0 tx=0x7fb56c005f90 comp rx=0 tx=0).stop 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.689+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 msgr2=0x7fb57c1976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.689+0000 7fb5818a7640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 0x7fb57c1976b0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fb57400eb10 tx=0x7fb57400c2d0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.689+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 shutdown_connections 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.689+0000 7fb5818a7640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb5680a2230 0x7fb5680a46f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.690+0000 7fb5818a7640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb57c102d70 0x7fb57c19dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.690+0000 7fb5818a7640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb57c10b820 0x7fb57c1976b0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.690+0000 7fb5818a7640 1 --2- 172.21.15.3:0/959430949 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb57c102450 0x7fb57c19d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.690+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 >> 172.21.15.3:0/959430949 conn(0x7fb57c0772b0 msgr2=0x7fb57c0fd230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.690+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 shutdown_connections 2024-02-09T23:03:01.694 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.690+0000 7fb5818a7640 1 -- 172.21.15.3:0/959430949 wait complete. 2024-02-09T23:03:01.705 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:01.705 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 -- 172.21.15.3:0/878013838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 msgr2=0x7feeac10b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 --2- 172.21.15.3:0/878013838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac10b250 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7feea0009900 tx=0x7feea002bcd0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 -- 172.21.15.3:0/878013838 shutdown_connections 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 --2- 172.21.15.3:0/878013838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeac10b820 0x7feeac10dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 --2- 172.21.15.3:0/878013838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac10b250 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 --2- 172.21.15.3:0/878013838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 0x7feeac102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.810+0000 7feeb3d77640 1 -- 172.21.15.3:0/878013838 >> 172.21.15.3:0/878013838 conn(0x7feeac077220 msgr2=0x7feeac077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 -- 172.21.15.3:0/878013838 shutdown_connections 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 -- 172.21.15.3:0/878013838 wait complete. 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 Processor -- start 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 -- start start 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 0x7feeac19d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac19db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeac10b820 0x7feeac1976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac197e40 con 0x7feeac102d70 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac197fb0 con 0x7feeac10b820 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb3d77640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac198170 con 0x7feeac102450 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb1312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac19db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb1b13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 0x7feeac19d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb1312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac19db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35880/0 (socket says 172.21.15.3:35880) 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.811+0000 7feeb1312640 1 -- 172.21.15.3:0/3622529484 learned_addr learned my addr 172.21.15.3:0/3622529484 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb2314640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeac10b820 0x7feeac1976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1b13640 1 -- 172.21.15.3:0/3622529484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeac10b820 msgr2=0x7feeac1976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1b13640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeac10b820 0x7feeac1976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1b13640 1 -- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 msgr2=0x7feeac19db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1b13640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac19db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1b13640 1 -- 172.21.15.3:0/3622529484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea0009560 con 0x7feeac102450 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1312640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac19db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb1b13640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 0x7feeac19d5f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9f0 tx=0x7feea400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeaaffd640 1 -- 172.21.15.3:0/3622529484 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea400ccc0 con 0x7feeac102450 2024-02-09T23:03:01.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac198450 con 0x7feeac102450 2024-02-09T23:03:01.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeaaffd640 1 -- 172.21.15.3:0/3622529484 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea400ce20 con 0x7feeac102450 2024-02-09T23:03:01.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeaaffd640 1 -- 172.21.15.3:0/3622529484 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4004a00 con 0x7feeac102450 2024-02-09T23:03:01.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.812+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac1a4570 con 0x7feeac102450 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.815+0000 7feeaaffd640 1 -- 172.21.15.3:0/3622529484 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7feea40050b0 con 0x7feeac102450 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.815+0000 7feeb6a29740 1 Processor -- start 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 -- start start 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268605be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x562686064990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562686017020 con 0x56268605ba30 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562685fca320 con 0x562686064f60 2024-02-09T23:03:01.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb6a29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562685fc7850 con 0x56268605c3e0 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb2314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268605be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x562686064990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x562686064990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54302/0 (socket says 172.21.15.3:54302) 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 -- 172.21.15.3:0/906226337 learned_addr learned my addr 172.21.15.3:0/906226337 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb2314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268605be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35892/0 (socket says 172.21.15.3:35892) 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeaaffd640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee940a2230 0x7fee940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1312640 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb2314640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee940a2230 0x7fee940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeaaffd640 1 -- 172.21.15.3:0/3622529484 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feea4014070 con 0x7feeac102450 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 -- 172.21.15.3:0/906226337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 msgr2=0x5626860673b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860673b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 -- 172.21.15.3:0/906226337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 msgr2=0x56268605be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268605be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.816+0000 7feeb1b13640 1 -- 172.21.15.3:0/906226337 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562685f85210 con 0x56268605c3e0 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb1b13640 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x562686064990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feea409a9b0 tx=0x7feea400cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e2b3f2cd9d6b7cb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb2314640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee940a2230 0x7fee940a46f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feeac06aa10 tx=0x7fee9c004ab0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7fee93fff640 1 -- 172.21.15.3:0/906226337 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea409a380 con 0x56268605c3e0 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7fee93fff640 1 -- 172.21.15.3:0/906226337 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feea409b120 con 0x56268605c3e0 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7fee93fff640 1 -- 172.21.15.3:0/906226337 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4097090 con 0x56268605c3e0 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 -- 172.21.15.3:0/906226337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 msgr2=0x562686064990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x562686064990 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7feea409a9b0 tx=0x7feea400cf90 comp rx=0 tx=0).stop 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 -- 172.21.15.3:0/906226337 shutdown_connections 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860673b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x562686064990 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 --2- 172.21.15.3:0/906226337 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268605be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.817+0000 7feeb6a29740 1 -- 172.21.15.3:0/906226337 >> 172.21.15.3:0/906226337 conn(0x562686056100 msgr2=0x5626860564e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 -- 172.21.15.3:0/906226337 shutdown_connections 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 -- 172.21.15.3:0/906226337 wait complete. 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 -- start start 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 Processor -- start 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605ba30 0x562686061f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x562686062470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860629b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562685fc3030 con 0x56268605c3e0 2024-02-09T23:03:01.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562685c96430 con 0x562686064f60 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562685c52e00 con 0x56268605ba30 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.818+0000 7feeb6a29740 10 client.0 _ll_register_callbacks cb 0x562685fb8030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x562686062470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1b13640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605ba30 0x562686061f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb2314640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860629b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x562686062470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35896/0 (socket says 172.21.15.3:35896) 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 -- 172.21.15.3:0/1754231779 learned_addr learned my addr 172.21.15.3:0/1754231779 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 -- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605ba30 msgr2=0x562686061f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605ba30 0x562686061f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 -- 172.21.15.3:0/1754231779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 msgr2=0x5626860629b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860629b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 -- 172.21.15.3:0/1754231779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac1a4570 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1b13640 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605ba30 0x562686061f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb1312640 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x562686062470 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7feea0036040 tx=0x7feea002c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e24080eee8a30aa7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7fee917fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7fee917fa640 1 -- 172.21.15.3:0/1754231779 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea0004410 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7fee917fa640 1 -- 172.21.15.3:0/1754231779 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feea0045070 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562685cc27e0 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7fee917fa640 1 -- 172.21.15.3:0/1754231779 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7feea0004df0 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb6a29740 10 client.19743 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.819+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x562685fc16b0 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7fee917fa640 1 -- 172.21.15.3:0/1754231779 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562685cc27e0 con 0x56268605c3e0 2024-02-09T23:03:01.832 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7fee917fa640 1 -- 172.21.15.3:0/1754231779 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7feea003c040 con 0x56268605c3e0 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb6a29740 10 client.19743 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb6a29740 10 client.19743 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb6a29740 20 client.19743 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5626860754d0 0x5626860778f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb6a29740 4 client.19743 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562685cff720 con 0x5626860754d0 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb1b13640 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5626860754d0 0x5626860778f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7fee917fa640 10 client.19743 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.820+0000 7feeb1b13640 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5626860754d0 0x5626860778f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=115d9915ea64d5d3 server_cookie=a48518aa2a1ad516 in_seq=0 out_seq=0 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7fee917fa640 1 -- 172.21.15.3:0/1754231779 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562685cff720 con 0x5626860754d0 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7fee917fa640 10 client.19743 handle_command_reply: tid=0 2024-02-09T23:03:01.833 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 client.19743 shutdown 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 msgr2=0x562686062470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x562686062470 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7feea0036040 tx=0x7feea002c720 comp rx=0 tx=0).stop 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 shutdown_connections 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5626860754d0 0x5626860778f0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686064f60 0x5626860629b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x562686062470 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7fee917fa640 0 client.19743 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1754231779 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605ba30 0x562686061f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 >> 172.21.15.3:0/1754231779 conn(0x562686056100 msgr2=0x562686057e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 shutdown_connections 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 1 -- 172.21.15.3:0/1754231779 wait complete. 2024-02-09T23:03:01.834 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.821+0000 7feeb6a29740 20 client.19743 trim_cache size 0 max 16384 2024-02-09T23:03:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 Processor -- start 2024-02-09T23:03:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 -- start start 2024-02-09T23:03:01.843 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562686056100 0x562686064660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56268605ba30 0x5626860627d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x56268605f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562685f48130 con 0x56268605c3e0 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562686071fc0 con 0x56268605ba30 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.833+0000 7feeb6a29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562686073cd0 con 0x562686056100 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x56268605f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x56268605f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35904/0 (socket says 172.21.15.3:35904) 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 -- 172.21.15.3:0/2286749433 learned_addr learned my addr 172.21.15.3:0/2286749433 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb1b13640 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562686056100 0x562686064660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 -- 172.21.15.3:0/2286749433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562686056100 msgr2=0x562686064660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562686056100 0x562686064660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.844 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 -- 172.21.15.3:0/2286749433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56268605ba30 msgr2=0x5626860627d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56268605ba30 0x5626860627d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 -- 172.21.15.3:0/2286749433 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562685f5dc90 con 0x56268605c3e0 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7feeb2314640 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x56268605f190 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fee9c012b20 tx=0x7fee9c009ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77762c0c00e6a3c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7fee937fe640 1 -- 172.21.15.3:0/2286749433 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c0142f0 con 0x56268605c3e0 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7fee937fe640 1 -- 172.21.15.3:0/2286749433 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee9c011ec0 con 0x56268605c3e0 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.834+0000 7fee937fe640 1 -- 172.21.15.3:0/2286749433 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee9c013400 con 0x56268605c3e0 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- 172.21.15.3:0/2286749433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 msgr2=0x56268605f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x56268605f190 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7fee9c012b20 tx=0x7fee9c009ab0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- 172.21.15.3:0/2286749433 shutdown_connections 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56268605ba30 0x5626860627d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605c3e0 0x56268605f190 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- 172.21.15.3:0/2286749433 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562686056100 0x562686064660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.845 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- 172.21.15.3:0/2286749433 >> 172.21.15.3:0/2286749433 conn(0x56268605cb30 msgr2=0x56268605d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- 172.21.15.3:0/2286749433 shutdown_connections 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- 172.21.15.3:0/2286749433 wait complete. 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- start start 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 Processor -- start 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686056100 0x56268606b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268606b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x5626860cba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56268606bc90 con 0x56268605ba30 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626860cbf60 con 0x562686056100 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.835+0000 7feeb6a29740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5626860cc0f0 con 0x56268605c3e0 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb6a29740 10 client.0 _ll_register_callbacks cb 0x7feea4096d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268606b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.846 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268606b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35908/0 (socket says 172.21.15.3:35908) 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 -- 172.21.15.3:0/1122704540 learned_addr learned my addr 172.21.15.3:0/1122704540 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb2314640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x5626860cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1b13640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686056100 0x56268606b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 -- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 msgr2=0x5626860cba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x5626860cba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 -- 172.21.15.3:0/1122704540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686056100 msgr2=0x56268606b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686056100 0x56268606b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 -- 172.21.15.3:0/1122704540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee9c012860 con 0x56268605ba30 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb1312640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268606b550 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7feea0003ae0 tx=0x7feea002ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b347b86586237ff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7fee917fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7fee917fa640 1 -- 172.21.15.3:0/1122704540 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea002c670 con 0x56268605ba30 2024-02-09T23:03:01.847 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7fee917fa640 1 -- 172.21.15.3:0/1122704540 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7feea0045070 con 0x56268605ba30 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562685d4e890 con 0x56268605ba30 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7fee917fa640 1 -- 172.21.15.3:0/1122704540 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7feea0003f60 con 0x56268605ba30 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb6a29740 10 client.19749 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.836+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x562685f83a70 con 0x56268605ba30 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7fee917fa640 1 -- 172.21.15.3:0/1122704540 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562685d4e890 con 0x56268605ba30 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7fee917fa640 1 -- 172.21.15.3:0/1122704540 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7feea003c040 con 0x56268605ba30 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 10 client.19749 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 10 client.19749 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 20 client.19749 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x562686064f60 0x5626860cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.848 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 4 client.19749 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562685c6e700 con 0x562686064f60 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb1b13640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x562686064f60 0x5626860cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7fee917fa640 10 client.19749 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb1b13640 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x562686064f60 0x5626860cf600 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=65c19355785b07d6 server_cookie=8218ba70428f855d in_seq=0 out_seq=0 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7fee917fa640 1 -- 172.21.15.3:0/1122704540 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562685c6e700 con 0x562686064f60 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7fee917fa640 10 client.19749 handle_command_reply: tid=0 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.837+0000 7feeb6a29740 1 client.19749 shutdown 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 msgr2=0x56268606b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268606b550 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7feea0003ae0 tx=0x7feea002ccd0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 shutdown_connections 2024-02-09T23:03:01.849 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x562686064f60 0x5626860cf600 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56268605c3e0 0x5626860cba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56268605ba30 0x56268606b550 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7fee917fa640 0 client.19749 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 --2- 172.21.15.3:0/1122704540 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562686056100 0x56268606b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 >> 172.21.15.3:0/1122704540 conn(0x56268605cb30 msgr2=0x5626860549b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 shutdown_connections 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 1 -- 172.21.15.3:0/1122704540 wait complete. 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb6a29740 20 client.19749 trim_cache size 0 max 16384 2024-02-09T23:03:01.850 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:01.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee940a2230 msgr2=0x7fee940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee940a2230 0x7fee940a46f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7feeac06aa10 tx=0x7fee9c004ab0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 msgr2=0x7feeac19d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.851 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 0x7feeac19d5f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7feea400e9f0 tx=0x7feea400eeb0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 shutdown_connections 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fee940a2230 0x7fee940a46f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7feeac10b820 0x7feeac1976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7feeac102d70 0x7feeac19db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 --2- 172.21.15.3:0/3622529484 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7feeac102450 0x7feeac19d5f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 >> 172.21.15.3:0/3622529484 conn(0x7feeac077220 msgr2=0x7feeac0fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 shutdown_connections 2024-02-09T23:03:01.852 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.838+0000 7feeb3d77640 1 -- 172.21.15.3:0/3622529484 wait complete. 2024-02-09T23:03:01.853 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:01.853 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:03:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 -- 172.21.15.3:0/3701340008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4105410 msgr2=0x7faab410cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 --2- 172.21.15.3:0/3701340008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4105410 0x7faab410cfe0 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7faaac009cd0 tx=0x7faaac02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 -- 172.21.15.3:0/3701340008 shutdown_connections 2024-02-09T23:03:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 --2- 172.21.15.3:0/3701340008 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4100cf0 0x7faab4104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 --2- 172.21.15.3:0/3701340008 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4105410 0x7faab410cfe0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 --2- 172.21.15.3:0/3701340008 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4100340 0x7faab4100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 -- 172.21.15.3:0/3701340008 >> 172.21.15.3:0/3701340008 conn(0x7faab40772e0 msgr2=0x7faab40776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 -- 172.21.15.3:0/3701340008 shutdown_connections 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 -- 172.21.15.3:0/3701340008 wait complete. 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.959+0000 7faab9d6c640 1 Processor -- start 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 -- start start 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 0x7faab419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4105410 0x7faab419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4197670 0x7faab4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4198140 con 0x7faab4100340 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab41982b0 con 0x7faab4105410 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab9d6c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab4198470 con 0x7faab4197670 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 0x7faab419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 0x7faab419d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35928/0 (socket says 172.21.15.3:35928) 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 -- 172.21.15.3:0/560537316 learned_addr learned my addr 172.21.15.3:0/560537316 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab3fff640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4197670 0x7faab4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab2ffd640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4105410 0x7faab419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 -- 172.21.15.3:0/560537316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4197670 msgr2=0x7faab4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4197670 0x7faab4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 -- 172.21.15.3:0/560537316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4105410 msgr2=0x7faab419daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4105410 0x7faab419daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab37fe640 1 -- 172.21.15.3:0/560537316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa8009490 con 0x7faab4100340 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab3fff640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4197670 0x7faab4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.960+0000 7faab2ffd640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4105410 0x7faab419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.961+0000 7faab37fe640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 0x7faab419d5b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7faaa400e9f0 tx=0x7faaa400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.961+0000 7faab0ff9640 1 -- 172.21.15.3:0/560537316 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4009700 con 0x7faab4100340 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.961+0000 7faab0ff9640 1 -- 172.21.15.3:0/560537316 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaa400cd80 con 0x7faab4100340 2024-02-09T23:03:01.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.961+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaac009900 con 0x7faab4100340 2024-02-09T23:03:01.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.961+0000 7faab0ff9640 1 -- 172.21.15.3:0/560537316 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa4018920 con 0x7faab4100340 2024-02-09T23:03:01.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.961+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab41a4760 con 0x7faab4100340 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.963+0000 7faab0ff9640 1 -- 172.21.15.3:0/560537316 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7faaa4018a80 con 0x7faab4100340 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.963+0000 7faabca19740 1 Processor -- start 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.963+0000 7faabca19740 1 -- start start 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.963+0000 7faab0ff9640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faa9c0a2230 0x7faa9c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faabca19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae206e40 0x559cae207220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faabca19740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab3fff640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faa9c0a2230 0x7faa9c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.974 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faabca19740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae210370 0x559cae2127c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faabca19740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae1be020 con 0x559cae2077f0 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faabca19740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae171320 con 0x559cae210370 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab0ff9640 1 -- 172.21.15.3:0/560537316 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7faaa4014070 con 0x7faab4100340 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faabca19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae16e850 con 0x559cae206e40 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35930/0 (socket says 172.21.15.3:35930) 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 -- 172.21.15.3:0/596104079 learned_addr learned my addr 172.21.15.3:0/596104079 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab37fe640 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae206e40 0x559cae207220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab3fff640 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae210370 0x559cae2127c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 -- 172.21.15.3:0/596104079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae206e40 msgr2=0x559cae207220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae206e40 0x559cae207220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 -- 172.21.15.3:0/596104079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae210370 msgr2=0x559cae2127c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae210370 0x559cae2127c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 -- 172.21.15.3:0/596104079 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cae12c210 con 0x559cae2077f0 2024-02-09T23:03:01.975 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab3fff640 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae210370 0x559cae2127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab3fff640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faa9c0a2230 0x7faa9c0a46f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faaac02c630 tx=0x7faaac032550 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faab2ffd640 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20fda0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7faaa80115a0 tx=0x7faaa8011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7c96d1713100d8a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.964+0000 7faa99ffb640 1 -- 172.21.15.3:0/596104079 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8005020 con 0x559cae2077f0 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faa99ffb640 1 -- 172.21.15.3:0/596104079 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaa8005660 con 0x559cae2077f0 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faa99ffb640 1 -- 172.21.15.3:0/596104079 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa800f980 con 0x559cae2077f0 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 -- 172.21.15.3:0/596104079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 msgr2=0x559cae20fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20fda0 secure :-1 s=READY pgs=1669 cs=0 l=1 rev1=1 crypto rx=0x7faaa80115a0 tx=0x7faaa8011a60 comp rx=0 tx=0).stop 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 -- 172.21.15.3:0/596104079 shutdown_connections 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae210370 0x559cae2127c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20fda0 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 --2- 172.21.15.3:0/596104079 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae206e40 0x559cae207220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 -- 172.21.15.3:0/596104079 >> 172.21.15.3:0/596104079 conn(0x559cae201510 msgr2=0x559cae2018f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.976 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 -- 172.21.15.3:0/596104079 shutdown_connections 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.965+0000 7faabca19740 1 -- 172.21.15.3:0/596104079 wait complete. 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 -- start start 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 Processor -- start 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae206e40 0x559cae20d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae20dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae16a030 con 0x559cae2077f0 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cade3d430 con 0x559cae206e40 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559caddf9e00 con 0x559cae210370 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faabca19740 10 client.0 _ll_register_callbacks cb 0x559cae15f030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab37fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae206e40 0x559cae20d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35942/0 (socket says 172.21.15.3:35942) 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 -- 172.21.15.3:0/1925309135 learned_addr learned my addr 172.21.15.3:0/1925309135 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.977 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab3fff640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae20dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 -- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 msgr2=0x559cae20dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae20dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 -- 172.21.15.3:0/1925309135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae206e40 msgr2=0x559cae20d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.966+0000 7faab2ffd640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae206e40 0x559cae20d270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faab2ffd640 1 -- 172.21.15.3:0/1925309135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa8009d90 con 0x559cae2077f0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faab37fe640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae206e40 0x559cae20d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faab2ffd640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20d7b0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7faaa8011570 tx=0x7faaa80042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96a6bd175ef8155e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faa92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faa92ffd640 1 -- 172.21.15.3:0/1925309135 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa80044e0 con 0x559cae2077f0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faa92ffd640 1 -- 172.21.15.3:0/1925309135 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7faaa8004b60 con 0x559cae2077f0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559cade697e0 con 0x559cae2077f0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faa92ffd640 1 -- 172.21.15.3:0/1925309135 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaa80051a0 con 0x559cae2077f0 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 10 client.19761 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:01.978 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x559cae1686b0 con 0x559cae2077f0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faa92ffd640 1 -- 172.21.15.3:0/1925309135 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559cade697e0 con 0x559cae2077f0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faa92ffd640 1 -- 172.21.15.3:0/1925309135 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faaa8021070 con 0x559cae2077f0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 10 client.19761 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 10 client.19761 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 20 client.19761 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae2207c0 0x559cae222be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.967+0000 7faabca19740 4 client.19761 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.968+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559cadea6720 con 0x559cae2207c0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.968+0000 7faab37fe640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae2207c0 0x559cae222be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.968+0000 7faab37fe640 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae2207c0 0x559cae222be0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=7dee5f07e18a59bb server_cookie=cbd9c5ce5d2f3ac8 in_seq=0 out_seq=0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.968+0000 7faa92ffd640 10 client.19761 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.968+0000 7faa92ffd640 1 -- 172.21.15.3:0/1925309135 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559cadea6720 con 0x559cae2207c0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faa92ffd640 10 client.19761 handle_command_reply: tid=0 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 client.19761 shutdown 2024-02-09T23:03:01.979 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 msgr2=0x559cae20d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20d7b0 secure :-1 s=READY pgs=1670 cs=0 l=1 rev1=1 crypto rx=0x7faaa8011570 tx=0x7faaa80042f0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 shutdown_connections 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae2207c0 0x559cae222be0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae20dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae2077f0 0x559cae20d7b0 unknown :-1 s=CLOSED pgs=1670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faa92ffd640 0 client.19761 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 --2- 172.21.15.3:0/1925309135 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae206e40 0x559cae20d270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 >> 172.21.15.3:0/1925309135 conn(0x559cae201510 msgr2=0x559cae1fc240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 shutdown_connections 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 1 -- 172.21.15.3:0/1925309135 wait complete. 2024-02-09T23:03:01.980 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.969+0000 7faabca19740 20 client.19761 trim_cache size 0 max 16384 2024-02-09T23:03:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 Processor -- start 2024-02-09T23:03:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 -- start start 2024-02-09T23:03:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae20db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2173d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae217910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.990 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae0ef130 con 0x559cae206e40 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae2061f0 con 0x559cae2077f0 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.981+0000 7faabca19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae227e30 con 0x559cae210370 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae20db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae20db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35952/0 (socket says 172.21.15.3:35952) 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 -- 172.21.15.3:0/2168743225 learned_addr learned my addr 172.21.15.3:0/2168743225 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab3fff640 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae217910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab2ffd640 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2173d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 -- 172.21.15.3:0/2168743225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 msgr2=0x559cae217910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae217910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 -- 172.21.15.3:0/2168743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 msgr2=0x559cae2173d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2173d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 -- 172.21.15.3:0/2168743225 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559cae104c90 con 0x559cae206e40 2024-02-09T23:03:01.991 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab3fff640 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae217910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faab37fe640 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae20db10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c6a0 tx=0x7faaa4097150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4bba90bc5c2c80c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faa997fa640 1 -- 172.21.15.3:0/2168743225 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa409d040 con 0x559cae206e40 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faa997fa640 1 -- 172.21.15.3:0/2168743225 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaa40991b0 con 0x559cae206e40 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.982+0000 7faa997fa640 1 -- 172.21.15.3:0/2168743225 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa40b3b30 con 0x559cae206e40 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 -- 172.21.15.3:0/2168743225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 msgr2=0x559cae20db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae20db10 secure :-1 s=READY pgs=1671 cs=0 l=1 rev1=1 crypto rx=0x7faaa400c6a0 tx=0x7faaa4097150 comp rx=0 tx=0).stop 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 -- 172.21.15.3:0/2168743225 shutdown_connections 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2173d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae217910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 --2- 172.21.15.3:0/2168743225 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae20db10 unknown :-1 s=CLOSED pgs=1671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 -- 172.21.15.3:0/2168743225 >> 172.21.15.3:0/2168743225 conn(0x559cae201510 msgr2=0x559cae20bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 -- 172.21.15.3:0/2168743225 shutdown_connections 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 -- 172.21.15.3:0/2168743225 wait complete. 2024-02-09T23:03:01.992 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 -- start start 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.983+0000 7faabca19740 1 Processor -- start 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae2161b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2166f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae276d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae216ff0 con 0x559cae206e40 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae277250 con 0x559cae2077f0 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559cae277390 con 0x559cae210370 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 10 client.0 _ll_register_callbacks cb 0x7faaa8009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae2161b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae2161b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35968/0 (socket says 172.21.15.3:35968) 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 -- 172.21.15.3:0/2088990271 learned_addr learned my addr 172.21.15.3:0/2088990271 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab2ffd640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab3fff640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae276d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.993 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 -- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 msgr2=0x559cae276d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae276d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 -- 172.21.15.3:0/2088990271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 msgr2=0x559cae2166f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2166f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 -- 172.21.15.3:0/2088990271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab410d7f0 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faab37fe640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae2161b0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7faaa409abb0 tx=0x7faaa4010f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de866f2a9dc2982d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faa92ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faa92ffd640 1 -- 172.21.15.3:0/2088990271 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa409d030 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faa92ffd640 1 -- 172.21.15.3:0/2088990271 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7faaa4097520 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faaa8005660 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faa92ffd640 1 -- 172.21.15.3:0/2088990271 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7faaa4085b70 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 10 client.19767 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.984+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x7faab41a4760 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faa92ffd640 1 -- 172.21.15.3:0/2088990271 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaa8005660 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faa92ffd640 1 -- 172.21.15.3:0/2088990271 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faaa408b050 con 0x559cae206e40 2024-02-09T23:03:01.994 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 10 client.19767 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 10 client.19767 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 20 client.19767 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae278730 0x559cae27ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 4 client.19767 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559cadef5890 con 0x559cae278730 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faab2ffd640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae278730 0x559cae27ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faa92ffd640 10 client.19767 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faab2ffd640 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae278730 0x559cae27ab20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d860bca78c83ad5b server_cookie=f0bb233ed9253647 in_seq=0 out_seq=0 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faa92ffd640 1 -- 172.21.15.3:0/2088990271 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559cadef5890 con 0x559cae278730 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faa92ffd640 10 client.19767 handle_command_reply: tid=0 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.985+0000 7faabca19740 1 client.19767 shutdown 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 msgr2=0x559cae2161b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae2161b0 secure :-1 s=READY pgs=1672 cs=0 l=1 rev1=1 crypto rx=0x7faaa409abb0 tx=0x7faaa4010f50 comp rx=0 tx=0).stop 2024-02-09T23:03:01.995 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 shutdown_connections 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x559cae278730 0x559cae27ab20 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559cae2077f0 0x559cae2166f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559cae210370 0x559cae276d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faa92ffd640 0 client.19767 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 --2- 172.21.15.3:0/2088990271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559cae206e40 0x559cae2161b0 unknown :-1 s=CLOSED pgs=1672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 >> 172.21.15.3:0/2088990271 conn(0x559cae201510 msgr2=0x559cae1fb9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 shutdown_connections 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 1 -- 172.21.15.3:0/2088990271 wait complete. 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faabca19740 20 client.19767 trim_cache size 0 max 16384 2024-02-09T23:03:01.996 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faa9c0a2230 msgr2=0x7faa9c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.997 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faa9c0a2230 0x7faa9c0a46f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faaac02c630 tx=0x7faaac032550 comp rx=0 tx=0).stop 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 msgr2=0x7faab419d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 0x7faab419d5b0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7faaa400e9f0 tx=0x7faaa400eeb0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 shutdown_connections 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7faa9c0a2230 0x7faa9c0a46f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7faab4105410 0x7faab419daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7faab4197670 0x7faab4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 --2- 172.21.15.3:0/560537316 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7faab4100340 0x7faab419d5b0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.986+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 >> 172.21.15.3:0/560537316 conn(0x7faab40772e0 msgr2=0x7faab40687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.987+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 shutdown_connections 2024-02-09T23:03:01.998 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:01.987+0000 7faab9d6c640 1 -- 172.21.15.3:0/560537316 wait complete. 2024-02-09T23:03:02.001 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:02.001 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2753071139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd284106500 msgr2=0x7fd28410e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2753071139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd284106500 0x7fd28410e0d0 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7fd274009cd0 tx=0x7fd27402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2753071139 shutdown_connections 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2753071139 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd284106500 0x7fd28410e0d0 unknown :-1 s=CLOSED pgs=1673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2753071139 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd284100360 0x7fd284105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2753071139 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 0x7fd2840ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2753071139 >> 172.21.15.3:0/2753071139 conn(0x7fd284076630 msgr2=0x7fd2840fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2753071139 shutdown_connections 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2753071139 wait complete. 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 Processor -- start 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.107+0000 7fd28bc0b640 1 -- start start 2024-02-09T23:03:02.113 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28bc0b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 0x7fd28419d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28bc0b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd284106500 0x7fd28419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28bc0b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 0x7fd284197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28bc0b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841981c0 con 0x7fd2841976f0 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28bc0b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd284198330 con 0x7fd2840ff9b0 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28bc0b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2841984f0 con 0x7fd284106500 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 0x7fd284197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 0x7fd28419d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 0x7fd284197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:35992/0 (socket says 172.21.15.3:35992) 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 -- 172.21.15.3:0/2120684822 learned_addr learned my addr 172.21.15.3:0/2120684822 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd2891a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd284106500 0x7fd28419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 0x7fd28419d630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:46796/0 (socket says 172.21.15.3:46796) 2024-02-09T23:03:02.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 -- 172.21.15.3:0/2120684822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd284106500 msgr2=0x7fd28419db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd284106500 0x7fd28419db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 -- 172.21.15.3:0/2120684822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 msgr2=0x7fd28419d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 0x7fd28419d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 -- 172.21.15.3:0/2120684822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd274009900 con 0x7fd2841976f0 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.108+0000 7fd28a1a8640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 0x7fd284197b70 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd274004af0 tx=0x7fd274032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.109+0000 7fd282ffd640 1 -- 172.21.15.3:0/2120684822 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2740328b0 con 0x7fd2841976f0 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.109+0000 7fd282ffd640 1 -- 172.21.15.3:0/2120684822 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd274003e50 con 0x7fd2841976f0 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.109+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2841a4430 con 0x7fd2841976f0 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.109+0000 7fd282ffd640 1 -- 172.21.15.3:0/2120684822 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd274004740 con 0x7fd2841976f0 2024-02-09T23:03:02.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.109+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2841a4900 con 0x7fd2841976f0 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.111+0000 7fd28e7d7740 1 Processor -- start 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.111+0000 7fd282ffd640 1 -- 172.21.15.3:0/2120684822 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd27400a040 con 0x7fd2841976f0 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 -- start start 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f4149e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f414a3e0 0x5604f4152990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f41553b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f4105020 con 0x5604f4152f60 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f40b8320 con 0x5604f414a3e0 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f40b5850 con 0x5604f4149a30 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f414a3e0 0x5604f4152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f4149e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd282ffd640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd26c0a22b0 0x7fd26c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f414a3e0 0x5604f4152990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:46804/0 (socket says 172.21.15.3:46804) 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2899a7640 1 -- 172.21.15.3:0/1665782966 learned_addr learned my addr 172.21.15.3:0/1665782966 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f41553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd28a1a8640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd26c0a22b0 0x7fd26c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 -- 172.21.15.3:0/1665782966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f414a3e0 msgr2=0x5604f4152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f414a3e0 0x5604f4152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 -- 172.21.15.3:0/1665782966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 msgr2=0x5604f41553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f41553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd2891a6640 1 -- 172.21.15.3:0/1665782966 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604f4073210 con 0x5604f4149a30 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.112+0000 7fd282ffd640 1 -- 172.21.15.3:0/2120684822 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd2740f06d0 con 0x7fd2841976f0 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.113+0000 7fd2891a6640 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f4149e10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd278009710 tx=0x7fd2780096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c293079d7b721dd server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.113+0000 7fd26bfff640 1 -- 172.21.15.3:0/1665782966 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278009da0 con 0x5604f4149a30 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.113+0000 7fd28a1a8640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd26c0a22b0 0x7fd26c0a4770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd2740edfe0 tx=0x7fd2740095c0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.113+0000 7fd26bfff640 1 -- 172.21.15.3:0/1665782966 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd278004510 con 0x5604f4149a30 2024-02-09T23:03:02.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.113+0000 7fd26bfff640 1 -- 172.21.15.3:0/1665782966 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278003850 con 0x5604f4149a30 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 -- 172.21.15.3:0/1665782966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 msgr2=0x5604f4149e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f4149e10 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fd278009710 tx=0x7fd2780096a0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 -- 172.21.15.3:0/1665782966 shutdown_connections 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f414a3e0 0x5604f4152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f41553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/1665782966 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f4149e10 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 -- 172.21.15.3:0/1665782966 >> 172.21.15.3:0/1665782966 conn(0x5604f4144100 msgr2=0x5604f41444e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 -- 172.21.15.3:0/1665782966 shutdown_connections 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 -- 172.21.15.3:0/1665782966 wait complete. 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 -- start start 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.114+0000 7fd28e7d7740 1 Processor -- start 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f414ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4150490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4152f60 0x5604f41509d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f40b1030 con 0x5604f414a3e0 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f3d84430 con 0x5604f4152f60 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f3d40e00 con 0x5604f4149a30 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd28e7d7740 10 client.0 _ll_register_callbacks cb 0x5604f40a6030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4150490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2891a6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f414ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4150490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36006/0 (socket says 172.21.15.3:36006) 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 -- 172.21.15.3:0/4193126536 learned_addr learned my addr 172.21.15.3:0/4193126536 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2891a6640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4152f60 0x5604f41509d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 -- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 msgr2=0x5604f414ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f414ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 -- 172.21.15.3:0/4193126536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4152f60 msgr2=0x5604f41509d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4152f60 0x5604f41509d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 -- 172.21.15.3:0/4193126536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd278020a60 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2891a6640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f414ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2899a7640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4150490 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fd27c012720 tx=0x7fd27c012be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa04b1092bf6dbd7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.115+0000 7fd2697fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd2697fa640 1 -- 172.21.15.3:0/4193126536 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c017070 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd2697fa640 1 -- 172.21.15.3:0/4193126536 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd27c004030 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5604f3db07e0 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd2697fa640 1 -- 172.21.15.3:0/4193126536 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd27c0041f0 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd28e7d7740 10 client.19779 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x5604f40af6b0 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd2697fa640 1 -- 172.21.15.3:0/4193126536 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5604f3db07e0 con 0x5604f414a3e0 2024-02-09T23:03:02.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.116+0000 7fd2697fa640 1 -- 172.21.15.3:0/4193126536 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd27c0039d0 con 0x5604f414a3e0 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd28e7d7740 10 client.19779 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd28e7d7740 10 client.19779 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd28e7d7740 20 client.19779 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f4163580 0x5604f41659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd28e7d7740 4 client.19779 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5604f3ded720 con 0x5604f4163580 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd2891a6640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f4163580 0x5604f41659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd2891a6640 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f4163580 0x5604f41659a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=536ed84156b5639d server_cookie=430b3e46221f1d9a in_seq=0 out_seq=0 2024-02-09T23:03:02.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.117+0000 7fd2697fa640 10 client.19779 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd2697fa640 1 -- 172.21.15.3:0/4193126536 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5604f3ded720 con 0x5604f4163580 2024-02-09T23:03:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd2697fa640 10 client.19779 handle_command_reply: tid=0 2024-02-09T23:03:02.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 client.19779 shutdown 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 msgr2=0x5604f4150490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4150490 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7fd27c012720 tx=0x7fd27c012be0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 shutdown_connections 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f4163580 0x5604f41659a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4152f60 0x5604f41509d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4150490 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd2697fa640 0 client.19779 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/4193126536 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4149a30 0x5604f414ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 >> 172.21.15.3:0/4193126536 conn(0x5604f4144100 msgr2=0x5604f4145e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 shutdown_connections 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 1 -- 172.21.15.3:0/4193126536 wait complete. 2024-02-09T23:03:02.128 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.118+0000 7fd28e7d7740 20 client.19779 trim_cache size 0 max 16384 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 Processor -- start 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 -- start start 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f4152680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f414a3e0 0x5604f41507f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f416a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f4036130 con 0x5604f4152f60 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f4148ac0 con 0x5604f4149a30 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f414bba0 con 0x5604f414a3e0 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd2891a6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f416a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd2891a6640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f416a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36016/0 (socket says 172.21.15.3:36016) 2024-02-09T23:03:02.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd2891a6640 1 -- 172.21.15.3:0/464720418 learned_addr learned my addr 172.21.15.3:0/464720418 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f414a3e0 0x5604f41507f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.130+0000 7fd2891a6640 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f4152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2891a6640 1 -- 172.21.15.3:0/464720418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f414a3e0 msgr2=0x5604f41507f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2891a6640 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f414a3e0 0x5604f41507f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2891a6640 1 -- 172.21.15.3:0/464720418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 msgr2=0x5604f4152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2891a6640 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f4152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2891a6640 1 -- 172.21.15.3:0/464720418 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604f404bc90 con 0x5604f4152f60 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2899a7640 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f414a3e0 0x5604f41507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd2891a6640 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f416a300 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x5604f413fd90 tx=0x7fd278017040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34acf9a0bdbe44df server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd26b7fe640 1 -- 172.21.15.3:0/464720418 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278036d80 con 0x5604f4152f60 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd26b7fe640 1 -- 172.21.15.3:0/464720418 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd278034070 con 0x5604f4152f60 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.131+0000 7fd26b7fe640 1 -- 172.21.15.3:0/464720418 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278036790 con 0x5604f4152f60 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 -- 172.21.15.3:0/464720418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 msgr2=0x5604f416a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.141 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f416a300 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x5604f413fd90 tx=0x7fd278017040 comp rx=0 tx=0).stop 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 -- 172.21.15.3:0/464720418 shutdown_connections 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f4152f60 0x5604f416a300 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f414a3e0 0x5604f41507f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/464720418 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f4152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 -- 172.21.15.3:0/464720418 >> 172.21.15.3:0/464720418 conn(0x5604f4144100 msgr2=0x5604f414a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 -- 172.21.15.3:0/464720418 shutdown_connections 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 -- 172.21.15.3:0/464720418 wait complete. 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 -- start start 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.132+0000 7fd28e7d7740 1 Processor -- start 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f415a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4158570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4152f60 0x5604f4158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.142 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f4158ff0 con 0x5604f414a3e0 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f4159160 con 0x5604f4149a30 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604f4159320 con 0x5604f4152f60 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 10 client.0 _ll_register_callbacks cb 0x7fd278015a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4158570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2891a6640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f415a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4158570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36024/0 (socket says 172.21.15.3:36024) 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 -- 172.21.15.3:0/3473441624 learned_addr learned my addr 172.21.15.3:0/3473441624 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2891a6640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4152f60 0x5604f4158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 -- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4152f60 msgr2=0x5604f4158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4152f60 0x5604f4158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.143 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 -- 172.21.15.3:0/3473441624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 msgr2=0x5604f415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f415a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 -- 172.21.15.3:0/3473441624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5604f40af6b0 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2891a6640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f415a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2899a7640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4158570 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x5604f414f7b0 tx=0x7fd27c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7215f789b806e929 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2697fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2697fa640 1 -- 172.21.15.3:0/3473441624 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c017040 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2697fa640 1 -- 172.21.15.3:0/3473441624 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fd27c02a020 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd278004510 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd2697fa640 1 -- 172.21.15.3:0/3473441624 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd27c009e50 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 10 client.19785 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.133+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x5604f4073210 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd2697fa640 1 -- 172.21.15.3:0/3473441624 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd278004510 con 0x5604f414a3e0 2024-02-09T23:03:02.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd2697fa640 1 -- 172.21.15.3:0/3473441624 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd27c025710 con 0x5604f414a3e0 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd28e7d7740 10 client.19785 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd28e7d7740 10 client.19785 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd28e7d7740 20 client.19785 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f41baf20 0x5604f41bd340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd28e7d7740 4 client.19785 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5604f3e3c890 con 0x5604f41baf20 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd2891a6640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f41baf20 0x5604f41bd340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd2697fa640 10 client.19785 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.134+0000 7fd2891a6640 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f41baf20 0x5604f41bd340 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=67c5749e89c28401 server_cookie=1a80ce5992d3cf35 in_seq=0 out_seq=0 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd2697fa640 1 -- 172.21.15.3:0/3473441624 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5604f3e3c890 con 0x5604f41baf20 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd2697fa640 10 client.19785 handle_command_reply: tid=0 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 client.19785 shutdown 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 msgr2=0x5604f4158570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4158570 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x5604f414f7b0 tx=0x7fd27c003820 comp rx=0 tx=0).stop 2024-02-09T23:03:02.145 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 shutdown_connections 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5604f41baf20 0x5604f41bd340 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5604f4152f60 0x5604f4158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5604f414a3e0 0x5604f4158570 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 --2- 172.21.15.3:0/3473441624 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5604f4149a30 0x5604f415a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd2697fa640 0 client.19785 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 >> 172.21.15.3:0/3473441624 conn(0x5604f4144100 msgr2=0x5604f416a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 shutdown_connections 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 1 -- 172.21.15.3:0/3473441624 wait complete. 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28e7d7740 20 client.19785 trim_cache size 0 max 16384 2024-02-09T23:03:02.146 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd26c0a22b0 msgr2=0x7fd26c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd26c0a22b0 0x7fd26c0a4770 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd2740edfe0 tx=0x7fd2740095c0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 msgr2=0x7fd284197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 0x7fd284197b70 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7fd274004af0 tx=0x7fd274032570 comp rx=0 tx=0).stop 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 shutdown_connections 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd26c0a22b0 0x7fd26c0a4770 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd2841976f0 0x7fd284197b70 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd284106500 0x7fd28419db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 --2- 172.21.15.3:0/2120684822 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd2840ff9b0 0x7fd28419d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.135+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 >> 172.21.15.3:0/2120684822 conn(0x7fd284076630 msgr2=0x7fd2841046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.136+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 shutdown_connections 2024-02-09T23:03:02.148 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.136+0000 7fd28bc0b640 1 -- 172.21.15.3:0/2120684822 wait complete. 2024-02-09T23:03:02.150 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:02.150 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:03:02.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 -- 172.21.15.3:0/3855362944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304100ce0 msgr2=0x7f7304104d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 --2- 172.21.15.3:0/3855362944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304100ce0 0x7f7304104d20 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009930 tx=0x7f72f002bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 -- 172.21.15.3:0/3855362944 shutdown_connections 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 --2- 172.21.15.3:0/3855362944 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7304105400 0x7f730410cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 --2- 172.21.15.3:0/3855362944 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304100ce0 0x7f7304104d20 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 --2- 172.21.15.3:0/3855362944 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7304100330 0x7f7304100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 -- 172.21.15.3:0/3855362944 >> 172.21.15.3:0/3855362944 conn(0x7f73040fbb90 msgr2=0x7f73040fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 -- 172.21.15.3:0/3855362944 shutdown_connections 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 -- 172.21.15.3:0/3855362944 wait complete. 2024-02-09T23:03:02.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.252+0000 7f730aceb640 1 Processor -- start 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 -- start start 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7304100330 0x7f7304198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 0x7f7304198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73041991d0 0x7f73041a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041a3e50 con 0x7f7304105400 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041a3fc0 con 0x7f7304100330 2024-02-09T23:03:02.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73041a4180 con 0x7f73041991d0 2024-02-09T23:03:02.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 0x7f7304198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 0x7f7304198c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36040/0 (socket says 172.21.15.3:36040) 2024-02-09T23:03:02.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 -- 172.21.15.3:0/4083305588 learned_addr learned my addr 172.21.15.3:0/4083305588 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7308a87640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7304100330 0x7f7304198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7309288640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73041991d0 0x7f73041a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 -- 172.21.15.3:0/4083305588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73041991d0 msgr2=0x7f73041a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73041991d0 0x7f73041a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 -- 172.21.15.3:0/4083305588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7304100330 msgr2=0x7f7304198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7304100330 0x7f7304198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 -- 172.21.15.3:0/4083305588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f0009560 con 0x7f7304105400 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7303fff640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 0x7f7304198c40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009230 tx=0x7f72f002c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7301ffb640 1 -- 172.21.15.3:0/4083305588 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0004350 con 0x7f7304105400 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7301ffb640 1 -- 172.21.15.3:0/4083305588 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72f00044b0 con 0x7f7304105400 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73041a4400 con 0x7f7304105400 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.253+0000 7f7301ffb640 1 -- 172.21.15.3:0/4083305588 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f0003da0 con 0x7f7304105400 2024-02-09T23:03:02.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.254+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73041a4910 con 0x7f7304105400 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7301ffb640 1 -- 172.21.15.3:0/4083305588 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f72f0004df0 con 0x7f7304105400 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 Processor -- start 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 -- start start 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563249477910 0x563249477cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 0x563249480df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x563249481820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563249433020 con 0x5632494813c0 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632493e6320 con 0x563249477910 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f730d996740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632493e3850 con 0x5632494782c0 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7301ffb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f72f40a2230 0x7f72f40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x563249481820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7308a87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 0x563249480df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x563249481820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36044/0 (socket says 172.21.15.3:36044) 2024-02-09T23:03:02.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7303fff640 1 -- 172.21.15.3:0/2121590991 learned_addr learned my addr 172.21.15.3:0/2121590991 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7308a87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 0x563249480df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54468/0 (socket says 172.21.15.3:54468) 2024-02-09T23:03:02.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.256+0000 7f7309288640 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563249477910 0x563249477cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7309288640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f72f40a2230 0x7f72f40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7301ffb640 1 -- 172.21.15.3:0/4083305588 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f72f00f0090 con 0x7f7304105400 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7308a87640 1 -- 172.21.15.3:0/2121590991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563249477910 msgr2=0x563249477cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7308a87640 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563249477910 0x563249477cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7308a87640 1 -- 172.21.15.3:0/2121590991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 msgr2=0x563249481820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7308a87640 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x563249481820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7308a87640 1 -- 172.21.15.3:0/2121590991 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632493a1210 con 0x5632494782c0 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7303fff640 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x563249481820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7308a87640 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 0x563249480df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009c20 tx=0x7f72f8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f569372fc7c32b1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f7309288640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f72f40a2230 0x7f72f40a46f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72fc005fd0 tx=0x7f72fc009500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f72eeffd640 1 -- 172.21.15.3:0/2121590991 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8004050 con 0x5632494782c0 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f72eeffd640 1 -- 172.21.15.3:0/2121590991 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72f8005500 con 0x5632494782c0 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.257+0000 7f72eeffd640 1 -- 172.21.15.3:0/2121590991 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f8004cd0 con 0x5632494782c0 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 -- 172.21.15.3:0/2121590991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 msgr2=0x563249480df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 0x563249480df0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72f8009c20 tx=0x7f72f8011810 comp rx=0 tx=0).stop 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 -- 172.21.15.3:0/2121590991 shutdown_connections 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x563249481820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5632494782c0 0x563249480df0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 --2- 172.21.15.3:0/2121590991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563249477910 0x563249477cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 -- 172.21.15.3:0/2121590991 >> 172.21.15.3:0/2121590991 conn(0x563249472100 msgr2=0x5632494724e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 -- 172.21.15.3:0/2121590991 shutdown_connections 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 -- 172.21.15.3:0/2121590991 wait complete. 2024-02-09T23:03:02.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 -- start start 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.258+0000 7f730d996740 1 Processor -- start 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x563249484dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x563249485300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632493df030 con 0x5632494782c0 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632490b2430 con 0x5632494813c0 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56324906ee00 con 0x563249477910 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 10 client.0 _ll_register_callbacks cb 0x5632493d4030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x563249485300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x563249485300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36060/0 (socket says 172.21.15.3:36060) 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7308a87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x563249484dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 -- 172.21.15.3:0/4266094399 learned_addr learned my addr 172.21.15.3:0/4266094399 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7309288640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 -- 172.21.15.3:0/4266094399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 msgr2=0x563249484dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x563249484dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 -- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 msgr2=0x563249485840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 -- 172.21.15.3:0/4266094399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f8009850 con 0x5632494782c0 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7308a87640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x563249484dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f7303fff640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x563249485300 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b8ff0 tx=0x7f72f0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b83d418c9bed18 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f72e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:02.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f72e3fff640 1 -- 172.21.15.3:0/4266094399 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00bd290 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f72e3fff640 1 -- 172.21.15.3:0/4266094399 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f72f00bcb90 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.259+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5632490de7e0 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f72e3fff640 1 -- 172.21.15.3:0/4266094399 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72f00c0ce0 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 10 client.19797 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x5632493dd6b0 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f72e3fff640 1 -- 172.21.15.3:0/4266094399 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72f00d4080 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f72e3fff640 1 -- 172.21.15.3:0/4266094399 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f72f003c070 con 0x5632494782c0 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 10 client.19797 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 10 client.19797 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 20 client.19797 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563249495590 0x5632494979a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 4 client.19797 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:02.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56324911b720 con 0x563249495590 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.260+0000 7f7308a87640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563249495590 0x5632494979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f7308a87640 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563249495590 0x5632494979a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5c5729674b6353bc server_cookie=fbd9615109a72de in_seq=0 out_seq=0 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f72e3fff640 10 client.19797 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f72e3fff640 1 -- 172.21.15.3:0/4266094399 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56324911b720 con 0x563249495590 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f72e3fff640 10 client.19797 handle_command_reply: tid=0 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 client.19797 shutdown 2024-02-09T23:03:02.286 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 msgr2=0x563249485300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x563249485300 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b8ff0 tx=0x7f72f0009260 comp rx=0 tx=0).stop 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 shutdown_connections 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x563249495590 0x5632494979a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f72e3fff640 0 client.19797 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x563249485300 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 --2- 172.21.15.3:0/4266094399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x563249484dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.261+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 >> 172.21.15.3:0/4266094399 conn(0x563249472100 msgr2=0x563249473e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.262+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 shutdown_connections 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.262+0000 7f730d996740 1 -- 172.21.15.3:0/4266094399 wait complete. 2024-02-09T23:03:02.287 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.262+0000 7f730d996740 20 client.19797 trim_cache size 0 max 16384 2024-02-09T23:03:02.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 Processor -- start 2024-02-09T23:03:02.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 -- start start 2024-02-09T23:03:02.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x5632494874f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x5632494827b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.293 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563249364130 con 0x5632494782c0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56324946da40 con 0x5632494813c0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f730d996740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563249477540 con 0x563249477910 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x5632494827b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f7303fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x5632494827b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36066/0 (socket says 172.21.15.3:36066) 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f7308a87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x5632494874f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f7303fff640 1 -- 172.21.15.3:0/1773776531 learned_addr learned my addr 172.21.15.3:0/1773776531 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.274+0000 7f7309288640 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f7303fff640 1 -- 172.21.15.3:0/1773776531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 msgr2=0x5632494874f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f7303fff640 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x5632494874f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f7303fff640 1 -- 172.21.15.3:0/1773776531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 msgr2=0x563249485660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f7303fff640 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f7303fff640 1 -- 172.21.15.3:0/1773776531 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563249379c90 con 0x5632494782c0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f7303fff640 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x5632494827b0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b9560 tx=0x7f72f002bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa1ae220eda780b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.294 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f72ee7fc640 1 -- 172.21.15.3:0/1773776531 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00b9200 con 0x5632494782c0 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f72ee7fc640 1 -- 172.21.15.3:0/1773776531 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5632490de7e0 con 0x5632494782c0 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.275+0000 7f72ee7fc640 1 -- 172.21.15.3:0/1773776531 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72f00c0ce0 con 0x5632494782c0 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 -- 172.21.15.3:0/1773776531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 msgr2=0x5632494827b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x5632494827b0 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f72f00b9560 tx=0x7f72f002bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 -- 172.21.15.3:0/1773776531 shutdown_connections 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494813c0 0x563249485660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494782c0 0x5632494827b0 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- 172.21.15.3:0/1773776531 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x5632494874f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 -- 172.21.15.3:0/1773776531 >> 172.21.15.3:0/1773776531 conn(0x563249472100 msgr2=0x56324947f050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 -- 172.21.15.3:0/1773776531 shutdown_connections 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 -- 172.21.15.3:0/1773776531 wait complete. 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 -- start start 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 Processor -- start 2024-02-09T23:03:02.295 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x56324948c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494782c0 0x56324948a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.276+0000 7f730d996740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x56324948adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56324948b300 con 0x5632494813c0 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56324948b470 con 0x5632494782c0 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56324948b630 con 0x563249477910 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 10 client.0 _ll_register_callbacks cb 0x7f72f80097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x56324948adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x56324948adc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36082/0 (socket says 172.21.15.3:36082) 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 -- 172.21.15.3:0/3548051283 learned_addr learned my addr 172.21.15.3:0/3548051283 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7308a87640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x56324948c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7303fff640 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494782c0 0x56324948a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.296 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 -- 172.21.15.3:0/3548051283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 msgr2=0x56324948c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x56324948c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 -- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494782c0 msgr2=0x56324948a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494782c0 0x56324948a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 -- 172.21.15.3:0/3548051283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632493dd6b0 con 0x5632494813c0 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f7309288640 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x56324948adc0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009e10 tx=0x7f72fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95a1e623999315ca server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f72e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f72e3fff640 1 -- 172.21.15.3:0/3548051283 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc025040 con 0x5632494813c0 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f72e3fff640 1 -- 172.21.15.3:0/3548051283 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(101..101 src has 1..101) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f72fc010d40 con 0x5632494813c0 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f72f8005500 con 0x5632494813c0 2024-02-09T23:03:02.297 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f72e3fff640 1 -- 172.21.15.3:0/3548051283 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f72fc004b20 con 0x5632494813c0 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 10 client.19803 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.277+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56324939fa70 con 0x5632494813c0 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f72e3fff640 1 -- 172.21.15.3:0/3548051283 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72fc037020 con 0x5632494813c0 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f72e3fff640 1 -- 172.21.15.3:0/3548051283 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f72fc005890 con 0x5632494813c0 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 10 client.19803 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 10 client.19803 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 20 client.19803 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:02.298 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5632494ecfe0 0x5632494ef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 4 client.19803 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56324916a890 con 0x5632494ecfe0 2024-02-09T23:03:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f7308a87640 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5632494ecfe0 0x5632494ef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f7308a87640 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5632494ecfe0 0x5632494ef400 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ea48b4796d621a13 server_cookie=5dcdf114ffe6d5e3 in_seq=0 out_seq=0 2024-02-09T23:03:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f72e3fff640 10 client.19803 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:02.299 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f72e3fff640 1 -- 172.21.15.3:0/3548051283 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56324916a890 con 0x5632494ecfe0 2024-02-09T23:03:02.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f72e3fff640 10 client.19803 handle_command_reply: tid=0 2024-02-09T23:03:02.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.278+0000 7f730d996740 1 client.19803 shutdown 2024-02-09T23:03:02.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 msgr2=0x56324948adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x56324948adc0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f72fc009e10 tx=0x7f72fc004ab0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 shutdown_connections 2024-02-09T23:03:02.300 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x5632494ecfe0 0x5632494ef400 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5632494782c0 0x56324948a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5632494813c0 0x56324948adc0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 --2- 172.21.15.3:0/3548051283 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563249477910 0x56324948c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f72e3fff640 0 client.19803 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 >> 172.21.15.3:0/3548051283 conn(0x563249472100 msgr2=0x5632494856a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 shutdown_connections 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 1 -- 172.21.15.3:0/3548051283 wait complete. 2024-02-09T23:03:02.301 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730d996740 20 client.19803 trim_cache size 0 max 16384 2024-02-09T23:03:02.302 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:02.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f72f40a2230 msgr2=0x7f72f40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f72f40a2230 0x7f72f40a46f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f72fc005fd0 tx=0x7f72fc009500 comp rx=0 tx=0).stop 2024-02-09T23:03:02.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 msgr2=0x7f7304198c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 0x7f7304198c40 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f72f0009230 tx=0x7f72f002c680 comp rx=0 tx=0).stop 2024-02-09T23:03:02.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 shutdown_connections 2024-02-09T23:03:02.303 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f72f40a2230 0x7f72f40a46f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f73041991d0 0x7f73041a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7304105400 0x7f7304198c40 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 --2- 172.21.15.3:0/4083305588 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7304100330 0x7f7304198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 >> 172.21.15.3:0/4083305588 conn(0x7f73040fbb90 msgr2=0x7f73040fcf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 shutdown_connections 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.279+0000 7f730aceb640 1 -- 172.21.15.3:0/4083305588 wait complete. 2024-02-09T23:03:02.304 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:02.304 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:03:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 -- 172.21.15.3:0/2277148781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 msgr2=0x7fc54810b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/2277148781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc54810b250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc53c009860 tx=0x7fc53c02bc00 comp rx=0 tx=0).stop 2024-02-09T23:03:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 -- 172.21.15.3:0/2277148781 shutdown_connections 2024-02-09T23:03:02.448 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/2277148781 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc54810dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/2277148781 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc54810b250 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/2277148781 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc548102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 -- 172.21.15.3:0/2277148781 >> 172.21.15.3:0/2277148781 conn(0x7fc5480772e0 msgr2=0x7fc5480776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 -- 172.21.15.3:0/2277148781 shutdown_connections 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.442+0000 7fc54dbd0640 1 -- 172.21.15.3:0/2277148781 wait complete. 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 Processor -- start 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 -- start start 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc5481000a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc5481005e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc548100b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5481a5c30 con 0x7fc54810b820 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5481a5da0 con 0x7fc548102450 2024-02-09T23:03:02.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc54dbd0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5481a5f60 con 0x7fc548102d70 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc547fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc548100b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc5481000a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc547fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc548100b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36090/0 (socket says 172.21.15.3:36090) 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc5481000a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:46918/0 (socket says 172.21.15.3:46918) 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc547fff640 1 -- 172.21.15.3:0/3803445682 learned_addr learned my addr 172.21.15.3:0/3803445682 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.443+0000 7fc546ffd640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc5481005e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc546ffd640 1 -- 172.21.15.3:0/3803445682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 msgr2=0x7fc5481000a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc546ffd640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc5481000a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc546ffd640 1 -- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 msgr2=0x7fc548100b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc546ffd640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc548100b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc546ffd640 1 -- 172.21.15.3:0/3803445682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc540009560 con 0x7fc548102d70 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc547fff640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc548100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc5477fe640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc5481000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc546ffd640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc5481005e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc53c037040 tx=0x7fc53c0346f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc544ff9640 1 -- 172.21.15.3:0/3803445682 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c02c7a0 con 0x7fc548102d70 2024-02-09T23:03:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc53c009490 con 0x7fc548102d70 2024-02-09T23:03:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc544ff9640 1 -- 172.21.15.3:0/3803445682 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53c0044b0 con 0x7fc548102d70 2024-02-09T23:03:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc544ff9640 1 -- 172.21.15.3:0/3803445682 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c03d3f0 con 0x7fc548102d70 2024-02-09T23:03:02.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.444+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5481a6550 con 0x7fc548102d70 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.446+0000 7fc544ff9640 1 -- 172.21.15.3:0/3803445682 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fc53c03b040 con 0x7fc548102d70 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 Processor -- start 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 -- start start 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663af5e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afe990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663b013b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663ab1020 con 0x55f663af63e0 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663a64320 con 0x55f663af5a30 2024-02-09T23:03:02.458 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5507a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663a61850 con 0x55f663afef60 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc544ff9640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5300a2230 0x7fc5300a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afe990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36092/0 (socket says 172.21.15.3:36092) 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 -- 172.21.15.3:0/2390934407 learned_addr learned my addr 172.21.15.3:0/2390934407 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc546ffd640 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663b013b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc547fff640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5300a2230 0x7fc5300a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc547fff640 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663af5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc544ff9640 1 -- 172.21.15.3:0/3803445682 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fc53c0ef950 con 0x7fc548102d70 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 -- 172.21.15.3:0/2390934407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 msgr2=0x55f663b013b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663b013b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 -- 172.21.15.3:0/2390934407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 msgr2=0x55f663af5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663af5e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.447+0000 7fc5477fe640 1 -- 172.21.15.3:0/2390934407 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f663a1f210 con 0x55f663af63e0 2024-02-09T23:03:02.459 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc547fff640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5300a2230 0x7fc5300a46f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc540013ef0 tx=0x7fc540009e50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afe990 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc538011860 tx=0x7fc538011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ca971b63e31667 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc52dffb640 1 -- 172.21.15.3:0/2390934407 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5380112f0 con 0x55f663af63e0 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc52dffb640 1 -- 172.21.15.3:0/2390934407 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc538003e50 con 0x55f663af63e0 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc52dffb640 1 -- 172.21.15.3:0/2390934407 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5380044f0 con 0x55f663af63e0 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc5507a3740 1 -- 172.21.15.3:0/2390934407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 msgr2=0x55f663afe990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.448+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afe990 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7fc538011860 tx=0x7fc538011d20 comp rx=0 tx=0).stop 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- 172.21.15.3:0/2390934407 shutdown_connections 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663b013b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afe990 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2390934407 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663af5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- 172.21.15.3:0/2390934407 >> 172.21.15.3:0/2390934407 conn(0x55f663af0100 msgr2=0x55f663af04e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- 172.21.15.3:0/2390934407 shutdown_connections 2024-02-09T23:03:02.460 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- 172.21.15.3:0/2390934407 wait complete. 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- start start 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 Processor -- start 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663afbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663afca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663a5d030 con 0x55f663af63e0 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663730430 con 0x55f663af5a30 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6636ece00 con 0x55f663afef60 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.449+0000 7fc5507a3740 10 client.0 _ll_register_callbacks cb 0x55f663a52030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663afbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36100/0 (socket says 172.21.15.3:36100) 2024-02-09T23:03:02.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 -- 172.21.15.3:0/2990097153 learned_addr learned my addr 172.21.15.3:0/2990097153 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc547fff640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663afca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 -- 172.21.15.3:0/2990097153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 msgr2=0x55f663afca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663afca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 -- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 msgr2=0x55f663afbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663afbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 -- 172.21.15.3:0/2990097153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5380114c0 con 0x55f663af63e0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc547fff640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663afca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663afbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc546ffd640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afc4c0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fc53c000c00 tx=0x7fc53c08b680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f9366edee7439a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc526ffd640 1 -- 172.21.15.3:0/2990097153 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c0ed500 con 0x55f663af63e0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc526ffd640 1 -- 172.21.15.3:0/2990097153 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fc53c0c0580 con 0x55f663af63e0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f66375c7e0 con 0x55f663af63e0 2024-02-09T23:03:02.462 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc526ffd640 1 -- 172.21.15.3:0/2990097153 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc53c003990 con 0x55f663af63e0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc5507a3740 10 client.19815 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55f663a5b6b0 con 0x55f663af63e0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.450+0000 7fc526ffd640 1 -- 172.21.15.3:0/2990097153 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f66375c7e0 con 0x55f663af63e0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc526ffd640 1 -- 172.21.15.3:0/2990097153 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc53c0c4860 con 0x55f663af63e0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5507a3740 10 client.19815 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5507a3740 10 client.19815 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5507a3740 20 client.19815 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b115c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5507a3740 4 client.19815 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f663799720 con 0x55f663b0f1d0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b115c0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19f1464076b1ac68 server_cookie=274d4625e73c6499 in_seq=0 out_seq=0 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.451+0000 7fc526ffd640 10 client.19815 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:02.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc526ffd640 1 -- 172.21.15.3:0/2990097153 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f663799720 con 0x55f663b0f1d0 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc526ffd640 10 client.19815 handle_command_reply: tid=0 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 client.19815 shutdown 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 msgr2=0x55f663afc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afc4c0 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7fc53c000c00 tx=0x7fc53c08b680 comp rx=0 tx=0).stop 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 shutdown_connections 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b115c0 crc :-1 s=READY pgs=323 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663afef60 0x55f663afca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc526ffd640 0 client.19815 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af63e0 0x55f663afc4c0 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2990097153 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af5a30 0x55f663afbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 >> 172.21.15.3:0/2990097153 conn(0x55f663af0100 msgr2=0x55f663af1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 shutdown_connections 2024-02-09T23:03:02.464 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 1 -- 172.21.15.3:0/2990097153 wait complete. 2024-02-09T23:03:02.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.452+0000 7fc5507a3740 20 client.19815 trim_cache size 0 max 16384 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.464+0000 7fc5507a3740 1 Processor -- start 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 -- start start 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 0x55f663afe6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af63e0 0x55f663afc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663afef60 0x55f663afa850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f6639e2130 con 0x55f663af5a30 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663af5380 con 0x55f663afef60 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5507a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663aeba40 con 0x55f663af63e0 2024-02-09T23:03:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 0x55f663afe6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 0x55f663afe6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36110/0 (socket says 172.21.15.3:36110) 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 -- 172.21.15.3:0/2428725284 learned_addr learned my addr 172.21.15.3:0/2428725284 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc546ffd640 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af63e0 0x55f663afc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc547fff640 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663afef60 0x55f663afa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 -- 172.21.15.3:0/2428725284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af63e0 msgr2=0x55f663afc820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af63e0 0x55f663afc820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 -- 172.21.15.3:0/2428725284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663afef60 msgr2=0x55f663afa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663afef60 0x55f663afa850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 -- 172.21.15.3:0/2428725284 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f6639f7c90 con 0x55f663af5a30 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc5477fe640 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 0x55f663afe6b0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc538011860 tx=0x7fc538004030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18667c6b7f7e9e0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc52d7fa640 1 -- 172.21.15.3:0/2428725284 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc538009500 con 0x55f663af5a30 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc52d7fa640 1 -- 172.21.15.3:0/2428725284 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc53800fb40 con 0x55f663af5a30 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.465+0000 7fc52d7fa640 1 -- 172.21.15.3:0/2428725284 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5380059c0 con 0x55f663af5a30 2024-02-09T23:03:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 -- 172.21.15.3:0/2428725284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 msgr2=0x55f663afe6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 0x55f663afe6b0 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc538011860 tx=0x7fc538004030 comp rx=0 tx=0).stop 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 -- 172.21.15.3:0/2428725284 shutdown_connections 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663afef60 0x55f663afa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af63e0 0x55f663afc820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 --2- 172.21.15.3:0/2428725284 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663af5a30 0x55f663afe6b0 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 -- 172.21.15.3:0/2428725284 >> 172.21.15.3:0/2428725284 conn(0x55f663af0100 msgr2=0x55f663af7210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 -- 172.21.15.3:0/2428725284 shutdown_connections 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 -- 172.21.15.3:0/2428725284 wait complete. 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 -- start start 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.466+0000 7fc5507a3740 1 Processor -- start 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af5a30 0x55f663b03da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af63e0 0x55f663b042e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 0x55f663b64680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663b04ab0 con 0x55f663afef60 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663b04c20 con 0x55f663af63e0 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f663b64bf0 con 0x55f663af5a30 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5507a3740 10 client.0 _ll_register_callbacks cb 0x7fc538009ee0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 0x55f663b64680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc546ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af63e0 0x55f663b042e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc5477fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af5a30 0x55f663b03da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 0x55f663b64680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36124/0 (socket says 172.21.15.3:36124) 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc546ffd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af63e0 0x55f663b042e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:46970/0 (socket says 172.21.15.3:46970) 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 -- 172.21.15.3:0/840663406 learned_addr learned my addr 172.21.15.3:0/840663406 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 -- 172.21.15.3:0/840663406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af5a30 msgr2=0x55f663b03da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af5a30 0x55f663b03da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 -- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af63e0 msgr2=0x55f663b042e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af63e0 0x55f663b042e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 -- 172.21.15.3:0/840663406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f663a1f210 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc547fff640 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 0x55f663b64680 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fc5400133d0 tx=0x7fc540004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72821f952fc81dc5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc526ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc526ffd640 1 -- 172.21.15.3:0/840663406 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc540027070 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.467+0000 7fc526ffd640 1 -- 172.21.15.3:0/840663406 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fc540012720 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc538003e50 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc526ffd640 1 -- 172.21.15.3:0/840663406 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc5400128e0 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 10 client.19821 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55f663a1da70 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc526ffd640 1 -- 172.21.15.3:0/840663406 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc540038020 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc526ffd640 1 -- 172.21.15.3:0/840663406 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc54002bb10 con 0x55f663afef60 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 10 client.19821 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 10 client.19821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 20 client.19821 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b68480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 4 client.19821 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f6637e8890 con 0x55f663b0f1d0 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.468+0000 7fc5477fe640 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b68480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc526ffd640 10 client.19821 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5477fe640 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b68480 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f66b3ab09a0b176b server_cookie=9f3605342a49cd88 in_seq=0 out_seq=0 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc526ffd640 1 -- 172.21.15.3:0/840663406 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f6637e8890 con 0x55f663b0f1d0 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc526ffd640 10 client.19821 handle_command_reply: tid=0 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 client.19821 shutdown 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 msgr2=0x55f663b64680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 0x55f663b64680 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7fc5400133d0 tx=0x7fc540004730 comp rx=0 tx=0).stop 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 shutdown_connections 2024-02-09T23:03:02.479 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f663b0f1d0 0x55f663b68480 crc :-1 s=READY pgs=324 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f663af63e0 0x55f663b042e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f663afef60 0x55f663b64680 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc526ffd640 0 client.19821 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 --2- 172.21.15.3:0/840663406 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f663af5a30 0x55f663b03da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 >> 172.21.15.3:0/840663406 conn(0x55f663af0100 msgr2=0x55f663af1550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 shutdown_connections 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 1 -- 172.21.15.3:0/840663406 wait complete. 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.469+0000 7fc5507a3740 20 client.19821 trim_cache size 0 max 16384 2024-02-09T23:03:02.480 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:02.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5300a2230 msgr2=0x7fc5300a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5300a2230 0x7fc5300a46f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc540013ef0 tx=0x7fc540009e50 comp rx=0 tx=0).stop 2024-02-09T23:03:02.481 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 msgr2=0x7fc5481005e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc5481005e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fc53c037040 tx=0x7fc53c0346f0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 shutdown_connections 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fc5300a2230 0x7fc5300a46f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fc54810b820 0x7fc548100b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fc548102d70 0x7fc5481005e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 --2- 172.21.15.3:0/3803445682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fc548102450 0x7fc5481000a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 >> 172.21.15.3:0/3803445682 conn(0x7fc5480772e0 msgr2=0x7fc5480fd250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 shutdown_connections 2024-02-09T23:03:02.482 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:02.470+0000 7fc54dbd0640 1 -- 172.21.15.3:0/3803445682 wait complete. 2024-02-09T23:03:02.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:02 smithi003 ceph-mon[32678]: pgmap v368: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 245 op/s 2024-02-09T23:03:02.483 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:02 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/959430949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:03:02.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 427365 3.51 MB/sec execute 536 sec latency 134.593 ms 2024-02-09T23:03:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:02 smithi064 ceph-mon[36638]: pgmap v368: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 245 op/s 2024-02-09T23:03:02.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:02 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/959430949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:03:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:02 smithi089 ceph-mon[36665]: pgmap v368: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 245 op/s 2024-02-09T23:03:02.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:02 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/959430949' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:03:03.485 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:03:03.486 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:03:03.486 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/2870089708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0068810 msgr2=0x7fcaf0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/2870089708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0068810 0x7fcaf0068bf0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7fcad8009930 tx=0x7fcad802bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/2870089708 shutdown_connections 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/2870089708 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcaf00691c0 0x7fcaf0105e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/2870089708 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0106570 0x7fcaf010e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/2870089708 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0068810 0x7fcaf0068bf0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/2870089708 >> 172.21.15.3:0/2870089708 conn(0x7fcaf0077230 msgr2=0x7fcaf0077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/2870089708 shutdown_connections 2024-02-09T23:03:03.613 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/2870089708 wait complete. 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 Processor -- start 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.607+0000 7fcaf6b8c640 1 -- start start 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf6b8c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0068810 0x7fcaf019d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf6b8c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcaf0106570 0x7fcaf019db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf6b8c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 0x7fcaf0197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf6b8c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0198150 con 0x7fcaf0197680 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf6b8c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf01982c0 con 0x7fcaf0106570 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf6b8c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf0198480 con 0x7fcaf0068810 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 0x7fcaf0197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0068810 0x7fcaf019d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 0x7fcaf0197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36152/0 (socket says 172.21.15.3:36152) 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0068810 0x7fcaf019d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54568/0 (socket says 172.21.15.3:54568) 2024-02-09T23:03:03.614 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 -- 172.21.15.3:0/17946322 learned_addr learned my addr 172.21.15.3:0/17946322 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaeffff640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcaf0106570 0x7fcaf019db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 -- 172.21.15.3:0/17946322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0068810 msgr2=0x7fcaf019d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0068810 0x7fcaf019d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 -- 172.21.15.3:0/17946322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcaf0106570 msgr2=0x7fcaf019db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcaf0106570 0x7fcaf019db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 -- 172.21.15.3:0/17946322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad8009560 con 0x7fcaf0197680 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaf5129640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 0x7fcaf0197b00 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcae800eaa0 tx=0x7fcae800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.608+0000 7fcaedffb640 1 -- 172.21.15.3:0/17946322 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae800cdb0 con 0x7fcaf0197680 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.609+0000 7fcaedffb640 1 -- 172.21.15.3:0/17946322 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcae8005020 con 0x7fcaf0197680 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.609+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf01a4420 con 0x7fcaf0197680 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.609+0000 7fcaedffb640 1 -- 172.21.15.3:0/17946322 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae80116b0 con 0x7fcaf0197680 2024-02-09T23:03:03.615 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.609+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf01a4900 con 0x7fcaf0197680 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.611+0000 7fcaedffb640 1 -- 172.21.15.3:0/17946322 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fcae8005520 con 0x7fcaf0197680 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 Processor -- start 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 -- start start 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x562787206570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x562787211a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562787174000 con 0x562787206ab0 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562787171530 con 0x562787206190 2024-02-09T23:03:03.622 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf9767740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56278716cd10 con 0x56278720f690 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaedffb640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcadc0a2230 0x7fcadc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36166/0 (socket says 172.21.15.3:36166) 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 -- 172.21.15.3:0/1273953530 learned_addr learned my addr 172.21.15.3:0/1273953530 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x562787206570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x562787211a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf5129640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcadc0a2230 0x7fcadc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaedffb640 1 -- 172.21.15.3:0/17946322 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fcae80ce240 con 0x7fcaf0197680 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 -- 172.21.15.3:0/1273953530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 msgr2=0x562787211a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x562787211a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 -- 172.21.15.3:0/1273953530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 msgr2=0x562787206570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.612+0000 7fcaf4928640 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x562787206570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.613+0000 7fcaf4928640 1 -- 172.21.15.3:0/1273953530 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562787113210 con 0x562787206ab0 2024-02-09T23:03:03.623 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.613+0000 7fcaf4928640 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720f150 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcad8036040 tx=0x7fcad802c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b71aecd9b72290 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.613+0000 7fcaf5129640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcadc0a2230 0x7fcadc0a46f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcae8097120 tx=0x7fcae80ce500 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.613+0000 7fcae6ffd640 1 -- 172.21.15.3:0/1273953530 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8004880 con 0x562787206ab0 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.613+0000 7fcae6ffd640 1 -- 172.21.15.3:0/1273953530 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcad80049e0 con 0x562787206ab0 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.613+0000 7fcae6ffd640 1 -- 172.21.15.3:0/1273953530 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8005150 con 0x562787206ab0 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 -- 172.21.15.3:0/1273953530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 msgr2=0x56278720f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720f150 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7fcad8036040 tx=0x7fcad802c680 comp rx=0 tx=0).stop 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 -- 172.21.15.3:0/1273953530 shutdown_connections 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x562787211a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720f150 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1273953530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x562787206570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 -- 172.21.15.3:0/1273953530 >> 172.21.15.3:0/1273953530 conn(0x5627872008c0 msgr2=0x562787200ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 -- 172.21.15.3:0/1273953530 shutdown_connections 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 -- 172.21.15.3:0/1273953530 wait complete. 2024-02-09T23:03:03.624 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 -- start start 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.614+0000 7fcaf9767740 1 Processor -- start 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562787111e60 con 0x562787206ab0 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562786e404b0 con 0x562787206190 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627870f2670 con 0x56278720f690 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 10 client.0 _ll_register_callbacks cb 0x56278710a190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36180/0 (socket says 172.21.15.3:36180) 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 -- 172.21.15.3:0/1972840439 learned_addr learned my addr 172.21.15.3:0/1972840439 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.625 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 -- 172.21.15.3:0/1972840439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 msgr2=0x56278720d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 -- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 msgr2=0x56278720c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 -- 172.21.15.3:0/1972840439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf010e930 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf4928640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720cb10 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fcad802c170 tx=0x7fcad8004570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f15743d16b08ad2 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcacffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcacffff640 1 -- 172.21.15.3:0/1972840439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad8004880 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcacffff640 1 -- 172.21.15.3:0/1972840439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fcad804d070 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x562786e6c860 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.615+0000 7fcacffff640 1 -- 172.21.15.3:0/1972840439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcad802cbe0 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 10 client.19833 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56278716b390 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcacffff640 1 -- 172.21.15.3:0/1972840439 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x562786e6c860 con 0x562787206ab0 2024-02-09T23:03:03.626 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcacffff640 1 -- 172.21.15.3:0/1972840439 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcad803c050 con 0x562787206ab0 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 10 client.19833 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 10 client.19833 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 20 client.19833 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56278721f270 0x562787221690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 4 client.19833 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562786ebd410 con 0x56278721f270 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.616+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56278721f270 0x562787221690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcacffff640 10 client.19833 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaeffff640 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56278721f270 0x562787221690 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d4024f15772646ac server_cookie=73a1069952ee33b9 in_seq=0 out_seq=0 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcacffff640 1 -- 172.21.15.3:0/1972840439 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562786ebd410 con 0x56278721f270 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcacffff640 10 client.19833 handle_command_reply: tid=0 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaf9767740 1 client.19833 shutdown 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 msgr2=0x56278720cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.627 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720cb10 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7fcad802c170 tx=0x7fcad8004570 comp rx=0 tx=0).stop 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 shutdown_connections 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56278721f270 0x562787221690 crc :-1 s=READY pgs=325 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.617+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720cb10 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcaf9767740 1 --2- 172.21.15.3:0/1972840439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcacffff640 0 client.19833 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 >> 172.21.15.3:0/1972840439 conn(0x5627872008c0 msgr2=0x562787202610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 shutdown_connections 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcaf9767740 1 -- 172.21.15.3:0/1972840439 wait complete. 2024-02-09T23:03:03.628 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.618+0000 7fcaf9767740 20 client.19833 trim_cache size 0 max 16384 2024-02-09T23:03:03.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:03 smithi003 ceph-mon[32678]: osdmap e102: 12 total, 12 up, 12 in 2024-02-09T23:03:03.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:03 smithi003 ceph-mon[32678]: pgmap v370: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.633+0000 7fcaf9767740 1 Processor -- start 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.633+0000 7fcaf9767740 1 -- start start 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720ce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf9767740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562786e6c890 con 0x562787206ab0 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf9767740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627872014c0 con 0x562787206190 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf9767740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x562787205500 con 0x56278720f690 2024-02-09T23:03:03.642 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36186/0 (socket says 172.21.15.3:36186) 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 -- 172.21.15.3:0/2499755758 learned_addr learned my addr 172.21.15.3:0/2499755758 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 -- 172.21.15.3:0/2499755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 msgr2=0x56278720b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720b5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 -- 172.21.15.3:0/2499755758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 msgr2=0x56278720ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 -- 172.21.15.3:0/2499755758 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562787107c70 con 0x562787206ab0 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcaf4928640 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720ce70 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fcad8006010 tx=0x7fcad802c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519088e709d72209 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.643 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcae67fc640 1 -- 172.21.15.3:0/2499755758 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad802caf0 con 0x562787206ab0 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcae67fc640 1 -- 172.21.15.3:0/2499755758 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcad802cc50 con 0x562787206ab0 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.634+0000 7fcae67fc640 1 -- 172.21.15.3:0/2499755758 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcad803f440 con 0x562787206ab0 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 -- 172.21.15.3:0/2499755758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 msgr2=0x56278720ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720ce70 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7fcad8006010 tx=0x7fcad802c3e0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 -- 172.21.15.3:0/2499755758 shutdown_connections 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56278720f690 0x56278720b5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x562787206ab0 0x56278720ce70 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2499755758 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206190 0x56278720ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 -- 172.21.15.3:0/2499755758 >> 172.21.15.3:0/2499755758 conn(0x5627872008c0 msgr2=0x56278720aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 -- 172.21.15.3:0/2499755758 shutdown_connections 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 -- 172.21.15.3:0/2499755758 wait complete. 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 -- start start 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.635+0000 7fcaf9767740 1 Processor -- start 2024-02-09T23:03:03.644 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562787206190 0x562787216850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206ab0 0x562787214bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 0x562787215130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x562787215800 con 0x56278720f690 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x562787215970 con 0x562787206ab0 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627872757a0 con 0x562787206190 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf9767740 10 client.0 _ll_register_callbacks cb 0x7fcad8009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 0x562787215130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 0x562787215130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36190/0 (socket says 172.21.15.3:36190) 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 -- 172.21.15.3:0/2907703594 learned_addr learned my addr 172.21.15.3:0/2907703594 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaf4928640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206ab0 0x562787214bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562787206190 0x562787216850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 -- 172.21.15.3:0/2907703594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562787206190 msgr2=0x562787216850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.645 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562787206190 0x562787216850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 -- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206ab0 msgr2=0x562787214bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206ab0 0x562787214bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 -- 172.21.15.3:0/2907703594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56278716b390 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcaeffff640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 0x562787215130 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcaf006a9d0 tx=0x7fcad8004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f64c6938cbf26441 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcacffff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.636+0000 7fcacffff640 1 -- 172.21.15.3:0/2907703594 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcae0005020 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcacffff640 1 -- 172.21.15.3:0/2907703594 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fcae0005180 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcad80049e0 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcacffff640 1 -- 172.21.15.3:0/2907703594 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcae0005340 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 10 client.19839 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x56278712dac0 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcacffff640 1 -- 172.21.15.3:0/2907703594 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcad80049e0 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcacffff640 1 -- 172.21.15.3:0/2907703594 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcae000f5a0 con 0x56278720f690 2024-02-09T23:03:03.646 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 10 client.19839 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 10 client.19839 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 20 client.19839 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562787276f70 0x562787279390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 4 client.19839 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562786ea97b0 con 0x562787276f70 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.637+0000 7fcaf4928640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562787276f70 0x562787279390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.638+0000 7fcacffff640 10 client.19839 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.647 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.638+0000 7fcaf4928640 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562787276f70 0x562787279390 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10d35c2a8542b6b2 server_cookie=c4d21b8768d56466 in_seq=0 out_seq=0 2024-02-09T23:03:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcacffff640 1 -- 172.21.15.3:0/2907703594 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562786ea97b0 con 0x562787276f70 2024-02-09T23:03:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcacffff640 10 client.19839 handle_command_reply: tid=0 2024-02-09T23:03:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 client.19839 shutdown 2024-02-09T23:03:03.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 msgr2=0x562787215130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 0x562787215130 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7fcaf006a9d0 tx=0x7fcad8004df0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 shutdown_connections 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x562787276f70 0x562787279390 crc :-1 s=READY pgs=326 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x562787206ab0 0x562787214bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56278720f690 0x562787215130 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 --2- 172.21.15.3:0/2907703594 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x562787206190 0x562787216850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 >> 172.21.15.3:0/2907703594 conn(0x5627872008c0 msgr2=0x5627871ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcacffff640 0 client.19839 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 shutdown_connections 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 1 -- 172.21.15.3:0/2907703594 wait complete. 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.639+0000 7fcaf9767740 20 client.19839 trim_cache size 0 max 16384 2024-02-09T23:03:03.649 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:03:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:03:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "242722c3-dd7f-442d-9718-94a45e85505b", 2024-02-09T23:03:03.650 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:03:03.650 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcadc0a2230 msgr2=0x7fcadc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcadc0a2230 0x7fcadc0a46f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fcae8097120 tx=0x7fcae80ce500 comp rx=0 tx=0).stop 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 msgr2=0x7fcaf0197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 0x7fcaf0197b00 secure :-1 s=READY pgs=1688 cs=0 l=1 rev1=1 crypto rx=0x7fcae800eaa0 tx=0x7fcae800ef60 comp rx=0 tx=0).stop 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 shutdown_connections 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcadc0a2230 0x7fcadc0a46f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcaf0106570 0x7fcaf019db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcaf0197680 0x7fcaf0197b00 unknown :-1 s=CLOSED pgs=1688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 --2- 172.21.15.3:0/17946322 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcaf0068810 0x7fcaf019d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 >> 172.21.15.3:0/17946322 conn(0x7fcaf0077230 msgr2=0x7fcaf00fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 shutdown_connections 2024-02-09T23:03:03.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.640+0000 7fcaf6b8c640 1 -- 172.21.15.3:0/17946322 wait complete. 2024-02-09T23:03:03.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:03:03.655 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:03:03.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 427368 3.50 MB/sec execute 537 sec latency 1132.965 ms 2024-02-09T23:03:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:03 smithi064 ceph-mon[36638]: osdmap e102: 12 total, 12 up, 12 in 2024-02-09T23:03:03.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:03 smithi064 ceph-mon[36638]: pgmap v370: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-02-09T23:03:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:03 smithi089 ceph-mon[36665]: osdmap e102: 12 total, 12 up, 12 in 2024-02-09T23:03:03.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:03 smithi089 ceph-mon[36665]: pgmap v370: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 -- 172.21.15.3:0/3388105693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c106500 msgr2=0x7f762c10e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 --2- 172.21.15.3:0/3388105693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c106500 0x7f762c10e0d0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f7624009cd0 tx=0x7f762402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 -- 172.21.15.3:0/3388105693 shutdown_connections 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 --2- 172.21.15.3:0/3388105693 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c106500 0x7f762c10e0d0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 --2- 172.21.15.3:0/3388105693 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c101de0 0x7f762c105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 --2- 172.21.15.3:0/3388105693 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f762c101430 0x7f762c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 -- 172.21.15.3:0/3388105693 >> 172.21.15.3:0/3388105693 conn(0x7f762c0fccd0 msgr2=0x7f762c0ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 -- 172.21.15.3:0/3388105693 shutdown_connections 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 -- 172.21.15.3:0/3388105693 wait complete. 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.760+0000 7f7631496640 1 Processor -- start 2024-02-09T23:03:03.766 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 -- start start 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 0x7f762c19d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f762c106500 0x7f762c19db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1976d0 0x7f762c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c1981a0 con 0x7f762c101430 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c198310 con 0x7f762c106500 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f7631496640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762c1984d0 con 0x7f762c1976d0 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 0x7f762c19d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 0x7f762c19d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36208/0 (socket says 172.21.15.3:36208) 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762affd640 1 -- 172.21.15.3:0/2601927315 learned_addr learned my addr 172.21.15.3:0/2601927315 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1976d0 0x7f762c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f762c106500 0x7f762c19db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 -- 172.21.15.3:0/2601927315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f762c106500 msgr2=0x7f762c19db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f762c106500 0x7f762c19db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.767 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 -- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 msgr2=0x7f762c19d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 0x7f762c19d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 -- 172.21.15.3:0/2601927315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7624009900 con 0x7f762c1976d0 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762affd640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 0x7f762c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.761+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1976d0 0x7f762c197b50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f762402c630 tx=0x7f7624034a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.762+0000 7f7613fff640 1 -- 172.21.15.3:0/2601927315 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240328b0 con 0x7f762c1976d0 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.762+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f762c1a43c0 con 0x7f762c1976d0 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.762+0000 7f7613fff640 1 -- 172.21.15.3:0/2601927315 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7624003e50 con 0x7f762c1976d0 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.762+0000 7f7613fff640 1 -- 172.21.15.3:0/2601927315 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7624004860 con 0x7f762c1976d0 2024-02-09T23:03:03.768 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.762+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c1a4840 con 0x7f762c1976d0 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7613fff640 1 -- 172.21.15.3:0/2601927315 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f762400a040 con 0x7f762c1976d0 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 Processor -- start 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 -- start start 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7613fff640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76140a2230 0x7f76140a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 0x55ba72fe8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.775 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72fa3d00 con 0x55ba72fe8980 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72f582f0 con 0x55ba72ff1eb0 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7634062740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72f55820 con 0x55ba72fe9330 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76140a2230 0x7f76140a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f762affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 0x55ba72fe8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f762affd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 0x55ba72fe8d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36218/0 (socket says 172.21.15.3:36218) 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f762affd640 1 -- 172.21.15.3:0/4288410477 learned_addr learned my addr 172.21.15.3:0/4288410477 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.764+0000 7f7613fff640 1 -- 172.21.15.3:0/2601927315 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f76240f0c70 con 0x7f762c1976d0 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762a7fc640 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762b7fe640 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762affd640 1 -- 172.21.15.3:0/4288410477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 msgr2=0x55ba72ff18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762affd640 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.776 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762affd640 1 -- 172.21.15.3:0/4288410477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 msgr2=0x55ba72ff4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762affd640 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762affd640 1 -- 172.21.15.3:0/4288410477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba72f12ff0 con 0x55ba72fe8980 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762b7fe640 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762affd640 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 0x55ba72fe8d60 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f761c011650 tx=0x7f761c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64713070f48382f0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76140a2230 0x7f76140a46f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f762c198ed0 tx=0x7f7620009a50 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f76117fa640 1 -- 172.21.15.3:0/4288410477 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c00f6c0 con 0x55ba72fe8980 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f76117fa640 1 -- 172.21.15.3:0/4288410477 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f761c00fd00 con 0x55ba72fe8980 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.765+0000 7f76117fa640 1 -- 172.21.15.3:0/4288410477 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c019e80 con 0x55ba72fe8980 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- 172.21.15.3:0/4288410477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 msgr2=0x55ba72fe8d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 0x55ba72fe8d60 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f761c011650 tx=0x7f761c009f90 comp rx=0 tx=0).stop 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- 172.21.15.3:0/4288410477 shutdown_connections 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.777 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- 172.21.15.3:0/4288410477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe8980 0x55ba72fe8d60 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- 172.21.15.3:0/4288410477 >> 172.21.15.3:0/4288410477 conn(0x55ba72fe3050 msgr2=0x55ba72fe3430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- 172.21.15.3:0/4288410477 shutdown_connections 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- 172.21.15.3:0/4288410477 wait complete. 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- start start 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 Processor -- start 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 0x55ba72feede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 0x55ba72fef320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72fef860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72f51000 con 0x55ba72fe9330 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72c24430 con 0x55ba72ff1eb0 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.766+0000 7f7634062740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72be0e00 con 0x55ba72fe8980 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f7634062740 10 client.0 _ll_register_callbacks cb 0x55ba72f457f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 0x55ba72fef320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.778 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 0x55ba72feede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762affd640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 0x55ba72feede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:54612/0 (socket says 172.21.15.3:54612) 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 0x55ba72fef320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36230/0 (socket says 172.21.15.3:36230) 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72fef860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762affd640 1 -- 172.21.15.3:0/2337441420 learned_addr learned my addr 172.21.15.3:0/2337441420 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 -- 172.21.15.3:0/2337441420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 msgr2=0x55ba72feede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 0x55ba72feede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 -- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 msgr2=0x55ba72fef860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72fef860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 -- 172.21.15.3:0/2337441420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c009e00 con 0x55ba72fe9330 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762affd640 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 0x55ba72feede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 0x55ba72fef320 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x55ba72fe9730 tx=0x7f762001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=905cc5ed341bdd21 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f760eff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f760eff5640 1 -- 172.21.15.3:0/2337441420 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620004520 con 0x55ba72fe9330 2024-02-09T23:03:03.779 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f760eff5640 1 -- 172.21.15.3:0/2337441420 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f7620013170 con 0x55ba72fe9330 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba72c507e0 con 0x55ba72fe9330 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f760eff5640 1 -- 172.21.15.3:0/2337441420 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7620003b40 con 0x55ba72fe9330 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f7634062740 10 client.19851 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.767+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55ba72f4f680 con 0x55ba72fe9330 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f760eff5640 1 -- 172.21.15.3:0/2337441420 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ba72c507e0 con 0x55ba72fe9330 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f760eff5640 1 -- 172.21.15.3:0/2337441420 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7620010c40 con 0x55ba72fe9330 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f7634062740 10 client.19851 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f7634062740 10 client.19851 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f7634062740 20 client.19851 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:03.780 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f7634062740 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73002330 0x55ba73004750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f7634062740 4 client.19851 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba72c8d720 con 0x55ba73002330 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f762affd640 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73002330 0x55ba73004750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f762affd640 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73002330 0x55ba73004750 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932d502abb218ff1 server_cookie=6676c81466ba882 in_seq=0 out_seq=0 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.768+0000 7f760eff5640 10 client.19851 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f760eff5640 1 -- 172.21.15.3:0/2337441420 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ba72c8d720 con 0x55ba73002330 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f760eff5640 10 client.19851 handle_command_reply: tid=0 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 client.19851 shutdown 2024-02-09T23:03:03.781 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 msgr2=0x55ba72fef320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 0x55ba72fef320 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x55ba72fe9730 tx=0x7f762001d000 comp rx=0 tx=0).stop 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 shutdown_connections 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73002330 0x55ba73004750 crc :-1 s=READY pgs=327 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72ff1eb0 0x55ba72fef860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72fe9330 0x55ba72fef320 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 --2- 172.21.15.3:0/2337441420 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe8980 0x55ba72feede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f760eff5640 0 client.19851 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 >> 172.21.15.3:0/2337441420 conn(0x55ba72fe3050 msgr2=0x55ba72fe4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 shutdown_connections 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.769+0000 7f7634062740 1 -- 172.21.15.3:0/2337441420 wait complete. 2024-02-09T23:03:03.782 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.770+0000 7f7634062740 20 client.19851 trim_cache size 0 max 16384 2024-02-09T23:03:03.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 Processor -- start 2024-02-09T23:03:03.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 -- start start 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff1510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72fea100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72fea640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72ed5fd0 con 0x55ba72ff1eb0 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72fe2ec0 con 0x55ba72fe8980 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f7634062740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72fb8ab0 con 0x55ba72fe9330 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72fea640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72fea640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36236/0 (socket says 172.21.15.3:36236) 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 -- 172.21.15.3:0/2537103399 learned_addr learned my addr 172.21.15.3:0/2537103399 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762a7fc640 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72fea100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762affd640 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 -- 172.21.15.3:0/2537103399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 msgr2=0x55ba72fea100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72fea100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 -- 172.21.15.3:0/2537103399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 msgr2=0x55ba72ff1510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff1510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 -- 172.21.15.3:0/2537103399 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba72eebb30 con 0x55ba72ff1eb0 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.787+0000 7f762b7fe640 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72fea640 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f76240ba120 tx=0x7f76240dd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f181a919a906b7bb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7610ff9640 1 -- 172.21.15.3:0/2537103399 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240bd750 con 0x55ba72ff1eb0 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7610ff9640 1 -- 172.21.15.3:0/2537103399 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76240c18a0 con 0x55ba72ff1eb0 2024-02-09T23:03:03.793 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7610ff9640 1 -- 172.21.15.3:0/2537103399 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240ee620 con 0x55ba72ff1eb0 2024-02-09T23:03:03.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 -- 172.21.15.3:0/2537103399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 msgr2=0x55ba72fea640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72fea640 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f76240ba120 tx=0x7f76240dd040 comp rx=0 tx=0).stop 2024-02-09T23:03:03.797 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 -- 172.21.15.3:0/2537103399 shutdown_connections 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72fea640 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72fea100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 --2- 172.21.15.3:0/2537103399 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff1510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 -- 172.21.15.3:0/2537103399 >> 172.21.15.3:0/2537103399 conn(0x55ba72fe3050 msgr2=0x55ba72fde080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.788+0000 7f7634062740 1 -- 172.21.15.3:0/2537103399 shutdown_connections 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 -- 172.21.15.3:0/2537103399 wait complete. 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 -- start start 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 Processor -- start 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72ff8150 con 0x55ba72ff1eb0 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba72ff82c0 con 0x55ba72fe8980 2024-02-09T23:03:03.798 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba73058880 con 0x55ba72fe9330 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f7634062740 10 client.0 _ll_register_callbacks cb 0x7f761c009c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f762b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:36248/0 (socket says 172.21.15.3:36248) 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f762b7fe640 1 -- 172.21.15.3:0/1952984149 learned_addr learned my addr 172.21.15.3:0/1952984149 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f762affd640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.789+0000 7f762a7fc640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762b7fe640 1 -- 172.21.15.3:0/1952984149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 msgr2=0x55ba72ff7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762b7fe640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762b7fe640 1 -- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 msgr2=0x55ba72ff91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762b7fe640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762b7fe640 1 -- 172.21.15.3:0/1952984149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762c1a4840 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762a7fc640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f762b7fe640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff7a80 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f7624006010 tx=0x7f7624004300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ebd20decaba10d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f760eff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f760eff5640 1 -- 172.21.15.3:0/1952984149 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76240c5ef0 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f760eff5640 1 -- 172.21.15.3:0/1952984149 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f76240bd750 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f761c00fd00 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f760eff5640 1 -- 172.21.15.3:0/1952984149 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76240d60d0 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f7634062740 10 client.19857 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55ba72f11850 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.790+0000 7f760eff5640 1 -- 172.21.15.3:0/1952984149 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f761c00fd00 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f760eff5640 1 -- 172.21.15.3:0/1952984149 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f76240c5a60 con 0x55ba72ff1eb0 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 10 client.19857 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:03.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 10 client.19857 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 20 client.19857 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73059af0 0x55ba7305bf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 4 client.19857 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ba72cdc890 con 0x55ba73059af0 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f762affd640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73059af0 0x55ba7305bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f762affd640 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73059af0 0x55ba7305bf10 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5b215e426b0c778 server_cookie=ffb3da92f2d9fe5 in_seq=0 out_seq=0 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f760eff5640 10 client.19857 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f760eff5640 1 -- 172.21.15.3:0/1952984149 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55ba72cdc890 con 0x55ba73059af0 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f760eff5640 10 client.19857 handle_command_reply: tid=0 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.791+0000 7f7634062740 1 client.19857 shutdown 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 msgr2=0x55ba72ff7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff7a80 secure :-1 s=READY pgs=1697 cs=0 l=1 rev1=1 crypto rx=0x7f7624006010 tx=0x7f7624004300 comp rx=0 tx=0).stop 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 shutdown_connections 2024-02-09T23:03:03.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55ba73059af0 0x55ba7305bf10 crc :-1 s=READY pgs=328 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55ba72ff1eb0 0x55ba72ff7a80 unknown :-1 s=CLOSED pgs=1697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55ba72fe9330 0x55ba72ff7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f760eff5640 0 client.19857 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 --2- 172.21.15.3:0/1952984149 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55ba72fe8980 0x55ba72ff91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 >> 172.21.15.3:0/1952984149 conn(0x55ba72fe3050 msgr2=0x55ba72fea100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 shutdown_connections 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 1 -- 172.21.15.3:0/1952984149 wait complete. 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7634062740 20 client.19857 trim_cache size 0 max 16384 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:03:03.802 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "242722c3-dd7f-442d-9718-94a45e85505b": { 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "242722c3-dd7f-442d-9718-94a45e85505b", 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:03:03.803 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76140a2230 msgr2=0x7f76140a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76140a2230 0x7f76140a46f0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f762c198ed0 tx=0x7f7620009a50 comp rx=0 tx=0).stop 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1976d0 msgr2=0x7f762c197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1976d0 0x7f762c197b50 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f762402c630 tx=0x7f7624034a90 comp rx=0 tx=0).stop 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 shutdown_connections 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f76140a2230 0x7f76140a46f0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f762c106500 0x7f762c19db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f762c1976d0 0x7f762c197b50 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 --2- 172.21.15.3:0/2601927315 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f762c101430 0x7f762c19d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.792+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 >> 172.21.15.3:0/2601927315 conn(0x7f762c0fccd0 msgr2=0x7f762c0ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.793+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 shutdown_connections 2024-02-09T23:03:03.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:03.793+0000 7f7631496640 1 -- 172.21.15.3:0/2601927315 wait complete. 2024-02-09T23:03:03.807 INFO:tasks.cephfs.filesystem:scrub status for tag:242722c3-dd7f-442d-9718-94a45e85505b - {'path': '/', 'tag': '242722c3-dd7f-442d-9718-94a45e85505b', 'options': 'recursive,force'} 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:04 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 429464 3.52 MB/sec execute 538 sec latency 29.574 ms 2024-02-09T23:03:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:03:04.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:04.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:04 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:04.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:04 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:05 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:05.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:05 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:05 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:05.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:05 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:05 smithi003 ceph-mon[32678]: pgmap v371: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-02-09T23:03:05.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 430728 3.52 MB/sec execute 539 sec latency 11.604 ms 2024-02-09T23:03:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:05 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:05 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:05 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:05 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:05 smithi064 ceph-mon[36638]: pgmap v371: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-02-09T23:03:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:05 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:05 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:05 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:05 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:05 smithi089 ceph-mon[36665]: pgmap v371: 97 pgs: 97 active+clean; 165 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.1 MiB/s wr, 211 op/s 2024-02-09T23:03:06.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 430914 3.51 MB/sec execute 540 sec latency 806.733 ms 2024-02-09T23:03:07.262 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 -- 172.21.15.3:0/2646637048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8100730 msgr2=0x7f35a8105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 --2- 172.21.15.3:0/2646637048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8100730 0x7f35a8105ed0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7f359c009930 tx=0x7f359c02bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 -- 172.21.15.3:0/2646637048 shutdown_connections 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 --2- 172.21.15.3:0/2646637048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a8106600 0x7f35a810e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 --2- 172.21.15.3:0/2646637048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8100730 0x7f35a8105ed0 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 --2- 172.21.15.3:0/2646637048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 0x7f35a8100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 -- 172.21.15.3:0/2646637048 >> 172.21.15.3:0/2646637048 conn(0x7f35a8075410 msgr2=0x7f35a8075820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 -- 172.21.15.3:0/2646637048 shutdown_connections 2024-02-09T23:03:07.263 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.258+0000 7f35ad177640 1 -- 172.21.15.3:0/2646637048 wait complete. 2024-02-09T23:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.259+0000 7f35ad177640 1 Processor -- start 2024-02-09T23:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.259+0000 7f35ad177640 1 -- start start 2024-02-09T23:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35ad177640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 0x7f35a819d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35ad177640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 0x7f35a819db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.265 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35ad177640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a81976e0 0x7f35a8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35ad177640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a81981b0 con 0x7f35a8106600 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35ad177640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a8198320 con 0x7f35a81976e0 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35ad177640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a81984e0 con 0x7f35a80ffd80 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 0x7f35a819db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 0x7f35a819db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54304/0 (socket says 172.21.15.3:54304) 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a759e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a81976e0 0x7f35a8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 -- 172.21.15.3:0/2310033305 learned_addr learned my addr 172.21.15.3:0/2310033305 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 0x7f35a819d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 -- 172.21.15.3:0/2310033305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 msgr2=0x7f35a819d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 0x7f35a819d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 -- 172.21.15.3:0/2310033305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a81976e0 msgr2=0x7f35a8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a81976e0 0x7f35a8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.266 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 -- 172.21.15.3:0/2310033305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359c009560 con 0x7f35a8106600 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a759e640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a81976e0 0x7f35a8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 0x7f35a819d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.260+0000 7f35a659c640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 0x7f35a819db60 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f359c02c170 tx=0x7f359c02c5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.261+0000 7f358ffff640 1 -- 172.21.15.3:0/2310033305 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c0042d0 con 0x7f35a8106600 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.261+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a81a4420 con 0x7f35a8106600 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.261+0000 7f358ffff640 1 -- 172.21.15.3:0/2310033305 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f359c004430 con 0x7f35a8106600 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.261+0000 7f358ffff640 1 -- 172.21.15.3:0/2310033305 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c003bf0 con 0x7f35a8106600 2024-02-09T23:03:07.267 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.261+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a81a4950 con 0x7f35a8106600 2024-02-09T23:03:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.263+0000 7f358ffff640 1 -- 172.21.15.3:0/2310033305 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f359c003d50 con 0x7f35a8106600 2024-02-09T23:03:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.263+0000 7f35afe27740 1 Processor -- start 2024-02-09T23:03:07.273 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 -- start start 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f358ffff640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35900a22a0 0x7f35900a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14b8490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14c1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14c3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce1426a00 con 0x55cce14c17a0 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce1423f30 con 0x55cce14b89d0 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35afe27740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce141f710 con 0x55cce14b80b0 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a759e640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35900a22a0 0x7f35900a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f358ffff640 1 -- 172.21.15.3:0/2310033305 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f359c0f0590 con 0x7f35a8106600 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14b8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14b8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:49634/0 (socket says 172.21.15.3:49634) 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 -- 172.21.15.3:0/290553491 learned_addr learned my addr 172.21.15.3:0/290553491 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:07.274 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a659c640 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14c1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a759e640 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14c3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 -- 172.21.15.3:0/290553491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 msgr2=0x55cce14c1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14c1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 -- 172.21.15.3:0/290553491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 msgr2=0x55cce14c3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14c3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 -- 172.21.15.3:0/290553491 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cce13e43b0 con 0x55cce14b80b0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a759e640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35900a22a0 0x7f35900a4760 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f35a00137e0 tx=0x7f35a0013be0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.264+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14b8490 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3598011860 tx=0x7f3598011d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62e45bb77116ff server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f358d7fa640 1 -- 172.21.15.3:0/290553491 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35980112f0 con 0x55cce14b80b0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f358d7fa640 1 -- 172.21.15.3:0/290553491 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3598003e50 con 0x55cce14b80b0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f358d7fa640 1 -- 172.21.15.3:0/290553491 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598004480 con 0x55cce14b80b0 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 -- 172.21.15.3:0/290553491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 msgr2=0x55cce14b8490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14b8490 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3598011860 tx=0x7f3598011d20 comp rx=0 tx=0).stop 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 -- 172.21.15.3:0/290553491 shutdown_connections 2024-02-09T23:03:07.275 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14c1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14c3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 --2- 172.21.15.3:0/290553491 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14b8490 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.265+0000 7f35afe27740 1 -- 172.21.15.3:0/290553491 >> 172.21.15.3:0/290553491 conn(0x55cce14b29d0 msgr2=0x55cce14b2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 -- 172.21.15.3:0/290553491 shutdown_connections 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 -- 172.21.15.3:0/290553491 wait complete. 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 -- start start 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 Processor -- start 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b80b0 0x55cce14be7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14bf260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce13c4b90 con 0x55cce14b80b0 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce10f3250 con 0x55cce14b89d0 2024-02-09T23:03:07.276 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce13a5200 con 0x55cce14c17a0 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35afe27740 10 client.0 _ll_register_callbacks cb 0x55cce13db5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35a6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b80b0 0x55cce14be7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35a759e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14bf260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35a659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35a759e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14bf260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:49640/0 (socket says 172.21.15.3:49640) 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35a659c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:44718/0 (socket says 172.21.15.3:44718) 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.266+0000 7f35a759e640 1 -- 172.21.15.3:0/3539728991 learned_addr learned my addr 172.21.15.3:0/3539728991 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a6d9d640 1 -- 172.21.15.3:0/3539728991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 msgr2=0x55cce14bf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14bf260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a6d9d640 1 -- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 msgr2=0x55cce14bed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a6d9d640 1 -- 172.21.15.3:0/3539728991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35980114c0 con 0x55cce14b80b0 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a659c640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:07.277 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a759e640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14bf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b80b0 0x55cce14be7e0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f359800a010 tx=0x7f3598011f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df6671e5fff1cc3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f358aff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f358aff5640 1 -- 172.21.15.3:0/3539728991 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359800fc50 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f358aff5640 1 -- 172.21.15.3:0/3539728991 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f3598004200 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55cce1445ee0 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f358aff5640 1 -- 172.21.15.3:0/3539728991 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35980043e0 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35afe27740 10 client.19869 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55cce141dd90 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.267+0000 7f358aff5640 1 -- 172.21.15.3:0/3539728991 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3598032020 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f358aff5640 1 -- 172.21.15.3:0/3539728991 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f359801ac40 con 0x55cce14b80b0 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35afe27740 10 client.19869 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35afe27740 10 client.19869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35afe27740 20 client.19869 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:07.278 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35afe27740 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce14d1920 0x55cce14d3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35afe27740 4 client.19869 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cce111f600 con 0x55cce14d1920 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35a659c640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce14d1920 0x55cce14d3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f358aff5640 10 client.19869 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.268+0000 7f35a659c640 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce14d1920 0x55cce14d3d40 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c2aef939d0361ec server_cookie=89a303af3403f64f in_seq=0 out_seq=0 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f358aff5640 1 -- 172.21.15.3:0/3539728991 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cce111f600 con 0x55cce14d1920 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f358aff5640 10 client.19869 handle_command_reply: tid=0 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 client.19869 shutdown 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b80b0 msgr2=0x55cce14be7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b80b0 0x55cce14be7e0 secure :-1 s=READY pgs=1700 cs=0 l=1 rev1=1 crypto rx=0x7f359800a010 tx=0x7f3598011f70 comp rx=0 tx=0).stop 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 shutdown_connections 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce14d1920 0x55cce14d3d40 crc :-1 s=READY pgs=329 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.279 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14bf260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f358aff5640 0 client.19869 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 --2- 172.21.15.3:0/3539728991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b80b0 0x55cce14be7e0 unknown :-1 s=CLOSED pgs=1700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 >> 172.21.15.3:0/3539728991 conn(0x55cce14b29d0 msgr2=0x55cce14b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 shutdown_connections 2024-02-09T23:03:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 1 -- 172.21.15.3:0/3539728991 wait complete. 2024-02-09T23:03:07.280 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.269+0000 7f35afe27740 20 client.19869 trim_cache size 0 max 16384 2024-02-09T23:03:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 Processor -- start 2024-02-09T23:03:07.290 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 -- start start 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14c0f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14bba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce14ae310 con 0x55cce14c17a0 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce14ce4d0 con 0x55cce14b89d0 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.285+0000 7f35afe27740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce14acec0 con 0x55cce14b80b0 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14bba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14bba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54338/0 (socket says 172.21.15.3:54338) 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 -- 172.21.15.3:0/3565662541 learned_addr learned my addr 172.21.15.3:0/3565662541 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14c0f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a659c640 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 -- 172.21.15.3:0/3565662541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 msgr2=0x55cce14c0f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14c0f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.291 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 -- 172.21.15.3:0/3565662541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 msgr2=0x55cce14bf080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bf080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 -- 172.21.15.3:0/3565662541 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cce13ba820 con 0x55cce14c17a0 2024-02-09T23:03:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f35a759e640 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14bba60 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f35a00123f0 tx=0x7f35a001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b76e53347a7ab405 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f358cff9640 1 -- 172.21.15.3:0/3565662541 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0018070 con 0x55cce14c17a0 2024-02-09T23:03:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f358cff9640 1 -- 172.21.15.3:0/3565662541 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35a0010430 con 0x55cce14c17a0 2024-02-09T23:03:07.292 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.286+0000 7f358cff9640 1 -- 172.21.15.3:0/3565662541 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35a0020640 con 0x55cce14c17a0 2024-02-09T23:03:07.295 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 -- 172.21.15.3:0/3565662541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 msgr2=0x55cce14bba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14bba60 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7f35a00123f0 tx=0x7f35a001f000 comp rx=0 tx=0).stop 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 -- 172.21.15.3:0/3565662541 shutdown_connections 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b89d0 0x55cce14bf080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14c17a0 0x55cce14bba60 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 --2- 172.21.15.3:0/3565662541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14b80b0 0x55cce14c0f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 -- 172.21.15.3:0/3565662541 >> 172.21.15.3:0/3565662541 conn(0x55cce14b29d0 msgr2=0x55cce14bd0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 -- 172.21.15.3:0/3565662541 shutdown_connections 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 -- 172.21.15.3:0/3565662541 wait complete. 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.287+0000 7f35afe27740 1 -- start start 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 Processor -- start 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b80b0 0x55cce14c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.296 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 0x55cce14c7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14c7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce14c8d80 con 0x55cce14b89d0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce1527e70 con 0x55cce14b80b0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cce1528000 con 0x55cce14c17a0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35afe27740 10 client.0 _ll_register_callbacks cb 0x7f359c0045a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35a659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 0x55cce14c7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35a659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 0x55cce14c7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54340/0 (socket says 172.21.15.3:54340) 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35a659c640 1 -- 172.21.15.3:0/2284497423 learned_addr learned my addr 172.21.15.3:0/2284497423 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35a759e640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.288+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b80b0 0x55cce14c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a659c640 1 -- 172.21.15.3:0/2284497423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 msgr2=0x55cce14c7860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a659c640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14c7860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a659c640 1 -- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b80b0 msgr2=0x55cce14c6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:07.297 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a659c640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b80b0 0x55cce14c6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a659c640 1 -- 172.21.15.3:0/2284497423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a0012180 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a759e640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35a659c640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 0x55cce14c7320 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f359c004650 tx=0x7f359c0bc970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3732bfa4f00ee5 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f358aff5640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f358aff5640 1 -- 172.21.15.3:0/2284497423 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359c0bd010 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f358aff5640 1 -- 172.21.15.3:0/2284497423 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f359c0c1160 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3598003e50 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f358aff5640 1 -- 172.21.15.3:0/2284497423 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f359c0c57b0 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35afe27740 10 client.19875 fetch_fsmap learned FSMap version 49 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) v3 -- 0x55cce13e06f0 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f358aff5640 1 -- 172.21.15.3:0/2284497423 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3598003e50 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.289+0000 7f358aff5640 1 -- 172.21.15.3:0/2284497423 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 49) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f359c03c070 con 0x55cce14b89d0 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35afe27740 10 client.19875 fetch_fsmap finished waiting for FSMap version 49 2024-02-09T23:03:07.298 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35afe27740 10 client.19875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35afe27740 20 client.19875 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35afe27740 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce1529510 0x55cce152b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35afe27740 4 client.19875 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55cce11701b0 con 0x55cce1529510 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce1529510 0x55cce152b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f35a6d9d640 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce1529510 0x55cce152b930 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5d5f52a4c514b5 server_cookie=6e4353fd5372b920 in_seq=0 out_seq=0 2024-02-09T23:03:07.299 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.290+0000 7f358aff5640 10 client.19875 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f358aff5640 1 -- 172.21.15.3:0/2284497423 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55cce11701b0 con 0x55cce1529510 2024-02-09T23:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f358aff5640 10 client.19875 handle_command_reply: tid=0 2024-02-09T23:03:07.327 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 client.19875 shutdown 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 msgr2=0x55cce14c7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 0x55cce14c7320 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f359c004650 tx=0x7f359c0bc970 comp rx=0 tx=0).stop 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 shutdown_connections 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55cce1529510 0x55cce152b930 crc :-1 s=READY pgs=330 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55cce14c17a0 0x55cce14c7860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55cce14b89d0 0x55cce14c7320 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f358aff5640 0 client.19875 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 --2- 172.21.15.3:0/2284497423 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55cce14b80b0 0x55cce14c6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 >> 172.21.15.3:0/2284497423 conn(0x55cce14b29d0 msgr2=0x55cce14b11a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 shutdown_connections 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 1 -- 172.21.15.3:0/2284497423 wait complete. 2024-02-09T23:03:07.328 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.323+0000 7f35afe27740 20 client.19875 trim_cache size 0 max 16384 2024-02-09T23:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:03:07.329 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35900a22a0 msgr2=0x7f35900a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35900a22a0 0x7f35900a4760 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f35a00137e0 tx=0x7f35a0013be0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 msgr2=0x7f35a819db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 0x7f35a819db60 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7f359c02c170 tx=0x7f359c02c5f0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 shutdown_connections 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f35900a22a0 0x7f35900a4760 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f35a81976e0 0x7f35a8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f35a8106600 0x7f35a819db60 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 --2- 172.21.15.3:0/2310033305 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f35a80ffd80 0x7f35a819d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 >> 172.21.15.3:0/2310033305 conn(0x7f35a8075410 msgr2=0x7f35a80fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:07.330 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 shutdown_connections 2024-02-09T23:03:07.331 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:07.324+0000 7f35ad177640 1 -- 172.21.15.3:0/2310033305 wait complete. 2024-02-09T23:03:07.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 432593 3.53 MB/sec execute 541 sec latency 830.257 ms 2024-02-09T23:03:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:07 smithi064 ceph-mon[36638]: pgmap v372: 97 pgs: 97 active+clean; 170 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 241 op/s 2024-02-09T23:03:07.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:07 smithi064 ceph-mon[36638]: daemon mds.e restarted 2024-02-09T23:03:07.685 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:03:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setuser ceph since I am not root 2024-02-09T23:03:07.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:03:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setgroup ceph since I am not root 2024-02-09T23:03:07.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:03:07 smithi064 ceph-mds[66607]: -- 172.21.15.64:0/2226532430 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f3f9861a0 con 0x556f3f95cc00 2024-02-09T23:03:07.686 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:03:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: starting mds.e at 2024-02-09T23:03:07.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:07 smithi003 ceph-mon[32678]: pgmap v372: 97 pgs: 97 active+clean; 170 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 241 op/s 2024-02-09T23:03:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:07 smithi003 ceph-mon[32678]: daemon mds.e restarted 2024-02-09T23:03:07.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:07 smithi089 ceph-mon[36665]: pgmap v372: 97 pgs: 97 active+clean; 170 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 2.2 MiB/s wr, 241 op/s 2024-02-09T23:03:07.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:07 smithi089 ceph-mon[36665]: daemon mds.e restarted 2024-02-09T23:03:08.666 INFO:tasks.workunit.client.0.smithi003.stdout: 1 433815 3.53 MB/sec execute 542 sec latency 22.409 ms 2024-02-09T23:03:08.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:08 smithi003 ceph-mon[32678]: osdmap e103: 12 total, 12 up, 12 in 2024-02-09T23:03:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:08 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6834/1843835937,v1:172.21.15.64:6835/1843835937] up:boot 2024-02-09T23:03:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:08 smithi003 ceph-mon[32678]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T23:03:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:08 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T23:03:08.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:08 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:08 smithi064 ceph-mon[36638]: osdmap e103: 12 total, 12 up, 12 in 2024-02-09T23:03:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:08 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6834/1843835937,v1:172.21.15.64:6835/1843835937] up:boot 2024-02-09T23:03:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:08 smithi064 ceph-mon[36638]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T23:03:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:08 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T23:03:08.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:08 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:08 smithi089 ceph-mon[36665]: osdmap e103: 12 total, 12 up, 12 in 2024-02-09T23:03:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:08 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6834/1843835937,v1:172.21.15.64:6835/1843835937] up:boot 2024-02-09T23:03:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:08 smithi089 ceph-mon[36665]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-02-09T23:03:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:08 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T23:03:08.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:08 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:09.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 434458 3.53 MB/sec execute 543 sec latency 634.968 ms 2024-02-09T23:03:10.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:10 smithi003 ceph-mon[32678]: pgmap v374: 97 pgs: 97 active+clean; 170 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 206 op/s 2024-02-09T23:03:10.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:10 smithi003 ceph-mon[32678]: osdmap e104: 12 total, 12 up, 12 in 2024-02-09T23:03:10.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 435777 3.54 MB/sec execute 544 sec latency 1073.260 ms 2024-02-09T23:03:10.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:10 smithi064 ceph-mon[36638]: pgmap v374: 97 pgs: 97 active+clean; 170 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 206 op/s 2024-02-09T23:03:10.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:10 smithi064 ceph-mon[36638]: osdmap e104: 12 total, 12 up, 12 in 2024-02-09T23:03:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:10 smithi089 ceph-mon[36665]: pgmap v374: 97 pgs: 97 active+clean; 170 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 928 KiB/s rd, 1.8 MiB/s wr, 206 op/s 2024-02-09T23:03:10.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:10 smithi089 ceph-mon[36665]: osdmap e104: 12 total, 12 up, 12 in 2024-02-09T23:03:11.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:11 smithi003 ceph-mon[32678]: pgmap v376: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-02-09T23:03:11.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 437216 3.54 MB/sec execute 545 sec latency 16.682 ms 2024-02-09T23:03:11.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:11 smithi064 ceph-mon[36638]: pgmap v376: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-02-09T23:03:11.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:11 smithi089 ceph-mon[36665]: pgmap v376: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-02-09T23:03:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:12.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:12 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:12.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 438005 3.55 MB/sec execute 546 sec latency 402.133 ms 2024-02-09T23:03:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:12.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:12 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:12.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:12.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:12 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:13.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:13 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:13.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:13 smithi003 ceph-mon[32678]: pgmap v377: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-02-09T23:03:13.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 440007 3.56 MB/sec execute 547 sec latency 523.335 ms 2024-02-09T23:03:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:13 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:13.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:13 smithi064 ceph-mon[36638]: pgmap v377: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-02-09T23:03:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:13 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:13 smithi089 ceph-mon[36665]: pgmap v377: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-02-09T23:03:14.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 441327 3.57 MB/sec execute 548 sec latency 13.640 ms 2024-02-09T23:03:15.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:15 smithi003 ceph-mon[32678]: pgmap v378: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-02-09T23:03:15.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 441547 3.56 MB/sec execute 549 sec latency 868.330 ms 2024-02-09T23:03:15.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:15 smithi064 ceph-mon[36638]: pgmap v378: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-02-09T23:03:15.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:15 smithi089 ceph-mon[36665]: pgmap v378: 97 pgs: 97 active+clean; 166 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 1.0 MiB/s wr, 117 op/s 2024-02-09T23:03:16.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 441547 3.56 MB/sec execute 550 sec latency 1868.417 ms 2024-02-09T23:03:17.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 443688 3.57 MB/sec execute 551 sec latency 1903.301 ms 2024-02-09T23:03:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:17 smithi003 ceph-mon[32678]: pgmap v379: 97 pgs: 97 active+clean; 173 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 231 op/s 2024-02-09T23:03:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:17 smithi064 ceph-mon[36638]: pgmap v379: 97 pgs: 97 active+clean; 173 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 231 op/s 2024-02-09T23:03:17.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:17 smithi089 ceph-mon[36665]: pgmap v379: 97 pgs: 97 active+clean; 173 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 231 op/s 2024-02-09T23:03:18.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 445053 3.58 MB/sec execute 552 sec latency 19.449 ms 2024-02-09T23:03:19.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 445096 3.57 MB/sec execute 553 sec latency 965.177 ms 2024-02-09T23:03:19.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:19 smithi003 ceph-mon[32678]: pgmap v380: 97 pgs: 97 active+clean; 173 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:03:19.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:19 smithi064 ceph-mon[36638]: pgmap v380: 97 pgs: 97 active+clean; 173 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:03:19.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:19 smithi089 ceph-mon[36665]: pgmap v380: 97 pgs: 97 active+clean; 173 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.9 MiB/s wr, 207 op/s 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 -- 172.21.15.3:0/3573692568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a80ffad0 msgr2=0x7fb4a80fff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 --2- 172.21.15.3:0/3573692568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a80ffad0 0x7fb4a80fff50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb498009890 tx=0x7fb49802bb70 comp rx=0 tx=0).stop 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 -- 172.21.15.3:0/3573692568 shutdown_connections 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 --2- 172.21.15.3:0/3573692568 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a80ffad0 0x7fb4a80fff50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 --2- 172.21.15.3:0/3573692568 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8100490 0x7fb4a810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 --2- 172.21.15.3:0/3573692568 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a8106270 0x7fb4a8106650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 -- 172.21.15.3:0/3573692568 >> 172.21.15.3:0/3573692568 conn(0x7fb4a8075450 msgr2=0x7fb4a8075860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 -- 172.21.15.3:0/3573692568 shutdown_connections 2024-02-09T23:03:20.427 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 -- 172.21.15.3:0/3573692568 wait complete. 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 Processor -- start 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.422+0000 7fb4af655640 1 -- start start 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4af655640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 0x7fb4a819d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4af655640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a8100490 0x7fb4a819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4af655640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8106270 0x7fb4a8197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4af655640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8197e90 con 0x7fb4a8106270 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4af655640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a8198000 con 0x7fb4a8100490 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4af655640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a81981c0 con 0x7fb4a80ffad0 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 0x7fb4a819d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 0x7fb4a819d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:51020/0 (socket says 172.21.15.3:51020) 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/2582073838 learned_addr learned my addr 172.21.15.3:0/2582073838 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8106270 0x7fb4a8197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.428 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a8100490 0x7fb4a819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/2582073838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a8100490 msgr2=0x7fb4a819db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a8100490 0x7fb4a819db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8106270 msgr2=0x7fb4a8197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8106270 0x7fb4a8197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/2582073838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0009900 con 0x7fb4a80ffad0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8106270 0x7fb4a8197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 0x7fb4a819d5f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400e9f0 tx=0x7fb4a400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb49e7fc640 1 -- 172.21.15.3:0/2582073838 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4009700 con 0x7fb4a80ffad0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.423+0000 7fb49e7fc640 1 -- 172.21.15.3:0/2582073838 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4a400cd80 con 0x7fb4a80ffad0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.424+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb498009490 con 0x7fb4a80ffad0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.424+0000 7fb49e7fc640 1 -- 172.21.15.3:0/2582073838 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4018920 con 0x7fb4a80ffad0 2024-02-09T23:03:20.429 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.424+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81a4570 con 0x7fb4a80ffad0 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.426+0000 7fb49e7fc640 1 -- 172.21.15.3:0/2582073838 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fb4a4018a80 con 0x7fb4a80ffad0 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.426+0000 7fb4b222b740 1 Processor -- start 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.426+0000 7fb4b222b740 1 -- start start 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.426+0000 7fb49e7fc640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4940a2220 0x7fb4940a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1dfe490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 0x564ad1e07260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4940a2220 0x7fb4940a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1e077a0 0x564ad1e09b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4b222b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1d6ca00 con 0x564ad1dfe0b0 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb49e7fc640 1 -- 172.21.15.3:0/2582073838 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb4a4014070 con 0x7fb4a80ffad0 2024-02-09T23:03:20.436 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4b222b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1d69f30 con 0x564ad1dfe9d0 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4b222b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1d65710 con 0x564ad1e077a0 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1dfe490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4acbf0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 0x564ad1e07260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1dfe490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56882/0 (socket says 172.21.15.3:56882) 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/1952805150 learned_addr learned my addr 172.21.15.3:0/1952805150 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 0x564ad1e07260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:40592/0 (socket says 172.21.15.3:40592) 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1e077a0 0x564ad1e09b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/1952805150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1e077a0 msgr2=0x564ad1e09b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1e077a0 0x564ad1e09b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/1952805150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 msgr2=0x564ad1e07260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 0x564ad1e07260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.437 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/1952805150 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ad1d2a3b0 con 0x564ad1dfe0b0 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 0x564ad1e07260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4940a2220 0x7fb4940a46e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009fd0 tx=0x7fb4a000c740 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.427+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1dfe490 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4097880 tx=0x7fb4a4097280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=889dadb7ad7b7538 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.428+0000 7fb4937fe640 1 -- 172.21.15.3:0/1952805150 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a409aeb0 con 0x564ad1dfe0b0 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.428+0000 7fb4937fe640 1 -- 172.21.15.3:0/1952805150 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4a409f000 con 0x564ad1dfe0b0 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.428+0000 7fb4937fe640 1 -- 172.21.15.3:0/1952805150 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a4055940 con 0x564ad1dfe0b0 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.428+0000 7fb4b222b740 1 -- 172.21.15.3:0/1952805150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 msgr2=0x564ad1dfe490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.428+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1dfe490 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4097880 tx=0x7fb4a4097280 comp rx=0 tx=0).stop 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- 172.21.15.3:0/1952805150 shutdown_connections 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe9d0 0x564ad1e07260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1e077a0 0x564ad1e09b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1952805150 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1dfe490 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- 172.21.15.3:0/1952805150 >> 172.21.15.3:0/1952805150 conn(0x564ad1df89d0 msgr2=0x564ad1df8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:20.438 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- 172.21.15.3:0/1952805150 shutdown_connections 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- 172.21.15.3:0/1952805150 wait complete. 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- start start 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 Processor -- start 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe0b0 0x564ad1e047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 0x564ad1e04d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e05240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1d0ab90 con 0x564ad1dfe9d0 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1a39250 con 0x564ad1e077a0 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1ceb200 con 0x564ad1dfe0b0 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4b222b740 10 client.0 _ll_register_callbacks cb 0x564ad1d215d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.429+0000 7fb4acbf0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 0x564ad1e04d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe0b0 0x564ad1e047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 0x564ad1e04d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56896/0 (socket says 172.21.15.3:56896) 2024-02-09T23:03:20.439 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 -- 172.21.15.3:0/234756719 learned_addr learned my addr 172.21.15.3:0/234756719 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e05240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 -- 172.21.15.3:0/234756719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe0b0 msgr2=0x564ad1e047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe0b0 0x564ad1e047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 -- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 msgr2=0x564ad1e05240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e05240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 -- 172.21.15.3:0/234756719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a81a4570 con 0x564ad1dfe9d0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe0b0 0x564ad1e047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 0x564ad1e04d00 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb498005b60 tx=0x7fb498004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb77585722b7d1d0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb490ff9640 1 -- 172.21.15.3:0/234756719 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498004520 con 0x564ad1dfe9d0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb490ff9640 1 -- 172.21.15.3:0/234756719 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb498034850 con 0x564ad1dfe9d0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ad1d8bee0 con 0x564ad1dfe9d0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb490ff9640 1 -- 172.21.15.3:0/234756719 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb498034a10 con 0x564ad1dfe9d0 2024-02-09T23:03:20.440 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4b222b740 10 client.19890 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.430+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x564ad1d63d90 con 0x564ad1dfe9d0 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb490ff9640 1 -- 172.21.15.3:0/234756719 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564ad1d8bee0 con 0x564ad1dfe9d0 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb490ff9640 1 -- 172.21.15.3:0/234756719 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb49802ca70 con 0x564ad1dfe9d0 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4b222b740 10 client.19890 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4b222b740 10 client.19890 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4b222b740 20 client.19890 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4b222b740 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e17820 0x564ad1e19c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4b222b740 4 client.19890 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564ad1a65600 con 0x564ad1e17820 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e17820 0x564ad1e19c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb490ff9640 10 client.19890 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.431+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e17820 0x564ad1e19c40 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656cc6fbfb462920 server_cookie=52d2ec255fd0cee5 in_seq=0 out_seq=0 2024-02-09T23:03:20.441 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb490ff9640 1 -- 172.21.15.3:0/234756719 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564ad1a65600 con 0x564ad1e17820 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb490ff9640 10 client.19890 handle_command_reply: tid=0 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 client.19890 shutdown 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 msgr2=0x564ad1e04d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 0x564ad1e04d00 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7fb498005b60 tx=0x7fb498004270 comp rx=0 tx=0).stop 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 shutdown_connections 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e17820 0x564ad1e19c40 crc :-1 s=READY pgs=331 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e05240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe9d0 0x564ad1e04d00 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 --2- 172.21.15.3:0/234756719 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe0b0 0x564ad1e047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 >> 172.21.15.3:0/234756719 conn(0x564ad1df89d0 msgr2=0x564ad1dfa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb490ff9640 0 client.19890 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 shutdown_connections 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 1 -- 172.21.15.3:0/234756719 wait complete. 2024-02-09T23:03:20.442 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.432+0000 7fb4b222b740 20 client.19890 trim_cache size 0 max 16384 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.446+0000 7fb4b222b740 1 Processor -- start 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.446+0000 7fb4b222b740 1 -- start start 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.446+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1e06ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.446+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e05060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.446+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e030b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4b222b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1e14430 con 0x564ad1dfe0b0 2024-02-09T23:03:20.454 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4b222b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1df2ec0 con 0x564ad1e077a0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4b222b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1e00410 con 0x564ad1dfe9d0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1e06ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e05060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1e06ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:56898/0 (socket says 172.21.15.3:56898) 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/1688423018 learned_addr learned my addr 172.21.15.3:0/1688423018 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4adbf2640 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e030b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 -- 172.21.15.3:0/1688423018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 msgr2=0x564ad1e030b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e030b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 -- 172.21.15.3:0/1688423018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 msgr2=0x564ad1e06ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1e06ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 -- 172.21.15.3:0/1688423018 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564ad1d00820 con 0x564ad1dfe9d0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1e06ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e05060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb49802c180 tx=0x7fb49802bb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=501cf18977f88705 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:20.455 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb492ffd640 1 -- 172.21.15.3:0/1688423018 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498056070 con 0x564ad1dfe9d0 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb492ffd640 1 -- 172.21.15.3:0/1688423018 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4980042e0 con 0x564ad1dfe9d0 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.447+0000 7fb492ffd640 1 -- 172.21.15.3:0/1688423018 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb498032860 con 0x564ad1dfe9d0 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 -- 172.21.15.3:0/1688423018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 msgr2=0x564ad1e05060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e05060 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fb49802c180 tx=0x7fb49802bb70 comp rx=0 tx=0).stop 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 -- 172.21.15.3:0/1688423018 shutdown_connections 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1e077a0 0x564ad1e030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e05060 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 --2- 172.21.15.3:0/1688423018 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1dfe0b0 0x564ad1e06ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 -- 172.21.15.3:0/1688423018 >> 172.21.15.3:0/1688423018 conn(0x564ad1df89d0 msgr2=0x564ad1e1e5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 -- 172.21.15.3:0/1688423018 shutdown_connections 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.448+0000 7fb4b222b740 1 -- 172.21.15.3:0/1688423018 wait complete. 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 -- start start 2024-02-09T23:03:20.456 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 Processor -- start 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 0x564ad1e0c410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e0c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1e077a0 0x564ad1e6cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1e0d250 con 0x564ad1e077a0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1e6d280 con 0x564ad1dfe0b0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x564ad1e6d3c0 con 0x564ad1dfe9d0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4b222b740 10 client.0 _ll_register_callbacks cb 0x7fb4a0009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 0x564ad1e0c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4adbf2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1e077a0 0x564ad1e6cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4ad3f1640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 0x564ad1e0c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:40610/0 (socket says 172.21.15.3:40610) 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/3735276741 learned_addr learned my addr 172.21.15.3:0/3735276741 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.449+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e0c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/3735276741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 msgr2=0x564ad1e0c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.457 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e0c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/3735276741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1e077a0 msgr2=0x564ad1e6cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1e077a0 0x564ad1e6cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4ad3f1640 1 -- 172.21.15.3:0/3735276741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564ad1d2a3b0 con 0x564ad1dfe0b0 2024-02-09T23:03:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e0c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:20.458 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4ad3f1640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 0x564ad1e0c410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4097830 tx=0x7fb4a40556c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8601cee12965450a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.64:3300/0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb490ff9640 1 -- 172.21.15.3:0/3735276741 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a40a3650 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb490ff9640 1 -- 172.21.15.3:0/3735276741 <== mon.2 v2:172.21.15.64:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fb4a40991b0 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564ad1ab61b0 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb490ff9640 1 -- 172.21.15.3:0/3735276741 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4a4010b80 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4b222b740 10 client.28724 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x564ad1d63d90 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.450+0000 7fb490ff9640 1 -- 172.21.15.3:0/3735276741 <== mon.2 v2:172.21.15.64:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4a40b2020 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb490ff9640 1 -- 172.21.15.3:0/3735276741 <== mon.2 v2:172.21.15.64:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb4a409a300 con 0x564ad1dfe0b0 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4b222b740 10 client.28724 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4b222b740 10 client.28724 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4b222b740 20 client.28724 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4b222b740 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e6e700 0x564ad1e70b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:20.459 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4b222b740 4 client.28724 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:03:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564ad1aa2550 con 0x564ad1e6e700 2024-02-09T23:03:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e6e700 0x564ad1e70b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb490ff9640 10 client.28724 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:20.460 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.451+0000 7fb4acbf0640 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e6e700 0x564ad1e70b20 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=791d1f18e9568cb4 server_cookie=ca4641f86c149b4e in_seq=0 out_seq=0 2024-02-09T23:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb490ff9640 1 -- 172.21.15.3:0/3735276741 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564ad1aa2550 con 0x564ad1e6e700 2024-02-09T23:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb490ff9640 10 client.28724 handle_command_reply: tid=0 2024-02-09T23:03:20.493 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 client.28724 shutdown 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 msgr2=0x564ad1e0c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 0x564ad1e0c410 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb4a4097830 tx=0x7fb4a40556c0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 shutdown_connections 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x564ad1e6e700 0x564ad1e70b20 crc :-1 s=READY pgs=332 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x564ad1e077a0 0x564ad1e6cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x564ad1dfe9d0 0x564ad1e0c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 --2- 172.21.15.3:0/3735276741 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x564ad1dfe0b0 0x564ad1e0c410 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb490ff9640 0 client.28724 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 >> 172.21.15.3:0/3735276741 conn(0x564ad1df89d0 msgr2=0x564ad1e03050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 shutdown_connections 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 1 -- 172.21.15.3:0/3735276741 wait complete. 2024-02-09T23:03:20.494 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.490+0000 7fb4b222b740 20 client.28724 trim_cache size 0 max 16384 2024-02-09T23:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:03:20.495 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4940a2220 msgr2=0x7fb4940a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4940a2220 0x7fb4940a46e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009fd0 tx=0x7fb4a000c740 comp rx=0 tx=0).stop 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 msgr2=0x7fb4a819d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 0x7fb4a819d5f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fb4a400e9f0 tx=0x7fb4a400eeb0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 shutdown_connections 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fb4940a2220 0x7fb4940a46e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fb4a8100490 0x7fb4a819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fb4a8106270 0x7fb4a8197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 --2- 172.21.15.3:0/2582073838 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fb4a80ffad0 0x7fb4a819d5f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 >> 172.21.15.3:0/2582073838 conn(0x7fb4a8075450 msgr2=0x7fb4a80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:20.496 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 shutdown_connections 2024-02-09T23:03:20.497 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:20.491+0000 7fb4af655640 1 -- 172.21.15.3:0/2582073838 wait complete. 2024-02-09T23:03:20.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 446383 3.58 MB/sec execute 554 sec latency 1406.834 ms 2024-02-09T23:03:21.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 447769 3.58 MB/sec execute 555 sec latency 17.243 ms 2024-02-09T23:03:21.778 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:21 smithi089 ceph-mon[36665]: pgmap v381: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-02-09T23:03:21.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:21 smithi003 ceph-mon[32678]: pgmap v381: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-02-09T23:03:21.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:21 smithi064 ceph-mon[36638]: pgmap v381: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 2.6 MiB/s wr, 255 op/s 2024-02-09T23:03:22.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 448638 3.59 MB/sec execute 556 sec latency 410.290 ms 2024-02-09T23:03:23.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 450285 3.60 MB/sec execute 557 sec latency 528.791 ms 2024-02-09T23:03:23.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:23 smithi003 ceph-mon[32678]: pgmap v382: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-02-09T23:03:23.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:23 smithi064 ceph-mon[36638]: pgmap v382: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-02-09T23:03:23.976 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:23 smithi089 ceph-mon[36665]: pgmap v382: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-02-09T23:03:24.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 451295 3.60 MB/sec execute 558 sec latency 165.465 ms 2024-02-09T23:03:25.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:25 smithi003 ceph-mon[32678]: pgmap v383: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-02-09T23:03:25.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 452187 3.60 MB/sec execute 559 sec latency 534.901 ms 2024-02-09T23:03:25.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:25 smithi064 ceph-mon[36638]: pgmap v383: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-02-09T23:03:25.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:25 smithi089 ceph-mon[36665]: pgmap v383: 97 pgs: 97 active+clean; 179 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.7 MiB/s wr, 158 op/s 2024-02-09T23:03:26.667 INFO:tasks.workunit.client.0.smithi003.stdout: 1 452187 3.59 MB/sec execute 560 sec latency 1534.947 ms 2024-02-09T23:03:27.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 454328 3.61 MB/sec execute 561 sec latency 1570.071 ms 2024-02-09T23:03:27.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:27 smithi003 ceph-mon[32678]: pgmap v384: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 271 op/s 2024-02-09T23:03:27.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:27 smithi064 ceph-mon[36638]: pgmap v384: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 271 op/s 2024-02-09T23:03:27.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:27 smithi089 ceph-mon[36665]: pgmap v384: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 271 op/s 2024-02-09T23:03:28.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 455678 3.62 MB/sec execute 562 sec latency 20.417 ms 2024-02-09T23:03:29.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 455730 3.61 MB/sec execute 563 sec latency 953.371 ms 2024-02-09T23:03:29.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:29 smithi003 ceph-mon[32678]: pgmap v385: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-02-09T23:03:29.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:29 smithi064 ceph-mon[36638]: pgmap v385: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-02-09T23:03:29.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:29 smithi089 ceph-mon[36665]: pgmap v385: 97 pgs: 97 active+clean; 178 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.9 MiB/s wr, 176 op/s 2024-02-09T23:03:30.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 457017 3.62 MB/sec execute 564 sec latency 1392.526 ms 2024-02-09T23:03:31.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 458322 3.62 MB/sec execute 565 sec latency 50.393 ms 2024-02-09T23:03:31.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:31 smithi003 ceph-mon[32678]: pgmap v386: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 238 op/s 2024-02-09T23:03:31.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:31 smithi064 ceph-mon[36638]: pgmap v386: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 238 op/s 2024-02-09T23:03:31.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:31 smithi089 ceph-mon[36665]: pgmap v386: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 2.6 MiB/s wr, 238 op/s 2024-02-09T23:03:32.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 459313 3.65 MB/sec execute 566 sec latency 73.146 ms 2024-02-09T23:03:32.836 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:32 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:03:32.836 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:32 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.836 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:32 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.836 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:32 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.837 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:32 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:32 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:03:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:32 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:32 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:32 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:32 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:32 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:03:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:32 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:32 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:32 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:32.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:32 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.588+0000 7f94a9fb5640 1 -- 172.21.15.3:0/1884829295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4106530 msgr2=0x7f94a410e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.588+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/1884829295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4106530 0x7f94a410e100 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x7f949c009cd0 tx=0x7f949c02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 -- 172.21.15.3:0/1884829295 shutdown_connections 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/1884829295 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4106530 0x7f94a410e100 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/1884829295 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94a4101e10 0x7f94a4105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/1884829295 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94a4101460 0x7f94a4101840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 -- 172.21.15.3:0/1884829295 >> 172.21.15.3:0/1884829295 conn(0x7f94a40fccc0 msgr2=0x7f94a40ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 -- 172.21.15.3:0/1884829295 shutdown_connections 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 -- 172.21.15.3:0/1884829295 wait complete. 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 Processor -- start 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 -- start start 2024-02-09T23:03:33.594 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94a4101460 0x7f94a419d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.589+0000 7f94a9fb5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94a4106530 0x7f94a419d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a9fb5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 0x7f94a4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a9fb5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a4198030 con 0x7f94a4197640 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a9fb5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a41981a0 con 0x7f94a4101460 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a9fb5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a4198360 con 0x7f94a4106530 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 0x7f94a4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 0x7f94a4197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52578/0 (socket says 172.21.15.3:52578) 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 -- 172.21.15.3:0/852620797 learned_addr learned my addr 172.21.15.3:0/852620797 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94a4106530 0x7f94a419d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a37fe640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94a4101460 0x7f94a419d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.595 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 -- 172.21.15.3:0/852620797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94a4106530 msgr2=0x7f94a419d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94a4106530 0x7f94a419d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 -- 172.21.15.3:0/852620797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94a4101460 msgr2=0x7f94a419d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94a4101460 0x7f94a419d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 -- 172.21.15.3:0/852620797 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f949c009900 con 0x7f94a4197640 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a3fff640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 0x7f94a4197af0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f949c0348f0 tx=0x7f949c034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a0ff9640 1 -- 172.21.15.3:0/852620797 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0328c0 con 0x7f94a4197640 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a0ff9640 1 -- 172.21.15.3:0/852620797 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f949c003e50 con 0x7f94a4197640 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a41a4300 con 0x7f94a4197640 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a0ff9640 1 -- 172.21.15.3:0/852620797 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c040450 con 0x7f94a4197640 2024-02-09T23:03:33.596 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.590+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a41a47b0 con 0x7f94a4197640 2024-02-09T23:03:33.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a0ff9640 1 -- 172.21.15.3:0/852620797 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f949c00a040 con 0x7f94a4197640 2024-02-09T23:03:33.603 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 Processor -- start 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 -- start start 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a0ff9640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f948c0a22a0 0x7f948c0a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b70b0 0x5631745b7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745c0260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 0x5631745c2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563174525a00 con 0x5631745c07a0 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563174522f30 con 0x5631745b70b0 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94acb81740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56317451e710 con 0x5631745b79d0 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f948c0a22a0 0x7f948c0a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.604 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 0x5631745c2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 0x5631745c2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52582/0 (socket says 172.21.15.3:52582) 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a3fff640 1 -- 172.21.15.3:0/609461603 learned_addr learned my addr 172.21.15.3:0/609461603 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a0ff9640 1 -- 172.21.15.3:0/852620797 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f949c0f0df0 con 0x7f94a4197640 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.593+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745c0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a37fe640 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b70b0 0x5631745b7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a3fff640 1 -- 172.21.15.3:0/609461603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 msgr2=0x5631745c0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a3fff640 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745c0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a3fff640 1 -- 172.21.15.3:0/609461603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b70b0 msgr2=0x5631745b7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a3fff640 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b70b0 0x5631745b7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a3fff640 1 -- 172.21.15.3:0/609461603 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631744e33b0 con 0x5631745c07a0 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f948c0a22a0 0x7f948c0a4760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9498005e10 tx=0x7f949800aa90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f94a3fff640 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 0x5631745c2b90 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f949c004af0 tx=0x7f949c051f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69525c0216e552db server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.605 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f9489ffb640 1 -- 172.21.15.3:0/609461603 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0bd250 con 0x5631745c07a0 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f9489ffb640 1 -- 172.21.15.3:0/609461603 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f949c0bdff0 con 0x5631745c07a0 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.594+0000 7f9489ffb640 1 -- 172.21.15.3:0/609461603 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949c0c1a40 con 0x5631745c07a0 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 -- 172.21.15.3:0/609461603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 msgr2=0x5631745c2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 0x5631745c2b90 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f949c004af0 tx=0x7f949c051f90 comp rx=0 tx=0).stop 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 -- 172.21.15.3:0/609461603 shutdown_connections 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745c07a0 0x5631745c2b90 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745c0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 --2- 172.21.15.3:0/609461603 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b70b0 0x5631745b7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 -- 172.21.15.3:0/609461603 >> 172.21.15.3:0/609461603 conn(0x5631745b19d0 msgr2=0x5631745b1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 -- 172.21.15.3:0/609461603 shutdown_connections 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 -- 172.21.15.3:0/609461603 wait complete. 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 -- start start 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.595+0000 7f94acb81740 1 Processor -- start 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.606 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745bdd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x5631745be280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631744c3b90 con 0x5631745b70b0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631741f2250 con 0x5631745c07a0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631744a4200 con 0x5631745b79d0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 10 client.0 _ll_register_callbacks cb 0x5631744da5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bd800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52584/0 (socket says 172.21.15.3:52584) 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 -- 172.21.15.3:0/4027103154 learned_addr learned my addr 172.21.15.3:0/4027103154 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745bdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a3fff640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x5631745be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 -- 172.21.15.3:0/4027103154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 msgr2=0x5631745bdd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745bdd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 -- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 msgr2=0x5631745be280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.607 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x5631745be280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 -- 172.21.15.3:0/4027103154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a41a47b0 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745bdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94a37fe640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bd800 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f9494009e70 tx=0x7f94940117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2741d65209b49545 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f9482ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f9482ffd640 1 -- 172.21.15.3:0/4027103154 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494009780 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f9482ffd640 1 -- 172.21.15.3:0/4027103154 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9494012860 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.596+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563174544ee0 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f9482ffd640 1 -- 172.21.15.3:0/4027103154 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9494012a20 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 10 client.19908 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x56317451cd90 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f9482ffd640 1 -- 172.21.15.3:0/4027103154 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563174544ee0 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f9482ffd640 1 -- 172.21.15.3:0/4027103154 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f949400c070 con 0x5631745b70b0 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 10 client.19908 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:33.608 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 10 client.19908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 20 client.19908 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5631745d4cc0 0x5631745d70e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 4 client.19908 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56317421e600 con 0x5631745d4cc0 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.597+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5631745d4cc0 0x5631745d70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f9482ffd640 10 client.19908 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5631745d4cc0 0x5631745d70e0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be3f8ed304781254 server_cookie=d4231d26ca14cc5d in_seq=0 out_seq=0 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f9482ffd640 1 -- 172.21.15.3:0/4027103154 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56317421e600 con 0x5631745d4cc0 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f9482ffd640 10 client.19908 handle_command_reply: tid=0 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 client.19908 shutdown 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 msgr2=0x5631745bd800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bd800 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f9494009e70 tx=0x7f94940117b0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 shutdown_connections 2024-02-09T23:03:33.609 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5631745d4cc0 0x5631745d70e0 crc :-1 s=READY pgs=333 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x5631745be280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b79d0 0x5631745bdd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f9482ffd640 0 client.19908 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 --2- 172.21.15.3:0/4027103154 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bd800 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 >> 172.21.15.3:0/4027103154 conn(0x5631745b19d0 msgr2=0x5631745b3720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.598+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 shutdown_connections 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.599+0000 7f94acb81740 1 -- 172.21.15.3:0/4027103154 wait complete. 2024-02-09T23:03:33.610 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.599+0000 7f94acb81740 20 client.19908 trim_cache size 0 max 16384 2024-02-09T23:03:33.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 Processor -- start 2024-02-09T23:03:33.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 -- start start 2024-02-09T23:03:33.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.624 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b79d0 0x5631745b8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745c07a0 0x5631745b9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631745d0e60 con 0x5631745b70b0 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631745d0fd0 con 0x5631745b79d0 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94acb81740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5631745dc190 con 0x5631745c07a0 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52592/0 (socket says 172.21.15.3:52592) 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 -- 172.21.15.3:0/2465092520 learned_addr learned my addr 172.21.15.3:0/2465092520 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a3fff640 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745c07a0 0x5631745b9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b79d0 0x5631745b8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 -- 172.21.15.3:0/2465092520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745c07a0 msgr2=0x5631745b9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745c07a0 0x5631745b9020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 -- 172.21.15.3:0/2465092520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b79d0 msgr2=0x5631745b8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.625 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b79d0 0x5631745b8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 -- 172.21.15.3:0/2465092520 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5631744b9820 con 0x5631745b70b0 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b79d0 0x5631745b8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.617+0000 7f94a37fe640 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bff30 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9494000c00 tx=0x7f9494005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38c10c32310c1b9d server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94897fa640 1 -- 172.21.15.3:0/2465092520 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949401a5f0 con 0x5631745b70b0 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94897fa640 1 -- 172.21.15.3:0/2465092520 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9494009760 con 0x5631745b70b0 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94897fa640 1 -- 172.21.15.3:0/2465092520 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9494012de0 con 0x5631745b70b0 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94acb81740 1 -- 172.21.15.3:0/2465092520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 msgr2=0x5631745bff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94acb81740 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bff30 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9494000c00 tx=0x7f9494005bd0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94acb81740 1 -- 172.21.15.3:0/2465092520 shutdown_connections 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94acb81740 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745b79d0 0x5631745b8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.626 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.618+0000 7f94acb81740 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745c07a0 0x5631745b9020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 --2- 172.21.15.3:0/2465092520 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b70b0 0x5631745bff30 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- 172.21.15.3:0/2465092520 >> 172.21.15.3:0/2465092520 conn(0x5631745b19d0 msgr2=0x5631745dba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- 172.21.15.3:0/2465092520 shutdown_connections 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- 172.21.15.3:0/2465092520 wait complete. 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- start start 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 Processor -- start 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b70b0 0x5631745c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 0x5631745c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x56317462a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56317462a740 con 0x5631745b79d0 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56317462a880 con 0x5631745c07a0 2024-02-09T23:03:33.627 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56317462a9c0 con 0x5631745b70b0 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.619+0000 7f94acb81740 10 client.0 _ll_register_callbacks cb 0x7f94940096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 0x5631745c9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a37fe640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b70b0 0x5631745c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 0x5631745c9d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52604/0 (socket says 172.21.15.3:52604) 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 -- 172.21.15.3:0/1129424218 learned_addr learned my addr 172.21.15.3:0/1129424218 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a3fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x56317462a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.628 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 -- 172.21.15.3:0/1129424218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b70b0 msgr2=0x5631745c97f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b70b0 0x5631745c97f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 -- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 msgr2=0x56317462a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x56317462a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 -- 172.21.15.3:0/1129424218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56317451cd90 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94a2ffd640 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 0x5631745c9d30 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f94a40fd110 tx=0x7f9498004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87fcc55b2810f456 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f9482ffd640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f9482ffd640 1 -- 172.21.15.3:0/1129424218 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9498025040 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f9482ffd640 1 -- 172.21.15.3:0/1129424218 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9498020b20 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56317426f1b0 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f9482ffd640 1 -- 172.21.15.3:0/1129424218 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9498020ce0 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94acb81740 10 client.19914 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.620+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x5631744df6f0 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f9482ffd640 1 -- 172.21.15.3:0/1129424218 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9498040080 con 0x5631745b79d0 2024-02-09T23:03:33.629 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f9482ffd640 1 -- 172.21.15.3:0/1129424218 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f949803a030 con 0x5631745b79d0 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94acb81740 10 client.19914 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94acb81740 10 client.19914 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94acb81740 20 client.19914 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94acb81740 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56317462c280 0x56317462e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94acb81740 4 client.19914 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56317425b550 con 0x56317462c280 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.621+0000 7f94a37fe640 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56317462c280 0x56317462e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.622+0000 7f94a37fe640 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56317462c280 0x56317462e6a0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f690ac924e24a8e6 server_cookie=e41c08d1c8a17d2d in_seq=0 out_seq=0 2024-02-09T23:03:33.630 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.622+0000 7f9482ffd640 10 client.19914 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 460033 3.65 MB/sec execute 567 sec latency 559.489 ms 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f9482ffd640 1 -- 172.21.15.3:0/1129424218 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56317425b550 con 0x56317462c280 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f9482ffd640 10 client.19914 handle_command_reply: tid=0 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 client.19914 shutdown 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 msgr2=0x5631745c9d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 0x5631745c9d30 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f94a40fd110 tx=0x7f9498004730 comp rx=0 tx=0).stop 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 shutdown_connections 2024-02-09T23:03:33.670 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x56317462c280 0x56317462e6a0 crc :-1 s=READY pgs=334 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f9482ffd640 0 client.19914 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5631745c07a0 0x56317462a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5631745b79d0 0x5631745c9d30 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 --2- 172.21.15.3:0/1129424218 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5631745b70b0 0x5631745c97f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.667+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 >> 172.21.15.3:0/1129424218 conn(0x5631745b19d0 msgr2=0x5631745b8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.668+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 shutdown_connections 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.668+0000 7f94acb81740 1 -- 172.21.15.3:0/1129424218 wait complete. 2024-02-09T23:03:33.671 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.668+0000 7f94acb81740 20 client.19914 trim_cache size 0 max 16384 2024-02-09T23:03:33.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:03:33.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:03:33.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:03:33.672 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.670+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f948c0a22a0 msgr2=0x7f948c0a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.670+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f948c0a22a0 0x7f948c0a4760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f9498005e10 tx=0x7f949800aa90 comp rx=0 tx=0).stop 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.670+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 msgr2=0x7f94a4197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.670+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 0x7f94a4197af0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f949c0348f0 tx=0x7f949c034cc0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.671+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 shutdown_connections 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.671+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f948c0a22a0 0x7f948c0a4760 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.671+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f94a4197640 0x7f94a4197af0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.673 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.671+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f94a4106530 0x7f94a419d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.671+0000 7f94a9fb5640 1 --2- 172.21.15.3:0/852620797 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f94a4101460 0x7f94a419d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.671+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 >> 172.21.15.3:0/852620797 conn(0x7f94a40fccc0 msgr2=0x7f94a40fd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.674 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.672+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 shutdown_connections 2024-02-09T23:03:33.675 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:33.672+0000 7f94a9fb5640 1 -- 172.21.15.3:0/852620797 wait complete. 2024-02-09T23:03:33.808 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:03:33.808 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:03:33.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:33 smithi003 ceph-mon[32678]: pgmap v387: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-02-09T23:03:33.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:33 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:03:33.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:33 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:03:33.833 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:33 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:33.921 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- 172.21.15.3:0/831600724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 msgr2=0x7fa674100720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- 172.21.15.3:0/831600724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674100720 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7fa66c009cd0 tx=0x7fa66c02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- 172.21.15.3:0/831600724 shutdown_connections 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- 172.21.15.3:0/831600724 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa674100cf0 0x7fa674104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- 172.21.15.3:0/831600724 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa674105410 0x7fa67410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- 172.21.15.3:0/831600724 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674100720 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- 172.21.15.3:0/831600724 >> 172.21.15.3:0/831600724 conn(0x7fa6740772e0 msgr2=0x7fa6740776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- 172.21.15.3:0/831600724 shutdown_connections 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- 172.21.15.3:0/831600724 wait complete. 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 Processor -- start 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- start start 2024-02-09T23:03:33.922 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa674105410 0x7fa674102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa674102850 0x7fa6741a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741a6050 con 0x7fa674100340 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741a61c0 con 0x7fa674102850 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.917+0000 7fa67b02c640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6741a6380 con 0x7fa674105410 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674101dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674101dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52622/0 (socket says 172.21.15.3:52622) 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 -- 172.21.15.3:0/622566972 learned_addr learned my addr 172.21.15.3:0/622566972 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa673fff640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa674105410 0x7fa674102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa6795c9640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa674102850 0x7fa6741a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 -- 172.21.15.3:0/622566972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa674105410 msgr2=0x7fa674102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa674105410 0x7fa674102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 -- 172.21.15.3:0/622566972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa674102850 msgr2=0x7fa6741a5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.923 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa674102850 0x7fa6741a5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 -- 172.21.15.3:0/622566972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa66c009900 con 0x7fa674100340 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa678dc8640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674101dd0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fa66c004af0 tx=0x7fa66c0325c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa671ffb640 1 -- 172.21.15.3:0/622566972 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c034e20 con 0x7fa674100340 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa671ffb640 1 -- 172.21.15.3:0/622566972 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa66c003e50 con 0x7fa674100340 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6741a6600 con 0x7fa674100340 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.918+0000 7fa671ffb640 1 -- 172.21.15.3:0/622566972 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c004880 con 0x7fa674100340 2024-02-09T23:03:33.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.919+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6741a6ab0 con 0x7fa674100340 2024-02-09T23:03:33.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa671ffb640 1 -- 172.21.15.3:0/622566972 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa66c00a040 con 0x7fa674100340 2024-02-09T23:03:33.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 Processor -- start 2024-02-09T23:03:33.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 -- start start 2024-02-09T23:03:33.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad08d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad09330 0x55843ad118e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad11eb0 0x55843ad14300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843acc3d00 con 0x55843ad09330 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843ac782f0 con 0x55843ad08980 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843ac75820 con 0x55843ad11eb0 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa671ffb640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa65c0a2260 0x7fa65c0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad09330 0x55843ad118e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad09330 0x55843ad118e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52626/0 (socket says 172.21.15.3:52626) 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa678dc8640 1 -- 172.21.15.3:0/2851888205 learned_addr learned my addr 172.21.15.3:0/2851888205 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa6795c9640 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad08d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.921+0000 7fa673fff640 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad11eb0 0x55843ad14300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.932 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa6795c9640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa65c0a2260 0x7fa65c0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa671ffb640 1 -- 172.21.15.3:0/622566972 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa66c0f0a30 con 0x7fa674100340 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa673fff640 1 -- 172.21.15.3:0/2851888205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 msgr2=0x55843ad08d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa673fff640 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad08d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa673fff640 1 -- 172.21.15.3:0/2851888205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad09330 msgr2=0x55843ad118e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa673fff640 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad09330 0x55843ad118e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa673fff640 1 -- 172.21.15.3:0/2851888205 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55843ac32ff0 con 0x55843ad11eb0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa6795c9640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa65c0a2260 0x7fa65c0a4720 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa664009fd0 tx=0x7fa664005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa673fff640 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad11eb0 0x55843ad14300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa67406a920 tx=0x7fa668011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a25b83a5c2f9a12 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa65affd640 1 -- 172.21.15.3:0/2851888205 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668004310 con 0x55843ad11eb0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa65affd640 1 -- 172.21.15.3:0/2851888205 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa668004470 con 0x55843ad11eb0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.922+0000 7fa65affd640 1 -- 172.21.15.3:0/2851888205 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668003850 con 0x55843ad11eb0 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 -- 172.21.15.3:0/2851888205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad11eb0 msgr2=0x55843ad14300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.933 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad11eb0 0x55843ad14300 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa67406a920 tx=0x7fa668011440 comp rx=0 tx=0).stop 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 -- 172.21.15.3:0/2851888205 shutdown_connections 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad11eb0 0x55843ad14300 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad09330 0x55843ad118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/2851888205 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad08d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 -- 172.21.15.3:0/2851888205 >> 172.21.15.3:0/2851888205 conn(0x55843ad03050 msgr2=0x55843ad03430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 -- 172.21.15.3:0/2851888205 shutdown_connections 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 -- 172.21.15.3:0/2851888205 wait complete. 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 -- start start 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.923+0000 7fa67dbfd740 1 Processor -- start 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 0x55843ad0ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad0f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.934 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad11eb0 0x55843ad0f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843ac71000 con 0x55843ad08980 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843a944430 con 0x55843ad11eb0 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843a900e00 con 0x55843ad09330 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa67dbfd740 10 client.0 _ll_register_callbacks cb 0x55843ac657f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 0x55843ad0ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 0x55843ad0ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52632/0 (socket says 172.21.15.3:52632) 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 -- 172.21.15.3:0/1485159814 learned_addr learned my addr 172.21.15.3:0/1485159814 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa673fff640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad0f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa6795c9640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad11eb0 0x55843ad0f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 -- 172.21.15.3:0/1485159814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 msgr2=0x55843ad0f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad0f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 -- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad11eb0 msgr2=0x55843ad0f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.935 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad11eb0 0x55843ad0f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.924+0000 7fa678dc8640 1 -- 172.21.15.3:0/1485159814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa668009850 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa678dc8640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 0x55843ad0ed80 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0bd4c0 tx=0x7fa66c0bd3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cbd7421a15b969 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa653fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa653fff640 1 -- 172.21.15.3:0/1485159814 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa66c0f0ea0 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa653fff640 1 -- 172.21.15.3:0/1485159814 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa66c0bdc30 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55843a9707e0 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa653fff640 1 -- 172.21.15.3:0/1485159814 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa66c0bd530 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 10 client.19926 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x55843ac6f680 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa653fff640 1 -- 172.21.15.3:0/1485159814 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55843a9707e0 con 0x55843ad08980 2024-02-09T23:03:33.936 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa653fff640 1 -- 172.21.15.3:0/1485159814 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa66c0328b0 con 0x55843ad08980 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 10 client.19926 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 10 client.19926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 20 client.19926 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad22110 0x55843ad24500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 4 client.19926 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.925+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55843a9ad720 con 0x55843ad22110 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.926+0000 7fa673fff640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad22110 0x55843ad24500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.926+0000 7fa653fff640 10 client.19926 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.926+0000 7fa673fff640 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad22110 0x55843ad24500 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7913dfb02b3a4f8 server_cookie=fc14cfdf8468d3c in_seq=0 out_seq=0 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa653fff640 1 -- 172.21.15.3:0/1485159814 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55843a9ad720 con 0x55843ad22110 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa653fff640 10 client.19926 handle_command_reply: tid=0 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 client.19926 shutdown 2024-02-09T23:03:33.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 msgr2=0x55843ad0ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 0x55843ad0ed80 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7fa66c0bd4c0 tx=0x7fa66c0bd3c0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 shutdown_connections 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad22110 0x55843ad24500 crc :-1 s=READY pgs=335 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad11eb0 0x55843ad0f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad0f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa653fff640 0 client.19926 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/1485159814 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad08980 0x55843ad0ed80 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 >> 172.21.15.3:0/1485159814 conn(0x55843ad03050 msgr2=0x55843ad04da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 shutdown_connections 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 1 -- 172.21.15.3:0/1485159814 wait complete. 2024-02-09T23:03:33.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.927+0000 7fa67dbfd740 20 client.19926 trim_cache size 0 max 16384 2024-02-09T23:03:33.939 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:33 smithi064 ceph-mon[36638]: pgmap v387: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-02-09T23:03:33.939 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:33 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:03:33.939 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:33 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:03:33.939 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:33 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 Processor -- start 2024-02-09T23:03:33.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 -- start start 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad08980 0x55843ad114b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad09330 0x55843ad0f620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad0bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843abf5fd0 con 0x55843ad11eb0 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843a9707e0 con 0x55843ad09330 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843acd8ab0 con 0x55843ad08980 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad0bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa678dc8640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad08980 0x55843ad114b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad0bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52646/0 (socket says 172.21.15.3:52646) 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 -- 172.21.15.3:0/981243437 learned_addr learned my addr 172.21.15.3:0/981243437 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa673fff640 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad09330 0x55843ad0f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 -- 172.21.15.3:0/981243437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad08980 msgr2=0x55843ad114b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad08980 0x55843ad114b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 -- 172.21.15.3:0/981243437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad09330 msgr2=0x55843ad0f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad09330 0x55843ad0f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 -- 172.21.15.3:0/981243437 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55843ac0bb30 con 0x55843ad11eb0 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa678dc8640 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad08980 0x55843ad114b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.943+0000 7fa6795c9640 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad0bfe0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fa664011630 tx=0x7fa66400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f48da4ad2c3dc6c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa65a7fc640 1 -- 172.21.15.3:0/981243437 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664026040 con 0x55843ad11eb0 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa65a7fc640 1 -- 172.21.15.3:0/981243437 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa664011030 con 0x55843ad11eb0 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa65a7fc640 1 -- 172.21.15.3:0/981243437 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa664010670 con 0x55843ad11eb0 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 -- 172.21.15.3:0/981243437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 msgr2=0x55843ad0bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad0bfe0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7fa664011630 tx=0x7fa66400e000 comp rx=0 tx=0).stop 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 -- 172.21.15.3:0/981243437 shutdown_connections 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad09330 0x55843ad0f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad0bfe0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/981243437 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad08980 0x55843ad114b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 -- 172.21.15.3:0/981243437 >> 172.21.15.3:0/981243437 conn(0x55843ad03050 msgr2=0x55843ad28e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 -- 172.21.15.3:0/981243437 shutdown_connections 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.944+0000 7fa67dbfd740 1 -- 172.21.15.3:0/981243437 wait complete. 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 -- start start 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 Processor -- start 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad18260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad72710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843ad72c50 con 0x55843ad11eb0 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843ad72dc0 con 0x55843ad08980 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55843ad72f80 con 0x55843ad09330 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa67dbfd740 10 client.0 _ll_register_callbacks cb 0x7fa6680097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:33.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa6795c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad72710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa6795c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad72710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52648/0 (socket says 172.21.15.3:52648) 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa6795c9640 1 -- 172.21.15.3:0/68863276 learned_addr learned my addr 172.21.15.3:0/68863276 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa678dc8640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad18260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.945+0000 7fa673fff640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa673fff640 1 -- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 msgr2=0x55843ad18260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa673fff640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad18260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa673fff640 1 -- 172.21.15.3:0/68863276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 msgr2=0x55843ad72710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa673fff640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad72710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa673fff640 1 -- 172.21.15.3:0/68863276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55843ac6f680 con 0x55843ad09330 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa6795c9640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad72710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa678dc8640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad18260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:33.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa673fff640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad187a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa668004d70 tx=0x7fa6680096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d42db7aeba52b49 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa653fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa653fff640 1 -- 172.21.15.3:0/68863276 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa668009310 con 0x55843ad09330 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa668004470 con 0x55843ad09330 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa653fff640 1 -- 172.21.15.3:0/68863276 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa6680049d0 con 0x55843ad09330 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa653fff640 1 -- 172.21.15.3:0/68863276 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa668004b90 con 0x55843ad09330 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa67dbfd740 10 client.28912 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x7fa664019040 con 0x55843ad09330 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.946+0000 7fa653fff640 1 -- 172.21.15.3:0/68863276 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa66803d080 con 0x55843ad09330 2024-02-09T23:03:33.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa653fff640 1 -- 172.21.15.3:0/68863276 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa668037030 con 0x55843ad09330 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 10 client.28912 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 10 client.28912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 20 client.28912 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad7a800 0x55843ad7cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 4 client.28912 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55843a9fc890 con 0x55843ad7a800 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa678dc8640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad7a800 0x55843ad7cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa678dc8640 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad7a800 0x55843ad7cc20 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7fad73c6aa9ed14 server_cookie=c06a571589508293 in_seq=0 out_seq=0 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa653fff640 10 client.28912 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa653fff640 1 -- 172.21.15.3:0/68863276 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55843a9fc890 con 0x55843ad7a800 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa653fff640 10 client.28912 handle_command_reply: tid=0 2024-02-09T23:03:33.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 client.28912 shutdown 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 msgr2=0x55843ad187a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad187a0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa668004d70 tx=0x7fa6680096d0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 shutdown_connections 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55843ad7a800 0x55843ad7cc20 crc :-1 s=READY pgs=336 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55843ad11eb0 0x55843ad72710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55843ad09330 0x55843ad187a0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa653fff640 0 client.28912 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 --2- 172.21.15.3:0/68863276 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55843ad08980 0x55843ad18260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.947+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 >> 172.21.15.3:0/68863276 conn(0x55843ad03050 msgr2=0x55843ad04460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 shutdown_connections 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67dbfd740 1 -- 172.21.15.3:0/68863276 wait complete. 2024-02-09T23:03:33.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67dbfd740 20 client.28912 trim_cache size 0 max 16384 2024-02-09T23:03:33.960 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:03:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:03:33.960 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:03:33.960 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa65c0a2260 msgr2=0x7fa65c0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa65c0a2260 0x7fa65c0a4720 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fa664009fd0 tx=0x7fa664005f90 comp rx=0 tx=0).stop 2024-02-09T23:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 msgr2=0x7fa674101dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674101dd0 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7fa66c004af0 tx=0x7fa66c0325c0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 shutdown_connections 2024-02-09T23:03:33.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa65c0a2260 0x7fa65c0a4720 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa674102850 0x7fa6741a5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa674105410 0x7fa674102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 --2- 172.21.15.3:0/622566972 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa674100340 0x7fa674101dd0 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 >> 172.21.15.3:0/622566972 conn(0x7fa6740772e0 msgr2=0x7fa6740688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 shutdown_connections 2024-02-09T23:03:33.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:33.948+0000 7fa67b02c640 1 -- 172.21.15.3:0/622566972 wait complete. 2024-02-09T23:03:33.963 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:03:33.963 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:03:33.976 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:33 smithi089 ceph-mon[36665]: pgmap v387: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-02-09T23:03:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:33 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:03:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:33 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:03:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:33 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:03:34.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 -- 172.21.15.3:0/2755455271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250100330 msgr2=0x7fe250100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 --2- 172.21.15.3:0/2755455271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250100330 0x7fe250100710 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009930 tx=0x7fe24c02bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:34.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 -- 172.21.15.3:0/2755455271 shutdown_connections 2024-02-09T23:03:34.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 --2- 172.21.15.3:0/2755455271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250100ce0 0x7fe250104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 --2- 172.21.15.3:0/2755455271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe250105400 0x7fe25010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 --2- 172.21.15.3:0/2755455271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250100330 0x7fe250100710 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 -- 172.21.15.3:0/2755455271 >> 172.21.15.3:0/2755455271 conn(0x7fe2500fbbb0 msgr2=0x7fe2500fdfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 -- 172.21.15.3:0/2755455271 shutdown_connections 2024-02-09T23:03:34.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.106+0000 7fe2582b5640 1 -- 172.21.15.3:0/2755455271 wait complete. 2024-02-09T23:03:34.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 Processor -- start 2024-02-09T23:03:34.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 -- start start 2024-02-09T23:03:34.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe250100330 0x7fe25019d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250105400 0x7fe25019db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.118 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 0x7fe250197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250198210 con 0x7fe250197760 2024-02-09T23:03:34.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250198380 con 0x7fe250105400 2024-02-09T23:03:34.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe2582b5640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe250198540 con 0x7fe250100330 2024-02-09T23:03:34.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe256852640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 0x7fe250197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe256852640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 0x7fe250197c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52670/0 (socket says 172.21.15.3:52670) 2024-02-09T23:03:34.119 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe256852640 1 -- 172.21.15.3:0/1226517319 learned_addr learned my addr 172.21.15.3:0/1226517319 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe256051640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe250100330 0x7fe25019d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.107+0000 7fe255850640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250105400 0x7fe25019db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe256852640 1 -- 172.21.15.3:0/1226517319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe250100330 msgr2=0x7fe25019d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe256852640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe250100330 0x7fe25019d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.120 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe256852640 1 -- 172.21.15.3:0/1226517319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250105400 msgr2=0x7fe25019db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe256852640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250105400 0x7fe25019db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe256852640 1 -- 172.21.15.3:0/1226517319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe240009490 con 0x7fe250197760 2024-02-09T23:03:34.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe256852640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 0x7fe250197c10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fe24800eb10 tx=0x7fe24800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248004b20 con 0x7fe250197760 2024-02-09T23:03:34.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe248004c80 con 0x7fe250197760 2024-02-09T23:03:34.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe24c009560 con 0x7fe250197760 2024-02-09T23:03:34.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe248010650 con 0x7fe250197760 2024-02-09T23:03:34.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.108+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2501a4800 con 0x7fe250197760 2024-02-09T23:03:34.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.109+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe250101800 con 0x7fe250197760 2024-02-09T23:03:34.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.110+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe248005020 con 0x7fe250197760 2024-02-09T23:03:34.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.111+0000 7fe2477fe640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe23c0a2230 0x7fe23c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.111+0000 7fe256051640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe23c0a2230 0x7fe23c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.111+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe248014070 con 0x7fe250197760 2024-02-09T23:03:34.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.112+0000 7fe256051640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe23c0a2230 0x7fe23c0a46f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe24c005ab0 tx=0x7fe24c005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.116+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe248097250 con 0x7fe250197760 2024-02-09T23:03:34.357 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.355+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe250198ca0 con 0x7fe250197760 2024-02-09T23:03:34.358 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.355+0000 7fe2477fe640 1 -- 172.21.15.3:0/1226517319 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+11364 (secure 0 0 0) 0x7fe24809a910 con 0x7fe250197760 2024-02-09T23:03:34.358 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:03:34.359 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":51,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":51,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:03:07.509267+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_28706":{"gid":28706,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/1843835937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1843835937},{"type":"v1","addr":"172.21.15.64:6835","nonce":1843835937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:03:34.359 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 51 2024-02-09T23:03:34.360 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.357+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe23c0a2230 msgr2=0x7fe23c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.357+0000 7fe2582b5640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe23c0a2230 0x7fe23c0a46f0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe24c005ab0 tx=0x7fe24c005fd0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.357+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 msgr2=0x7fe250197c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.357+0000 7fe2582b5640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 0x7fe250197c10 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7fe24800eb10 tx=0x7fe24800c2d0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 shutdown_connections 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe23c0a2230 0x7fe23c0a46f0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe250105400 0x7fe25019db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe250197760 0x7fe250197c10 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 --2- 172.21.15.3:0/1226517319 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe250100330 0x7fe25019d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 >> 172.21.15.3:0/1226517319 conn(0x7fe2500fbbb0 msgr2=0x7fe250069cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 shutdown_connections 2024-02-09T23:03:34.361 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.358+0000 7fe2582b5640 1 -- 172.21.15.3:0/1226517319 wait complete. 2024-02-09T23:03:34.372 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:34.372 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:03:34.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 -- 172.21.15.3:0/2276157124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28100d10 msgr2=0x7f3b28104e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/2276157124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28100d10 0x7f3b28104e60 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f3b18009930 tx=0x7f3b1802bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:34.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 -- 172.21.15.3:0/2276157124 shutdown_connections 2024-02-09T23:03:34.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/2276157124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28105540 0x7f3b2810d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.483 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/2276157124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28100d10 0x7f3b28104e60 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/2276157124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b28100360 0x7f3b28100740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 -- 172.21.15.3:0/2276157124 >> 172.21.15.3:0/2276157124 conn(0x7f3b280772a0 msgr2=0x7f3b280776b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 -- 172.21.15.3:0/2276157124 shutdown_connections 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.478+0000 7f3b2f82a640 1 -- 172.21.15.3:0/2276157124 wait complete. 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 Processor -- start 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 -- start start 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28100360 0x7f3b2819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b28105540 0x7f3b2819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28197840 0x7f3b28197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281982f0 con 0x7f3b28197840 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b28198460 con 0x7f3b28105540 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2f82a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b281985d0 con 0x7f3b28100360 2024-02-09T23:03:34.484 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28197840 0x7f3b28197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28197840 0x7f3b28197cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52684/0 (socket says 172.21.15.3:52684) 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2d5c6640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28100360 0x7f3b2819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2cdc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b28105540 0x7f3b2819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.479+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3152647067 learned_addr learned my addr 172.21.15.3:0/3152647067 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2d5c6640 1 -- 172.21.15.3:0/3152647067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b28105540 msgr2=0x7f3b2819dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2d5c6640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b28105540 0x7f3b2819dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2d5c6640 1 -- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28197840 msgr2=0x7f3b28197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2d5c6640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28197840 0x7f3b28197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2d5c6640 1 -- 172.21.15.3:0/3152647067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b18009560 con 0x7f3b28100360 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2d5c6640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28100360 0x7f3b2819d6d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400e9f0 tx=0x7f3b2400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b1e7fc640 1 -- 172.21.15.3:0/3152647067 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2400ccc0 con 0x7f3b28100360 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b1e7fc640 1 -- 172.21.15.3:0/3152647067 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b2400ce20 con 0x7f3b28100360 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b1e7fc640 1 -- 172.21.15.3:0/3152647067 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b24018920 con 0x7f3b28100360 2024-02-09T23:03:34.485 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b281a4580 con 0x7f3b28100360 2024-02-09T23:03:34.486 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.480+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b281a4a60 con 0x7f3b28100360 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.482+0000 7f3b1e7fc640 1 -- 172.21.15.3:0/3152647067 <== mon.1 v2:172.21.15.89:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3b24018a80 con 0x7f3b28100360 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 Processor -- start 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 -- start start 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b1e7fc640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b140a22b0 0x7f3b140a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677d2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677d2f60 0x559a677d53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a67785020 con 0x559a677c9a30 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a67738320 con 0x559a677d2f60 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b32403740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a67735850 con 0x559a677ca3e0 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2d5c6640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b140a22b0 0x7f3b140a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677c9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52686/0 (socket says 172.21.15.3:52686) 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/456718910 learned_addr learned my addr 172.21.15.3:0/456718910 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2cdc5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677d2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b1e7fc640 1 -- 172.21.15.3:0/3152647067 <== mon.1 v2:172.21.15.89:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3b24014070 con 0x7f3b28100360 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677d2f60 0x559a677d53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2cdc5640 1 -- 172.21.15.3:0/456718910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677d2f60 msgr2=0x559a677d53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677d2f60 0x559a677d53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2cdc5640 1 -- 172.21.15.3:0/456718910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 msgr2=0x559a677c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.493 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.483+0000 7f3b2cdc5640 1 -- 172.21.15.3:0/456718910 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a676f3210 con 0x559a677ca3e0 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.484+0000 7f3b2d5c6640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b140a22b0 0x7f3b140a4770 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cbaf0 tx=0x7f3b2400cf90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.484+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677d2990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b1802c170 tx=0x7f3b1802c5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52a95c540f3507ab server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.484+0000 7f3b137fe640 1 -- 172.21.15.3:0/456718910 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18004780 con 0x559a677ca3e0 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.484+0000 7f3b137fe640 1 -- 172.21.15.3:0/456718910 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b180048e0 con 0x559a677ca3e0 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.484+0000 7f3b137fe640 1 -- 172.21.15.3:0/456718910 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b18005050 con 0x559a677ca3e0 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 -- 172.21.15.3:0/456718910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 msgr2=0x559a677d2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677d2990 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3b1802c170 tx=0x7f3b1802c5d0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 -- 172.21.15.3:0/456718910 shutdown_connections 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677d2f60 0x559a677d53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677d2990 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 --2- 172.21.15.3:0/456718910 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 -- 172.21.15.3:0/456718910 >> 172.21.15.3:0/456718910 conn(0x559a677c4100 msgr2=0x559a677c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 -- 172.21.15.3:0/456718910 shutdown_connections 2024-02-09T23:03:34.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 -- 172.21.15.3:0/456718910 wait complete. 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 -- start start 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.485+0000 7f3b32403740 1 Processor -- start 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 0x559a677cff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677d0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a67731030 con 0x559a677d2f60 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a67404430 con 0x559a677ca3e0 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a673c0e00 con 0x559a677c9a30 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 10 client.0 _ll_register_callbacks cb 0x559a67726030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 0x559a677cff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2cdc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 0x559a677cff10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:55932/0 (socket says 172.21.15.3:55932) 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/1785537828 learned_addr learned my addr 172.21.15.3:0/1785537828 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2cdc5640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52070/0 (socket says 172.21.15.3:52070) 2024-02-09T23:03:34.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/1785537828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 msgr2=0x559a677d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/1785537828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 msgr2=0x559a677d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/1785537828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b2810d920 con 0x559a677c9a30 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 0x559a677cff10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3b20013f90 tx=0x7f3b20009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60b9294835a77245 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b10ff9640 1 -- 172.21.15.3:0/1785537828 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20004b20 con 0x559a677c9a30 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b10ff9640 1 -- 172.21.15.3:0/1785537828 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3b20004c80 con 0x559a677c9a30 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.486+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559a674307e0 con 0x559a677c9a30 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b10ff9640 1 -- 172.21.15.3:0/1785537828 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b20004e40 con 0x559a677c9a30 2024-02-09T23:03:34.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 10 client.28930 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x559a6772f6b0 con 0x559a677c9a30 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b10ff9640 1 -- 172.21.15.3:0/1785537828 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559a674307e0 con 0x559a677c9a30 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b10ff9640 1 -- 172.21.15.3:0/1785537828 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3b2002c750 con 0x559a677c9a30 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 10 client.28930 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 10 client.28930 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 20 client.28930 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a677e33f0 0x559a677e5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 4 client.28930 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559a6746d720 con 0x559a677e33f0 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.487+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a677e33f0 0x559a677e5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b10ff9640 10 client.28930 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a677e33f0 0x559a677e5810 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2d5456d042ce39eb server_cookie=5ba9b154d4cd2631 in_seq=0 out_seq=0 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b10ff9640 1 -- 172.21.15.3:0/1785537828 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559a6746d720 con 0x559a677e33f0 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b10ff9640 10 client.28930 handle_command_reply: tid=0 2024-02-09T23:03:34.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 client.28930 shutdown 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 msgr2=0x559a677cff10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 0x559a677cff10 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3b20013f90 tx=0x7f3b20009500 comp rx=0 tx=0).stop 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 shutdown_connections 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a677e33f0 0x559a677e5810 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677d0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b10ff9640 0 client.28930 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 --2- 172.21.15.3:0/1785537828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677c9a30 0x559a677cff10 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 >> 172.21.15.3:0/1785537828 conn(0x559a677c4100 msgr2=0x559a677c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 shutdown_connections 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.488+0000 7f3b32403740 1 -- 172.21.15.3:0/1785537828 wait complete. 2024-02-09T23:03:34.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.489+0000 7f3b32403740 20 client.28930 trim_cache size 0 max 16384 2024-02-09T23:03:34.508 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 Processor -- start 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 -- start start 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677c9a30 0x559a677d2640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a676b6130 con 0x559a677d2f60 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a674307e0 con 0x559a677c9a30 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b32403740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a677d0ee0 con 0x559a677ca3e0 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677cb6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52698/0 (socket says 172.21.15.3:52698) 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3233468877 learned_addr learned my addr 172.21.15.3:0/3233468877 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2cdc5640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677c9a30 0x559a677d2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3233468877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 msgr2=0x559a677cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.509 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3233468877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677c9a30 msgr2=0x559a677d2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677c9a30 0x559a677d2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3233468877 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559a676cbc90 con 0x559a677d2f60 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677cb6f0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x559a677d1320 tx=0x7f3b200021d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d273b5260b283758 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b12ffd640 1 -- 172.21.15.3:0/3233468877 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20017070 con 0x559a677d2f60 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b12ffd640 1 -- 172.21.15.3:0/3233468877 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b20013e20 con 0x559a677d2f60 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.501+0000 7f3b12ffd640 1 -- 172.21.15.3:0/3233468877 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b2002c8c0 con 0x559a677d2f60 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 -- 172.21.15.3:0/3233468877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 msgr2=0x559a677cb6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677cb6f0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x559a677d1320 tx=0x7f3b200021d0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 -- 172.21.15.3:0/3233468877 shutdown_connections 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677d2f60 0x559a677cb6f0 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677ca3e0 0x559a677cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 --2- 172.21.15.3:0/3233468877 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677c9a30 0x559a677d2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.510 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 -- 172.21.15.3:0/3233468877 >> 172.21.15.3:0/3233468877 conn(0x559a677c4100 msgr2=0x559a677c44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 -- 172.21.15.3:0/3233468877 shutdown_connections 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 -- 172.21.15.3:0/3233468877 wait complete. 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.502+0000 7f3b32403740 1 -- start start 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 Processor -- start 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677d8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d91b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677d2f60 0x559a678398e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a677d9940 con 0x559a677c9a30 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a677d9a80 con 0x559a677ca3e0 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x559a67839e20 con 0x559a677d2f60 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b32403740 10 client.0 _ll_register_callbacks cb 0x7f3b200093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677d8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677d8c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52700/0 (socket says 172.21.15.3:52700) 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3368016054 learned_addr learned my addr 172.21.15.3:0/3368016054 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.511 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677d2f60 msgr2=0x559a678398e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677d2f60 0x559a678398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3368016054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 msgr2=0x559a677d91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d91b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.503+0000 7f3b2ddc7640 1 -- 172.21.15.3:0/3368016054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559a6772f6b0 con 0x559a677c9a30 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b2ddc7640 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677d8c70 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f3b20005c90 tx=0x7f3b20013d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93c2bc575cad9ae7 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b10ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b10ff9640 1 -- 172.21.15.3:0/3368016054 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20017040 con 0x559a677c9a30 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b10ff9640 1 -- 172.21.15.3:0/3368016054 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3b20004be0 con 0x559a677c9a30 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3b180048e0 con 0x559a677c9a30 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b10ff9640 1 -- 172.21.15.3:0/3368016054 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3b20009d00 con 0x559a677c9a30 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 10 client.19956 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.512 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x559a676f1a70 con 0x559a677c9a30 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b10ff9640 1 -- 172.21.15.3:0/3368016054 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b20044020 con 0x559a677c9a30 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b10ff9640 1 -- 172.21.15.3:0/3368016054 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3b20005070 con 0x559a677c9a30 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 10 client.19956 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 10 client.19956 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 20 client.19956 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a6783b2f0 0x559a6783d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 4 client.19956 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559a674bc890 con 0x559a6783b2f0 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.504+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a6783b2f0 0x559a6783d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b2cdc5640 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a6783b2f0 0x559a6783d710 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d630e0da6d343adb server_cookie=186dffc359ad53b9 in_seq=0 out_seq=0 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b10ff9640 10 client.19956 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b10ff9640 1 -- 172.21.15.3:0/3368016054 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559a674bc890 con 0x559a6783b2f0 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b10ff9640 10 client.19956 handle_command_reply: tid=0 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 client.19956 shutdown 2024-02-09T23:03:34.513 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 msgr2=0x559a677d8c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677d8c70 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f3b20005c90 tx=0x7f3b20013d90 comp rx=0 tx=0).stop 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 shutdown_connections 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x559a6783b2f0 0x559a6783d710 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x559a677ca3e0 0x559a677d91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b10ff9640 0 client.19956 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x559a677d2f60 0x559a678398e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 --2- 172.21.15.3:0/3368016054 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x559a677c9a30 0x559a677d8c70 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 >> 172.21.15.3:0/3368016054 conn(0x559a677c4100 msgr2=0x559a677cb1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 shutdown_connections 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 1 -- 172.21.15.3:0/3368016054 wait complete. 2024-02-09T23:03:34.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.505+0000 7f3b32403740 20 client.19956 trim_cache size 0 max 16384 2024-02-09T23:03:34.515 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b140a22b0 msgr2=0x7f3b140a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b140a22b0 0x7f3b140a4770 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3b240cbaf0 tx=0x7f3b2400cf90 comp rx=0 tx=0).stop 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28100360 msgr2=0x7f3b2819d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28100360 0x7f3b2819d6d0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3b2400e9f0 tx=0x7f3b2400eeb0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 shutdown_connections 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f3b140a22b0 0x7f3b140a4770 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3b28105540 0x7f3b2819dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3b28197840 0x7f3b28197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 --2- 172.21.15.3:0/3152647067 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3b28100360 0x7f3b2819d6d0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 >> 172.21.15.3:0/3152647067 conn(0x7f3b280772a0 msgr2=0x7f3b28069c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 shutdown_connections 2024-02-09T23:03:34.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.506+0000 7f3b2f82a640 1 -- 172.21.15.3:0/3152647067 wait complete. 2024-02-09T23:03:34.520 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:34.520 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 -- 172.21.15.3:0/1874980883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 msgr2=0x7f316410e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1874980883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316410e160 secure :-1 s=READY pgs=1720 cs=0 l=1 rev1=1 crypto rx=0x7f315c009cd0 tx=0x7f315c02c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 -- 172.21.15.3:0/1874980883 shutdown_connections 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1874980883 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3164100250 0x7f3164105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1874980883 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316410e160 unknown :-1 s=CLOSED pgs=1720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1874980883 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640ff8a0 0x7f31640ffc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 -- 172.21.15.3:0/1874980883 >> 172.21.15.3:0/1874980883 conn(0x7f31640776a0 msgr2=0x7f31640fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 -- 172.21.15.3:0/1874980883 shutdown_connections 2024-02-09T23:03:34.639 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.634+0000 7f316a5ac640 1 -- 172.21.15.3:0/1874980883 wait complete. 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 Processor -- start 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 -- start start 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640ff8a0 0x7f316419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31641976b0 0x7f3164197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3164198180 con 0x7f3164106550 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641982f0 con 0x7f31641976b0 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f316a5ac640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31641984b0 con 0x7f31640ff8a0 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316419db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52722/0 (socket says 172.21.15.3:52722) 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 -- 172.21.15.3:0/1974623346 learned_addr learned my addr 172.21.15.3:0/1974623346 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f3163fff640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640ff8a0 0x7f316419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f3168b49640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31641976b0 0x7f3164197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.640 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 -- 172.21.15.3:0/1974623346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640ff8a0 msgr2=0x7f316419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640ff8a0 0x7f316419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 -- 172.21.15.3:0/1974623346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31641976b0 msgr2=0x7f3164197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31641976b0 0x7f3164197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.635+0000 7f31637fe640 1 -- 172.21.15.3:0/1974623346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f315c009900 con 0x7f3164106550 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.636+0000 7f31637fe640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316419db30 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f315800e9c0 tx=0x7f315800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.636+0000 7f31617fa640 1 -- 172.21.15.3:0/1974623346 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158009700 con 0x7f3164106550 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.636+0000 7f31617fa640 1 -- 172.21.15.3:0/1974623346 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315800cd80 con 0x7f3164106550 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.636+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31641a4450 con 0x7f3164106550 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.636+0000 7f31617fa640 1 -- 172.21.15.3:0/1974623346 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3158010430 con 0x7f3164106550 2024-02-09T23:03:34.641 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.636+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31641a4930 con 0x7f3164106550 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.638+0000 7f31617fa640 1 -- 172.21.15.3:0/1974623346 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3158005000 con 0x7f3164106550 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 Processor -- start 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 -- start start 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f31617fa640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f314c0a2230 0x7f314c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e203e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e20cf60 0x55663e20f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.648 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e1bf020 con 0x55663e20cf60 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e172320 con 0x55663e203a30 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f316d17c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e16f850 con 0x55663e2043e0 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f31617fa640 1 -- 172.21.15.3:0/1974623346 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3158014070 con 0x7f3164106550 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3168b49640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f314c0a2230 0x7f314c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e203e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3168b49640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e20cf60 0x55663e20f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e203e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52114/0 (socket says 172.21.15.3:52114) 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 -- 172.21.15.3:0/1985565488 learned_addr learned my addr 172.21.15.3:0/1985565488 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f31637fe640 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 -- 172.21.15.3:0/1985565488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 msgr2=0x55663e20c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 -- 172.21.15.3:0/1985565488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e20cf60 msgr2=0x55663e20f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e20cf60 0x55663e20f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.649 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f3163fff640 1 -- 172.21.15.3:0/1985565488 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55663e12d210 con 0x55663e203a30 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.639+0000 7f31637fe640 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f3168b49640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f314c0a2230 0x7f314c0a46f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f315c005cc0 tx=0x7f315c032f20 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f3163fff640 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e203e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3154009d20 tx=0x7f3154011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=98b04767d781443a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f314a7fc640 1 -- 172.21.15.3:0/1985565488 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154019630 con 0x55663e203a30 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f314a7fc640 1 -- 172.21.15.3:0/1985565488 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3154019c70 con 0x55663e203a30 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f314a7fc640 1 -- 172.21.15.3:0/1985565488 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f315400eb90 con 0x55663e203a30 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f316d17c740 1 -- 172.21.15.3:0/1985565488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 msgr2=0x55663e203e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.640+0000 7f316d17c740 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e203e10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f3154009d20 tx=0x7f3154011810 comp rx=0 tx=0).stop 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- 172.21.15.3:0/1985565488 shutdown_connections 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e20cf60 0x55663e20f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 --2- 172.21.15.3:0/1985565488 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e203e10 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- 172.21.15.3:0/1985565488 >> 172.21.15.3:0/1985565488 conn(0x55663e1fe100 msgr2=0x55663e1fe4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.650 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- 172.21.15.3:0/1985565488 shutdown_connections 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- 172.21.15.3:0/1985565488 wait complete. 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- start start 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 Processor -- start 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e209f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e20a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e20cf60 0x55663e20a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e16b030 con 0x55663e2043e0 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663de3e430 con 0x55663e203a30 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663ddfae00 con 0x55663e20cf60 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f316d17c740 10 client.0 _ll_register_callbacks cb 0x55663e160030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.641+0000 7f31637fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e20a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e20a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52736/0 (socket says 172.21.15.3:52736) 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 -- 172.21.15.3:0/2073392346 learned_addr learned my addr 172.21.15.3:0/2073392346 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.651 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f3163fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e209f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f3168b49640 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e20cf60 0x55663e20a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 -- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e20cf60 msgr2=0x55663e20a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e20cf60 0x55663e20a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 -- 172.21.15.3:0/2073392346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 msgr2=0x55663e209f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e209f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 -- 172.21.15.3:0/2073392346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3154009950 con 0x55663e2043e0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31637fe640 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e20a490 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f31580cbbe0 tx=0x7f315809b5e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f062a9e1f99dbfb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31437fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31437fe640 1 -- 172.21.15.3:0/2073392346 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580a35c0 con 0x55663e2043e0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31437fe640 1 -- 172.21.15.3:0/2073392346 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f3158097490 con 0x55663e2043e0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55663de6a7e0 con 0x55663e2043e0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31437fe640 1 -- 172.21.15.3:0/2073392346 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3158097650 con 0x55663e2043e0 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f316d17c740 10 client.19968 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.652 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x55663e1696b0 con 0x55663e2043e0 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.642+0000 7f31437fe640 1 -- 172.21.15.3:0/2073392346 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55663de6a7e0 con 0x55663e2043e0 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f31437fe640 1 -- 172.21.15.3:0/2073392346 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f315801f060 con 0x55663e2043e0 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f316d17c740 10 client.19968 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f316d17c740 10 client.19968 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f316d17c740 20 client.19968 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f316d17c740 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e21d330 0x55663e21f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f316d17c740 4 client.19968 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55663dea7720 con 0x55663e21d330 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f3163fff640 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e21d330 0x55663e21f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f3163fff640 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e21d330 0x55663e21f720 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=767d13d64150414a server_cookie=e630361e35a4bcec in_seq=0 out_seq=0 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.643+0000 7f31437fe640 10 client.19968 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f31437fe640 1 -- 172.21.15.3:0/2073392346 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55663dea7720 con 0x55663e21d330 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f31437fe640 10 client.19968 handle_command_reply: tid=0 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 client.19968 shutdown 2024-02-09T23:03:34.653 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 msgr2=0x55663e20a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e20a490 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f31580cbbe0 tx=0x7f315809b5e0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 shutdown_connections 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e21d330 0x55663e21f720 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e20cf60 0x55663e20a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e20a490 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f31437fe640 0 client.19968 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 --2- 172.21.15.3:0/2073392346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e203a30 0x55663e209f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 >> 172.21.15.3:0/2073392346 conn(0x55663e1fe100 msgr2=0x55663e1ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 shutdown_connections 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 1 -- 172.21.15.3:0/2073392346 wait complete. 2024-02-09T23:03:34.654 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.644+0000 7f316d17c740 20 client.19968 trim_cache size 0 max 16384 2024-02-09T23:03:34.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 Processor -- start 2024-02-09T23:03:34.665 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 -- start start 2024-02-09T23:03:34.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 0x55663e2240c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.666 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e20a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e0f0130 con 0x55663e203a30 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663de6a7e0 con 0x55663e20cf60 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f316d17c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e20b3e0 con 0x55663e2043e0 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f3163fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 0x55663e2240c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f3163fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 0x55663e2240c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52752/0 (socket says 172.21.15.3:52752) 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f3163fff640 1 -- 172.21.15.3:0/792915062 learned_addr learned my addr 172.21.15.3:0/792915062 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f31637fe640 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.657+0000 7f3168b49640 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e20a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3163fff640 1 -- 172.21.15.3:0/792915062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 msgr2=0x55663e20a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.667 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3163fff640 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3163fff640 1 -- 172.21.15.3:0/792915062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 msgr2=0x55663e20a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3163fff640 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e20a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3163fff640 1 -- 172.21.15.3:0/792915062 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55663e105c90 con 0x55663e203a30 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3163fff640 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 0x55663e2240c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3154009c40 tx=0x7f3154009660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a60041ac1e3d3f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3149ffb640 1 -- 172.21.15.3:0/792915062 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31540050c0 con 0x55663e203a30 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3149ffb640 1 -- 172.21.15.3:0/792915062 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f315402c070 con 0x55663e203a30 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.658+0000 7f3149ffb640 1 -- 172.21.15.3:0/792915062 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3154019690 con 0x55663e203a30 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- 172.21.15.3:0/792915062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 msgr2=0x55663e2240c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 0x55663e2240c0 secure :-1 s=READY pgs=1723 cs=0 l=1 rev1=1 crypto rx=0x7f3154009c40 tx=0x7f3154009660 comp rx=0 tx=0).stop 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- 172.21.15.3:0/792915062 shutdown_connections 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e20a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e2043e0 0x55663e20a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.668 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- 172.21.15.3:0/792915062 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e203a30 0x55663e2240c0 unknown :-1 s=CLOSED pgs=1723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- 172.21.15.3:0/792915062 >> 172.21.15.3:0/792915062 conn(0x55663e1fe100 msgr2=0x55663e2047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- 172.21.15.3:0/792915062 shutdown_connections 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- 172.21.15.3:0/792915062 wait complete. 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- start start 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 Processor -- start 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e203a30 0x55663e212970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e212eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e2133f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e2148c0 con 0x55663e2043e0 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e273880 con 0x55663e20cf60 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.659+0000 7f316d17c740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55663e273a40 con 0x55663e203a30 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f316d17c740 10 client.0 _ll_register_callbacks cb 0x7f3154009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e212eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.669 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e212eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52760/0 (socket says 172.21.15.3:52760) 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f3163fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e203a30 0x55663e212970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 -- 172.21.15.3:0/2925243854 learned_addr learned my addr 172.21.15.3:0/2925243854 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f3168b49640 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e2133f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 -- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e203a30 msgr2=0x55663e212970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e203a30 0x55663e212970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 -- 172.21.15.3:0/2925243854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 msgr2=0x55663e2133f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e2133f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 -- 172.21.15.3:0/2925243854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55663e12d210 con 0x55663e2043e0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31637fe640 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e212eb0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f31580a3130 tx=0x7f3158004a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bde1ef2080af90db server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31437fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31437fe640 1 -- 172.21.15.3:0/2925243854 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31580a3320 con 0x55663e2043e0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31437fe640 1 -- 172.21.15.3:0/2925243854 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f31580a35c0 con 0x55663e2043e0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3154019c70 con 0x55663e2043e0 2024-02-09T23:03:34.670 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f31437fe640 1 -- 172.21.15.3:0/2925243854 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f31580054c0 con 0x55663e2043e0 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f316d17c740 10 client.19974 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.660+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x55663e12ba70 con 0x55663e2043e0 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f31437fe640 1 -- 172.21.15.3:0/2925243854 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3154019c70 con 0x55663e2043e0 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f31437fe640 1 -- 172.21.15.3:0/2925243854 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f315801f070 con 0x55663e2043e0 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 10 client.19974 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 10 client.19974 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 20 client.19974 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e274d10 0x55663e277100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 4 client.19974 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55663def6890 con 0x55663e274d10 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f3163fff640 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e274d10 0x55663e277100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f31437fe640 10 client.19974 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f3163fff640 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e274d10 0x55663e277100 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=34d5f24c40d5e91c server_cookie=209d63c1778b3d8a in_seq=0 out_seq=0 2024-02-09T23:03:34.671 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f31437fe640 1 -- 172.21.15.3:0/2925243854 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55663def6890 con 0x55663e274d10 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f31437fe640 10 client.19974 handle_command_reply: tid=0 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.661+0000 7f316d17c740 1 client.19974 shutdown 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 msgr2=0x55663e212eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e212eb0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f31580a3130 tx=0x7f3158004a50 comp rx=0 tx=0).stop 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 shutdown_connections 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55663e274d10 0x55663e277100 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55663e20cf60 0x55663e2133f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55663e2043e0 0x55663e212eb0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 --2- 172.21.15.3:0/2925243854 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55663e203a30 0x55663e212970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f31437fe640 0 client.19974 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 >> 172.21.15.3:0/2925243854 conn(0x55663e1fe100 msgr2=0x55663e2240c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 shutdown_connections 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 1 -- 172.21.15.3:0/2925243854 wait complete. 2024-02-09T23:03:34.672 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316d17c740 20 client.19974 trim_cache size 0 max 16384 2024-02-09T23:03:34.673 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f314c0a2230 msgr2=0x7f314c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f314c0a2230 0x7f314c0a46f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f315c005cc0 tx=0x7f315c032f20 comp rx=0 tx=0).stop 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 msgr2=0x7f316419db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316419db30 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f315800e9c0 tx=0x7f315800ee80 comp rx=0 tx=0).stop 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 shutdown_connections 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f314c0a2230 0x7f314c0a46f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f31641976b0 0x7f3164197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3164106550 0x7f316419db30 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 --2- 172.21.15.3:0/1974623346 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f31640ff8a0 0x7f316419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 >> 172.21.15.3:0/1974623346 conn(0x7f31640776a0 msgr2=0x7f3164104730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 shutdown_connections 2024-02-09T23:03:34.674 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.662+0000 7f316a5ac640 1 -- 172.21.15.3:0/1974623346 wait complete. 2024-02-09T23:03:34.675 INFO:tasks.workunit.client.0.smithi003.stdout: 1 462222 3.67 MB/sec execute 568 sec latency 33.921 ms 2024-02-09T23:03:34.677 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:34.677 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:03:34.787 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 -- 172.21.15.3:0/1972885690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100cf0 msgr2=0x7f62b8104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 --2- 172.21.15.3:0/1972885690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100cf0 0x7f62b8104d30 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f62a8009930 tx=0x7f62a802bc30 comp rx=0 tx=0).stop 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 -- 172.21.15.3:0/1972885690 shutdown_connections 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 --2- 172.21.15.3:0/1972885690 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62b8105410 0x7f62b810cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 --2- 172.21.15.3:0/1972885690 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100cf0 0x7f62b8104d30 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 --2- 172.21.15.3:0/1972885690 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8100340 0x7f62b8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 -- 172.21.15.3:0/1972885690 >> 172.21.15.3:0/1972885690 conn(0x7f62b80772e0 msgr2=0x7f62b80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 -- 172.21.15.3:0/1972885690 shutdown_connections 2024-02-09T23:03:34.788 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 -- 172.21.15.3:0/1972885690 wait complete. 2024-02-09T23:03:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.785+0000 7f62bd20b640 1 Processor -- start 2024-02-09T23:03:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 -- start start 2024-02-09T23:03:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 0x7f62b819d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.790 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8105410 0x7f62b819db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62b8197720 0x7f62b8197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b81981f0 con 0x7f62b8100340 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b8198360 con 0x7f62b8105410 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62bd20b640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62b8198520 con 0x7f62b8197720 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 0x7f62b819d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 0x7f62b819d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52774/0 (socket says 172.21.15.3:52774) 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 -- 172.21.15.3:0/2485031974 learned_addr learned my addr 172.21.15.3:0/2485031974 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b659c640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8105410 0x7f62b819db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b759e640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62b8197720 0x7f62b8197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 -- 172.21.15.3:0/2485031974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62b8197720 msgr2=0x7f62b8197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62b8197720 0x7f62b8197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 -- 172.21.15.3:0/2485031974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8105410 msgr2=0x7f62b819db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8105410 0x7f62b819db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.791 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 -- 172.21.15.3:0/2485031974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a8009560 con 0x7f62b8100340 2024-02-09T23:03:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.786+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 0x7f62b819d610 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9f0 tx=0x7f62a000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.787+0000 7f62affff640 1 -- 172.21.15.3:0/2485031974 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0009700 con 0x7f62b8100340 2024-02-09T23:03:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.787+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62b81a4470 con 0x7f62b8100340 2024-02-09T23:03:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.787+0000 7f62affff640 1 -- 172.21.15.3:0/2485031974 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62a000cd80 con 0x7f62b8100340 2024-02-09T23:03:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.787+0000 7f62affff640 1 -- 172.21.15.3:0/2485031974 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a0018920 con 0x7f62b8100340 2024-02-09T23:03:34.792 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.787+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b81a49a0 con 0x7f62b8100340 2024-02-09T23:03:34.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.789+0000 7f62affff640 1 -- 172.21.15.3:0/2485031974 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f62a0018a80 con 0x7f62b8100340 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.789+0000 7f62bfdd7740 1 Processor -- start 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.789+0000 7f62bfdd7740 1 -- start start 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa63db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa64ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa64d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa5f9020 con 0x5564aa63d7b0 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62affff640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62a40a2230 0x7f62a40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa5ac320 con 0x5564aa64b370 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa5a9850 con 0x5564aa63e160 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa63db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa63db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52784/0 (socket says 172.21.15.3:52784) 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 -- 172.21.15.3:0/2790841553 learned_addr learned my addr 172.21.15.3:0/2790841553 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b659c640 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa64d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa64ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62a40a2230 0x7f62a40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62affff640 1 -- 172.21.15.3:0/2485031974 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f62a0014070 con 0x7f62b8100340 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 -- 172.21.15.3:0/2790841553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 msgr2=0x5564aa64ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa64ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 -- 172.21.15.3:0/2790841553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 msgr2=0x5564aa64d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa64d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 -- 172.21.15.3:0/2790841553 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564aa567210 con 0x5564aa63d7b0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b659c640 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa64d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa63db90 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f62b0011da0 tx=0x7f62b000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13206a851e1efc0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.790+0000 7f62b759e640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62a40a2230 0x7f62a40a46f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009fd0 tx=0x7f62b001e000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62ad7fa640 1 -- 172.21.15.3:0/2790841553 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b0009e50 con 0x5564aa63d7b0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62ad7fa640 1 -- 172.21.15.3:0/2790841553 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62b0011600 con 0x5564aa63d7b0 2024-02-09T23:03:34.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62ad7fa640 1 -- 172.21.15.3:0/2790841553 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62b00042b0 con 0x5564aa63d7b0 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 -- 172.21.15.3:0/2790841553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 msgr2=0x5564aa63db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa63db90 secure :-1 s=READY pgs=1727 cs=0 l=1 rev1=1 crypto rx=0x7f62b0011da0 tx=0x7f62b000f790 comp rx=0 tx=0).stop 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 -- 172.21.15.3:0/2790841553 shutdown_connections 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa64d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa64ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/2790841553 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa63db90 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 -- 172.21.15.3:0/2790841553 >> 172.21.15.3:0/2790841553 conn(0x5564aa638100 msgr2=0x5564aa6384e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 -- 172.21.15.3:0/2790841553 shutdown_connections 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.791+0000 7f62bfdd7740 1 -- 172.21.15.3:0/2790841553 wait complete. 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 -- start start 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 Processor -- start 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63d7b0 0x5564aa6483a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6488e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.802 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa648e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa5a5030 con 0x5564aa63e160 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa278430 con 0x5564aa64b370 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa234e00 con 0x5564aa63d7b0 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62bfdd7740 10 client.0 _ll_register_callbacks cb 0x5564aa59a030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62b659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6488e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62b6d9d640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63d7b0 0x5564aa6483a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62b659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6488e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52796/0 (socket says 172.21.15.3:52796) 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62b659c640 1 -- 172.21.15.3:0/679077223 learned_addr learned my addr 172.21.15.3:0/679077223 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.792+0000 7f62b759e640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa648e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b659c640 1 -- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63d7b0 msgr2=0x5564aa6483a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b659c640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63d7b0 0x5564aa6483a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b659c640 1 -- 172.21.15.3:0/679077223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 msgr2=0x5564aa648e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b659c640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa648e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b659c640 1 -- 172.21.15.3:0/679077223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b0019040 con 0x5564aa63e160 2024-02-09T23:03:34.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63d7b0 0x5564aa6483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62b659c640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6488e0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005ab0 tx=0x7f62a802ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98dbf820bfca348 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f629e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f629e7fc640 1 -- 172.21.15.3:0/679077223 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a8004040 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f629e7fc640 1 -- 172.21.15.3:0/679077223 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f62a80041a0 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5564aa2a47e0 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f629e7fc640 1 -- 172.21.15.3:0/679077223 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62a8004360 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62bfdd7740 10 client.19986 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x5564aa5a36b0 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f629e7fc640 1 -- 172.21.15.3:0/679077223 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5564aa2a47e0 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.793+0000 7f629e7fc640 1 -- 172.21.15.3:0/679077223 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f62a8050020 con 0x5564aa63e160 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62bfdd7740 10 client.19986 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62bfdd7740 10 client.19986 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:03:34.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62bfdd7740 20 client.19986 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa65b8f0 0x5564aa65dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62bfdd7740 4 client.19986 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5564aa2e1720 con 0x5564aa65b8f0 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa65b8f0 0x5564aa65dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa65b8f0 0x5564aa65dd10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7eea99c6bc8312f4 server_cookie=183004fe106fc5e2 in_seq=0 out_seq=0 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.794+0000 7f629e7fc640 10 client.19986 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f629e7fc640 1 -- 172.21.15.3:0/679077223 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5564aa2e1720 con 0x5564aa65b8f0 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f629e7fc640 10 client.19986 handle_command_reply: tid=0 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 client.19986 shutdown 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 msgr2=0x5564aa6488e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6488e0 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f62a8005ab0 tx=0x7f62a802ca10 comp rx=0 tx=0).stop 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 shutdown_connections 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa65b8f0 0x5564aa65dd10 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa648e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6488e0 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/679077223 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63d7b0 0x5564aa6483a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 >> 172.21.15.3:0/679077223 conn(0x5564aa638100 msgr2=0x5564aa639e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f629e7fc640 0 client.19986 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 shutdown_connections 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 1 -- 172.21.15.3:0/679077223 wait complete. 2024-02-09T23:03:34.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.795+0000 7f62bfdd7740 20 client.19986 trim_cache size 0 max 16384 2024-02-09T23:03:34.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 Processor -- start 2024-02-09T23:03:34.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 -- start start 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa648c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa639bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa652340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa52a130 con 0x5564aa63d7b0 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa2a4810 con 0x5564aa64b370 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa6379c0 con 0x5564aa63e160 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa648c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62b6d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa648c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52800/0 (socket says 172.21.15.3:52800) 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62b6d9d640 1 -- 172.21.15.3:0/1349685440 learned_addr learned my addr 172.21.15.3:0/1349685440 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62b659c640 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa639bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.807+0000 7f62b759e640 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa652340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62b6d9d640 1 -- 172.21.15.3:0/1349685440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 msgr2=0x5564aa639bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa639bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62b6d9d640 1 -- 172.21.15.3:0/1349685440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 msgr2=0x5564aa652340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa652340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62b6d9d640 1 -- 172.21.15.3:0/1349685440 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5564aa53fc90 con 0x5564aa63d7b0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa648c40 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f62a009ae80 tx=0x7f62a001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec0d9f8334baa21 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62acff9640 1 -- 172.21.15.3:0/1349685440 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a00a0230 con 0x5564aa63d7b0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62acff9640 1 -- 172.21.15.3:0/1349685440 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62a00aee90 con 0x5564aa63d7b0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.808+0000 7f62acff9640 1 -- 172.21.15.3:0/1349685440 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a00a3150 con 0x5564aa63d7b0 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 -- 172.21.15.3:0/1349685440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 msgr2=0x5564aa648c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa648c40 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f62a009ae80 tx=0x7f62a001a040 comp rx=0 tx=0).stop 2024-02-09T23:03:34.817 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 -- 172.21.15.3:0/1349685440 shutdown_connections 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa64b370 0x5564aa652340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa63e160 0x5564aa639bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/1349685440 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63d7b0 0x5564aa648c40 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 -- 172.21.15.3:0/1349685440 >> 172.21.15.3:0/1349685440 conn(0x5564aa638100 msgr2=0x5564aa6394e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 -- 172.21.15.3:0/1349685440 shutdown_connections 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 -- 172.21.15.3:0/1349685440 wait complete. 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 -- start start 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 Processor -- start 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa63d7b0 0x5564aa6ac260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6ac7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.809+0000 7f62bfdd7740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa64b370 0x5564aa651700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa651cc0 con 0x5564aa63e160 2024-02-09T23:03:34.818 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa651e30 con 0x5564aa63d7b0 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5564aa651ff0 con 0x5564aa64b370 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 10 client.0 _ll_register_callbacks cb 0x7f62b0011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6ac7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6ac7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52812/0 (socket says 172.21.15.3:52812) 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 -- 172.21.15.3:0/471401124 learned_addr learned my addr 172.21.15.3:0/471401124 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b759e640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa64b370 0x5564aa651700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa63d7b0 0x5564aa6ac260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 -- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa64b370 msgr2=0x5564aa651700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa64b370 0x5564aa651700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 -- 172.21.15.3:0/471401124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa63d7b0 msgr2=0x5564aa6ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa63d7b0 0x5564aa6ac260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 -- 172.21.15.3:0/471401124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62b81a49a0 con 0x5564aa63e160 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62b659c640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6ac7a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x5564aa647c00 tx=0x7f62a8003c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d78876f5a255b2af server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f629e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f629e7fc640 1 -- 172.21.15.3:0/471401124 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62a8004160 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f629e7fc640 1 -- 172.21.15.3:0/471401124 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f62a8000e40 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f62b0011600 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f629e7fc640 1 -- 172.21.15.3:0/471401124 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f62a80042c0 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 10 client.19992 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.810+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x5564aa565a70 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f629e7fc640 1 -- 172.21.15.3:0/471401124 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f62b0011600 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f629e7fc640 1 -- 172.21.15.3:0/471401124 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f62a803c040 con 0x5564aa63e160 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62bfdd7740 10 client.19992 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62bfdd7740 10 client.19992 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62bfdd7740 20 client.19992 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa6b3e40 0x5564aa6b6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62bfdd7740 4 client.19992 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5564aa330890 con 0x5564aa6b3e40 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa6b3e40 0x5564aa6b6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f629e7fc640 10 client.19992 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.811+0000 7f62b6d9d640 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa6b3e40 0x5564aa6b6260 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9276704671b2e9a8 server_cookie=1c652ed57c3728fe in_seq=0 out_seq=0 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f629e7fc640 1 -- 172.21.15.3:0/471401124 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5564aa330890 con 0x5564aa6b3e40 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f629e7fc640 10 client.19992 handle_command_reply: tid=0 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 client.19992 shutdown 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 msgr2=0x5564aa6ac7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6ac7a0 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x5564aa647c00 tx=0x7f62a8003c90 comp rx=0 tx=0).stop 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 shutdown_connections 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x5564aa6b3e40 0x5564aa6b6260 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5564aa64b370 0x5564aa651700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.821 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5564aa63e160 0x5564aa6ac7a0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f629e7fc640 0 client.19992 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 --2- 172.21.15.3:0/471401124 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5564aa63d7b0 0x5564aa6ac260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 >> 172.21.15.3:0/471401124 conn(0x5564aa638100 msgr2=0x5564aa637240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 shutdown_connections 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 1 -- 172.21.15.3:0/471401124 wait complete. 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bfdd7740 20 client.19992 trim_cache size 0 max 16384 2024-02-09T23:03:34.822 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62a40a2230 msgr2=0x7f62a40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62a40a2230 0x7f62a40a46f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f62b0009fd0 tx=0x7f62b001e000 comp rx=0 tx=0).stop 2024-02-09T23:03:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 msgr2=0x7f62b819d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 0x7f62b819d610 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x7f62a000e9f0 tx=0x7f62a000eeb0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 shutdown_connections 2024-02-09T23:03:34.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f62a40a2230 0x7f62a40a46f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f62b8105410 0x7f62b819db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f62b8197720 0x7f62b8197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 --2- 172.21.15.3:0/2485031974 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f62b8100340 0x7f62b819d610 unknown :-1 s=CLOSED pgs=1726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.812+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 >> 172.21.15.3:0/2485031974 conn(0x7f62b80772e0 msgr2=0x7f62b81040f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.813+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 shutdown_connections 2024-02-09T23:03:34.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.813+0000 7f62bd20b640 1 -- 172.21.15.3:0/2485031974 wait complete. 2024-02-09T23:03:34.826 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:34.826 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:03:34.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 -- 172.21.15.3:0/3072173800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f306410b7b0 msgr2=0x7f306410dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3072173800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f306410b7b0 0x7f306410dc50 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f305c009ca0 tx=0x7f305c02c180 comp rx=0 tx=0).stop 2024-02-09T23:03:34.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 -- 172.21.15.3:0/3072173800 shutdown_connections 2024-02-09T23:03:34.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3072173800 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3064102d90 0x7f306410b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3072173800 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f306410b7b0 0x7f306410dc50 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.937 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3072173800 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 0x7f3064102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 -- 172.21.15.3:0/3072173800 >> 172.21.15.3:0/3072173800 conn(0x7f30640772e0 msgr2=0x7f30640776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 -- 172.21.15.3:0/3072173800 shutdown_connections 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.932+0000 7f306a6fb640 1 -- 172.21.15.3:0/3072173800 wait complete. 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 Processor -- start 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 -- start start 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 0x7f30641000c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 0x7f3064100600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f306410b7b0 0x7f3064100b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641a5c60 con 0x7f3064102d90 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641a5dd0 con 0x7f306410b7b0 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f306a6fb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30641a5f90 con 0x7f3064102470 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 0x7f3064100600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.938 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3063fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 0x7f30641000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 0x7f3064100600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52830/0 (socket says 172.21.15.3:52830) 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3068c98640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f306410b7b0 0x7f3064100b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f305bfff640 1 -- 172.21.15.3:0/3552066254 learned_addr learned my addr 172.21.15.3:0/3552066254 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3068c98640 1 -- 172.21.15.3:0/3552066254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 msgr2=0x7f30641000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3068c98640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 0x7f30641000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3068c98640 1 -- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 msgr2=0x7f3064100600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3068c98640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 0x7f3064100600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3068c98640 1 -- 172.21.15.3:0/3552066254 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c009490 con 0x7f306410b7b0 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f3063fff640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 0x7f30641000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.933+0000 7f305bfff640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 0x7f3064100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.934+0000 7f3068c98640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f306410b7b0 0x7f3064100b40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f305c02c790 tx=0x7f305c0346f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.934+0000 7f3061ffb640 1 -- 172.21.15.3:0/3552066254 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c003d80 con 0x7f306410b7b0 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.934+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f305c009900 con 0x7f306410b7b0 2024-02-09T23:03:34.939 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.934+0000 7f3061ffb640 1 -- 172.21.15.3:0/3552066254 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f305c0348e0 con 0x7f306410b7b0 2024-02-09T23:03:34.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.934+0000 7f3061ffb640 1 -- 172.21.15.3:0/3552066254 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c004790 con 0x7f306410b7b0 2024-02-09T23:03:34.940 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.934+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30641a6550 con 0x7f306410b7b0 2024-02-09T23:03:34.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 Processor -- start 2024-02-09T23:03:34.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f3061ffb640 1 -- 172.21.15.3:0/3552066254 <== mon.2 v2:172.21.15.64:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f305c00a040 con 0x7f306410b7b0 2024-02-09T23:03:34.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 -- start start 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57ea9a30 0x555e57ea9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb2990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 0x555e57eb53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57e65020 con 0x555e57eb2f60 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57e18320 con 0x555e57ea9a30 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f306d3ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57e15850 con 0x555e57eaa3e0 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.936+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 0x555e57eb53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb2990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:56076/0 (socket says 172.21.15.3:56076) 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 0x555e57eb53b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52846/0 (socket says 172.21.15.3:52846) 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 -- 172.21.15.3:0/728210614 learned_addr learned my addr 172.21.15.3:0/728210614 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f305bfff640 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57ea9a30 0x555e57ea9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3061ffb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f30500a2230 0x7f30500a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3068c98640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f30500a2230 0x7f30500a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 -- 172.21.15.3:0/728210614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57ea9a30 msgr2=0x555e57ea9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57ea9a30 0x555e57ea9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 -- 172.21.15.3:0/728210614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 msgr2=0x555e57eb53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 0x555e57eb53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 -- 172.21.15.3:0/728210614 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e57dd3210 con 0x555e57eaa3e0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f305bfff640 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 0x555e57eb53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3061ffb640 1 -- 172.21.15.3:0/3552066254 <== mon.2 v2:172.21.15.64:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f305c0f09e0 con 0x7f306410b7b0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3063fff640 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb2990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f30540115c0 tx=0x7f3054011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f619be9c52b674 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f3068c98640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f30500a2230 0x7f30500a46f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f305c0ee2f0 tx=0x7f305c08f550 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f305a7fc640 1 -- 172.21.15.3:0/728210614 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305400f640 con 0x555e57eaa3e0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f305a7fc640 1 -- 172.21.15.3:0/728210614 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f305400fc80 con 0x555e57eaa3e0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.937+0000 7f305a7fc640 1 -- 172.21.15.3:0/728210614 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054019c40 con 0x555e57eaa3e0 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 -- 172.21.15.3:0/728210614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 msgr2=0x555e57eb2990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb2990 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f30540115c0 tx=0x7f3054011a80 comp rx=0 tx=0).stop 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 -- 172.21.15.3:0/728210614 shutdown_connections 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57eb2f60 0x555e57eb53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb2990 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 --2- 172.21.15.3:0/728210614 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57ea9a30 0x555e57ea9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 -- 172.21.15.3:0/728210614 >> 172.21.15.3:0/728210614 conn(0x555e57ea4100 msgr2=0x555e57ea44e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 -- 172.21.15.3:0/728210614 shutdown_connections 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 -- 172.21.15.3:0/728210614 wait complete. 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.938+0000 7f306d3ae740 1 -- start start 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 Processor -- start 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eaff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57eb0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb09a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57e11030 con 0x555e57ea9a30 2024-02-09T23:03:34.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57ae4430 con 0x555e57eaa3e0 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57aa0e00 con 0x555e57eb2f60 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f306d3ae740 10 client.0 _ll_register_callbacks cb 0x555e57e06030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eaff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eaff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52862/0 (socket says 172.21.15.3:52862) 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 -- 172.21.15.3:0/3949837402 learned_addr learned my addr 172.21.15.3:0/3949837402 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f3063fff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57eb0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb09a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 -- 172.21.15.3:0/3949837402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 msgr2=0x555e57eb09a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb09a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 -- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 msgr2=0x555e57eb0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57eb0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 -- 172.21.15.3:0/3949837402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3054009d40 con 0x555e57ea9a30 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f305bfff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eaff20 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f304c011d80 tx=0x7f304c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e3643f1c07643a9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.950 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f30437fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.939+0000 7f30437fe640 1 -- 172.21.15.3:0/3949837402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c00e5d0 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f30437fe640 1 -- 172.21.15.3:0/3949837402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f304c00e730 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555e57b107e0 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f30437fe640 1 -- 172.21.15.3:0/3949837402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f304c00e8f0 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 10 client.20004 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x555e57e0f6b0 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f30437fe640 1 -- 172.21.15.3:0/3949837402 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x555e57b107e0 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f30437fe640 1 -- 172.21.15.3:0/3949837402 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f304c026070 con 0x555e57ea9a30 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 10 client.20004 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 10 client.20004 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 20 client.20004 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57ec3570 0x555e57ec5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 4 client.20004 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:34.951 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555e57b4d720 con 0x555e57ec3570 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.940+0000 7f3063fff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57ec3570 0x555e57ec5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f30437fe640 10 client.20004 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f3063fff640 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57ec3570 0x555e57ec5990 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4bf46b6e87a1edf3 server_cookie=8d4cb61019592d1b in_seq=0 out_seq=0 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f30437fe640 1 -- 172.21.15.3:0/3949837402 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555e57b4d720 con 0x555e57ec3570 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f30437fe640 10 client.20004 handle_command_reply: tid=0 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 client.20004 shutdown 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 msgr2=0x555e57eaff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eaff20 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f304c011d80 tx=0x7f304c004730 comp rx=0 tx=0).stop 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 shutdown_connections 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57ec3570 0x555e57ec5990 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57eb0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb09a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.952 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f30437fe640 0 client.20004 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:34.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3949837402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eaff20 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 >> 172.21.15.3:0/3949837402 conn(0x555e57ea4100 msgr2=0x555e57ea5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 shutdown_connections 2024-02-09T23:03:34.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.941+0000 7f306d3ae740 1 -- 172.21.15.3:0/3949837402 wait complete. 2024-02-09T23:03:34.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.942+0000 7f306d3ae740 20 client.20004 trim_cache size 0 max 16384 2024-02-09T23:03:34.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.954+0000 7f306d3ae740 1 Processor -- start 2024-02-09T23:03:34.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.954+0000 7f306d3ae740 1 -- start start 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57ea30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f306d3ae740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57d96130 con 0x555e57ea9a30 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f306d3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57ea9380 con 0x555e57eaa3e0 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f306d3ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57e9f870 con 0x555e57eb2f60 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57ea30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52866/0 (socket says 172.21.15.3:52866) 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f305bfff640 1 -- 172.21.15.3:0/3053986216 learned_addr learned my addr 172.21.15.3:0/3053986216 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f305bfff640 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 -- 172.21.15.3:0/3053986216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 msgr2=0x555e57eb9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb9f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 -- 172.21.15.3:0/3053986216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 msgr2=0x555e57eb2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 -- 172.21.15.3:0/3053986216 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555e57dabc90 con 0x555e57eaa3e0 2024-02-09T23:03:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.955+0000 7f3063fff640 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57ea30c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x555e57ea5170 tx=0x7f3054005040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9db645a7dc5787e1 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f3059ffb640 1 -- 172.21.15.3:0/3053986216 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054009be0 con 0x555e57eaa3e0 2024-02-09T23:03:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f3059ffb640 1 -- 172.21.15.3:0/3053986216 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30540198d0 con 0x555e57eaa3e0 2024-02-09T23:03:34.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f3059ffb640 1 -- 172.21.15.3:0/3053986216 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3054005380 con 0x555e57eaa3e0 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f306d3ae740 1 -- 172.21.15.3:0/3053986216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 msgr2=0x555e57ea30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57ea30c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x555e57ea5170 tx=0x7f3054005040 comp rx=0 tx=0).stop 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f306d3ae740 1 -- 172.21.15.3:0/3053986216 shutdown_connections 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.956+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eaa3e0 0x555e57ea30c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eb2f60 0x555e57eb9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3053986216 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- 172.21.15.3:0/3053986216 >> 172.21.15.3:0/3053986216 conn(0x555e57ea4100 msgr2=0x555e57ea29b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- 172.21.15.3:0/3053986216 shutdown_connections 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- 172.21.15.3:0/3053986216 wait complete. 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- start start 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 Processor -- start 2024-02-09T23:03:34.965 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb9480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb99c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eb2f60 0x555e57f12a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57f12f80 con 0x555e57ea9a30 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57f130f0 con 0x555e57eb2f60 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x555e57f132b0 con 0x555e57eaa3e0 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f306d3ae740 10 client.0 _ll_register_callbacks cb 0x7f3054009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f305bfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb9480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52882/0 (socket says 172.21.15.3:52882) 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f3063fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.957+0000 7f305bfff640 1 -- 172.21.15.3:0/3010032802 learned_addr learned my addr 172.21.15.3:0/3010032802 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eb2f60 0x555e57f12a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 -- 172.21.15.3:0/3010032802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 msgr2=0x555e57eb99c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.966 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb99c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 -- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eb2f60 msgr2=0x555e57f12a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eb2f60 0x555e57f12a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 -- 172.21.15.3:0/3010032802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555e57dd3210 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f305bfff640 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb9480 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f304c005730 tx=0x7f304c001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98d300aa268f3d3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f30437fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f30437fe640 1 -- 172.21.15.3:0/3010032802 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c004460 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f30437fe640 1 -- 172.21.15.3:0/3010032802 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f304c011600 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f305400fc80 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f30437fe640 1 -- 172.21.15.3:0/3010032802 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7f304c0045c0 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f306d3ae740 10 client.20010 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x555e57dd1a70 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.958+0000 7f30437fe640 1 -- 172.21.15.3:0/3010032802 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f304c044080 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f30437fe640 1 -- 172.21.15.3:0/3010032802 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f304c028420 con 0x555e57ea9a30 2024-02-09T23:03:34.967 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 10 client.20010 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 10 client.20010 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 20 client.20010 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57f1bd80 0x555e57f1e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 4 client.20010 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x555e57b9c890 con 0x555e57f1bd80 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f3063fff640 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57f1bd80 0x555e57f1e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f30437fe640 10 client.20010 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f3063fff640 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57f1bd80 0x555e57f1e1a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b447ca50594a72e2 server_cookie=b3103b8c912ce0fa in_seq=0 out_seq=0 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f30437fe640 1 -- 172.21.15.3:0/3010032802 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x555e57b9c890 con 0x555e57f1bd80 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f30437fe640 10 client.20010 handle_command_reply: tid=0 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.959+0000 7f306d3ae740 1 client.20010 shutdown 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 msgr2=0x555e57eb9480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.968 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb9480 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f304c005730 tx=0x7f304c001e50 comp rx=0 tx=0).stop 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 shutdown_connections 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x555e57f1bd80 0x555e57f1e1a0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x555e57eb2f60 0x555e57f12a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f30437fe640 0 client.20010 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x555e57eaa3e0 0x555e57eb99c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 --2- 172.21.15.3:0/3010032802 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x555e57ea9a30 0x555e57eb9480 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 >> 172.21.15.3:0/3010032802 conn(0x555e57ea4100 msgr2=0x555e57eb0750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 shutdown_connections 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 1 -- 172.21.15.3:0/3010032802 wait complete. 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306d3ae740 20 client.20010 trim_cache size 0 max 16384 2024-02-09T23:03:34.969 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f30500a2230 msgr2=0x7f30500a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f30500a2230 0x7f30500a46f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f305c0ee2f0 tx=0x7f305c08f550 comp rx=0 tx=0).stop 2024-02-09T23:03:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f306410b7b0 msgr2=0x7f3064100b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f306410b7b0 0x7f3064100b40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f305c02c790 tx=0x7f305c0346f0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.970 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 shutdown_connections 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f30500a2230 0x7f30500a46f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f306410b7b0 0x7f3064100b40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3064102d90 0x7f3064100600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 --2- 172.21.15.3:0/3552066254 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3064102470 0x7f30641000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 >> 172.21.15.3:0/3552066254 conn(0x7f30640772e0 msgr2=0x7f30640fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 shutdown_connections 2024-02-09T23:03:34.971 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:34.960+0000 7f306a6fb640 1 -- 172.21.15.3:0/3552066254 wait complete. 2024-02-09T23:03:34.974 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:03:34.974 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:03:35.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 -- 172.21.15.3:0/1273606554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8100360 msgr2=0x7fa4a8100740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 --2- 172.21.15.3:0/1273606554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8100360 0x7fa4a8100740 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4009cd0 tx=0x7fa4a402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.089 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 -- 172.21.15.3:0/1273606554 shutdown_connections 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 --2- 172.21.15.3:0/1273606554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8105540 0x7fa4a810d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 --2- 172.21.15.3:0/1273606554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a8100d10 0x7fa4a8104e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 --2- 172.21.15.3:0/1273606554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8100360 0x7fa4a8100740 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.084+0000 7fa4af713640 1 -- 172.21.15.3:0/1273606554 >> 172.21.15.3:0/1273606554 conn(0x7fa4a80772a0 msgr2=0x7fa4a80776b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 -- 172.21.15.3:0/1273606554 shutdown_connections 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 -- 172.21.15.3:0/1273606554 wait complete. 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 Processor -- start 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 -- start start 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8100360 0x7fa4a819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 0x7fa4a819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a81977e0 0x7fa4a8197c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.090 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8198290 con 0x7fa4a8105540 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8198400 con 0x7fa4a8100360 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4af713640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4a8198570 con 0x7fa4a81977e0 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4accae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 0x7fa4a819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4adcb0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a81977e0 0x7fa4a8197c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4accae640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 0x7fa4a819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52896/0 (socket says 172.21.15.3:52896) 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.085+0000 7fa4accae640 1 -- 172.21.15.3:0/3707224667 learned_addr learned my addr 172.21.15.3:0/3707224667 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8100360 0x7fa4a819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4accae640 1 -- 172.21.15.3:0/3707224667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a81977e0 msgr2=0x7fa4a8197c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4accae640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a81977e0 0x7fa4a8197c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4accae640 1 -- 172.21.15.3:0/3707224667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8100360 msgr2=0x7fa4a819d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4accae640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8100360 0x7fa4a819d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4accae640 1 -- 172.21.15.3:0/3707224667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a4009900 con 0x7fa4a8105540 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a81977e0 0x7fa4a8197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:35.091 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8100360 0x7fa4a819d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4accae640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 0x7fa4a819dbb0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fa49800e9c0 tx=0x7fa49800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa49e7fc640 1 -- 172.21.15.3:0/3707224667 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498009700 con 0x7fa4a8105540 2024-02-09T23:03:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa49e7fc640 1 -- 172.21.15.3:0/3707224667 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa49800cd80 con 0x7fa4a8105540 2024-02-09T23:03:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa49e7fc640 1 -- 172.21.15.3:0/3707224667 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa498010430 con 0x7fa4a8105540 2024-02-09T23:03:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a81a4520 con 0x7fa4a8105540 2024-02-09T23:03:35.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.086+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a81a49b0 con 0x7fa4a8105540 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 Processor -- start 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa49e7fc640 1 -- 172.21.15.3:0/3707224667 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa498005000 con 0x7fa4a8105540 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 -- start start 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 0x5563f2575e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 0x5563f257e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f25813b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f2531020 con 0x5563f25763e0 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f24e4320 con 0x5563f2575a30 2024-02-09T23:03:35.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f24e1850 con 0x5563f257ef60 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 0x5563f257e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 0x5563f2575e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 0x5563f257e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52904/0 (socket says 172.21.15.3:52904) 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 0x5563f2575e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:52304/0 (socket says 172.21.15.3:52304) 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4ad4af640 1 -- 172.21.15.3:0/4067745072 learned_addr learned my addr 172.21.15.3:0/4067745072 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4accae640 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f25813b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 -- 172.21.15.3:0/4067745072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 msgr2=0x5563f25813b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f25813b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 -- 172.21.15.3:0/4067745072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 msgr2=0x5563f257e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 0x5563f257e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa49e7fc640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4940a2230 0x7fa4940a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4adcb0640 1 -- 172.21.15.3:0/4067745072 --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563f249f210 con 0x5563f2575a30 2024-02-09T23:03:35.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.090+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 0x5563f257e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.091+0000 7fa49e7fc640 1 -- 172.21.15.3:0/3707224667 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa498014070 con 0x7fa4a8105540 2024-02-09T23:03:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.091+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4940a2230 0x7fa4940a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.091+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 0x5563f2575e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009f20 tx=0x7fa4a00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69bdbc8528524227 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.091+0000 7fa4937fe640 1 -- 172.21.15.3:0/4067745072 <== mon.2 v2:172.21.15.64:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a0020030 con 0x5563f2575a30 2024-02-09T23:03:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.091+0000 7fa4937fe640 1 -- 172.21.15.3:0/4067745072 <== mon.2 v2:172.21.15.64:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4a0019b20 con 0x5563f2575a30 2024-02-09T23:03:35.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.091+0000 7fa4937fe640 1 -- 172.21.15.3:0/4067745072 <== mon.2 v2:172.21.15.64:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a000f9a0 con 0x5563f2575a30 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 -- 172.21.15.3:0/4067745072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 msgr2=0x5563f2575e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 0x5563f2575e10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0009f20 tx=0x7fa4a00118d0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4940a2230 0x7fa4940a46f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0020000 tx=0x7fa4a0021000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 -- 172.21.15.3:0/4067745072 shutdown_connections 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f25813b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f25763e0 0x5563f257e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/4067745072 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f2575a30 0x5563f2575e10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 -- 172.21.15.3:0/4067745072 >> 172.21.15.3:0/4067745072 conn(0x5563f2570100 msgr2=0x5563f25704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 -- 172.21.15.3:0/4067745072 shutdown_connections 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 -- 172.21.15.3:0/4067745072 wait complete. 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 -- start start 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.092+0000 7fa4b23c9740 1 Processor -- start 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257bf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f25763e0 0x5563f257c490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f257ef60 0x5563f257c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f24dd030 con 0x5563f2575a30 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f21b0430 con 0x5563f257ef60 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f216ce00 con 0x5563f25763e0 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4b23c9740 10 client.0 _ll_register_callbacks cb 0x5563f24d2030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:35.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257bf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52916/0 (socket says 172.21.15.3:52916) 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1387561087 learned_addr learned my addr 172.21.15.3:0/1387561087 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f257ef60 0x5563f257c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4accae640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f25763e0 0x5563f257c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1387561087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f25763e0 msgr2=0x5563f257c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f25763e0 0x5563f257c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f257ef60 msgr2=0x5563f257c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f257ef60 0x5563f257c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1387561087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a0009bd0 con 0x5563f2575a30 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.093+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f257ef60 0x5563f257c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257bf50 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fa4a402c630 tx=0x7fa4a4034440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1acf7be7ec96fb8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa490ff9640 1 -- 172.21.15.3:0/1387561087 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4034c90 con 0x5563f2575a30 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa490ff9640 1 -- 172.21.15.3:0/1387561087 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa4a4045070 con 0x5563f2575a30 2024-02-09T23:03:35.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5563f21dc7e0 con 0x5563f2575a30 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa490ff9640 1 -- 172.21.15.3:0/1387561087 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4a4034df0 con 0x5563f2575a30 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa4b23c9740 10 client.20022 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x5563f24db6b0 con 0x5563f2575a30 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa490ff9640 1 -- 172.21.15.3:0/1387561087 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5563f21dc7e0 con 0x5563f2575a30 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.094+0000 7fa490ff9640 1 -- 172.21.15.3:0/1387561087 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa4a4004160 con 0x5563f2575a30 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4b23c9740 10 client.20022 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4b23c9740 10 client.20022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4b23c9740 20 client.20022 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f2591930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4b23c9740 4 client.20022 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5563f2219720 con 0x5563f258f540 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4accae640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f2591930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa490ff9640 10 client.20022 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.095+0000 7fa4accae640 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f2591930 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd1d663075cdced1 server_cookie=51ff00a6b10cbe55 in_seq=0 out_seq=0 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa490ff9640 1 -- 172.21.15.3:0/1387561087 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5563f2219720 con 0x5563f258f540 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa490ff9640 10 client.20022 handle_command_reply: tid=0 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 client.20022 shutdown 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 msgr2=0x5563f257bf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257bf50 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7fa4a402c630 tx=0x7fa4a4034440 comp rx=0 tx=0).stop 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 shutdown_connections 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f2591930 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa490ff9640 0 client.20022 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f257ef60 0x5563f257c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f25763e0 0x5563f257c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1387561087 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257bf50 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 >> 172.21.15.3:0/1387561087 conn(0x5563f2570100 msgr2=0x5563f2571e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:35.104 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 shutdown_connections 2024-02-09T23:03:35.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1387561087 wait complete. 2024-02-09T23:03:35.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.096+0000 7fa4b23c9740 20 client.20022 trim_cache size 0 max 16384 2024-02-09T23:03:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 Processor -- start 2024-02-09T23:03:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 -- start start 2024-02-09T23:03:35.114 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f257c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f257a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f256b8b0 con 0x5563f2575a30 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f2462130 con 0x5563f25763e0 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f258c040 con 0x5563f257ef60 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52926/0 (socket says 172.21.15.3:52926) 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1399885129 learned_addr learned my addr 172.21.15.3:0/1399885129 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4accae640 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f257c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f257a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1399885129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 msgr2=0x5563f257a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f257a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.115 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1399885129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 msgr2=0x5563f257c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f257c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.110+0000 7fa4ad4af640 1 -- 172.21.15.3:0/1399885129 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563f2477c90 con 0x5563f2575a30 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.111+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257e680 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40348e0 tx=0x7fa4a402c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=351b41124c2e009f server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.111+0000 7fa492ffd640 1 -- 172.21.15.3:0/1399885129 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4034a90 con 0x5563f2575a30 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.111+0000 7fa492ffd640 1 -- 172.21.15.3:0/1399885129 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4a4043070 con 0x5563f2575a30 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.111+0000 7fa492ffd640 1 -- 172.21.15.3:0/1399885129 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4003d30 con 0x5563f2575a30 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.111+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1399885129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 msgr2=0x5563f257e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.111+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257e680 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7fa4a40348e0 tx=0x7fa4a402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1399885129 shutdown_connections 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f257c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f257ef60 0x5563f257a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/1399885129 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f2575a30 0x5563f257e680 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1399885129 >> 172.21.15.3:0/1399885129 conn(0x5563f2570100 msgr2=0x5563f25704e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:35.116 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1399885129 shutdown_connections 2024-02-09T23:03:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 -- 172.21.15.3:0/1399885129 wait complete. 2024-02-09T23:03:35.117 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 -- start start 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 Processor -- start 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.112+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 0x5563f2584160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f25846a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 0x5563f25e4a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f25e4f90 con 0x5563f257ef60 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f25e50d0 con 0x5563f25763e0 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563f25e5210 con 0x5563f2575a30 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 10 client.0 _ll_register_callbacks cb 0x7fa4a0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4adcb0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 0x5563f25e4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.121 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 0x5563f2584160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 0x5563f2584160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:56110/0 (socket says 172.21.15.3:56110) 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4adcb0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 0x5563f25e4a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52928/0 (socket says 172.21.15.3:52928) 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 -- 172.21.15.3:0/3007440729 learned_addr learned my addr 172.21.15.3:0/3007440729 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4accae640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f25846a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 -- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 msgr2=0x5563f25846a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f25846a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 -- 172.21.15.3:0/3007440729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 msgr2=0x5563f25e4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 0x5563f25e4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 -- 172.21.15.3:0/3007440729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563f24db6b0 con 0x5563f2575a30 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4adcb0640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 0x5563f25e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4ad4af640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 0x5563f2584160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4004570 tx=0x7fa4a4005df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6cb407db8c49ba6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa490ff9640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa490ff9640 1 -- 172.21.15.3:0/3007440729 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4032ac0 con 0x5563f2575a30 2024-02-09T23:03:35.122 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa4a0019b20 con 0x5563f2575a30 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.113+0000 7fa490ff9640 1 -- 172.21.15.3:0/3007440729 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa4a4045040 con 0x5563f2575a30 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa490ff9640 1 -- 172.21.15.3:0/3007440729 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa4a4004b40 con 0x5563f2575a30 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 10 client.29005 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x5563f249da70 con 0x5563f2575a30 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa490ff9640 1 -- 172.21.15.3:0/3007440729 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa4a0019b20 con 0x5563f2575a30 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa490ff9640 1 -- 172.21.15.3:0/3007440729 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa4a4052430 con 0x5563f2575a30 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 10 client.29005 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 10 client.29005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 20 client.29005 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f25e8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 4 client.29005 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5563f2268890 con 0x5563f258f540 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4accae640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f25e8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:35.123 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa4accae640 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f25e8620 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9909bd8738cf8831 server_cookie=7c012e9c7333562e in_seq=0 out_seq=0 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.114+0000 7fa490ff9640 10 client.29005 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa490ff9640 1 -- 172.21.15.3:0/3007440729 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5563f2268890 con 0x5563f258f540 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa490ff9640 10 client.29005 handle_command_reply: tid=0 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 client.29005 shutdown 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 msgr2=0x5563f2584160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 0x5563f2584160 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fa4a4004570 tx=0x7fa4a4005df0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 shutdown_connections 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x5563f258f540 0x5563f25e8620 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x5563f25763e0 0x5563f25846a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x5563f257ef60 0x5563f25e4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa490ff9640 0 client.29005 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 --2- 172.21.15.3:0/3007440729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x5563f2575a30 0x5563f2584160 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.124 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 >> 172.21.15.3:0/3007440729 conn(0x5563f2570100 msgr2=0x5563f257a3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:35.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 shutdown_connections 2024-02-09T23:03:35.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 1 -- 172.21.15.3:0/3007440729 wait complete. 2024-02-09T23:03:35.125 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4b23c9740 20 client.29005 trim_cache size 0 max 16384 2024-02-09T23:03:35.125 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4940a2230 msgr2=0x7fa4940a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4940a2230 0x7fa4940a46f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fa4a0020000 tx=0x7fa4a0021000 comp rx=0 tx=0).stop 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 msgr2=0x7fa4a819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 0x7fa4a819dbb0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7fa49800e9c0 tx=0x7fa49800ee80 comp rx=0 tx=0).stop 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 shutdown_connections 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa4940a2230 0x7fa4940a46f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa4a81977e0 0x7fa4a8197c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.126 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa4a8105540 0x7fa4a819dbb0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 --2- 172.21.15.3:0/3707224667 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa4a8100360 0x7fa4a819d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:35.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.115+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 >> 172.21.15.3:0/3707224667 conn(0x7fa4a80772a0 msgr2=0x7fa4a80fe3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:35.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.116+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 shutdown_connections 2024-02-09T23:03:35.127 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:35.116+0000 7fa4af713640 1 -- 172.21.15.3:0/3707224667 wait complete. 2024-02-09T23:03:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:35 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1226517319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:03:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:35 smithi003 ceph-mon[32678]: pgmap v388: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-02-09T23:03:35.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 463595 3.68 MB/sec execute 569 sec latency 125.892 ms 2024-02-09T23:03:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:35 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1226517319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:03:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:35 smithi064 ceph-mon[36638]: pgmap v388: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-02-09T23:03:35.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:35 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1226517319' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:03:35.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:35 smithi089 ceph-mon[36665]: pgmap v388: 97 pgs: 97 active+clean; 183 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 1.8 MiB/s wr, 174 op/s 2024-02-09T23:03:36.130 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-02-09T23:03:36.130 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:03:36.130 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-02-09T23:03:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 -- 172.21.15.3:0/2113806669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c1053e0 msgr2=0x7fa78c10cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 --2- 172.21.15.3:0/2113806669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c1053e0 0x7fa78c10cfb0 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7fa784009cd0 tx=0x7fa78402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 -- 172.21.15.3:0/2113806669 shutdown_connections 2024-02-09T23:03:36.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 --2- 172.21.15.3:0/2113806669 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa78c100cc0 0x7fa78c104d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 --2- 172.21.15.3:0/2113806669 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c1053e0 0x7fa78c10cfb0 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 --2- 172.21.15.3:0/2113806669 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78c100310 0x7fa78c1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 -- 172.21.15.3:0/2113806669 >> 172.21.15.3:0/2113806669 conn(0x7fa78c077230 msgr2=0x7fa78c077640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 -- 172.21.15.3:0/2113806669 shutdown_connections 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.242+0000 7fa7927d0640 1 -- 172.21.15.3:0/2113806669 wait complete. 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 Processor -- start 2024-02-09T23:03:36.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 -- start start 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa78c100310 0x7fa78c196710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78c1053e0 0x7fa78c196c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 0x7fa78c19b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c19b9c0 con 0x7fa78c19afd0 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c19bb30 con 0x7fa78c100310 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa7927d0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa78c19bca0 con 0x7fa78c1053e0 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 0x7fa78c19b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 0x7fa78c19b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52944/0 (socket says 172.21.15.3:52944) 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 -- 172.21.15.3:0/1881402729 learned_addr learned my addr 172.21.15.3:0/1881402729 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78c1053e0 0x7fa78c196c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa78bfff640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa78c100310 0x7fa78c196710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 -- 172.21.15.3:0/1881402729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78c1053e0 msgr2=0x7fa78c196c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78c1053e0 0x7fa78c196c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 -- 172.21.15.3:0/1881402729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa78c100310 msgr2=0x7fa78c196710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa78c100310 0x7fa78c196710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.243+0000 7fa790d6d640 1 -- 172.21.15.3:0/1881402729 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa784009900 con 0x7fa78c19afd0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.244+0000 7fa790d6d640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 0x7fa78c19b480 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fa784004af0 tx=0x7fa784032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.244+0000 7fa7897fa640 1 -- 172.21.15.3:0/1881402729 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7840328b0 con 0x7fa78c19afd0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.244+0000 7fa7897fa640 1 -- 172.21.15.3:0/1881402729 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa784003e50 con 0x7fa78c19afd0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.244+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa78c1a3280 con 0x7fa78c19afd0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.244+0000 7fa7897fa640 1 -- 172.21.15.3:0/1881402729 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa784004740 con 0x7fa78c19afd0 2024-02-09T23:03:36.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.244+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa78c1a36a0 con 0x7fa78c19afd0 2024-02-09T23:03:36.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.246+0000 7fa7897fa640 1 -- 172.21.15.3:0/1881402729 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa78400a040 con 0x7fa78c19afd0 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.246+0000 7fa7953a3740 1 Processor -- start 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 -- start start 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2ec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f5150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2f7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a25a000 con 0x563f4a2f5690 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a257530 con 0x563f4a2ecab0 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7953a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a252d10 con 0x563f4a2ec190 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7897fa640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa7740a2230 0x7fa7740a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2f7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa790d6d640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa7740a2230 0x7fa7740a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2f7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52960/0 (socket says 172.21.15.3:52960) 2024-02-09T23:03:36.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3578521234 learned_addr learned my addr 172.21.15.3:0/3578521234 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa7897fa640 1 -- 172.21.15.3:0/1881402729 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa7840f0bb0 con 0x7fa78c19afd0 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3578521234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 msgr2=0x563f4a2ec570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2ec570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3578521234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 msgr2=0x563f4a2f5150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f5150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3578521234 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f4a1f9210 con 0x563f4a2f5690 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:36.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa790d6d640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa7740a2230 0x7fa7740a46f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa784009430 tx=0x7fa7840a8040 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.247+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2f7a80 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa780009560 tx=0x7fa780004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=796f45d88ae9bee3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7727fc640 1 -- 172.21.15.3:0/3578521234 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780004050 con 0x563f4a2f5690 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7727fc640 1 -- 172.21.15.3:0/3578521234 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa780005500 con 0x563f4a2f5690 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7727fc640 1 -- 172.21.15.3:0/3578521234 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780004c10 con 0x563f4a2f5690 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 -- 172.21.15.3:0/3578521234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 msgr2=0x563f4a2f7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2f7a80 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa780009560 tx=0x7fa780004730 comp rx=0 tx=0).stop 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 -- 172.21.15.3:0/3578521234 shutdown_connections 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2f7a80 unknown :-1 s=CLOSED pgs=1740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3578521234 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 -- 172.21.15.3:0/3578521234 >> 172.21.15.3:0/3578521234 conn(0x563f4a2e68c0 msgr2=0x563f4a2e6ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 -- 172.21.15.3:0/3578521234 shutdown_connections 2024-02-09T23:03:36.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.248+0000 7fa7953a3740 1 -- 172.21.15.3:0/3578521234 wait complete. 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 -- start start 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 Processor -- start 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 0x563f4a2f2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2f5690 0x563f4a2f3110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a1f7e60 con 0x563f4a2ec190 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f49f264b0 con 0x563f4a2ecab0 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a1d8670 con 0x563f4a2f5690 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa7953a3740 10 client.0 _ll_register_callbacks cb 0x563f4a1f0190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 0x563f4a2f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 0x563f4a2f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52966/0 (socket says 172.21.15.3:52966) 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3670265682 learned_addr learned my addr 172.21.15.3:0/3670265682 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.249+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2f5690 0x563f4a2f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3670265682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2f5690 msgr2=0x563f4a2f3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2f5690 0x563f4a2f3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 msgr2=0x563f4a2f2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3670265682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa780019040 con 0x563f4a2ec190 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 0x563f4a2f2690 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa7800054d0 tx=0x7fa7800048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1223e9261eb02b3e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa76b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3670265682 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780004c60 con 0x563f4a2ec190 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3670265682 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa780000e40 con 0x563f4a2ec190 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563f49f52860 con 0x563f4a2ec190 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3670265682 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa780004dc0 con 0x563f4a2ec190 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa7953a3740 10 client.20040 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:36.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x563f4a251390 con 0x563f4a2ec190 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.250+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3670265682 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563f49f52860 con 0x563f4a2ec190 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3670265682 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa78001f6b0 con 0x563f4a2ec190 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa7953a3740 10 client.20040 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa7953a3740 10 client.20040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa7953a3740 20 client.20040 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a305670 0x563f4a307a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa7953a3740 4 client.20040 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563f49fa3410 con 0x563f4a305670 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a305670 0x563f4a307a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a305670 0x563f4a307a90 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1e4aeaad97bf4106 server_cookie=bb6d20b9054cedeb in_seq=0 out_seq=0 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.251+0000 7fa76b7fe640 10 client.20040 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.252+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3670265682 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563f49fa3410 con 0x563f4a305670 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.252+0000 7fa76b7fe640 10 client.20040 handle_command_reply: tid=0 2024-02-09T23:03:36.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.252+0000 7fa7953a3740 1 client.20040 shutdown 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.252+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 msgr2=0x563f4a2f2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.252+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 0x563f4a2f2690 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa7800054d0 tx=0x7fa7800048b0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 shutdown_connections 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a305670 0x563f4a307a90 crc :-1 s=READY pgs=339 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2f5690 0x563f4a2f3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa76b7fe640 0 client.20040 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3670265682 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2ec190 0x563f4a2f2690 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 >> 172.21.15.3:0/3670265682 conn(0x563f4a2e68c0 msgr2=0x563f4a2e8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 shutdown_connections 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 1 -- 172.21.15.3:0/3670265682 wait complete. 2024-02-09T23:03:36.262 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.253+0000 7fa7953a3740 20 client.20040 trim_cache size 0 max 16384 2024-02-09T23:03:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 Processor -- start 2024-02-09T23:03:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 -- start start 2024-02-09T23:03:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2f4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fc540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a302200 con 0x563f4a2f5690 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a2ebe20 con 0x563f4a2ecab0 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa7953a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a30cd20 con 0x563f4a2ec190 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fc540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52970/0 (socket says 172.21.15.3:52970) 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 -- 172.21.15.3:0/2408874281 learned_addr learned my addr 172.21.15.3:0/2408874281 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2f4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78bfff640 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 -- 172.21.15.3:0/2408874281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 msgr2=0x563f4a2f4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2f4dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 -- 172.21.15.3:0/2408874281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 msgr2=0x563f4a2f2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.268+0000 7fa78b7fe640 1 -- 172.21.15.3:0/2408874281 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563f4a1edc70 con 0x563f4a2f5690 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fc540 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fa7800097c0 tx=0x7fa780005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3178c2c398930d0e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa771ffb640 1 -- 172.21.15.3:0/2408874281 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78001fbe0 con 0x563f4a2f5690 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa771ffb640 1 -- 172.21.15.3:0/2408874281 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa780004c60 con 0x563f4a2f5690 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa771ffb640 1 -- 172.21.15.3:0/2408874281 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa780004430 con 0x563f4a2f5690 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa7953a3740 1 -- 172.21.15.3:0/2408874281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 msgr2=0x563f4a2fc540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa7953a3740 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fc540 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fa7800097c0 tx=0x7fa780005970 comp rx=0 tx=0).stop 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa7953a3740 1 -- 172.21.15.3:0/2408874281 shutdown_connections 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa7953a3740 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a2f2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa7953a3740 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fc540 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.269+0000 7fa7953a3740 1 --2- 172.21.15.3:0/2408874281 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a2f4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- 172.21.15.3:0/2408874281 >> 172.21.15.3:0/2408874281 conn(0x563f4a2e68c0 msgr2=0x563f4a2f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.277 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- 172.21.15.3:0/2408874281 shutdown_connections 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- 172.21.15.3:0/2408874281 wait complete. 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- start start 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 Processor -- start 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a3560c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a356600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fb990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a2fbf50 con 0x563f4a2f5690 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a2fc0c0 con 0x563f4a2ecab0 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x563f4a2fc280 con 0x563f4a2ec190 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa7953a3740 10 client.0 _ll_register_callbacks cb 0x7fa780009a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa78b7fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:52984/0 (socket says 172.21.15.3:52984) 2024-02-09T23:03:36.278 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.270+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3674609455 learned_addr learned my addr 172.21.15.3:0/3674609455 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a3560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a356600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3674609455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 msgr2=0x563f4a3560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a3560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 msgr2=0x563f4a356600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a356600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 -- 172.21.15.3:0/3674609455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563f4a251390 con 0x563f4a2f5690 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa78b7fe640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fb990 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fa78000a010 tx=0x7fa780009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15a38f5673a10a5a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa76b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3674609455 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa78001f7a0 con 0x563f4a2f5690 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa780005500 con 0x563f4a2f5690 2024-02-09T23:03:36.279 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3674609455 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa780000e40 con 0x563f4a2f5690 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3674609455 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=51) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa78000f790 con 0x563f4a2f5690 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa7953a3740 10 client.20046 fetch_fsmap learned FSMap version 51 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=51,monmap=4+}) v3 -- 0x563f4a213ac0 con 0x563f4a2f5690 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.271+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3674609455 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa780005500 con 0x563f4a2f5690 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3674609455 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 51) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa780004590 con 0x563f4a2f5690 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa7953a3740 10 client.20046 fetch_fsmap finished waiting for FSMap version 51 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa7953a3740 10 client.20046 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa7953a3740 20 client.20046 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a35db00 0x563f4a35fef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa7953a3740 4 client.20046 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563f49f8f7b0 con 0x563f4a35db00 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a35db00 0x563f4a35fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.280 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa78bfff640 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a35db00 0x563f4a35fef0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ccf53eecbe2a3d3 server_cookie=c63987dabd29707a in_seq=0 out_seq=0 2024-02-09T23:03:36.281 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.272+0000 7fa76b7fe640 10 client.20046 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.273+0000 7fa76b7fe640 1 -- 172.21.15.3:0/3674609455 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563f49f8f7b0 con 0x563f4a35db00 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.273+0000 7fa76b7fe640 10 client.20046 handle_command_reply: tid=0 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.273+0000 7fa7953a3740 1 client.20046 shutdown 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 msgr2=0x563f4a2fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fb990 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fa78000a010 tx=0x7fa780009f10 comp rx=0 tx=0).stop 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 shutdown_connections 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x563f4a35db00 0x563f4a35fef0 crc :-1 s=READY pgs=340 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x563f4a2ecab0 0x563f4a356600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x563f4a2f5690 0x563f4a2fb990 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.282 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 --2- 172.21.15.3:0/3674609455 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x563f4a2ec190 0x563f4a3560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa76b7fe640 0 client.20046 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 >> 172.21.15.3:0/3674609455 conn(0x563f4a2e68c0 msgr2=0x563f4a2e5920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 shutdown_connections 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 1 -- 172.21.15.3:0/3674609455 wait complete. 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7953a3740 20 client.20046 trim_cache size 0 max 16384 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stdout: "return_code": 0, 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_tag": "f3fd76ec-f52d-4613-bd22-3ceaa8b7f0cb", 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": "asynchronous" 2024-02-09T23:03:36.283 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:03:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa7740a2230 msgr2=0x7fa7740a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa7740a2230 0x7fa7740a46f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fa784009430 tx=0x7fa7840a8040 comp rx=0 tx=0).stop 2024-02-09T23:03:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 msgr2=0x7fa78c19b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.284 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 0x7fa78c19b480 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7fa784004af0 tx=0x7fa784032570 comp rx=0 tx=0).stop 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 shutdown_connections 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa7740a2230 0x7fa7740a46f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa78c19afd0 0x7fa78c19b480 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa78c1053e0 0x7fa78c196c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 --2- 172.21.15.3:0/1881402729 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa78c100310 0x7fa78c196710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 >> 172.21.15.3:0/1881402729 conn(0x7fa78c077230 msgr2=0x7fa78c0fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.274+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 shutdown_connections 2024-02-09T23:03:36.285 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.275+0000 7fa7927d0640 1 -- 172.21.15.3:0/1881402729 wait complete. 2024-02-09T23:03:36.288 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:03:36.288 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:03:36.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 -- 172.21.15.3:0/165091379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 msgr2=0x7f0f5810e0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/165091379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5810e0e0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f0f50009cd0 tx=0x7f0f5002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 -- 172.21.15.3:0/165091379 shutdown_connections 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/165091379 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f58101df0 0x7f0f58105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/165091379 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5810e0e0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/165091379 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f58101440 0x7f0f58101820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 -- 172.21.15.3:0/165091379 >> 172.21.15.3:0/165091379 conn(0x7f0f580fccc0 msgr2=0x7f0f580ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 -- 172.21.15.3:0/165091379 shutdown_connections 2024-02-09T23:03:36.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.399+0000 7f0f5e88f640 1 -- 172.21.15.3:0/165091379 wait complete. 2024-02-09T23:03:36.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.400+0000 7f0f5e88f640 1 Processor -- start 2024-02-09T23:03:36.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.400+0000 7f0f5e88f640 1 -- start start 2024-02-09T23:03:36.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5e88f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f58101440 0x7f0f5819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5e88f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5e88f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f58197780 0x7f0f58197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5e88f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f58198250 con 0x7f0f58106510 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5e88f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f581983c0 con 0x7f0f58197780 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5e88f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f58198580 con 0x7f0f58101440 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5819dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53006/0 (socket says 172.21.15.3:53006) 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 -- 172.21.15.3:0/3957577720 learned_addr learned my addr 172.21.15.3:0/3957577720 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f57fff640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f58101440 0x7f0f5819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f58197780 0x7f0f58197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 -- 172.21.15.3:0/3957577720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f58101440 msgr2=0x7f0f5819d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f58101440 0x7f0f5819d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 -- 172.21.15.3:0/3957577720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f58197780 msgr2=0x7f0f58197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f58197780 0x7f0f58197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 -- 172.21.15.3:0/3957577720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c009490 con 0x7f0f58106510 2024-02-09T23:03:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.401+0000 7f0f577fe640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5819dbb0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00e9a0 tx=0x7f0f4c00ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.402+0000 7f0f557fa640 1 -- 172.21.15.3:0/3957577720 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c00cd80 con 0x7f0f58106510 2024-02-09T23:03:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.402+0000 7f0f557fa640 1 -- 172.21.15.3:0/3957577720 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f4c0035f0 con 0x7f0f58106510 2024-02-09T23:03:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.402+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f50009900 con 0x7f0f58106510 2024-02-09T23:03:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.402+0000 7f0f557fa640 1 -- 172.21.15.3:0/3957577720 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c010690 con 0x7f0f58106510 2024-02-09T23:03:36.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.402+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f581a4870 con 0x7f0f58106510 2024-02-09T23:03:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.404+0000 7f0f557fa640 1 -- 172.21.15.3:0/3957577720 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f0f4c003ad0 con 0x7f0f58106510 2024-02-09T23:03:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.404+0000 7f0f6146f740 1 Processor -- start 2024-02-09T23:03:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.404+0000 7f0f6146f740 1 -- start start 2024-02-09T23:03:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.404+0000 7f0f557fa640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f400a22b0 0x7f0f400a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f400a22b0 0x7f0f400a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f557fa640 1 -- 172.21.15.3:0/3957577720 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0f4c014070 con 0x7f0f58106510 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef45ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef467cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef4682c0 0x560aef46a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f6146f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef415d00 con 0x560aef4682c0 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f6146f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef3ca2f0 con 0x560aef45b1d0 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f6146f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef3c7820 con 0x560aef45a820 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef45ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f5ce2c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef4682c0 0x560aef46a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef45ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:56182/0 (socket says 172.21.15.3:56182) 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef467cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 -- 172.21.15.3:0/2651553530 learned_addr learned my addr 172.21.15.3:0/2651553530 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 -- 172.21.15.3:0/2651553530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 msgr2=0x560aef467cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef467cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 -- 172.21.15.3:0/2651553530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef4682c0 msgr2=0x560aef46a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef4682c0 0x560aef46a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 -- 172.21.15.3:0/2651553530 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560aef384ff0 con 0x560aef45a820 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f400a22b0 0x7f0f400a4770 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0f50005cc0 tx=0x7f0f500325d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.405+0000 7f0f57fff640 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef45ac00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009d20 tx=0x7f0f48011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40c000247793bf6b server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f3e7fc640 1 -- 172.21.15.3:0/2651553530 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48019630 con 0x560aef45a820 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f3e7fc640 1 -- 172.21.15.3:0/2651553530 <== mon.1 v2:172.21.15.89:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f48019c70 con 0x560aef45a820 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f3e7fc640 1 -- 172.21.15.3:0/2651553530 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4800ec00 con 0x560aef45a820 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 -- 172.21.15.3:0/2651553530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 msgr2=0x560aef45ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef45ac00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009d20 tx=0x7f0f48011810 comp rx=0 tx=0).stop 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 -- 172.21.15.3:0/2651553530 shutdown_connections 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef467cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef4682c0 0x560aef46a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2651553530 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef45ac00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 -- 172.21.15.3:0/2651553530 >> 172.21.15.3:0/2651553530 conn(0x560aef455050 msgr2=0x560aef455430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 -- 172.21.15.3:0/2651553530 shutdown_connections 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.406+0000 7f0f6146f740 1 -- 172.21.15.3:0/2651553530 wait complete. 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 -- start start 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 Processor -- start 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef4651b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 0x560aef4656f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef4682c0 0x560aef465c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef3c3000 con 0x560aef45b1d0 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef096430 con 0x560aef4682c0 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef052e00 con 0x560aef45a820 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f6146f740 10 client.0 _ll_register_callbacks cb 0x560aef3b77f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 0x560aef4656f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef4651b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 0x560aef4656f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53032/0 (socket says 172.21.15.3:53032) 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f577fe640 1 -- 172.21.15.3:0/1807788367 learned_addr learned my addr 172.21.15.3:0/1807788367 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.407+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef4682c0 0x560aef465c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f577fe640 1 -- 172.21.15.3:0/1807788367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 msgr2=0x560aef4651b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f577fe640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef4651b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f577fe640 1 -- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef4682c0 msgr2=0x560aef465c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f577fe640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef4682c0 0x560aef465c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f577fe640 1 -- 172.21.15.3:0/1807788367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f48009950 con 0x560aef45b1d0 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef4682c0 0x560aef465c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f577fe640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 0x560aef4656f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c097250 tx=0x7f0f4c003890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e6aafd002f465c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f377fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f377fe640 1 -- 172.21.15.3:0/1807788367 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c0b7070 con 0x560aef45b1d0 2024-02-09T23:03:36.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f377fe640 1 -- 172.21.15.3:0/1807788367 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0f4c0aee40 con 0x560aef45b1d0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560aef0c27e0 con 0x560aef45b1d0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f377fe640 1 -- 172.21.15.3:0/1807788367 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f4c09ef40 con 0x560aef45b1d0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 10 client.20058 fetch_fsmap learned FSMap version 52 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x560aef3c1680 con 0x560aef45b1d0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f377fe640 1 -- 172.21.15.3:0/1807788367 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560aef0c27e0 con 0x560aef45b1d0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f377fe640 1 -- 172.21.15.3:0/1807788367 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 52) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0f4c01f060 con 0x560aef45b1d0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 10 client.20058 fetch_fsmap finished waiting for FSMap version 52 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 10 client.20058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 20 client.20058 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef478770 0x560aef47ab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 4 client.20058 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.408+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560aef0ff720 con 0x560aef478770 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.409+0000 7f0f57fff640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef478770 0x560aef47ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.409+0000 7f0f57fff640 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef478770 0x560aef47ab60 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=389e1bbd45e6baa7 server_cookie=349a6e483b13b3b7 in_seq=0 out_seq=0 2024-02-09T23:03:36.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.409+0000 7f0f377fe640 10 client.20058 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f377fe640 1 -- 172.21.15.3:0/1807788367 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560aef0ff720 con 0x560aef478770 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f377fe640 10 client.20058 handle_command_reply: tid=0 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 client.20058 shutdown 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 msgr2=0x560aef4656f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 0x560aef4656f0 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c097250 tx=0x7f0f4c003890 comp rx=0 tx=0).stop 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 shutdown_connections 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef478770 0x560aef47ab60 crc :-1 s=READY pgs=341 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef4682c0 0x560aef465c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45b1d0 0x560aef4656f0 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 --2- 172.21.15.3:0/1807788367 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef45a820 0x560aef4651b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f377fe640 0 client.20058 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 >> 172.21.15.3:0/1807788367 conn(0x560aef455050 msgr2=0x560aef456da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 shutdown_connections 2024-02-09T23:03:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.410+0000 7f0f6146f740 1 -- 172.21.15.3:0/1807788367 wait complete. 2024-02-09T23:03:36.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.411+0000 7f0f6146f740 20 client.20058 trim_cache size 0 max 16384 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 Processor -- start 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 -- start start 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef4678e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef465a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef462430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef347fd0 con 0x560aef45a820 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef45a480 con 0x560aef45b1d0 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f6146f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef460dc0 con 0x560aef4682c0 2024-02-09T23:03:36.430 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef4678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef4678e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53040/0 (socket says 172.21.15.3:53040) 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.426+0000 7f0f57fff640 1 -- 172.21.15.3:0/2139446324 learned_addr learned my addr 172.21.15.3:0/2139446324 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef462430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f577fe640 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef465a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f57fff640 1 -- 172.21.15.3:0/2139446324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 msgr2=0x560aef462430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f57fff640 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef462430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f57fff640 1 -- 172.21.15.3:0/2139446324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 msgr2=0x560aef465a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f57fff640 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef465a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f57fff640 1 -- 172.21.15.3:0/2139446324 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560aef35db30 con 0x560aef45a820 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f57fff640 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef4678e0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009c40 tx=0x7f0f48005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4872dc903812c289 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f3dffb640 1 -- 172.21.15.3:0/2139446324 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48009ed0 con 0x560aef45a820 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f3dffb640 1 -- 172.21.15.3:0/2139446324 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f4800edd0 con 0x560aef45a820 2024-02-09T23:03:36.431 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.427+0000 7f0f3dffb640 1 -- 172.21.15.3:0/2139446324 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f48019690 con 0x560aef45a820 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 -- 172.21.15.3:0/2139446324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 msgr2=0x560aef4678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef4678e0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f0f48009c40 tx=0x7f0f48005f90 comp rx=0 tx=0).stop 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 -- 172.21.15.3:0/2139446324 shutdown_connections 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef465a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef462430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 --2- 172.21.15.3:0/2139446324 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef4678e0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 -- 172.21.15.3:0/2139446324 >> 172.21.15.3:0/2139446324 conn(0x560aef455050 msgr2=0x560aef47f4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 -- 172.21.15.3:0/2139446324 shutdown_connections 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 -- 172.21.15.3:0/2139446324 wait complete. 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 -- start start 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.428+0000 7f0f6146f740 1 Processor -- start 2024-02-09T23:03:36.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef46dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef46e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef46e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef4cec50 con 0x560aef45a820 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef4ced90 con 0x560aef45b1d0 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x560aef4ceed0 con 0x560aef4682c0 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 10 client.0 _ll_register_callbacks cb 0x7f0f48009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef46dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef46dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:53044/0 (socket says 172.21.15.3:53044) 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 -- 172.21.15.3:0/3121987691 learned_addr learned my addr 172.21.15.3:0/3121987691 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f577fe640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef46e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f5ce2c640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef46e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 -- 172.21.15.3:0/3121987691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 msgr2=0x560aef46e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef46e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 -- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 msgr2=0x560aef46e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef46e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 -- 172.21.15.3:0/3121987691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560aef384ff0 con 0x560aef45a820 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f5ce2c640 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef46e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f57fff640 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef46dcd0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0f48011d10 tx=0x7f0f4800efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd9a78637a49fa0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f377fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f377fe640 1 -- 172.21.15.3:0/3121987691 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f480052b0 con 0x560aef45a820 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f377fe640 1 -- 172.21.15.3:0/3121987691 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f0f48005a70 con 0x560aef45a820 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.429+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0f48019c70 con 0x560aef45a820 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f377fe640 1 -- 172.21.15.3:0/3121987691 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0f48009ed0 con 0x560aef45a820 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 10 client.20064 fetch_fsmap learned FSMap version 52 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x560aef383850 con 0x560aef45a820 2024-02-09T23:03:36.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f377fe640 1 -- 172.21.15.3:0/3121987691 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0f48019c70 con 0x560aef45a820 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f377fe640 1 -- 172.21.15.3:0/3121987691 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 52) v1 ==== 7894+0+0 (secure 0 0 0) 0x7f0f4803f070 con 0x560aef45a820 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 10 client.20064 fetch_fsmap finished waiting for FSMap version 52 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 10 client.20064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 20 client.20064 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef4d02a0 0x560aef4d2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 4 client.20064 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560aef14e890 con 0x560aef4d02a0 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.430+0000 7f0f577fe640 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef4d02a0 0x560aef4d2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f577fe640 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef4d02a0 0x560aef4d2690 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac076ba7863e9291 server_cookie=674aacd41208237a in_seq=0 out_seq=0 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f377fe640 10 client.20064 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f377fe640 1 -- 172.21.15.3:0/3121987691 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560aef14e890 con 0x560aef4d02a0 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f377fe640 10 client.20064 handle_command_reply: tid=0 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 client.20064 shutdown 2024-02-09T23:03:36.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 msgr2=0x560aef46dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef46dcd0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f0f48011d10 tx=0x7f0f4800efd0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 shutdown_connections 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x560aef4d02a0 0x560aef4d2690 crc :-1 s=READY pgs=342 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x560aef45b1d0 0x560aef46e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x560aef4682c0 0x560aef46e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f377fe640 0 client.20064 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 --2- 172.21.15.3:0/3121987691 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x560aef45a820 0x560aef46dcd0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 >> 172.21.15.3:0/3121987691 conn(0x560aef455050 msgr2=0x560aef453960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 shutdown_connections 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 1 -- 172.21.15.3:0/3121987691 wait complete. 2024-02-09T23:03:36.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.431+0000 7f0f6146f740 20 client.20064 trim_cache size 0 max 16384 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "IDLE (waiting for more scrubs)", 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": { 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: "f3fd76ec-f52d-4613-bd22-3ceaa8b7f0cb": { 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: "path": "/", 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: "tag": "f3fd76ec-f52d-4613-bd22-3ceaa8b7f0cb", 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: "options": "recursive,force" 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:03:36.441 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:03:36.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f400a22b0 msgr2=0x7f0f400a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f400a22b0 0x7f0f400a4770 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0f50005cc0 tx=0x7f0f500325d0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 msgr2=0x7f0f5819dbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5819dbb0 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c00e9a0 tx=0x7f0f4c00ee60 comp rx=0 tx=0).stop 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 shutdown_connections 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f0f400a22b0 0x7f0f400a4770 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0f58197780 0x7f0f58197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0f58106510 0x7f0f5819dbb0 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 --2- 172.21.15.3:0/3957577720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f0f58101440 0x7f0f5819d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 >> 172.21.15.3:0/3957577720 conn(0x7f0f580fccc0 msgr2=0x7f0f580ff010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 shutdown_connections 2024-02-09T23:03:36.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:03:36.432+0000 7f0f5e88f640 1 -- 172.21.15.3:0/3957577720 wait complete. 2024-02-09T23:03:36.446 INFO:tasks.cephfs.filesystem:scrub status for tag:f3fd76ec-f52d-4613-bd22-3ceaa8b7f0cb - {'path': '/', 'tag': 'f3fd76ec-f52d-4613-bd22-3ceaa8b7f0cb', 'options': 'recursive,force'} 2024-02-09T23:03:36.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 464456 3.67 MB/sec execute 570 sec latency 783.608 ms 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub queued for path: / 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: active paths [/] 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:37.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.634 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:37 smithi003 ceph-mon[32678]: pgmap v389: 97 pgs: 97 active+clean; 181 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-02-09T23:03:37.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 465264 3.68 MB/sec execute 571 sec latency 594.762 ms 2024-02-09T23:03:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub queued for path: / 2024-02-09T23:03:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: active paths [/] 2024-02-09T23:03:37.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:37.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.687 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:37 smithi064 ceph-mon[36638]: pgmap v389: 97 pgs: 97 active+clean; 181 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-02-09T23:03:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub queued for path: / 2024-02-09T23:03:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.728 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: active paths [/] 2024-02-09T23:03:37.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:37.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: scrub summary: idle+waiting paths [/] 2024-02-09T23:03:37.729 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:37 smithi089 ceph-mon[36665]: pgmap v389: 97 pgs: 97 active+clean; 181 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 MiB/s rd, 2.7 MiB/s wr, 295 op/s 2024-02-09T23:03:38.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 465264 3.67 MB/sec execute 572 sec latency 1594.816 ms 2024-02-09T23:03:39.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 465264 3.67 MB/sec execute 573 sec latency 2594.933 ms 2024-02-09T23:03:39.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:39 smithi003 ceph-mon[32678]: pgmap v390: 97 pgs: 97 active+clean; 181 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 183 op/s 2024-02-09T23:03:39.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:39 smithi064 ceph-mon[36638]: pgmap v390: 97 pgs: 97 active+clean; 181 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 183 op/s 2024-02-09T23:03:39.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:39 smithi089 ceph-mon[36665]: pgmap v390: 97 pgs: 97 active+clean; 181 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 1.6 MiB/s wr, 183 op/s 2024-02-09T23:03:40.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 465264 3.66 MB/sec execute 574 sec latency 3595.039 ms 2024-02-09T23:03:41.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 465264 3.65 MB/sec execute 575 sec latency 4595.138 ms 2024-02-09T23:03:41.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:41 smithi003 ceph-mon[32678]: pgmap v391: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 223 op/s 2024-02-09T23:03:41.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:41 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:41 smithi064 ceph-mon[36638]: pgmap v391: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 223 op/s 2024-02-09T23:03:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:41 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:41 smithi089 ceph-mon[36665]: pgmap v391: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 223 op/s 2024-02-09T23:03:41.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:41 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:42.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 466581 3.66 MB/sec execute 576 sec latency 4621.009 ms 2024-02-09T23:03:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:42.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:42 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:42.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:42 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:42.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:42 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:43.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 468439 3.67 MB/sec execute 577 sec latency 77.022 ms 2024-02-09T23:03:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:43 smithi003 ceph-mon[32678]: scrub summary: idle 2024-02-09T23:03:43.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:43 smithi003 ceph-mon[32678]: pgmap v392: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-02-09T23:03:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:43 smithi064 ceph-mon[36638]: scrub summary: idle 2024-02-09T23:03:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:43 smithi064 ceph-mon[36638]: pgmap v392: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-02-09T23:03:43.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:43 smithi089 ceph-mon[36665]: scrub summary: idle 2024-02-09T23:03:43.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:43 smithi089 ceph-mon[36665]: pgmap v392: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-02-09T23:03:44.668 INFO:tasks.workunit.client.0.smithi003.stdout: 1 469795 3.68 MB/sec execute 578 sec latency 16.947 ms 2024-02-09T23:03:45.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:45 smithi003 ceph-mon[32678]: pgmap v393: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-02-09T23:03:45.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 470685 3.68 MB/sec execute 579 sec latency 513.409 ms 2024-02-09T23:03:45.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:45 smithi064 ceph-mon[36638]: pgmap v393: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-02-09T23:03:45.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:45 smithi089 ceph-mon[36665]: pgmap v393: 97 pgs: 97 active+clean; 188 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 1.7 MiB/s wr, 161 op/s 2024-02-09T23:03:46.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:46 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:46.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 470685 3.67 MB/sec execute 580 sec latency 1513.466 ms 2024-02-09T23:03:46.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:46 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:46.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:46 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:46.780 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 -- 172.21.15.3:0/783007821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 msgr2=0x7fcf9810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/783007821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf9810cfe0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009cd0 tx=0x7fcf9002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 -- 172.21.15.3:0/783007821 shutdown_connections 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/783007821 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf98100cf0 0x7fcf98104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/783007821 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf9810cfe0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/783007821 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf98100340 0x7fcf98100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.777+0000 7fcf9ec23640 1 -- 172.21.15.3:0/783007821 >> 172.21.15.3:0/783007821 conn(0x7fcf98077220 msgr2=0x7fcf98077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 -- 172.21.15.3:0/783007821 shutdown_connections 2024-02-09T23:03:46.781 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 -- 172.21.15.3:0/783007821 wait complete. 2024-02-09T23:03:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 Processor -- start 2024-02-09T23:03:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 -- start start 2024-02-09T23:03:46.783 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf98100340 0x7fcf98198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf98198c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf981991f0 0x7fcf981a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981a3e70 con 0x7fcf98105410 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981a3fe0 con 0x7fcf98100340 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.778+0000 7fcf9ec23640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf981a41a0 con 0x7fcf981991f0 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf98198c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf98198c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42858/0 (socket says 172.21.15.3:42858) 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 -- 172.21.15.3:0/3293930844 learned_addr learned my addr 172.21.15.3:0/3293930844 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf981991f0 0x7fcf981a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf98100340 0x7fcf98198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 -- 172.21.15.3:0/3293930844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf981991f0 msgr2=0x7fcf981a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf981991f0 0x7fcf981a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.784 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 -- 172.21.15.3:0/3293930844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf98100340 msgr2=0x7fcf98198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf98100340 0x7fcf98198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 -- 172.21.15.3:0/3293930844 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf90009900 con 0x7fcf98105410 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf981991f0 0x7fcf981a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf97fff640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf98198c60 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e9c0 tx=0x7fcf8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf95ffb640 1 -- 172.21.15.3:0/3293930844 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c009700 con 0x7fcf98105410 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf95ffb640 1 -- 172.21.15.3:0/3293930844 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8c00cd80 con 0x7fcf98105410 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.779+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf981a4480 con 0x7fcf98105410 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.780+0000 7fcf95ffb640 1 -- 172.21.15.3:0/3293930844 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c010430 con 0x7fcf98105410 2024-02-09T23:03:46.785 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.780+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf981a4990 con 0x7fcf98105410 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.782+0000 7fcf95ffb640 1 -- 172.21.15.3:0/3293930844 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fcf8c005000 con 0x7fcf98105410 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.782+0000 7fcf95ffb640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf800a2220 0x7fcf800a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.782+0000 7fcfa17f8740 1 Processor -- start 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.782+0000 7fcfa17f8740 1 -- start start 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.782+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf800a2220 0x7fcf800a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.782+0000 7fcf95ffb640 1 -- 172.21.15.3:0/3293930844 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fcf8c014070 con 0x7fcf98105410 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284afb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 0x557284b04670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b06fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284a65a00 con 0x557284afbde0 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284a62f30 con 0x557284afb4c0 2024-02-09T23:03:46.791 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284a5e710 con 0x557284b04bb0 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 0x557284b04670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 0x557284b04670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42862/0 (socket says 172.21.15.3:42862) 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf9c9bf640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284afb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 -- 172.21.15.3:0/1420398859 learned_addr learned my addr 172.21.15.3:0/1420398859 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b06fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf800a2220 0x7fcf800a46e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009c20 tx=0x7fcf88005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 -- 172.21.15.3:0/1420398859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 msgr2=0x557284b06fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b06fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 -- 172.21.15.3:0/1420398859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 msgr2=0x557284afb8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284afb8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 -- 172.21.15.3:0/1420398859 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557284a233b0 con 0x557284afbde0 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b06fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:46.792 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284afb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.783+0000 7fcf97fff640 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 0x557284b04670 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c097270 tx=0x7fcf8c09ad10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab3da9fe0dd2da43 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcf7effd640 1 -- 172.21.15.3:0/1420398859 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c0a35b0 con 0x557284afbde0 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcf7effd640 1 -- 172.21.15.3:0/1420398859 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf8c09d030 con 0x557284afbde0 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcf7effd640 1 -- 172.21.15.3:0/1420398859 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf8c005510 con 0x557284afbde0 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1420398859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 msgr2=0x557284b04670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 0x557284b04670 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c097270 tx=0x7fcf8c09ad10 comp rx=0 tx=0).stop 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1420398859 shutdown_connections 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b06fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afbde0 0x557284b04670 unknown :-1 s=CLOSED pgs=1751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1420398859 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284afb8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1420398859 >> 172.21.15.3:0/1420398859 conn(0x557284af5de0 msgr2=0x557284af61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1420398859 shutdown_connections 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.784+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1420398859 wait complete. 2024-02-09T23:03:46.793 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 -- start start 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 Processor -- start 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afb4c0 0x557284b01c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b02180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b026c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284a03b90 con 0x557284afb4c0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284732250 con 0x557284afbde0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5572849e4200 con 0x557284b04bb0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.785+0000 7fcfa17f8740 10 client.0 _ll_register_callbacks cb 0x557284a1a5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9c9bf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afb4c0 0x557284b01c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b026c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf97fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b02180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf97fff640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b02180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.64:3300/0 says I am v2:172.21.15.3:48786/0 (socket says 172.21.15.3:48786) 2024-02-09T23:03:46.794 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b026c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:33508/0 (socket says 172.21.15.3:33508) 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf97fff640 1 -- 172.21.15.3:0/1729056983 learned_addr learned my addr 172.21.15.3:0/1729056983 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 msgr2=0x557284b02180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b02180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1729056983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afb4c0 msgr2=0x557284b01c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afb4c0 0x557284b01c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1729056983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf981a4990 con 0x557284b04bb0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afb4c0 0x557284b01c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b026c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcf9002c630 tx=0x7fcf90034a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=210b8d2c7694f7e9 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf77fff640 10 client.0 ms_handle_connect on v2:172.21.15.89:3300/0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf77fff640 1 -- 172.21.15.3:0/1729056983 <== mon.1 v2:172.21.15.89:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900328b0 con 0x557284b04bb0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.786+0000 7fcf77fff640 1 -- 172.21.15.3:0/1729056983 <== mon.1 v2:172.21.15.89:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fcf90046070 con 0x557284b04bb0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x557284a84ee0 con 0x557284b04bb0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcf77fff640 1 -- 172.21.15.3:0/1729056983 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf90032a10 con 0x557284b04bb0 2024-02-09T23:03:46.795 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 10 client.29050 fetch_fsmap learned FSMap version 54 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) v3 -- 0x557284a5cd90 con 0x557284b04bb0 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcf77fff640 1 -- 172.21.15.3:0/1729056983 <== mon.1 v2:172.21.15.89:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x557284a84ee0 con 0x557284b04bb0 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcf77fff640 1 -- 172.21.15.3:0/1729056983 <== mon.1 v2:172.21.15.89:3300/0 5 ==== fsmap(e 54) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fcf900046c0 con 0x557284b04bb0 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 10 client.29050 fetch_fsmap finished waiting for FSMap version 54 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 10 client.29050 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 20 client.29050 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b14e70 0x557284b17290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 4 client.29050 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.787+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55728475e600 con 0x557284b14e70 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.788+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b14e70 0x557284b17290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.788+0000 7fcf77fff640 10 client.29050 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:46.796 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.788+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b14e70 0x557284b17290 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f48d04d65a40f67d server_cookie=45ce59b0458c4b13 in_seq=0 out_seq=0 2024-02-09T23:03:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcf77fff640 1 -- 172.21.15.3:0/1729056983 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55728475e600 con 0x557284b14e70 2024-02-09T23:03:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcf77fff640 10 client.29050 handle_command_reply: tid=0 2024-02-09T23:03:46.797 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 client.29050 shutdown 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 msgr2=0x557284b026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b026c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcf9002c630 tx=0x7fcf90034a90 comp rx=0 tx=0).stop 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 shutdown_connections 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b14e70 0x557284b17290 crc :-1 s=READY pgs=343 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b02180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcf77fff640 0 client.29050 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284b04bb0 0x557284b026c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1729056983 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284afb4c0 0x557284b01c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 >> 172.21.15.3:0/1729056983 conn(0x557284af5de0 msgr2=0x557284af7b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 shutdown_connections 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.789+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1729056983 wait complete. 2024-02-09T23:03:46.798 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.790+0000 7fcfa17f8740 20 client.29050 trim_cache size 0 max 16384 2024-02-09T23:03:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 Processor -- start 2024-02-09T23:03:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 -- start start 2024-02-09T23:03:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284b04370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afbde0 0x557284afcf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.808 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284afd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284aed110 con 0x557284b04bb0 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284af6830 con 0x557284afb4c0 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284af56b0 con 0x557284afbde0 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284afd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284afd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42876/0 (socket says 172.21.15.3:42876) 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1001739119 learned_addr learned my addr 172.21.15.3:0/1001739119 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284b04370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf97fff640 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afbde0 0x557284afcf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1001739119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afbde0 msgr2=0x557284afcf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afbde0 0x557284afcf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1001739119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 msgr2=0x557284b04370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284b04370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.809 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/1001739119 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5572849f9820 con 0x557284b04bb0 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf97fff640 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afbde0 0x557284afcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.804+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284afd490 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fcf9002c770 tx=0x7fcf90032d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6988e90c90d6779c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcf7e7fc640 1 -- 172.21.15.3:0/1001739119 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf900328b0 con 0x557284b04bb0 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcf7e7fc640 1 -- 172.21.15.3:0/1001739119 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf90044070 con 0x557284b04bb0 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcf7e7fc640 1 -- 172.21.15.3:0/1001739119 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90003d70 con 0x557284b04bb0 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1001739119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 msgr2=0x557284afd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284afd490 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fcf9002c770 tx=0x7fcf90032d70 comp rx=0 tx=0).stop 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1001739119 shutdown_connections 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284afd490 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afbde0 0x557284afcf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/1001739119 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afb4c0 0x557284b04370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1001739119 >> 172.21.15.3:0/1001739119 conn(0x557284af5de0 msgr2=0x557284afc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1001739119 shutdown_connections 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.805+0000 7fcfa17f8740 1 -- 172.21.15.3:0/1001739119 wait complete. 2024-02-09T23:03:46.810 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 -- start start 2024-02-09T23:03:46.811 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 Processor -- start 2024-02-09T23:03:46.813 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afb4c0 0x557284b0bc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b09fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284b0a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284b0abe0 con 0x557284b04bb0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284b0ad50 con 0x557284afbde0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x557284b6b380 con 0x557284afb4c0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcfa17f8740 10 client.0 _ll_register_callbacks cb 0x7fcf88005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284b0a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcf9c9bf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afb4c0 0x557284b0bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcf9d1c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284b0a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:42886/0 (socket says 172.21.15.3:42886) 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/978619439 learned_addr learned my addr 172.21.15.3:0/978619439 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcf9c9bf640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afb4c0 0x557284b0bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.89:3300/0 says I am v2:172.21.15.3:33540/0 (socket says 172.21.15.3:33540) 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.806+0000 7fcf97fff640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b09fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/978619439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afb4c0 msgr2=0x557284b0bc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.814 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afb4c0 0x557284b0bc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 msgr2=0x557284b09fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b09fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf9d1c0640 1 -- 172.21.15.3:0/978619439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557284a5cd90 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf97fff640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b09fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf9d1c0640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284b0a4e0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fcf9002c700 tx=0x7fcf90032b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ae2737ee6901f69 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf77fff640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf77fff640 1 -- 172.21.15.3:0/978619439 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90053bc0 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf77fff640 1 -- 172.21.15.3:0/978619439 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fcf90046040 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf8c09d030 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcf77fff640 1 -- 172.21.15.3:0/978619439 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=54) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf90034a90 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcfa17f8740 10 client.20082 fetch_fsmap learned FSMap version 54 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.807+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=54,monmap=4+}) v3 -- 0x557284a1f6f0 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcf77fff640 1 -- 172.21.15.3:0/978619439 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf9006d020 con 0x557284b04bb0 2024-02-09T23:03:46.815 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcf77fff640 1 -- 172.21.15.3:0/978619439 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 54) v1 ==== 7894+0+0 (secure 0 0 0) 0x7fcf9005c410 con 0x557284b04bb0 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcfa17f8740 10 client.20082 fetch_fsmap finished waiting for FSMap version 54 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcfa17f8740 10 client.20082 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcfa17f8740 20 client.20082 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b6c550 0x557284b6e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcfa17f8740 4 client.20082 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5572847af1b0 con 0x557284b6c550 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b6c550 0x557284b6e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.808+0000 7fcf77fff640 10 client.20082 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:46.816 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.809+0000 7fcf9c9bf640 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b6c550 0x557284b6e970 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=134eaa42d5ddd5d3 server_cookie=a32a8e6af1305ed7 in_seq=0 out_seq=0 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcf77fff640 1 -- 172.21.15.3:0/978619439 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5572847af1b0 con 0x557284b6c550 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcf77fff640 10 client.20082 handle_command_reply: tid=0 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 client.20082 shutdown 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 msgr2=0x557284b0a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284b0a4e0 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fcf9002c700 tx=0x7fcf90032b10 comp rx=0 tx=0).stop 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 shutdown_connections 2024-02-09T23:03:46.859 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x557284b6c550 0x557284b6e970 crc :-1 s=READY pgs=344 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x557284afbde0 0x557284b09fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x557284b04bb0 0x557284b0a4e0 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcf77fff640 0 client.20082 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 --2- 172.21.15.3:0/978619439 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x557284afb4c0 0x557284b0bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 >> 172.21.15.3:0/978619439 conn(0x557284af5de0 msgr2=0x557284afcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 shutdown_connections 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 1 -- 172.21.15.3:0/978619439 wait complete. 2024-02-09T23:03:46.860 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.855+0000 7fcfa17f8740 20 client.20082 trim_cache size 0 max 16384 2024-02-09T23:03:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:03:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:03:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:03:46.861 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf800a2220 msgr2=0x7fcf800a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf800a2220 0x7fcf800a46e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fcf88009c20 tx=0x7fcf88005f90 comp rx=0 tx=0).stop 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 msgr2=0x7fcf98198c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf98198c60 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fcf8c00e9c0 tx=0x7fcf8c00ee80 comp rx=0 tx=0).stop 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 shutdown_connections 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fcf800a2220 0x7fcf800a46e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fcf981991f0 0x7fcf981a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fcf98105410 0x7fcf98198c60 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.862 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 --2- 172.21.15.3:0/3293930844 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fcf98100340 0x7fcf98198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 >> 172.21.15.3:0/3293930844 conn(0x7fcf98077220 msgr2=0x7fcf980fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 shutdown_connections 2024-02-09T23:03:46.863 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:46.856+0000 7fcf9ec23640 1 -- 172.21.15.3:0/3293930844 wait complete. 2024-02-09T23:03:47.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:47 smithi003 ceph-mon[32678]: pgmap v394: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 255 op/s 2024-02-09T23:03:47.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 472787 3.69 MB/sec execute 581 sec latency 1550.266 ms 2024-02-09T23:03:47.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:47 smithi064 ceph-mon[36638]: pgmap v394: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 255 op/s 2024-02-09T23:03:47.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:47 smithi089 ceph-mon[36665]: pgmap v394: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 255 op/s 2024-02-09T23:03:48.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 474148 3.69 MB/sec execute 582 sec latency 12.494 ms 2024-02-09T23:03:49.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 474231 3.69 MB/sec execute 583 sec latency 943.364 ms 2024-02-09T23:03:49.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:49 smithi003 ceph-mon[32678]: pgmap v395: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.8 MiB/s wr, 134 op/s 2024-02-09T23:03:49.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:49 smithi064 ceph-mon[36638]: pgmap v395: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.8 MiB/s wr, 134 op/s 2024-02-09T23:03:49.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:49 smithi089 ceph-mon[36665]: pgmap v395: 97 pgs: 97 active+clean; 186 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 1.8 MiB/s wr, 134 op/s 2024-02-09T23:03:50.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 475525 3.69 MB/sec execute 584 sec latency 1382.765 ms 2024-02-09T23:03:51.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 477230 3.70 MB/sec execute 585 sec latency 13.320 ms 2024-02-09T23:03:51.778 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:51 smithi089 ceph-mon[36665]: pgmap v396: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 2.5 MiB/s wr, 205 op/s 2024-02-09T23:03:51.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:51 smithi003 ceph-mon[32678]: pgmap v396: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 2.5 MiB/s wr, 205 op/s 2024-02-09T23:03:51.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:51 smithi064 ceph-mon[36638]: pgmap v396: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 2.5 MiB/s wr, 205 op/s 2024-02-09T23:03:52.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 477780 3.70 MB/sec execute 586 sec latency 709.428 ms 2024-02-09T23:03:53.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 479941 3.72 MB/sec execute 587 sec latency 830.280 ms 2024-02-09T23:03:53.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:53 smithi003 ceph-mon[32678]: pgmap v397: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-02-09T23:03:53.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:53 smithi064 ceph-mon[36638]: pgmap v397: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-02-09T23:03:53.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:53 smithi089 ceph-mon[36665]: pgmap v397: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-02-09T23:03:54.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 481360 3.72 MB/sec execute 588 sec latency 63.653 ms 2024-02-09T23:03:55.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:55 smithi003 ceph-mon[32678]: pgmap v398: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-02-09T23:03:55.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 481360 3.71 MB/sec execute 589 sec latency 1063.739 ms 2024-02-09T23:03:55.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:55 smithi064 ceph-mon[36638]: pgmap v398: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-02-09T23:03:55.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:55 smithi089 ceph-mon[36665]: pgmap v398: 97 pgs: 97 active+clean; 189 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 690 KiB/s rd, 1.7 MiB/s wr, 164 op/s 2024-02-09T23:03:56.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:56 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:56.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 481360 3.71 MB/sec execute 590 sec latency 2063.774 ms 2024-02-09T23:03:56.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:56 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:56.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:56 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:03:57.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:57 smithi003 ceph-mon[32678]: pgmap v399: 97 pgs: 97 active+clean; 193 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 260 op/s 2024-02-09T23:03:57.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 483487 3.72 MB/sec execute 591 sec latency 2091.109 ms 2024-02-09T23:03:57.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:57 smithi064 ceph-mon[36638]: pgmap v399: 97 pgs: 97 active+clean; 193 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 260 op/s 2024-02-09T23:03:57.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:57 smithi089 ceph-mon[36665]: pgmap v399: 97 pgs: 97 active+clean; 193 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.7 MiB/s wr, 260 op/s 2024-02-09T23:03:58.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 484911 3.73 MB/sec execute 592 sec latency 93.790 ms 2024-02-09T23:03:59.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 484911 3.72 MB/sec execute 593 sec latency 1093.903 ms 2024-02-09T23:03:59.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:03:59 smithi003 ceph-mon[32678]: pgmap v400: 97 pgs: 97 active+clean; 193 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-02-09T23:03:59.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:03:59 smithi064 ceph-mon[36638]: pgmap v400: 97 pgs: 97 active+clean; 193 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-02-09T23:03:59.958 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 -- 172.21.15.3:0/3054517501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d80ffae0 msgr2=0x7f43d80fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 --2- 172.21.15.3:0/3054517501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d80ffae0 0x7f43d80fff60 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f43c8009930 tx=0x7f43c802bcd0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 -- 172.21.15.3:0/3054517501 shutdown_connections 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 --2- 172.21.15.3:0/3054517501 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f43d81004a0 0x7f43d810e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 --2- 172.21.15.3:0/3054517501 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d80ffae0 0x7f43d80fff60 unknown :-1 s=CLOSED pgs=1754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 --2- 172.21.15.3:0/3054517501 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d8107420 0x7f43d8107800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.953+0000 7f43dff02640 1 -- 172.21.15.3:0/3054517501 >> 172.21.15.3:0/3054517501 conn(0x7f43d80753d0 msgr2=0x7f43d80757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 -- 172.21.15.3:0/3054517501 shutdown_connections 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 -- 172.21.15.3:0/3054517501 wait complete. 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 Processor -- start 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 -- start start 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80ffae0 0x7f43d8100cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 0x7f43d8101200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.959 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f43d8107420 0x7f43d8101740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81a5c50 con 0x7f43d81004a0 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81a5dc0 con 0x7f43d8107420 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dff02640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43d81a5f80 con 0x7f43d80ffae0 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.954+0000 7f43dd49d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 0x7f43d8101200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 0x7f43d8101200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54158/0 (socket says 172.21.15.3:54158) 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 -- 172.21.15.3:0/390442988 learned_addr learned my addr 172.21.15.3:0/390442988 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80ffae0 0x7f43d8100cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43de49f640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f43d8107420 0x7f43d8101740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 -- 172.21.15.3:0/390442988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80ffae0 msgr2=0x7f43d8100cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80ffae0 0x7f43d8100cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 -- 172.21.15.3:0/390442988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f43d8107420 msgr2=0x7f43d8101740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:59.960 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f43d8107420 0x7f43d8101740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 -- 172.21.15.3:0/390442988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c8009560 con 0x7f43d81004a0 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dd49d640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 0x7f43d8101200 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005940 tx=0x7f43c802c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43ceffd640 1 -- 172.21.15.3:0/390442988 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8004350 con 0x7f43d81004a0 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43ceffd640 1 -- 172.21.15.3:0/390442988 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43c80044b0 con 0x7f43d81004a0 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43d81a6200 con 0x7f43d81004a0 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43ceffd640 1 -- 172.21.15.3:0/390442988 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c8003d70 con 0x7f43d81004a0 2024-02-09T23:03:59.961 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.955+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d81a6710 con 0x7f43d81004a0 2024-02-09T23:03:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.957+0000 7f43ceffd640 1 -- 172.21.15.3:0/390442988 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f43c8004df0 con 0x7f43d81004a0 2024-02-09T23:03:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 Processor -- start 2024-02-09T23:03:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 -- start start 2024-02-09T23:03:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f969193490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919c260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.968 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f96919eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43ceffd640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f43c40a2250 0x7f43c40a4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f969101a00 con 0x55f96919c7a0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9690fef30 con 0x55f9691939d0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9690fa710 con 0x55f9691930b0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43dd49d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f96919eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43de49f640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f43c40a2250 0x7f43c40a4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43dd49d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f96919eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54168/0 (socket says 172.21.15.3:54168) 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43dd49d640 1 -- 172.21.15.3:0/483937572 learned_addr learned my addr 172.21.15.3:0/483937572 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.958+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43de49f640 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f969193490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43ceffd640 1 -- 172.21.15.3:0/390442988 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f43c80f0670 con 0x7f43d81004a0 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43dd49d640 1 -- 172.21.15.3:0/483937572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 msgr2=0x55f969193490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.969 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43dd49d640 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f969193490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43dd49d640 1 -- 172.21.15.3:0/483937572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 msgr2=0x55f96919c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43dd49d640 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43dd49d640 1 -- 172.21.15.3:0/483937572 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9690bf3b0 con 0x55f96919c7a0 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43de49f640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f43c40a2250 0x7f43c40a4710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f43d0005fd0 tx=0x7f43d0009de0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43dd49d640 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f96919eb90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005ec0 tx=0x7f43c802bcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8df3e4905678606c server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43c3fff640 1 -- 172.21.15.3:0/483937572 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80bca70 con 0x55f96919c7a0 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43c3fff640 1 -- 172.21.15.3:0/483937572 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43c80bd810 con 0x55f96919c7a0 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.959+0000 7f43c3fff640 1 -- 172.21.15.3:0/483937572 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43c80b9780 con 0x55f96919c7a0 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 -- 172.21.15.3:0/483937572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 msgr2=0x55f96919eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f96919eb90 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005ec0 tx=0x7f43c802bcd0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 -- 172.21.15.3:0/483937572 shutdown_connections 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.970 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f96919eb90 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/483937572 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f969193490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 -- 172.21.15.3:0/483937572 >> 172.21.15.3:0/483937572 conn(0x55f96918d9d0 msgr2=0x55f96918ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 -- 172.21.15.3:0/483937572 shutdown_connections 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 -- 172.21.15.3:0/483937572 wait complete. 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.960+0000 7f43e2bb4740 1 -- start start 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 Processor -- start 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 0x55f9691997f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f969199d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f96919c7a0 0x55f96919a270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f96909fb90 con 0x55f9691930b0 2024-02-09T23:03:59.971 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f968dce250 con 0x55f9691939d0 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f969080200 con 0x55f96919c7a0 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43e2bb4740 10 client.0 _ll_register_callbacks cb 0x55f9690b65d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 0x55f9691997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 0x55f9691997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54172/0 (socket says 172.21.15.3:54172) 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 -- 172.21.15.3:0/3225802732 learned_addr learned my addr 172.21.15.3:0/3225802732 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43dd49d640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f969199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43de49f640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f96919c7a0 0x55f96919a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 -- 172.21.15.3:0/3225802732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f96919c7a0 msgr2=0x55f96919a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f96919c7a0 0x55f96919a270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 -- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 msgr2=0x55f969199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f969199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.972 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.961+0000 7f43ddc9e640 1 -- 172.21.15.3:0/3225802732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d81a6710 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 0x55f9691997f0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f43d4012710 tx=0x7f43d4012bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb8de7fdb4f3589 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43c17fa640 1 -- 172.21.15.3:0/3225802732 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d4017070 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43c17fa640 1 -- 172.21.15.3:0/3225802732 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f43d400f920 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f969120ee0 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43c17fa640 1 -- 172.21.15.3:0/3225802732 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43d40122e0 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 10 client.20094 fetch_fsmap learned FSMap version 55 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55f9690f8d90 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43c17fa640 1 -- 172.21.15.3:0/3225802732 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43d4028080 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43c17fa640 1 -- 172.21.15.3:0/3225802732 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f43d4022030 con 0x55f9691930b0 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 10 client.20094 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 10 client.20094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 20 client.20094 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:59.973 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9691ac7f0 0x55f9691aec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 4 client.20094 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.962+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f968dfa600 con 0x55f9691ac7f0 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.963+0000 7f43dd49d640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9691ac7f0 0x55f9691aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.963+0000 7f43dd49d640 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9691ac7f0 0x55f9691aec30 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bd8fb00152dc50 server_cookie=630c87a88b0c1e0d in_seq=0 out_seq=0 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.963+0000 7f43c17fa640 10 client.20094 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43c17fa640 1 -- 172.21.15.3:0/3225802732 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f968dfa600 con 0x55f9691ac7f0 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43c17fa640 10 client.20094 handle_command_reply: tid=0 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 client.20094 shutdown 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 msgr2=0x55f9691997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 0x55f9691997f0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7f43d4012710 tx=0x7f43d4012bd0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 shutdown_connections 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9691ac7f0 0x55f9691aec30 crc :-1 s=READY pgs=345 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.974 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f969199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f96919c7a0 0x55f96919a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/3225802732 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f9691930b0 0x55f9691997f0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43c17fa640 0 client.20094 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 >> 172.21.15.3:0/3225802732 conn(0x55f96918d9d0 msgr2=0x55f96918f720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 shutdown_connections 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 1 -- 172.21.15.3:0/3225802732 wait complete. 2024-02-09T23:03:59.975 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.964+0000 7f43e2bb4740 20 client.20094 trim_cache size 0 max 16384 2024-02-09T23:03:59.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:03:59 smithi089 ceph-mon[36665]: pgmap v400: 97 pgs: 97 active+clean; 193 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 755 KiB/s rd, 1.7 MiB/s wr, 165 op/s 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 Processor -- start 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 -- start start 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f96919bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691980c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f96918d500 con 0x55f96919c7a0 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f96919eb90 con 0x55f9691939d0 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9691b3e60 con 0x55f9691930b0 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.985 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691980c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54176/0 (socket says 172.21.15.3:54176) 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 -- 172.21.15.3:0/1204294370 learned_addr learned my addr 172.21.15.3:0/1204294370 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43dd49d640 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f96919bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 -- 172.21.15.3:0/1204294370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 msgr2=0x55f96919bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f96919bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 -- 172.21.15.3:0/1204294370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 msgr2=0x55f96919a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 -- 172.21.15.3:0/1204294370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f969095820 con 0x55f96919c7a0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f96919bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.978+0000 7f43de49f640 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691980c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f43d0012620 tx=0x7f43d001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f7937abe0c10fc server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43c37fe640 1 -- 172.21.15.3:0/1204294370 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0004b20 con 0x55f96919c7a0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43c37fe640 1 -- 172.21.15.3:0/1204294370 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43d0025070 con 0x55f96919c7a0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43c37fe640 1 -- 172.21.15.3:0/1204294370 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d00047b0 con 0x55f96919c7a0 2024-02-09T23:03:59.986 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 -- 172.21.15.3:0/1204294370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 msgr2=0x55f9691980c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691980c0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f43d0012620 tx=0x7f43d001f000 comp rx=0 tx=0).stop 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 -- 172.21.15.3:0/1204294370 shutdown_connections 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691939d0 0x55f96919a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691980c0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/1204294370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691930b0 0x55f96919bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 -- 172.21.15.3:0/1204294370 >> 172.21.15.3:0/1204294370 conn(0x55f96918d9d0 msgr2=0x55f969194ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 -- 172.21.15.3:0/1204294370 shutdown_connections 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.979+0000 7f43e2bb4740 1 -- 172.21.15.3:0/1204294370 wait complete. 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 -- start start 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 Processor -- start 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691930b0 0x55f9691a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691939d0 0x55f9691a1e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691a23b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f969202db0 con 0x55f96919c7a0 2024-02-09T23:03:59.987 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f969202f20 con 0x55f9691930b0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9692030e0 con 0x55f9691939d0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43e2bb4740 10 client.0 _ll_register_callbacks cb 0x7f43d4009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43de49f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691a23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43de49f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691a23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54180/0 (socket says 172.21.15.3:54180) 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43de49f640 1 -- 172.21.15.3:0/2040762116 learned_addr learned my addr 172.21.15.3:0/2040762116 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691930b0 0x55f9691a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.980+0000 7f43dd49d640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691939d0 0x55f9691a1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43de49f640 1 -- 172.21.15.3:0/2040762116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691939d0 msgr2=0x55f9691a1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43de49f640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691939d0 0x55f9691a1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43de49f640 1 -- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691930b0 msgr2=0x55f9691a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43de49f640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691930b0 0x55f9691a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43de49f640 1 -- 172.21.15.3:0/2040762116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43d001a040 con 0x55f96919c7a0 2024-02-09T23:03:59.988 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43de49f640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691a23b0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f43d0004a80 tx=0x7f43d0004190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a96c5a116b78495 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43c17fa640 1 -- 172.21.15.3:0/2040762116 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43d0004d30 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43c17fa640 1 -- 172.21.15.3:0/2040762116 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f43d0004530 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f43c80bd810 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43c17fa640 1 -- 172.21.15.3:0/2040762116 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f43d00046f0 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 10 client.20100 fetch_fsmap learned FSMap version 55 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55f9690bb6f0 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43c17fa640 1 -- 172.21.15.3:0/2040762116 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f43c80bd810 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43c17fa640 1 -- 172.21.15.3:0/2040762116 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f43d0005650 con 0x55f96919c7a0 2024-02-09T23:03:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 10 client.20100 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 10 client.20100 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 20 client.20100 populate_metadata read hostname 'smithi003' 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9692047a0 0x55f969206bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 4 client.20100 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.981+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f968e4b1b0 con 0x55f9692047a0 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.982+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9692047a0 0x55f969206bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.982+0000 7f43ddc9e640 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9692047a0 0x55f969206bc0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b55505686c5ad0e6 server_cookie=d9d842aea2a64388 in_seq=0 out_seq=0 2024-02-09T23:03:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:03:59.982+0000 7f43c17fa640 10 client.20100 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:00.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.031+0000 7f43c17fa640 1 -- 172.21.15.3:0/2040762116 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f968e4b1b0 con 0x55f9692047a0 2024-02-09T23:04:00.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.031+0000 7f43c17fa640 10 client.20100 handle_command_reply: tid=0 2024-02-09T23:04:00.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.031+0000 7f43e2bb4740 1 client.20100 shutdown 2024-02-09T23:04:00.035 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 msgr2=0x55f9691a23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691a23b0 secure :-1 s=READY pgs=1759 cs=0 l=1 rev1=1 crypto rx=0x7f43d0004a80 tx=0x7f43d0004190 comp rx=0 tx=0).stop 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 shutdown_connections 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55f9692047a0 0x55f969206bc0 crc :-1 s=READY pgs=346 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55f96919c7a0 0x55f9691a23b0 unknown :-1 s=CLOSED pgs=1759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55f9691939d0 0x55f9691a1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43c17fa640 0 client.20100 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 --2- 172.21.15.3:0/2040762116 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55f9691930b0 0x55f9691a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 >> 172.21.15.3:0/2040762116 conn(0x55f96918d9d0 msgr2=0x55f969198050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 shutdown_connections 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 1 -- 172.21.15.3:0/2040762116 wait complete. 2024-02-09T23:04:00.036 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43e2bb4740 20 client.20100 trim_cache size 0 max 16384 2024-02-09T23:04:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:{ 2024-02-09T23:04:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "message": "", 2024-02-09T23:04:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout: "return_code": 0 2024-02-09T23:04:00.037 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stdout:} 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f43c40a2250 msgr2=0x7f43c40a4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43dff02640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f43c40a2250 0x7f43c40a4710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f43d0005fd0 tx=0x7f43d0009de0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 msgr2=0x7f43d8101200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.032+0000 7f43dff02640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 0x7f43d8101200 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f43c8005940 tx=0x7f43c802c760 comp rx=0 tx=0).stop 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 shutdown_connections 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f43c40a2250 0x7f43c40a4710 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f43d8107420 0x7f43d8101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f43d81004a0 0x7f43d8101200 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 --2- 172.21.15.3:0/390442988 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f43d80ffae0 0x7f43d8100cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 >> 172.21.15.3:0/390442988 conn(0x7f43d80753d0 msgr2=0x7f43d80fe7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 shutdown_connections 2024-02-09T23:04:00.038 INFO:teuthology.task.background_exec.ubuntu@smithi003.front.sepia.ceph.com.smithi003.stderr:2024-02-09T23:04:00.033+0000 7f43dff02640 1 -- 172.21.15.3:0/390442988 wait complete. 2024-02-09T23:04:00.669 INFO:tasks.workunit.client.0.smithi003.stdout: 1 486450 3.73 MB/sec execute 594 sec latency 1444.448 ms 2024-02-09T23:04:01.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:04:01.233 INFO:tasks.workunit.client.0.smithi003.stdout:[486931] write failed on handle 11658 (Resource temporarily unavailable) 2024-02-09T23:04:01.233 INFO:tasks.workunit.client.0.smithi003.stdout:Child failed with status 1 2024-02-09T23:04:01.233 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-02-09T23:04:01.234 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-02-09T23:04:01.551 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:01 smithi003 ceph-mon[32678]: pgmap v401: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 243 op/s 2024-02-09T23:04:01.552 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-02-09T23:04:01.813 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=52cff9affa5048c38ff62349d63dfe6b Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_ceph_ceph-c_290212a1bfd33563ba76ee5b7f0b944fdb6c5d90/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_d9fdb2209e15b39d9f061fd85399f352ce0f0894/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh' 2024-02-09T23:04:01.816 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-02-09T23:04:01.826 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T23:04:01.841 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-02-09T23:04:01.868 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-02-09T23:04:01.868 INFO:teuthology.task.exec:Executing custom commands... 2024-02-09T23:04:01.868 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T23:04:01.868 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-02-09T23:04:01.911 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-02-09T23:04:01.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:01 smithi064 ceph-mon[36638]: pgmap v401: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 243 op/s 2024-02-09T23:04:01.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:01 smithi089 ceph-mon[36665]: pgmap v401: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 243 op/s 2024-02-09T23:04:01.981 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-02-09T23:04:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.124+0000 7f334fa4e640 1 -- 172.21.15.3:0/4047608125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348068d40 msgr2=0x7f3348069120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.124+0000 7f334fa4e640 1 --2- 172.21.15.3:0/4047608125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348068d40 0x7f3348069120 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7f3344009930 tx=0x7f334402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 -- 172.21.15.3:0/4047608125 shutdown_connections 2024-02-09T23:04:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 --2- 172.21.15.3:0/4047608125 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f33480696f0 0x7f3348105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 --2- 172.21.15.3:0/4047608125 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3348106510 0x7f334810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.130 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 --2- 172.21.15.3:0/4047608125 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348068d40 0x7f3348069120 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 -- 172.21.15.3:0/4047608125 >> 172.21.15.3:0/4047608125 conn(0x7f3348077220 msgr2=0x7f3348077630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 -- 172.21.15.3:0/4047608125 shutdown_connections 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 -- 172.21.15.3:0/4047608125 wait complete. 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 Processor -- start 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 -- start start 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3348068d40 0x7f33481000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3348106510 0x7f3348100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.131 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.125+0000 7f334fa4e640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 0x7f33481a5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334fa4e640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a6060 con 0x7f3348100b70 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334fa4e640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a61d0 con 0x7f3348068d40 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334fa4e640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33481a6390 con 0x7f3348106510 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 0x7f33481a5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 0x7f33481a5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54196/0 (socket says 172.21.15.3:54196) 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 -- 172.21.15.3:0/1881683997 learned_addr learned my addr 172.21.15.3:0/1881683997 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334d7ea640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3348068d40 0x7f33481000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.132 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334cfe9640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3348106510 0x7f3348100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 -- 172.21.15.3:0/1881683997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3348106510 msgr2=0x7f3348100630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3348106510 0x7f3348100630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 -- 172.21.15.3:0/1881683997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3348068d40 msgr2=0x7f33481000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3348068d40 0x7f33481000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 -- 172.21.15.3:0/1881683997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3344009560 con 0x7f3348100b70 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334d7ea640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3348068d40 0x7f33481000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334dfeb640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 0x7f33481a5a90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f334000cad0 tx=0x7f334000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334000ae20 con 0x7f3348100b70 2024-02-09T23:04:02.133 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3340005020 con 0x7f3348100b70 2024-02-09T23:04:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33481a6670 con 0x7f3348100b70 2024-02-09T23:04:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334000f6b0 con 0x7f3348100b70 2024-02-09T23:04:02.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.126+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33481a6bd0 con 0x7f3348100b70 2024-02-09T23:04:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.127+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334804f7e0 con 0x7f3348100b70 2024-02-09T23:04:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.128+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f3340005520 con 0x7f3348100b70 2024-02-09T23:04:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.129+0000 7f333e7fc640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f33340a22d0 0x7f33340a4790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.129+0000 7f334d7ea640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f33340a22d0 0x7f33340a4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.129+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f33400ce3f0 con 0x7f3348100b70 2024-02-09T23:04:02.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.129+0000 7f334d7ea640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f33340a22d0 0x7f33340a4790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3344006010 tx=0x7f3344004880 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:02.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.133+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f3340097020 con 0x7f3348100b70 2024-02-09T23:04:02.343 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.340+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 --> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f33480696f0 con 0x7f33340a22d0 2024-02-09T23:04:02.351 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.348+0000 7f333e7fc640 1 -- 172.21.15.3:0/1881683997 <== mgr.15519 v2:172.21.15.3:6800/4119047166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f33480696f0 con 0x7f33340a22d0 2024-02-09T23:04:02.351 INFO:teuthology.orchestra.run.smithi003.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-02-09T00:00:00", "created": "2024-02-09T22:51:15", "first": "2024-02-09T22:52:00", "last": "2024-02-09T23:04:00", "last_pruned": "2024-02-09T23:04:00", "created_count": 13, "pruned_count": 7, "active": true} 2024-02-09T23:04:02.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f33340a22d0 msgr2=0x7f33340a4790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.353 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f33340a22d0 0x7f33340a4790 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f3344006010 tx=0x7f3344004880 comp rx=0 tx=0).stop 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 msgr2=0x7f33481a5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 0x7f33481a5a90 secure :-1 s=READY pgs=1761 cs=0 l=1 rev1=1 crypto rx=0x7f334000cad0 tx=0x7f334000cf90 comp rx=0 tx=0).stop 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 shutdown_connections 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f33340a22d0 0x7f33340a4790 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f3348100b70 0x7f33481a5a90 unknown :-1 s=CLOSED pgs=1761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f3348106510 0x7f3348100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 --2- 172.21.15.3:0/1881683997 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f3348068d40 0x7f33481000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.350+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 >> 172.21.15.3:0/1881683997 conn(0x7f3348077220 msgr2=0x7f33480fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:02.354 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.351+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 shutdown_connections 2024-02-09T23:04:02.355 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.351+0000 7f334fa4e640 1 -- 172.21.15.3:0/1881683997 wait complete. 2024-02-09T23:04:02.364 DEBUG:teuthology.orchestra.run.smithi003:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-02-09T23:04:02.471 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 -- 172.21.15.3:0/3766317895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0100370 msgr2=0x7f92d0100750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 --2- 172.21.15.3:0/3766317895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0100370 0x7f92d0100750 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7f92cc009930 tx=0x7f92cc02bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 -- 172.21.15.3:0/3766317895 shutdown_connections 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 --2- 172.21.15.3:0/3766317895 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d0105440 0x7f92d010d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 --2- 172.21.15.3:0/3766317895 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0100d20 0x7f92d0104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 --2- 172.21.15.3:0/3766317895 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0100370 0x7f92d0100750 unknown :-1 s=CLOSED pgs=1762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 -- 172.21.15.3:0/3766317895 >> 172.21.15.3:0/3766317895 conn(0x7f92d0077310 msgr2=0x7f92d0077720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 -- 172.21.15.3:0/3766317895 shutdown_connections 2024-02-09T23:04:02.472 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.466+0000 7f92d77e2640 1 -- 172.21.15.3:0/3766317895 wait complete. 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 Processor -- start 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 -- start start 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d0100370 0x7f92d019d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 0x7f92d019dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0197880 0x7f92d0197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d0198270 con 0x7f92d0105440 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d01983e0 con 0x7f92d0100370 2024-02-09T23:04:02.473 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d77e2640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d01985a0 con 0x7f92d0197880 2024-02-09T23:04:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d4d7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 0x7f92d019dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d4d7d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 0x7f92d019dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:54218/0 (socket says 172.21.15.3:54218) 2024-02-09T23:04:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d4d7d640 1 -- 172.21.15.3:0/2156282917 learned_addr learned my addr 172.21.15.3:0/2156282917 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.467+0000 7f92d557e640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d0100370 0x7f92d019d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.474 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d4d7d640 1 -- 172.21.15.3:0/2156282917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0197880 msgr2=0x7f92d0197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d4d7d640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0197880 0x7f92d0197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d4d7d640 1 -- 172.21.15.3:0/2156282917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d0100370 msgr2=0x7f92d019d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d4d7d640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d0100370 0x7f92d019d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d4d7d640 1 -- 172.21.15.3:0/2156282917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc009560 con 0x7f92d0105440 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d4d7d640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 0x7f92d019dc10 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f92c000b450 tx=0x7f92c000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c0011890 con 0x7f92d0105440 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92c00103e0 con 0x7f92d0105440 2024-02-09T23:04:02.475 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d01a4590 con 0x7f92d0105440 2024-02-09T23:04:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c000f580 con 0x7f92d0105440 2024-02-09T23:04:02.476 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.468+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d01a4a70 con 0x7f92d0105440 2024-02-09T23:04:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.469+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d0101840 con 0x7f92d0105440 2024-02-09T23:04:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.471+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f92c00108e0 con 0x7f92d0105440 2024-02-09T23:04:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.471+0000 7f92c67fc640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f92bc0a22a0 0x7f92bc0a4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.472+0000 7f92d557e640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f92bc0a22a0 0x7f92bc0a4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.472+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f92c00ce0c0 con 0x7f92d0105440 2024-02-09T23:04:02.477 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.472+0000 7f92d557e640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f92bc0a22a0 0x7f92bc0a4760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f92cc005ab0 tx=0x7f92cc005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:02.478 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.475+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f92c0096f00 con 0x7f92d0105440 2024-02-09T23:04:02.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.747+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 --> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f92d0100d20 con 0x7f92bc0a22a0 2024-02-09T23:04:02.756 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.753+0000 7f92c67fc640 1 -- 172.21.15.3:0/2156282917 <== mgr.15519 v2:172.21.15.3:6800/4119047166 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f92d0100d20 con 0x7f92bc0a22a0 2024-02-09T23:04:02.756 INFO:teuthology.orchestra.run.smithi003.stdout:/ 1m 6m3h 2024-02-09T23:04:02.758 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f92bc0a22a0 msgr2=0x7f92bc0a4760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f92bc0a22a0 0x7f92bc0a4760 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f92cc005ab0 tx=0x7f92cc005fd0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 msgr2=0x7f92d019dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 0x7f92d019dc10 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7f92c000b450 tx=0x7f92c000b910 comp rx=0 tx=0).stop 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 shutdown_connections 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f92bc0a22a0 0x7f92bc0a4760 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f92d0197880 0x7f92d0197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.755+0000 7f92d77e2640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f92d0105440 0x7f92d019dc10 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.759 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.756+0000 7f92d77e2640 1 --2- 172.21.15.3:0/2156282917 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f92d0100370 0x7f92d019d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.756+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 >> 172.21.15.3:0/2156282917 conn(0x7f92d0077310 msgr2=0x7f92d0069cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.756+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 shutdown_connections 2024-02-09T23:04:02.760 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:02.756+0000 7f92d77e2640 1 -- 172.21.15.3:0/2156282917 wait complete. 2024-02-09T23:04:02.769 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-02-09T23:04:02.780 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-02-09T23:04:03.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:03 smithi003 ceph-mon[32678]: from='client.20106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T23:04:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:03 smithi003 ceph-mon[32678]: pgmap v402: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-02-09T23:04:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:03 smithi003 ceph-mon[32678]: osdmap e105: 12 total, 12 up, 12 in 2024-02-09T23:04:03.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:03 smithi003 ceph-mon[32678]: from='client.20112 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T23:04:03.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:03 smithi064 ceph-mon[36638]: from='client.20106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T23:04:03.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:03 smithi064 ceph-mon[36638]: pgmap v402: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-02-09T23:04:03.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:03 smithi064 ceph-mon[36638]: osdmap e105: 12 total, 12 up, 12 in 2024-02-09T23:04:03.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:03 smithi064 ceph-mon[36638]: from='client.20112 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T23:04:03.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:03 smithi089 ceph-mon[36665]: from='client.20106 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T23:04:03.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:03 smithi089 ceph-mon[36665]: pgmap v402: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 1.7 MiB/s wr, 173 op/s 2024-02-09T23:04:03.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:03 smithi089 ceph-mon[36665]: osdmap e105: 12 total, 12 up, 12 in 2024-02-09T23:04:03.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:03 smithi089 ceph-mon[36665]: from='client.20112 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-02-09T23:04:05.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:05 smithi003 ceph-mon[32678]: pgmap v404: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-02-09T23:04:05.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:05 smithi064 ceph-mon[36638]: pgmap v404: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-02-09T23:04:05.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:05 smithi089 ceph-mon[36665]: pgmap v404: 97 pgs: 97 active+clean; 190 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 933 KiB/s rd, 2.1 MiB/s wr, 207 op/s 2024-02-09T23:04:06.447 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-02-09T23:04:06.447 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-02-09T23:04:06.567 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 -- 172.21.15.3:0/3708691908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f3410b820 msgr2=0x7f1f3410dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/3708691908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f3410b820 0x7f1f3410dc50 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009ca0 tx=0x7f1f2c02c180 comp rx=0 tx=0).stop 2024-02-09T23:04:06.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 -- 172.21.15.3:0/3708691908 shutdown_connections 2024-02-09T23:04:06.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/3708691908 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f34102d70 0x7f1f3410b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/3708691908 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f3410b820 0x7f1f3410dc50 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/3708691908 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f34102450 0x7f1f34102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.568 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 -- 172.21.15.3:0/3708691908 >> 172.21.15.3:0/3708691908 conn(0x7f1f34077250 msgr2=0x7f1f34077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 -- 172.21.15.3:0/3708691908 shutdown_connections 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 -- 172.21.15.3:0/3708691908 wait complete. 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 Processor -- start 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.563+0000 7f1f3aa17640 1 -- start start 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f3aa17640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f34102450 0x7f1f3419d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f3aa17640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 0x7f1f3419d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f3aa17640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f3410b820 0x7f1f34197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.569 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f3aa17640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34197c60 con 0x7f1f34102d70 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f3aa17640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34197dd0 con 0x7f1f3410b820 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f3aa17640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f34197f90 con 0x7f1f34102450 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 0x7f1f3419d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 0x7f1f3419d950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45568/0 (socket says 172.21.15.3:45568) 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 -- 172.21.15.3:0/1669057102 learned_addr learned my addr 172.21.15.3:0/1669057102 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f3410b820 0x7f1f34197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f33fff640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f34102450 0x7f1f3419d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 -- 172.21.15.3:0/1669057102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f34102450 msgr2=0x7f1f3419d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.570 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f34102450 0x7f1f3419d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 -- 172.21.15.3:0/1669057102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f3410b820 msgr2=0x7f1f34197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f3410b820 0x7f1f34197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 -- 172.21.15.3:0/1669057102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c009900 con 0x7f1f34102d70 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.564+0000 7f1f337fe640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 0x7f1f3419d950 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800c920 tx=0x7f1f2800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.565+0000 7f1f317fa640 1 -- 172.21.15.3:0/1669057102 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28007c70 con 0x7f1f34102d70 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.565+0000 7f1f317fa640 1 -- 172.21.15.3:0/1669057102 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f28007dd0 con 0x7f1f34102d70 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.565+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f34198270 con 0x7f1f34102d70 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.565+0000 7f1f317fa640 1 -- 172.21.15.3:0/1669057102 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28018860 con 0x7f1f34102d70 2024-02-09T23:04:06.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.565+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f341a44b0 con 0x7f1f34102d70 2024-02-09T23:04:06.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.567+0000 7f1f317fa640 1 -- 172.21.15.3:0/1669057102 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1f2800fac0 con 0x7f1f34102d70 2024-02-09T23:04:06.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.567+0000 7f1f3d5df740 1 Processor -- start 2024-02-09T23:04:06.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.567+0000 7f1f3d5df740 1 -- start start 2024-02-09T23:04:06.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a880209d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a88020a330 0x55a8802128e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a880212eb0 0x55a880215300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8801c4d00 con 0x55a880209980 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8801792f0 con 0x55a88020a330 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a880176820 con 0x55a880212eb0 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f317fa640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1f1c0a22b0 0x7f1f1c0a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a880209d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a880209d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45582/0 (socket says 172.21.15.3:45582) 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 -- 172.21.15.3:0/3798147464 learned_addr learned my addr 172.21.15.3:0/3798147464 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f337fe640 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a880212eb0 0x55a880215300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1f1c0a22b0 0x7f1f1c0a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f33fff640 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a88020a330 0x55a8802128e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f317fa640 1 -- 172.21.15.3:0/1669057102 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f1f280ce1a0 con 0x7f1f34102d70 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 -- 172.21.15.3:0/3798147464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a880212eb0 msgr2=0x55a880215300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a880212eb0 0x55a880215300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 -- 172.21.15.3:0/3798147464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a88020a330 msgr2=0x55a8802128e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a88020a330 0x55a8802128e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 -- 172.21.15.3:0/3798147464 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a880133ff0 con 0x55a880209980 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a880209d60 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f1f3406a9b0 tx=0x7f1f2c003aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267df3a614170ac4 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.568+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1f1c0a22b0 0x7f1f1c0a4770 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c032730 tx=0x7f1f2c03c000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f1a7fc640 1 -- 172.21.15.3:0/3798147464 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c048070 con 0x55a880209980 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f1a7fc640 1 -- 172.21.15.3:0/3798147464 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f2c032490 con 0x55a880209980 2024-02-09T23:04:06.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f1a7fc640 1 -- 172.21.15.3:0/3798147464 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c032e90 con 0x55a880209980 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 -- 172.21.15.3:0/3798147464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 msgr2=0x55a880209d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a880209d60 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7f1f3406a9b0 tx=0x7f1f2c003aa0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 -- 172.21.15.3:0/3798147464 shutdown_connections 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a88020a330 0x55a8802128e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a880212eb0 0x55a880215300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/3798147464 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a880209d60 unknown :-1 s=CLOSED pgs=1766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 -- 172.21.15.3:0/3798147464 >> 172.21.15.3:0/3798147464 conn(0x55a880204050 msgr2=0x55a880204430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 -- 172.21.15.3:0/3798147464 shutdown_connections 2024-02-09T23:04:06.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.569+0000 7f1f3d5df740 1 -- 172.21.15.3:0/3798147464 wait complete. 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 -- start start 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 Processor -- start 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a88020fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a8802103a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880212eb0 0x55a8802108e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a880172000 con 0x55a880209980 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a87fe45430 con 0x55a880212eb0 2024-02-09T23:04:06.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a87fe01e00 con 0x55a88020a330 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f3d5df740 10 client.0 _ll_register_callbacks cb 0x55a8801667f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a88020fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a88020fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45592/0 (socket says 172.21.15.3:45592) 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f33fff640 1 -- 172.21.15.3:0/456779438 learned_addr learned my addr 172.21.15.3:0/456779438 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.570+0000 7f1f337fe640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a8802103a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880212eb0 0x55a8802108e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f33fff640 1 -- 172.21.15.3:0/456779438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 msgr2=0x55a8802103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f33fff640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a8802103a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f33fff640 1 -- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880212eb0 msgr2=0x55a8802108e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f33fff640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880212eb0 0x55a8802108e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f33fff640 1 -- 172.21.15.3:0/456779438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f3410e8d0 con 0x55a880209980 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f38fb4640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880212eb0 0x55a8802108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f337fe640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a8802103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f33fff640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a88020fe60 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f1f24011760 tx=0x7f1f24011c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183608c6a61cb903 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f137fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:06.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f137fe640 1 -- 172.21.15.3:0/456779438 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400fe10 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f137fe640 1 -- 172.21.15.3:0/456779438 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f1f24005020 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a87feae720 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f137fe640 1 -- 172.21.15.3:0/456779438 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f240051e0 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f3d5df740 10 client.20124 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55a880170680 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.571+0000 7f1f137fe640 1 -- 172.21.15.3:0/456779438 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a87feae720 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f137fe640 1 -- 172.21.15.3:0/456779438 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1f24019480 con 0x55a880209980 2024-02-09T23:04:06.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f3d5df740 10 client.20124 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f3d5df740 10 client.20124 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f3d5df740 20 client.20124 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a8802234d0 0x55a880225910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f3d5df740 4 client.20124 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a87fefd890 con 0x55a8802234d0 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f337fe640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a8802234d0 0x55a880225910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f337fe640 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a8802234d0 0x55a880225910 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1237b9b9c826321b server_cookie=c86845f149f1e8b8 in_seq=0 out_seq=0 2024-02-09T23:04:06.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.572+0000 7f1f137fe640 10 client.20124 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f137fe640 1 -- 172.21.15.3:0/456779438 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a87fefd890 con 0x55a8802234d0 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f137fe640 10 client.20124 handle_command_reply: tid=0 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 client.20124 shutdown 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 msgr2=0x55a88020fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a88020fe60 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7f1f24011760 tx=0x7f1f24011c20 comp rx=0 tx=0).stop 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 shutdown_connections 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a8802234d0 0x55a880225910 crc :-1 s=READY pgs=347 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880212eb0 0x55a8802108e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f137fe640 0 client.20124 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a8802103a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/456779438 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a88020fe60 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 >> 172.21.15.3:0/456779438 conn(0x55a880204050 msgr2=0x55a880205da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 shutdown_connections 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.573+0000 7f1f3d5df740 1 -- 172.21.15.3:0/456779438 wait complete. 2024-02-09T23:04:06.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.574+0000 7f1f3d5df740 20 client.20124 trim_cache size 0 max 16384 2024-02-09T23:04:06.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 Processor -- start 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 -- start start 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880204050 0x55a8802103b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a8802108f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a88020d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8800f6fd0 con 0x55a880209980 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a880210e30 con 0x55a880204050 2024-02-09T23:04:06.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a88022ab60 con 0x55a88020a330 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a8802108f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a8802108f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45606/0 (socket says 172.21.15.3:45606) 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 -- 172.21.15.3:0/1383043029 learned_addr learned my addr 172.21.15.3:0/1383043029 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 -- 172.21.15.3:0/1383043029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 msgr2=0x55a88020d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a88020d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f33fff640 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880204050 0x55a8802103b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 -- 172.21.15.3:0/1383043029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880204050 msgr2=0x55a8802103b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880204050 0x55a8802103b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 -- 172.21.15.3:0/1383043029 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a88010cb30 con 0x55a880209980 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.589+0000 7f1f337fe640 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a8802108f0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1f280974e0 tx=0x7f1f2801cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20045f71fe1b4f72 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f19ffb640 1 -- 172.21.15.3:0/1383043029 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f280131b0 con 0x55a880209980 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f19ffb640 1 -- 172.21.15.3:0/1383043029 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f2809b210 con 0x55a880209980 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f19ffb640 1 -- 172.21.15.3:0/1383043029 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f28097180 con 0x55a880209980 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 -- 172.21.15.3:0/1383043029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 msgr2=0x55a8802108f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a8802108f0 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7f1f280974e0 tx=0x7f1f2801cea0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.599 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 -- 172.21.15.3:0/1383043029 shutdown_connections 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a88020d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880209980 0x55a8802108f0 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/1383043029 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880204050 0x55a8802103b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 -- 172.21.15.3:0/1383043029 >> 172.21.15.3:0/1383043029 conn(0x55a88020aa80 msgr2=0x55a88020b160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 -- 172.21.15.3:0/1383043029 shutdown_connections 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 -- 172.21.15.3:0/1383043029 wait complete. 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.590+0000 7f1f3d5df740 1 -- start start 2024-02-09T23:04:06.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 Processor -- start 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 0x55a8802192f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880209980 0x55a880219830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a880272a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a880272f40 con 0x55a880204050 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a8802730b0 con 0x55a880209980 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a880273270 con 0x55a88020a330 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f3d5df740 10 client.0 _ll_register_callbacks cb 0x7f1f2c009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:06.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 0x55a8802192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 0x55a8802192f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45610/0 (socket says 172.21.15.3:45610) 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 -- 172.21.15.3:0/4254350786 learned_addr learned my addr 172.21.15.3:0/4254350786 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f337fe640 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880209980 0x55a880219830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 -- 172.21.15.3:0/4254350786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 msgr2=0x55a880272a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a880272a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 -- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880209980 msgr2=0x55a880219830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880209980 0x55a880219830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.602 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.591+0000 7f1f33fff640 1 -- 172.21.15.3:0/4254350786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f341a44b0 con 0x55a880204050 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f33fff640 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 0x55a8802192f0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1f24019860 tx=0x7f1f2400fec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9aadcac1a52db01 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f137fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f137fe640 1 -- 172.21.15.3:0/4254350786 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400f790 con 0x55a880204050 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f137fe640 1 -- 172.21.15.3:0/4254350786 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f1f24000e40 con 0x55a880204050 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a87fe717e0 con 0x55a880204050 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f137fe640 1 -- 172.21.15.3:0/4254350786 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f24019980 con 0x55a880204050 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 10 client.20130 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:06.603 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55a880132850 con 0x55a880204050 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f137fe640 1 -- 172.21.15.3:0/4254350786 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f24038020 con 0x55a880204050 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f137fe640 1 -- 172.21.15.3:0/4254350786 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f1f24020040 con 0x55a880204050 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 10 client.20130 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 10 client.20130 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 20 client.20130 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a880212eb0 0x55a88027dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 4 client.20130 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-02-09T23:04:06.604 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.592+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55a87fe1d700 con 0x55a880212eb0 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f337fe640 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a880212eb0 0x55a88027dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f337fe640 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a880212eb0 0x55a88027dd70 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8663be3ee3314ef5 server_cookie=9ac5db2ffc5fa51 in_seq=0 out_seq=0 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f137fe640 10 client.20130 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f137fe640 1 -- 172.21.15.3:0/4254350786 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55a87fe1d700 con 0x55a880212eb0 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f137fe640 10 client.20130 handle_command_reply: tid=0 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 client.20130 shutdown 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 msgr2=0x55a8802192f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.605 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 0x55a8802192f0 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7f1f24019860 tx=0x7f1f2400fec0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 shutdown_connections 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a880212eb0 0x55a88027dd70 crc :-1 s=READY pgs=348 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a880209980 0x55a880219830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f137fe640 0 client.20130 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a88020a330 0x55a880272a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 --2- 172.21.15.3:0/4254350786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a880204050 0x55a8802192f0 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.606 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.593+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 >> 172.21.15.3:0/4254350786 conn(0x55a88020aa80 msgr2=0x55a880202960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 shutdown_connections 2024-02-09T23:04:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3d5df740 1 -- 172.21.15.3:0/4254350786 wait complete. 2024-02-09T23:04:06.607 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3d5df740 20 client.20130 trim_cache size 0 max 16384 2024-02-09T23:04:06.607 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:04:06.608 INFO:teuthology.orchestra.run.smithi003.stdout: "status": "no active scrubs running", 2024-02-09T23:04:06.608 INFO:teuthology.orchestra.run.smithi003.stdout: "scrubs": {} 2024-02-09T23:04:06.608 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:04:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1f1c0a22b0 msgr2=0x7f1f1c0a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.609 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1f1c0a22b0 0x7f1f1c0a4770 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c032730 tx=0x7f1f2c03c000 comp rx=0 tx=0).stop 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 msgr2=0x7f1f3419d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 0x7f1f3419d950 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f1f2800c920 tx=0x7f1f2800cde0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 shutdown_connections 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1f1c0a22b0 0x7f1f1c0a4770 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1f3410b820 0x7f1f34197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1f34102d70 0x7f1f3419d950 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.610 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 --2- 172.21.15.3:0/1669057102 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1f34102450 0x7f1f3419d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 >> 172.21.15.3:0/1669057102 conn(0x7f1f34077250 msgr2=0x7f1f340fd190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 shutdown_connections 2024-02-09T23:04:06.611 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.594+0000 7f1f3aa17640 1 -- 172.21.15.3:0/1669057102 wait complete. 2024-02-09T23:04:06.611 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-02-09T23:04:06.611 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:04:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.711+0000 7f1dee163640 1 -- 172.21.15.3:0/73635308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8105420 msgr2=0x7f1de810cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.711+0000 7f1dee163640 1 --2- 172.21.15.3:0/73635308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8105420 0x7f1de810cff0 secure :-1 s=READY pgs=1770 cs=0 l=1 rev1=1 crypto rx=0x7f1de0009cd0 tx=0x7f1de002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- 172.21.15.3:0/73635308 shutdown_connections 2024-02-09T23:04:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 --2- 172.21.15.3:0/73635308 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100d00 0x7f1de8104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 --2- 172.21.15.3:0/73635308 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8105420 0x7f1de810cff0 unknown :-1 s=CLOSED pgs=1770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.717 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 --2- 172.21.15.3:0/73635308 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de8100350 0x7f1de8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- 172.21.15.3:0/73635308 >> 172.21.15.3:0/73635308 conn(0x7f1de80772b0 msgr2=0x7f1de80776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- 172.21.15.3:0/73635308 shutdown_connections 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- 172.21.15.3:0/73635308 wait complete. 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 Processor -- start 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- start start 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100350 0x7f1de819d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de8105420 0x7f1de819db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 0x7f1de8197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.718 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8198150 con 0x7f1de8197680 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de81982c0 con 0x7f1de8100350 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.712+0000 7f1dee163640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de8198480 con 0x7f1de8105420 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 0x7f1de8197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 0x7f1de8197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45632/0 (socket says 172.21.15.3:45632) 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 -- 172.21.15.3:0/2250124775 learned_addr learned my addr 172.21.15.3:0/2250124775 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de77fe640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100350 0x7f1de819d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de6ffd640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de8105420 0x7f1de819db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 -- 172.21.15.3:0/2250124775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de8105420 msgr2=0x7f1de819db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de8105420 0x7f1de819db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 -- 172.21.15.3:0/2250124775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100350 msgr2=0x7f1de819d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100350 0x7f1de819d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 -- 172.21.15.3:0/2250124775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de0009900 con 0x7f1de8197680 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de77fe640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100350 0x7f1de819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de7fff640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 0x7f1de8197b00 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f1de00348f0 tx=0x7f1de0034cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de00328c0 con 0x7f1de8197680 2024-02-09T23:04:06.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de0003e50 con 0x7f1de8197680 2024-02-09T23:04:06.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de81a43c0 con 0x7f1de8197680 2024-02-09T23:04:06.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de0040450 con 0x7f1de8197680 2024-02-09T23:04:06.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.713+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de81a4840 con 0x7f1de8197680 2024-02-09T23:04:06.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.714+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de8101820 con 0x7f1de8197680 2024-02-09T23:04:06.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.715+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1de000a040 con 0x7f1de8197680 2024-02-09T23:04:06.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.716+0000 7f1de4ff9640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1dd00a2230 0x7f1dd00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:06.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.716+0000 7f1de77fe640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1dd00a2230 0x7f1dd00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:06.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.716+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f1de00f12a0 con 0x7f1de8197680 2024-02-09T23:04:06.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.717+0000 7f1de77fe640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1dd00a2230 0x7f1dd00a46f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80058c0 tx=0x7f1dd8005850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:06.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.720+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f1de00b9fc0 con 0x7f1de8197680 2024-02-09T23:04:06.942 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.940+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1de80680b0 con 0x7f1de8197680 2024-02-09T23:04:06.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.940+0000 7f1de4ff9640 1 -- 172.21.15.3:0/2250124775 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 55 v55) v1 ==== 94+0+11364 (secure 0 0 0) 0x7f1de00bd680 con 0x7f1de8197680 2024-02-09T23:04:06.944 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:06.944 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":55,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":55,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:03:55.088898+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_28706":{"gid":28706,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/1843835937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1843835937},{"type":"v1","addr":"172.21.15.64:6835","nonce":1843835937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:04:06.944 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 55 2024-02-09T23:04:06.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1dd00a2230 msgr2=0x7f1dd00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1dd00a2230 0x7f1dd00a46f0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f1dd80058c0 tx=0x7f1dd8005850 comp rx=0 tx=0).stop 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 msgr2=0x7f1de8197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 0x7f1de8197b00 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f1de00348f0 tx=0x7f1de0034cc0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 shutdown_connections 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f1dd00a2230 0x7f1dd00a46f0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1de8197680 0x7f1de8197b00 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1de8105420 0x7f1de819db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 --2- 172.21.15.3:0/2250124775 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1de8100350 0x7f1de819d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 >> 172.21.15.3:0/2250124775 conn(0x7f1de80772b0 msgr2=0x7f1de80687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 shutdown_connections 2024-02-09T23:04:06.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:06.943+0000 7f1dee163640 1 -- 172.21.15.3:0/2250124775 wait complete. 2024-02-09T23:04:06.957 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:04:06.957 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-02-09T23:04:07.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2037771362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 msgr2=0x7f6b98100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2037771362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b98100ee0 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009930 tx=0x7f6b9402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2037771362 shutdown_connections 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2037771362 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b981014b0 0x7f6b98104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2037771362 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b98105370 0x7f6b9810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2037771362 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b98100ee0 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2037771362 >> 172.21.15.3:0/2037771362 conn(0x7f6b98076630 msgr2=0x7f6b980fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2037771362 shutdown_connections 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.059+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2037771362 wait complete. 2024-02-09T23:04:07.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 Processor -- start 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 -- start start 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b9810d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b98105370 0x7f6b9810ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b9810e330 0x7f6b981a5a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981a6000 con 0x7f6b98100b00 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981a6170 con 0x7f6b9810e330 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9fff4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b981a6330 con 0x7f6b98105370 2024-02-09T23:04:07.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b9810d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b9810d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45654/0 (socket says 172.21.15.3:45654) 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 -- 172.21.15.3:0/2646050207 learned_addr learned my addr 172.21.15.3:0/2646050207 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9e591640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b9810e330 0x7f6b981a5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 -- 172.21.15.3:0/2646050207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b98105370 msgr2=0x7f6b9810ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b98105370 0x7f6b9810ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 -- 172.21.15.3:0/2646050207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b9810e330 msgr2=0x7f6b981a5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b9810e330 0x7f6b981a5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.060+0000 7f6b9dd90640 1 -- 172.21.15.3:0/2646050207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b94009560 con 0x7f6b98100b00 2024-02-09T23:04:07.067 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.061+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b9810d8b0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009230 tx=0x7f6b9402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.061+0000 7f6b8effd640 1 -- 172.21.15.3:0/2646050207 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94004350 con 0x7f6b98100b00 2024-02-09T23:04:07.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.061+0000 7f6b8effd640 1 -- 172.21.15.3:0/2646050207 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b940044b0 con 0x7f6b98100b00 2024-02-09T23:04:07.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.061+0000 7f6b8effd640 1 -- 172.21.15.3:0/2646050207 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b94003d70 con 0x7f6b98100b00 2024-02-09T23:04:07.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.061+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b981a65b0 con 0x7f6b98100b00 2024-02-09T23:04:07.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.061+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b981a6b10 con 0x7f6b98100b00 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.063+0000 7f6b8effd640 1 -- 172.21.15.3:0/2646050207 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6b94004df0 con 0x7f6b98100b00 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.063+0000 7f6ba2bc8740 1 Processor -- start 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.063+0000 7f6ba2bc8740 1 -- start start 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.063+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x558352139e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.063+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352142990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x5583521453b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583520f5020 con 0x558352142f60 2024-02-09T23:04:07.075 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583520a8320 con 0x558352139a30 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583520a5850 con 0x55835213a3e0 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b8effd640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6b840a22e0 0x7f6b840a47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x5583521453b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x5583521453b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45660/0 (socket says 172.21.15.3:45660) 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 -- 172.21.15.3:0/133719370 learned_addr learned my addr 172.21.15.3:0/133719370 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9e591640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6b840a22e0 0x7f6b840a47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b8effd640 1 -- 172.21.15.3:0/2646050207 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f6b940f0180 con 0x7f6b98100b00 2024-02-09T23:04:07.076 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352142990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9e591640 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x558352139e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 -- 172.21.15.3:0/133719370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 msgr2=0x558352142990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352142990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 -- 172.21.15.3:0/133719370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 msgr2=0x558352139e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x558352139e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 -- 172.21.15.3:0/133719370 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558352063210 con 0x558352142f60 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.077 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x5583521453b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009bb0 tx=0x7f6b8800e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e42691bdafbdd570 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b9e591640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6b840a22e0 0x7f6b840a47a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6b90005c40 tx=0x7f6b900094f0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b83fff640 1 -- 172.21.15.3:0/133719370 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88009730 con 0x558352142f60 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b83fff640 1 -- 172.21.15.3:0/133719370 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88005500 con 0x558352142f60 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.064+0000 7f6b83fff640 1 -- 172.21.15.3:0/133719370 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b88010690 con 0x558352142f60 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/133719370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 msgr2=0x5583521453b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x5583521453b0 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f6b88009bb0 tx=0x7f6b8800e720 comp rx=0 tx=0).stop 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/133719370 shutdown_connections 2024-02-09T23:04:07.078 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x5583521453b0 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352142990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/133719370 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x558352139e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/133719370 >> 172.21.15.3:0/133719370 conn(0x558352134100 msgr2=0x5583521344e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/133719370 shutdown_connections 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.065+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/133719370 wait complete. 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 -- start start 2024-02-09T23:04:07.079 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 Processor -- start 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x55835213ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352140450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x558352140990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583520a1030 con 0x558352142f60 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558351d74430 con 0x558352139a30 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x558351d30e00 con 0x55835213a3e0 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6ba2bc8740 10 client.0 _ll_register_callbacks cb 0x558352096030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.080 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x558352140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x558352140990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45666/0 (socket says 172.21.15.3:45666) 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 -- 172.21.15.3:0/1781762192 learned_addr learned my addr 172.21.15.3:0/1781762192 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x55835213ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352140450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 -- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 msgr2=0x558352140450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352140450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 -- 172.21.15.3:0/1781762192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 msgr2=0x55835213ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x55835213ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.081 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9e591640 1 -- 172.21.15.3:0/1781762192 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b88009890 con 0x558352142f60 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.066+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b9e591640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x558352140990 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f6b900109d0 tx=0x7f6b90004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12f8fba064bc3302 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b817fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b817fa640 1 -- 172.21.15.3:0/1781762192 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90011d70 con 0x558352142f60 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b817fa640 1 -- 172.21.15.3:0/1781762192 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f6b90024070 con 0x558352142f60 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558351da07e0 con 0x558352142f60 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b817fa640 1 -- 172.21.15.3:0/1781762192 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b90011ed0 con 0x558352142f60 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 10 client.20148 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.082 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55835209f6b0 con 0x558352142f60 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b817fa640 1 -- 172.21.15.3:0/1781762192 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x558351da07e0 con 0x558352142f60 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6b817fa640 1 -- 172.21.15.3:0/1781762192 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6b9000c070 con 0x558352142f60 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 10 client.20148 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 10 client.20148 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 20 client.20148 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x558352153520 0x558352155940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 4 client.20148 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:04:07.083 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.067+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558351ddd720 con 0x558352153520 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.068+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x558352153520 0x558352155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.068+0000 7f6b817fa640 10 client.20148 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.068+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x558352153520 0x558352155940 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=103b25dc7579198a server_cookie=db38ea837e7885de in_seq=0 out_seq=0 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6b817fa640 1 -- 172.21.15.3:0/1781762192 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558351ddd720 con 0x558352153520 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6b817fa640 10 client.20148 handle_command_reply: tid=0 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 client.20148 shutdown 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 msgr2=0x558352140990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.084 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x558352140990 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f6b900109d0 tx=0x7f6b90004ab0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 shutdown_connections 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x558352153520 0x558352155940 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x558352140990 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x558352140450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6b817fa640 0 client.20148 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1781762192 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352139a30 0x55835213ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 >> 172.21.15.3:0/1781762192 conn(0x558352134100 msgr2=0x558352135e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.085 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 shutdown_connections 2024-02-09T23:04:07.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1781762192 wait complete. 2024-02-09T23:04:07.086 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.069+0000 7f6ba2bc8740 20 client.20148 trim_cache size 0 max 16384 2024-02-09T23:04:07.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 Processor -- start 2024-02-09T23:04:07.092 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 -- start start 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558352139a30 0x558352142640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55835213a3e0 0x5583521407b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x55835213d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x558352026130 con 0x558352142f60 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x558351da07e0 con 0x55835213a3e0 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583521339c0 con 0x558352139a30 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x55835213d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.093 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x55835213d190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45676/0 (socket says 172.21.15.3:45676) 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 -- 172.21.15.3:0/3446787355 learned_addr learned my addr 172.21.15.3:0/3446787355 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55835213a3e0 0x5583521407b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558352139a30 0x558352142640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 -- 172.21.15.3:0/3446787355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558352139a30 msgr2=0x558352142640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558352139a30 0x558352142640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 -- 172.21.15.3:0/3446787355 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55835213a3e0 msgr2=0x5583521407b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55835213a3e0 0x5583521407b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 -- 172.21.15.3:0/3446787355 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55835203bc90 con 0x558352142f60 2024-02-09T23:04:07.094 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558352139a30 0x558352142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.081+0000 7f6b9e591640 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x55835213d190 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f6b90004a80 tx=0x7f6b90009d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb92ae9a27d8d67 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6b837fe640 1 -- 172.21.15.3:0/3446787355 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90012520 con 0x558352142f60 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6b837fe640 1 -- 172.21.15.3:0/3446787355 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b90022070 con 0x558352142f60 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6b837fe640 1 -- 172.21.15.3:0/3446787355 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b90005280 con 0x558352142f60 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/3446787355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 msgr2=0x55835213d190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x55835213d190 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7f6b90004a80 tx=0x7f6b90009d10 comp rx=0 tx=0).stop 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/3446787355 shutdown_connections 2024-02-09T23:04:07.095 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55835213a3e0 0x5583521407b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352142f60 0x55835213d190 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/3446787355 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x558352139a30 0x558352142640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/3446787355 >> 172.21.15.3:0/3446787355 conn(0x558352134100 msgr2=0x55835215a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/3446787355 shutdown_connections 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.082+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/3446787355 wait complete. 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 -- start start 2024-02-09T23:04:07.096 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 Processor -- start 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 0x558352148c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x5583521491d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352142f60 0x558352149750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583521a9c30 con 0x558352139a30 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583521a9da0 con 0x558352142f60 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x5583521a9f60 con 0x55835213a3e0 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6ba2bc8740 10 client.0 _ll_register_callbacks cb 0x7f6b88009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.097 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 0x558352148c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 0x558352148c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45678/0 (socket says 172.21.15.3:45678) 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 -- 172.21.15.3:0/1290386048 learned_addr learned my addr 172.21.15.3:0/1290386048 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x5583521491d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9e591640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352142f60 0x558352149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 -- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 msgr2=0x5583521491d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x5583521491d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 -- 172.21.15.3:0/1290386048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352142f60 msgr2=0x558352149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.098 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352142f60 0x558352149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.083+0000 7f6b9dd90640 1 -- 172.21.15.3:0/1290386048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55835209f6b0 con 0x558352139a30 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b9dd90640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 0x558352148c90 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f6b940ed9c0 tx=0x7f6b9402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f2f95735d3baf87 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b817fa640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b817fa640 1 -- 172.21.15.3:0/1290386048 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b940baf00 con 0x558352139a30 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b817fa640 1 -- 172.21.15.3:0/1290386048 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f6b940b9270 con 0x558352139a30 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6b88005500 con 0x558352139a30 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b817fa640 1 -- 172.21.15.3:0/1290386048 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6b940b9430 con 0x558352139a30 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 10 client.20154 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.099 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x558352061a70 con 0x558352139a30 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b817fa640 1 -- 172.21.15.3:0/1290386048 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6b88005500 con 0x558352139a30 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b817fa640 1 -- 172.21.15.3:0/1290386048 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6b9403c070 con 0x558352139a30 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 10 client.20154 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 10 client.20154 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.f 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 20 client.20154 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5583521ab540 0x5583521ad930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 4 client.20154 mds_command: new command op to 24475 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:04:07.100 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 --> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558351e2c890 con 0x5583521ab540 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.084+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5583521ab540 0x5583521ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6b817fa640 10 client.20154 ms_handle_connect on v2:172.21.15.89:6834/4241363743 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6b9d58f640 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5583521ab540 0x5583521ad930 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=4eb35e30890d62d8 server_cookie=1d7a8f7d3cdb12ab in_seq=0 out_seq=0 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6b817fa640 1 -- 172.21.15.3:0/1290386048 <== mds.4 v2:172.21.15.89:6834/4241363743 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558351e2c890 con 0x5583521ab540 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6b817fa640 10 client.20154 handle_command_reply: tid=0 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 client.20154 shutdown 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 msgr2=0x558352148c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.101 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 0x558352148c90 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7f6b940ed9c0 tx=0x7f6b9402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 shutdown_connections 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:6834/4241363743,v1:172.21.15.89:6835/4241363743] conn(0x5583521ab540 0x5583521ad930 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x558352142f60 0x558352149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55835213a3e0 0x5583521491d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 --2- 172.21.15.3:0/1290386048 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x558352139a30 0x558352148c90 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6b817fa640 0 client.20154 ms_handle_reset on v2:172.21.15.89:6834/4241363743 2024-02-09T23:04:07.102 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 >> 172.21.15.3:0/1290386048 conn(0x558352134100 msgr2=0x5583521329b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 shutdown_connections 2024-02-09T23:04:07.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 1 -- 172.21.15.3:0/1290386048 wait complete. 2024-02-09T23:04:07.103 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.085+0000 7f6ba2bc8740 20 client.20154 trim_cache size 0 max 16384 2024-02-09T23:04:07.103 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:04:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6b840a22e0 msgr2=0x7f6b840a47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6b840a22e0 0x7f6b840a47a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f6b90005c40 tx=0x7f6b900094f0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 msgr2=0x7f6b9810d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b9810d8b0 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f6b94009230 tx=0x7f6b9402c6c0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.105 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 shutdown_connections 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6b840a22e0 0x7f6b840a47a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6b9810e330 0x7f6b981a5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6b98105370 0x7f6b9810ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 --2- 172.21.15.3:0/2646050207 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6b98100b00 0x7f6b9810d8b0 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 >> 172.21.15.3:0/2646050207 conn(0x7f6b98076630 msgr2=0x7f6b98103560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 shutdown_connections 2024-02-09T23:04:07.106 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.086+0000 7f6b9fff4640 1 -- 172.21.15.3:0/2646050207 wait complete. 2024-02-09T23:04:07.107 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:04:07.107 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-02-09T23:04:07.214 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 -- 172.21.15.3:0/1070389778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 msgr2=0x7f5154101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 --2- 172.21.15.3:0/1070389778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f5154101810 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f5144009930 tx=0x7f514402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 -- 172.21.15.3:0/1070389778 shutdown_connections 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 --2- 172.21.15.3:0/1070389778 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5154101de0 0x7f5154105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 --2- 172.21.15.3:0/1070389778 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154106500 0x7f515410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 --2- 172.21.15.3:0/1070389778 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f5154101810 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 -- 172.21.15.3:0/1070389778 >> 172.21.15.3:0/1070389778 conn(0x7f51540fccf0 msgr2=0x7f51540ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.215 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 -- 172.21.15.3:0/1070389778 shutdown_connections 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.209+0000 7f5158cf4640 1 -- 172.21.15.3:0/1070389778 wait complete. 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 Processor -- start 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 -- start start 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f515419d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5154106500 0x7f515419dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154197780 0x7f5154197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154198250 con 0x7f5154101430 2024-02-09T23:04:07.216 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51541983c0 con 0x7f5154106500 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5158cf4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154198580 con 0x7f5154197780 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f515259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f515419d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f515259c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f515419d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45692/0 (socket says 172.21.15.3:45692) 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f515259c640 1 -- 172.21.15.3:0/600247868 learned_addr learned my addr 172.21.15.3:0/600247868 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f5152d9d640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154197780 0x7f5154197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f515259c640 1 -- 172.21.15.3:0/600247868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154197780 msgr2=0x7f5154197c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.210+0000 7f515259c640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154197780 0x7f5154197c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.217 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f515259c640 1 -- 172.21.15.3:0/600247868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5154106500 msgr2=0x7f515419dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f515259c640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5154106500 0x7f515419dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f515259c640 1 -- 172.21.15.3:0/600247868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5144009560 con 0x7f5154101430 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f5152d9d640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154197780 0x7f5154197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f515259c640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f515419d670 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f5144009230 tx=0x7f5144004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f514b7fe640 1 -- 172.21.15.3:0/600247868 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144004440 con 0x7f5154101430 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f514b7fe640 1 -- 172.21.15.3:0/600247868 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51440052d0 con 0x7f5154101430 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51541a4470 con 0x7f5154101430 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f514b7fe640 1 -- 172.21.15.3:0/600247868 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144003d70 con 0x7f5154101430 2024-02-09T23:04:07.218 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.211+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51541a4950 con 0x7f5154101430 2024-02-09T23:04:07.226 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.213+0000 7f514b7fe640 1 -- 172.21.15.3:0/600247868 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5144005440 con 0x7f5154101430 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.213+0000 7f515b99b740 1 Processor -- start 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 -- start start 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a6910 0x55b19a3a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f514b7fe640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f51400a2230 0x7f51400a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 0x55b19a3b0820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a362020 con 0x55b19a3b03c0 2024-02-09T23:04:07.227 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a315320 con 0x55b19a3a72c0 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515b99b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a312850 con 0x55b19a3a6910 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515259c640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f51400a2230 0x7f51400a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 0x55b19a3b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 0x55b19a3b0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45700/0 (socket says 172.21.15.3:45700) 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 -- 172.21.15.3:0/532501574 learned_addr learned my addr 172.21.15.3:0/532501574 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f514b7fe640 1 -- 172.21.15.3:0/600247868 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f51440f02c0 con 0x7f5154101430 2024-02-09T23:04:07.228 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a6910 0x55b19a3a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f514bfff640 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 -- 172.21.15.3:0/532501574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a6910 msgr2=0x55b19a3a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a6910 0x55b19a3a6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 -- 172.21.15.3:0/532501574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 msgr2=0x55b19a3afdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3afdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f5152d9d640 1 -- 172.21.15.3:0/532501574 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b19a2d0210 con 0x55b19a3b03c0 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.214+0000 7f515259c640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f51400a2230 0x7f51400a46f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5144000c00 tx=0x7f514408ed60 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f5152d9d640 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 0x55b19a3b0820 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f514c009f70 tx=0x7f514c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6f3c1c5fea21445 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.229 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f5148ff9640 1 -- 172.21.15.3:0/532501574 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c013d50 con 0x55b19a3b03c0 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f5148ff9640 1 -- 172.21.15.3:0/532501574 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f514c011900 con 0x55b19a3b03c0 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f5148ff9640 1 -- 172.21.15.3:0/532501574 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c005310 con 0x55b19a3b03c0 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 -- 172.21.15.3:0/532501574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 msgr2=0x55b19a3b0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 0x55b19a3b0820 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7f514c009f70 tx=0x7f514c009810 comp rx=0 tx=0).stop 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 -- 172.21.15.3:0/532501574 shutdown_connections 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3afdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3b03c0 0x55b19a3b0820 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.230 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 --2- 172.21.15.3:0/532501574 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a6910 0x55b19a3a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.215+0000 7f515b99b740 1 -- 172.21.15.3:0/532501574 >> 172.21.15.3:0/532501574 conn(0x55b19a3a1100 msgr2=0x55b19a3a14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 -- 172.21.15.3:0/532501574 shutdown_connections 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 -- 172.21.15.3:0/532501574 wait complete. 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 -- start start 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 Processor -- start 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3b4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3b03c0 0x55b19a3b4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.231 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a30e030 con 0x55b19a3a6910 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b199fe1430 con 0x55b19a3a72c0 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b199f9de00 con 0x55b19a3b03c0 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f515b99b740 10 client.0 _ll_register_callbacks cb 0x55b19a303030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45710/0 (socket says 172.21.15.3:45710) 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 -- 172.21.15.3:0/3417042543 learned_addr learned my addr 172.21.15.3:0/3417042543 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3b03c0 0x55b19a3b4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.232 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f514bfff640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 -- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3b03c0 msgr2=0x55b19a3b4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3b03c0 0x55b19a3b4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 -- 172.21.15.3:0/3417042543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 msgr2=0x55b19a3b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.216+0000 7f5152d9d640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5152d9d640 1 -- 172.21.15.3:0/3417042543 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c013400 con 0x55b19a3a6910 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f514bfff640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5152d9d640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b3ec0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f514c0092f0 tx=0x7f514c009560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc7cf7b88ee66713 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.233 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5139ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5139ffb640 1 -- 172.21.15.3:0/3417042543 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c004b80 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5139ffb640 1 -- 172.21.15.3:0/3417042543 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f514c004ce0 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b19a00d7e0 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5139ffb640 1 -- 172.21.15.3:0/3417042543 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f514c004ea0 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 10 client.20166 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55b19a30c6b0 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5139ffb640 1 -- 172.21.15.3:0/3417042543 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b19a00d7e0 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f5139ffb640 1 -- 172.21.15.3:0/3417042543 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f514c01e800 con 0x55b19a3a6910 2024-02-09T23:04:07.234 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 10 client.20166 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 10 client.20166 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 20 client.20166 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a3c4820 0x55b19a3c6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 4 client.20166 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.217+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b19a04a720 con 0x55b19a3c4820 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.218+0000 7f514bfff640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a3c4820 0x55b19a3c6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.218+0000 7f5139ffb640 10 client.20166 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:04:07.235 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.218+0000 7f514bfff640 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a3c4820 0x55b19a3c6c40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=be1bd54732345c4 server_cookie=8cc2bf4e5832d41f in_seq=0 out_seq=0 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f5139ffb640 1 -- 172.21.15.3:0/3417042543 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b19a04a720 con 0x55b19a3c4820 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f5139ffb640 10 client.20166 handle_command_reply: tid=0 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 client.20166 shutdown 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 msgr2=0x55b19a3b3ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b3ec0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7f514c0092f0 tx=0x7f514c009560 comp rx=0 tx=0).stop 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 shutdown_connections 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a3c4820 0x55b19a3c6c40 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.236 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3a72c0 0x55b19a3b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3b03c0 0x55b19a3b4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f5139ffb640 0 client.20166 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 --2- 172.21.15.3:0/3417042543 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b3ec0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 >> 172.21.15.3:0/3417042543 conn(0x55b19a3a1100 msgr2=0x55b19a3a2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 shutdown_connections 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 1 -- 172.21.15.3:0/3417042543 wait complete. 2024-02-09T23:04:07.237 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.219+0000 7f515b99b740 20 client.20166 trim_cache size 0 max 16384 2024-02-09T23:04:07.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.230+0000 7f515b99b740 1 Processor -- start 2024-02-09T23:04:07.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 -- start start 2024-02-09T23:04:07.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3b2790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.243 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a293130 con 0x55b19a3a6910 2024-02-09T23:04:07.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a3adf80 con 0x55b19a3b03c0 2024-02-09T23:04:07.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f515b99b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a3a5d40 con 0x55b19a3a72c0 2024-02-09T23:04:07.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45720/0 (socket says 172.21.15.3:45720) 2024-02-09T23:04:07.244 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 -- 172.21.15.3:0/2986794816 learned_addr learned my addr 172.21.15.3:0/2986794816 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f514bfff640 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 -- 172.21.15.3:0/2986794816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 msgr2=0x55b19a3b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3b17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 -- 172.21.15.3:0/2986794816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 msgr2=0x55b19a3b2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.245 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3b2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 -- 172.21.15.3:0/2986794816 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b19a2a8c90 con 0x55b19a3a6910 2024-02-09T23:04:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f514bfff640 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b65a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f514c016020 tx=0x7f514c0136d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1d03498d11bf0c0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f513bfff640 1 -- 172.21.15.3:0/2986794816 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c011750 con 0x55b19a3a6910 2024-02-09T23:04:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f513bfff640 1 -- 172.21.15.3:0/2986794816 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f514c009770 con 0x55b19a3a6910 2024-02-09T23:04:07.246 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.231+0000 7f513bfff640 1 -- 172.21.15.3:0/2986794816 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c02d6d0 con 0x55b19a3a6910 2024-02-09T23:04:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- 172.21.15.3:0/2986794816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 msgr2=0x55b19a3b65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b65a0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7f514c016020 tx=0x7f514c0136d0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- 172.21.15.3:0/2986794816 shutdown_connections 2024-02-09T23:04:07.247 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3b2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- 172.21.15.3:0/2986794816 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b65a0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- 172.21.15.3:0/2986794816 >> 172.21.15.3:0/2986794816 conn(0x55b19a3a1100 msgr2=0x55b19a3ae180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.248 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- 172.21.15.3:0/2986794816 shutdown_connections 2024-02-09T23:04:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- 172.21.15.3:0/2986794816 wait complete. 2024-02-09T23:04:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- start start 2024-02-09T23:04:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 Processor -- start 2024-02-09T23:04:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.249 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3ba310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3ba850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a3bbd70 con 0x55b19a3a6910 2024-02-09T23:04:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a41ad30 con 0x55b19a3b03c0 2024-02-09T23:04:07.250 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b19a41aea0 con 0x55b19a3a72c0 2024-02-09T23:04:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.232+0000 7f515b99b740 10 client.0 _ll_register_callbacks cb 0x7f514c009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b9dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45728/0 (socket says 172.21.15.3:45728) 2024-02-09T23:04:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 -- 172.21.15.3:0/2591735991 learned_addr learned my addr 172.21.15.3:0/2591735991 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.251 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f514bfff640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3ba310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3ba850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 -- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 msgr2=0x55b19a3ba310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.252 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3ba310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 -- 172.21.15.3:0/2591735991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 msgr2=0x55b19a3ba850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.255 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3ba850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 -- 172.21.15.3:0/2591735991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b19a30c6b0 con 0x55b19a3a6910 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5152d9d640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b9dd0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f514c0135f0 tx=0x7f514c004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37732e2ef7ee22c3 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5139ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5139ffb640 1 -- 172.21.15.3:0/2591735991 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c01e670 con 0x55b19a3a6910 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5139ffb640 1 -- 172.21.15.3:0/2591735991 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f514c01e7d0 con 0x55b19a3a6910 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f514c011900 con 0x55b19a3a6910 2024-02-09T23:04:07.256 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5139ffb640 1 -- 172.21.15.3:0/2591735991 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f514c01e990 con 0x55b19a3a6910 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f515b99b740 10 client.20172 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55b19a2cea70 con 0x55b19a3a6910 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.233+0000 7f5139ffb640 1 -- 172.21.15.3:0/2591735991 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f514c011900 con 0x55b19a3a6910 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f5139ffb640 1 -- 172.21.15.3:0/2591735991 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f514c038430 con 0x55b19a3a6910 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f515b99b740 10 client.20172 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f515b99b740 10 client.20172 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.c 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f515b99b740 20 client.20172 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f515b99b740 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a41c300 0x55b19a41e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.257 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f515b99b740 4 client.20172 mds_command: new command op to 24481 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 --> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b19a099890 con 0x55b19a41c300 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f514bfff640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a41c300 0x55b19a41e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f514bfff640 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a41c300 0x55b19a41e720 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=ecbbc54e0441b783 server_cookie=179950814683e3cb in_seq=0 out_seq=0 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.234+0000 7f5139ffb640 10 client.20172 ms_handle_connect on v2:172.21.15.89:6836/1935794058 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5139ffb640 1 -- 172.21.15.3:0/2591735991 <== mds.3 v2:172.21.15.89:6836/1935794058 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b19a099890 con 0x55b19a41c300 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5139ffb640 10 client.20172 handle_command_reply: tid=0 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 client.20172 shutdown 2024-02-09T23:04:07.258 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 msgr2=0x55b19a3b9dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b9dd0 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f514c0135f0 tx=0x7f514c004d30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 shutdown_connections 2024-02-09T23:04:07.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:6836/1935794058,v1:172.21.15.89:6837/1935794058] conn(0x55b19a41c300 0x55b19a41e720 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55b19a3b03c0 0x55b19a3ba850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55b19a3a72c0 0x55b19a3ba310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.259 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 --2- 172.21.15.3:0/2591735991 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55b19a3a6910 0x55b19a3b9dd0 unknown :-1 s=CLOSED pgs=1783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 >> 172.21.15.3:0/2591735991 conn(0x55b19a3a1100 msgr2=0x55b19a3b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5139ffb640 0 client.20172 ms_handle_reset on v2:172.21.15.89:6836/1935794058 2024-02-09T23:04:07.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 shutdown_connections 2024-02-09T23:04:07.260 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 1 -- 172.21.15.3:0/2591735991 wait complete. 2024-02-09T23:04:07.261 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f515b99b740 20 client.20172 trim_cache size 0 max 16384 2024-02-09T23:04:07.261 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:04:07.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f51400a2230 msgr2=0x7f51400a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f51400a2230 0x7f51400a46f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5144000c00 tx=0x7f514408ed60 comp rx=0 tx=0).stop 2024-02-09T23:04:07.264 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 msgr2=0x7f515419d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f515419d670 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f5144009230 tx=0x7f5144004040 comp rx=0 tx=0).stop 2024-02-09T23:04:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 shutdown_connections 2024-02-09T23:04:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f51400a2230 0x7f51400a46f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5154106500 0x7f515419dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.265 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5154197780 0x7f5154197c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 --2- 172.21.15.3:0/600247868 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5154101430 0x7f515419d670 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 >> 172.21.15.3:0/600247868 conn(0x7f51540fccf0 msgr2=0x7f51540fe050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 shutdown_connections 2024-02-09T23:04:07.266 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.235+0000 7f5158cf4640 1 -- 172.21.15.3:0/600247868 wait complete. 2024-02-09T23:04:07.267 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:04:07.267 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-02-09T23:04:07.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 -- 172.21.15.3:0/2907152696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d30100cf0 msgr2=0x7f5d30104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.390 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 --2- 172.21.15.3:0/2907152696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d30100cf0 0x7f5d30104d30 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f5d20009930 tx=0x7f5d2002bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 -- 172.21.15.3:0/2907152696 shutdown_connections 2024-02-09T23:04:07.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 --2- 172.21.15.3:0/2907152696 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 0x7f5d3010cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.391 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 --2- 172.21.15.3:0/2907152696 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d30100cf0 0x7f5d30104d30 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 --2- 172.21.15.3:0/2907152696 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d30100340 0x7f5d30100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 -- 172.21.15.3:0/2907152696 >> 172.21.15.3:0/2907152696 conn(0x7f5d300772e0 msgr2=0x7f5d300776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 -- 172.21.15.3:0/2907152696 shutdown_connections 2024-02-09T23:04:07.392 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 -- 172.21.15.3:0/2907152696 wait complete. 2024-02-09T23:04:07.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 Processor -- start 2024-02-09T23:04:07.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.370+0000 7f5d38415640 1 -- start start 2024-02-09T23:04:07.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d30100340 0x7f5d3019d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.393 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 0x7f5d3019db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 0x7f5d30197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d301981b0 con 0x7f5d301976e0 2024-02-09T23:04:07.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d30198320 con 0x7f5d30100340 2024-02-09T23:04:07.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d301984e0 con 0x7f5d30105410 2024-02-09T23:04:07.394 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 0x7f5d30197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 0x7f5d30197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45754/0 (socket says 172.21.15.3:45754) 2024-02-09T23:04:07.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 -- 172.21.15.3:0/2715328839 learned_addr learned my addr 172.21.15.3:0/2715328839 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d361b1640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d30100340 0x7f5d3019d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.395 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d359b0640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 0x7f5d3019db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 -- 172.21.15.3:0/2715328839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 msgr2=0x7f5d3019db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 0x7f5d3019db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 -- 172.21.15.3:0/2715328839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d30100340 msgr2=0x7f5d3019d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.396 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d30100340 0x7f5d3019d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 -- 172.21.15.3:0/2715328839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d20009560 con 0x7f5d301976e0 2024-02-09T23:04:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d359b0640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 0x7f5d3019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d369b2640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 0x7f5d30197b60 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800e960 tx=0x7f5d2800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d277fe640 1 -- 172.21.15.3:0/2715328839 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d28004d20 con 0x7f5d301976e0 2024-02-09T23:04:07.397 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d301a4480 con 0x7f5d301976e0 2024-02-09T23:04:07.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d277fe640 1 -- 172.21.15.3:0/2715328839 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d28004e80 con 0x7f5d301976e0 2024-02-09T23:04:07.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d277fe640 1 -- 172.21.15.3:0/2715328839 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d280116d0 con 0x7f5d301976e0 2024-02-09T23:04:07.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.371+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d301a4910 con 0x7f5d301976e0 2024-02-09T23:04:07.398 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.374+0000 7f5d277fe640 1 -- 172.21.15.3:0/2715328839 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f5d28005340 con 0x7f5d301976e0 2024-02-09T23:04:07.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.374+0000 7f5d3afe1740 1 Processor -- start 2024-02-09T23:04:07.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.374+0000 7f5d3afe1740 1 -- start start 2024-02-09T23:04:07.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d20be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.399 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d214990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 0x56034d2173b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d1c7020 con 0x56034d214f60 2024-02-09T23:04:07.400 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d17a320 con 0x56034d20c3e0 2024-02-09T23:04:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d177850 con 0x56034d20ba30 2024-02-09T23:04:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d277fe640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5d1c0a2230 0x7f5d1c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 0x56034d2173b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 0x56034d2173b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45762/0 (socket says 172.21.15.3:45762) 2024-02-09T23:04:07.401 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 -- 172.21.15.3:0/1422859130 learned_addr learned my addr 172.21.15.3:0/1422859130 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d369b2640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5d1c0a2230 0x7f5d1c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d214990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d20be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.402 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d277fe640 1 -- 172.21.15.3:0/2715328839 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f5d280ce260 con 0x7f5d301976e0 2024-02-09T23:04:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 -- 172.21.15.3:0/1422859130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 msgr2=0x56034d20be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d20be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 -- 172.21.15.3:0/1422859130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 msgr2=0x56034d214990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.403 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d214990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 -- 172.21.15.3:0/1422859130 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56034d135210 con 0x56034d214f60 2024-02-09T23:04:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d214990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:04:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d369b2640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5d1c0a2230 0x7f5d1c0a46f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5d2805c070 tx=0x7f5d2806dca0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.404 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.375+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 0x56034d2173b0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005ec0 tx=0x7f5d20005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502cc1c09416dbfb server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d24bfe640 1 -- 172.21.15.3:0/1422859130 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2002c9e0 con 0x56034d214f60 2024-02-09T23:04:07.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d24bfe640 1 -- 172.21.15.3:0/1422859130 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d20004510 con 0x56034d214f60 2024-02-09T23:04:07.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d24bfe640 1 -- 172.21.15.3:0/1422859130 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d200055c0 con 0x56034d214f60 2024-02-09T23:04:07.405 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1422859130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 msgr2=0x56034d2173b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 0x56034d2173b0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005ec0 tx=0x7f5d20005fd0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1422859130 shutdown_connections 2024-02-09T23:04:07.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d214990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.406 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d214f60 0x56034d2173b0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1422859130 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d20be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.376+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1422859130 >> 172.21.15.3:0/1422859130 conn(0x56034d206100 msgr2=0x56034d2064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1422859130 shutdown_connections 2024-02-09T23:04:07.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1422859130 wait complete. 2024-02-09T23:04:07.407 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 -- start start 2024-02-09T23:04:07.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 Processor -- start 2024-02-09T23:04:07.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d211f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.408 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d212490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2129d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d173030 con 0x56034d20c3e0 2024-02-09T23:04:07.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034ce46430 con 0x56034d214f60 2024-02-09T23:04:07.409 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034ce02e00 con 0x56034d20ba30 2024-02-09T23:04:07.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d3afe1740 10 client.0 _ll_register_callbacks cb 0x56034d168030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d361b1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d212490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d361b1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d212490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45770/0 (socket says 172.21.15.3:45770) 2024-02-09T23:04:07.410 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d361b1640 1 -- 172.21.15.3:0/1828041329 learned_addr learned my addr 172.21.15.3:0/1828041329 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.377+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d211f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2129d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d361b1640 1 -- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 msgr2=0x56034d211f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.411 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d211f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d361b1640 1 -- 172.21.15.3:0/1828041329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 msgr2=0x56034d2129d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2129d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d361b1640 1 -- 172.21.15.3:0/1828041329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3010d7f0 con 0x56034d20c3e0 2024-02-09T23:04:07.412 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d212490 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00bad0 tx=0x7f5d2c00bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=286834e1812f5a6e server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1828041329 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c016070 con 0x56034d20c3e0 2024-02-09T23:04:07.413 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1828041329 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f5d2c003e00 con 0x56034d20c3e0 2024-02-09T23:04:07.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56034ce727e0 con 0x56034d20c3e0 2024-02-09T23:04:07.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1828041329 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d2c011440 con 0x56034d20c3e0 2024-02-09T23:04:07.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d3afe1740 10 client.20184 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.414 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x56034d1716b0 con 0x56034d20c3e0 2024-02-09T23:04:07.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.378+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1828041329 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d2c027080 con 0x56034d20c3e0 2024-02-09T23:04:07.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1828041329 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5d2c0237f0 con 0x56034d20c3e0 2024-02-09T23:04:07.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d3afe1740 10 client.20184 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.415 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d3afe1740 10 client.20184 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:04:07.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d3afe1740 20 client.20184 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d225620 0x56034d227a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d3afe1740 4 client.20184 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:04:07.416 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56034ceaf720 con 0x56034d225620 2024-02-09T23:04:07.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d225620 0x56034d227a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d225620 0x56034d227a40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a2141674214e8cf9 server_cookie=ddab1d80a392fe28 in_seq=0 out_seq=0 2024-02-09T23:04:07.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.379+0000 7f5d19ffb640 10 client.20184 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:04:07.417 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1828041329 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56034ceaf720 con 0x56034d225620 2024-02-09T23:04:07.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d19ffb640 10 client.20184 handle_command_reply: tid=0 2024-02-09T23:04:07.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 client.20184 shutdown 2024-02-09T23:04:07.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 msgr2=0x56034d212490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.418 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d212490 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c00bad0 tx=0x7f5d2c00bf90 comp rx=0 tx=0).stop 2024-02-09T23:04:07.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 shutdown_connections 2024-02-09T23:04:07.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d225620 0x56034d227a40 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2129d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.419 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d19ffb640 0 client.20184 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:04:07.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d212490 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1828041329 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d211f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.420 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 >> 172.21.15.3:0/1828041329 conn(0x56034d206100 msgr2=0x56034d207e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 shutdown_connections 2024-02-09T23:04:07.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1828041329 wait complete. 2024-02-09T23:04:07.421 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.380+0000 7f5d3afe1740 20 client.20184 trim_cache size 0 max 16384 2024-02-09T23:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 Processor -- start 2024-02-09T23:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 -- start start 2024-02-09T23:04:07.433 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d214680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d201420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2127f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d0f8130 con 0x56034d20c3e0 2024-02-09T23:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d2073a0 con 0x56034d214f60 2024-02-09T23:04:07.434 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d201b00 con 0x56034d20ba30 2024-02-09T23:04:07.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d361b1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d201420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d361b1640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d201420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45784/0 (socket says 172.21.15.3:45784) 2024-02-09T23:04:07.435 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.392+0000 7f5d361b1640 1 -- 172.21.15.3:0/157245621 learned_addr learned my addr 172.21.15.3:0/157245621 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d361b1640 1 -- 172.21.15.3:0/157245621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 msgr2=0x56034d214680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d361b1640 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d214680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d361b1640 1 -- 172.21.15.3:0/157245621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 msgr2=0x56034d2127f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.436 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d361b1640 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2127f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d361b1640 1 -- 172.21.15.3:0/157245621 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56034d10dc90 con 0x56034d20c3e0 2024-02-09T23:04:07.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d361b1640 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d201420 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c004340 tx=0x7f5d2c005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9a277412c22da8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d1bfff640 1 -- 172.21.15.3:0/157245621 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c016070 con 0x56034d20c3e0 2024-02-09T23:04:07.437 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d1bfff640 1 -- 172.21.15.3:0/157245621 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d2c02b070 con 0x56034d20c3e0 2024-02-09T23:04:07.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d1bfff640 1 -- 172.21.15.3:0/157245621 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c0055a0 con 0x56034d20c3e0 2024-02-09T23:04:07.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d3afe1740 1 -- 172.21.15.3:0/157245621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 msgr2=0x56034d201420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.393+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d201420 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c004340 tx=0x7f5d2c005900 comp rx=0 tx=0).stop 2024-02-09T23:04:07.438 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- 172.21.15.3:0/157245621 shutdown_connections 2024-02-09T23:04:07.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d214f60 0x56034d2127f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20c3e0 0x56034d201420 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/157245621 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d20ba30 0x56034d214680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.439 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- 172.21.15.3:0/157245621 >> 172.21.15.3:0/157245621 conn(0x56034d206100 msgr2=0x56034d210820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- 172.21.15.3:0/157245621 shutdown_connections 2024-02-09T23:04:07.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- 172.21.15.3:0/157245621 wait complete. 2024-02-09T23:04:07.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- start start 2024-02-09T23:04:07.440 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 Processor -- start 2024-02-09T23:04:07.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 0x56034d21c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d21a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d214f60 0x56034d21afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.441 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d21b670 con 0x56034d20ba30 2024-02-09T23:04:07.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d27bb30 con 0x56034d20c3e0 2024-02-09T23:04:07.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56034d27bc70 con 0x56034d214f60 2024-02-09T23:04:07.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d3afe1740 10 client.0 _ll_register_callbacks cb 0x7f5d20009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d359b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 0x56034d21c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d359b0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 0x56034d21c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45788/0 (socket says 172.21.15.3:45788) 2024-02-09T23:04:07.442 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d359b0640 1 -- 172.21.15.3:0/1209913477 learned_addr learned my addr 172.21.15.3:0/1209913477 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.394+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d21a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d359b0640 1 -- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d214f60 msgr2=0x56034d21afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d214f60 0x56034d21afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d359b0640 1 -- 172.21.15.3:0/1209913477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 msgr2=0x56034d21a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d21a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d359b0640 1 -- 172.21.15.3:0/1209913477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56034d1716b0 con 0x56034d20ba30 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d21a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.443 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d359b0640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 0x56034d21c470 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005ec0 tx=0x7f5d200050d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1806b9ea669204 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1209913477 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d20058b80 con 0x56034d20ba30 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1209913477 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f5d20000e40 con 0x56034d20ba30 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d20004510 con 0x56034d20ba30 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1209913477 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d200056b0 con 0x56034d20ba30 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 10 client.20190 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x56034d133a70 con 0x56034d20ba30 2024-02-09T23:04:07.444 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1209913477 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d20064020 con 0x56034d20ba30 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1209913477 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5d20058400 con 0x56034d20ba30 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 10 client.20190 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 10 client.20190 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 20 client.20190 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d27d190 0x56034d27f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 4 client.20190 mds_command: new command op to 24496 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.395+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 --> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56034cefe890 con 0x56034d27d190 2024-02-09T23:04:07.445 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d27d190 0x56034d27f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d361b1640 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d27d190 0x56034d27f5b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a6db50ec75d6019 server_cookie=a02a7ab089127a00 in_seq=0 out_seq=0 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d19ffb640 10 client.20190 ms_handle_connect on v2:172.21.15.89:6838/305807401 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d19ffb640 1 -- 172.21.15.3:0/1209913477 <== mds.2 v2:172.21.15.89:6838/305807401 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56034cefe890 con 0x56034d27d190 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d19ffb640 10 client.20190 handle_command_reply: tid=0 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 client.20190 shutdown 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 msgr2=0x56034d21c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 0x56034d21c470 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f5d20005ec0 tx=0x7f5d200050d0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 shutdown_connections 2024-02-09T23:04:07.446 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:6838/305807401,v1:172.21.15.89:6839/305807401] conn(0x56034d27d190 0x56034d27f5b0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56034d20c3e0 0x56034d21a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d19ffb640 0 client.20190 ms_handle_reset on v2:172.21.15.89:6838/305807401 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56034d214f60 0x56034d21afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 --2- 172.21.15.3:0/1209913477 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56034d20ba30 0x56034d21c470 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 >> 172.21.15.3:0/1209913477 conn(0x56034d206100 msgr2=0x56034d22c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 shutdown_connections 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 1 -- 172.21.15.3:0/1209913477 wait complete. 2024-02-09T23:04:07.447 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.396+0000 7f5d3afe1740 20 client.20190 trim_cache size 0 max 16384 2024-02-09T23:04:07.448 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:04:07.449 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5d1c0a2230 msgr2=0x7f5d1c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5d1c0a2230 0x7f5d1c0a46f0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5d2805c070 tx=0x7f5d2806dca0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 msgr2=0x7f5d30197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 0x7f5d30197b60 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7f5d2800e960 tx=0x7f5d2800ee20 comp rx=0 tx=0).stop 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 shutdown_connections 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f5d1c0a2230 0x7f5d1c0a46f0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f5d301976e0 0x7f5d30197b60 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f5d30105410 0x7f5d3019db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.450 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 --2- 172.21.15.3:0/2715328839 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f5d30100340 0x7f5d3019d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 >> 172.21.15.3:0/2715328839 conn(0x7f5d300772e0 msgr2=0x7f5d300688a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 shutdown_connections 2024-02-09T23:04:07.451 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.397+0000 7f5d38415640 1 -- 172.21.15.3:0/2715328839 wait complete. 2024-02-09T23:04:07.451 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:04:07.451 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-02-09T23:04:07.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 -- 172.21.15.3:0/633917571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 msgr2=0x7f650810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 --2- 172.21.15.3:0/633917571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f650810cfe0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f6500009930 tx=0x7f650002bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.558 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 -- 172.21.15.3:0/633917571 shutdown_connections 2024-02-09T23:04:07.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 --2- 172.21.15.3:0/633917571 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f650810cfe0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 --2- 172.21.15.3:0/633917571 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6508100cf0 0x7f6508104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 --2- 172.21.15.3:0/633917571 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6508100340 0x7f6508100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 -- 172.21.15.3:0/633917571 >> 172.21.15.3:0/633917571 conn(0x7f65080772e0 msgr2=0x7f65080776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 -- 172.21.15.3:0/633917571 shutdown_connections 2024-02-09T23:04:07.559 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.555+0000 7f650c888640 1 -- 172.21.15.3:0/633917571 wait complete. 2024-02-09T23:04:07.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 Processor -- start 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 -- start start 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6508100340 0x7f65081986e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f6508198c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6508199200 0x7f65081a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65081a3e50 con 0x7f6508105410 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65081a3fc0 con 0x7f6508100340 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650c888640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65081a4180 con 0x7f6508199200 2024-02-09T23:04:07.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f6508198c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f6508198c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45806/0 (socket says 172.21.15.3:45806) 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 -- 172.21.15.3:0/458985733 learned_addr learned my addr 172.21.15.3:0/458985733 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6506d9d640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6508199200 0x7f65081a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f650659c640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6508100340 0x7f65081986e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 -- 172.21.15.3:0/458985733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6508199200 msgr2=0x7f65081a3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6508199200 0x7f65081a3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 -- 172.21.15.3:0/458985733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6508100340 msgr2=0x7f65081986e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6508100340 0x7f65081986e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.556+0000 7f6505d9b640 1 -- 172.21.15.3:0/458985733 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6500009560 con 0x7f6508105410 2024-02-09T23:04:07.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.557+0000 7f6505d9b640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f6508198c20 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00d740 tx=0x7f64fc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.557+0000 7f64f77fe640 1 -- 172.21.15.3:0/458985733 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc009c30 con 0x7f6508105410 2024-02-09T23:04:07.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.557+0000 7f64f77fe640 1 -- 172.21.15.3:0/458985733 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64fc00de50 con 0x7f6508105410 2024-02-09T23:04:07.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.557+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65081a4460 con 0x7f6508105410 2024-02-09T23:04:07.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.557+0000 7f64f77fe640 1 -- 172.21.15.3:0/458985733 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc010460 con 0x7f6508105410 2024-02-09T23:04:07.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.557+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65081a4920 con 0x7f6508105410 2024-02-09T23:04:07.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.559+0000 7f64f77fe640 1 -- 172.21.15.3:0/458985733 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f64fc004f10 con 0x7f6508105410 2024-02-09T23:04:07.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.559+0000 7f650f467740 1 Processor -- start 2024-02-09T23:04:07.571 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 -- start start 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfbfe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfcb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cf7b020 con 0x56160cfbfa30 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cf2e320 con 0x56160cfc8f60 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650f467740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cf2b850 con 0x56160cfc03e0 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f64f77fe640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f64f00a2230 0x7f64f00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.572 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfbfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfbfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45818/0 (socket says 172.21.15.3:45818) 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 -- 172.21.15.3:0/4202688515 learned_addr learned my addr 172.21.15.3:0/4202688515 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f64f00a2230 0x7f64f00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f64f77fe640 1 -- 172.21.15.3:0/458985733 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f64fc0cda30 con 0x7f6508105410 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650659c640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6505d9b640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfcb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 -- 172.21.15.3:0/4202688515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 msgr2=0x56160cfc8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.573 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 -- 172.21.15.3:0/4202688515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 msgr2=0x56160cfcb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfcb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6506d9d640 1 -- 172.21.15.3:0/4202688515 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56160cee9210 con 0x56160cfbfa30 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f650659c640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.560+0000 7f6505d9b640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfcb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f6506d9d640 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfbfe10 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6500009230 tx=0x7f650002cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6827da20804ff3a server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f6506d9d640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f64f00a2230 0x7f64f00a46f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f650003b000 tx=0x7f650004a000 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f64f4ff9640 1 -- 172.21.15.3:0/4202688515 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500048070 con 0x56160cfbfa30 2024-02-09T23:04:07.574 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f64f4ff9640 1 -- 172.21.15.3:0/4202688515 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f650002c900 con 0x56160cfbfa30 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f64f4ff9640 1 -- 172.21.15.3:0/4202688515 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500003830 con 0x56160cfbfa30 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 -- 172.21.15.3:0/4202688515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 msgr2=0x56160cfbfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfbfe10 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f6500009230 tx=0x7f650002cc40 comp rx=0 tx=0).stop 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 -- 172.21.15.3:0/4202688515 shutdown_connections 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfcb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 --2- 172.21.15.3:0/4202688515 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfbfe10 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.561+0000 7f650f467740 1 -- 172.21.15.3:0/4202688515 >> 172.21.15.3:0/4202688515 conn(0x56160cfba100 msgr2=0x56160cfba4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.575 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 -- 172.21.15.3:0/4202688515 shutdown_connections 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 -- 172.21.15.3:0/4202688515 wait complete. 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 -- start start 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 Processor -- start 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfbfa30 0x56160cfc5f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 0x56160cfc6490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfc69d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cf27030 con 0x56160cfc03e0 2024-02-09T23:04:07.576 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cbfa430 con 0x56160cfc8f60 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cbb6e00 con 0x56160cfbfa30 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650f467740 10 client.0 _ll_register_callbacks cb 0x56160cf1c030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f6505d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 0x56160cfc6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f6505d9b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 0x56160cfc6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45832/0 (socket says 172.21.15.3:45832) 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f6505d9b640 1 -- 172.21.15.3:0/2518361554 learned_addr learned my addr 172.21.15.3:0/2518361554 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f650659c640 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfbfa30 0x56160cfc5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.562+0000 7f6505d9b640 1 -- 172.21.15.3:0/2518361554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfbfa30 msgr2=0x56160cfc5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.577 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f6505d9b640 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfbfa30 0x56160cfc5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f6505d9b640 1 -- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 msgr2=0x56160cfc69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f6505d9b640 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfc69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f6505d9b640 1 -- 172.21.15.3:0/2518361554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f650810d7f0 con 0x56160cfc03e0 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f6505d9b640 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 0x56160cfc6490 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f64fc096860 tx=0x7f64fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=711232e9ff5231de server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f64edffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f64edffb640 1 -- 172.21.15.3:0/2518361554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64fc0b4060 con 0x56160cfc03e0 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f64edffb640 1 -- 172.21.15.3:0/2518361554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f64fc09e550 con 0x56160cfc03e0 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56160cc63720 con 0x56160cfc03e0 2024-02-09T23:04:07.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f64edffb640 1 -- 172.21.15.3:0/2518361554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64fc0a2ba0 con 0x56160cfc03e0 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 10 client.20202 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x56160cf256b0 con 0x56160cfc03e0 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f64edffb640 1 -- 172.21.15.3:0/2518361554 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56160cc63720 con 0x56160cfc03e0 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f64edffb640 1 -- 172.21.15.3:0/2518361554 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f64fc09e200 con 0x56160cfc03e0 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 10 client.20202 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 10 client.20202 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 20 client.20202 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160cfd9480 0x56160cfdb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 4 client.20202 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.563+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56160ccb2890 con 0x56160cfd9480 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.564+0000 7f650659c640 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160cfd9480 0x56160cfdb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.564+0000 7f64edffb640 10 client.20202 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.564+0000 7f650659c640 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160cfd9480 0x56160cfdb8a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=164c2e74edcba558 server_cookie=fc603324adfe8cb5 in_seq=0 out_seq=0 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f64edffb640 1 -- 172.21.15.3:0/2518361554 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56160ccb2890 con 0x56160cfd9480 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f64edffb640 10 client.20202 handle_command_reply: tid=0 2024-02-09T23:04:07.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 client.20202 shutdown 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 msgr2=0x56160cfc6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 0x56160cfc6490 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f64fc096860 tx=0x7f64fc0046c0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 shutdown_connections 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160cfd9480 0x56160cfdb8a0 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc8f60 0x56160cfc69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc03e0 0x56160cfc6490 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 --2- 172.21.15.3:0/2518361554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfbfa30 0x56160cfc5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 >> 172.21.15.3:0/2518361554 conn(0x56160cfba100 msgr2=0x56160cfbbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f64edffb640 0 client.20202 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:04:07.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 shutdown_connections 2024-02-09T23:04:07.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 1 -- 172.21.15.3:0/2518361554 wait complete. 2024-02-09T23:04:07.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.565+0000 7f650f467740 20 client.20202 trim_cache size 0 max 16384 2024-02-09T23:04:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 Processor -- start 2024-02-09T23:04:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 -- start start 2024-02-09T23:04:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfbfa30 0x56160cfc8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc67f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 0x56160cfc1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cfb58b0 con 0x56160cfc8f60 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160ceac130 con 0x56160cfbfa30 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f650f467740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cfb5a40 con 0x56160cfc03e0 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 0x56160cfc1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 0x56160cfc1250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45844/0 (socket says 172.21.15.3:45844) 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 -- 172.21.15.3:0/219249554 learned_addr learned my addr 172.21.15.3:0/219249554 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 -- 172.21.15.3:0/219249554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 msgr2=0x56160cfc67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 -- 172.21.15.3:0/219249554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfbfa30 msgr2=0x56160cfc8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfbfa30 0x56160cfc8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 -- 172.21.15.3:0/219249554 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56160cec1c90 con 0x56160cfc8f60 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.577+0000 7f6506d9d640 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 0x56160cfc1250 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f650002cde0 tx=0x7f6500009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1147c0d4366a8ca8 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f64effff640 1 -- 172.21.15.3:0/219249554 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500048050 con 0x56160cfc8f60 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f64effff640 1 -- 172.21.15.3:0/219249554 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6500049040 con 0x56160cfc8f60 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f64effff640 1 -- 172.21.15.3:0/219249554 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6500003ab0 con 0x56160cfc8f60 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 -- 172.21.15.3:0/219249554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 msgr2=0x56160cfc1250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 0x56160cfc1250 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7f650002cde0 tx=0x7f6500009260 comp rx=0 tx=0).stop 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 -- 172.21.15.3:0/219249554 shutdown_connections 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfc8f60 0x56160cfc1250 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc03e0 0x56160cfc67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 --2- 172.21.15.3:0/219249554 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfbfa30 0x56160cfc8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 -- 172.21.15.3:0/219249554 >> 172.21.15.3:0/219249554 conn(0x56160cfba100 msgr2=0x56160cfe0200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 -- 172.21.15.3:0/219249554 shutdown_connections 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 -- 172.21.15.3:0/219249554 wait complete. 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 -- start start 2024-02-09T23:04:07.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.578+0000 7f650f467740 1 Processor -- start 2024-02-09T23:04:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfcffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc03e0 0x56160cfce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc8f60 0x56160cfcea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cfd0850 con 0x56160cfbfa30 2024-02-09T23:04:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cfcef90 con 0x56160cfc03e0 2024-02-09T23:04:07.592 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x56160cfcf150 con 0x56160cfc8f60 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 10 client.0 _ll_register_callbacks cb 0x7f6500009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfcffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfcffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45848/0 (socket says 172.21.15.3:45848) 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 -- 172.21.15.3:0/1848463391 learned_addr learned my addr 172.21.15.3:0/1848463391 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f6506d9d640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc8f60 0x56160cfcea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 -- 172.21.15.3:0/1848463391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc8f60 msgr2=0x56160cfcea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc8f60 0x56160cfcea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.593 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f6505d9b640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc03e0 0x56160cfce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 -- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc03e0 msgr2=0x56160cfce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc03e0 0x56160cfce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 -- 172.21.15.3:0/1848463391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56160cee9210 con 0x56160cfbfa30 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f6506d9d640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc8f60 0x56160cfcea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f6505d9b640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc03e0 0x56160cfce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650659c640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfcffa0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x56160cfbf500 tx=0x7f64f8012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b7369e75ebf9d6 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f64edffb640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.594 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f64edffb640 1 -- 172.21.15.3:0/1848463391 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64f8017040 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f64edffb640 1 -- 172.21.15.3:0/1848463391 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f64f8019e40 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56160cc267e0 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.579+0000 7f64edffb640 1 -- 172.21.15.3:0/1848463391 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7f64f8004ab0 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 10 client.20208 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x56160cee7a70 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f64edffb640 1 -- 172.21.15.3:0/1848463391 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x56160cc267e0 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f64edffb640 1 -- 172.21.15.3:0/1848463391 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f64f8010c10 con 0x56160cfbfa30 2024-02-09T23:04:07.595 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 10 client.20208 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 10 client.20208 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.h 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 20 client.20208 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160d035200 0x56160d037620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 4 client.20208 mds_command: new command op to 24503 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 --> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56160cbd2700 con 0x56160d035200 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f6505d9b640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160d035200 0x56160d037620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f64edffb640 10 client.20208 ms_handle_connect on v2:172.21.15.64:6836/3217313563 2024-02-09T23:04:07.596 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.580+0000 7f6505d9b640 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160d035200 0x56160d037620 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fb2701b20f286735 server_cookie=d1ffb9b78ff7fd70 in_seq=0 out_seq=0 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f64edffb640 1 -- 172.21.15.3:0/1848463391 <== mds.1 v2:172.21.15.64:6836/3217313563 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56160cbd2700 con 0x56160d035200 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f64edffb640 10 client.20208 handle_command_reply: tid=0 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 client.20208 shutdown 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 msgr2=0x56160cfcffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfcffa0 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x56160cfbf500 tx=0x7f64f8012b00 comp rx=0 tx=0).stop 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 shutdown_connections 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:6836/3217313563,v1:172.21.15.64:6837/3217313563] conn(0x56160d035200 0x56160d037620 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.597 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x56160cfc03e0 0x56160cfce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f64edffb640 0 client.20208 ms_handle_reset on v2:172.21.15.64:6836/3217313563 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x56160cfc8f60 0x56160cfcea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 --2- 172.21.15.3:0/1848463391 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x56160cfbfa30 0x56160cfcffa0 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 >> 172.21.15.3:0/1848463391 conn(0x56160cfba100 msgr2=0x56160cfc43f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 shutdown_connections 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 1 -- 172.21.15.3:0/1848463391 wait complete. 2024-02-09T23:04:07.598 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650f467740 20 client.20208 trim_cache size 0 max 16384 2024-02-09T23:04:07.599 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:04:07.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f64f00a2230 msgr2=0x7f64f00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f64f00a2230 0x7f64f00a46f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f650003b000 tx=0x7f650004a000 comp rx=0 tx=0).stop 2024-02-09T23:04:07.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 msgr2=0x7f6508198c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.600 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f6508198c20 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f64fc00d740 tx=0x7f64fc00dc00 comp rx=0 tx=0).stop 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 shutdown_connections 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f64f00a2230 0x7f64f00a46f0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6508199200 0x7f65081a3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6508105410 0x7f6508198c20 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 --2- 172.21.15.3:0/458985733 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6508100340 0x7f65081986e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 >> 172.21.15.3:0/458985733 conn(0x7f65080772e0 msgr2=0x7f65081065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.581+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 shutdown_connections 2024-02-09T23:04:07.601 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.582+0000 7f650c888640 1 -- 172.21.15.3:0/458985733 wait complete. 2024-02-09T23:04:07.602 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-02-09T23:04:07.602 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-02-09T23:04:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 -- 172.21.15.3:0/405618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd174101430 msgr2=0x7fd174101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 --2- 172.21.15.3:0/405618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd174101430 0x7fd174101810 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7fd164009930 tx=0x7fd16402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 -- 172.21.15.3:0/405618070 shutdown_connections 2024-02-09T23:04:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 --2- 172.21.15.3:0/405618070 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd174101de0 0x7fd174105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.709 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 --2- 172.21.15.3:0/405618070 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd174106500 0x7fd17410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 --2- 172.21.15.3:0/405618070 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd174101430 0x7fd174101810 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 -- 172.21.15.3:0/405618070 >> 172.21.15.3:0/405618070 conn(0x7fd1740fccd0 msgr2=0x7fd1740ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 -- 172.21.15.3:0/405618070 shutdown_connections 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 -- 172.21.15.3:0/405618070 wait complete. 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 Processor -- start 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.704+0000 7fd17a525640 1 -- start start 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd17a525640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd174101430 0x7fd17419d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd17a525640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd174106500 0x7fd17419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.710 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd17a525640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 0x7fd174197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd17a525640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd174198170 con 0x7fd1741976a0 2024-02-09T23:04:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd17a525640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741982e0 con 0x7fd174101430 2024-02-09T23:04:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd17a525640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1741984a0 con 0x7fd174106500 2024-02-09T23:04:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 0x7fd174197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 0x7fd174197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45874/0 (socket says 172.21.15.3:45874) 2024-02-09T23:04:07.711 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 -- 172.21.15.3:0/1443928900 learned_addr learned my addr 172.21.15.3:0/1443928900 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd1737fe640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd174106500 0x7fd17419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd173fff640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd174101430 0x7fd17419d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 -- 172.21.15.3:0/1443928900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd174106500 msgr2=0x7fd17419db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd174106500 0x7fd17419db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 -- 172.21.15.3:0/1443928900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd174101430 msgr2=0x7fd17419d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd174101430 0x7fd17419d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 -- 172.21.15.3:0/1443928900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd164009560 con 0x7fd1741976a0 2024-02-09T23:04:07.712 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd178ac2640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 0x7fd174197b20 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009fd0 tx=0x7fd16c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.705+0000 7fd1717fa640 1 -- 172.21.15.3:0/1443928900 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c00bea0 con 0x7fd1741976a0 2024-02-09T23:04:07.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.706+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1741a4440 con 0x7fd1741976a0 2024-02-09T23:04:07.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.706+0000 7fd1717fa640 1 -- 172.21.15.3:0/1443928900 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd16c009940 con 0x7fd1741976a0 2024-02-09T23:04:07.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.706+0000 7fd1717fa640 1 -- 172.21.15.3:0/1443928900 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16c0106c0 con 0x7fd1741976a0 2024-02-09T23:04:07.713 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.706+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1741a48d0 con 0x7fd1741976a0 2024-02-09T23:04:07.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1717fa640 1 -- 172.21.15.3:0/1443928900 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fd16c004240 con 0x7fd1741976a0 2024-02-09T23:04:07.719 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 Processor -- start 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 -- start start 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcaa30 0x55a78ddcae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd3990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddd63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dd86020 con 0x55a78ddd3f60 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dd39320 con 0x55a78ddcaa30 2024-02-09T23:04:07.720 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dd36850 con 0x55a78ddcb3e0 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1717fa640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd15c0a2230 0x7fd15c0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddd63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd178ac2640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd15c0a2230 0x7fd15c0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddd63b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45876/0 (socket says 172.21.15.3:45876) 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1737fe640 1 -- 172.21.15.3:0/1812517028 learned_addr learned my addr 172.21.15.3:0/1812517028 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1737fe640 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcaa30 0x55a78ddcae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.708+0000 7fd1717fa640 1 -- 172.21.15.3:0/1443928900 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fd16c0cde60 con 0x7fd1741976a0 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd173fff640 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.721 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd1737fe640 1 -- 172.21.15.3:0/1812517028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 msgr2=0x55a78ddd3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd1737fe640 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd1737fe640 1 -- 172.21.15.3:0/1812517028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcaa30 msgr2=0x55a78ddcae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd1737fe640 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcaa30 0x55a78ddcae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd1737fe640 1 -- 172.21.15.3:0/1812517028 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a78dcf4210 con 0x55a78ddd3f60 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd178ac2640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd15c0a2230 0x7fd15c0a46f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd16c005c40 tx=0x7fd16c004640 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd1737fe640 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddd63b0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd168014990 tx=0x7fd16800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a34560e5b2aa839 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd15a7fc640 1 -- 172.21.15.3:0/1812517028 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1680142d0 con 0x55a78ddd3f60 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd15a7fc640 1 -- 172.21.15.3:0/1812517028 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd16801a020 con 0x55a78ddd3f60 2024-02-09T23:04:07.722 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.709+0000 7fd15a7fc640 1 -- 172.21.15.3:0/1812517028 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168012840 con 0x55a78ddd3f60 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 -- 172.21.15.3:0/1812517028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 msgr2=0x55a78ddd63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddd63b0 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fd168014990 tx=0x7fd16800c490 comp rx=0 tx=0).stop 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 -- 172.21.15.3:0/1812517028 shutdown_connections 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddd63b0 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/1812517028 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcaa30 0x55a78ddcae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 -- 172.21.15.3:0/1812517028 >> 172.21.15.3:0/1812517028 conn(0x55a78ddc5100 msgr2=0x55a78ddc54e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.723 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 -- 172.21.15.3:0/1812517028 shutdown_connections 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 -- 172.21.15.3:0/1812517028 wait complete. 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 -- start start 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 Processor -- start 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 0x55a78ddd0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.710+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dd32030 con 0x55a78ddcaa30 2024-02-09T23:04:07.724 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78da05430 con 0x55a78ddd3f60 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78d9c1e00 con 0x55a78ddcb3e0 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 10 client.0 _ll_register_callbacks cb 0x55a78dd27030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 0x55a78ddd0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 0x55a78ddd0f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45886/0 (socket says 172.21.15.3:45886) 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 -- 172.21.15.3:0/3466608804 learned_addr learned my addr 172.21.15.3:0/3466608804 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd173fff640 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 -- 172.21.15.3:0/3466608804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 msgr2=0x55a78ddd1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.725 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 -- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 msgr2=0x55a78ddd19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 -- 172.21.15.3:0/3466608804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd168014640 con 0x55a78ddcaa30 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1737fe640 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 0x55a78ddd0f30 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd16800c430 tx=0x7fd1680124c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee28f75a4e6e9617 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1537fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1537fe640 1 -- 172.21.15.3:0/3466608804 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168004230 con 0x55a78ddcaa30 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1537fe640 1 -- 172.21.15.3:0/3466608804 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fd16801b030 con 0x55a78ddcaa30 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a78da317e0 con 0x55a78ddcaa30 2024-02-09T23:04:07.726 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd1537fe640 1 -- 172.21.15.3:0/3466608804 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd168004390 con 0x55a78ddcaa30 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 10 client.20220 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.711+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55a78dd306b0 con 0x55a78ddcaa30 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd1537fe640 1 -- 172.21.15.3:0/3466608804 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55a78da317e0 con 0x55a78ddcaa30 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd1537fe640 1 -- 172.21.15.3:0/3466608804 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd168003b40 con 0x55a78ddcaa30 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd17d1d3740 10 client.20220 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd17d1d3740 10 client.20220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd17d1d3740 20 client.20220 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.727 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78dde43b0 0x55a78dde67a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd17d1d3740 4 client.20220 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a78da6e720 con 0x55a78dde43b0 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd173fff640 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78dde43b0 0x55a78dde67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd1537fe640 10 client.20220 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.712+0000 7fd173fff640 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78dde43b0 0x55a78dde67a0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e92381be8fc562b server_cookie=8305b9444d77ab98 in_seq=0 out_seq=0 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd1537fe640 1 -- 172.21.15.3:0/3466608804 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a78da6e720 con 0x55a78dde43b0 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd1537fe640 10 client.20220 handle_command_reply: tid=0 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 client.20220 shutdown 2024-02-09T23:04:07.728 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 msgr2=0x55a78ddd0f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 0x55a78ddd0f30 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fd16800c430 tx=0x7fd1680124c0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 shutdown_connections 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78dde43b0 0x55a78dde67a0 crc :-1 s=READY pgs=349 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/3466608804 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcaa30 0x55a78ddd0f30 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd1537fe640 0 client.20220 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:07.729 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 >> 172.21.15.3:0/3466608804 conn(0x55a78ddc5100 msgr2=0x55a78ddc6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 shutdown_connections 2024-02-09T23:04:07.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 1 -- 172.21.15.3:0/3466608804 wait complete. 2024-02-09T23:04:07.730 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.713+0000 7fd17d1d3740 20 client.20220 trim_cache size 0 max 16384 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 Processor -- start 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 -- start start 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78ddd3660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcb3e0 0x55a78ddcf800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddcfd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dcb7130 con 0x55a78ddd3f60 2024-02-09T23:04:07.736 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78ddc0ab0 con 0x55a78ddcb3e0 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78dde1020 con 0x55a78ddcaa30 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.725+0000 7fd1737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddcfd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddcfd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45896/0 (socket says 172.21.15.3:45896) 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 -- 172.21.15.3:0/2680354880 learned_addr learned my addr 172.21.15.3:0/2680354880 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd173fff640 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcb3e0 0x55a78ddcf800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78ddd3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 -- 172.21.15.3:0/2680354880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 msgr2=0x55a78ddd3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.737 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78ddd3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 -- 172.21.15.3:0/2680354880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcb3e0 msgr2=0x55a78ddcf800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcb3e0 0x55a78ddcf800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 -- 172.21.15.3:0/2680354880 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a78dcccc90 con 0x55a78ddd3f60 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddcfd40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fd1680059f0 tx=0x7fd168012450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0b5e04cc68dc96 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd159ffb640 1 -- 172.21.15.3:0/2680354880 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168012530 con 0x55a78ddd3f60 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd159ffb640 1 -- 172.21.15.3:0/2680354880 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd168026070 con 0x55a78ddd3f60 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.726+0000 7fd159ffb640 1 -- 172.21.15.3:0/2680354880 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd168003fd0 con 0x55a78ddd3f60 2024-02-09T23:04:07.738 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2680354880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 msgr2=0x55a78ddcfd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddcfd40 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7fd1680059f0 tx=0x7fd168012450 comp rx=0 tx=0).stop 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2680354880 shutdown_connections 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddcb3e0 0x55a78ddcf800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddd3f60 0x55a78ddcfd40 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2680354880 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78ddd3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2680354880 >> 172.21.15.3:0/2680354880 conn(0x55a78ddc5100 msgr2=0x55a78ddeb0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2680354880 shutdown_connections 2024-02-09T23:04:07.739 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2680354880 wait complete. 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- start start 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 Processor -- start 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78dddb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78ddd9e90 con 0x55a78ddcb3e0 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.727+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78ddda000 con 0x55a78ddd3f60 2024-02-09T23:04:07.740 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd17d1d3740 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a78ddda1c0 con 0x55a78ddcaa30 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd17d1d3740 10 client.0 _ll_register_callbacks cb 0x7fd16800ca50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45906/0 (socket says 172.21.15.3:45906) 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 -- 172.21.15.3:0/2503531479 learned_addr learned my addr 172.21.15.3:0/2503531479 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78dddb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 -- 172.21.15.3:0/2503531479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 msgr2=0x55a78dddb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78dddb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.741 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 -- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 msgr2=0x55a78ddd9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 -- 172.21.15.3:0/2503531479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a78dd306b0 con 0x55a78ddcb3e0 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd173fff640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd9410 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fd164000c00 tx=0x7fd164004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12807f829a64f7da server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1537fe640 10 client.0 ms_handle_connect on v2:172.21.15.3:3300/0 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1537fe640 1 -- 172.21.15.3:0/2503531479 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164004040 con 0x55a78ddcb3e0 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1537fe640 1 -- 172.21.15.3:0/2503531479 <== mon.0 v2:172.21.15.3:3300/0 2 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7fd164043040 con 0x55a78ddcb3e0 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd16801a020 con 0x55a78ddcb3e0 2024-02-09T23:04:07.742 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1537fe640 1 -- 172.21.15.3:0/2503531479 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_get_version_reply(handle=1 version=55) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1640041a0 con 0x55a78ddcb3e0 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd17d1d3740 10 client.20226 fetch_fsmap learned FSMap version 55 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,fsmap=55,monmap=4+}) v3 -- 0x55a78dcf2a70 con 0x55a78ddcb3e0 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.728+0000 7fd1537fe640 1 -- 172.21.15.3:0/2503531479 <== mon.0 v2:172.21.15.3:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd16801a020 con 0x55a78ddcb3e0 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd1537fe640 1 -- 172.21.15.3:0/2503531479 <== mon.0 v2:172.21.15.3:3300/0 5 ==== fsmap(e 55) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd164046760 con 0x55a78ddcb3e0 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd17d1d3740 10 client.20226 fetch_fsmap finished waiting for FSMap version 55 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd17d1d3740 10 client.20226 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd17d1d3740 20 client.20226 populate_metadata read hostname 'smithi003' 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78de3be00 0x55a78de3e220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:07.743 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd17d1d3740 4 client.20226 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 --> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55a78dabd890 con 0x55a78de3be00 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78de3be00 0x55a78de3e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd1737fe640 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78de3be00 0x55a78de3e220 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7399e25d2952c4b4 server_cookie=4d72e5ac2ebebdc2 in_seq=0 out_seq=0 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.729+0000 7fd1537fe640 10 client.20226 ms_handle_connect on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd1537fe640 1 -- 172.21.15.3:0/2503531479 <== mds.0 v2:172.21.15.64:6838/4207075446 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55a78dabd890 con 0x55a78de3be00 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd1537fe640 10 client.20226 handle_command_reply: tid=0 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 client.20226 shutdown 2024-02-09T23:04:07.744 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 msgr2=0x55a78ddd9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd9410 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7fd164000c00 tx=0x7fd164004a80 comp rx=0 tx=0).stop 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 shutdown_connections 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:6838/4207075446,v1:172.21.15.64:6839/4207075446] conn(0x55a78de3be00 0x55a78de3e220 crc :-1 s=READY pgs=350 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x55a78ddd3f60 0x55a78ddd9950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x55a78ddcb3e0 0x55a78ddd9410 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd1537fe640 0 client.20226 ms_handle_reset on v2:172.21.15.64:6838/4207075446 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 --2- 172.21.15.3:0/2503531479 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x55a78ddcaa30 0x55a78dddb070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.745 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 >> 172.21.15.3:0/2503531479 conn(0x55a78ddc5100 msgr2=0x55a78ddcf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 shutdown_connections 2024-02-09T23:04:07.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 1 -- 172.21.15.3:0/2503531479 wait complete. 2024-02-09T23:04:07.746 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17d1d3740 20 client.20226 trim_cache size 0 max 16384 2024-02-09T23:04:07.746 INFO:teuthology.orchestra.run.smithi003.stdout:[] 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd15c0a2230 msgr2=0x7fd15c0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17a525640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd15c0a2230 0x7fd15c0a46f0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd16c005c40 tx=0x7fd16c004640 comp rx=0 tx=0).stop 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 msgr2=0x7fd174197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.730+0000 7fd17a525640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 0x7fd174197b20 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fd16c009fd0 tx=0x7fd16c00b260 comp rx=0 tx=0).stop 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 shutdown_connections 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fd15c0a2230 0x7fd15c0a46f0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.748 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fd1741976a0 0x7fd174197b20 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fd174106500 0x7fd17419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 --2- 172.21.15.3:0/1443928900 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fd174101430 0x7fd17419d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 >> 172.21.15.3:0/1443928900 conn(0x7fd1740fccd0 msgr2=0x7fd1740fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 shutdown_connections 2024-02-09T23:04:07.749 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:07.731+0000 7fd17a525640 1 -- 172.21.15.3:0/1443928900 wait complete. 2024-02-09T23:04:07.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:07 smithi003 ceph-mon[32678]: pgmap v405: 97 pgs: 97 active+clean; 209 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 3.6 MiB/s wr, 188 op/s 2024-02-09T23:04:07.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:07 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2250124775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:07.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:07 smithi064 ceph-mon[36638]: pgmap v405: 97 pgs: 97 active+clean; 209 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 3.6 MiB/s wr, 188 op/s 2024-02-09T23:04:07.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:07 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2250124775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:07.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:07 smithi089 ceph-mon[36665]: pgmap v405: 97 pgs: 97 active+clean; 209 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 3.6 MiB/s wr, 188 op/s 2024-02-09T23:04:07.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:07 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2250124775' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:08.750 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-02-09T23:04:08.750 INFO:tasks.fwd_scrub:done joining 2024-02-09T23:04:08.750 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-02-09T23:04:08.761 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T23:04:08.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 -- 172.21.15.3:0/163304541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0101430 msgr2=0x7f56e0101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:08.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 --2- 172.21.15.3:0/163304541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0101430 0x7f56e0101810 secure :-1 s=READY pgs=1802 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009930 tx=0x7f56d402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 -- 172.21.15.3:0/163304541 shutdown_connections 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 --2- 172.21.15.3:0/163304541 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e0106500 0x7f56e010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 --2- 172.21.15.3:0/163304541 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e0101de0 0x7f56e0105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 --2- 172.21.15.3:0/163304541 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0101430 0x7f56e0101810 unknown :-1 s=CLOSED pgs=1802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 -- 172.21.15.3:0/163304541 >> 172.21.15.3:0/163304541 conn(0x7f56e00fccd0 msgr2=0x7f56e00ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 -- 172.21.15.3:0/163304541 shutdown_connections 2024-02-09T23:04:08.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.872+0000 7f56e6c98640 1 -- 172.21.15.3:0/163304541 wait complete. 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 Processor -- start 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 -- start start 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e0101430 0x7f56e019d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 0x7f56e019dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e0197800 0x7f56e0197cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e01982b0 con 0x7f56e0106500 2024-02-09T23:04:08.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e0198420 con 0x7f56e0101430 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e6c98640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e0198590 con 0x7f56e0197800 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56dffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 0x7f56e019dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56dffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 0x7f56e019dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45938/0 (socket says 172.21.15.3:45938) 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56dffff640 1 -- 172.21.15.3:0/3211235012 learned_addr learned my addr 172.21.15.3:0/3211235012 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e5235640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e0197800 0x7f56e0197cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.873+0000 7f56e4a34640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e0101430 0x7f56e019d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56dffff640 1 -- 172.21.15.3:0/3211235012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e0197800 msgr2=0x7f56e0197cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56dffff640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e0197800 0x7f56e0197cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:08.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56dffff640 1 -- 172.21.15.3:0/3211235012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e0101430 msgr2=0x7f56e019d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56dffff640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e0101430 0x7f56e019d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56dffff640 1 -- 172.21.15.3:0/3211235012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d4009560 con 0x7f56e0106500 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56dffff640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 0x7f56e019dbd0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00b450 tx=0x7f56cc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc011890 con 0x7f56e0106500 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56cc0103e0 con 0x7f56e0106500 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e01a4540 con 0x7f56e0106500 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00f580 con 0x7f56e0106500 2024-02-09T23:04:08.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.874+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e01a4a20 con 0x7f56e0106500 2024-02-09T23:04:08.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.875+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e0102900 con 0x7f56e0106500 2024-02-09T23:04:08.882 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.876+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f56cc0108e0 con 0x7f56e0106500 2024-02-09T23:04:08.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.877+0000 7f56ddffb640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56d00a2230 0x7f56d00a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:08.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.877+0000 7f56e4a34640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56d00a2230 0x7f56d00a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:08.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.877+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f56cc0cdd30 con 0x7f56e0106500 2024-02-09T23:04:08.883 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.878+0000 7f56e4a34640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56d00a2230 0x7f56d00a46f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f56d4005ab0 tx=0x7f56d4005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:08.887 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:08.882+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f56cc096a50 con 0x7f56e0106500 2024-02-09T23:04:09.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.064+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56e0109730 con 0x7f56e0106500 2024-02-09T23:04:09.068 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.065+0000 7f56ddffb640 1 -- 172.21.15.3:0/3211235012 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+22990 (secure 0 0 0) 0x7f56cc09a110 con 0x7f56e0106500 2024-02-09T23:04:09.068 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:09.069 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":105,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T23:04:02.514620+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/284621866":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6800/284621866":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/3470258541":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2951038011":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2619736972":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2265484823":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T23:04:09.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56d00a2230 msgr2=0x7f56d00a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.070 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56d00a2230 0x7f56d00a46f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f56d4005ab0 tx=0x7f56d4005fd0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 msgr2=0x7f56e019dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 0x7f56e019dbd0 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00b450 tx=0x7f56cc00b910 comp rx=0 tx=0).stop 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 shutdown_connections 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f56d00a2230 0x7f56d00a46f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f56e0197800 0x7f56e0197cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f56e0106500 0x7f56e019dbd0 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 --2- 172.21.15.3:0/3211235012 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f56e0101430 0x7f56e019d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.071 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 >> 172.21.15.3:0/3211235012 conn(0x7f56e00fccd0 msgr2=0x7f56e00fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:09.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 shutdown_connections 2024-02-09T23:04:09.072 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.067+0000 7f56e6c98640 1 -- 172.21.15.3:0/3211235012 wait complete. 2024-02-09T23:04:09.082 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T23:04:09.083 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T23:04:09.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- 172.21.15.3:0/2169040271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc100cf0 msgr2=0x7f97dc104d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- 172.21.15.3:0/2169040271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc100cf0 0x7f97dc104d30 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f97d0009930 tx=0x7f97d002bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:09.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- 172.21.15.3:0/2169040271 shutdown_connections 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- 172.21.15.3:0/2169040271 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97dc105410 0x7f97dc10cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- 172.21.15.3:0/2169040271 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc100cf0 0x7f97dc104d30 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- 172.21.15.3:0/2169040271 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97dc100340 0x7f97dc100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- 172.21.15.3:0/2169040271 >> 172.21.15.3:0/2169040271 conn(0x7f97dc0772e0 msgr2=0x7f97dc0776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- 172.21.15.3:0/2169040271 shutdown_connections 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- 172.21.15.3:0/2169040271 wait complete. 2024-02-09T23:04:09.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 Processor -- start 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- start start 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97dc100340 0x7f97dc1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97dc105410 0x7f97dc198c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 0x7f97dc1a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1a3e40 con 0x7f97dc1991c0 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1a3fb0 con 0x7f97dc100340 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.186+0000 7f97e2c64640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97dc1a4170 con 0x7f97dc105410 2024-02-09T23:04:09.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 0x7f97dc1a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 0x7f97dc1a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45958/0 (socket says 172.21.15.3:45958) 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 -- 172.21.15.3:0/573171905 learned_addr learned my addr 172.21.15.3:0/573171905 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e0a00640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97dc100340 0x7f97dc1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97dbfff640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97dc105410 0x7f97dc198c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 -- 172.21.15.3:0/573171905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97dc105410 msgr2=0x7f97dc198c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97dc105410 0x7f97dc198c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 -- 172.21.15.3:0/573171905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97dc100340 msgr2=0x7f97dc1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.194 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97dc100340 0x7f97dc1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 -- 172.21.15.3:0/573171905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d0009560 con 0x7f97dc1991c0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e1201640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 0x7f97dc1a3770 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c720 tx=0x7f97d400cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4007bb0 con 0x7f97dc1991c0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97dc1a4450 con 0x7f97dc1991c0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97d400ce30 con 0x7f97dc1991c0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d4004db0 con 0x7f97dc1991c0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.187+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97dc1a4930 con 0x7f97dc1991c0 2024-02-09T23:04:09.195 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.188+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc0680b0 con 0x7f97dc1991c0 2024-02-09T23:04:09.196 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.189+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f97d4005020 con 0x7f97dc1991c0 2024-02-09T23:04:09.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.190+0000 7f97d9ffb640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f97c40a2230 0x7f97c40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.190+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 8557+0+0 (secure 0 0 0) 0x7f97d40cdb90 con 0x7f97dc1991c0 2024-02-09T23:04:09.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.190+0000 7f97e0a00640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f97c40a2230 0x7f97c40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.190+0000 7f97e0a00640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f97c40a2230 0x7f97c40a46f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f97cc005fd0 tx=0x7f97cc009680 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:09.197 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.194+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f97d40968b0 con 0x7f97dc1991c0 2024-02-09T23:04:09.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.380+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f97dc108640 con 0x7f97dc1991c0 2024-02-09T23:04:09.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.380+0000 7f97d9ffb640 1 -- 172.21.15.3:0/573171905 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f97d4099f70 con 0x7f97dc1991c0 2024-02-09T23:04:09.383 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T23:04:09.385 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f97c40a2230 msgr2=0x7f97c40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f97c40a2230 0x7f97c40a46f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f97cc005fd0 tx=0x7f97cc009680 comp rx=0 tx=0).stop 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 msgr2=0x7f97dc1a3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 0x7f97dc1a3770 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f97d400c720 tx=0x7f97d400cbe0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 shutdown_connections 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f97c40a2230 0x7f97c40a46f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.382+0000 7f97e2c64640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f97dc1991c0 0x7f97dc1a3770 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.383+0000 7f97e2c64640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f97dc105410 0x7f97dc198c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.386 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.383+0000 7f97e2c64640 1 --2- 172.21.15.3:0/573171905 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f97dc100340 0x7f97dc1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.383+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 >> 172.21.15.3:0/573171905 conn(0x7f97dc0772e0 msgr2=0x7f97dc0fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.383+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 shutdown_connections 2024-02-09T23:04:09.387 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.383+0000 7f97e2c64640 1 -- 172.21.15.3:0/573171905 wait complete. 2024-02-09T23:04:09.406 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T23:04:09.514 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 -- 172.21.15.3:0/61288230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 msgr2=0x7f6fec100ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 --2- 172.21.15.3:0/61288230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec100ee0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009930 tx=0x7f6fe802bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 -- 172.21.15.3:0/61288230 shutdown_connections 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 --2- 172.21.15.3:0/61288230 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6fec1014b0 0x7f6fec104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 --2- 172.21.15.3:0/61288230 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec105370 0x7f6fec10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 --2- 172.21.15.3:0/61288230 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec100ee0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 -- 172.21.15.3:0/61288230 >> 172.21.15.3:0/61288230 conn(0x7f6fec076630 msgr2=0x7f6fec0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:09.515 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 -- 172.21.15.3:0/61288230 shutdown_connections 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 -- 172.21.15.3:0/61288230 wait complete. 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 Processor -- start 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.510+0000 7f6ff380f640 1 -- start start 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff380f640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec0fba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff380f640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec105370 0x7f6fec0fbfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff380f640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6fec0fc4f0 0x7f6fec1a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff380f640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec1a6070 con 0x7f6fec100b00 2024-02-09T23:04:09.516 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff380f640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec1a61e0 con 0x7f6fec0fc4f0 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff380f640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fec1a63a0 con 0x7f6fec105370 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec0fba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec0fba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45976/0 (socket says 172.21.15.3:45976) 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 -- 172.21.15.3:0/2571480236 learned_addr learned my addr 172.21.15.3:0/2571480236 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff1dac640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6fec0fc4f0 0x7f6fec1a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff0daa640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec105370 0x7f6fec0fbfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 -- 172.21.15.3:0/2571480236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec105370 msgr2=0x7f6fec0fbfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.517 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec105370 0x7f6fec0fbfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 -- 172.21.15.3:0/2571480236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6fec0fc4f0 msgr2=0x7f6fec1a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6fec0fc4f0 0x7f6fec1a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 -- 172.21.15.3:0/2571480236 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fe8009560 con 0x7f6fec100b00 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6ff15ab640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec0fba70 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009230 tx=0x7f6fe8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.511+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8004440 con 0x7f6fec100b00 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.512+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6fe80052d0 con 0x7f6fec100b00 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.512+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fec1a6620 con 0x7f6fec100b00 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.512+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8003d70 con 0x7f6fec100b00 2024-02-09T23:04:09.518 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.512+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fec1a6b30 con 0x7f6fec100b00 2024-02-09T23:04:09.519 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.512+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fec0fcfa0 con 0x7f6fec100b00 2024-02-09T23:04:09.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.513+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6fe8005440 con 0x7f6fec100b00 2024-02-09T23:04:09.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.514+0000 7f6fe27fc640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6fd80a2260 0x7f6fd80a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.514+0000 7f6ff0daa640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6fd80a2260 0x7f6fd80a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.514+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6fe80f0110 con 0x7f6fec100b00 2024-02-09T23:04:09.520 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.515+0000 7f6ff0daa640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6fd80a2260 0x7f6fd80a4720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0fd500 tx=0x7f6fdc00b390 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:09.521 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.518+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6fe80b8eb0 con 0x7f6fec100b00 2024-02-09T23:04:09.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.700+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6fec1014b0 con 0x7f6fec100b00 2024-02-09T23:04:09.703 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.700+0000 7f6fe27fc640 1 -- 172.21.15.3:0/2571480236 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6fe80bc570 con 0x7f6fec100b00 2024-02-09T23:04:09.703 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T23:04:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6fd80a2260 msgr2=0x7f6fd80a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.705 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6fd80a2260 0x7f6fd80a4720 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6fec0fd500 tx=0x7f6fdc00b390 comp rx=0 tx=0).stop 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 msgr2=0x7f6fec0fba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec0fba70 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7f6fe8009230 tx=0x7f6fe8004040 comp rx=0 tx=0).stop 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 shutdown_connections 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6fd80a2260 0x7f6fd80a4720 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f6fec0fc4f0 0x7f6fec1a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f6fec105370 0x7f6fec0fbfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 --2- 172.21.15.3:0/2571480236 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f6fec100b00 0x7f6fec0fba70 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.706 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 >> 172.21.15.3:0/2571480236 conn(0x7f6fec076630 msgr2=0x7f6fec0ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 shutdown_connections 2024-02-09T23:04:09.707 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.702+0000 7f6ff380f640 1 -- 172.21.15.3:0/2571480236 wait complete. 2024-02-09T23:04:09.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:09 smithi003 ceph-mon[32678]: pgmap v406: 97 pgs: 97 active+clean; 209 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 3.6 MiB/s wr, 188 op/s 2024-02-09T23:04:09.715 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3211235012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:09.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:09 smithi003 ceph-mon[32678]: osdmap e106: 12 total, 12 up, 12 in 2024-02-09T23:04:09.716 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:09 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/573171905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T23:04:09.717 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T23:04:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:09 smithi089 ceph-mon[36665]: pgmap v406: 97 pgs: 97 active+clean; 209 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 3.6 MiB/s wr, 188 op/s 2024-02-09T23:04:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3211235012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:09 smithi089 ceph-mon[36665]: osdmap e106: 12 total, 12 up, 12 in 2024-02-09T23:04:09.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:09 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/573171905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T23:04:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 -- 172.21.15.3:0/3401707017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c1065a0 msgr2=0x7f7f1c10e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 --2- 172.21.15.3:0/3401707017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c10e120 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f7f14009cd0 tx=0x7f7f1402c0f0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 -- 172.21.15.3:0/3401707017 shutdown_connections 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 --2- 172.21.15.3:0/3401707017 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c10e120 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 --2- 172.21.15.3:0/3401707017 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c100340 0x7f7f1c105e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 --2- 172.21.15.3:0/3401707017 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c0ffd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 -- 172.21.15.3:0/3401707017 >> 172.21.15.3:0/3401707017 conn(0x7f7f1c076620 msgr2=0x7f7f1c0fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 -- 172.21.15.3:0/3401707017 shutdown_connections 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.820+0000 7f7f24113640 1 -- 172.21.15.3:0/3401707017 wait complete. 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 Processor -- start 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 -- start start 2024-02-09T23:04:09.826 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c19d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c19daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c1976c0 0x7f7f1c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c198190 con 0x7f7f1c0ff990 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c198300 con 0x7f7f1c1976c0 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f24113640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f1c1984c0 con 0x7f7f1c1065a0 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c19d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c19d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45994/0 (socket says 172.21.15.3:45994) 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 -- 172.21.15.3:0/2848796852 learned_addr learned my addr 172.21.15.3:0/2848796852 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:09.827 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f226b0640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c1976c0 0x7f7f1c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f216ae640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c19daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 -- 172.21.15.3:0/2848796852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c1065a0 msgr2=0x7f7f1c19daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c19daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 -- 172.21.15.3:0/2848796852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c1976c0 msgr2=0x7f7f1c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c1976c0 0x7f7f1c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 -- 172.21.15.3:0/2848796852 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f14009900 con 0x7f7f1c0ff990 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f216ae640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:09.828 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f226b0640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c1976c0 0x7f7f1c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.821+0000 7f7f21eaf640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c19d5b0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800c890 tx=0x7f7f1800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.822+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18004220 con 0x7f7f1c0ff990 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.822+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7f18007e90 con 0x7f7f1c0ff990 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.822+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1c1a4410 con 0x7f7f1c0ff990 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.822+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f18004a00 con 0x7f7f1c0ff990 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.822+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f1c1a48f0 con 0x7f7f1c0ff990 2024-02-09T23:04:09.829 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.823+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f1c102a40 con 0x7f7f1c0ff990 2024-02-09T23:04:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.824+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f7f180079b0 con 0x7f7f1c0ff990 2024-02-09T23:04:09.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.825+0000 7f7f12ffd640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7f080a2230 0x7f7f080a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.825+0000 7f7f216ae640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7f080a2230 0x7f7f080a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.825+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f7f180ce2f0 con 0x7f7f1c0ff990 2024-02-09T23:04:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.826+0000 7f7f216ae640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7f080a2230 0x7f7f080a46f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005fd0 tx=0x7f7f0c0094d0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:09.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:09.829+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f7f18096f20 con 0x7f7f1c0ff990 2024-02-09T23:04:09.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:09 smithi064 ceph-mon[36638]: pgmap v406: 97 pgs: 97 active+clean; 209 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 3.6 MiB/s wr, 188 op/s 2024-02-09T23:04:09.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3211235012' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:09.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:09 smithi064 ceph-mon[36638]: osdmap e106: 12 total, 12 up, 12 in 2024-02-09T23:04:09.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:09 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/573171905' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T23:04:10.014 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.011+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f1c068500 con 0x7f7f1c0ff990 2024-02-09T23:04:10.015 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.012+0000 7f7f12ffd640 1 -- 172.21.15.3:0/2848796852 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7f1801d020 con 0x7f7f1c0ff990 2024-02-09T23:04:10.015 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T23:04:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7f080a2230 msgr2=0x7f7f080a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7f080a2230 0x7f7f080a46f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f7f0c005fd0 tx=0x7f7f0c0094d0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 msgr2=0x7f7f1c19d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c19d5b0 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f7f1800c890 tx=0x7f7f1800cd50 comp rx=0 tx=0).stop 2024-02-09T23:04:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 shutdown_connections 2024-02-09T23:04:10.017 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f7f080a2230 0x7f7f080a46f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7f1c1976c0 0x7f7f1c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7f1c1065a0 0x7f7f1c19daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 --2- 172.21.15.3:0/2848796852 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7f1c0ff990 0x7f7f1c19d5b0 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 >> 172.21.15.3:0/2848796852 conn(0x7f7f1c076620 msgr2=0x7f7f1c1047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 shutdown_connections 2024-02-09T23:04:10.018 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.014+0000 7f7f24113640 1 -- 172.21.15.3:0/2848796852 wait complete. 2024-02-09T23:04:10.028 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-02-09T23:04:10.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 -- 172.21.15.3:0/2539568158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 msgr2=0x7ff0fc10cf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:10.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 --2- 172.21.15.3:0/2539568158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc10cf10 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7ff0ec009930 tx=0x7ff0ec02bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:10.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 -- 172.21.15.3:0/2539568158 shutdown_connections 2024-02-09T23:04:10.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 --2- 172.21.15.3:0/2539568158 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc10cf10 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 --2- 172.21.15.3:0/2539568158 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0fc100380 0x7ff0fc100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.134 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 --2- 172.21.15.3:0/2539568158 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0fc106250 0x7ff0fc106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 -- 172.21.15.3:0/2539568158 >> 172.21.15.3:0/2539568158 conn(0x7ff0fc076ec0 msgr2=0x7ff0fc0772d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 -- 172.21.15.3:0/2539568158 shutdown_connections 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 -- 172.21.15.3:0/2539568158 wait complete. 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 Processor -- start 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.129+0000 7ff102af4640 1 -- start start 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0fc100380 0x7ff0fc19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc19dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0fc106250 0x7ff0fc197750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:10.135 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc197ee0 con 0x7ff0fc100d40 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc198050 con 0x7ff0fc100380 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0fc198210 con 0x7ff0fc106250 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc19dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc19dbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:46012/0 (socket says 172.21.15.3:46012) 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 -- 172.21.15.3:0/514141444 learned_addr learned my addr 172.21.15.3:0/514141444 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 -- 172.21.15.3:0/514141444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0fc106250 msgr2=0x7ff0fc197750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0fc106250 0x7ff0fc197750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.136 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 -- 172.21.15.3:0/514141444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0fc100380 msgr2=0x7ff0fc19d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff100890640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0fc100380 0x7ff0fc19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0fc100380 0x7ff0fc19d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 -- 172.21.15.3:0/514141444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ec009560 con 0x7ff0fc100d40 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0fbfff640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc19dbd0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000d770 tx=0x7ff0f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0009c30 con 0x7ff0fc100d40 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0fc1984f0 con 0x7ff0fc100d40 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0f000de80 con 0x7ff0fc100d40 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f0004e80 con 0x7ff0fc100d40 2024-02-09T23:04:10.137 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.130+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0fc1a4610 con 0x7ff0fc100d40 2024-02-09T23:04:10.138 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.131+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0fc1017f0 con 0x7ff0fc100d40 2024-02-09T23:04:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.132+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff0f0005020 con 0x7ff0fc100d40 2024-02-09T23:04:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.133+0000 7ff0f9ffb640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0e40a2230 0x7ff0e40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.133+0000 7ff100890640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0e40a2230 0x7ff0e40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.133+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7ff0f00cdc00 con 0x7ff0fc100d40 2024-02-09T23:04:10.139 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.134+0000 7ff100890640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0e40a2230 0x7ff0e40a46f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4005fd0 tx=0x7ff0f4009200 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:10.140 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.137+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff0f0096830 con 0x7ff0fc100d40 2024-02-09T23:04:10.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.373+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7ff0fc0680b0 con 0x7ff0fc100d40 2024-02-09T23:04:10.383 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.381+0000 7ff0f9ffb640 1 -- 172.21.15.3:0/514141444 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+736934 (secure 0 0 0) 0x7ff0f0099ef0 con 0x7ff0fc100d40 2024-02-09T23:04:10.395 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:10.396 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:04:10.396 INFO:teuthology.orchestra.run.smithi003.stdout: "epoch": 26, 2024-02-09T23:04:10.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.396 INFO:teuthology.orchestra.run.smithi003.stdout: "active_gid": 15519, 2024-02-09T23:04:10.396 INFO:teuthology.orchestra.run.smithi003.stdout: "active_name": "x", 2024-02-09T23:04:10.396 INFO:teuthology.orchestra.run.smithi003.stdout: "active_addrs": { 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6800", 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 4119047166 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v1", 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:6801", 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 4119047166 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: "active_addr": "172.21.15.3:6801/4119047166", 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: "active_change": "2024-02-09T22:51:14.251736+0000", 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: "active_mgr_features": 4540701547738038271, 2024-02-09T23:04:10.398 INFO:teuthology.orchestra.run.smithi003.stdout: "available": true, 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: "standbys": [ 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: "gid": 25093, 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "z", 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_features": 4540701547738038271, 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: "available_modules": [ 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.399 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "alerts", 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.400 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How frequently to reexamine health status", 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.402 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.403 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.404 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.405 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.406 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.407 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.408 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.408 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.408 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.408 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.409 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_destination": { 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_destination", 2024-02-09T23:04:10.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email address to send alerts to", 2024-02-09T23:04:10.411 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_from_name": { 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_from_name", 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.412 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "Ceph", 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email From: name", 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.413 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_host": { 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_host", 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.414 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP server", 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.415 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_password": { 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_password", 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.416 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Password to authenticate with", 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_port": { 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_port", 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.417 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "465", 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP port", 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.418 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_sender": { 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_sender", 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.419 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP envelope sender", 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.420 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_ssl": { 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_ssl", 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.421 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use SSL to connect to SMTP server", 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_user": { 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_user", 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.422 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "User to authenticate as", 2024-02-09T23:04:10.423 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "balancer", 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.424 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "active": { 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "active", 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.425 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically balance PGs across cluster", 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_time": { 2024-02-09T23:04:10.426 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_time", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0000", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "beginning time of day to automatically balance", 2024-02-09T23:04:10.427 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_weekday": { 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_weekday", 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.428 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.429 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_max_iterations": { 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_max_iterations", 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "25", 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "250", 2024-02-09T23:04:10.430 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_metrics": { 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_metrics", 2024-02-09T23:04:10.431 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "pgs,objects,bytes", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-02-09T23:04:10.432 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_step": { 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_step", 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.5", 2024-02-09T23:04:10.433 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0.001", 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "0.999", 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "aggressiveness of optimization", 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.434 INFO:teuthology.orchestra.run.smithi003.stdout: "end_time": { 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_time", 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2359", 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.435 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ending time of day to automatically balance", 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "end_weekday": { 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_weekday", 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:10.436 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.437 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.438 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.439 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.440 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.441 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.442 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.443 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.444 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.445 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "min_score": { 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_score", 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.446 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "upmap", 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.448 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-compat", 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "none", 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "read", 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap", 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap-read" 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Balancer mode", 2024-02-09T23:04:10.449 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_ids": { 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_ids", 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.450 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.451 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.452 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_deviation": { 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_deviation", 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.453 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "deviation below which no optimization is attempted", 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-02-09T23:04:10.454 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_optimizations": { 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_optimizations", 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.455 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm", 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_down_multiplier": { 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_down_multiplier", 2024-02-09T23:04:10.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.458 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_refresh_rate": { 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_refresh_rate", 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.459 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "20", 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.460 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_starting_port": { 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_starting_port", 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4721", 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.461 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_ptrace": { 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_ptrace", 2024-02-09T23:04:10.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-02-09T23:04:10.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_interval": { 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_interval", 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.464 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to autotune daemon memory", 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_memory_target_ratio": { 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_memory_target_ratio", 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.7", 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.466 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_log_destination": { 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm_log_destination", 2024-02-09T23:04:10.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "file", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "file,syslog", 2024-02-09T23:04:10.468 INFO:teuthology.orchestra.run.smithi003.stdout: "syslog" 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "cgroups_split": { 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cgroups_split", 2024-02-09T23:04:10.469 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.470 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "config_checks_enabled": { 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_checks_enabled", 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.471 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.472 INFO:teuthology.orchestra.run.smithi003.stdout: "config_dashboard": { 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_dashboard", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-02-09T23:04:10.473 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_alertmanager": { 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_alertmanager", 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_base": { 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_base", 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph", 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.476 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Container image name, without the tag", 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_elasticsearch": { 2024-02-09T23:04:10.477 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_elasticsearch", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "elasticsearch container image", 2024-02-09T23:04:10.478 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_grafana": { 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_grafana", 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.479 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.480 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_haproxy": { 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_haproxy", 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-02-09T23:04:10.481 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "HAproxy container image", 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.482 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_agent": { 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_agent", 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.483 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger agent container image", 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_collector": { 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_collector", 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.484 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger collector container image", 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.485 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_query": { 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_query", 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-02-09T23:04:10.486 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger query container image", 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_keepalived": { 2024-02-09T23:04:10.487 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_keepalived", 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.488 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Keepalived container image", 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_loki": { 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_loki", 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.489 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Loki container image", 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.490 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_node_exporter": { 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_node_exporter", 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.491 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_nvmeof": { 2024-02-09T23:04:10.492 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_nvmeof", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Nvme-of container image", 2024-02-09T23:04:10.493 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_prometheus": { 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_prometheus", 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.494 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.495 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_promtail": { 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_promtail", 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.496 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Promtail container image", 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_snmp_gateway": { 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_snmp_gateway", 2024-02-09T23:04:10.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SNMP Gateway container image", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.498 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: "container_init": { 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_init", 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.499 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Run podman/docker with `--init`", 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.500 INFO:teuthology.orchestra.run.smithi003.stdout: "daemon_cache_timeout": { 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "daemon_cache_timeout", 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.501 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "default_cephadm_command_timeout": { 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_cephadm_command_timeout", 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.502 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "900", 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-02-09T23:04:10.503 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "default_registry": { 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_registry", 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.504 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io", 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "device_cache_timeout": { 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_cache_timeout", 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1800", 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.506 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache device inventory", 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "device_enhanced_scan": { 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_enhanced_scan", 2024-02-09T23:04:10.507 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use libstoragemgmt during device scans", 2024-02-09T23:04:10.508 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "facts_cache_timeout": { 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "facts_cache_timeout", 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.509 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache host facts data", 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.510 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "host_check_interval": { 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "host_check_interval", 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.511 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to perform a host check", 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "hw_monitoring": { 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hw_monitoring", 2024-02-09T23:04:10.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-02-09T23:04:10.513 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "inventory_list_all": { 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "inventory_list_all", 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.514 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.515 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.516 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.517 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "log_refresh_metadata": { 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_refresh_metadata", 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.518 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.519 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.520 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.522 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.523 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.524 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.525 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf": { 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.526 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.527 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.528 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.528 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "*", 2024-02-09T23:04:10.528 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.528 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.531 INFO:teuthology.orchestra.run.smithi003.stdout: "max_count_per_host": { 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_count_per_host", 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.532 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of daemons per service per host", 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd_draining_count": { 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_osd_draining_count", 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.533 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.534 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "migration_current": { 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "migration_current", 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.535 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "internal - do not modify", 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.536 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "root", 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.537 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm-package", 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "root" 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "mode for remote execution of cephadm", 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "oob_default_addr": { 2024-02-09T23:04:10.538 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "oob_default_addr", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "169.254.1.1", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default address for RedFish API (oob management).", 2024-02-09T23:04:10.539 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "prometheus_alerts_path": { 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus_alerts_path", 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.540 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.541 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_insecure": { 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_insecure", 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.542 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_password": { 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_password", 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.543 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.544 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_url": { 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_url", 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.545 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.546 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_username": { 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_username", 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.547 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "secure_monitoring_stack": { 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "secure_monitoring_stack", 2024-02-09T23:04:10.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.549 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "service_discovery_port": { 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "service_discovery_port", 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.550 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8765", 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "cephadm service discovery port", 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.551 INFO:teuthology.orchestra.run.smithi003.stdout: "ssh_config_file": { 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssh_config_file", 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.552 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "use_agent": { 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_agent", 2024-02-09T23:04:10.553 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-02-09T23:04:10.554 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "use_repo_digest": { 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_repo_digest", 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.555 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.556 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_failed_host_check": { 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_failed_host_check", 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.557 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if the host check fails", 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_daemons": { 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_daemons", 2024-02-09T23:04:10.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.559 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_hosts": { 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_hosts", 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.560 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.561 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crash", 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.563 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.564 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.565 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.566 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.567 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.568 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.569 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.570 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "retain_interval": { 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retain_interval", 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "31536000", 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.571 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain crashes before pruning them", 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_recent_interval": { 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_recent_interval", 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.572 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1209600", 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "time interval in which to warn about recent crashes", 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.573 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dashboard", 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.574 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.575 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_HOST": { 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_HOST", 2024-02-09T23:04:10.576 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.577 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.578 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_ENABLED": { 2024-02-09T23:04:10.579 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_ENABLED", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.580 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "ENABLE_BROWSABLE_API": { 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ENABLE_BROWSABLE_API", 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.583 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.584 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.585 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.586 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.587 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.588 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.589 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.590 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_NFS": { 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.591 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RBD": { 2024-02-09T23:04:10.592 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.593 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RGW": { 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.594 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.595 INFO:teuthology.orchestra.run.smithi003.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.596 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_PASSWORD": { 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_PASSWORD", 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.597 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.598 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.599 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_URL": { 2024-02-09T23:04:10.600 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_URL", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.601 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_USERNAME": { 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_USERNAME", 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.602 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.604 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-02-09T23:04:10.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.606 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.607 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.608 INFO:teuthology.orchestra.run.smithi003.stdout: "ISSUE_TRACKER_API_KEY": { 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.609 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_HOST": { 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_HOST", 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.610 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.611 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.612 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-02-09T23:04:10.613 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.614 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.615 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.616 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.617 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-02-09T23:04:10.618 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.619 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.620 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.622 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-02-09T23:04:10.623 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.624 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_ENABLED": { 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_ENABLED", 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.625 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.626 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.627 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.628 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.629 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.630 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8", 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.631 INFO:teuthology.orchestra.run.smithi003.stdout: "REST_REQUESTS_TIMEOUT": { 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "45", 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.632 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ACCESS_KEY": { 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ACCESS_KEY", 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.633 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.634 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.635 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SECRET_KEY": { 2024-02-09T23:04:10.636 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SECRET_KEY", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.637 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SSL_VERIFY": { 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SSL_VERIFY", 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.638 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.639 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:10.640 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.641 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.642 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.643 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.644 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "cross_origin_url": { 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cross_origin_url", 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.645 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "crt_file": { 2024-02-09T23:04:10.646 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crt_file", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.647 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "debug": { 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "debug", 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable/disable debug options", 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "jwt_token_ttl": { 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "jwt_token_ttl", 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "28800", 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.650 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.651 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.652 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.653 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.654 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.655 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.656 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.657 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.658 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.660 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "motd": { 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "motd", 2024-02-09T23:04:10.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "The message of the day", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.662 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect_resolve_ip_addr": { 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "redirect_resolve_ip_addr", 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.663 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.664 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.665 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.666 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8080", 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.668 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.669 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl_server_port": { 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl_server_port", 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8443", 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.670 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.671 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "redirect", 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect" 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.672 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.674 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "url_prefix": { 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url_prefix", 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.675 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.676 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_monitoring": { 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_monitoring", 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.677 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "monitor device health metrics", 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.679 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.680 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.681 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.683 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.684 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.685 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.686 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "mark_out_threshold": { 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mark_out_threshold", 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2419200", 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.687 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": { 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_name", 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.688 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "device_health_metrics", 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name of pool in which to store device health metrics", 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "retention_period": { 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retention_period", 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15552000", 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.690 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain device health metrics", 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_frequency": { 2024-02-09T23:04:10.691 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_frequency", 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.692 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to scrape device health metrics", 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "self_heal": { 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "self_heal", 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.693 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.694 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.695 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and check device health", 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_threshold": { 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_threshold", 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.696 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7257600", 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.697 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "diskprediction_local", 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.698 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.699 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.700 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.702 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.703 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.704 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.705 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.706 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "predict_interval": { 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predict_interval", 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.707 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "predictor_model": { 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predictor_model", 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "prophetstor", 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.709 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.710 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.711 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "influx", 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": false, 2024-02-09T23:04:10.712 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "influxdb python module not found", 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "batch_size": { 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "batch_size", 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5000", 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.713 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "database": { 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "database", 2024-02-09T23:04:10.714 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "ceph", 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-02-09T23:04:10.715 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": { 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hostname", 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.716 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server hostname", 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.717 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "30", 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "5", 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.718 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.720 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.721 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.722 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.723 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.724 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.725 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.726 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "password": { 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "password", 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.728 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "password of InfluxDB server user", 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "port": { 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "port", 2024-02-09T23:04:10.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8086", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server port", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.730 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "false", 2024-02-09T23:04:10.731 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "threads": { 2024-02-09T23:04:10.732 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threads", 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "32", 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.733 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "username": { 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "username", 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "username of InfluxDB server user", 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "verify_ssl": { 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "verify_ssl", 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "true", 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.736 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.737 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "insights", 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.738 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.739 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.740 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.741 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.743 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.744 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.745 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.746 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "iostat", 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.748 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.750 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.751 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.752 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.753 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.754 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.755 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "k8sevents", 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_event_retention_days": { 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ceph_event_retention_days", 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.756 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7", 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Days to hold ceph event information within local cache", 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.757 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "config_check_secs": { 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_check_secs", 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.758 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "10", 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.759 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.760 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.761 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.762 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.763 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.764 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.765 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.766 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.767 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "localpool", 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "failure_domain": { 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "failure_domain", 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.768 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "host", 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "failure domain for any created local pool", 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-02-09T23:04:10.769 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.770 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.771 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.773 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.774 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.775 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.776 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.777 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": { 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_size", 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.778 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default min_size for any created local pool", 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.779 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "num_rep": { 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "num_rep", 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3", 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.780 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default replica count for any created local pool", 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": { 2024-02-09T23:04:10.781 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_num", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "128", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default pg_num for any created local pool", 2024-02-09T23:04:10.782 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "prefix": { 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prefix", 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.783 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name prefix for any created local pool", 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.784 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "subtree": { 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "subtree", 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "rack", 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.785 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "CRUSH level for which to create a local pool", 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mds_autoscaler", 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.787 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.788 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.789 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.790 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.791 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.792 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.793 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.794 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.795 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirroring", 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.796 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.797 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.798 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.799 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.800 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.801 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.802 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.803 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.804 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "nfs", 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.805 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.806 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.807 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.808 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.809 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.810 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.812 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.813 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "fail_fs": { 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "fail_fs", 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.814 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.815 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.816 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.817 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.818 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.819 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.820 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.821 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.822 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.823 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator": { 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.824 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "rook", 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "test_orchestrator" 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Orchestrator backend", 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.825 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_perf_query", 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.826 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.827 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.828 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.829 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.830 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.831 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.832 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.833 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.834 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_support", 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.835 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.836 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.837 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.838 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.839 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.840 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.841 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.842 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.843 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_autoscaler", 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.844 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.845 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.846 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.847 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.848 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.849 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.850 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.851 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.852 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:10.853 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "threshold": { 2024-02-09T23:04:10.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threshold", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1.0", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "scaling threshold", 2024-02-09T23:04:10.855 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "progress", 2024-02-09T23:04:10.856 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_pg_recovery_event": { 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_pg_recovery_event", 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.857 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow the module to show pg recovery progress", 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.858 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.859 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.860 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.861 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.862 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.863 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.864 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.865 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.866 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.867 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "max_completed_events": { 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_completed_events", 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.868 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "50", 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "number of past completed events to remember", 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.869 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.870 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long the module is going to sleep", 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.871 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus", 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cache", 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.872 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.873 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "exclude_perf_counters": { 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "exclude_perf_counters", 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.874 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.875 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.876 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.877 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.878 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.879 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.880 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.881 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.882 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.883 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools": { 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools", 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.884 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools_refresh_interval": { 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.885 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300", 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.886 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_interval": { 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_interval", 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15.0", 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.887 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:10.888 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.889 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "9283", 2024-02-09T23:04:10.890 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.891 INFO:teuthology.orchestra.run.smithi003.stdout: "stale_cache_strategy": { 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stale_cache_strategy", 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "log", 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.892 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.893 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "default", 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "default", 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "error" 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.894 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.895 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.896 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.897 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.898 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.899 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.900 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.901 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.902 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.903 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.905 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_snap_create": { 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_snap_create", 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.906 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "mirror_snapshot_schedule": { 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirror_snapshot_schedule", 2024-02-09T23:04:10.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.908 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "trash_purge_schedule": { 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "trash_purge_schedule", 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.909 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.910 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "restful", 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_auth": { 2024-02-09T23:04:10.911 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_auth", 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.912 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.915 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.916 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.917 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.918 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.918 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.918 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.918 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.918 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.918 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.919 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.920 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.921 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.922 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.923 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.924 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.925 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rgw", 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.927 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.928 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.930 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.932 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.933 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.934 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.935 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rook", 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "drive_group_interval": { 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "drive_group_interval", 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.936 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300.0", 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.937 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.938 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.939 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.940 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.941 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.943 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.943 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.943 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.943 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.943 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.944 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.945 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.946 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.947 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_class": { 2024-02-09T23:04:10.948 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "storage_class", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "local", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-02-09T23:04:10.949 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "selftest", 2024-02-09T23:04:10.950 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.951 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.952 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.953 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.955 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.956 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.957 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.958 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "roption1": { 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption1", 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.959 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.960 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "roption2": { 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption2", 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "xyz", 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.961 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption1": { 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption1", 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.962 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.963 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption2": { 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption2", 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.964 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.965 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption3": { 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption3", 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.966 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption4": { 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption4", 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.967 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.968 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption5": { 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption5", 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.969 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption6": { 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption6", 2024-02-09T23:04:10.970 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.971 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption7": { 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption7", 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.972 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "42", 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.973 INFO:teuthology.orchestra.run.smithi003.stdout: "testkey": { 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testkey", 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.974 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "testlkey": { 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testlkey", 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.976 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "testnewline": { 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testnewline", 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.977 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.978 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_m_granularity": { 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_m_granularity", 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.979 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow minute scheduled snapshots", 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.980 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "dump_on_update": { 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dump_on_update", 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.981 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "dump database to debug log on update", 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.982 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.983 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.984 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.985 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.986 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.987 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.988 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.989 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.990 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stats", 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:10.991 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.992 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.995 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:10.996 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.997 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:10.998 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:10.999 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "status", 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.000 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.001 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.002 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.003 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.004 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.005 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.006 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.007 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telegraf", 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "address": { 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "address", 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.009 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.010 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15", 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.011 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.012 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.013 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.014 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.015 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.016 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.017 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.018 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.019 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.020 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telemetry", 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_basic": { 2024-02-09T23:04:11.021 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_basic", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share basic cluster information (size, version)", 2024-02-09T23:04:11.022 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_crash": { 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_crash", 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.023 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_device": { 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_device", 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.025 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_ident": { 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_ident", 2024-02-09T23:04:11.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.027 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_perf": { 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_perf", 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.028 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share various performance metrics of a cluster", 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "contact": { 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "contact", 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.030 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "description": { 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "description", 2024-02-09T23:04:11.031 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.032 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "device_url": { 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_url", 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.033 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.035 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.036 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "24", 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "8", 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.037 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "last_opt_revision": { 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "last_opt_revision", 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1", 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.038 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard": { 2024-02-09T23:04:11.039 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard", 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.040 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard_description": { 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard_description", 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.041 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.043 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.044 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.045 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.046 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.047 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.048 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.049 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.050 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "organization": { 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "organization", 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.051 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "proxy": { 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "proxy", 2024-02-09T23:04:11.052 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.053 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "url": { 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url", 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-02-09T23:04:11.054 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "test_orchestrator", 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.057 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.058 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.059 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.060 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.061 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.062 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.063 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.065 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.066 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.067 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.068 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.069 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.070 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.071 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.072 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_clones": { 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_clones", 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.073 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4", 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Number of asynchronous cloner threads", 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "periodic_async_work": { 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "periodic_async_work", 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.075 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Periodically check for async work", 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.076 INFO:teuthology.orchestra.run.smithi003.stdout: "snapshot_clone_delay": { 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snapshot_clone_delay", 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.077 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.078 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix", 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "discovery_interval": { 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "discovery_interval", 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.079 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "100", 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.080 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "identifier": { 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "identifier", 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.081 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.082 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.083 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.084 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.085 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.086 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.088 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.089 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.090 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.091 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_host": { 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_host", 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.092 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_port": { 2024-02-09T23:04:11.093 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_port", 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10051", 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.094 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_sender": { 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_sender", 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.095 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.096 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: "gid": 25096, 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "y", 2024-02-09T23:04:11.097 INFO:teuthology.orchestra.run.smithi003.stdout: "mgr_features": 4540701547738038271, 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "available_modules": [ 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "alerts", 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.098 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How frequently to reexamine health status", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.099 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.100 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.101 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.102 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.103 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.104 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.105 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.106 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.107 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_destination": { 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_destination", 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.108 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email address to send alerts to", 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.109 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_from_name": { 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_from_name", 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "Ceph", 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.110 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email From: name", 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_host": { 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_host", 2024-02-09T23:04:11.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP server", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.112 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_password": { 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_password", 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.113 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Password to authenticate with", 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.114 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_port": { 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_port", 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "465", 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.115 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP port", 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_sender": { 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_sender", 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.116 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP envelope sender", 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.117 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_ssl": { 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_ssl", 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.118 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use SSL to connect to SMTP server", 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.119 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_user": { 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_user", 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.120 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "User to authenticate as", 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.121 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "balancer", 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "active": { 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "active", 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.122 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically balance PGs across cluster", 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.123 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_time": { 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_time", 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0000", 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.124 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "beginning time of day to automatically balance", 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_weekday": { 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_weekday", 2024-02-09T23:04:11.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-02-09T23:04:11.126 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_max_iterations": { 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_max_iterations", 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.127 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "25", 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "250", 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.128 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_metrics": { 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_metrics", 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "pgs,objects,bytes", 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.129 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_step": { 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_step", 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.130 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.5", 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0.001", 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "0.999", 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "aggressiveness of optimization", 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.131 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "end_time": { 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_time", 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2359", 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.132 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ending time of day to automatically balance", 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "end_weekday": { 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_weekday", 2024-02-09T23:04:11.133 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-02-09T23:04:11.134 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.135 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.136 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.137 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.138 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.139 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.140 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.141 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.142 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "min_score": { 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_score", 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.143 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "upmap", 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.145 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-compat", 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "none", 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "read", 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap", 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap-read" 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Balancer mode", 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.146 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_ids": { 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_ids", 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.147 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.148 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.149 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_deviation": { 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_deviation", 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.150 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "deviation below which no optimization is attempted", 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.151 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_optimizations": { 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_optimizations", 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.152 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.153 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm", 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_down_multiplier": { 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_down_multiplier", 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.154 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.155 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_refresh_rate": { 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_refresh_rate", 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "20", 2024-02-09T23:04:11.156 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_starting_port": { 2024-02-09T23:04:11.157 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_starting_port", 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4721", 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.158 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_ptrace": { 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_ptrace", 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.159 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.160 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_interval": { 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_interval", 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.161 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to autotune daemon memory", 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_memory_target_ratio": { 2024-02-09T23:04:11.162 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_memory_target_ratio", 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.7", 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.163 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_log_destination": { 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm_log_destination", 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.164 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "file", 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "file,syslog", 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "syslog" 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.165 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "cgroups_split": { 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cgroups_split", 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.166 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.167 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "config_checks_enabled": { 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_checks_enabled", 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.168 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "config_dashboard": { 2024-02-09T23:04:11.169 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_dashboard", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-02-09T23:04:11.170 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_alertmanager": { 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_alertmanager", 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.171 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.172 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_base": { 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_base", 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph", 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.173 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Container image name, without the tag", 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_elasticsearch": { 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_elasticsearch", 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.174 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "elasticsearch container image", 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.175 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_grafana": { 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_grafana", 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-02-09T23:04:11.176 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_haproxy": { 2024-02-09T23:04:11.177 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_haproxy", 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.178 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "HAproxy container image", 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_agent": { 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_agent", 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.179 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger agent container image", 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.180 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_collector": { 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_collector", 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.181 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger collector container image", 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_query": { 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_query", 2024-02-09T23:04:11.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger query container image", 2024-02-09T23:04:11.183 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_keepalived": { 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_keepalived", 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.184 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Keepalived container image", 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.185 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_loki": { 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_loki", 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.186 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Loki container image", 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_node_exporter": { 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_node_exporter", 2024-02-09T23:04:11.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.188 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_nvmeof": { 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_nvmeof", 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.189 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Nvme-of container image", 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.190 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_prometheus": { 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_prometheus", 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.191 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_promtail": { 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_promtail", 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.192 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Promtail container image", 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.193 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_snmp_gateway": { 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_snmp_gateway", 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.194 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SNMP Gateway container image", 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "container_init": { 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_init", 2024-02-09T23:04:11.195 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Run podman/docker with `--init`", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.196 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "daemon_cache_timeout": { 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "daemon_cache_timeout", 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.197 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.198 INFO:teuthology.orchestra.run.smithi003.stdout: "default_cephadm_command_timeout": { 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_cephadm_command_timeout", 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "900", 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.199 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "default_registry": { 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_registry", 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.200 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io", 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.201 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "device_cache_timeout": { 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_cache_timeout", 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.202 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1800", 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache device inventory", 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.203 INFO:teuthology.orchestra.run.smithi003.stdout: "device_enhanced_scan": { 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_enhanced_scan", 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.204 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use libstoragemgmt during device scans", 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "facts_cache_timeout": { 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "facts_cache_timeout", 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.205 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache host facts data", 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.206 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "host_check_interval": { 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "host_check_interval", 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.207 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to perform a host check", 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "hw_monitoring": { 2024-02-09T23:04:11.208 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hw_monitoring", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-02-09T23:04:11.209 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "inventory_list_all": { 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "inventory_list_all", 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.210 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.212 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.213 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "log_refresh_metadata": { 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_refresh_metadata", 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.214 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.215 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.216 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.217 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.218 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.219 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.220 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf": { 2024-02-09T23:04:11.221 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.222 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-02-09T23:04:11.223 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "*", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.224 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "max_count_per_host": { 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_count_per_host", 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.225 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of daemons per service per host", 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.226 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd_draining_count": { 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_osd_draining_count", 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.227 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "migration_current": { 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "migration_current", 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.228 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "internal - do not modify", 2024-02-09T23:04:11.229 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.230 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "root", 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm-package", 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "root" 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.231 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "mode for remote execution of cephadm", 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "oob_default_addr": { 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "oob_default_addr", 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.232 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "169.254.1.1", 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default address for RedFish API (oob management).", 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.233 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "prometheus_alerts_path": { 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus_alerts_path", 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.234 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_insecure": { 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_insecure", 2024-02-09T23:04:11.235 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-02-09T23:04:11.236 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_password": { 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_password", 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.237 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.238 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_url": { 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_url", 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.239 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_username": { 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_username", 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.240 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.241 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "secure_monitoring_stack": { 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "secure_monitoring_stack", 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.242 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "service_discovery_port": { 2024-02-09T23:04:11.243 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "service_discovery_port", 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8765", 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.244 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "cephadm service discovery port", 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "ssh_config_file": { 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssh_config_file", 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.245 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "use_agent": { 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_agent", 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.247 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "use_repo_digest": { 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_repo_digest", 2024-02-09T23:04:11.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.249 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_failed_host_check": { 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_failed_host_check", 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.250 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if the host check fails", 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.251 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_daemons": { 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_daemons", 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.252 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_hosts": { 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_hosts", 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.253 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.254 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crash", 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.255 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.256 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.258 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.259 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.260 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.261 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.262 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.263 INFO:teuthology.orchestra.run.smithi003.stdout: "retain_interval": { 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retain_interval", 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "31536000", 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.264 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain crashes before pruning them", 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_recent_interval": { 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_recent_interval", 2024-02-09T23:04:11.265 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1209600", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "time interval in which to warn about recent crashes", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.266 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dashboard", 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.267 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.268 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_HOST": { 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_HOST", 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.269 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.270 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.271 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_ENABLED": { 2024-02-09T23:04:11.272 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_ENABLED", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.273 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.274 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.275 INFO:teuthology.orchestra.run.smithi003.stdout: "ENABLE_BROWSABLE_API": { 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ENABLE_BROWSABLE_API", 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.276 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.277 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.278 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.279 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-02-09T23:04:11.280 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.281 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.282 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.283 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_NFS": { 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.284 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RBD": { 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-02-09T23:04:11.285 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.286 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RGW": { 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.287 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.288 INFO:teuthology.orchestra.run.smithi003.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.289 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_PASSWORD": { 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_PASSWORD", 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.290 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.292 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_URL": { 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_URL", 2024-02-09T23:04:11.293 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.294 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_USERNAME": { 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_USERNAME", 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.295 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.296 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.297 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.298 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.299 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.300 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "ISSUE_TRACKER_API_KEY": { 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-02-09T23:04:11.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.302 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_HOST": { 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_HOST", 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.303 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.305 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-02-09T23:04:11.306 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.307 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.308 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.309 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.310 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-02-09T23:04:11.311 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.312 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.313 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.314 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.315 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.316 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.317 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_ENABLED": { 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_ENABLED", 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.318 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-02-09T23:04:11.319 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.320 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.321 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.322 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8", 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.323 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "REST_REQUESTS_TIMEOUT": { 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.324 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "45", 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.325 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ACCESS_KEY": { 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ACCESS_KEY", 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.326 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-02-09T23:04:11.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.328 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SECRET_KEY": { 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SECRET_KEY", 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.329 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.330 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SSL_VERIFY": { 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SSL_VERIFY", 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.331 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.333 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.334 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-02-09T23:04:11.335 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.336 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "cross_origin_url": { 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cross_origin_url", 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.337 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "crt_file": { 2024-02-09T23:04:11.338 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crt_file", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.339 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "debug": { 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "debug", 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.340 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable/disable debug options", 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.341 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "jwt_token_ttl": { 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "jwt_token_ttl", 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "28800", 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.342 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.343 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.344 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.345 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.346 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.347 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.348 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.351 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.352 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.353 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.354 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "motd": { 2024-02-09T23:04:11.355 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "motd", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "The message of the day", 2024-02-09T23:04:11.356 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect_resolve_ip_addr": { 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "redirect_resolve_ip_addr", 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.357 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.358 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.359 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:11.360 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8080", 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.361 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.362 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl_server_port": { 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl_server_port", 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8443", 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.364 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2024-02-09T23:04:11.365 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "redirect", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.366 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect" 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2024-02-09T23:04:11.367 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.368 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "url_prefix": { 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url_prefix", 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.369 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.370 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_monitoring": { 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_monitoring", 2024-02-09T23:04:11.371 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "monitor device health metrics", 2024-02-09T23:04:11.372 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.373 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.374 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.375 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.376 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.377 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.378 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.379 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.380 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "mark_out_threshold": { 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mark_out_threshold", 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.381 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2419200", 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": { 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_name", 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "device_health_metrics", 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.383 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name of pool in which to store device health metrics", 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "retention_period": { 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retention_period", 2024-02-09T23:04:11.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15552000", 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain device health metrics", 2024-02-09T23:04:11.385 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_frequency": { 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_frequency", 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.386 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to scrape device health metrics", 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.387 INFO:teuthology.orchestra.run.smithi003.stdout: "self_heal": { 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "self_heal", 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.388 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.389 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and check device health", 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.390 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_threshold": { 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_threshold", 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7257600", 2024-02-09T23:04:11.391 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "diskprediction_local", 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.394 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.396 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.398 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.399 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.400 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "predict_interval": { 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predict_interval", 2024-02-09T23:04:11.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.402 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "predictor_model": { 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predictor_model", 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.403 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "prophetstor", 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.405 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.406 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "influx", 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": false, 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "influxdb python module not found", 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "batch_size": { 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "batch_size", 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.407 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5000", 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "database": { 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "database", 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "ceph", 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.409 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": { 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hostname", 2024-02-09T23:04:11.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server hostname", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.411 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.412 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "30", 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "5", 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.414 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.415 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.416 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.417 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.418 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.419 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.420 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.421 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "password": { 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "password", 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.422 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "password of InfluxDB server user", 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "port": { 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "port", 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8086", 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.424 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server port", 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2024-02-09T23:04:11.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "false", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.426 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "threads": { 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threads", 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:11.427 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "32", 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.428 INFO:teuthology.orchestra.run.smithi003.stdout: "username": { 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "username", 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.429 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "username of InfluxDB server user", 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "verify_ssl": { 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "verify_ssl", 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.430 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "true", 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.431 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "insights", 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.432 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.433 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.434 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.435 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.436 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.437 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.438 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.439 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.440 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "iostat", 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.441 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.442 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.443 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.444 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.445 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.446 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.447 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.448 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.449 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "k8sevents", 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_event_retention_days": { 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ceph_event_retention_days", 2024-02-09T23:04:11.450 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Days to hold ceph event information within local cache", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.451 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "config_check_secs": { 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_check_secs", 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.452 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "10", 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.453 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.454 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.455 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.456 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.457 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.458 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.459 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.460 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "localpool", 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "failure_domain": { 2024-02-09T23:04:11.462 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "failure_domain", 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "host", 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.463 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "failure domain for any created local pool", 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.465 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.466 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.467 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.468 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.469 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.470 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.471 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": { 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_size", 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.472 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default min_size for any created local pool", 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.473 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "num_rep": { 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "num_rep", 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3", 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.474 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default replica count for any created local pool", 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": { 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_num", 2024-02-09T23:04:11.475 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "128", 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default pg_num for any created local pool", 2024-02-09T23:04:11.476 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "prefix": { 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prefix", 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.477 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name prefix for any created local pool", 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.478 INFO:teuthology.orchestra.run.smithi003.stdout: "subtree": { 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "subtree", 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "rack", 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.479 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "CRUSH level for which to create a local pool", 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.480 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mds_autoscaler", 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.481 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.482 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.483 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.483 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.483 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.483 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.483 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.483 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.484 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.485 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.486 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.487 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.488 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirroring", 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.490 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.491 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.492 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.493 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.494 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.495 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.496 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.497 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "nfs", 2024-02-09T23:04:11.498 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.499 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.500 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.501 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.502 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.503 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.504 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.505 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.506 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.507 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "fail_fs": { 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "fail_fs", 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.508 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.509 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.510 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.511 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.512 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.513 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.514 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.515 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.516 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.517 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator": { 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.518 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: "rook", 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: "test_orchestrator" 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Orchestrator backend", 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.519 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_perf_query", 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.520 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.521 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.522 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.523 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.524 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.525 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.526 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.527 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.528 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_support", 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.529 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.530 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.531 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.532 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.533 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.534 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.535 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.536 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.537 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_autoscaler", 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.538 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.539 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.540 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.541 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.542 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.543 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.544 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.545 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.546 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "threshold": { 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threshold", 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1.0", 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.548 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "scaling threshold", 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.549 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "progress", 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_pg_recovery_event": { 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_pg_recovery_event", 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.550 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow the module to show pg recovery progress", 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.552 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.555 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.555 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.555 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.555 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.555 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.556 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.557 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.558 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.559 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.560 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.561 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.562 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.563 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "max_completed_events": { 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_completed_events", 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "50", 2024-02-09T23:04:11.564 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "number of past completed events to remember", 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.565 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.566 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long the module is going to sleep", 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus", 2024-02-09T23:04:11.567 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cache", 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.568 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.569 INFO:teuthology.orchestra.run.smithi003.stdout: "exclude_perf_counters": { 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "exclude_perf_counters", 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.570 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.571 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.572 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.574 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.575 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.576 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.577 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.578 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools": { 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools", 2024-02-09T23:04:11.579 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.580 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools_refresh_interval": { 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.581 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300", 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.582 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_interval": { 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_interval", 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15.0", 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.583 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.584 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "9283", 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.586 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "stale_cache_strategy": { 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stale_cache_strategy", 2024-02-09T23:04:11.587 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "log", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.588 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "default", 2024-02-09T23:04:11.589 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "default", 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "error" 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.590 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2024-02-09T23:04:11.591 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.592 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.593 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.594 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.595 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.596 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.597 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.598 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.599 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.600 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_snap_create": { 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_snap_create", 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.601 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.602 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "mirror_snapshot_schedule": { 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirror_snapshot_schedule", 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.603 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "trash_purge_schedule": { 2024-02-09T23:04:11.604 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "trash_purge_schedule", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.605 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "restful", 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.606 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_auth": { 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_auth", 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.607 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2024-02-09T23:04:11.608 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.609 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.610 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.611 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.612 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.613 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.614 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.615 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.616 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.617 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.618 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.619 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.620 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.621 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rgw", 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.622 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.623 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.625 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.626 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.627 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.629 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.630 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rook", 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "drive_group_interval": { 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "drive_group_interval", 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.631 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300.0", 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.632 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.633 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.634 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.635 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.636 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.637 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.638 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.639 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.640 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_class": { 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "storage_class", 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "local", 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.641 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.642 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "selftest", 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.643 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.644 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.645 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.646 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.647 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.648 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.649 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.650 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "roption1": { 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption1", 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.651 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.652 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "roption2": { 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption2", 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "xyz", 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.653 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption1": { 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption1", 2024-02-09T23:04:11.654 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.655 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption2": { 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption2", 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.656 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.657 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption3": { 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption3", 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.658 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption4": { 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption4", 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.659 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.660 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption5": { 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption5", 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.661 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption6": { 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption6", 2024-02-09T23:04:11.662 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.663 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption7": { 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption7", 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.664 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "42", 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.665 INFO:teuthology.orchestra.run.smithi003.stdout: "testkey": { 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testkey", 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.666 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "testlkey": { 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testlkey", 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.667 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.668 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "testnewline": { 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testnewline", 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.669 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.670 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_m_granularity": { 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_m_granularity", 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.671 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow minute scheduled snapshots", 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.672 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "dump_on_update": { 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dump_on_update", 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.673 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "dump database to debug log on update", 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.674 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.675 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.676 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.677 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.678 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.679 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.680 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.681 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.682 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stats", 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.683 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.684 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.685 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.686 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.687 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.688 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.689 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.690 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.691 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "status", 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.692 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.693 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.694 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.695 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.696 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.697 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.698 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.699 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.700 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telegraf", 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "address": { 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "address", 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.701 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.702 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15", 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.703 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.704 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.705 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.706 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.707 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.708 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.709 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.710 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.711 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telemetry", 2024-02-09T23:04:11.712 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_basic": { 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_basic", 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.713 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share basic cluster information (size, version)", 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.714 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_crash": { 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_crash", 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.715 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_device": { 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_device", 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.716 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.717 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.717 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.717 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.717 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.717 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.718 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-02-09T23:04:11.718 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.718 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.718 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.718 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.719 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_ident": { 2024-02-09T23:04:11.719 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_ident", 2024-02-09T23:04:11.719 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.719 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.719 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.719 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.720 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.720 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.720 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.720 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-02-09T23:04:11.720 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.721 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.721 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.721 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.721 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_perf": { 2024-02-09T23:04:11.721 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_perf", 2024-02-09T23:04:11.721 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share various performance metrics of a cluster", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.722 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "contact": { 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "contact", 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.723 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "description": { 2024-02-09T23:04:11.724 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "description", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.725 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "device_url": { 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_url", 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.726 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.727 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.728 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.729 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "24", 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "8", 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.730 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "last_opt_revision": { 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "last_opt_revision", 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1", 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.731 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard": { 2024-02-09T23:04:11.732 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.733 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard_description": { 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard_description", 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.734 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.735 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.736 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.737 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.738 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.739 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.740 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.741 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.742 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.743 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "organization": { 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "organization", 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.744 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "proxy": { 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "proxy", 2024-02-09T23:04:11.745 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.746 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "url": { 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url", 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.747 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.748 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "test_orchestrator", 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.749 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.750 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.751 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.752 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.753 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.754 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.755 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.756 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.757 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.759 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.759 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.759 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.760 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.761 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.762 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.763 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.764 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.765 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.766 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.767 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_clones": { 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_clones", 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.768 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4", 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Number of asynchronous cloner threads", 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.769 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "periodic_async_work": { 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "periodic_async_work", 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.770 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Periodically check for async work", 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "snapshot_clone_delay": { 2024-02-09T23:04:11.771 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snapshot_clone_delay", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-02-09T23:04:11.772 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix", 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.773 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "discovery_interval": { 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "discovery_interval", 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "100", 2024-02-09T23:04:11.774 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "identifier": { 2024-02-09T23:04:11.775 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "identifier", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.776 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.777 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.778 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.779 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.780 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.781 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.782 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.783 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.784 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.785 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.786 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_host": { 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_host", 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.787 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_port": { 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_port", 2024-02-09T23:04:11.788 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10051", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.789 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_sender": { 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_sender", 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-02-09T23:04:11.790 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.791 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: "modules": [ 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: "dashboard", 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: "iostat", 2024-02-09T23:04:11.792 INFO:teuthology.orchestra.run.smithi003.stdout: "nfs", 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: "restful", 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: "snap_schedule" 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: "available_modules": [ 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "alerts", 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.793 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.794 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How frequently to reexamine health status", 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.795 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.796 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.797 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.798 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.799 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.800 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.801 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.802 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_destination": { 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_destination", 2024-02-09T23:04:11.803 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email address to send alerts to", 2024-02-09T23:04:11.804 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_from_name": { 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_from_name", 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.805 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "Ceph", 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Email From: name", 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.806 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_host": { 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_host", 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.807 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP server", 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_password": { 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_password", 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.808 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Password to authenticate with", 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.809 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_port": { 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_port", 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "465", 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.810 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP port", 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_sender": { 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_sender", 2024-02-09T23:04:11.811 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SMTP envelope sender", 2024-02-09T23:04:11.812 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_ssl": { 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_ssl", 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.813 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use SSL to connect to SMTP server", 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.814 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "smtp_user": { 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "smtp_user", 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.815 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "User to authenticate as", 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.816 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "balancer", 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "active": { 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "active", 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.817 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically balance PGs across cluster", 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.818 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_time": { 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_time", 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0000", 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.819 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "beginning time of day to automatically balance", 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "begin_weekday": { 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "begin_weekday", 2024-02-09T23:04:11.820 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-02-09T23:04:11.821 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_max_iterations": { 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_max_iterations", 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.822 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "25", 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "250", 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.823 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_metrics": { 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_metrics", 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "pgs,objects,bytes", 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.824 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "crush_compat_step": { 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crush_compat_step", 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.825 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.5", 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0.001", 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "0.999", 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "aggressiveness of optimization", 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-02-09T23:04:11.826 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "end_time": { 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_time", 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2359", 2024-02-09T23:04:11.827 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ending time of day to automatically balance", 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.828 INFO:teuthology.orchestra.run.smithi003.stdout: "end_weekday": { 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "end_weekday", 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "0", 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "6", 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.829 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.830 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.831 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.832 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.833 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.834 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.835 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.836 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.837 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "min_score": { 2024-02-09T23:04:11.838 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_score", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-02-09T23:04:11.839 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.840 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "upmap", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "crush-compat", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "none", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "read", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap", 2024-02-09T23:04:11.841 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap-read" 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Balancer mode", 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_ids": { 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_ids", 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.842 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.843 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.844 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_deviation": { 2024-02-09T23:04:11.845 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_deviation", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "deviation below which no optimization is attempted", 2024-02-09T23:04:11.846 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "upmap_max_optimizations": { 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "upmap_max_optimizations", 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.847 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.848 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm", 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.849 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_down_multiplier": { 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_down_multiplier", 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.850 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_refresh_rate": { 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_refresh_rate", 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.851 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "20", 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.852 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "agent_starting_port": { 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "agent_starting_port", 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4721", 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.853 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_ptrace": { 2024-02-09T23:04:11.854 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_ptrace", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-02-09T23:04:11.855 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_interval": { 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_interval", 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.856 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to autotune daemon memory", 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.857 INFO:teuthology.orchestra.run.smithi003.stdout: "autotune_memory_target_ratio": { 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "autotune_memory_target_ratio", 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0.7", 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.858 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm_log_destination": { 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cephadm_log_destination", 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.859 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "file", 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "file,syslog", 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: "syslog" 2024-02-09T23:04:11.860 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "cgroups_split": { 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cgroups_split", 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.861 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.862 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "config_checks_enabled": { 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_checks_enabled", 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.863 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.864 INFO:teuthology.orchestra.run.smithi003.stdout: "config_dashboard": { 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_dashboard", 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.865 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_alertmanager": { 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_alertmanager", 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.866 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.867 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_base": { 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_base", 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph", 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.868 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Container image name, without the tag", 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_elasticsearch": { 2024-02-09T23:04:11.869 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_elasticsearch", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "elasticsearch container image", 2024-02-09T23:04:11.870 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_grafana": { 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_grafana", 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.871 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.872 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_haproxy": { 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_haproxy", 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.873 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "HAproxy container image", 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_agent": { 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_agent", 2024-02-09T23:04:11.874 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger agent container image", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.875 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_collector": { 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_collector", 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.876 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger collector container image", 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_jaeger_query": { 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_jaeger_query", 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.878 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Jaeger query container image", 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_keepalived": { 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_keepalived", 2024-02-09T23:04:11.879 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Keepalived container image", 2024-02-09T23:04:11.880 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_loki": { 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_loki", 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.881 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Loki container image", 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.882 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_node_exporter": { 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_node_exporter", 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.883 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_nvmeof": { 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_nvmeof", 2024-02-09T23:04:11.884 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/ceph/nvmeof:latest", 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Nvme-of container image", 2024-02-09T23:04:11.885 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_prometheus": { 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_prometheus", 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.886 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Prometheus container image", 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.887 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_promtail": { 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_promtail", 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.888 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Promtail container image", 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "container_image_snmp_gateway": { 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_image_snmp_gateway", 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.889 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "SNMP Gateway container image", 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.890 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "container_init": { 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "container_init", 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.891 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Run podman/docker with `--init`", 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "daemon_cache_timeout": { 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "daemon_cache_timeout", 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.892 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.893 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "default_cephadm_command_timeout": { 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_cephadm_command_timeout", 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "900", 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.894 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "default_registry": { 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "default_registry", 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.895 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "docker.io", 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.896 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "device_cache_timeout": { 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_cache_timeout", 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1800", 2024-02-09T23:04:11.897 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache device inventory", 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "device_enhanced_scan": { 2024-02-09T23:04:11.898 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_enhanced_scan", 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.899 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use libstoragemgmt during device scans", 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "facts_cache_timeout": { 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "facts_cache_timeout", 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.900 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "seconds to cache host facts data", 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.901 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "host_check_interval": { 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "host_check_interval", 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:11.902 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to perform a host check", 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "hw_monitoring": { 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hw_monitoring", 2024-02-09T23:04:11.903 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-02-09T23:04:11.904 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "inventory_list_all": { 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "inventory_list_all", 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.905 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.906 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.907 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.908 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "log_refresh_metadata": { 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_refresh_metadata", 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.909 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.910 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.911 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.912 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.913 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.914 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.915 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf": { 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.916 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.917 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "*", 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.918 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "max_count_per_host": { 2024-02-09T23:04:11.919 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_count_per_host", 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.920 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of daemons per service per host", 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "max_osd_draining_count": { 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_osd_draining_count", 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.921 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.922 INFO:teuthology.orchestra.run.smithi003.stdout: "migration_current": { 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "migration_current", 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.923 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "internal - do not modify", 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "mode": { 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mode", 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.924 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "root", 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm-package", 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: "root" 2024-02-09T23:04:11.925 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "mode for remote execution of cephadm", 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "oob_default_addr": { 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "oob_default_addr", 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.926 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "169.254.1.1", 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Default address for RedFish API (oob management).", 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.927 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: "prometheus_alerts_path": { 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus_alerts_path", 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.928 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.929 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_insecure": { 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_insecure", 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.930 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_password": { 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_password", 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.931 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.932 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_url": { 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_url", 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.933 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.934 INFO:teuthology.orchestra.run.smithi003.stdout: "registry_username": { 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "registry_username", 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.935 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "secure_monitoring_stack": { 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "secure_monitoring_stack", 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.936 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.937 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "service_discovery_port": { 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "service_discovery_port", 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8765", 2024-02-09T23:04:11.938 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "cephadm service discovery port", 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "ssh_config_file": { 2024-02-09T23:04:11.939 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssh_config_file", 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.940 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "use_agent": { 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_agent", 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.941 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.942 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "use_repo_digest": { 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "use_repo_digest", 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.943 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_failed_host_check": { 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_failed_host_check", 2024-02-09T23:04:11.944 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if the host check fails", 2024-02-09T23:04:11.945 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_daemons": { 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_daemons", 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.946 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.947 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_on_stray_hosts": { 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_on_stray_hosts", 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.948 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.949 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crash", 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.950 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.951 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:11.952 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.953 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.954 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:11.955 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.956 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.957 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "retain_interval": { 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retain_interval", 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.958 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "31536000", 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain crashes before pruning them", 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.959 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_recent_interval": { 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_recent_interval", 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1209600", 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.960 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.961 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.963 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "time interval in which to warn about recent crashes", 2024-02-09T23:04:11.963 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.963 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.963 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.963 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dashboard", 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-02-09T23:04:11.964 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.965 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_HOST": { 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_HOST", 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.966 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.967 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.968 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_ENABLED": { 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_ENABLED", 2024-02-09T23:04:11.969 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.970 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.971 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.972 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "ENABLE_BROWSABLE_API": { 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ENABLE_BROWSABLE_API", 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.973 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.974 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.975 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.976 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.977 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.978 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.979 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.980 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_NFS": { 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.981 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.982 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RBD": { 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.983 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "FEATURE_TOGGLE_RGW": { 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.984 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.985 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.986 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_PASSWORD": { 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_PASSWORD", 2024-02-09T23:04:11.987 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.988 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.989 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.990 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_URL": { 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_URL", 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.991 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_API_USERNAME": { 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_API_USERNAME", 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.992 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.993 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.994 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-02-09T23:04:11.995 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.996 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.997 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:11.998 INFO:teuthology.orchestra.run.smithi003.stdout: "ISSUE_TRACKER_API_KEY": { 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:11.999 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_HOST": { 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_HOST", 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.000 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.001 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.002 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-02-09T23:04:12.003 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.004 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.005 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.006 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.007 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.008 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.009 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.010 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-02-09T23:04:12.011 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.012 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.013 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.014 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_ENABLED": { 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_ENABLED", 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.015 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.016 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.017 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.018 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-02-09T23:04:12.019 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.020 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "REST_REQUESTS_TIMEOUT": { 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.021 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "45", 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.022 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ACCESS_KEY": { 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ACCESS_KEY", 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.023 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-02-09T23:04:12.024 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "admin", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.025 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SECRET_KEY": { 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SECRET_KEY", 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.026 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.027 INFO:teuthology.orchestra.run.smithi003.stdout: "RGW_API_SSL_VERIFY": { 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "RGW_API_SSL_VERIFY", 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.028 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-02-09T23:04:12.029 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.030 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.031 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.033 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "cross_origin_url": { 2024-02-09T23:04:12.034 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cross_origin_url", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.035 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "crt_file": { 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "crt_file", 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.036 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.037 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "debug": { 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "debug", 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.038 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Enable/disable debug options", 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "jwt_token_ttl": { 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "jwt_token_ttl", 2024-02-09T23:04:12.039 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "28800", 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.040 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.041 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.042 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.043 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.044 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.045 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.046 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.047 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.048 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.049 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.050 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "motd": { 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "motd", 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.051 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "The message of the day", 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect_resolve_ip_addr": { 2024-02-09T23:04:12.052 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "redirect_resolve_ip_addr", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.053 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.054 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.055 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8080", 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.056 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2024-02-09T23:04:12.057 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.058 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl_server_port": { 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl_server_port", 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.059 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8443", 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.060 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "redirect", 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.061 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "redirect" 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2024-02-09T23:04:12.062 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.063 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "url_prefix": { 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url_prefix", 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.064 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_monitoring": { 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_monitoring", 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.066 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "monitor device health metrics", 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.067 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.068 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.069 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.070 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.071 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.072 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.073 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.074 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.075 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "mark_out_threshold": { 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mark_out_threshold", 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.076 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "2419200", 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.077 INFO:teuthology.orchestra.run.smithi003.stdout: "pool_name": { 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pool_name", 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "device_health_metrics", 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.078 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name of pool in which to store device health metrics", 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "retention_period": { 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "retention_period", 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.079 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15552000", 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long to retain device health metrics", 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.080 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_frequency": { 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_frequency", 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.081 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to scrape device health metrics", 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "self_heal": { 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "self_heal", 2024-02-09T23:04:12.082 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-02-09T23:04:12.083 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.084 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how frequently to wake up and check device health", 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.085 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "warn_threshold": { 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "warn_threshold", 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7257600", 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.086 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.087 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "diskprediction_local", 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.088 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.089 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.090 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.091 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.092 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.093 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.094 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.095 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.096 INFO:teuthology.orchestra.run.smithi003.stdout: "predict_interval": { 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predict_interval", 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "86400", 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.097 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "predictor_model": { 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "predictor_model", 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.098 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "prophetstor", 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.099 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "600", 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.100 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.101 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "influx", 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": false, 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "influxdb python module not found", 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "batch_size": { 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "batch_size", 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.102 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5000", 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.103 INFO:teuthology.orchestra.run.smithi003.stdout: "database": { 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "database", 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "ceph", 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.104 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "hostname": { 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "hostname", 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.105 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server hostname", 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.106 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "30", 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "5", 2024-02-09T23:04:12.107 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.108 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.109 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.110 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.111 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.112 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.113 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.114 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.115 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "password": { 2024-02-09T23:04:12.116 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "password", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "password of InfluxDB server user", 2024-02-09T23:04:12.117 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "port": { 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "port", 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.118 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "8086", 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "InfluxDB server port", 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.119 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "ssl": { 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ssl", 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "false", 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.120 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "threads": { 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threads", 2024-02-09T23:04:12.121 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "32", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.122 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "username": { 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "username", 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.123 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "username of InfluxDB server user", 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "verify_ssl": { 2024-02-09T23:04:12.124 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "verify_ssl", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "true", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-02-09T23:04:12.125 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "insights", 2024-02-09T23:04:12.126 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.127 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.127 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.127 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.127 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.127 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.127 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.128 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.129 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.130 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.131 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.132 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.133 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.134 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.135 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "iostat", 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.136 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.137 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.138 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.139 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.140 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.141 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.142 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.143 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.144 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "k8sevents", 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_event_retention_days": { 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "ceph_event_retention_days", 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.145 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "7", 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Days to hold ceph event information within local cache", 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.146 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "config_check_secs": { 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "config_check_secs", 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "10", 2024-02-09T23:04:12.147 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.148 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.149 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.150 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.151 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.152 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.153 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.154 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.155 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.156 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "localpool", 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "failure_domain": { 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "failure_domain", 2024-02-09T23:04:12.157 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "host", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "failure domain for any created local pool", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-02-09T23:04:12.158 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.159 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.160 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.161 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.162 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.163 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.164 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.165 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.166 INFO:teuthology.orchestra.run.smithi003.stdout: "min_size": { 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "min_size", 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.167 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default min_size for any created local pool", 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "num_rep": { 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "num_rep", 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.168 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3", 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default replica count for any created local pool", 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.169 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_num": { 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_num", 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "128", 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.170 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "default pg_num for any created local pool", 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "prefix": { 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prefix", 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.171 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "name prefix for any created local pool", 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.172 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "subtree": { 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "subtree", 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "rack", 2024-02-09T23:04:12.173 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "CRUSH level for which to create a local pool", 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.174 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mds_autoscaler", 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.175 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.176 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.177 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.178 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.179 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.180 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.181 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.182 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.183 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirroring", 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.184 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.185 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.186 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.187 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.188 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.189 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.190 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.191 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.192 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "nfs", 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.193 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.194 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.195 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.196 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.197 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.198 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.199 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.200 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2024-02-09T23:04:12.201 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "fail_fs": { 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "fail_fs", 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.202 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.203 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.204 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.205 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.206 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.207 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.208 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.209 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.210 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.211 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator": { 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "orchestrator", 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.212 INFO:teuthology.orchestra.run.smithi003.stdout: "cephadm", 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "rook", 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "test_orchestrator" 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Orchestrator backend", 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.213 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_perf_query", 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.214 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.215 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.216 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.217 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.218 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.219 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.220 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.221 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.222 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "osd_support", 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.223 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.224 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.225 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.226 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.227 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.228 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.229 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.230 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.231 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "pg_autoscaler", 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.232 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.233 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.234 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.235 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.236 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.237 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.238 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.239 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:12.240 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.241 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "threshold": { 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "threshold", 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "3.0", 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1.0", 2024-02-09T23:04:12.242 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "scaling threshold", 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.243 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "progress", 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_pg_recovery_event": { 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_pg_recovery_event", 2024-02-09T23:04:12.244 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow the module to show pg recovery progress", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.245 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.246 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.247 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.248 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.249 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.250 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.251 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.252 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.253 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.254 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.255 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "max_completed_events": { 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_completed_events", 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "50", 2024-02-09T23:04:12.256 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "number of past completed events to remember", 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "sleep_interval": { 2024-02-09T23:04:12.257 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "sleep_interval", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "5", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "how long the module is going to sleep", 2024-02-09T23:04:12.258 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "prometheus", 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.259 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "cache": { 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "cache", 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.260 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "exclude_perf_counters": { 2024-02-09T23:04:12.261 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "exclude_perf_counters", 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.262 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.263 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.264 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.265 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.266 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.267 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.268 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.269 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.270 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools": { 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools", 2024-02-09T23:04:12.271 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.272 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_stats_pools_refresh_interval": { 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.273 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300", 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.274 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "scrape_interval": { 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "scrape_interval", 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15.0", 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.275 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.276 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "::", 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.277 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "9283", 2024-02-09T23:04:12.278 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "stale_cache_strategy": { 2024-02-09T23:04:12.279 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stale_cache_strategy", 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "log", 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.280 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_behaviour": { 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_behaviour", 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.281 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "default", 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "default", 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "error" 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.282 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "standby_error_status_code": { 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "standby_error_status_code", 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.283 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "500", 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "400", 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "599", 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.284 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.285 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.286 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.287 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.288 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.289 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.290 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.291 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.292 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_snap_create": { 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_snap_create", 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.293 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10", 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.294 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "mirror_snapshot_schedule": { 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "mirror_snapshot_schedule", 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.295 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "trash_purge_schedule": { 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "trash_purge_schedule", 2024-02-09T23:04:12.296 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.297 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "restful", 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.298 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "enable_auth": { 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enable_auth", 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.299 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.300 INFO:teuthology.orchestra.run.smithi003.stdout: "key_file": { 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "key_file", 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.301 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.302 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.303 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.304 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.305 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.306 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.307 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.308 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.309 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "server_addr": { 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_addr", 2024-02-09T23:04:12.310 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.311 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "server_port": { 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "server_port", 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.312 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.313 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rgw", 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.314 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.315 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.316 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.317 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.318 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.319 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.320 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.321 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.322 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rook", 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "drive_group_interval": { 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "drive_group_interval", 2024-02-09T23:04:12.323 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "300.0", 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "interval in seconds between re-application of applied drive_groups", 2024-02-09T23:04:12.324 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.325 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.326 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.327 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.328 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.329 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.330 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.331 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.332 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "storage_class": { 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "storage_class", 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.333 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "local", 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.334 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "selftest", 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.335 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.336 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.337 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.338 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.339 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.340 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.341 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.342 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "roption1": { 2024-02-09T23:04:12.343 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption1", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.344 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "roption2": { 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "roption2", 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.345 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "xyz", 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.346 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption1": { 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption1", 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.347 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption2": { 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption2", 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.348 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.349 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption3": { 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption3", 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "float", 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.350 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption4": { 2024-02-09T23:04:12.351 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption4", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.352 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption5": { 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption5", 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.353 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.354 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption6": { 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption6", 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.355 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "rwoption7": { 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rwoption7", 2024-02-09T23:04:12.356 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "1", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "42", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.357 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "testkey": { 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testkey", 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.358 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.359 INFO:teuthology.orchestra.run.smithi003.stdout: "testlkey": { 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testlkey", 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.360 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "testnewline": { 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "testnewline", 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.361 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.362 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "allow_m_granularity": { 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "allow_m_granularity", 2024-02-09T23:04:12.363 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "allow minute scheduled snapshots", 2024-02-09T23:04:12.364 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "dump_on_update": { 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "dump_on_update", 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.365 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "dump database to debug log on update", 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.366 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.367 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.368 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.369 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.370 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.371 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.372 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.373 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.374 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "stats", 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.375 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.376 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.377 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.378 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.379 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.380 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.381 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.382 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.383 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "status", 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.384 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.385 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.386 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.387 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.388 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.389 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.390 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.391 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.392 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telegraf", 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "address": { 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "address", 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.393 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.394 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "15", 2024-02-09T23:04:12.395 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.396 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.397 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.398 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.399 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.400 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.401 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.402 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.403 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.404 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "telemetry", 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_basic": { 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_basic", 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.405 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share basic cluster information (size, version)", 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.406 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_crash": { 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_crash", 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.407 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_device": { 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_device", 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.408 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "True", 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.409 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_ident": { 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_ident", 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.410 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "channel_perf": { 2024-02-09T23:04:12.411 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "channel_perf", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Share various performance metrics of a cluster", 2024-02-09T23:04:12.412 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "contact": { 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "contact", 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.413 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.414 INFO:teuthology.orchestra.run.smithi003.stdout: "description": { 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "description", 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.415 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "device_url": { 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "device_url", 2024-02-09T23:04:12.416 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.417 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "enabled": { 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "enabled", 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.418 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.419 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "24", 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "8", 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.420 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "last_opt_revision": { 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "last_opt_revision", 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.421 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "1", 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.422 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard": { 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard", 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.423 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "leaderboard_description": { 2024-02-09T23:04:12.424 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "leaderboard_description", 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.425 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.426 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.427 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.428 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.429 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.430 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.431 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.432 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.433 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "organization": { 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "organization", 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.434 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "proxy": { 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "proxy", 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.436 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "url": { 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "url", 2024-02-09T23:04:12.437 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.438 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "test_orchestrator", 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.439 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.440 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.441 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.442 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.443 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.444 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.445 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.446 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.447 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.448 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.449 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.450 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.451 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.452 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.453 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.454 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.455 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.456 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "max_concurrent_clones": { 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "max_concurrent_clones", 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "4", 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.457 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Number of asynchronous cloner threads", 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "periodic_async_work": { 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "periodic_async_work", 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.458 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Periodically check for async work", 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.459 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "snapshot_clone_delay": { 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snapshot_clone_delay", 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "0", 2024-02-09T23:04:12.460 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.461 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix", 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "can_run": true, 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "error_string": "", 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "module_options": { 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "discovery_interval": { 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "discovery_interval", 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "uint", 2024-02-09T23:04:12.462 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "100", 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.463 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "identifier": { 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "identifier", 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.464 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.465 INFO:teuthology.orchestra.run.smithi003.stdout: "interval": { 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "interval", 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "secs", 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "60", 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.466 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "log_level": { 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_level", 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.467 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.468 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.469 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster": { 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster", 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.470 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_cluster_level": { 2024-02-09T23:04:12.471 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_cluster_level", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "info", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [ 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "", 2024-02-09T23:04:12.472 INFO:teuthology.orchestra.run.smithi003.stdout: "critical", 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "debug", 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "error", 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "info", 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "warning" 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.473 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "log_to_file": { 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "log_to_file", 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "bool", 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 1, 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "False", 2024-02-09T23:04:12.474 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_host": { 2024-02-09T23:04:12.475 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_host", 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "", 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.476 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_port": { 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_port", 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "int", 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.477 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "10051", 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.478 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "zabbix_sender": { 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "zabbix_sender", 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "str", 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "level": "advanced", 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "flags": 0, 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "min": "", 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "max": "", 2024-02-09T23:04:12.479 INFO:teuthology.orchestra.run.smithi003.stdout: "enum_allowed": [], 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: "desc": "", 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: "long_desc": "", 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: "tags": [], 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: "see_also": [] 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.480 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "services": { 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "dashboard": "https://172.21.15.3:8443/" 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "always_on_modules": { 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "octopus": [ 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2024-02-09T23:04:12.481 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "pacific": [ 2024-02-09T23:04:12.482 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2024-02-09T23:04:12.483 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "quincy": [ 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2024-02-09T23:04:12.484 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "reef": [ 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2024-02-09T23:04:12.485 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: ], 2024-02-09T23:04:12.486 INFO:teuthology.orchestra.run.smithi003.stdout: "squid": [ 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "balancer", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "crash", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "devicehealth", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "orchestrator", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "pg_autoscaler", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "progress", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "rbd_support", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "status", 2024-02-09T23:04:12.487 INFO:teuthology.orchestra.run.smithi003.stdout: "telemetry", 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: "volumes" 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: "last_failure_osd_epoch": 89, 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: "active_clients": [ 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "devicehealth", 2024-02-09T23:04:12.488 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3687433321 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.489 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "libcephsqlite", 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1352911915 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.490 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "rbd_support", 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 1179535959 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.491 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 4253938165 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.492 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "snap_schedule", 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 3982508169 2024-02-09T23:04:12.493 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: "name": "volumes", 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addrvec": [ 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: { 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: "type": "v2", 2024-02-09T23:04:12.494 INFO:teuthology.orchestra.run.smithi003.stdout: "addr": "172.21.15.3:0", 2024-02-09T23:04:12.495 INFO:teuthology.orchestra.run.smithi003.stdout: "nonce": 206643333 2024-02-09T23:04:12.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.495 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:12.495 INFO:teuthology.orchestra.run.smithi003.stdout: ] 2024-02-09T23:04:12.495 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:04:12.497 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2571480236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T23:04:12.497 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2848796852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T23:04:12.497 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:10 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/514141444' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-02-09T23:04:12.497 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:11 smithi064 ceph-mon[36638]: pgmap v408: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 3.8 MiB/s wr, 150 op/s 2024-02-09T23:04:12.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2571480236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T23:04:12.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2848796852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T23:04:12.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:10 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/514141444' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-02-09T23:04:12.498 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:11 smithi089 ceph-mon[36665]: pgmap v408: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 3.8 MiB/s wr, 150 op/s 2024-02-09T23:04:12.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0e40a2230 msgr2=0x7ff0e40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0e40a2230 0x7ff0e40a46f0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7ff0f4005fd0 tx=0x7ff0f4009200 comp rx=0 tx=0).stop 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 msgr2=0x7ff0fc19dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc19dbd0 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000d770 tx=0x7ff0f000dc30 comp rx=0 tx=0).stop 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 shutdown_connections 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff0e40a2230 0x7ff0e40a46f0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff0fc106250 0x7ff0fc197750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:12.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff0fc100d40 0x7ff0fc19dbd0 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:12.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 --2- 172.21.15.3:0/514141444 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff0fc100380 0x7ff0fc19d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:12.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 >> 172.21.15.3:0/514141444 conn(0x7ff0fc076ec0 msgr2=0x7ff0fc0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:12.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 shutdown_connections 2024-02-09T23:04:12.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:10.386+0000 7ff102af4640 1 -- 172.21.15.3:0/514141444 wait complete. 2024-02-09T23:04:12.508 DEBUG:tasks.check_counter:Getting stats from a 2024-02-09T23:04:12.508 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.a perf dump 2024-02-09T23:04:12.508 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2571480236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T23:04:12.509 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2848796852' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T23:04:12.509 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:10 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/514141444' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-02-09T23:04:12.509 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:11 smithi003 ceph-mon[32678]: pgmap v408: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 3.8 MiB/s wr, 150 op/s 2024-02-09T23:04:12.764 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T23:04:13.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:13 smithi089 ceph-mon[36665]: pgmap v409: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-02-09T23:04:13.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:13 smithi003 ceph-mon[32678]: pgmap v409: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-02-09T23:04:13.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:13 smithi064 ceph-mon[36638]: pgmap v409: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 292 KiB/s rd, 3.1 MiB/s wr, 121 op/s 2024-02-09T23:04:14.542 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:04:14.542 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 1479, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 1479, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 8846299, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 459174, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 7, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.140489518, 2024-02-09T23:04:14.543 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.045007952, 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.040167131, 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.042137228, 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 1478, 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.051500344, 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000034844 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.544 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 448 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.545 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 1116, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 1301, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 24502663, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 838746, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 8, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 6, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.117349632, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.035502459, 2024-02-09T23:04:14.546 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.043478040, 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.028563554, 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 1298, 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.046553886, 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000035865 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.547 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:14.548 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.548 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 320, 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 492224 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 699, 2024-02-09T23:04:14.549 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 656, 2024-02-09T23:04:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 10296616, 2024-02-09T23:04:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 271518, 2024-02-09T23:04:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2024-02-09T23:04:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:14.550 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.084313124, 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.028301587, 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.049283512, 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.014061492, 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 655, 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.023338596, 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000035631 2024-02-09T23:04:14.551 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1808112, 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 151472 2024-02-09T23:04:14.552 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 1, 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "finisher-MDSRank": { 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "queue_len": 0, 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "complete_latency": { 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 5082, 2024-02-09T23:04:14.553 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.121094783, 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000023828 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: "queue_len": 0, 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: "complete_latency": { 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.554 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": { 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "request": 0, 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "reply": 0, 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "reply_latency": { 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.555 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "slow_reply": 0, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "forward": 0, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_commit": 0, 2024-02-09T23:04:14.556 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_split": 0, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_merge": 0, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes": 3654, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_top": 17, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_bottom": 3637, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_pin_tail": 0, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_pinned": 3630, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_expired": 0, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_with_caps": 0, 2024-02-09T23:04:14.557 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "subtrees": 5, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_hit": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_forward": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_discover": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_lock": 0, 2024-02-09T23:04:14.558 INFO:teuthology.orchestra.run.smithi003.stdout: "load_cent": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "q": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "exported": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "exported_inodes": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "imported": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "imported_inodes": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:14.559 INFO:teuthology.orchestra.run.smithi003.stdout: "openino_backtrace_fetch": 0, 2024-02-09T23:04:14.560 INFO:teuthology.orchestra.run.smithi003.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:14.560 INFO:teuthology.orchestra.run.smithi003.stdout: "root_rfiles": 0, 2024-02-09T23:04:14.560 INFO:teuthology.orchestra.run.smithi003.stdout: "root_rbytes": 0, 2024-02-09T23:04:14.560 INFO:teuthology.orchestra.run.smithi003.stdout: "root_rsnaps": 0, 2024-02-09T23:04:14.560 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_backtrace_fetch": 0, 2024-02-09T23:04:14.561 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:14.561 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:14.561 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:14.561 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:14.561 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_file_inodes": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_revoke": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_grant": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:14.562 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_caps": 0, 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_caps_dirty": 0, 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_cap_release": 0, 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "process_request_cap_release": 0 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache": { 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_update": 0, 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_update_receipt": 0, 2024-02-09T23:04:14.563 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_try_discover": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_send_discover": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_handle_discover": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "num_strays": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_created": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_enqueued": 0, 2024-02-09T23:04:14.564 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_migrated": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_started": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_completed": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_exportdir": 0, 2024-02-09T23:04:14.565 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_flush": 0, 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_inodestats": 0 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log": { 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "evlrg": 0, 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "evadd": 0, 2024-02-09T23:04:14.566 INFO:teuthology.orchestra.run.smithi003.stdout: "evex": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "evtrm": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "ev": 22976, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "evexg": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "evexd": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "segadd": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "segex": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "segtrm": 0, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "seg": 29, 2024-02-09T23:04:14.567 INFO:teuthology.orchestra.run.smithi003.stdout: "segmjr": 3, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "segexg": 0, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "segexd": 0, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "expos": 4194304, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "wrpos": 45178072, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "rdpos": 45170781, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "jlat": { 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.568 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "replayed": 22976 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_mem": { 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "ino": 278, 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "ino+": 3651, 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "ino-": 3373, 2024-02-09T23:04:14.569 INFO:teuthology.orchestra.run.smithi003.stdout: "dir": 24, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "dir+": 25, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "dir-": 1, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "dn": 3654, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "dn+": 3669, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "dn-": 15, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "cap": 0, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "cap+": 0, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "cap-": 0, 2024-02-09T23:04:14.570 INFO:teuthology.orchestra.run.smithi003.stdout: "rss": 54508, 2024-02-09T23:04:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "heap": 215420 2024-02-09T23:04:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server": { 2024-02-09T23:04:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "dispatch_client_request": 0, 2024-02-09T23:04:14.571 INFO:teuthology.orchestra.run.smithi003.stdout: "dispatch_server_request": 0, 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_request": 0, 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_session": 0, 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_peer_request": 0, 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "req_create_latency": { 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.572 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "req_getattr_latency": { 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.573 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "req_link_latency": { 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.574 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookup_latency": { 2024-02-09T23:04:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:14.575 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupino_latency": { 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.576 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupname_latency": { 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.577 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lssnap_latency": { 2024-02-09T23:04:14.578 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "req_mkdir_latency": { 2024-02-09T23:04:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.579 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.580 INFO:teuthology.orchestra.run.smithi003.stdout: "req_mknod_latency": { 2024-02-09T23:04:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "req_mksnap_latency": { 2024-02-09T23:04:14.581 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "req_open_latency": { 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.582 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "req_readdir_latency": { 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rename_latency": { 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.583 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:14.584 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rmdir_latency": { 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.585 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.586 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setattr_latency": { 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:14.587 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.588 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setlayout_latency": { 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setxattr_latency": { 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.589 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "req_symlink_latency": { 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.590 INFO:teuthology.orchestra.run.smithi003.stdout: "req_unlink_latency": { 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.591 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.592 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sessions": { 2024-02-09T23:04:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "session_count": 1, 2024-02-09T23:04:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "session_add": 1, 2024-02-09T23:04:14.593 INFO:teuthology.orchestra.run.smithi003.stdout: "session_remove": 0, 2024-02-09T23:04:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "sessions_open": 1, 2024-02-09T23:04:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "sessions_stale": 0, 2024-02-09T23:04:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "total_load": 0, 2024-02-09T23:04:14.594 INFO:teuthology.orchestra.run.smithi003.stdout: "average_load": 0, 2024-02-09T23:04:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: "avg_session_uptime": 2250, 2024-02-09T23:04:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.595 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2024-02-09T23:04:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:14.596 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:14.597 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:14.598 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:14.599 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:14.600 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:14.601 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:14.602 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:14.603 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 224975, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 331, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:14.604 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 167, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2024-02-09T23:04:14.605 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 3297110, 2024-02-09T23:04:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 30877, 2024-02-09T23:04:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2024-02-09T23:04:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2024-02-09T23:04:14.606 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter": { 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_active": 0, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_laggy": 0, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_send": 2975, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_send_bytes": 21, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_resend": 3, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_reply": 2975, 2024-02-09T23:04:14.607 INFO:teuthology.orchestra.run.smithi003.stdout: "op_latency": { 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 2975, 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 1.621980250, 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000545203 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "op_inflight": 0, 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "oplen_avg": { 2024-02-09T23:04:14.608 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 2975, 2024-02-09T23:04:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 2976 2024-02-09T23:04:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "op": 2975, 2024-02-09T23:04:14.609 INFO:teuthology.orchestra.run.smithi003.stdout: "op_r": 2975, 2024-02-09T23:04:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "op_w": 0, 2024-02-09T23:04:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "op_rmw": 0, 2024-02-09T23:04:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "op_pg": 0, 2024-02-09T23:04:14.610 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_stat": 1674, 2024-02-09T23:04:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_create": 0, 2024-02-09T23:04:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_read": 1299, 2024-02-09T23:04:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_write": 0, 2024-02-09T23:04:14.611 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_writefull": 0, 2024-02-09T23:04:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_writesame": 0, 2024-02-09T23:04:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_append": 0, 2024-02-09T23:04:14.612 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_zero": 0, 2024-02-09T23:04:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_truncate": 0, 2024-02-09T23:04:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_delete": 0, 2024-02-09T23:04:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_mapext": 0, 2024-02-09T23:04:14.613 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_getxattr": 1, 2024-02-09T23:04:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_setxattr": 0, 2024-02-09T23:04:14.614 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_call": 0, 2024-02-09T23:04:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_watch": 0, 2024-02-09T23:04:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_notify": 0, 2024-02-09T23:04:14.615 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:14.616 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_pgls": 0, 2024-02-09T23:04:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:14.619 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_other": 0, 2024-02-09T23:04:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_active": 0, 2024-02-09T23:04:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_send": 0, 2024-02-09T23:04:14.620 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_resend": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_ping": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "poolop_active": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "poolop_send": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "poolop_resend": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "poolstat_active": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "poolstat_send": 0, 2024-02-09T23:04:14.621 INFO:teuthology.orchestra.run.smithi003.stdout: "poolstat_resend": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "statfs_active": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "statfs_send": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "statfs_resend": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "command_active": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "command_send": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "command_resend": 0, 2024-02-09T23:04:14.622 INFO:teuthology.orchestra.run.smithi003.stdout: "map_epoch": 106, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "map_full": 0, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "map_inc": 20, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_sessions": 8, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_session_open": 8, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_session_close": 0, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_laggy": 0, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_wr": 0, 2024-02-09T23:04:14.623 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_rd": 2, 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_del": 0 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "oft": { 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_objs": 0, 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_updates": 0, 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_removes": 0 2024-02-09T23:04:14.624 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "osd-slow-ops": { 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "slow_ops_count": 0 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "purge_queue": { 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing_ops_high_water": 0, 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing": 0, 2024-02-09T23:04:14.625 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing_high_water": 0, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executed_ops": 0, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executed": 0, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:14.626 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 3290, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 43330719, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 3290, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 3290, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 43330719, 2024-02-09T23:04:14.627 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:14.628 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2024-02-09T23:04:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:14.629 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 2975, 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 41004834, 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 2975, 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 41004834, 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.630 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 1024, 2024-02-09T23:04:14.631 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 2975, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 2975, 2024-02-09T23:04:14.632 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 2975, 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 2975, 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.633 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 3758096384, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:14.634 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 0, 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 0, 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:14.635 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-write_buf_throttle-0x564635bb0960": { 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 3758096384, 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:14.636 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 0, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 0, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:14.637 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.638 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:14.638 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:04:14.871 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:14 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:04:14.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:14 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:04:14.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:14 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:04:15.453 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-02-09T23:04:15.453 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-02-09T23:04:15.453 INFO:tasks.check_counter:Daemon mds.a mds.root_rsnaps=0 2024-02-09T23:04:15.453 INFO:tasks.check_counter:Daemon mds.a mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:15.453 INFO:tasks.check_counter:Daemon mds.a mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:15.453 DEBUG:tasks.check_counter:Getting stats from d 2024-02-09T23:04:15.454 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.d perf dump 2024-02-09T23:04:15.710 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T23:04:15.744 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:15 smithi003 ceph-mon[32678]: pgmap v410: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-02-09T23:04:15.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:15 smithi064 ceph-mon[36638]: pgmap v410: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-02-09T23:04:15.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:15 smithi089 ceph-mon[36665]: pgmap v410: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 290 KiB/s rd, 3.1 MiB/s wr, 120 op/s 2024-02-09T23:04:17.430 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:04:17.430 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:17.430 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 4, 2024-02-09T23:04:17.430 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 4, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 2992, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 448, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 0, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.001495648, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.000041756, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.000590834, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-02-09T23:04:17.431 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 3, 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000746479, 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000248826 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:17.432 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 448 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 262, 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 251, 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 1580544, 2024-02-09T23:04:17.433 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 150864, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 2, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 1, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.048491500, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.015803787, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.036856910, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.004955887, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 250, 2024-02-09T23:04:17.434 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.011309597, 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000045238 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.435 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1580544, 2024-02-09T23:04:17.436 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 150864 2024-02-09T23:04:17.436 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.436 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:17.436 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 2, 2024-02-09T23:04:17.436 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 192, 2024-02-09T23:04:17.436 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 320, 2024-02-09T23:04:17.437 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 188496, 2024-02-09T23:04:17.437 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 5, 2024-02-09T23:04:17.437 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:17.437 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.015530991, 2024-02-09T23:04:17.437 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.011126073, 2024-02-09T23:04:17.437 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.001833024, 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 189, 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.016740459, 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000088573 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:17.438 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 320, 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 188496 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2024-02-09T23:04:17.439 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 0, 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:17.440 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:17.441 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:17.441 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:17.441 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:17.441 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:17.441 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:17.441 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:17.442 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:17.443 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:17.444 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 148977, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 30, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:17.445 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2024-02-09T23:04:17.446 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 928, 2024-02-09T23:04:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 2, 2024-02-09T23:04:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2024-02-09T23:04:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2024-02-09T23:04:17.447 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 264, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 1543579, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:17.448 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 264, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 264, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 1543579, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:17.449 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:17.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:17.450 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:17.450 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:04:17.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:17 smithi003 ceph-mon[32678]: pgmap v411: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 726 KiB/s wr, 45 op/s 2024-02-09T23:04:17.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:17 smithi064 ceph-mon[36638]: pgmap v411: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 726 KiB/s wr, 45 op/s 2024-02-09T23:04:17.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:17 smithi089 ceph-mon[36665]: pgmap v411: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 726 KiB/s wr, 45 op/s 2024-02-09T23:04:18.044 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-02-09T23:04:18.044 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-02-09T23:04:18.044 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-02-09T23:04:18.044 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-02-09T23:04:18.044 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-02-09T23:04:18.044 DEBUG:tasks.check_counter:Getting stats from g 2024-02-09T23:04:18.045 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.g perf dump 2024-02-09T23:04:18.296 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T23:04:19.633 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:19 smithi003 ceph-mon[32678]: pgmap v412: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 726 KiB/s wr, 45 op/s 2024-02-09T23:04:19.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:19 smithi064 ceph-mon[36638]: pgmap v412: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 726 KiB/s wr, 45 op/s 2024-02-09T23:04:19.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:19 smithi089 ceph-mon[36665]: pgmap v412: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 726 KiB/s wr, 45 op/s 2024-02-09T23:04:19.984 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 1610, 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 1565, 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 10606239, 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 554669, 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 6, 2024-02-09T23:04:19.985 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 2, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.173877746, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.064830108, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.070487384, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.038440439, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 1565, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.058182867, 2024-02-09T23:04:19.986 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000037177 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.987 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1811648, 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 153168 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 1577, 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 1764, 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 21378908, 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 984058, 2024-02-09T23:04:19.988 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 7, 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.133838874, 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.037333442, 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.050528493, 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.036231934, 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:19.989 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 1761, 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.049311513, 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000028001 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:19.990 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 320, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 494176 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 198, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 200, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 4515341, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 61590, 2024-02-09T23:04:19.991 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 5, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 4, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.018834411, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.004162034, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.008469286, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.004787402, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 198, 2024-02-09T23:04:19.992 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.006432794, 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000032488 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.993 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 0, 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 0 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 1, 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "finisher-MDSRank": { 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "queue_len": 0, 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "complete_latency": { 2024-02-09T23:04:19.994 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 5286, 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.111377953, 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000021070 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "queue_len": 0, 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "complete_latency": { 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:19.995 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "mds": { 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "request": 0, 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "reply": 0, 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "reply_latency": { 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:19.996 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: "slow_reply": 0, 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: "forward": 0, 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:19.997 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_commit": 0, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_split": 0, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_merge": 0, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes": 2588, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_top": 33, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_bottom": 2555, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_pin_tail": 0, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_pinned": 2541, 2024-02-09T23:04:19.998 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_expired": 0, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "inodes_with_caps": 0, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "caps": 0, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "subtrees": 6, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse": 0, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_hit": 0, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_forward": 0, 2024-02-09T23:04:19.999 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_discover": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "traverse_lock": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "load_cent": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "q": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "exported": 0, 2024-02-09T23:04:20.000 INFO:teuthology.orchestra.run.smithi003.stdout: "exported_inodes": 0, 2024-02-09T23:04:20.001 INFO:teuthology.orchestra.run.smithi003.stdout: "imported": 0, 2024-02-09T23:04:20.001 INFO:teuthology.orchestra.run.smithi003.stdout: "imported_inodes": 0, 2024-02-09T23:04:20.001 INFO:teuthology.orchestra.run.smithi003.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:20.001 INFO:teuthology.orchestra.run.smithi003.stdout: "openino_backtrace_fetch": 0, 2024-02-09T23:04:20.001 INFO:teuthology.orchestra.run.smithi003.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:20.001 INFO:teuthology.orchestra.run.smithi003.stdout: "root_rfiles": 0, 2024-02-09T23:04:20.002 INFO:teuthology.orchestra.run.smithi003.stdout: "root_rbytes": 0, 2024-02-09T23:04:20.002 INFO:teuthology.orchestra.run.smithi003.stdout: "root_rsnaps": 0, 2024-02-09T23:04:20.002 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_backtrace_fetch": 0, 2024-02-09T23:04:20.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:20.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:20.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:20.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:20.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:20.003 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "scrub_file_inodes": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_revoke": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_grant": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:20.004 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_caps": 0, 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_caps_dirty": 0, 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_cap_release": 0, 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "process_request_cap_release": 0 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_cache": { 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_update": 0, 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_update_receipt": 0, 2024-02-09T23:04:20.005 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_try_discover": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_send_discover": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "dir_handle_discover": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "num_strays": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_created": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_enqueued": 0, 2024-02-09T23:04:20.006 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "strays_migrated": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_started": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "recovery_completed": 0, 2024-02-09T23:04:20.007 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:20.008 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_exportdir": 0, 2024-02-09T23:04:20.008 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_flush": 0, 2024-02-09T23:04:20.008 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:20.008 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:20.008 INFO:teuthology.orchestra.run.smithi003.stdout: "ireq_inodestats": 0 2024-02-09T23:04:20.008 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.009 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_log": { 2024-02-09T23:04:20.009 INFO:teuthology.orchestra.run.smithi003.stdout: "evlrg": 0, 2024-02-09T23:04:20.009 INFO:teuthology.orchestra.run.smithi003.stdout: "evadd": 0, 2024-02-09T23:04:20.009 INFO:teuthology.orchestra.run.smithi003.stdout: "evex": 0, 2024-02-09T23:04:20.009 INFO:teuthology.orchestra.run.smithi003.stdout: "evtrm": 0, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "ev": 20458, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "evexg": 0, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "evexd": 0, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "segadd": 0, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "segex": 0, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "segtrm": 0, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "seg": 26, 2024-02-09T23:04:20.010 INFO:teuthology.orchestra.run.smithi003.stdout: "segmjr": 3, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "segexg": 0, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "segexd": 0, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "expos": 4194304, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "wrpos": 38019172, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "rdpos": 38014255, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "jlat": { 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.011 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: "replayed": 20458 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_mem": { 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: "ino": 232, 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: "ino+": 2582, 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: "ino-": 2350, 2024-02-09T23:04:20.012 INFO:teuthology.orchestra.run.smithi003.stdout: "dir": 25, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "dir+": 28, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "dir-": 3, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "dn": 2588, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "dn+": 2610, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "dn-": 22, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "cap": 0, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "cap+": 0, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "cap-": 0, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "rss": 53580, 2024-02-09T23:04:20.013 INFO:teuthology.orchestra.run.smithi003.stdout: "heap": 215420 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_server": { 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "dispatch_client_request": 0, 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "dispatch_server_request": 0, 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_request": 0, 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_client_session": 0, 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "handle_peer_request": 0, 2024-02-09T23:04:20.014 INFO:teuthology.orchestra.run.smithi003.stdout: "req_create_latency": { 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "req_getattr_latency": { 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.015 INFO:teuthology.orchestra.run.smithi003.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "req_link_latency": { 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.016 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookup_latency": { 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.017 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupino_latency": { 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.018 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupname_latency": { 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.019 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "req_lssnap_latency": { 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.020 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.021 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.021 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.021 INFO:teuthology.orchestra.run.smithi003.stdout: "req_mkdir_latency": { 2024-02-09T23:04:20.021 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.021 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: "req_mknod_latency": { 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.022 INFO:teuthology.orchestra.run.smithi003.stdout: "req_mksnap_latency": { 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "req_open_latency": { 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.023 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "req_readdir_latency": { 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rename_latency": { 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.024 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.025 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rmdir_latency": { 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.026 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.027 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.027 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.027 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:20.027 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.027 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.028 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setattr_latency": { 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.029 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.030 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setlayout_latency": { 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "req_setxattr_latency": { 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.031 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: "req_symlink_latency": { 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: "req_unlink_latency": { 2024-02-09T23:04:20.032 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.033 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_sessions": { 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: "session_count": 1, 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: "session_add": 3, 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: "session_remove": 2, 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: "sessions_open": 3, 2024-02-09T23:04:20.034 INFO:teuthology.orchestra.run.smithi003.stdout: "sessions_stale": 0, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "total_load": 0, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "average_load": 0, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "avg_session_uptime": 5195, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:20.035 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:20.036 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:20.036 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:20.037 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:20.037 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:20.037 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:20.037 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:20.038 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:20.038 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:20.038 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:20.038 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:20.038 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:20.039 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:20.040 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:20.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:20.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:20.041 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:20.042 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:20.042 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:20.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:20.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2024-02-09T23:04:20.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:20.043 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:20.044 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:20.044 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:20.044 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 214867, 2024-02-09T23:04:20.044 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 292, 2024-02-09T23:04:20.045 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:20.045 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:20.045 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2024-02-09T23:04:20.045 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2024-02-09T23:04:20.046 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:20.046 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:20.046 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:20.046 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:20.047 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:20.047 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 167, 2024-02-09T23:04:20.047 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:20.047 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:20.048 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:20.048 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2024-02-09T23:04:20.048 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 2463311, 2024-02-09T23:04:20.048 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 25363, 2024-02-09T23:04:20.049 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:20.049 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2024-02-09T23:04:20.049 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:20.049 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2024-02-09T23:04:20.049 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.050 INFO:teuthology.orchestra.run.smithi003.stdout: "objecter": { 2024-02-09T23:04:20.050 INFO:teuthology.orchestra.run.smithi003.stdout: "op_active": 0, 2024-02-09T23:04:20.050 INFO:teuthology.orchestra.run.smithi003.stdout: "op_laggy": 0, 2024-02-09T23:04:20.050 INFO:teuthology.orchestra.run.smithi003.stdout: "op_send": 3064, 2024-02-09T23:04:20.051 INFO:teuthology.orchestra.run.smithi003.stdout: "op_send_bytes": 21, 2024-02-09T23:04:20.051 INFO:teuthology.orchestra.run.smithi003.stdout: "op_resend": 3, 2024-02-09T23:04:20.051 INFO:teuthology.orchestra.run.smithi003.stdout: "op_reply": 3064, 2024-02-09T23:04:20.051 INFO:teuthology.orchestra.run.smithi003.stdout: "op_latency": { 2024-02-09T23:04:20.052 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 3064, 2024-02-09T23:04:20.052 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 1.566980939, 2024-02-09T23:04:20.052 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000511416 2024-02-09T23:04:20.052 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.052 INFO:teuthology.orchestra.run.smithi003.stdout: "op_inflight": 0, 2024-02-09T23:04:20.052 INFO:teuthology.orchestra.run.smithi003.stdout: "oplen_avg": { 2024-02-09T23:04:20.053 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 3064, 2024-02-09T23:04:20.054 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 3065 2024-02-09T23:04:20.054 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.055 INFO:teuthology.orchestra.run.smithi003.stdout: "op": 3064, 2024-02-09T23:04:20.055 INFO:teuthology.orchestra.run.smithi003.stdout: "op_r": 3064, 2024-02-09T23:04:20.055 INFO:teuthology.orchestra.run.smithi003.stdout: "op_w": 0, 2024-02-09T23:04:20.056 INFO:teuthology.orchestra.run.smithi003.stdout: "op_rmw": 0, 2024-02-09T23:04:20.056 INFO:teuthology.orchestra.run.smithi003.stdout: "op_pg": 0, 2024-02-09T23:04:20.056 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_stat": 1701, 2024-02-09T23:04:20.056 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_create": 0, 2024-02-09T23:04:20.057 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_read": 1361, 2024-02-09T23:04:20.057 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_write": 0, 2024-02-09T23:04:20.057 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_writefull": 0, 2024-02-09T23:04:20.058 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_writesame": 0, 2024-02-09T23:04:20.058 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_append": 0, 2024-02-09T23:04:20.058 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_zero": 0, 2024-02-09T23:04:20.058 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_truncate": 0, 2024-02-09T23:04:20.059 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_delete": 0, 2024-02-09T23:04:20.059 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_mapext": 0, 2024-02-09T23:04:20.059 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:20.059 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:20.060 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_getxattr": 1, 2024-02-09T23:04:20.060 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_setxattr": 0, 2024-02-09T23:04:20.060 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:20.060 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:20.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:20.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_call": 0, 2024-02-09T23:04:20.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_watch": 0, 2024-02-09T23:04:20.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_notify": 0, 2024-02-09T23:04:20.061 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_pgls": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "osdop_other": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_active": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_send": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_resend": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "linger_ping": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "poolop_active": 0, 2024-02-09T23:04:20.062 INFO:teuthology.orchestra.run.smithi003.stdout: "poolop_send": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "poolop_resend": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "poolstat_active": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "poolstat_send": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "poolstat_resend": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "statfs_active": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "statfs_send": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "statfs_resend": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "command_active": 0, 2024-02-09T23:04:20.063 INFO:teuthology.orchestra.run.smithi003.stdout: "command_send": 0, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "command_resend": 0, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "map_epoch": 106, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "map_full": 0, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "map_inc": 20, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_sessions": 7, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_session_open": 7, 2024-02-09T23:04:20.064 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_session_close": 0, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_laggy": 0, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_wr": 0, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_rd": 2, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_del": 0 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "oft": { 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_objs": 0, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:20.065 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_updates": 0, 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: "omap_total_removes": 0 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: "osd-slow-ops": { 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: "slow_ops_count": 0 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: "purge_queue": { 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:20.066 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing_ops_high_water": 0, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing": 0, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executing_high_water": 0, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executed_ops": 0, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_executed": 0, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:20.067 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 3381, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 36177335, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 3381, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:20.068 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 3381, 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 36177335, 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:20.069 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 3064, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 33837240, 2024-02-09T23:04:20.070 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 3064, 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 33837240, 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.071 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 1024, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 3064, 2024-02-09T23:04:20.072 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 3064, 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 3064, 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 3064, 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.073 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 3758096384, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:20.074 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 0, 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 0, 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.075 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-write_buf_throttle-0x55cf19054960": { 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 3758096384, 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 0, 2024-02-09T23:04:20.076 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:20.077 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:20.078 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:20.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.078 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:20.078 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:04:20.927 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-02-09T23:04:20.928 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-02-09T23:04:20.928 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=0 2024-02-09T23:04:20.928 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:20.928 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:20.928 DEBUG:tasks.check_counter:Getting stats from j 2024-02-09T23:04:20.928 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.j perf dump 2024-02-09T23:04:21.175 INFO:teuthology.orchestra.run.smithi003.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.a/config 2024-02-09T23:04:21.611 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:21 smithi003 ceph-mon[32678]: pgmap v413: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1024 KiB/s wr, 62 op/s 2024-02-09T23:04:21.779 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:21 smithi089 ceph-mon[36665]: pgmap v413: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1024 KiB/s wr, 62 op/s 2024-02-09T23:04:21.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:21 smithi064 ceph-mon[36638]: pgmap v413: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 1024 KiB/s wr, 62 op/s 2024-02-09T23:04:22.871 INFO:teuthology.orchestra.run.smithi003.stdout:{ 2024-02-09T23:04:22.871 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:22.871 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 0, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 2, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 0, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 0, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 4, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 0, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.001189503, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.000000000, 2024-02-09T23:04:22.872 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.000215214, 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:22.873 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 0, 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 0 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:22.874 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 264, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 253, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 1581088, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 152112, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 2, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 1, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.048810416, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.015965855, 2024-02-09T23:04:22.875 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.036438126, 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.004765319, 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 252, 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.010319574, 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000040950 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:22.876 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 1581088, 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 152112 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_messages": 6, 2024-02-09T23:04:22.877 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages": 195, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_bytes": 3312, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_bytes": 189120, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_created_connections": 5, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_total_time": 0.015915116, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_send_time": 0.010695943, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_recv_time": 0.002735758, 2024-02-09T23:04:22.878 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-02-09T23:04:22.879 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:22.879 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 193, 2024-02-09T23:04:22.879 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.015444492, 2024-02-09T23:04:22.879 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000080023 2024-02-09T23:04:22.879 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.879 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: "msgr_send_encrypted_bytes": 189120 2024-02-09T23:04:22.880 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "cct": { 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "total_workers": 0, 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "unhealthy_workers": 0 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "mempool": { 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:22.881 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:22.882 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:22.883 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:22.884 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_items": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_bytes": 148994, 2024-02-09T23:04:22.885 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_anon_items": 31, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_bytes": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_items": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:22.886 INFO:teuthology.orchestra.run.smithi003.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_bytes": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_items": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "pgmap_items": 0, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_bytes": 928, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "mds_co_items": 2, 2024-02-09T23:04:22.887 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_1_items": 0, 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "unittest_2_items": 0 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: }, 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "val": 0, 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "max": 104857600, 2024-02-09T23:04:22.888 INFO:teuthology.orchestra.run.smithi003.stdout: "get_started": 0, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "get": 266, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "get_sum": 1543839, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "get_or_fail_success": 266, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "take": 0, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "take_sum": 0, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "put": 266, 2024-02-09T23:04:22.889 INFO:teuthology.orchestra.run.smithi003.stdout: "put_sum": 1543839, 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout: "wait": { 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout: "avgcount": 0, 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout: "sum": 0.000000000, 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout: "avgtime": 0.000000000 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout: } 2024-02-09T23:04:22.890 INFO:teuthology.orchestra.run.smithi003.stdout:} 2024-02-09T23:04:23.541 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-02-09T23:04:23.541 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-02-09T23:04:23.541 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-02-09T23:04:23.541 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-02-09T23:04:23.541 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-02-09T23:04:23.541 DEBUG:tasks.check_counter:Getting stats from b 2024-02-09T23:04:23.542 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.b perf dump 2024-02-09T23:04:23.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:23 smithi003 ceph-mon[32678]: pgmap v414: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 652 KiB/s wr, 37 op/s 2024-02-09T23:04:23.804 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T23:04:23.926 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:23 smithi064 ceph-mon[36638]: pgmap v414: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 652 KiB/s wr, 37 op/s 2024-02-09T23:04:23.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:23 smithi089 ceph-mon[36665]: pgmap v414: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 652 KiB/s wr, 37 op/s 2024-02-09T23:04:25.495 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:25 smithi064 ceph-mon[36638]: pgmap v415: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 652 KiB/s wr, 37 op/s 2024-02-09T23:04:25.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:25 smithi003 ceph-mon[32678]: pgmap v415: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 652 KiB/s wr, 37 op/s 2024-02-09T23:04:25.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:25 smithi089 ceph-mon[36665]: pgmap v415: 97 pgs: 97 active+clean; 208 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 652 KiB/s wr, 37 op/s 2024-02-09T23:04:25.764 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 46367, 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 55174, 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 13031878, 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 78234476, 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 13, 2024-02-09T23:04:25.765 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 6, 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 3.300930702, 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 1.456936145, 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 1.807685289, 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.251758930, 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 55173, 2024-02-09T23:04:25.766 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 2.530758715, 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000045869 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 81379, 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.013772354, 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000169 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.767 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1847936, 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 565696 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 101855, 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 124441, 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 27646644, 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 113960056, 2024-02-09T23:04:25.768 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 26, 2024-02-09T23:04:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 5.804177561, 2024-02-09T23:04:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 2.284883792, 2024-02-09T23:04:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 3.709511382, 2024-02-09T23:04:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.348896710, 2024-02-09T23:04:25.769 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 124443, 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 4.922210588, 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000039553 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 202548, 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.029744195, 2024-02-09T23:04:25.770 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000146 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 441072, 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 254112 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 136376, 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 163399, 2024-02-09T23:04:25.771 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 37128715, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 175228247, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 309, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 9, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 7.890391204, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 3.305917847, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 5.211332169, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.385346460, 2024-02-09T23:04:25.772 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 163393, 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 4.017970432, 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000024590 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 248246, 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.036748089, 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000148 2024-02-09T23:04:25.773 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 98064, 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 30720 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-02-09T23:04:25.774 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 51977, 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 23.280701016, 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000447903 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:25.775 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 4847, 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.372836695, 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000076921 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 51152, 2024-02-09T23:04:25.776 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 42599, 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 42599, 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 421.456290700, 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.009893572 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 8553, 2024-02-09T23:04:25.777 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 11, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 3, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 1022, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 1, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 1, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 247, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 64, 2024-02-09T23:04:25.778 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 155, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 28, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 155, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 874, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 62, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 62, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 17, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 80932, 2024-02-09T23:04:25.779 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 61554, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 8545, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 176, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 213, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 24438, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 0, 2024-02-09T23:04:25.780 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 15, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 249, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 3, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 39, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 922, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 87, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 20477221, 2024-02-09T23:04:25.781 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 6, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 922, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 461, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 24, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 437, 2024-02-09T23:04:25.782 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 461, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 15, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 19733, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 9661, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 70, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 24737, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 2801, 2024-02-09T23:04:25.783 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 1389, 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 18519 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 8, 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 0, 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 0, 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 176, 2024-02-09T23:04:25.784 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 154, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 14, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 2427, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 2413, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:25.785 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 24, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 16, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 2, 2024-02-09T23:04:25.786 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 34429, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 34097, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 34097, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 332, 2024-02-09T23:04:25.787 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 857, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 112, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 111, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 111, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 1, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 1, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-02-09T23:04:25.788 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 3, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 214315458, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 216347274, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 0, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 25829, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 52.248981519, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.002022880 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.789 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 0 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 182, 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 3165, 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 2983, 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 62, 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 80, 2024-02-09T23:04:25.790 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 18, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 253, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 5963, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 5710, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 40, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 3036, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 2996, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 68996, 2024-02-09T23:04:25.791 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 215420 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 76887, 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 20565, 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 51152, 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 392, 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 38935, 2024-02-09T23:04:25.792 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 2601, 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 4.210188100, 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001618680 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 21079, 2024-02-09T23:04:25.793 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 100.750854275, 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.004779679 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.794 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 4290, 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 297.991773040, 2024-02-09T23:04:25.795 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.069461951 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.796 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.797 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.798 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.799 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.799 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.799 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 13, 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.002556969, 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000196689 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 20, 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.026989415, 2024-02-09T23:04:25.800 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001349470 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 13, 2024-02-09T23:04:25.801 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 1.008890220, 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.077606940 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-02-09T23:04:25.802 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 11806, 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 10.597213602, 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000897612 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 295, 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 3.093166642, 2024-02-09T23:04:25.803 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.010485310 2024-02-09T23:04:25.804 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.804 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:25.804 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.804 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-02-09T23:04:25.805 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 7, 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.050173004, 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.007167572 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.806 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 1, 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.001195596, 2024-02-09T23:04:25.807 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001195596 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 40, 2024-02-09T23:04:25.808 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.013258015, 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000331450 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-02-09T23:04:25.809 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 7, 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.020039102, 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.002862728 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.810 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 2427, 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 3.689992720, 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001520392 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:25.811 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 2, 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 5, 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 3, 2024-02-09T23:04:25.812 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 2, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 6635, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 3317, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 771, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:25.813 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:25.814 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:25.815 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:25.816 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:25.817 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 605404, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 318, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 88, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 1, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:25.818 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 167, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-02-09T23:04:25.819 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 1007310, 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 17585, 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-02-09T23:04:25.820 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 34412, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 218759566, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 32, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 34412, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 34412, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 116.031579898, 2024-02-09T23:04:25.821 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.003371834 2024-02-09T23:04:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-02-09T23:04:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-02-09T23:04:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 34412, 2024-02-09T23:04:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 41389 2024-02-09T23:04:25.822 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 34412, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 1232, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 33180, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 981, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 2084, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 272, 2024-02-09T23:04:25.823 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 26077, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 391, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 1, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 3027, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-02-09T23:04:25.824 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 959, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 4168, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-02-09T23:04:25.825 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-02-09T23:04:25.826 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-02-09T23:04:25.827 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 106, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 21, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 12, 2024-02-09T23:04:25.828 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 12, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 0, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 2926, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 28, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 475 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 1, 2024-02-09T23:04:25.829 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 0 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:25.830 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 6, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 6, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 2413, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 2413, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:25.831 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 284594, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 52114444, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 284594, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 284594, 2024-02-09T23:04:25.832 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 52114444, 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:25.833 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 34412, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 219003539, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 34412, 2024-02-09T23:04:25.834 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 219003539, 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:25.835 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 34412, 2024-02-09T23:04:25.836 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 34412, 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 34412, 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 34412, 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.837 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 2413, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 244161, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 2413, 2024-02-09T23:04:25.838 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 248, 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 244161, 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.839 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x5651d4e94640": { 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 13178, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 34429, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 212152970, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:25.840 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 34429, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 25829, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 212139792, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:25.841 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.842 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:25.842 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-09T23:04:26.571 INFO:tasks.check_counter:Daemon mds.b mds.exported=15 2024-02-09T23:04:26.571 INFO:tasks.check_counter:Daemon mds.b mds.imported=3 2024-02-09T23:04:26.572 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-02-09T23:04:26.572 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=13 2024-02-09T23:04:26.572 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=7 2024-02-09T23:04:26.572 DEBUG:tasks.check_counter:Getting stats from e 2024-02-09T23:04:26.572 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.e perf dump 2024-02-09T23:04:26.829 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T23:04:27.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:27 smithi003 ceph-mon[32678]: pgmap v416: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.2 MiB/s wr, 63 op/s 2024-02-09T23:04:27.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:27 smithi064 ceph-mon[36638]: pgmap v416: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.2 MiB/s wr, 63 op/s 2024-02-09T23:04:27.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:27 smithi089 ceph-mon[36665]: pgmap v416: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 1.2 MiB/s wr, 63 op/s 2024-02-09T23:04:28.486 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-09T23:04:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:28.486 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 107, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 109, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 4232619, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 33296, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 7, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.012147023, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.003504855, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.004122462, 2024-02-09T23:04:28.487 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.002548916, 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 107, 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.006542633, 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000061146 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:28.488 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:28.489 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 197, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 191, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 15295174, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 64486, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 4, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.023697616, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.004331250, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.015400788, 2024-02-09T23:04:28.490 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.004162402, 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 190, 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.007864015, 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000041389 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.491 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 196288, 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 16224 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 62, 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 79, 2024-02-09T23:04:28.492 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 8406799, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 135264, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 5, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 4, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.011146772, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.001734754, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.007644954, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.001263184, 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:28.493 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 78, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.004118698, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000052803 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-02-09T23:04:28.494 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 117536 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:28.495 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 533, 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.015898802, 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000029828 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:28.496 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 0, 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 0, 2024-02-09T23:04:28.497 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 0, 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:28.498 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 0, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 0, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 0, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 310, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 154, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 156, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 0, 2024-02-09T23:04:28.499 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 148, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 0, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 0, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 0, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 14, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 0, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 0, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 0, 2024-02-09T23:04:28.500 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 0, 2024-02-09T23:04:28.501 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 0, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 0, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 87, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 21257509, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 6, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 0, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:28.502 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 0, 2024-02-09T23:04:28.503 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 0, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 0 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.504 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:28.505 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-02-09T23:04:28.506 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 0, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.507 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 0, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 0, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 0, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 3584, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 0, 2024-02-09T23:04:28.508 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 0, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 0, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 0, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 2, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 5, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 0, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 214315458, 2024-02-09T23:04:28.509 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 216455732, 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 216445142, 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 3584 2024-02-09T23:04:28.510 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 92, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 318, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 226, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 36, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 36, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 0, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 310, 2024-02-09T23:04:28.511 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 310, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 0, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 0, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 0, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 0, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 47096, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 215396 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.512 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 0, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 0, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 0, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 0, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 0, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.513 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.514 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.515 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.516 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.517 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.518 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-02-09T23:04:28.519 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-02-09T23:04:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.520 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.521 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-02-09T23:04:28.522 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.523 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:28.524 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.525 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.526 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.527 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.528 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-02-09T23:04:28.529 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.530 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.531 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.532 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 2, 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 2, 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 0, 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 0, 2024-02-09T23:04:28.533 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 0, 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 0, 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 2494, 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-02-09T23:04:28.534 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:28.535 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:28.536 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:28.537 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:28.538 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 219194, 2024-02-09T23:04:28.539 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 233, 2024-02-09T23:04:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-02-09T23:04:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-02-09T23:04:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:28.540 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 167, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-02-09T23:04:28.541 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 594857, 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 6749, 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-02-09T23:04:28.542 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 319, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 26, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 4, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 319, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 319, 2024-02-09T23:04:28.543 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.206995714, 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000648889 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 319, 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 320 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.544 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 319, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 319, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 0, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 165, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 0, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 151, 2024-02-09T23:04:28.545 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-02-09T23:04:28.546 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 2, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-02-09T23:04:28.547 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-02-09T23:04:28.548 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-02-09T23:04:28.549 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 106, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 3, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 8, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 8, 2024-02-09T23:04:28.550 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 0, 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 0, 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 2, 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 0 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 0, 2024-02-09T23:04:28.551 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 0 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:28.552 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 0, 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 0, 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 0, 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 0, 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:28.553 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 362, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 27896905, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 362, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:28.554 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 362, 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 27896905, 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:28.555 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 319, 2024-02-09T23:04:28.556 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 27641649, 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 319, 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 27641649, 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.557 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:28.558 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 319, 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 319, 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 319, 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 319, 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.559 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:28.560 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.561 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x556f408c8960": { 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:28.562 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:28.563 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.564 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:28.564 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-09T23:04:29.508 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-02-09T23:04:29.508 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-02-09T23:04:29.508 INFO:tasks.check_counter:Daemon mds.e mds.root_rsnaps=6 2024-02-09T23:04:29.509 INFO:tasks.check_counter:Daemon mds.e mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:29.509 INFO:tasks.check_counter:Daemon mds.e mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:29.509 DEBUG:tasks.check_counter:Getting stats from h 2024-02-09T23:04:29.509 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.h perf dump 2024-02-09T23:04:29.765 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T23:04:29.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:29 smithi003 ceph-mon[32678]: pgmap v417: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 878 KiB/s wr, 46 op/s 2024-02-09T23:04:29.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:29 smithi003 ceph-mon[32678]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:04:29.887 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:29 smithi064 ceph-mon[36638]: pgmap v417: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 878 KiB/s wr, 46 op/s 2024-02-09T23:04:29.888 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:29 smithi064 ceph-mon[36638]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:04:29.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:29 smithi089 ceph-mon[36665]: pgmap v417: 97 pgs: 97 active+clean; 210 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 878 KiB/s wr, 46 op/s 2024-02-09T23:04:29.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:29 smithi089 ceph-mon[36665]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-02-09T23:04:31.386 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-09T23:04:31.386 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:31.386 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 51132, 2024-02-09T23:04:31.386 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 42588, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 18864292, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 28735342, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 10, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 4, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 2.999013683, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 1.306325531, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 2.022719000, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.131474008, 2024-02-09T23:04:31.387 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 42587, 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.713656554, 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000016757 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 82773, 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.012397698, 2024-02-09T23:04:31.388 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000149 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1849808, 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 180432 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 40046, 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 67369, 2024-02-09T23:04:31.389 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 12471362, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 51825312, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 20, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 2.413937452, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 1.083816585, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 1.368410455, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.073956055, 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:31.390 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 67367, 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 1.047747022, 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000015552 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 90495, 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.013655721, 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000150 2024-02-09T23:04:31.391 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 5248, 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 504368 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 4918, 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 4933, 2024-02-09T23:04:31.392 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 1906808, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 15648390, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 55, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 9, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.379034702, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.122271717, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.106676942, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.123666865, 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:31.393 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 4931, 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.150340380, 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000030488 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 234, 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000055710, 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000238 2024-02-09T23:04:31.394 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 0, 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 0 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.395 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 11448, 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 6.993684969, 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000610908 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.396 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 9892, 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.517713421, 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000052336 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.397 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 29666, 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 28560, 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 28560, 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 26.213744414, 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000917848 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-02-09T23:04:31.398 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 1106, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 11, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 0, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 0, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 214, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 39, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 113, 2024-02-09T23:04:31.399 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 62, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 157, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 70, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 14, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 14, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 10, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 54491, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 22719, 2024-02-09T23:04:31.400 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 1102, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 11, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 1, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 5053, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 1, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 2, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 39, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 5, 2024-02-09T23:04:31.401 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 185, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 81, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 87, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 6407151, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 6, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 81, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:31.402 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 81, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 2693, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 18313, 2024-02-09T23:04:31.403 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 21, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 3994, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 467, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 323, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 27362 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-02-09T23:04:31.404 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 21, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 6, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 4, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 14, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 29, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 0, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 4959, 2024-02-09T23:04:31.405 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 4959, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:31.406 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 2, 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-02-09T23:04:31.407 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 21326, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 0, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 0, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 21326, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 0, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 26, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 0, 2024-02-09T23:04:31.408 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 0, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 26, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 3, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 0, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 4194304, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 38822612, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 0, 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-02-09T23:04:31.409 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 5517, 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 10.493953375, 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001902112 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 0 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 165, 2024-02-09T23:04:31.410 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 5169, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 5004, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 33, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 45, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 12, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 212, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 7796, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 7584, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 14, 2024-02-09T23:04:31.411 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 5121, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 5107, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 58152, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 207228 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 48742, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 79, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 29666, 2024-02-09T23:04:31.412 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 22, 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 5176, 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 4964, 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 9.754111481, 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001964970 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-02-09T23:04:31.413 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 138, 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.092680690, 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000671599 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.414 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 6021, 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 5.836312612, 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000969326 2024-02-09T23:04:31.415 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.416 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:31.417 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.418 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-02-09T23:04:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.419 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.420 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.421 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-02-09T23:04:31.422 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 12478, 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 5.062688278, 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000405729 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.423 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.424 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:31.425 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.426 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.427 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.428 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-02-09T23:04:31.429 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 4959, 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 5.467951353, 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001102631 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.430 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-02-09T23:04:31.431 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 1, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 2, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 1, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 1, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 5234, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 5234, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 738, 2024-02-09T23:04:31.432 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:31.433 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:31.434 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:31.435 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-02-09T23:04:31.436 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 365569, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 272, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 176, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 2, 2024-02-09T23:04:31.437 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 167, 2024-02-09T23:04:31.438 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 641561, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 5367, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-02-09T23:04:31.439 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 12051, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 35605593, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 33, 2024-02-09T23:04:31.440 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 12051, 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 12051, 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 22.938522784, 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.001903453 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-02-09T23:04:31.441 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 12051, 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 12604 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 12051, 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 485, 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 11566, 2024-02-09T23:04:31.442 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 0, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 10, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 404, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 5921, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 269, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-02-09T23:04:31.443 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 0, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 4979, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 81, 2024-02-09T23:04:31.444 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 20, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-02-09T23:04:31.445 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-02-09T23:04:31.446 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-02-09T23:04:31.447 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 106, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 20, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 12, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 12, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 0, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 664, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 0, 2024-02-09T23:04:31.448 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 256 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 1, 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-02-09T23:04:31.449 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 0 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 6, 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 6, 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 4959, 2024-02-09T23:04:31.450 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 4959, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 96092, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 24571176, 2024-02-09T23:04:31.451 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 96092, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 96092, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 24571176, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.452 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:31.453 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 12051, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 36106636, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 12051, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 36106636, 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:31.454 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:31.455 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 12051, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 12051, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 12051, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 12051, 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:31.456 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:31.457 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 4959, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 501043, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 4959, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 404, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 501043, 2024-02-09T23:04:31.458 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x556a46b8c640": { 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 599, 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:31.459 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 21326, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 34628308, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 21326, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 5517, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 34627709, 2024-02-09T23:04:31.460 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:31.461 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:31.461 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:31.461 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:31.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.461 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:31.461 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-09T23:04:31.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:31 smithi064 ceph-mon[36638]: pgmap v418: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-02-09T23:04:31.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:31 smithi003 ceph-mon[32678]: pgmap v418: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-02-09T23:04:31.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:31 smithi089 ceph-mon[36665]: pgmap v418: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 1.1 MiB/s wr, 65 op/s 2024-02-09T23:04:32.098 INFO:tasks.check_counter:Daemon mds.h mds.exported=2 2024-02-09T23:04:32.099 INFO:tasks.check_counter:Daemon mds.h mds.imported=5 2024-02-09T23:04:32.099 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=6 2024-02-09T23:04:32.099 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:32.099 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:32.099 DEBUG:tasks.check_counter:Getting stats from k 2024-02-09T23:04:32.099 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.k perf dump 2024-02-09T23:04:32.350 INFO:teuthology.orchestra.run.smithi064.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.b/config 2024-02-09T23:04:33.789 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:33 smithi064 ceph-mon[36638]: pgmap v419: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 811 KiB/s wr, 45 op/s 2024-02-09T23:04:33.789 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:33 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:04:33.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:33 smithi003 ceph-mon[32678]: pgmap v419: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 811 KiB/s wr, 45 op/s 2024-02-09T23:04:33.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:33 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:04:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:33 smithi089 ceph-mon[36665]: pgmap v419: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 811 KiB/s wr, 45 op/s 2024-02-09T23:04:33.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:33 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-02-09T23:04:34.018 INFO:teuthology.orchestra.run.smithi064.stdout:{ 2024-02-09T23:04:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 2504, 2024-02-09T23:04:34.018 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 2504, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 22764416, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 777955, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 7, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.201401970, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.060125935, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.064169101, 2024-02-09T23:04:34.019 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.060676815, 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 2503, 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.071431913, 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000028538 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.020 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 448 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 730, 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 686, 2024-02-09T23:04:34.021 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 10308094, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 284353, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 4, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 3, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.091702989, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.024553226, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.058651933, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.017094043, 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:34.022 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 685, 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.023015014, 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000033598 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.023 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 1817488, 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 158336 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_messages": 142, 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages": 336, 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_bytes": 4239720, 2024-02-09T23:04:34.024 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_bytes": 543748, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_created_connections": 8, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_total_time": 0.023662163, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_send_time": 0.010515230, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_recv_time": 0.006963724, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_running_fast_dispatch_time": 0.003133932, 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:34.025 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 333, 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.019433424, 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000058358 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.026 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_recv_encrypted_bytes": 320, 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "msgr_send_encrypted_bytes": 500208 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "cct": { 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "total_workers": 1, 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "unhealthy_workers": 0 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-MDSRank": { 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:34.027 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 5184, 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.111797738, 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000021565 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "queue_len": 0, 2024-02-09T23:04:34.028 INFO:teuthology.orchestra.run.smithi064.stdout: "complete_latency": { 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "mds": { 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "request": 0, 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "reply": 0, 2024-02-09T23:04:34.029 INFO:teuthology.orchestra.run.smithi064.stdout: "reply_latency": { 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_reply": 0, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "forward": 0, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:34.030 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_commit": 0, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_split": 0, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_merge": 0, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes": 5166, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_top": 37, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_bottom": 5129, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pin_tail": 0, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_pinned": 5112, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_expired": 0, 2024-02-09T23:04:34.031 INFO:teuthology.orchestra.run.smithi064.stdout: "inodes_with_caps": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "caps": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "subtrees": 5, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_hit": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_forward": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_discover": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:34.032 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "traverse_lock": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "load_cent": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "q": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "exported": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "exported_inodes": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "imported": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "imported_inodes": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:34.033 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_backtrace_fetch": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rfiles": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rbytes": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "root_rsnaps": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_fetch": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:34.034 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "scrub_file_inodes": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_revoke": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_grant": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-02-09T23:04:34.035 INFO:teuthology.orchestra.run.smithi064.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps": 0, 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_caps_dirty": 0, 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_cap_release": 0, 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "process_request_cap_release": 0 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_cache": { 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update": 0, 2024-02-09T23:04:34.036 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_update_receipt": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_try_discover": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_send_discover": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "dir_handle_discover": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_created": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_enqueued": 0, 2024-02-09T23:04:34.037 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "strays_migrated": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_started": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "recovery_completed": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_exportdir": 0, 2024-02-09T23:04:34.038 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_flush": 0, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "ireq_inodestats": 0 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_log": { 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "evlrg": 0, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "evadd": 0, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "evex": 0, 2024-02-09T23:04:34.039 INFO:teuthology.orchestra.run.smithi064.stdout: "evtrm": 0, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "ev": 21325, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "evexg": 0, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "evexd": 0, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "segadd": 0, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "segex": 0, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "segtrm": 0, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "seg": 26, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "segmjr": 3, 2024-02-09T23:04:34.040 INFO:teuthology.orchestra.run.smithi064.stdout: "segexg": 0, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "segexd": 0, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "expos": 4194304, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "wrpos": 38822013, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "rdpos": 38814722, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "jlat": { 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.041 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "replayed": 21325 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_mem": { 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "ino": 160, 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "ino+": 5151, 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "ino-": 4991, 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "dir": 24, 2024-02-09T23:04:34.042 INFO:teuthology.orchestra.run.smithi064.stdout: "dir+": 26, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "dir-": 2, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "dn": 5166, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "dn+": 5207, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "dn-": 41, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cap": 0, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cap+": 0, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "cap-": 0, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "rss": 52732, 2024-02-09T23:04:34.043 INFO:teuthology.orchestra.run.smithi064.stdout: "heap": 215420 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_server": { 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_client_request": 0, 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "dispatch_server_request": 0, 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_request": 0, 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_client_session": 0, 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "handle_peer_request": 0, 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "req_create_latency": { 2024-02-09T23:04:34.044 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getattr_latency": { 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.045 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "req_link_latency": { 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.046 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookup_latency": { 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.047 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupino_latency": { 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.048 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupname_latency": { 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.049 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "req_lssnap_latency": { 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.050 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mkdir_latency": { 2024-02-09T23:04:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.051 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mknod_latency": { 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.052 INFO:teuthology.orchestra.run.smithi064.stdout: "req_mksnap_latency": { 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "req_open_latency": { 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.053 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "req_readdir_latency": { 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rename_latency": { 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.054 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.055 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmdir_latency": { 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.056 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:34.057 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setattr_latency": { 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.058 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.059 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setlayout_latency": { 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.060 INFO:teuthology.orchestra.run.smithi064.stdout: "req_setxattr_latency": { 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "req_symlink_latency": { 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.061 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "req_unlink_latency": { 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:34.062 INFO:teuthology.orchestra.run.smithi064.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_sessions": { 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "session_count": 1, 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "session_add": 2, 2024-02-09T23:04:34.063 INFO:teuthology.orchestra.run.smithi064.stdout: "session_remove": 1, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_open": 1, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "sessions_stale": 0, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "total_load": 0, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "average_load": 0, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "avg_session_uptime": 2496, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "mempool": { 2024-02-09T23:04:34.064 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:34.065 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:34.066 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:34.067 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_items": 0, 2024-02-09T23:04:34.068 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_bytes": 212799, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_anon_items": 217, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_bytes": 0, 2024-02-09T23:04:34.069 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_items": 0, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_items": 167, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:34.070 INFO:teuthology.orchestra.run.smithi064.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "pgmap_items": 0, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_bytes": 4085171, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "mds_co_items": 24593, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_1_items": 0, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: "unittest_2_items": 0 2024-02-09T23:04:34.071 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "objecter": { 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_active": 0, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_laggy": 0, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send": 3045, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_send_bytes": 21, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_resend": 3, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_reply": 3045, 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "op_latency": { 2024-02-09T23:04:34.072 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3045, 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 1.548967740, 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000508692 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "op_inflight": 0, 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "oplen_avg": { 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 3045, 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 3046 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.073 INFO:teuthology.orchestra.run.smithi064.stdout: "op": 3045, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "op_r": 3045, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "op_w": 0, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "op_rmw": 0, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "op_pg": 0, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_stat": 1752, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_create": 0, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_read": 1291, 2024-02-09T23:04:34.074 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_write": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writefull": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_writesame": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_append": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_zero": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_truncate": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_delete": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_mapext": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:34.075 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_getxattr": 1, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_setxattr": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_call": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_watch": 0, 2024-02-09T23:04:34.076 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_notify": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "osdop_other": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_active": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_send": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_resend": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "linger_ping": 0, 2024-02-09T23:04:34.077 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_active": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_send": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolop_resend": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_active": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_send": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "poolstat_resend": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_active": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_send": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "statfs_resend": 0, 2024-02-09T23:04:34.078 INFO:teuthology.orchestra.run.smithi064.stdout: "command_active": 0, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "command_send": 0, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "command_resend": 0, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "map_epoch": 106, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "map_full": 0, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "map_inc": 20, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_sessions": 7, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_open": 8, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_session_close": 1, 2024-02-09T23:04:34.079 INFO:teuthology.orchestra.run.smithi064.stdout: "osd_laggy": 0, 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_wr": 0, 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_rd": 2, 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_del": 0 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "oft": { 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_objs": 0, 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:34.080 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_updates": 0, 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "omap_total_removes": 0 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "osd-slow-ops": { 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "slow_ops_count": 0 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "purge_queue": { 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_ops_high_water": 0, 2024-02-09T23:04:34.081 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing": 0, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executing_high_water": 0, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed_ops": 0, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_executed": 0, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:34.082 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 3372, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 36989372, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 3372, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 3372, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 36989372, 2024-02-09T23:04:34.083 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 104857600, 2024-02-09T23:04:34.084 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 3045, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 34643068, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 3045, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 34643068, 2024-02-09T23:04:34.085 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:34.086 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 1024, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 3045, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 3045, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 3045, 2024-02-09T23:04:34.087 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 3045, 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:34.088 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:34.089 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: }, 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "throttle-write_buf_throttle-0x55e5fee62960": { 2024-02-09T23:04:34.090 INFO:teuthology.orchestra.run.smithi064.stdout: "val": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "max": 3758096384, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "get_started": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "get": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "get_sum": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "take": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "take_sum": 0, 2024-02-09T23:04:34.091 INFO:teuthology.orchestra.run.smithi064.stdout: "put": 0, 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "put_sum": 0, 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "wait": { 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "avgcount": 0, 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "sum": 0.000000000, 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: "avgtime": 0.000000000 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout: } 2024-02-09T23:04:34.092 INFO:teuthology.orchestra.run.smithi064.stdout:} 2024-02-09T23:04:34.847 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-02-09T23:04:34.847 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-02-09T23:04:34.847 INFO:tasks.check_counter:Daemon mds.k mds.root_rsnaps=0 2024-02-09T23:04:34.848 INFO:tasks.check_counter:Daemon mds.k mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:34.848 INFO:tasks.check_counter:Daemon mds.k mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:34.848 DEBUG:tasks.check_counter:Getting stats from c 2024-02-09T23:04:34.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.c perf dump 2024-02-09T23:04:35.107 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T23:04:35.569 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:35 smithi089 ceph-mon[36665]: pgmap v420: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 811 KiB/s wr, 45 op/s 2024-02-09T23:04:35.569 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:35 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:04:35.569 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:35 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:04:35.569 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:35 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:04:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:35 smithi003 ceph-mon[32678]: pgmap v420: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 811 KiB/s wr, 45 op/s 2024-02-09T23:04:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:35 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:04:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:35 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:04:35.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:35 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:04:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:35 smithi064 ceph-mon[36638]: pgmap v420: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 811 KiB/s wr, 45 op/s 2024-02-09T23:04:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:35 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-02-09T23:04:35.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:35 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-02-09T23:04:35.686 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:35 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' 2024-02-09T23:04:36.943 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 102067, 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 109253, 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 34701319, 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 71347266, 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 10, 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:36.944 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 4.717768724, 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 1.996944973, 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 3.520204202, 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.084111471, 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 109252, 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.623515608, 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000014860 2024-02-09T23:04:36.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 193270, 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.023820991, 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000123 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1841872, 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 183472 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.946 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 5211, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 5222, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1585783, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 27131325, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 14, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.379570112, 2024-02-09T23:04:36.947 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.148695541, 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.088938825, 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.119974095, 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5224, 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.147295821, 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000028195 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:36.948 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 338, 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000085280, 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000252 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 160, 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 128624 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 4770, 2024-02-09T23:04:36.949 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 3977, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1474217, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 10493236, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 60, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 12, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.290144702, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.110611285, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.119920833, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.061300461, 2024-02-09T23:04:36.950 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3971, 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.107359545, 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000027035 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2936, 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000353648, 2024-02-09T23:04:36.951 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000120 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 374336 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.952 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 11519, 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 7.040481635, 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000611205 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:36.953 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 6992, 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.350874362, 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000050182 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-09T23:04:36.954 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 25817, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 24242, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 24242, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 25.370778551, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001046562 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 0, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 1575, 2024-02-09T23:04:36.955 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 11, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 0, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 197, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 11, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 5, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 181, 2024-02-09T23:04:36.956 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 181, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 84, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 32, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 32, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 10, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 54629, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 28725, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 1575, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 11, 2024-02-09T23:04:36.957 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 1277, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 6138, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 0, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 2, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 75, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 4, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 63, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:36.958 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 384, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 87, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 6407151, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 6, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 384, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:36.959 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 384, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 5719, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 19708, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:36.960 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 124, 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 10738, 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 2637, 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 1217, 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 23323 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-02-09T23:04:36.961 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 1, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 21, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 12, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 29, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 1, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 0, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 3460, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 3460, 2024-02-09T23:04:36.962 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 2, 2024-02-09T23:04:36.963 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 23201, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 0, 2024-02-09T23:04:36.964 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 0, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 23201, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 0, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 29, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 0, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 0, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 29, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 3, 2024-02-09T23:04:36.965 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 0, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 4194304, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 45703299, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 0, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5413, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 8.813404319, 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001628192 2024-02-09T23:04:36.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 0 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 154, 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 3672, 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 3518, 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 31, 2024-02-09T23:04:36.967 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 47, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 16, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 163, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 5445, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 5282, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 28, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 3615, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 3587, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 62212, 2024-02-09T23:04:36.968 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 207228 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 42857, 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 8, 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 25817, 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 521, 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 5011, 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-02-09T23:04:36.969 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3459, 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 6.065412545, 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001753516 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 20, 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.118946500, 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.005947325 2024-02-09T23:04:36.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.971 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 6657, 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 8.302746010, 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001247220 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:36.972 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1, 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000127769, 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000127769 2024-02-09T23:04:36.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.974 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-02-09T23:04:36.975 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-02-09T23:04:36.976 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.977 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 9360, 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 4.473067558, 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000477891 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1277, 2024-02-09T23:04:36.979 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 2.441719658, 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001912074 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:36.980 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.982 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8, 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.020428396, 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002553549 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.983 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.984 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-02-09T23:04:36.987 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3460, 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3.948330115, 2024-02-09T23:04:36.989 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001141135 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:36.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 1, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 1, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 0, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 1, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 4701, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 4701, 2024-02-09T23:04:36.991 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 796, 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:36.992 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:36.993 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:36.994 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:36.995 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 714515, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 346, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:36.996 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 167, 2024-02-09T23:04:36.997 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 725524, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 12348, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:36.998 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 10832, 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 42293053, 2024-02-09T23:04:36.999 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 33, 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 10832, 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 10832, 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 18.906572181, 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001745436 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-02-09T23:04:37.000 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 10832, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 11450 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 10832, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 798, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 10034, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-02-09T23:04:37.001 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 0, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 10, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 414, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 5827, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 269, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 0, 2024-02-09T23:04:37.002 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 3481, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 384, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 20, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:37.003 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-02-09T23:04:37.004 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-02-09T23:04:37.005 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 106, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 20, 2024-02-09T23:04:37.006 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 12, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 12, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 0, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 804, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 0, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 241 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-02-09T23:04:37.007 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 1, 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 0 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-02-09T23:04:37.008 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 6, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 6, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 3460, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 3460, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:37.009 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 112044, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 27654701, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 112044, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:37.010 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 112044, 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 27654701, 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:37.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:37.012 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 10832, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 42643521, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 10832, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 42643521, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:37.013 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:37.014 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 10832, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 10832, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 10832, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 10832, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:37.015 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 3460, 2024-02-09T23:04:37.016 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 350468, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 3460, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 414, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 350468, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:37.017 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x5565ad110640": { 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 2396, 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:37.018 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 23201, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 41508995, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 23201, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 5413, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 41506599, 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:37.019 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:37.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:37.020 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-09T23:04:37.733 INFO:tasks.check_counter:Daemon mds.c mds.exported=2 2024-02-09T23:04:37.733 INFO:tasks.check_counter:Daemon mds.c mds.imported=4 2024-02-09T23:04:37.734 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-02-09T23:04:37.734 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:37.734 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:37.734 DEBUG:tasks.check_counter:Getting stats from f 2024-02-09T23:04:37.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.f perf dump 2024-02-09T23:04:37.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:37 smithi003 ceph-mon[32678]: pgmap v421: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-02-09T23:04:37.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:37 smithi064 ceph-mon[36638]: pgmap v421: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-02-09T23:04:37.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:37 smithi089 ceph-mon[36665]: pgmap v421: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 1.0 MiB/s wr, 60 op/s 2024-02-09T23:04:37.978 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T23:04:39.742 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:39 smithi089 ceph-mon[36665]: pgmap v422: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 524 KiB/s wr, 34 op/s 2024-02-09T23:04:39.743 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-09T23:04:39.743 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 41313, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 55847, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 14754695, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 61345494, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 15, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 2.172083734, 2024-02-09T23:04:39.744 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.945465436, 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 1.561962069, 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.081409367, 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 55848, 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.015830715, 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000018189 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:39.745 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 83243, 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.010646219, 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000127 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1836096, 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 568688 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:39.746 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 51342, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 39012, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 17195504, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 14762402, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 19, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 6, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 2.370985597, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.939422478, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 1.797366925, 2024-02-09T23:04:39.747 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.043048472, 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 39010, 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.708919436, 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000018172 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:39.748 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 86163, 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.011310934, 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000131 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 5296, 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 132272 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 5106, 2024-02-09T23:04:39.749 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 6071, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1449340, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 20206938, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 52, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 8, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.407545401, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.173566288, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.120938758, 2024-02-09T23:04:39.750 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.095118205, 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 6067, 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.155557481, 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000025639 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2942, 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000433004, 2024-02-09T23:04:39.751 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000147 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1856, 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 2864 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-02-09T23:04:39.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 9183, 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 5.224145968, 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000568893 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.753 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5497, 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.259020333, 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000047120 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.754 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 26476, 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 26240, 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 26240, 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 158.366396202, 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.006035304 2024-02-09T23:04:39.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 0, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 236, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 1, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 11, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 0, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-02-09T23:04:39.756 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 197, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 33, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 148, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 16, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 149, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 133, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 20, 2024-02-09T23:04:39.757 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 20, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 8, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 48142, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 29945, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 234, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 103, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:39.758 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 757, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 5182, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 1, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 2, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 19, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 6, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 84, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:39.759 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 275, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 87, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 6407151, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 6, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 275, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:39.760 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 16, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 16, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 259, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 2, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 2588, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 12715, 2024-02-09T23:04:39.761 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 135, 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 7612, 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 2738, 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 1249, 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 25414 2024-02-09T23:04:39.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 0, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 23, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 13, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 123, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 59, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 0, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:39.763 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 2408, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 2408, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:39.764 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 3, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.765 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 20627, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 0, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 0, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 20627, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 0, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 26, 2024-02-09T23:04:39.766 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 0, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 0, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 26, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 3, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 0, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 4194304, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 38451434, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 0, 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-02-09T23:04:39.767 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4402, 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 8.895923537, 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002020882 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 0 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 164, 2024-02-09T23:04:39.768 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 2601, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 2437, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 32, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 48, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 16, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 187, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 4637, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 4450, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 20, 2024-02-09T23:04:39.769 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 2564, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 2544, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 59172, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 207228 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 41711, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 1, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 26476, 2024-02-09T23:04:39.770 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 220, 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 4340, 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2316, 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 4.802500746, 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002073618 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 272, 2024-02-09T23:04:39.771 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.189510298, 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000696729 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.772 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4470, 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 12.065169196, 2024-02-09T23:04:39.773 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002699142 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1, 2024-02-09T23:04:39.774 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000149049, 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000149049 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.775 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.776 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.777 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.778 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.779 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 15924, 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 6.364000802, 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000399648 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-02-09T23:04:39.780 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 849, 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 131.057971114, 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.154367457 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-02-09T23:04:39.782 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.783 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.784 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.785 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.786 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.787 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2408, 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3.887094997, 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001614242 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:39.788 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 1, 2024-02-09T23:04:39.789 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 3, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 2, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 1, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 5271, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 5271, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 815, 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:39.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:39.791 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:39.792 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:39.793 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:39.794 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 484674, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 272, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 176, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 2, 2024-02-09T23:04:39.795 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 167, 2024-02-09T23:04:39.796 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 652332, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 6716, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:39.797 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 8741, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 34825620, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 33, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 8741, 2024-02-09T23:04:39.798 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8741, 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 16.859618463, 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001928797 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8741, 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 9376 2024-02-09T23:04:39.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 8741, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 713, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 8028, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 0, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 10, 2024-02-09T23:04:39.800 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 437, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 4839, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 277, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 0, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 2428, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-02-09T23:04:39.801 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 275, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 20, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-02-09T23:04:39.802 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-02-09T23:04:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-02-09T23:04:39.804 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 106, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 20, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 12, 2024-02-09T23:04:39.805 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 12, 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 0, 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 858, 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 2, 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 230 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-02-09T23:04:39.806 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 1, 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 0 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.807 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 6, 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 6, 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 2408, 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 2408, 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:39.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 97757, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 24578192, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 97757, 2024-02-09T23:04:39.809 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 97757, 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 24578192, 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:39.811 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:39.812 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:39.812 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 8741, 2024-02-09T23:04:39.812 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 35069564, 2024-02-09T23:04:39.812 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 8741, 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 35069564, 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:39.813 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 8741, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 8741, 2024-02-09T23:04:39.814 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 8741, 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 8741, 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.815 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 2408, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 243960, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 2408, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:39.816 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 437, 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 243960, 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:39.817 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x563056e40640": { 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 4093, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 20627, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 34257130, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 20627, 2024-02-09T23:04:39.818 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 4402, 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 34253037, 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:39.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:39.820 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-09T23:04:39.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:39 smithi003 ceph-mon[32678]: pgmap v422: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 524 KiB/s wr, 34 op/s 2024-02-09T23:04:39.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:39 smithi064 ceph-mon[36638]: pgmap v422: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 524 KiB/s wr, 34 op/s 2024-02-09T23:04:40.566 INFO:tasks.check_counter:Daemon mds.f mds.exported=2 2024-02-09T23:04:40.566 INFO:tasks.check_counter:Daemon mds.f mds.imported=6 2024-02-09T23:04:40.566 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=6 2024-02-09T23:04:40.566 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:40.567 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:40.567 DEBUG:tasks.check_counter:Getting stats from i 2024-02-09T23:04:40.567 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.i perf dump 2024-02-09T23:04:40.805 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T23:04:41.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:41 smithi003 ceph-mon[32678]: pgmap v423: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 532 KiB/s wr, 42 op/s 2024-02-09T23:04:41.887 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:41 smithi089 ceph-mon[36665]: pgmap v423: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 532 KiB/s wr, 42 op/s 2024-02-09T23:04:41.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:41 smithi064 ceph-mon[36638]: pgmap v423: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 532 KiB/s wr, 42 op/s 2024-02-09T23:04:42.505 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-09T23:04:42.505 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 45782, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 65814, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 14028320, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 70119157, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 60, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 8, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 2.484966004, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 1.076762212, 2024-02-09T23:04:42.506 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 1.710191468, 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.125393627, 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 65812, 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.147471369, 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000017435 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:42.507 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 93635, 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.012686952, 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000135 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:42.508 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 6613, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 6827, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 1709109, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 24733717, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 15, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 7, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.476961322, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.191447513, 2024-02-09T23:04:42.509 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.103118567, 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.148246864, 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 6822, 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.183320248, 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000026871 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.510 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 42, 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000008782, 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000209 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 2208, 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 511264 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:42.511 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 66822, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 55753, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 23212996, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 19947273, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 10, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 6, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 3.079337113, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 1.233152422, 2024-02-09T23:04:42.512 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 2.225890066, 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.091918027, 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 55755, 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.916762152, 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000016442 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:42.513 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 113669, 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.014370274, 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000126 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1849072, 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 192416 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-02-09T23:04:42.514 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 17713, 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 8.148300464, 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000460018 2024-02-09T23:04:42.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8098, 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.427003817, 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000052729 2024-02-09T23:04:42.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 25223, 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 24294, 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 24294, 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 23.794596956, 2024-02-09T23:04:42.517 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000979443 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 0, 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 929, 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 11, 2024-02-09T23:04:42.518 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 0, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 198, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 17, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 157, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 24, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 174, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 41, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 24, 2024-02-09T23:04:42.519 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 24, 2024-02-09T23:04:42.520 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 10, 2024-02-09T23:04:42.520 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 53582, 2024-02-09T23:04:42.520 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 30251, 2024-02-09T23:04:42.520 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 929, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 6, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 1085, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 2594, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 1, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 1, 2024-02-09T23:04:42.521 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 40, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 4, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 51, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 313, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 87, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 6407151, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 6, 2024-02-09T23:04:42.522 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 313, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 313, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 2, 2024-02-09T23:04:42.523 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 2401, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 17331, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 110, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 11497, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 5630, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 1116, 2024-02-09T23:04:42.524 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 24851 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 0, 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 21, 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 12, 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 23, 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 122, 2024-02-09T23:04:42.525 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 0, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 3731, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 3731, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-02-09T23:04:42.527 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 1, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-02-09T23:04:42.528 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 26766, 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 0, 2024-02-09T23:04:42.529 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 0, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 26766, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 0, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 35, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 0, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 0, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 35, 2024-02-09T23:04:42.530 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 4, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 0, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 4194304, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 51020236, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 0, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 8717, 2024-02-09T23:04:42.531 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 16.621995882, 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001906848 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 0 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 177, 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 3944, 2024-02-09T23:04:42.532 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 3767, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 30, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 43, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 13, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 188, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 4089, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 3901, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 15, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 4011, 2024-02-09T23:04:42.533 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 3996, 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 59952, 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 207228 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 47320, 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 2, 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 25223, 2024-02-09T23:04:42.534 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 1420, 2024-02-09T23:04:42.535 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 5584, 2024-02-09T23:04:42.535 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-02-09T23:04:42.535 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3732, 2024-02-09T23:04:42.535 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 6.948299231, 2024-02-09T23:04:42.535 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001861816 2024-02-09T23:04:42.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 110, 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.072053441, 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000655031 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.536 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.537 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 4834, 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3.471104395, 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000718060 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.538 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.539 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.540 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:42.541 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.541 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.541 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.541 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-02-09T23:04:42.541 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.542 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.542 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.543 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.544 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1, 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000526442, 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000526442 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 10227, 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 4.709047049, 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000460452 2024-02-09T23:04:42.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1144, 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 2.398173750, 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002096305 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.546 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.547 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-02-09T23:04:42.549 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 7, 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.018359767, 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.002622823 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 508, 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.250013988, 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000492153 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.551 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.552 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3731, 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 5.927018893, 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001588587 2024-02-09T23:04:42.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 2, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 2, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 0, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 2, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 4696, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 2348, 2024-02-09T23:04:42.555 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 751, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:42.556 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:42.557 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:42.558 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:42.559 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 454246, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 270, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 88, 2024-02-09T23:04:42.560 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 1, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 167, 2024-02-09T23:04:42.561 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 683995, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 6166, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:42.562 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 14784, 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 47671011, 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 28, 2024-02-09T23:04:42.563 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 14784, 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 14784, 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 28.053365182, 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.001897549 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-02-09T23:04:42.564 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 14784, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 15565 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 14784, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 873, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 13911, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-02-09T23:04:42.565 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 0, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 10, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 560, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 9277, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 360, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-02-09T23:04:42.566 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 0, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 3754, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 313, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 20, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:42.567 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:42.568 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-02-09T23:04:42.569 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-02-09T23:04:42.570 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 106, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 20, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 12, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 12, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 0, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 916, 2024-02-09T23:04:42.571 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 0, 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 355 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 1, 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.572 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 0 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 6, 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 6, 2024-02-09T23:04:42.573 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 3731, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 3731, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 119213, 2024-02-09T23:04:42.574 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 28198297, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 119213, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 119213, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 28198297, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.575 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:42.576 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 14784, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 48048754, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 14784, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 48048754, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.577 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:42.578 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 14784, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 14784, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 14784, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 14784, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.579 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:42.580 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 3731, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 377743, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 3731, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 560, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 377743, 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:42.581 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x55fcb1434640": { 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:42.582 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 26766, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 46825932, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 26766, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 8717, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 46825932, 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:42.583 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:42.584 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:42.584 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:42.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:42.584 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-09T23:04:43.426 INFO:tasks.check_counter:Daemon mds.i mds.exported=1 2024-02-09T23:04:43.426 INFO:tasks.check_counter:Daemon mds.i mds.imported=4 2024-02-09T23:04:43.427 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-02-09T23:04:43.427 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:43.427 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:43.427 DEBUG:tasks.check_counter:Getting stats from l 2024-02-09T23:04:43.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 shell --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 -- ceph daemon mds.l perf dump 2024-02-09T23:04:43.669 INFO:teuthology.orchestra.run.smithi089.stderr:Inferring config /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/mon.c/config 2024-02-09T23:04:43.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:43 smithi089 ceph-mon[36665]: pgmap v424: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 252 KiB/s wr, 22 op/s 2024-02-09T23:04:43.881 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:43 smithi003 ceph-mon[32678]: pgmap v424: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 252 KiB/s wr, 22 op/s 2024-02-09T23:04:43.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:43 smithi064 ceph-mon[36638]: pgmap v424: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 252 KiB/s wr, 22 op/s 2024-02-09T23:04:45.531 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:45 smithi089 ceph-mon[36665]: pgmap v425: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 252 KiB/s wr, 22 op/s 2024-02-09T23:04:45.532 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-02-09T23:04:45.532 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-0": { 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 505, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 461, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 6057868, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 215574, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 7, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 2, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.069215760, 2024-02-09T23:04:45.533 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.020152876, 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.047396181, 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.010332886, 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 460, 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.017384917, 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000037793 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:45.534 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 1818576, 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 160832 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-1": { 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 871, 2024-02-09T23:04:45.535 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 1067, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 21192860, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 775180, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 8, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 6, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.090382742, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.029698486, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.029383853, 2024-02-09T23:04:45.536 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.023134381, 2024-02-09T23:04:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 1065, 2024-02-09T23:04:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.042786237, 2024-02-09T23:04:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000040174 2024-02-09T23:04:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.537 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 506160 2024-02-09T23:04:45.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "AsyncMessenger::Worker-2": { 2024-02-09T23:04:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_messages": 2008, 2024-02-09T23:04:45.539 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages": 2010, 2024-02-09T23:04:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_bytes": 22272221, 2024-02-09T23:04:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_bytes": 624546, 2024-02-09T23:04:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_created_connections": 6, 2024-02-09T23:04:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_active_connections": 5, 2024-02-09T23:04:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_total_time": 0.190797188, 2024-02-09T23:04:45.540 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_send_time": 0.051484648, 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_recv_time": 0.063083928, 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_running_fast_dispatch_time": 0.060432670, 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_messages_queue_lat": { 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 2008, 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.071241602, 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000035478 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.541 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_handle_ack_lat": { 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_recv_encrypted_bytes": 0, 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "msgr_send_encrypted_bytes": 0 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.542 INFO:teuthology.orchestra.run.smithi089.stdout: "cct": { 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "total_workers": 1, 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "unhealthy_workers": 0 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-MDSRank": { 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 5271, 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.133153688, 2024-02-09T23:04:45.543 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000025261 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "finisher-PurgeQueue": { 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "queue_len": 0, 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "complete_latency": { 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.544 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "mds": { 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "request": 0, 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "reply": 0, 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "reply_latency": { 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.545 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_reply": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "forward": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_complete": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_fetch_keys": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_commit": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_split": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_merge": 0, 2024-02-09T23:04:45.546 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes": 3915, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_top": 15, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_bottom": 3900, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pin_tail": 0, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_pinned": 3894, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_expired": 0, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "inodes_with_caps": 0, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "caps": 0, 2024-02-09T23:04:45.547 INFO:teuthology.orchestra.run.smithi089.stdout: "subtrees": 5, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_hit": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_forward": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_discover": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_dir_fetch": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_remote_ino": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "traverse_lock": 0, 2024-02-09T23:04:45.548 INFO:teuthology.orchestra.run.smithi089.stdout: "load_cent": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "q": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "exported": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "exported_inodes": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "imported": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "imported_inodes": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_dir_fetch": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_backtrace_fetch": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "openino_peer_discover": 0, 2024-02-09T23:04:45.549 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rfiles": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rbytes": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "root_rsnaps": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_fetch": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_set_tag": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_backtrace_repaired": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_inotable_repaired": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_inodes": 0, 2024-02-09T23:04:45.550 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dir_base_inodes": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_dirfrag_rstats": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "scrub_file_inodes": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_inode_file_caps": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_revoke": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_grant": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_trunc": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "ceph_cap_op_flush_ack": 0, 2024-02-09T23:04:45.551 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps": 0, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_caps_dirty": 0, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_cap_release": 0, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "process_request_cap_release": 0 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache": { 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update": 0, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_update_receipt": 0, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_try_discover": 0, 2024-02-09T23:04:45.552 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_send_discover": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "dir_handle_discover": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_delayed": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "num_strays_enqueuing": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_created": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_enqueued": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_reintegrated": 0, 2024-02-09T23:04:45.553 INFO:teuthology.orchestra.run.smithi089.stdout: "strays_migrated": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_processing": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_enqueued": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "num_recovering_prioritized": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_started": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_completed": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_enqueue_scrub": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_exportdir": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_flush": 0, 2024-02-09T23:04:45.554 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragmentdir": 0, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_fragstats": 0, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "ireq_inodestats": 0 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log": { 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "evlrg": 0, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "evadd": 0, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "evex": 0, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "evtrm": 0, 2024-02-09T23:04:45.555 INFO:teuthology.orchestra.run.smithi089.stdout: "ev": 26766, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "evexg": 0, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "evexd": 0, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "segadd": 0, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "segex": 0, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "segtrm": 0, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "seg": 35, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "segmjr": 4, 2024-02-09T23:04:45.556 INFO:teuthology.orchestra.run.smithi089.stdout: "segexg": 0, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "segexd": 0, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "expos": 4194304, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "wrpos": 51020236, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "rdpos": 51019662, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "jlat": { 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "replayed": 26766 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mem": { 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "ino": 171, 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "ino+": 3942, 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "ino-": 3771, 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "dir": 24, 2024-02-09T23:04:45.558 INFO:teuthology.orchestra.run.smithi089.stdout: "dir+": 25, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "dir-": 1, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "dn": 3915, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "dn+": 3955, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "dn-": 40, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "cap": 0, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "cap+": 0, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "cap-": 0, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "rss": 53388, 2024-02-09T23:04:45.559 INFO:teuthology.orchestra.run.smithi089.stdout: "heap": 215420 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_server": { 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_client_request": 0, 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "dispatch_server_request": 0, 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_request": 0, 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_client_session": 0, 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "handle_peer_request": 0, 2024-02-09T23:04:45.560 INFO:teuthology.orchestra.run.smithi089.stdout: "req_create_latency": { 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getattr_latency": { 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.561 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getfilelock_latency": { 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "req_link_latency": { 2024-02-09T23:04:45.562 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookup_latency": { 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookuphash_latency": { 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupino_latency": { 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.564 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupname_latency": { 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupparent_latency": { 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lookupsnap_latency": { 2024-02-09T23:04:45.566 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "req_lssnap_latency": { 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.567 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mkdir_latency": { 2024-02-09T23:04:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.568 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mknod_latency": { 2024-02-09T23:04:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.569 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "req_mksnap_latency": { 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "req_open_latency": { 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "req_readdir_latency": { 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.571 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rename_latency": { 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "req_renamesnap_latency": { 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "req_snapdiff_latency": { 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.573 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmdir_latency": { 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmsnap_latency": { 2024-02-09T23:04:45.574 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "req_rmxattr_latency": { 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setattr_latency": { 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setdirlayout_latency": { 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.576 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setfilelock_latency": { 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.577 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setlayout_latency": { 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "req_setxattr_latency": { 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "req_symlink_latency": { 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "req_unlink_latency": { 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.579 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_revoke_eviction": 0, 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "cap_acquisition_throttle": 0, 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "req_getvxattr_latency": { 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.580 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessions": { 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "session_count": 2, 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "session_add": 2, 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "session_remove": 0, 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_open": 2, 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "sessions_stale": 0, 2024-02-09T23:04:45.581 INFO:teuthology.orchestra.run.smithi089.stdout: "total_load": 0, 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "average_load": 0, 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "avg_session_uptime": 2498, 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "mdthresh_evicted": 0 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool": { 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_bytes": 0, 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "bloom_filter_items": 0, 2024-02-09T23:04:45.582 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_bytes": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_items": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_bytes": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_data_items": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_bytes": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_onode_items": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_bytes": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_items": 0, 2024-02-09T23:04:45.583 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_bytes": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_other_items": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_bytes": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_buffer_items": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_bytes": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_items": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_bytes": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blob_items": 0, 2024-02-09T23:04:45.584 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_bytes": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_shared_blob_items": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_bytes": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_inline_bl_items": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_bytes": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_items": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_bytes": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_txc_items": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_bytes": 0, 2024-02-09T23:04:45.585 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_deferred_items": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_bytes": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_writing_items": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_bytes": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_items": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_bytes": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_reader_items": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_bytes": 0, 2024-02-09T23:04:45.586 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_file_writer_items": 0, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_bytes": 231615, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_anon_items": 231, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_bytes": 0, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "buffer_meta_items": 0, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bytes": 0, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_items": 0, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_bytes": 0, 2024-02-09T23:04:45.587 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mapbl_items": 0, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_bytes": 0, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pglog_items": 0, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_bytes": 8032, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_items": 167, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_bytes": 0, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "osdmap_mapping_items": 0, 2024-02-09T23:04:45.588 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_bytes": 0, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "pgmap_items": 0, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_bytes": 3234096, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_co_items": 19462, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_bytes": 0, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_1_items": 0, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_bytes": 0, 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: "unittest_2_items": 0 2024-02-09T23:04:45.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter": { 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_active": 0, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_laggy": 0, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send": 3049, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_send_bytes": 21, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_resend": 3, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_reply": 3049, 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "op_latency": { 2024-02-09T23:04:45.590 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3049, 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 1.819958836, 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000596903 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "op_inflight": 0, 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "oplen_avg": { 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 3049, 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 3050 2024-02-09T23:04:45.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "op": 3049, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "op_r": 3049, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "op_w": 0, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "op_rmw": 0, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "op_pg": 0, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_stat": 1690, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_create": 0, 2024-02-09T23:04:45.592 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_read": 1357, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_write": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writefull": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_writesame": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_append": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_zero": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_truncate": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_delete": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_mapext": 0, 2024-02-09T23:04:45.593 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_sparse_read": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_clonerange": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_getxattr": 1, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_setxattr": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_cmpxattr": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_rmxattr": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_resetxattrs": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_call": 0, 2024-02-09T23:04:45.594 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_watch": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_notify": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_src_cmpxattr": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_pgls_filter": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "osdop_other": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_active": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_send": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_resend": 0, 2024-02-09T23:04:45.595 INFO:teuthology.orchestra.run.smithi089.stdout: "linger_ping": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_active": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_send": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "poolop_resend": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_active": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_send": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "poolstat_resend": 0, 2024-02-09T23:04:45.596 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_active": 0, 2024-02-09T23:04:45.597 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_send": 0, 2024-02-09T23:04:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "statfs_resend": 0, 2024-02-09T23:04:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "command_active": 0, 2024-02-09T23:04:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "command_send": 0, 2024-02-09T23:04:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "command_resend": 0, 2024-02-09T23:04:45.599 INFO:teuthology.orchestra.run.smithi089.stdout: "map_epoch": 106, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "map_full": 0, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "map_inc": 20, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_sessions": 9, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_open": 10, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_session_close": 1, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_laggy": 0, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_wr": 0, 2024-02-09T23:04:45.600 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_rd": 2, 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_del": 0 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "oft": { 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_objs": 0, 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_kv_pairs": 0, 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_updates": 0, 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: "omap_total_removes": 0 2024-02-09T23:04:45.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "osd-slow-ops": { 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "slow_ops_count": 0 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "purge_queue": { 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops": 0, 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_ops_high_water": 0, 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing": 0, 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executing_high_water": 0, 2024-02-09T23:04:45.602 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed_ops": 0, 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_executed": 0, 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "pq_item_in_journal": 0 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:45.603 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 3380, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 49199163, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 3380, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 3380, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 49199163, 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:45.604 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_bytes": { 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 104857600, 2024-02-09T23:04:45.605 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 3049, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 46850629, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 3049, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 46850629, 2024-02-09T23:04:45.606 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-objecter_ops": { 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:45.607 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 1024, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 3049, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 3049, 2024-02-09T23:04:45.608 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 3049, 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 3049, 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.609 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle": { 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:45.610 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 0, 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 0, 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "throttle-write_buf_throttle-0x55fce77e6960": { 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "val": 0, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "max": 3758096384, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "get_started": 0, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "get": 0, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "get_sum": 0, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_fail": 0, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "get_or_fail_success": 0, 2024-02-09T23:04:45.612 INFO:teuthology.orchestra.run.smithi089.stdout: "take": 0, 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "take_sum": 0, 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "put": 0, 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "put_sum": 0, 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "wait": { 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "avgcount": 0, 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "sum": 0.000000000, 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: "avgtime": 0.000000000 2024-02-09T23:04:45.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-02-09T23:04:45.614 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-02-09T23:04:45.631 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:45 smithi003 ceph-mon[32678]: pgmap v425: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 252 KiB/s wr, 22 op/s 2024-02-09T23:04:45.685 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:45 smithi064 ceph-mon[36638]: pgmap v425: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 53 KiB/s rd, 252 KiB/s wr, 22 op/s 2024-02-09T23:04:46.308 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-02-09T23:04:46.308 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-02-09T23:04:46.308 INFO:tasks.check_counter:Daemon mds.l mds.root_rsnaps=0 2024-02-09T23:04:46.308 INFO:tasks.check_counter:Daemon mds.l mds_server.req_mksnap_latency.avgcount=0 2024-02-09T23:04:46.308 INFO:tasks.check_counter:Daemon mds.l mds_server.req_rmsnap_latency.avgcount=0 2024-02-09T23:04:46.309 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-02-09T23:04:46.320 INFO:tasks.kclient:Unmounting kernel clients... 2024-02-09T23:04:46.320 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T23:04:46.320 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T23:04:46.340 DEBUG:teuthology.orchestra.run.smithi003:> set -ex 2024-02-09T23:04:46.340 DEBUG:teuthology.orchestra.run.smithi003:> dd if=/proc/self/mounts of=/dev/stdout 2024-02-09T23:04:46.401 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T23:04:46.560 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 -- 172.21.15.3:0/3792268202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100d00 msgr2=0x7f53bc104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 --2- 172.21.15.3:0/3792268202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100d00 0x7f53bc104d40 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7f53a8009930 tx=0x7f53a802bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 -- 172.21.15.3:0/3792268202 shutdown_connections 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 --2- 172.21.15.3:0/3792268202 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53bc105420 0x7f53bc10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 --2- 172.21.15.3:0/3792268202 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100d00 0x7f53bc104d40 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 --2- 172.21.15.3:0/3792268202 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53bc100350 0x7f53bc100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 -- 172.21.15.3:0/3792268202 >> 172.21.15.3:0/3792268202 conn(0x7f53bc0772b0 msgr2=0x7f53bc0776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:46.561 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 -- 172.21.15.3:0/3792268202 shutdown_connections 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.555+0000 7f53c3264640 1 -- 172.21.15.3:0/3792268202 wait complete. 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 Processor -- start 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 -- start start 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 0x7f53bc101de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53bc105420 0x7f53bc102320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53bc102860 0x7f53bc1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1a6030 con 0x7f53bc100350 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1a61a0 con 0x7f53bc105420 2024-02-09T23:04:46.562 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c3264640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53bc1a6360 con 0x7f53bc102860 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 0x7f53bc101de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 0x7f53bc101de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45110/0 (socket says 172.21.15.3:45110) 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 -- 172.21.15.3:0/2862492828 learned_addr learned my addr 172.21.15.3:0/2862492828 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53bbfff640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53bc105420 0x7f53bc102320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1801640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53bc102860 0x7f53bc1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 -- 172.21.15.3:0/2862492828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53bc102860 msgr2=0x7f53bc1a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53bc102860 0x7f53bc1a5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.563 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 -- 172.21.15.3:0/2862492828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53bc105420 msgr2=0x7f53bc102320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53bc105420 0x7f53bc102320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.556+0000 7f53c1000640 1 -- 172.21.15.3:0/2862492828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a8009560 con 0x7f53bc100350 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.557+0000 7f53c1000640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 0x7f53bc101de0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f53b000ea80 tx=0x7f53b000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.557+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b000ce40 con 0x7f53bc100350 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.557+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f53b0005020 con 0x7f53bc100350 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.557+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b00116d0 con 0x7f53bc100350 2024-02-09T23:04:46.564 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.557+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53bc1a6640 con 0x7f53bc100350 2024-02-09T23:04:46.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.557+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53bc1a6ad0 con 0x7f53bc100350 2024-02-09T23:04:46.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.558+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53bc0680b0 con 0x7f53bc100350 2024-02-09T23:04:46.565 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.559+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f53b0005190 con 0x7f53bc100350 2024-02-09T23:04:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.559+0000 7f53b9ffb640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53ac0a2260 0x7f53ac0a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.560+0000 7f53bbfff640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53ac0a2260 0x7f53ac0a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.560+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f53b00ce390 con 0x7f53bc100350 2024-02-09T23:04:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.560+0000 7f53bbfff640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53ac0a2260 0x7f53ac0a4720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f53a8000c00 tx=0x7f53a8005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:46.566 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.564+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f53b009aaf0 con 0x7f53bc100350 2024-02-09T23:04:46.750 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.747+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53bc103310 con 0x7f53bc100350 2024-02-09T23:04:46.751 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.749+0000 7f53b9ffb640 1 -- 172.21.15.3:0/2862492828 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+22923 (secure 0 0 0) 0x7f53b001f090 con 0x7f53bc100350 2024-02-09T23:04:46.752 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:46.752 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":106,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T23:04:09.305034+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/284621866":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6800/284621866":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/3470258541":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2951038011":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2619736972":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2265484823":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T23:04:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53ac0a2260 msgr2=0x7f53ac0a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53ac0a2260 0x7f53ac0a4720 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f53a8000c00 tx=0x7f53a8005fd0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 msgr2=0x7f53bc101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 0x7f53bc101de0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7f53b000ea80 tx=0x7f53b000ef40 comp rx=0 tx=0).stop 2024-02-09T23:04:46.754 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 shutdown_connections 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f53ac0a2260 0x7f53ac0a4720 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f53bc105420 0x7f53bc102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f53bc102860 0x7f53bc1a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 --2- 172.21.15.3:0/2862492828 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f53bc100350 0x7f53bc101de0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 >> 172.21.15.3:0/2862492828 conn(0x7f53bc0772b0 msgr2=0x7f53bc068860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 shutdown_connections 2024-02-09T23:04:46.755 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.751+0000 7f53c3264640 1 -- 172.21.15.3:0/2862492828 wait complete. 2024-02-09T23:04:46.765 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-02-09T22:44:32.790973+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-02-09T22:49:04.801615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-02-09T22:49:05.816157+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '105', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 21, 'snap_epoch': 105, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-02-09T23:04:46.766 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-02-09T23:04:46.874 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 -- 172.21.15.3:0/1477598619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0514106200 msgr2=0x7f05141065e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 --2- 172.21.15.3:0/1477598619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0514106200 0x7f05141065e0 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f0504009930 tx=0x7f050402bc20 comp rx=0 tx=0).stop 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 -- 172.21.15.3:0/1477598619 shutdown_connections 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 --2- 172.21.15.3:0/1477598619 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f05140ff700 0x7f05140ffb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 --2- 172.21.15.3:0/1477598619 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05141000c0 0x7f051410e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 --2- 172.21.15.3:0/1477598619 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f0514106200 0x7f05141065e0 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 -- 172.21.15.3:0/1477598619 >> 172.21.15.3:0/1477598619 conn(0x7f0514076630 msgr2=0x7f05140fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 -- 172.21.15.3:0/1477598619 shutdown_connections 2024-02-09T23:04:46.875 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 -- 172.21.15.3:0/1477598619 wait complete. 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.869+0000 7f051a015640 1 Processor -- start 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 -- start start 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 0x7f051419d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05141000c0 0x7f051419d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0514106200 0x7f0514197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0514197c50 con 0x7f05140ff700 2024-02-09T23:04:46.876 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0514197dc0 con 0x7f0514106200 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f051a015640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0514197f80 con 0x7f05141000c0 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 0x7f051419d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 0x7f051419d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45134/0 (socket says 172.21.15.3:45134) 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 -- 172.21.15.3:0/1977819565 learned_addr learned my addr 172.21.15.3:0/1977819565 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f0512ffd640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05141000c0 0x7f051419d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f0513fff640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0514106200 0x7f0514197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 -- 172.21.15.3:0/1977819565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05141000c0 msgr2=0x7f051419d990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.877 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05141000c0 0x7f051419d990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 -- 172.21.15.3:0/1977819565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0514106200 msgr2=0x7f0514197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0514106200 0x7f0514197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 -- 172.21.15.3:0/1977819565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0504009560 con 0x7f05140ff700 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f0513fff640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0514106200 0x7f0514197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f05137fe640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 0x7f051419d450 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f0504005940 tx=0x7f05040042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.870+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05040044e0 con 0x7f05140ff700 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.871+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0504004aa0 con 0x7f05140ff700 2024-02-09T23:04:46.878 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.871+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0514198200 con 0x7f05140ff700 2024-02-09T23:04:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.871+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504003e50 con 0x7f05140ff700 2024-02-09T23:04:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.871+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05141a44f0 con 0x7f05140ff700 2024-02-09T23:04:46.879 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.872+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0514068500 con 0x7f05140ff700 2024-02-09T23:04:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.872+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f05040035f0 con 0x7f05140ff700 2024-02-09T23:04:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.873+0000 7f0510ff9640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04fc0a2230 0x7f04fc0a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.873+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f05040f02c0 con 0x7f05140ff700 2024-02-09T23:04:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.873+0000 7f0512ffd640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04fc0a2230 0x7f04fc0a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:46.880 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.874+0000 7f0512ffd640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04fc0a2230 0x7f04fc0a46f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0508005e10 tx=0x7f0508005da0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:46.881 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:46.878+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f05040b9060 con 0x7f05140ff700 2024-02-09T23:04:47.061 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.059+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05141113e0 con 0x7f05140ff700 2024-02-09T23:04:47.062 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.059+0000 7f0510ff9640 1 -- 172.21.15.3:0/1977819565 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v106) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05040bc720 con 0x7f05140ff700 2024-02-09T23:04:47.062 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 1 2024-02-09T23:04:47.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04fc0a2230 msgr2=0x7f04fc0a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04fc0a2230 0x7f04fc0a46f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f0508005e10 tx=0x7f0508005da0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.064 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 msgr2=0x7f051419d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 0x7f051419d450 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f0504005940 tx=0x7f05040042f0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 shutdown_connections 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f04fc0a2230 0x7f04fc0a46f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f0514106200 0x7f0514197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f05141000c0 0x7f051419d990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 --2- 172.21.15.3:0/1977819565 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f05140ff700 0x7f051419d450 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 >> 172.21.15.3:0/1977819565 conn(0x7f0514076630 msgr2=0x7f0514104060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:47.065 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 shutdown_connections 2024-02-09T23:04:47.066 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.061+0000 7f051a015640 1 -- 172.21.15.3:0/1977819565 wait complete. 2024-02-09T23:04:47.075 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-02-09T23:04:47.185 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 -- 172.21.15.3:0/1116281977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c100330 msgr2=0x7f958c100710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 --2- 172.21.15.3:0/1116281977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c100330 0x7f958c100710 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f9588009930 tx=0x7f958802bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:47.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 -- 172.21.15.3:0/1116281977 shutdown_connections 2024-02-09T23:04:47.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 --2- 172.21.15.3:0/1116281977 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c105400 0x7f958c10cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 --2- 172.21.15.3:0/1116281977 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c100ce0 0x7f958c104d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.186 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 --2- 172.21.15.3:0/1116281977 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c100330 0x7f958c100710 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 -- 172.21.15.3:0/1116281977 >> 172.21.15.3:0/1116281977 conn(0x7f958c077250 msgr2=0x7f958c077660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 -- 172.21.15.3:0/1116281977 shutdown_connections 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.179+0000 7f95933c9640 1 -- 172.21.15.3:0/1116281977 wait complete. 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 Processor -- start 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 -- start start 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c100330 0x7f958c19f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c105400 0x7f958c19fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 0x7f958c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.187 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19a290 con 0x7f958c1998a0 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19a400 con 0x7f958c100330 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f95933c9640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f958c19a5c0 con 0x7f958c105400 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f9591966640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 0x7f958c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f9591966640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 0x7f958c199d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45152/0 (socket says 172.21.15.3:45152) 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f9591966640 1 -- 172.21.15.3:0/816866339 learned_addr learned my addr 172.21.15.3:0/816866339 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f9590964640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c105400 0x7f958c19fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.188 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.180+0000 7f9591165640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c100330 0x7f958c19f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591966640 1 -- 172.21.15.3:0/816866339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c105400 msgr2=0x7f958c19fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591966640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c105400 0x7f958c19fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591966640 1 -- 172.21.15.3:0/816866339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c100330 msgr2=0x7f958c19f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591966640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c100330 0x7f958c19f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591966640 1 -- 172.21.15.3:0/816866339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9588009560 con 0x7f958c1998a0 2024-02-09T23:04:47.189 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591165640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c100330 0x7f958c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f9591966640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 0x7f958c199d50 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f958400eb10 tx=0x7f958400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584004b20 con 0x7f958c1998a0 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f958c1a65b0 con 0x7f958c1998a0 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9584004c80 con 0x7f958c1998a0 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9584010650 con 0x7f958c1998a0 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.181+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f958c1a6ac0 con 0x7f958c1998a0 2024-02-09T23:04:47.190 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.182+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f958c068500 con 0x7f958c1998a0 2024-02-09T23:04:47.191 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.183+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f95840107f0 con 0x7f958c1998a0 2024-02-09T23:04:47.192 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.184+0000 7f95827fc640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95780a2230 0x7f95780a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.184+0000 7f9591165640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95780a2230 0x7f95780a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.184+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f9584014070 con 0x7f958c1998a0 2024-02-09T23:04:47.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.184+0000 7f9591165640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95780a2230 0x7f95780a46f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9588036040 tx=0x7f9588000bc0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:47.193 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.188+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f95840974f0 con 0x7f958c1998a0 2024-02-09T23:04:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.368+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f958c100ce0 con 0x7f958c1998a0 2024-02-09T23:04:47.371 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.368+0000 7f95827fc640 1 -- 172.21.15.3:0/816866339 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v106) v1 ==== 104+0+11 (secure 0 0 0) 0x7f958409abb0 con 0x7f958c1998a0 2024-02-09T23:04:47.371 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 32 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.370+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95780a2230 msgr2=0x7f95780a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95780a2230 0x7f95780a46f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9588036040 tx=0x7f9588000bc0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 msgr2=0x7f958c199d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 0x7f958c199d50 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f958400eb10 tx=0x7f958400c2d0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 shutdown_connections 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95780a2230 0x7f95780a46f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f958c1998a0 0x7f958c199d50 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.374 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f958c105400 0x7f958c19fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 --2- 172.21.15.3:0/816866339 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f958c100330 0x7f958c19f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 >> 172.21.15.3:0/816866339 conn(0x7f958c077250 msgr2=0x7f958c0ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 shutdown_connections 2024-02-09T23:04:47.375 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.371+0000 7f95933c9640 1 -- 172.21.15.3:0/816866339 wait complete. 2024-02-09T23:04:47.385 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-02-09T23:04:47.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3239540341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 msgr2=0x7fa1f00ffb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3239540341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f00ffb20 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4009930 tx=0x7fa1e402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:47.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3239540341 shutdown_connections 2024-02-09T23:04:47.491 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3239540341 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f0100060 0x7fa1f010e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3239540341 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f00ffb20 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3239540341 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f01061e0 0x7fa1f01065c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3239540341 >> 172.21.15.3:0/3239540341 conn(0x7fa1f0076620 msgr2=0x7fa1f00fd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:47.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3239540341 shutdown_connections 2024-02-09T23:04:47.492 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.488+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3239540341 wait complete. 2024-02-09T23:04:47.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 Processor -- start 2024-02-09T23:04:47.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 -- start start 2024-02-09T23:04:47.494 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f019d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f0100060 0x7fa1f019dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f01061e0 0x7fa1f01976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0197e40 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.495 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0197fb0 con 0x7fa1f01061e0 2024-02-09T23:04:47.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1f4aeb640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1f0198170 con 0x7fa1f0100060 2024-02-09T23:04:47.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f019d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f019d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45176/0 (socket says 172.21.15.3:45176) 2024-02-09T23:04:47.496 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 -- 172.21.15.3:0/3846214031 learned_addr learned my addr 172.21.15.3:0/3846214031 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:47.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1eed9d640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f01061e0 0x7fa1f01976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 -- 172.21.15.3:0/3846214031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f0100060 msgr2=0x7fa1f019dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f0100060 0x7fa1f019dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.497 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 -- 172.21.15.3:0/3846214031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f01061e0 msgr2=0x7fa1f01976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f01061e0 0x7fa1f01976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.489+0000 7fa1ee59c640 1 -- 172.21.15.3:0/3846214031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e4009560 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.490+0000 7fa1ee59c640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f019d5a0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000d770 tx=0x7fa1e000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.490+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0009c30 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.490+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1e000de80 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.490+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1f0198450 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.490+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1e0004e80 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.490+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1f01a4520 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.498 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.491+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1f01029c0 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.492+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fa1e0005000 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.499 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.492+0000 7fa1df7fe640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d80a22b0 0x7fa1d80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.493+0000 7fa1edd9b640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d80a22b0 0x7fa1d80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.493+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fa1e00cdfb0 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.493+0000 7fa1edd9b640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d80a22b0 0x7fa1d80a4770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4036040 tx=0x7fa1e4000bc0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:47.500 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.496+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fa1e0096c60 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.680 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.677+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1f01096e0 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.681 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.678+0000 7fa1df7fe640 1 -- 172.21.15.3:0/3846214031 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v106) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1e0017070 con 0x7fa1f00ff6a0 2024-02-09T23:04:47.681 INFO:teuthology.orchestra.run.smithi003.stdout:pg_num: 64 2024-02-09T23:04:47.682 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.679+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d80a22b0 msgr2=0x7fa1d80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.679+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d80a22b0 0x7fa1d80a4770 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fa1e4036040 tx=0x7fa1e4000bc0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 msgr2=0x7fa1f019d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f019d5a0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fa1e000d770 tx=0x7fa1e000dc30 comp rx=0 tx=0).stop 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 shutdown_connections 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fa1d80a22b0 0x7fa1d80a4770 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fa1f01061e0 0x7fa1f01976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.683 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fa1f0100060 0x7fa1f019dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 --2- 172.21.15.3:0/3846214031 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fa1f00ff6a0 0x7fa1f019d5a0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 >> 172.21.15.3:0/3846214031 conn(0x7fa1f0076620 msgr2=0x7fa1f0104140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 shutdown_connections 2024-02-09T23:04:47.684 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.680+0000 7fa1f4aeb640 1 -- 172.21.15.3:0/3846214031 wait complete. 2024-02-09T23:04:47.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:47 smithi003 ceph-mon[32678]: pgmap v426: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 259 KiB/s wr, 28 op/s 2024-02-09T23:04:47.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:47 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2862492828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:47.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:47 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1977819565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T23:04:47.693 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:47 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/816866339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T23:04:47.694 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-02-09T23:04:47.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 -- 172.21.15.3:0/3962265045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 msgr2=0x7f115810b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.799 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 --2- 172.21.15.3:0/3962265045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f115810b060 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7f1140009900 tx=0x7f114002bcd0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 -- 172.21.15.3:0/3962265045 shutdown_connections 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 --2- 172.21.15.3:0/3962265045 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f115810b630 0x7f115810dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 --2- 172.21.15.3:0/3962265045 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f115810b060 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 --2- 172.21.15.3:0/3962265045 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f1158102260 0x7f1158102640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 -- 172.21.15.3:0/3962265045 >> 172.21.15.3:0/3962265045 conn(0x7f11580fba20 msgr2=0x7f11580fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 -- 172.21.15.3:0/3962265045 shutdown_connections 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.796+0000 7f115da02640 1 -- 172.21.15.3:0/3962265045 wait complete. 2024-02-09T23:04:47.800 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 Processor -- start 2024-02-09T23:04:47.801 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 -- start start 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1158102260 0x7f1158197680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f1158197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115810b630 0x7f115819bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115819c540 con 0x7f1158102b80 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115819c6b0 con 0x7f115810b630 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f115da02640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f115819c870 con 0x7f1158102260 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f1158197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.803 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f1158197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45184/0 (socket says 172.21.15.3:45184) 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 -- 172.21.15.3:0/1321267224 learned_addr learned my addr 172.21.15.3:0/1321267224 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f1156ffd640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1158102260 0x7f1158197680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f11577fe640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115810b630 0x7f115819bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 -- 172.21.15.3:0/1321267224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1158102260 msgr2=0x7f1158197680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1158102260 0x7f1158197680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 -- 172.21.15.3:0/1321267224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115810b630 msgr2=0x7f115819bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115810b630 0x7f115819bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f114ffff640 1 -- 172.21.15.3:0/1321267224 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1140009560 con 0x7f1158102b80 2024-02-09T23:04:47.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.797+0000 7f1156ffd640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1158102260 0x7f1158197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.798+0000 7f114ffff640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f1158197bc0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f1140005940 tx=0x7f1140004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.798+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140004440 con 0x7f1158102b80 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.798+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11400052d0 con 0x7f1158102b80 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.798+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f115819caf0 con 0x7f1158102b80 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.798+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1140003da0 con 0x7f1158102b80 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.798+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11581a43d0 con 0x7f1158102b80 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.799+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1158069f40 con 0x7f1158102b80 2024-02-09T23:04:47.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.800+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f1140005440 con 0x7f1158102b80 2024-02-09T23:04:47.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.801+0000 7f115ca00640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11440a2230 0x7f11440a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:47.806 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.801+0000 7f1156ffd640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11440a2230 0x7f11440a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:47.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.801+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f11400f0f40 con 0x7f1158102b80 2024-02-09T23:04:47.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.801+0000 7f1156ffd640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11440a2230 0x7f11440a46f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f11480058c0 tx=0x7f1148005850 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:47.807 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.804+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f11400b9ce0 con 0x7f1158102b80 2024-02-09T23:04:47.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:47 smithi064 ceph-mon[36638]: pgmap v426: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 259 KiB/s wr, 28 op/s 2024-02-09T23:04:47.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:47 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2862492828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:47.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:47 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1977819565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T23:04:47.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:47 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/816866339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T23:04:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:47 smithi089 ceph-mon[36665]: pgmap v426: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 259 KiB/s wr, 28 op/s 2024-02-09T23:04:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:47 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2862492828' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:47.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:47 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1977819565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-02-09T23:04:47.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:47 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/816866339' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-02-09T23:04:48.000 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.998+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f1158106720 con 0x7f1158102b80 2024-02-09T23:04:48.001 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:47.998+0000 7f115ca00640 1 -- 172.21.15.3:0/1321267224 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v106) v1 ==== 81+0+925 (secure 0 0 0) 0x7f11400bd3a0 con 0x7f1158102b80 2024-02-09T23:04:48.001 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/284621866 2024-02-10T22:51:14.251460+0000 2024-02-09T23:04:48.001 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/284621866 2024-02-10T22:51:14.251460+0000 2024-02-09T23:04:48.001 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/3470258541 2024-02-10T22:51:14.251460+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/2045974484 2024-02-10T22:42:06.452795+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/2951038011 2024-02-10T22:51:14.251460+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/2045974484 2024-02-10T22:42:06.452795+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/2619736972 2024-02-10T22:51:14.251460+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1324827090 2024-02-10T22:42:06.452795+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/2265484823 2024-02-10T22:51:14.251460+0000 2024-02-09T23:04:48.002 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/858232672 2024-02-10T22:41:39.084868+0000 2024-02-09T23:04:48.003 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/715011297 2024-02-10T22:42:06.452795+0000 2024-02-09T23:04:48.003 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6800/2830848370 2024-02-10T22:41:39.084868+0000 2024-02-09T23:04:48.003 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:6801/2830848370 2024-02-10T22:41:39.084868+0000 2024-02-09T23:04:48.003 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1807309503 2024-02-10T22:41:39.084868+0000 2024-02-09T23:04:48.003 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/222016476 2024-02-10T22:41:39.084868+0000 2024-02-09T23:04:48.003 INFO:teuthology.orchestra.run.smithi003.stdout:172.21.15.3:0/1462820865 2024-02-10T22:42:06.452795+0000 2024-02-09T23:04:48.004 INFO:teuthology.orchestra.run.smithi003.stderr:listed 16 entries 2024-02-09T23:04:48.004 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11440a2230 msgr2=0x7f11440a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11440a2230 0x7f11440a46f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f11480058c0 tx=0x7f1148005850 comp rx=0 tx=0).stop 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 msgr2=0x7f1158197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f1158197bc0 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7f1140005940 tx=0x7f1140004040 comp rx=0 tx=0).stop 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 shutdown_connections 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f11440a2230 0x7f11440a46f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f115810b630 0x7f115819bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f1158102b80 0x7f1158197bc0 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 --2- 172.21.15.3:0/1321267224 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f1158102260 0x7f1158197680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.005 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 >> 172.21.15.3:0/1321267224 conn(0x7f11580fba20 msgr2=0x7f11580fc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 shutdown_connections 2024-02-09T23:04:48.006 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.000+0000 7f115da02640 1 -- 172.21.15.3:0/1321267224 wait complete. 2024-02-09T23:04:48.014 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-02-09T23:04:48.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:04:48.014 DEBUG:teuthology.orchestra.run.smithi003:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-02-09T23:04:48.215 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T23:04:48.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:04:48.216 DEBUG:teuthology.orchestra.run.smithi003:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-02-09T23:04:48.273 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:04:48.273 DEBUG:teuthology.orchestra.run.smithi003:> ip netns list 2024-02-09T23:04:48.296 INFO:teuthology.orchestra.run.smithi003.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-02-09T23:04:48.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:04:48.297 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-02-09T23:04:48.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-02-09T23:04:48.365 DEBUG:teuthology.orchestra.run.smithi003:> sudo ip link delete ceph-brx 2024-02-09T23:04:48.456 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-02-09T23:04:48.470 INFO:tasks.cephadm:Teardown begin 2024-02-09T23:04:48.471 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-02-09T23:04:48.471 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:04:48.578 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 -- 172.21.15.3:0/919763577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f700810b820 msgr2=0x7f700810dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 --2- 172.21.15.3:0/919763577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f700810b820 0x7f700810dc70 secure :-1 s=READY pgs=1822 cs=0 l=1 rev1=1 crypto rx=0x7f7000009ca0 tx=0x7f700002c180 comp rx=0 tx=0).stop 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 -- 172.21.15.3:0/919763577 shutdown_connections 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 --2- 172.21.15.3:0/919763577 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f700810b820 0x7f700810dc70 unknown :-1 s=CLOSED pgs=1822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 --2- 172.21.15.3:0/919763577 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7008102d70 0x7f700810b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 --2- 172.21.15.3:0/919763577 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f7008102450 0x7f7008102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 -- 172.21.15.3:0/919763577 >> 172.21.15.3:0/919763577 conn(0x7f70080772b0 msgr2=0x7f70080776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 -- 172.21.15.3:0/919763577 shutdown_connections 2024-02-09T23:04:48.579 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.573+0000 7f70107bc640 1 -- 172.21.15.3:0/919763577 wait complete. 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 Processor -- start 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 -- start start 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7008102450 0x7f700819d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 0x7f700819d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f700810b820 0x7f7008197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008197c50 con 0x7f7008102d70 2024-02-09T23:04:48.580 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008197dc0 con 0x7f700810b820 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f70107bc640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7008197f80 con 0x7f7008102450 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 0x7f700819d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 0x7f700819d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45198/0 (socket says 172.21.15.3:45198) 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 -- 172.21.15.3:0/829547235 learned_addr learned my addr 172.21.15.3:0/829547235 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700ed59640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f700810b820 0x7f7008197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700e558640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7008102450 0x7f700819d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 -- 172.21.15.3:0/829547235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7008102450 msgr2=0x7f700819d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.581 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7008102450 0x7f700819d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 -- 172.21.15.3:0/829547235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f700810b820 msgr2=0x7f7008197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f700810b820 0x7f7008197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 -- 172.21.15.3:0/829547235 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000009900 con 0x7f7008102d70 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.574+0000 7f700dd57640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 0x7f700819d990 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d950 tx=0x7f6ff800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.575+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8009a30 con 0x7f7008102d70 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.575+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ff8009b90 con 0x7f7008102d70 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.575+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7008198260 con 0x7f7008102d70 2024-02-09T23:04:48.582 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.575+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff8004a00 con 0x7f7008102d70 2024-02-09T23:04:48.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.575+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70081a44f0 con 0x7f7008102d70 2024-02-09T23:04:48.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.576+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70080680b0 con 0x7f7008102d70 2024-02-09T23:04:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.577+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f6ff8009d00 con 0x7f7008102d70 2024-02-09T23:04:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.577+0000 7f6fff7fe640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6ff40a22b0 0x7f6ff40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.577+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f6ff80cdfb0 con 0x7f7008102d70 2024-02-09T23:04:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.577+0000 7f700e558640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6ff40a22b0 0x7f6ff40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.578+0000 7f700e558640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6ff40a22b0 0x7f6ff40a4770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7004009bd0 tx=0x7f7004005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:48.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.581+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f6ff8096cc0 con 0x7f7008102d70 2024-02-09T23:04:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:48 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/3846214031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T23:04:48.632 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:48 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1321267224' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-09T23:04:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:48 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/3846214031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T23:04:48.727 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:48 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1321267224' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-09T23:04:48.804 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.801+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70081069a0 con 0x7f7008102d70 2024-02-09T23:04:48.805 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.802+0000 7f6fff7fe640 1 -- 172.21.15.3:0/829547235 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+11374 (secure 0 0 0) 0x7f70081069a0 con 0x7f7008102d70 2024-02-09T23:04:48.805 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:48.806 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:04:28.046058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_28706":{"gid":28706,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/1843835937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1843835937},{"type":"v1","addr":"172.21.15.64:6835","nonce":1843835937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:04:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 57 2024-02-09T23:04:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6ff40a22b0 msgr2=0x7f6ff40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6ff40a22b0 0x7f6ff40a4770 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7004009bd0 tx=0x7f7004005f90 comp rx=0 tx=0).stop 2024-02-09T23:04:48.808 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 msgr2=0x7f700819d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 0x7f700819d990 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f6ff800d950 tx=0x7f6ff800de10 comp rx=0 tx=0).stop 2024-02-09T23:04:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 shutdown_connections 2024-02-09T23:04:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f6ff40a22b0 0x7f6ff40a4770 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f700810b820 0x7f7008197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f7008102d70 0x7f700819d990 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 --2- 172.21.15.3:0/829547235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f7008102450 0x7f700819d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 >> 172.21.15.3:0/829547235 conn(0x7f70080772b0 msgr2=0x7f70080fd1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:48.810 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 shutdown_connections 2024-02-09T23:04:48.811 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.804+0000 7f70107bc640 1 -- 172.21.15.3:0/829547235 wait complete. 2024-02-09T23:04:48.818 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-02-09T23:04:48.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1847882937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8100d00 msgr2=0x7fe1e8104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.924 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1847882937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8100d00 0x7fe1e8104d40 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fe1d0009930 tx=0x7fe1d002bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1847882937 shutdown_connections 2024-02-09T23:04:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1847882937 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 0x7fe1e810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1847882937 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8100d00 0x7fe1e8104d40 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1847882937 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e8100350 0x7fe1e8100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1847882937 >> 172.21.15.3:0/1847882937 conn(0x7fe1e80772b0 msgr2=0x7fe1e80776c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:48.925 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1847882937 shutdown_connections 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.918+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1847882937 wait complete. 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 Processor -- start 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 -- start start 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e8100350 0x7fe1e819d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 0x7fe1e819dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 0x7fe1e8197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.926 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8198210 con 0x7fe1e8197740 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8198380 con 0x7fe1e8105420 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1ed53a640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1e8198540 con 0x7fe1e8100350 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 0x7fe1e8197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1e77fe640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 0x7fe1e8197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45210/0 (socket says 172.21.15.3:45210) 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1e77fe640 1 -- 172.21.15.3:0/1454286402 learned_addr learned my addr 172.21.15.3:0/1454286402 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.919+0000 7fe1e6ffd640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e8100350 0x7fe1e819d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1dffff640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 0x7fe1e819dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.927 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e77fe640 1 -- 172.21.15.3:0/1454286402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e8100350 msgr2=0x7fe1e819d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e77fe640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e8100350 0x7fe1e819d680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e77fe640 1 -- 172.21.15.3:0/1454286402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 msgr2=0x7fe1e819dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e77fe640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 0x7fe1e819dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e77fe640 1 -- 172.21.15.3:0/1454286402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d0009560 con 0x7fe1e8197740 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1dffff640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 0x7fe1e819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e77fe640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 0x7fe1e8197bc0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000eaa0 tx=0x7fe1e000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e000cdb0 con 0x7fe1e8197740 2024-02-09T23:04:48.928 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1e0005020 con 0x7fe1e8197740 2024-02-09T23:04:48.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e00116b0 con 0x7fe1e8197740 2024-02-09T23:04:48.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1e81a44e0 con 0x7fe1e8197740 2024-02-09T23:04:48.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.920+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1e81a4970 con 0x7fe1e8197740 2024-02-09T23:04:48.929 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.921+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1e8101820 con 0x7fe1e8197740 2024-02-09T23:04:48.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.922+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe1e0005520 con 0x7fe1e8197740 2024-02-09T23:04:48.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.923+0000 7fe1e4ff9640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1d40a2230 0x7fe1d40a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:48.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.923+0000 7fe1e6ffd640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1d40a2230 0x7fe1d40a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:48.930 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.923+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe1e00ce630 con 0x7fe1e8197740 2024-02-09T23:04:48.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.923+0000 7fe1e6ffd640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1d40a2230 0x7fe1d40a46f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005fd0 tx=0x7fe1d800b390 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:48.931 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:48.926+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe1e00972c0 con 0x7fe1e8197740 2024-02-09T23:04:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:48 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/3846214031' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-02-09T23:04:48.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:48 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1321267224' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-02-09T23:04:49.150 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.148+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1e8109660 con 0x7fe1e8197740 2024-02-09T23:04:49.151 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.148+0000 7fe1e4ff9640 1 -- 172.21.15.3:0/1454286402 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+11374 (secure 0 0 0) 0x7fe1e001f020 con 0x7fe1e8197740 2024-02-09T23:04:49.151 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:49.151 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14550,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6835/977042251","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6834","nonce":977042251},{"type":"v1","addr":"172.21.15.3:6835","nonce":977042251}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.3:6837/1424466335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6836","nonce":1424466335},{"type":"v1","addr":"172.21.15.3:6837","nonce":1424466335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":57,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-02-09T22:49:06.610270+0000","modified":"2024-02-09T23:04:28.046058+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24533,"mds_1":24503,"mds_2":24496,"mds_3":24481,"mds_4":24475},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14634":{"gid":14634,"name":"g","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6839/3695104270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6838","nonce":3695104270},{"type":"v1","addr":"172.21.15.3:6839","nonce":3695104270}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_14646":{"gid":14646,"name":"a","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.3:6841/3916323155","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6840","nonce":3916323155},{"type":"v1","addr":"172.21.15.3:6841","nonce":3916323155}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24427":{"gid":24427,"name":"l","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.89:6833/2685783987","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2685783987},{"type":"v1","addr":"172.21.15.89:6833","nonce":2685783987}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24437":{"gid":24437,"name":"k","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6833/1710343231","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6832","nonce":1710343231},{"type":"v1","addr":"172.21.15.64:6833","nonce":1710343231}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24475":{"gid":24475,"name":"f","rank":4,"incarnation":22,"state":"up:active","state_seq":8,"addr":"172.21.15.89:6835/4241363743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6834","nonce":4241363743},{"type":"v1","addr":"172.21.15.89:6835","nonce":4241363743}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24481":{"gid":24481,"name":"c","rank":3,"incarnation":20,"state":"up:active","state_seq":7,"addr":"172.21.15.89:6837/1935794058","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6836","nonce":1935794058},{"type":"v1","addr":"172.21.15.89:6837","nonce":1935794058}]},"join_fscid":-1,"export_targets":[0,1,2,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24496":{"gid":24496,"name":"i","rank":2,"incarnation":18,"state":"up:active","state_seq":6,"addr":"172.21.15.89:6839/305807401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6838","nonce":305807401},{"type":"v1","addr":"172.21.15.89:6839","nonce":305807401}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24503":{"gid":24503,"name":"h","rank":1,"incarnation":16,"state":"up:active","state_seq":6,"addr":"172.21.15.64:6837/3217313563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6836","nonce":3217313563},{"type":"v1","addr":"172.21.15.64:6837","nonce":3217313563}]},"join_fscid":-1,"export_targets":[0,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24533":{"gid":24533,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.64:6839/4207075446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6838","nonce":4207075446},{"type":"v1","addr":"172.21.15.64:6839","nonce":4207075446}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_28706":{"gid":28706,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.64:6835/1843835937","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6834","nonce":1843835937},{"type":"v1","addr":"172.21.15.64:6835","nonce":1843835937}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-02-09T23:04:49.152 INFO:teuthology.orchestra.run.smithi003.stderr:dumped fsmap epoch 57 2024-02-09T23:04:49.153 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.150+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1d40a2230 msgr2=0x7fe1d40a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.150+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1d40a2230 0x7fe1d40a46f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8005fd0 tx=0x7fe1d800b390 comp rx=0 tx=0).stop 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.150+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 msgr2=0x7fe1e8197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.150+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 0x7fe1e8197bc0 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fe1e000eaa0 tx=0x7fe1e000ef60 comp rx=0 tx=0).stop 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 shutdown_connections 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1d40a2230 0x7fe1d40a46f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1e8105420 0x7fe1e819dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1e8197740 0x7fe1e8197bc0 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.154 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 --2- 172.21.15.3:0/1454286402 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1e8100350 0x7fe1e819d680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 >> 172.21.15.3:0/1454286402 conn(0x7fe1e80772b0 msgr2=0x7fe1e80688e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:49.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 shutdown_connections 2024-02-09T23:04:49.155 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.151+0000 7fe1ed53a640 1 -- 172.21.15.3:0/1454286402 wait complete. 2024-02-09T23:04:49.165 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.264+0000 7fe1b4e15640 1 -- 172.21.15.3:0/2719270671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100d00 msgr2=0x7fe1b0104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.264+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/2719270671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100d00 0x7fe1b0104d40 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009930 tx=0x7fe1a402bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 -- 172.21.15.3:0/2719270671 shutdown_connections 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/2719270671 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1b0105420 0x7fe1b010cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/2719270671 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100d00 0x7fe1b0104d40 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/2719270671 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1b0100350 0x7fe1b0100730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.270 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 -- 172.21.15.3:0/2719270671 >> 172.21.15.3:0/2719270671 conn(0x7fe1b00fbbd0 msgr2=0x7fe1b00fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 -- 172.21.15.3:0/2719270671 shutdown_connections 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 -- 172.21.15.3:0/2719270671 wait complete. 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 Processor -- start 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.265+0000 7fe1b4e15640 1 -- start start 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 0x7fe1b010d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1b0105420 0x7fe1b010de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1b010e350 0x7fe1b01a5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01a6140 con 0x7fe1b0100350 2024-02-09T23:04:49.271 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01a62b0 con 0x7fe1b0105420 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01a6470 con 0x7fe1b010e350 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 0x7fe1b010d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 0x7fe1b010d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45234/0 (socket says 172.21.15.3:45234) 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 -- 172.21.15.3:0/4226252720 learned_addr learned my addr 172.21.15.3:0/4226252720 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1aed9d640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1b010e350 0x7fe1b01a5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1add9b640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1b0105420 0x7fe1b010de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 -- 172.21.15.3:0/4226252720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1b010e350 msgr2=0x7fe1b01a5bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.272 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1b010e350 0x7fe1b01a5bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 -- 172.21.15.3:0/4226252720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1b0105420 msgr2=0x7fe1b010de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1b0105420 0x7fe1b010de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 -- 172.21.15.3:0/4226252720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a4009560 con 0x7fe1b0100350 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1ae59c640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 0x7fe1b010d8d0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800b4a0 tx=0x7fe1a800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8010610 con 0x7fe1b0100350 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b01a6750 con 0x7fe1b0100350 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1a8010c50 con 0x7fe1b0100350 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.266+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8017e20 con 0x7fe1b0100350 2024-02-09T23:04:49.273 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.267+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b01a6c80 con 0x7fe1b0100350 2024-02-09T23:04:49.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.268+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b0068500 con 0x7fe1b0100350 2024-02-09T23:04:49.274 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.269+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7fe1a8017480 con 0x7fe1b0100350 2024-02-09T23:04:49.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.269+0000 7fe19f7fe640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1980a2230 0x7fe1980a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.269+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7fe1a80cd610 con 0x7fe1b0100350 2024-02-09T23:04:49.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.269+0000 7fe1add9b640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1980a2230 0x7fe1980a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.275 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.270+0000 7fe1add9b640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1980a2230 0x7fe1980a46f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009900 tx=0x7fe1a4005fd0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:49.276 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.274+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7fe1a80963b0 con 0x7fe1b0100350 2024-02-09T23:04:49.461 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.458+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1b0108650 con 0x7fe1b0100350 2024-02-09T23:04:49.463 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.460+0000 7fe19f7fe640 1 -- 172.21.15.3:0/4226252720 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v106) v1 ==== 74+0+22923 (secure 0 0 0) 0x7fe1a8099a70 con 0x7fe1b0100350 2024-02-09T23:04:49.463 INFO:teuthology.orchestra.run.smithi003.stdout: 2024-02-09T23:04:49.463 INFO:teuthology.orchestra.run.smithi003.stdout:{"epoch":106,"fsid":"2bf8a73a-c79c-11ee-95b8-87774f69a715","created":"2024-02-09T22:41:08.917818+0000","modified":"2024-02-09T23:04:09.305034+0000","last_up_change":"2024-02-09T22:48:16.457999+0000","last_in_change":"2024-02-09T22:47:56.222846+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-02-09T22:44:32.790973+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-02-09T22:49:04.801615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-02-09T22:49:05.816157+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":21,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c1313805-be20-4bd1-a375-4d8c740e1ff2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6802","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6803","nonce":103055907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6804","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6805","nonce":103055907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6808","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6809","nonce":103055907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6806","nonce":103055907},{"type":"v1","addr":"172.21.15.3:6807","nonce":103055907}]},"public_addr":"172.21.15.3:6803/103055907","cluster_addr":"172.21.15.3:6805/103055907","heartbeat_back_addr":"172.21.15.3:6809/103055907","heartbeat_front_addr":"172.21.15.3:6807/103055907","state":["exists","up"]},{"osd":1,"uuid":"b4780ed6-656c-4d81-9520-d56cc32e39f9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6800","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6801","nonce":368704973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6802","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6803","nonce":368704973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6806","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6807","nonce":368704973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6804","nonce":368704973},{"type":"v1","addr":"172.21.15.64:6805","nonce":368704973}]},"public_addr":"172.21.15.64:6801/368704973","cluster_addr":"172.21.15.64:6803/368704973","heartbeat_back_addr":"172.21.15.64:6807/368704973","heartbeat_front_addr":"172.21.15.64:6805/368704973","state":["exists","up"]},{"osd":2,"uuid":"d1c14346-6703-4597-8214-860019453569","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6801","nonce":2362328563}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6803","nonce":2362328563}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6807","nonce":2362328563}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6804","nonce":2362328563},{"type":"v1","addr":"172.21.15.89:6805","nonce":2362328563}]},"public_addr":"172.21.15.89:6801/2362328563","cluster_addr":"172.21.15.89:6803/2362328563","heartbeat_back_addr":"172.21.15.89:6807/2362328563","heartbeat_front_addr":"172.21.15.89:6805/2362328563","state":["exists","up"]},{"osd":3,"uuid":"c7129351-e0c4-4563-8e98-d818d30d891a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6810","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6811","nonce":3084237347}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6812","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6813","nonce":3084237347}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6816","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6817","nonce":3084237347}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6814","nonce":3084237347},{"type":"v1","addr":"172.21.15.3:6815","nonce":3084237347}]},"public_addr":"172.21.15.3:6811/3084237347","cluster_addr":"172.21.15.3:6813/3084237347","heartbeat_back_addr":"172.21.15.3:6817/3084237347","heartbeat_front_addr":"172.21.15.3:6815/3084237347","state":["exists","up"]},{"osd":4,"uuid":"bef8649b-bcc5-40c9-8df2-b59957282a8d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6808","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6809","nonce":285015639}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6810","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6811","nonce":285015639}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6814","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6815","nonce":285015639}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6812","nonce":285015639},{"type":"v1","addr":"172.21.15.64:6813","nonce":285015639}]},"public_addr":"172.21.15.64:6809/285015639","cluster_addr":"172.21.15.64:6811/285015639","heartbeat_back_addr":"172.21.15.64:6815/285015639","heartbeat_front_addr":"172.21.15.64:6813/285015639","state":["exists","up"]},{"osd":5,"uuid":"245a8e5f-2535-4346-8457-dd94e211574d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6808","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6809","nonce":708499709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6811","nonce":708499709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6815","nonce":708499709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6812","nonce":708499709},{"type":"v1","addr":"172.21.15.89:6813","nonce":708499709}]},"public_addr":"172.21.15.89:6809/708499709","cluster_addr":"172.21.15.89:6811/708499709","heartbeat_back_addr":"172.21.15.89:6815/708499709","heartbeat_front_addr":"172.21.15.89:6813/708499709","state":["exists","up"]},{"osd":6,"uuid":"392d6f64-4fbc-472e-96ef-dfb2b5291327","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6818","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6819","nonce":1303593428}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6820","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6821","nonce":1303593428}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6824","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6825","nonce":1303593428}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6822","nonce":1303593428},{"type":"v1","addr":"172.21.15.3:6823","nonce":1303593428}]},"public_addr":"172.21.15.3:6819/1303593428","cluster_addr":"172.21.15.3:6821/1303593428","heartbeat_back_addr":"172.21.15.3:6825/1303593428","heartbeat_front_addr":"172.21.15.3:6823/1303593428","state":["exists","up"]},{"osd":7,"uuid":"c96f9dfa-d559-4d7f-9fde-00d3fbbe856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6816","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6817","nonce":867242656}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6818","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6819","nonce":867242656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6822","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6823","nonce":867242656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6820","nonce":867242656},{"type":"v1","addr":"172.21.15.64:6821","nonce":867242656}]},"public_addr":"172.21.15.64:6817/867242656","cluster_addr":"172.21.15.64:6819/867242656","heartbeat_back_addr":"172.21.15.64:6823/867242656","heartbeat_front_addr":"172.21.15.64:6821/867242656","state":["exists","up"]},{"osd":8,"uuid":"49491de9-7b30-46a5-9869-b3ff6d2285f7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6817","nonce":4011062821}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6819","nonce":4011062821}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6823","nonce":4011062821}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":4011062821},{"type":"v1","addr":"172.21.15.89:6821","nonce":4011062821}]},"public_addr":"172.21.15.89:6817/4011062821","cluster_addr":"172.21.15.89:6819/4011062821","heartbeat_back_addr":"172.21.15.89:6823/4011062821","heartbeat_front_addr":"172.21.15.89:6821/4011062821","state":["exists","up"]},{"osd":9,"uuid":"46d134d1-6b63-4909-8a0a-c5c431c9dfdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6826","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6827","nonce":2163219868}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6828","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6829","nonce":2163219868}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6832","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6833","nonce":2163219868}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.3:6830","nonce":2163219868},{"type":"v1","addr":"172.21.15.3:6831","nonce":2163219868}]},"public_addr":"172.21.15.3:6827/2163219868","cluster_addr":"172.21.15.3:6829/2163219868","heartbeat_back_addr":"172.21.15.3:6833/2163219868","heartbeat_front_addr":"172.21.15.3:6831/2163219868","state":["exists","up"]},{"osd":10,"uuid":"c065623e-ebf8-4100-8d19-26bcc2a837aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6824","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6825","nonce":451880154}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6826","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6827","nonce":451880154}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6830","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6831","nonce":451880154}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.64:6828","nonce":451880154},{"type":"v1","addr":"172.21.15.64:6829","nonce":451880154}]},"public_addr":"172.21.15.64:6825/451880154","cluster_addr":"172.21.15.64:6827/451880154","heartbeat_back_addr":"172.21.15.64:6831/451880154","heartbeat_front_addr":"172.21.15.64:6829/451880154","state":["exists","up"]},{"osd":11,"uuid":"33affb4f-8f05-46cd-b347-fe905769c48c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6824","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6825","nonce":2302784119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6827","nonce":2302784119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6831","nonce":2302784119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":2302784119},{"type":"v1","addr":"172.21.15.89:6829","nonce":2302784119}]},"public_addr":"172.21.15.89:6825/2302784119","cluster_addr":"172.21.15.89:6827/2302784119","heartbeat_back_addr":"172.21.15.89:6831/2302784119","heartbeat_front_addr":"172.21.15.89:6829/2302784119","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:43:42.676599+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:07.047687+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:30.066748+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:44:54.964745+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:20.287496+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:45:43.873421+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:08.667693+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:33.743883+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:46:57.689078+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:22.870296+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:47:48.543772+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-02-09T22:48:14.568158+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.b","mappings":[{"from":2,"to":4}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.11","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.31","mappings":[{"from":4,"to":6}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.3:6801/284621866":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6800/284621866":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/3470258541":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6801/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2951038011":"2024-02-10T22:51:14.251460+0000","172.21.15.3:6800/2045974484":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2619736972":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/1324827090":"2024-02-10T22:42:06.452795+0000","172.21.15.3:0/2265484823":"2024-02-10T22:51:14.251460+0000","172.21.15.3:0/858232672":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/715011297":"2024-02-10T22:42:06.452795+0000","172.21.15.3:6800/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:6801/2830848370":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1807309503":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/222016476":"2024-02-10T22:41:39.084868+0000","172.21.15.3:0/1462820865":"2024-02-10T22:42:06.452795+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":8,"length":1},{"begin":21,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-02-09T23:04:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1980a2230 msgr2=0x7fe1980a46f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1980a2230 0x7fe1980a46f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4009900 tx=0x7fe1a4005fd0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 msgr2=0x7fe1b010d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.465 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 0x7fe1b010d8d0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800b4a0 tx=0x7fe1a800b960 comp rx=0 tx=0).stop 2024-02-09T23:04:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 shutdown_connections 2024-02-09T23:04:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7fe1980a2230 0x7fe1980a46f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7fe1b0105420 0x7fe1b010de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7fe1b010e350 0x7fe1b01a5bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 --2- 172.21.15.3:0/4226252720 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7fe1b0100350 0x7fe1b010d8d0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.466 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 >> 172.21.15.3:0/4226252720 conn(0x7fe1b00fbbd0 msgr2=0x7fe1b00fcf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:49.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 shutdown_connections 2024-02-09T23:04:49.467 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.462+0000 7fe1b4e15640 1 -- 172.21.15.3:0/4226252720 wait complete. 2024-02-09T23:04:49.477 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-02-09T23:04:49.583 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 -- 172.21.15.3:0/1828365181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4101430 msgr2=0x7f95a4101810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 --2- 172.21.15.3:0/1828365181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4101430 0x7f95a4101810 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7f95a0009930 tx=0x7f95a002bc20 comp rx=0 tx=0).stop 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 -- 172.21.15.3:0/1828365181 shutdown_connections 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 --2- 172.21.15.3:0/1828365181 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95a4106500 0x7f95a410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 --2- 172.21.15.3:0/1828365181 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a4101de0 0x7f95a4105e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 --2- 172.21.15.3:0/1828365181 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4101430 0x7f95a4101810 unknown :-1 s=CLOSED pgs=1828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 -- 172.21.15.3:0/1828365181 >> 172.21.15.3:0/1828365181 conn(0x7f95a40fccd0 msgr2=0x7f95a40ff0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:49.584 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 -- 172.21.15.3:0/1828365181 shutdown_connections 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.578+0000 7f95ab913640 1 -- 172.21.15.3:0/1828365181 wait complete. 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 Processor -- start 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 -- start start 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95a4101430 0x7f95a419d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a4106500 0x7f95a419db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 0x7f95a4197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a41981d0 con 0x7f95a4197700 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4198340 con 0x7f95a4101430 2024-02-09T23:04:49.585 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95ab913640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a4198500 con 0x7f95a4106500 2024-02-09T23:04:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 0x7f95a4197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 0x7f95a4197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45250/0 (socket says 172.21.15.3:45250) 2024-02-09T23:04:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 -- 172.21.15.3:0/2509280105 learned_addr learned my addr 172.21.15.3:0/2509280105 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a8eae640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a4106500 0x7f95a419db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.586 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a96af640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95a4101430 0x7f95a419d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 -- 172.21.15.3:0/2509280105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a4106500 msgr2=0x7f95a419db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a4106500 0x7f95a419db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 -- 172.21.15.3:0/2509280105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95a4101430 msgr2=0x7f95a419d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.587 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95a4101430 0x7f95a419d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 -- 172.21.15.3:0/2509280105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a0009560 con 0x7f95a4197700 2024-02-09T23:04:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.579+0000 7f95a9eb0640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 0x7f95a4197b80 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f959c00cce0 tx=0x7f959c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.580+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c00f450 con 0x7f95a4197700 2024-02-09T23:04:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.580+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f959c00fa90 con 0x7f95a4197700 2024-02-09T23:04:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.580+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a41a4450 con 0x7f95a4197700 2024-02-09T23:04:49.588 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.580+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c018b90 con 0x7f95a4197700 2024-02-09T23:04:49.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.580+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41a4930 con 0x7f95a4197700 2024-02-09T23:04:49.589 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.581+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a4102900 con 0x7f95a4197700 2024-02-09T23:04:49.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.582+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f959c00fc00 con 0x7f95a4197700 2024-02-09T23:04:49.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.583+0000 7f959a7fc640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95900a22b0 0x7f95900a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.583+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 8513+0+0 (secure 0 0 0) 0x7f959c0ce8a0 con 0x7f95a4197700 2024-02-09T23:04:49.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.583+0000 7f95a96af640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95900a22b0 0x7f95900a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.590 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.583+0000 7f95a96af640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95900a22b0 0x7f95900a4770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f95a0006010 tx=0x7f95a0000bc0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:49.591 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.587+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f959c097640 con 0x7f95a4197700 2024-02-09T23:04:49.809 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.806+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f95a4109730 con 0x7f95a4197700 2024-02-09T23:04:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-mon[32678]: pgmap v427: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-02-09T23:04:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/829547235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1454286402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.811 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/4226252720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.808+0000 7f959a7fc640 1 -- 172.21.15.3:0/2509280105 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f959c09ad00 con 0x7f95a4197700 2024-02-09T23:04:49.812 INFO:teuthology.orchestra.run.smithi003.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-02-09T23:04:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95900a22b0 msgr2=0x7f95900a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95900a22b0 0x7f95900a4770 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f95a0006010 tx=0x7f95a0000bc0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 msgr2=0x7f95a4197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 0x7f95a4197b80 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f959c00cce0 tx=0x7f959c00a620 comp rx=0 tx=0).stop 2024-02-09T23:04:49.815 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 shutdown_connections 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f95900a22b0 0x7f95900a4770 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f95a4197700 0x7f95a4197b80 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f95a4106500 0x7f95a419db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 --2- 172.21.15.3:0/2509280105 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f95a4101430 0x7f95a419d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 >> 172.21.15.3:0/2509280105 conn(0x7f95a40fccd0 msgr2=0x7f95a40ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 shutdown_connections 2024-02-09T23:04:49.816 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.812+0000 7f95ab913640 1 -- 172.21.15.3:0/2509280105 wait complete. 2024-02-09T23:04:49.831 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-02-09T23:04:49.859 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mon[36638]: pgmap v427: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-02-09T23:04:49.859 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/829547235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.859 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1454286402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.859 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/4226252720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mon[36665]: pgmap v427: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 15 KiB/s wr, 12 op/s 2024-02-09T23:04:49.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/829547235' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1454286402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.868 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/4226252720' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-02-09T23:04:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.937+0000 7ff90f023640 1 -- 172.21.15.3:0/1398480786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff908100350 msgr2=0x7ff908100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.937+0000 7ff90f023640 1 --2- 172.21.15.3:0/1398480786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff908100350 0x7ff908100730 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc009930 tx=0x7ff8fc02bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.937+0000 7ff90f023640 1 -- 172.21.15.3:0/1398480786 shutdown_connections 2024-02-09T23:04:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 --2- 172.21.15.3:0/1398480786 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff908105420 0x7ff90810cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.943 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 --2- 172.21.15.3:0/1398480786 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908100d00 0x7ff908104d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 --2- 172.21.15.3:0/1398480786 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff908100350 0x7ff908100730 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- 172.21.15.3:0/1398480786 >> 172.21.15.3:0/1398480786 conn(0x7ff908077280 msgr2=0x7ff908077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- 172.21.15.3:0/1398480786 shutdown_connections 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- 172.21.15.3:0/1398480786 wait complete. 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 Processor -- start 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- start start 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff908100350 0x7ff908198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.944 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908105420 0x7ff908198c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 0x7ff9081a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9081a3e50 con 0x7ff9081991d0 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9081a3fc0 con 0x7ff908100350 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90f023640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9081a4180 con 0x7ff908105420 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90d5c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 0x7ff9081a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90d5c0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 0x7ff9081a3780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45308/0 (socket says 172.21.15.3:45308) 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff90d5c0640 1 -- 172.21.15.3:0/2862681811 learned_addr learned my addr 172.21.15.3:0/2862681811 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:49.945 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.938+0000 7ff907fff640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908105420 0x7ff908198c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90cdbf640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff908100350 0x7ff908198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90d5c0640 1 -- 172.21.15.3:0/2862681811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908105420 msgr2=0x7ff908198c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90d5c0640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908105420 0x7ff908198c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90d5c0640 1 -- 172.21.15.3:0/2862681811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff908100350 msgr2=0x7ff908198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90d5c0640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff908100350 0x7ff908198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90d5c0640 1 -- 172.21.15.3:0/2862681811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc009560 con 0x7ff9081991d0 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff907fff640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908105420 0x7ff908198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90d5c0640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 0x7ff9081a3780 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7ff90000eaa0 tx=0x7ff90000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:49.946 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff90000cdb0 con 0x7ff9081991d0 2024-02-09T23:04:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff900005020 con 0x7ff9081991d0 2024-02-09T23:04:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9081a4460 con 0x7ff9081991d0 2024-02-09T23:04:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9000116b0 con 0x7ff9081991d0 2024-02-09T23:04:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.939+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9081a4970 con 0x7ff9081991d0 2024-02-09T23:04:49.947 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.940+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff908067820 con 0x7ff9081991d0 2024-02-09T23:04:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.941+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7ff900005520 con 0x7ff9081991d0 2024-02-09T23:04:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.942+0000 7ff905ffb640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff8f80a22b0 0x7ff8f80a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.942+0000 7ff90cdbf640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff8f80a22b0 0x7ff8f80a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.942+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 8899+0+0 (secure 0 0 0) 0x7ff9000ce440 con 0x7ff9081991d0 2024-02-09T23:04:49.948 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.942+0000 7ff90cdbf640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff8f80a22b0 0x7ff8f80a4770 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc006010 tx=0x7ff8fc000bc0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:49.949 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:49.946+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7ff900096ef0 con 0x7ff9081991d0 2024-02-09T23:04:50.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a[32674]: 2024-02-09T23:04:49.807+0000 7f6c76e34700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T23:04:50.132 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a[73656]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.132 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a[73656]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.132 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-mds[73662]: -- 172.21.15.3:0/682766546 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bbebf3e1a0 con 0x55bbebf15000 2024-02-09T23:04:50.133 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a[73656]: starting mds.a at 2024-02-09T23:04:50.133 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g[73280]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.133 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g[73280]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.133 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-mds[73303]: -- 172.21.15.3:0/2201114022 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561ecb04a1a0 con 0x561ecb021000 2024-02-09T23:04:50.133 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:04:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g[73280]: starting mds.g at 2024-02-09T23:04:50.167 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.165+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff908108650 con 0x7ff9081991d0 2024-02-09T23:04:50.185 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.185 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.186 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mds[67237]: -- 172.21.15.64:0/1667951897 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56336204c1a0 con 0x563362022800 2024-02-09T23:04:50.186 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: starting mds.b at 2024-02-09T23:04:50.186 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.186 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.186 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mds[66607]: -- 172.21.15.64:0/1873144159 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561e31e361a0 con 0x561e31e0cc00 2024-02-09T23:04:50.186 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: starting mds.e at 2024-02-09T23:04:50.187 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k[66281]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.187 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k[66281]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.187 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mds[66285]: -- 172.21.15.64:0/1372360767 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f6a01e81a0 con 0x55f6a01bec00 2024-02-09T23:04:50.187 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k[66281]: starting mds.k at 2024-02-09T23:04:50.187 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h[66914]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.188 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h[66914]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.188 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-mds[66918]: -- 172.21.15.64:0/2036498417 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55de5bae01a0 con 0x55de5bab6c00 2024-02-09T23:04:50.188 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:04:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h[66914]: starting mds.h at 2024-02-09T23:04:50.227 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f[66706]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.227 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f[66706]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.227 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mds[66710]: -- 172.21.15.89:0/1394252180 <== mon.2 v2:172.21.15.64:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56016ca401a0 con 0x56016ca17000 2024-02-09T23:04:50.227 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f[66706]: starting mds.f at 2024-02-09T23:04:50.228 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c[67015]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.228 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c[67015]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.228 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mds[67019]: -- 172.21.15.89:0/1474108295 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55edbfbf41a0 con 0x55edbfbca800 2024-02-09T23:04:50.228 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c[67015]: starting mds.c at 2024-02-09T23:04:50.229 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i[67337]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.229 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i[67337]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.229 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mds[67341]: -- 172.21.15.89:0/3578654369 <== mon.1 v2:172.21.15.89:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56399cce7080 con 0x56399ccb0800 2024-02-09T23:04:50.229 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mds[67341]: -- 172.21.15.89:0/3578654369 <== mon.1 v2:172.21.15.89:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56399ccda1a0 con 0x56399ccb0800 2024-02-09T23:04:50.229 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i[67337]: starting mds.i at 2024-02-09T23:04:50.229 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l[66400]: ignoring --setuser ceph since I am not root 2024-02-09T23:04:50.230 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l[66400]: ignoring --setgroup ceph since I am not root 2024-02-09T23:04:50.230 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-mds[66404]: -- 172.21.15.89:0/3034060067 <== mon.0 v2:172.21.15.3:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b2da6ae1a0 con 0x55b2da684800 2024-02-09T23:04:50.230 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:04:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l[66400]: starting mds.l at 2024-02-09T23:04:50.819 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.816+0000 7ff905ffb640 1 -- 172.21.15.3:0/2862681811 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff90009a5b0 con 0x7ff9081991d0 2024-02-09T23:04:50.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff8f80a22b0 msgr2=0x7ff8f80a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:50.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff8f80a22b0 0x7ff8f80a4770 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff8fc006010 tx=0x7ff8fc000bc0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 msgr2=0x7ff9081a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:50.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 0x7ff9081a3780 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7ff90000eaa0 tx=0x7ff90000ef60 comp rx=0 tx=0).stop 2024-02-09T23:04:50.822 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 shutdown_connections 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7ff8f80a22b0 0x7ff8f80a4770 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7ff9081991d0 0x7ff9081a3780 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7ff908105420 0x7ff908198c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 --2- 172.21.15.3:0/2862681811 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7ff908100350 0x7ff908198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 >> 172.21.15.3:0/2862681811 conn(0x7ff908077280 msgr2=0x7ff9080fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 shutdown_connections 2024-02-09T23:04:50.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.819+0000 7ff90f023640 1 -- 172.21.15.3:0/2862681811 wait complete. 2024-02-09T23:04:50.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2509280105' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-02-09T23:04:50.837 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-09T23:04:50.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T23:04:50.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2509280105' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-02-09T23:04:50.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: osdmap e107: 12 total, 12 up, 12 in 2024-02-09T23:04:50.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-02-09T23:04:50.838 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:50 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2862681811' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T23:04:50.839 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-02-09T23:04:50.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2509280105' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-02-09T23:04:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-09T23:04:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T23:04:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2509280105' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-02-09T23:04:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: osdmap e107: 12 total, 12 up, 12 in 2024-02-09T23:04:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-02-09T23:04:50.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:50 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2862681811' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T23:04:50.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 -- 172.21.15.3:0/3078161909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c8105410 msgr2=0x7f17c810cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 --2- 172.21.15.3:0/3078161909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c8105410 0x7f17c810cfe0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f17c0009cd0 tx=0x7f17c002c0f0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 -- 172.21.15.3:0/3078161909 shutdown_connections 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 --2- 172.21.15.3:0/3078161909 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c8105410 0x7f17c810cfe0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 --2- 172.21.15.3:0/3078161909 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100cf0 0x7f17c8104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 --2- 172.21.15.3:0/3078161909 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17c8100340 0x7f17c8100720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 -- 172.21.15.3:0/3078161909 >> 172.21.15.3:0/3078161909 conn(0x7f17c80772e0 msgr2=0x7f17c80776f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 -- 172.21.15.3:0/3078161909 shutdown_connections 2024-02-09T23:04:50.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.949+0000 7f17cde35640 1 -- 172.21.15.3:0/3078161909 wait complete. 2024-02-09T23:04:50.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 Processor -- start 2024-02-09T23:04:50.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 -- start start 2024-02-09T23:04:50.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100340 0x7f17c810d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:50.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17c8105410 0x7f17c810de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:50.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 0x7f17c81a5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:50.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a5ff0 con 0x7f17c810e340 2024-02-09T23:04:50.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a6160 con 0x7f17c8105410 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17cde35640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c81a6320 con 0x7f17c8100340 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17c7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 0x7f17c81a5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17c7fff640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 0x7f17c81a5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45334/0 (socket says 172.21.15.3:45334) 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.950+0000 7f17c7fff640 1 -- 172.21.15.3:0/1614065006 learned_addr learned my addr 172.21.15.3:0/1614065006 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17bffff640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17c8105410 0x7f17c810de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c77fe640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100340 0x7f17c810d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c7fff640 1 -- 172.21.15.3:0/1614065006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100340 msgr2=0x7f17c810d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:50.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c7fff640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100340 0x7f17c810d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c7fff640 1 -- 172.21.15.3:0/1614065006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17c8105410 msgr2=0x7f17c810de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c7fff640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17c8105410 0x7f17c810de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c7fff640 1 -- 172.21.15.3:0/1614065006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c0009900 con 0x7f17c810e340 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c77fe640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100340 0x7f17c810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c7fff640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 0x7f17c81a5a80 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f17c0004af0 tx=0x7f17c0032570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c00328b0 con 0x7f17c810e340 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17c0003e50 con 0x7f17c810e340 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c81a65a0 con 0x7f17c810e340 2024-02-09T23:04:50.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c0004740 con 0x7f17c810e340 2024-02-09T23:04:50.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.951+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c81a6aa0 con 0x7f17c810e340 2024-02-09T23:04:50.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.952+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c80680b0 con 0x7f17c810e340 2024-02-09T23:04:50.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.953+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f17c000a040 con 0x7f17c810e340 2024-02-09T23:04:50.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.954+0000 7f17c57fa640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17b40a22b0 0x7f17b40a4770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:50.961 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.954+0000 7f17c77fe640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17b40a22b0 0x7f17b40a4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:50.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.954+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 8899+0+0 (secure 0 0 0) 0x7f17c00f08f0 con 0x7f17c810e340 2024-02-09T23:04:50.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.955+0000 7f17c77fe640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17b40a22b0 0x7f17b40a4770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009b50 tx=0x7f17b8005f90 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:50.962 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:50.958+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f17c00b94b0 con 0x7f17c810e340 2024-02-09T23:04:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2509280105' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-02-09T23:04:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-02-09T23:04:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-02-09T23:04:50.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2509280105' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-02-09T23:04:50.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: osdmap e107: 12 total, 12 up, 12 in 2024-02-09T23:04:50.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-02-09T23:04:50.978 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:50 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2862681811' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-02-09T23:04:51.144 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.141+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f17c81086a0 con 0x7f17c810e340 2024-02-09T23:04:51.820 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.817+0000 7f17c57fa640 1 -- 172.21.15.3:0/1614065006 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v108) v1 ==== 181+0+0 (secure 0 0 0) 0x7f17c00bcb70 con 0x7f17c810e340 2024-02-09T23:04:51.820 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_metadata' removed 2024-02-09T23:04:51.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.819+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17b40a22b0 msgr2=0x7f17b40a4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:51.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17b40a22b0 0x7f17b40a4770 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f17b8009b50 tx=0x7f17b8005f90 comp rx=0 tx=0).stop 2024-02-09T23:04:51.823 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 msgr2=0x7f17c81a5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 0x7f17c81a5a80 secure :-1 s=READY pgs=1841 cs=0 l=1 rev1=1 crypto rx=0x7f17c0004af0 tx=0x7f17c0032570 comp rx=0 tx=0).stop 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 shutdown_connections 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f17b40a22b0 0x7f17b40a4770 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f17c8105410 0x7f17c810de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f17c810e340 0x7f17c81a5a80 unknown :-1 s=CLOSED pgs=1841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 --2- 172.21.15.3:0/1614065006 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f17c8100340 0x7f17c810d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.824 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 >> 172.21.15.3:0/1614065006 conn(0x7f17c80772e0 msgr2=0x7f17c80fe080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:51.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 shutdown_connections 2024-02-09T23:04:51.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.820+0000 7f17cde35640 1 -- 172.21.15.3:0/1614065006 wait complete. 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: pgmap v429: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: Cluster is now healthy 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/2862681811' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6832/4237854227,v1:172.21.15.64:6835/4237854227] up:boot 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.3:6838/864268316,v1:172.21.15.3:6839/864268316] up:boot 2024-02-09T23:04:51.839 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6835/3152647195,v1:172.21.15.89:6839/3152647195] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.3:6840/3618687587,v1:172.21.15.3:6841/3618687587] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6833/12127441,v1:172.21.15.64:6834/12127441] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6832/2387041801,v1:172.21.15.89:6836/2387041801] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6834/2671775291,v1:172.21.15.89:6838/2671775291] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.89:6833/371429967,v1:172.21.15.89:6837/371429967] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6836/4058690701,v1:172.21.15.64:6838/4058690701] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: mds.? [v2:172.21.15.64:6837/526794098,v1:172.21.15.64:6839/526794098] up:boot 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: fsmap 12 up:standby 2024-02-09T23:04:51.840 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T23:04:51.841 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T23:04:51.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T23:04:51.842 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:51 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1614065006' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T23:04:51.843 DEBUG:teuthology.orchestra.run.smithi003:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-02-09T23:04:51.879 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: pgmap v429: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-02-09T23:04:51.879 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-09T23:04:51.879 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-09T23:04:51.879 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: Cluster is now healthy 2024-02-09T23:04:51.879 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/2862681811' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6832/4237854227,v1:172.21.15.64:6835/4237854227] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.3:6838/864268316,v1:172.21.15.3:6839/864268316] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6835/3152647195,v1:172.21.15.89:6839/3152647195] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.3:6840/3618687587,v1:172.21.15.3:6841/3618687587] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6833/12127441,v1:172.21.15.64:6834/12127441] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6832/2387041801,v1:172.21.15.89:6836/2387041801] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6834/2671775291,v1:172.21.15.89:6838/2671775291] up:boot 2024-02-09T23:04:51.880 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.89:6833/371429967,v1:172.21.15.89:6837/371429967] up:boot 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6836/4058690701,v1:172.21.15.64:6838/4058690701] up:boot 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: mds.? [v2:172.21.15.64:6837/526794098,v1:172.21.15.64:6839/526794098] up:boot 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: fsmap 12 up:standby 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T23:04:51.881 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T23:04:51.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T23:04:51.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T23:04:51.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T23:04:51.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T23:04:51.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T23:04:51.882 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:51 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1614065006' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T23:04:51.935 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: pgmap v429: 97 pgs: 97 active+clean; 207 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 18 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: Cluster is now healthy 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/2862681811' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6832/4237854227,v1:172.21.15.64:6835/4237854227] up:boot 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.3:6838/864268316,v1:172.21.15.3:6839/864268316] up:boot 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6835/3152647195,v1:172.21.15.89:6839/3152647195] up:boot 2024-02-09T23:04:51.936 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.3:6840/3618687587,v1:172.21.15.3:6841/3618687587] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6833/12127441,v1:172.21.15.64:6834/12127441] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6832/2387041801,v1:172.21.15.89:6836/2387041801] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6834/2671775291,v1:172.21.15.89:6838/2671775291] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.89:6833/371429967,v1:172.21.15.89:6837/371429967] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6836/4058690701,v1:172.21.15.64:6838/4058690701] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: mds.? [v2:172.21.15.64:6837/526794098,v1:172.21.15.64:6839/526794098] up:boot 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: fsmap 12 up:standby 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-02-09T23:04:51.937 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-02-09T23:04:51.938 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='mgr.15519 172.21.15.3:0/4069469988' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-02-09T23:04:51.939 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:51 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1614065006' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T23:04:51.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 -- 172.21.15.3:0/138071628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c101440 msgr2=0x7f387c101820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:51.953 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 --2- 172.21.15.3:0/138071628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c101440 0x7f387c101820 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7f3878009930 tx=0x7f387802bc30 comp rx=0 tx=0).stop 2024-02-09T23:04:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 -- 172.21.15.3:0/138071628 shutdown_connections 2024-02-09T23:04:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 --2- 172.21.15.3:0/138071628 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101df0 0x7f387c105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 --2- 172.21.15.3:0/138071628 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f387c106510 0x7f387c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 --2- 172.21.15.3:0/138071628 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c101440 0x7f387c101820 unknown :-1 s=CLOSED pgs=1842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 -- 172.21.15.3:0/138071628 >> 172.21.15.3:0/138071628 conn(0x7f387c0fccc0 msgr2=0x7f387c0ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:51.954 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 -- 172.21.15.3:0/138071628 shutdown_connections 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.948+0000 7f38841d0640 1 -- 172.21.15.3:0/138071628 wait complete. 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 Processor -- start 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 -- start start 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 --2- >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101440 0x7f387c073530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 0x7f387c073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 --2- >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f387c073fb0 0x7f387c1a5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 -- --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c1a6030 con 0x7f387c106510 2024-02-09T23:04:51.955 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 -- --> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c1a61a0 con 0x7f387c101440 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f38841d0640 1 -- --> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387c1a6360 con 0x7f387c073fb0 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f388176b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 0x7f387c073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f388176b640 1 --2- >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 0x7f387c073a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.3:3300/0 says I am v2:172.21.15.3:45348/0 (socket says 172.21.15.3:45348) 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f388176b640 1 -- 172.21.15.3:0/869472221 learned_addr learned my addr 172.21.15.3:0/869472221 (peer_addr_for_me v2:172.21.15.3:0/0) 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f388276d640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f387c073fb0 0x7f387c1a5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f3881f6c640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101440 0x7f387c073530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.949+0000 7f388176b640 1 -- 172.21.15.3:0/869472221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f387c073fb0 msgr2=0x7f387c1a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:51.956 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f388176b640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f387c073fb0 0x7f387c1a5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f388176b640 1 -- 172.21.15.3:0/869472221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101440 msgr2=0x7f387c073530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f388176b640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101440 0x7f387c073530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f388176b640 1 -- 172.21.15.3:0/869472221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3878009560 con 0x7f387c106510 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f3881f6c640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101440 0x7f387c073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f388176b640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 0x7f387c073a70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f386c005fd0 tx=0x7f386c00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c009c30 con 0x7f387c106510 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f386c009d90 con 0x7f387c106510 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f387c1a6640 con 0x7f387c106510 2024-02-09T23:04:51.957 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f386c010460 con 0x7f387c106510 2024-02-09T23:04:51.958 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.950+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f387c1a6b50 con 0x7f387c106510 2024-02-09T23:04:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.951+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f387c0680b0 con 0x7f387c106510 2024-02-09T23:04:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.952+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 4 ==== mgrmap(e 26) v1 ==== 136898+0+0 (secure 0 0 0) 0x7f386c003f20 con 0x7f387c106510 2024-02-09T23:04:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.953+0000 7f3872ffd640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38680a2260 0x7f38680a4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-02-09T23:04:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.953+0000 7f3881f6c640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38680a2260 0x7f38680a4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-02-09T23:04:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.953+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8374+0+0 (secure 0 0 0) 0x7f386c0cdc90 con 0x7f387c106510 2024-02-09T23:04:51.959 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.953+0000 7f3881f6c640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38680a2260 0x7f38680a4720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3878006010 tx=0x7f38780047a0 comp rx=0 tx=0).ready entity=mgr.15519 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-02-09T23:04:51.960 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:51.957+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+183564 (secure 0 0 0) 0x7f386c0969b0 con 0x7f387c106510 2024-02-09T23:04:52.163 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.160+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 --> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f387c109740 con 0x7f387c106510 2024-02-09T23:04:52.825 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.823+0000 7f3872ffd640 1 -- 172.21.15.3:0/869472221 <== mon.0 v2:172.21.15.3:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v109) v1 ==== 169+0+0 (secure 0 0 0) 0x7f386c09a070 con 0x7f387c106510 2024-02-09T23:04:52.826 INFO:teuthology.orchestra.run.smithi003.stderr:pool 'cephfs_data' removed 2024-02-09T23:04:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.826+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38680a2260 msgr2=0x7f38680a4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.826+0000 7f38841d0640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38680a2260 0x7f38680a4720 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f3878006010 tx=0x7f38780047a0 comp rx=0 tx=0).stop 2024-02-09T23:04:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 msgr2=0x7f387c073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-02-09T23:04:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 0x7f387c073a70 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f386c005fd0 tx=0x7f386c00dde0 comp rx=0 tx=0).stop 2024-02-09T23:04:52.830 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 shutdown_connections 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:6800/4119047166,v1:172.21.15.3:6801/4119047166] conn(0x7f38680a2260 0x7f38680a4720 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f387c073fb0 0x7f387c1a5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.3:3300/0,v1:172.21.15.3:6789/0] conn(0x7f387c106510 0x7f387c073a70 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 --2- 172.21.15.3:0/869472221 >> [v2:172.21.15.64:3300/0,v1:172.21.15.64:6789/0] conn(0x7f387c101440 0x7f387c073530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 >> 172.21.15.3:0/869472221 conn(0x7f387c0fccc0 msgr2=0x7f387c0fe000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 shutdown_connections 2024-02-09T23:04:52.831 INFO:teuthology.orchestra.run.smithi003.stderr:2024-02-09T23:04:52.827+0000 7f38841d0640 1 -- 172.21.15.3:0/869472221 wait complete. 2024-02-09T23:04:52.852 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T23:04:52.893 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T23:04:52.929 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T23:04:52.962 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-02-09T23:04:52.962 DEBUG:teuthology.orchestra.run.smithi003:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T23:04:52.981 DEBUG:teuthology.orchestra.run.smithi064:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T23:04:52.996 DEBUG:teuthology.orchestra.run.smithi089:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-02-09T23:04:53.016 INFO:tasks.cephadm:Stopping all daemons... 2024-02-09T23:04:53.017 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-02-09T23:04:53.017 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a 2024-02-09T23:04:53.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:52 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/1614065006' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T23:04:53.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:52 smithi003 ceph-mon[32678]: osdmap e108: 12 total, 12 up, 12 in 2024-02-09T23:04:53.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:52 smithi003 ceph-mon[32678]: from='client.? 172.21.15.3:0/869472221' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T23:04:53.132 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:53 smithi003 systemd[1]: Stopping Ceph mon.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:04:53.157 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:52 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/1614065006' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T23:04:53.157 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:52 smithi089 ceph-mon[36665]: osdmap e108: 12 total, 12 up, 12 in 2024-02-09T23:04:53.157 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:52 smithi089 ceph-mon[36665]: from='client.? 172.21.15.3:0/869472221' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T23:04:53.187 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:52 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/1614065006' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-02-09T23:04:53.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:52 smithi064 ceph-mon[36638]: osdmap e108: 12 total, 12 up, 12 in 2024-02-09T23:04:53.188 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:52 smithi064 ceph-mon[36638]: from='client.? 172.21.15.3:0/869472221' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-02-09T23:04:53.627 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:53 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a[32674]: 2024-02-09T23:04:53.332+0000 7f6c7c63f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:04:53.627 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:53 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a[32674]: 2024-02-09T23:04:53.332+0000 7f6c7c63f700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-02-09T23:04:53.628 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:53 smithi003 podman[108093]: 2024-02-09 23:04:53.363844237 +0000 UTC m=+0.141199762 container died 1d1b447cf16c1daa0cdca5ec400e57a5384f9272d0625114939f13e26b5ef894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-02-09T23:04:53.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:53 smithi003 podman[108093]: 2024-02-09 23:04:53.732870478 +0000 UTC m=+0.510226002 container cleanup 1d1b447cf16c1daa0cdca5ec400e57a5384f9272d0625114939f13e26b5ef894 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a, io.buildah.version=1.33.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0) 2024-02-09T23:04:53.882 INFO:journalctl@ceph.mon.a.smithi003.stdout:Feb 09 23:04:53 smithi003 bash[108093]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-a 2024-02-09T23:04:54.222 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.a.service' 2024-02-09T23:04:54.269 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:04:54.269 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-02-09T23:04:54.270 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-02-09T23:04:54.270 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.b 2024-02-09T23:04:54.599 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:54 smithi064 systemd[1]: Stopping Ceph mon.b for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:04:54.874 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b[36634]: 2024-02-09T23:04:54.595+0000 7f059acbb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:04:54.874 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b[36634]: 2024-02-09T23:04:54.595+0000 7f059acbb700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-02-09T23:04:54.874 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:54 smithi064 podman[74130]: 2024-02-09 23:04:54.67828332 +0000 UTC m=+0.196062690 container died aea8794133553767c48cb5a79324f9325b077a82821c997c5b82c73e1d66f940 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-09T23:04:55.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:54 smithi064 podman[74130]: 2024-02-09 23:04:54.979160128 +0000 UTC m=+0.496939491 container cleanup aea8794133553767c48cb5a79324f9325b077a82821c997c5b82c73e1d66f940 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131) 2024-02-09T23:04:55.185 INFO:journalctl@ceph.mon.b.smithi064.stdout:Feb 09 23:04:54 smithi064 bash[74130]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-b 2024-02-09T23:04:55.330 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.b.service' 2024-02-09T23:04:55.380 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:04:55.381 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-02-09T23:04:55.381 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-02-09T23:04:55.381 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.c 2024-02-09T23:04:55.692 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:55 smithi089 systemd[1]: Stopping Ceph mon.c for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:04:55.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-c[36661]: 2024-02-09T23:04:55.689+0000 7f480fc93700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:04:55.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-c[36661]: 2024-02-09T23:04:55.689+0000 7f480fc93700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-02-09T23:04:55.977 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:55 smithi089 podman[73700]: 2024-02-09 23:04:55.771440233 +0000 UTC m=+0.191167259 container died c1741322896adb8410774b3faf23269c3896d014b14ccca443a81a1642b50d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-02-09T23:04:56.343 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:56 smithi089 podman[73700]: 2024-02-09 23:04:56.115118155 +0000 UTC m=+0.534845181 container cleanup c1741322896adb8410774b3faf23269c3896d014b14ccca443a81a1642b50d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2) 2024-02-09T23:04:56.343 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:56 smithi089 bash[73700]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-c 2024-02-09T23:04:56.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.c.service' 2024-02-09T23:04:56.603 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:56 smithi089 podman[73714]: 2024-02-09 23:04:56.341406617 +0000 UTC m=+0.566293491 container remove c1741322896adb8410774b3faf23269c3896d014b14ccca443a81a1642b50d78 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mon-c, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-09T23:04:56.603 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:56 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.c.service: Deactivated successfully. 2024-02-09T23:04:56.603 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:56 smithi089 systemd[1]: Stopped Ceph mon.c for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T23:04:56.603 INFO:journalctl@ceph.mon.c.smithi089.stdout:Feb 09 23:04:56 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mon.c.service: Consumed 16.233s CPU time. 2024-02-09T23:04:56.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:04:56.625 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-02-09T23:04:56.625 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-02-09T23:04:56.625 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x 2024-02-09T23:04:56.950 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 23:04:56 smithi003 systemd[1]: Stopping Ceph mgr.x for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:04:57.224 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 23:04:56 smithi003 podman[108214]: 2024-02-09 23:04:56.951224169 +0000 UTC m=+0.134247248 container died 0ae12abd5b181113b24a7287707a0239b73ba19a6b6bbc8ae26b568588cee8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-09T23:04:57.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 23:04:57 smithi003 podman[108214]: 2024-02-09 23:04:57.304202307 +0000 UTC m=+0.487225382 container cleanup 0ae12abd5b181113b24a7287707a0239b73ba19a6b6bbc8ae26b568588cee8e5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:290212a1bfd33563ba76ee5b7f0b944fdb6c5d90, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-09T23:04:57.632 INFO:journalctl@ceph.mgr.x.smithi003.stdout:Feb 09 23:04:57 smithi003 bash[108214]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-x 2024-02-09T23:04:57.781 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.x.service' 2024-02-09T23:04:57.818 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:04:57.818 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-02-09T23:04:57.819 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-02-09T23:04:57.819 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.y 2024-02-09T23:04:58.122 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 23:04:57 smithi064 systemd[1]: Stopping Ceph mgr.y for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:04:58.395 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 23:04:58 smithi064 podman[74251]: 2024-02-09 23:04:58.120580692 +0000 UTC m=+0.108455010 container died ea86b2fc3cf50a3df681a27ee124d57c6aa3d85a53849eae84fb3c157b6b6af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True) 2024-02-09T23:04:58.685 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 23:04:58 smithi064 podman[74251]: 2024-02-09 23:04:58.475651858 +0000 UTC m=+0.463526163 container cleanup ea86b2fc3cf50a3df681a27ee124d57c6aa3d85a53849eae84fb3c157b6b6af0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:04:58.686 INFO:journalctl@ceph.mgr.y.smithi064.stdout:Feb 09 23:04:58 smithi064 bash[74251]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-y 2024-02-09T23:04:58.911 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.y.service' 2024-02-09T23:04:58.954 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:04:58.954 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-02-09T23:04:58.954 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-02-09T23:04:58.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.z 2024-02-09T23:04:59.275 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:04:59 smithi089 systemd[1]: Stopping Ceph mgr.z for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:04:59.530 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:04:59 smithi089 podman[73824]: 2024-02-09 23:04:59.273546358 +0000 UTC m=+0.118127069 container died bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-09T23:04:59.789 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:04:59 smithi089 podman[73824]: 2024-02-09 23:04:59.602384962 +0000 UTC m=+0.446965670 container cleanup bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-09T23:04:59.789 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:04:59 smithi089 bash[73824]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z 2024-02-09T23:05:00.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.z.service' 2024-02-09T23:05:00.042 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:04:59 smithi089 podman[73839]: 2024-02-09 23:04:59.787074221 +0000 UTC m=+0.507219049 container remove bfdaf36675bf28a8e1c4c4b3999eca5a20efe4ccc761e38052d3e5e47bc5b76f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-09T23:05:00.042 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:04:59 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-02-09T23:05:00.043 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:05:00 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.z.service: Failed with result 'exit-code'. 2024-02-09T23:05:00.043 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:05:00 smithi089 systemd[1]: Stopped Ceph mgr.z for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T23:05:00.043 INFO:journalctl@ceph.mgr.z.smithi089.stdout:Feb 09 23:05:00 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mgr.z.service: Consumed 20.414s CPU time. 2024-02-09T23:05:00.055 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:00.056 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-02-09T23:05:00.056 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-02-09T23:05:00.056 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.0 2024-02-09T23:05:00.353 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:00 smithi003 systemd[1]: Stopping Ceph osd.0 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:00.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0[46960]: 2024-02-09T23:05:00.350+0000 7f2d0d13d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:00.632 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0[46960]: 2024-02-09T23:05:00.350+0000 7f2d0d13d700 -1 osd.0 109 *** Got signal Terminated *** 2024-02-09T23:05:00.633 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0[46960]: 2024-02-09T23:05:00.350+0000 7f2d0d13d700 -1 osd.0 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:06.054 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:05 smithi003 podman[108335]: 2024-02-09 23:05:05.761163256 +0000 UTC m=+5.514806396 container died 4e68f37a564fb4b86d572dd16ea5b0810ba3afeb31ae94b6073fff94ec079e9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-02-09T23:05:06.381 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:06 smithi003 podman[108335]: 2024-02-09 23:05:06.133849788 +0000 UTC m=+5.887492926 container cleanup 4e68f37a564fb4b86d572dd16ea5b0810ba3afeb31ae94b6073fff94ec079e9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-09T23:05:06.382 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:06 smithi003 bash[108335]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0 2024-02-09T23:05:06.797 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:06 smithi003 podman[108359]: 2024-02-09 23:05:06.452092019 +0000 UTC m=+0.684620378 container remove 4e68f37a564fb4b86d572dd16ea5b0810ba3afeb31ae94b6073fff94ec079e9f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-09T23:05:07.124 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:06 smithi003 podman[108451]: 2024-02-09 23:05:06.794457311 +0000 UTC m=+0.105897247 container create 15e364248c78d76f4eaeb001ccdf70d467fde1b6c69c09a306c0dcf4fa0fbec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-09T23:05:07.124 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:06 smithi003 podman[108451]: 2024-02-09 23:05:06.70565435 +0000 UTC m=+0.017094270 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:07.124 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:07 smithi003 podman[108451]: 2024-02-09 23:05:07.02249168 +0000 UTC m=+0.333931588 container init 15e364248c78d76f4eaeb001ccdf70d467fde1b6c69c09a306c0dcf4fa0fbec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-02-09T23:05:07.124 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:07 smithi003 podman[108451]: 2024-02-09 23:05:07.025877293 +0000 UTC m=+0.337317190 container start 15e364248c78d76f4eaeb001ccdf70d467fde1b6c69c09a306c0dcf4fa0fbec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-09T23:05:07.124 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:07 smithi003 podman[108451]: 2024-02-09 23:05:07.121721815 +0000 UTC m=+0.433161718 container attach 15e364248c78d76f4eaeb001ccdf70d467fde1b6c69c09a306c0dcf4fa0fbec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-02-09T23:05:07.382 INFO:journalctl@ceph.osd.0.smithi003.stdout:Feb 09 23:05:07 smithi003 podman[108451]: 2024-02-09 23:05:07.326240369 +0000 UTC m=+0.637680263 container died 15e364248c78d76f4eaeb001ccdf70d467fde1b6c69c09a306c0dcf4fa0fbec6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-0-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a) 2024-02-09T23:05:07.976 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.0.service' 2024-02-09T23:05:08.021 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:08.021 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-02-09T23:05:08.022 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-02-09T23:05:08.022 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.1 2024-02-09T23:05:08.324 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:08 smithi064 systemd[1]: Stopping Ceph osd.1 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:08.685 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1[41263]: 2024-02-09T23:05:08.320+0000 7f0731515700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:08.685 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1[41263]: 2024-02-09T23:05:08.320+0000 7f0731515700 -1 osd.1 109 *** Got signal Terminated *** 2024-02-09T23:05:08.686 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1[41263]: 2024-02-09T23:05:08.320+0000 7f0731515700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:14.034 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:13 smithi064 podman[74385]: 2024-02-09 23:05:13.747578306 +0000 UTC m=+5.532987196 container died 28c3e92bda39fec4e300af422ad11ea66df9e76f18dbd09fbee6338b7abbfc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-09T23:05:14.375 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 podman[74385]: 2024-02-09 23:05:14.113006016 +0000 UTC m=+5.898414905 container cleanup 28c3e92bda39fec4e300af422ad11ea66df9e76f18dbd09fbee6338b7abbfc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-02-09T23:05:14.375 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 bash[74385]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1 2024-02-09T23:05:14.685 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 podman[74403]: 2024-02-09 23:05:14.37297325 +0000 UTC m=+0.617566772 container remove 28c3e92bda39fec4e300af422ad11ea66df9e76f18dbd09fbee6338b7abbfc9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True) 2024-02-09T23:05:15.003 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 podman[74495]: 2024-02-09 23:05:14.638027351 +0000 UTC m=+0.018286968 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:15.003 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 podman[74495]: 2024-02-09 23:05:14.740393295 +0000 UTC m=+0.120652915 container create 0bdc2fd0143c84bb791a1fce8ad0555f1e66c59e6769e4663cc749ea815614e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:15.003 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 podman[74495]: 2024-02-09 23:05:14.917471926 +0000 UTC m=+0.297731541 container init 0bdc2fd0143c84bb791a1fce8ad0555f1e66c59e6769e4663cc749ea815614e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-09T23:05:15.003 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:14 smithi064 podman[74495]: 2024-02-09 23:05:14.92076398 +0000 UTC m=+0.301023573 container start 0bdc2fd0143c84bb791a1fce8ad0555f1e66c59e6769e4663cc749ea815614e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a) 2024-02-09T23:05:15.003 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:15 smithi064 podman[74495]: 2024-02-09 23:05:15.000688506 +0000 UTC m=+0.380948118 container attach 0bdc2fd0143c84bb791a1fce8ad0555f1e66c59e6769e4663cc749ea815614e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, ceph=True) 2024-02-09T23:05:15.435 INFO:journalctl@ceph.osd.1.smithi064.stdout:Feb 09 23:05:15 smithi064 podman[74495]: 2024-02-09 23:05:15.226933702 +0000 UTC m=+0.607193313 container died 0bdc2fd0143c84bb791a1fce8ad0555f1e66c59e6769e4663cc749ea815614e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-1-deactivate, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T23:05:15.949 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.1.service' 2024-02-09T23:05:15.997 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:15.997 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-02-09T23:05:15.997 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-02-09T23:05:15.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.2 2024-02-09T23:05:16.284 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:16 smithi089 systemd[1]: Stopping Ceph osd.2 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:16.727 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2[41795]: 2024-02-09T23:05:16.280+0000 7fb2307a6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:16.727 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2[41795]: 2024-02-09T23:05:16.280+0000 7fb2307a6700 -1 osd.2 109 *** Got signal Terminated *** 2024-02-09T23:05:16.727 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2[41795]: 2024-02-09T23:05:16.280+0000 7fb2307a6700 -1 osd.2 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:21.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:21 smithi089 podman[73948]: 2024-02-09 23:05:21.724486895 +0000 UTC m=+5.542559327 container died 6360825648cbdc3ae2c251cbb126722f580c3a2e5709b5de8e70f06d0fd1e60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:05:22.276 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 podman[73948]: 2024-02-09 23:05:22.072776659 +0000 UTC m=+5.890849082 container cleanup 6360825648cbdc3ae2c251cbb126722f580c3a2e5709b5de8e70f06d0fd1e60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=) 2024-02-09T23:05:22.277 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 bash[73948]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2 2024-02-09T23:05:22.684 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 podman[73966]: 2024-02-09 23:05:22.273888355 +0000 UTC m=+0.542524740 container remove 6360825648cbdc3ae2c251cbb126722f580c3a2e5709b5de8e70f06d0fd1e60b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-02-09T23:05:22.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 podman[74056]: 2024-02-09 23:05:22.585218361 +0000 UTC m=+0.017501383 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:22.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 podman[74056]: 2024-02-09 23:05:22.699796195 +0000 UTC m=+0.132079220 container create 0d5753ade46735d689c68b123a6c0ac7bce6bbda488aadcb3e59bd7093baf260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, ceph=True) 2024-02-09T23:05:22.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 podman[74056]: 2024-02-09 23:05:22.927405038 +0000 UTC m=+0.359688069 container init 0d5753ade46735d689c68b123a6c0ac7bce6bbda488aadcb3e59bd7093baf260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T23:05:22.977 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:22 smithi089 podman[74056]: 2024-02-09 23:05:22.930532017 +0000 UTC m=+0.362815035 container start 0d5753ade46735d689c68b123a6c0ac7bce6bbda488aadcb3e59bd7093baf260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-02-09T23:05:23.231 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:23 smithi089 podman[74056]: 2024-02-09 23:05:23.010757063 +0000 UTC m=+0.443040093 container attach 0d5753ade46735d689c68b123a6c0ac7bce6bbda488aadcb3e59bd7093baf260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:05:23.231 INFO:journalctl@ceph.osd.2.smithi089.stdout:Feb 09 23:05:23 smithi089 podman[74056]: 2024-02-09 23:05:23.219047623 +0000 UTC m=+0.651330640 container died 0d5753ade46735d689c68b123a6c0ac7bce6bbda488aadcb3e59bd7093baf260 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.2) 2024-02-09T23:05:23.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.2.service' 2024-02-09T23:05:23.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:23.868 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-02-09T23:05:23.868 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-02-09T23:05:23.869 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.3 2024-02-09T23:05:24.180 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:23 smithi003 systemd[1]: Stopping Ceph osd.3 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:24.631 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:24 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T23:05:24.176+0000 7fce370e9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:24.632 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:24 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T23:05:24.176+0000 7fce370e9700 -1 osd.3 109 *** Got signal Terminated *** 2024-02-09T23:05:24.632 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:24 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T23:05:24.176+0000 7fce370e9700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:26.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:26 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:26.662+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:27.881 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:27 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T23:05:27.458+0000 7fce2fed6700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:00.095398+0000 front 2024-02-09T23:05:00.095353+0000 (oldest deadline 2024-02-09T23:05:26.496319+0000) 2024-02-09T23:05:27.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:27 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:27.675+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:28.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:27.767+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:28.816 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:28 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3[53925]: 2024-02-09T23:05:28.499+0000 7fce2fed6700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:00.095398+0000 front 2024-02-09T23:05:00.095353+0000 (oldest deadline 2024-02-09T23:05:26.496319+0000) 2024-02-09T23:05:29.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:29 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:29.006+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:29.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:28 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:28.814+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:29.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:28 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:28.718+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:29.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:28.774+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:29.882 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:29 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:29.787+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:29.882 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:29 smithi003 podman[108561]: 2024-02-09 23:05:29.628755994 +0000 UTC m=+5.567028853 container died 7b8ce729ef9ae4d57bed7c3b0149c801b6079fb806d84eb930e199012686306f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240131) 2024-02-09T23:05:30.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:29 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:29.735+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:30.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:29.745+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:30.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:29.841+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:30.272 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:29 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:29.983+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:30.272 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:29 smithi003 podman[108561]: 2024-02-09 23:05:29.98519154 +0000 UTC m=+5.923464399 container cleanup 7b8ce729ef9ae4d57bed7c3b0149c801b6079fb806d84eb930e199012686306f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-02-09T23:05:30.272 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:29 smithi003 bash[108561]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3 2024-02-09T23:05:30.604 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:30 smithi003 podman[108578]: 2024-02-09 23:05:30.269748746 +0000 UTC m=+0.643057590 container remove 7b8ce729ef9ae4d57bed7c3b0149c801b6079fb806d84eb930e199012686306f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-09T23:05:30.876 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:30 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:30.768+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:30.876 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:30 smithi003 podman[108668]: 2024-02-09 23:05:30.505332943 +0000 UTC m=+0.017466842 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:30.876 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:30 smithi003 podman[108668]: 2024-02-09 23:05:30.645903991 +0000 UTC m=+0.158037915 container create 8b8ae538bb3e87a2265c005f401cc5f2ea5b30df56ac64ce2c3b5aeb19732a70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:31.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:30 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:30.960+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:31.132 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:30 smithi003 podman[108668]: 2024-02-09 23:05:30.87385922 +0000 UTC m=+0.385993108 container init 8b8ae538bb3e87a2265c005f401cc5f2ea5b30df56ac64ce2c3b5aeb19732a70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-02-09T23:05:31.132 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:30 smithi003 podman[108668]: 2024-02-09 23:05:30.877835649 +0000 UTC m=+0.389969529 container start 8b8ae538bb3e87a2265c005f401cc5f2ea5b30df56ac64ce2c3b5aeb19732a70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T23:05:31.132 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:31 smithi003 podman[108668]: 2024-02-09 23:05:31.007202198 +0000 UTC m=+0.519336078 container attach 8b8ae538bb3e87a2265c005f401cc5f2ea5b30df56ac64ce2c3b5aeb19732a70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-02-09T23:05:31.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:30 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:30.743+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:31.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:30.838+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:31.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:30.751+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:31.471 INFO:journalctl@ceph.osd.3.smithi003.stdout:Feb 09 23:05:31 smithi003 podman[108668]: 2024-02-09 23:05:31.182766667 +0000 UTC m=+0.694900546 container died 8b8ae538bb3e87a2265c005f401cc5f2ea5b30df56ac64ce2c3b5aeb19732a70 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-3-deactivate, RELEASE=main-290212a, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:31.771 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.3.service' 2024-02-09T23:05:31.813 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:31.813 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-02-09T23:05:31.813 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-02-09T23:05:31.814 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.4 2024-02-09T23:05:32.127 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:31 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:31.710+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:32.128 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:31 smithi064 systemd[1]: Stopping Ceph osd.4 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:32.131 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:31 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:31.816+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:32.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:31 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:31.917+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:32.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:31.794+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:32.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:31.788+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:32.435 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T23:05:32.125+0000 7f2fb664d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:32.435 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T23:05:32.125+0000 7f2fb664d700 -1 osd.4 109 *** Got signal Terminated *** 2024-02-09T23:05:32.436 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T23:05:32.125+0000 7f2fb664d700 -1 osd.4 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:33.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:32 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:32.791+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:33.133 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:32 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:32.954+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:33.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:32 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:32.692+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:33.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:32 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:32.787+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:33.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:32 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:32.809+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:34.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:33.769+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:34.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:33.975+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:34.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:33 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:33.975+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:34.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:33 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:33.693+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:34.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:33 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:33.826+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:34.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:33 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:33.806+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:34.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:34 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:34.678+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:35.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:34.813+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:35.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:34.953+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:35.133 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:34 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:34.953+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:35.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:34 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:34.793+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:35.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:34 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:34.840+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:35.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:34 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:34.840+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:35.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:35 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:35.653+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:36.131 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:35.999+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:36.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:35.999+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:36.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:35.779+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:36.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:35.779+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:36.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:35 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:35.762+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:36.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:35 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:35.762+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:36.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:35 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:35.881+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:36.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:35 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:35.881+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:36.435 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:36 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T23:05:36.046+0000 7f2faf43a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.294217+0000 front 2024-02-09T23:05:11.294241+0000 (oldest deadline 2024-02-09T23:05:35.394423+0000) 2024-02-09T23:05:36.772 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:36 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:36.511+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:36.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:36 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:36.682+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:37.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:37 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:37.013+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:37.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:37 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:37.013+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:37.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:36.819+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:37.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:36 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:36.819+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:37.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:36 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:36.769+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:37.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:36 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:36.769+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:37.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:36 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:36.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:37.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:36 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:36.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:37.435 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:37 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4[47804]: 2024-02-09T23:05:37.072+0000 7f2faf43a700 -1 osd.4 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.294217+0000 front 2024-02-09T23:05:11.294241+0000 (oldest deadline 2024-02-09T23:05:35.394423+0000) 2024-02-09T23:05:37.703 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:37 smithi064 podman[74601]: 2024-02-09 23:05:37.553212515 +0000 UTC m=+5.538075456 container died ddec16151a008a7ebf2b4986d6f7991a9174211609d1f0632c181bfe4aa62c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-02-09T23:05:37.761 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:37 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:37.503+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:38.131 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:38.053+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:38.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:38.053+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:38.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:37 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:37.805+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:38.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:37 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:37.805+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:38.170 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:37 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:37.748+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:38.171 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:37 smithi064 podman[74601]: 2024-02-09 23:05:37.925216627 +0000 UTC m=+5.910079571 container cleanup ddec16151a008a7ebf2b4986d6f7991a9174211609d1f0632c181bfe4aa62c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:38.171 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:37 smithi064 bash[74601]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4 2024-02-09T23:05:38.171 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:37 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:37.699+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:38.171 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:37 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:37.699+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:38.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:37 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:37.758+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:38.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:37 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:37.758+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:38.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:37 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:37.884+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:38.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:37 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:37.884+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:38.436 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:38 smithi064 podman[74619]: 2024-02-09 23:05:38.1679333 +0000 UTC m=+0.616543005 container remove ddec16151a008a7ebf2b4986d6f7991a9174211609d1f0632c181bfe4aa62c2b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-02-09T23:05:38.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:38 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:38.462+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:38.763 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:38 smithi064 podman[74723]: 2024-02-09 23:05:38.414148567 +0000 UTC m=+0.018527921 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:38.764 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:38 smithi064 podman[74723]: 2024-02-09 23:05:38.518766355 +0000 UTC m=+0.123145700 container create 71713d39b02e0c3c757467b7c35f8fafbc1a8f1ddf97cb755fad585118ec4b7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4-deactivate, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-02-09T23:05:38.764 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:38 smithi064 podman[74723]: 2024-02-09 23:05:38.720909171 +0000 UTC m=+0.325288510 container init 71713d39b02e0c3c757467b7c35f8fafbc1a8f1ddf97cb755fad585118ec4b7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T23:05:38.764 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:38 smithi064 podman[74723]: 2024-02-09 23:05:38.724160837 +0000 UTC m=+0.328540177 container start 71713d39b02e0c3c757467b7c35f8fafbc1a8f1ddf97cb755fad585118ec4b7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-09T23:05:38.764 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:38 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:38.712+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:38.764 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:38 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:38.712+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:39.030 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:38 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:38.760+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:39.030 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:38 smithi064 podman[74723]: 2024-02-09 23:05:38.796045426 +0000 UTC m=+0.400424783 container attach 71713d39b02e0c3c757467b7c35f8fafbc1a8f1ddf97cb755fad585118ec4b7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:05:39.065 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:38.763+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:39.066 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:38 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:38.763+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:39.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:38 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:38.753+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:39.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:38 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:38.753+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:39.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:38 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:38.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:39.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:38 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:38.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:39.309 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:39 smithi064 podman[74723]: 2024-02-09 23:05:39.028100222 +0000 UTC m=+0.632479568 container died 71713d39b02e0c3c757467b7c35f8fafbc1a8f1ddf97cb755fad585118ec4b7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.2) 2024-02-09T23:05:39.381 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:39.062+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:39.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:39.062+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:39.559 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.4.service' 2024-02-09T23:05:39.591 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:39 smithi064 podman[74750]: 2024-02-09 23:05:39.541297212 +0000 UTC m=+0.504628642 container remove 71713d39b02e0c3c757467b7c35f8fafbc1a8f1ddf97cb755fad585118ec4b7b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:39.592 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:39 smithi064 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.4.service: Deactivated successfully. 2024-02-09T23:05:39.592 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:39 smithi064 systemd[1]: Stopped Ceph osd.4 for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T23:05:39.592 INFO:journalctl@ceph.osd.4.smithi064.stdout:Feb 09 23:05:39 smithi064 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.4.service: Consumed 35.598s CPU time. 2024-02-09T23:05:39.610 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:39.610 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-02-09T23:05:39.611 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-02-09T23:05:39.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.5 2024-02-09T23:05:39.727 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:39 smithi089 systemd[1]: Stopping Ceph osd.5 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:39.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:39.448+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:39.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:39 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:39.780+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:39.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:39 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:39.694+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:39.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:39 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:39.694+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:40.082 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:39.790+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:40.082 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:39.790+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:40.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:39.888+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:40.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:39.888+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:40.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:39.789+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:40.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:39.789+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:40.228 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:39.907+0000 7f34a6f50700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:40.228 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:39.907+0000 7f34a6f50700 -1 osd.5 109 *** Got signal Terminated *** 2024-02-09T23:05:40.228 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:39 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:39.907+0000 7f34a6f50700 -1 osd.5 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:40.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:40 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:40.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:40.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:40 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:40.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:40.772 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:40 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:40.498+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:41.094 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:40 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:40.782+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:41.094 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:40 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:40.782+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:41.185 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:40 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:40.782+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:41.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:40 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:40.682+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:41.186 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:40 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:40.682+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:41.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:40 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:40.770+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:41.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:40 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:40.770+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:41.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:40 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:40.914+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:41.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:40 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:40.914+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:41.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:41 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:41.092+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:41.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:41 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:41.092+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:41.815 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:41 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:41.519+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:41.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:41 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:41.779+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:41.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:41 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:41.634+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:41.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:41 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:41.634+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:42.131 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:42 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:42.047+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:42.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:42 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:42.047+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:42.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:41 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:41.824+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:42.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:41 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:41.824+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:42.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:41 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:41.813+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:42.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:41 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:41.813+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:42.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:41 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:41.886+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:42.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:41 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:41.886+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:42.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:42 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:42.471+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:42.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:42 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:42.825+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:42.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:42 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:42.591+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:42.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:42 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:42.591+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:43.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:43.049+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:43.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:43.049+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:43.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:42 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:42.817+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:43.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:42 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:42.817+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:43.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:42 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:42.799+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:43.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:42 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:42.799+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:43.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:42 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:42.934+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:43.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:42 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:42.934+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:43.821 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:43 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:43.497+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:43.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:43 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:43.786+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:43.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:43 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:43.786+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:43.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:43 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:43.555+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:43.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:43 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:43.555+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:44.131 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:44.041+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:44.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:44.041+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:44.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:43.846+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:44.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:43.846+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:44.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:43 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:43.846+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:44.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:43 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:43.818+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:44.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:43 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:43.818+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:44.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:43 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:43.900+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:44.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:43 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:43.900+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:44.820 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:44 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:44.483+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:44.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:44 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:44.821+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:44.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:44 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:44.821+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:44.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:44 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:44.575+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:44.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:44 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:44.575+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:44.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:44 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:44.575+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:45.131 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:45 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:45.034+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:45.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:45 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:45.034+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:45.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:44.811+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:45.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:44.811+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:45.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:44.811+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:45.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:44 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:44.818+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.460153+0000 front 2024-02-09T23:05:04.460003+0000 (oldest deadline 2024-02-09T23:05:29.160021+0000) 2024-02-09T23:05:45.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:44 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5[48804]: 2024-02-09T23:05:44.818+0000 7f349fd3d700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:10.861596+0000 front 2024-02-09T23:05:10.861504+0000 (oldest deadline 2024-02-09T23:05:35.561485+0000) 2024-02-09T23:05:45.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:44 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:44.873+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:45.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:44 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:44.873+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:45.536 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:45 smithi089 podman[74176]: 2024-02-09 23:05:45.280782034 +0000 UTC m=+5.480393743 container died f98239c4bbf762f49af42de75ee8084c6c60b0a2864eb3b8695be45502e76fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-09T23:05:45.536 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:45 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:45.493+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:45.836 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:45 smithi089 podman[74176]: 2024-02-09 23:05:45.625030532 +0000 UTC m=+5.824642200 container cleanup f98239c4bbf762f49af42de75ee8084c6c60b0a2864eb3b8695be45502e76fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-09T23:05:45.837 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:45 smithi089 bash[74176]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5 2024-02-09T23:05:45.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:45 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:45.774+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:45.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:45 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:45.774+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:45.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:45 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:45.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:45.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:45 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:45.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:45.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:45 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:45.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:46.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:46 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:46.024+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:46.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:46 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:46.024+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:46.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:46 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:46.024+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:46.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:45 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:45.825+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:46.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:45 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:45.825+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:46.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:45 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:45.825+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:46.227 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:45 smithi089 podman[74194]: 2024-02-09 23:05:45.834379338 +0000 UTC m=+0.551338853 container remove f98239c4bbf762f49af42de75ee8084c6c60b0a2864eb3b8695be45502e76fb1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-02-09T23:05:46.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:45 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:45.836+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:46.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:45 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:45.836+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:46.498 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:46 smithi089 podman[74281]: 2024-02-09 23:05:46.131410428 +0000 UTC m=+0.017317810 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:46.498 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:46 smithi089 podman[74281]: 2024-02-09 23:05:46.327817787 +0000 UTC m=+0.213725156 container create 5358572bc7ae32be6e6f13a99dc478941d9087e55ce45046f058b22803aa31f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-02-09T23:05:46.498 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:46 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:46.471+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:46.499 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:46 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:46.471+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:46.856 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:46 smithi089 podman[74281]: 2024-02-09 23:05:46.604388049 +0000 UTC m=+0.490295428 container init 5358572bc7ae32be6e6f13a99dc478941d9087e55ce45046f058b22803aa31f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:46.856 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:46 smithi089 podman[74281]: 2024-02-09 23:05:46.60802571 +0000 UTC m=+0.493933079 container start 5358572bc7ae32be6e6f13a99dc478941d9087e55ce45046f058b22803aa31f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-09T23:05:46.857 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:46 smithi089 podman[74281]: 2024-02-09 23:05:46.679093085 +0000 UTC m=+0.565000473 container attach 5358572bc7ae32be6e6f13a99dc478941d9087e55ce45046f058b22803aa31f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5-deactivate, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-09T23:05:46.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:46 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:46.761+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:46.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:46 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:46.761+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:46.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:46 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:46.530+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:46.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:46 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:46.530+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:46.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:46 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:46.530+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:47.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:47.063+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:47.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:47.063+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:47.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:47.063+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:47.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:46 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:46.859+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:47.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:46 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:46.859+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:47.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:46 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:46.859+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:47.201 INFO:journalctl@ceph.osd.5.smithi089.stdout:Feb 09 23:05:46 smithi089 podman[74281]: 2024-02-09 23:05:46.898262899 +0000 UTC m=+0.784170268 container died 5358572bc7ae32be6e6f13a99dc478941d9087e55ce45046f058b22803aa31f5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-5-deactivate, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True) 2024-02-09T23:05:47.201 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:46 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:46.852+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:47.201 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:46 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:46.852+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:47.202 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:46 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:46.852+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:47.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.5.service' 2024-02-09T23:05:47.531 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:47.531 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-02-09T23:05:47.532 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-02-09T23:05:47.532 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.6 2024-02-09T23:05:47.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:47 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:47.463+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:47.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:47 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:47.463+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:47.865 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 systemd[1]: Stopping Ceph osd.6 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:47.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:47 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:47.721+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:47.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:47 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:47.721+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:47.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:47 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:47.502+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:47.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:47 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:47.502+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:47.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:47 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:47.502+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:48.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:48 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:48.095+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:48.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:48 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:48.095+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:48.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:48 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:48.095+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:48.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:47.861+0000 7f2ca1d08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:48.132 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:47.861+0000 7f2ca1d08700 -1 osd.6 109 *** Got signal Terminated *** 2024-02-09T23:05:48.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:47.861+0000 7f2ca1d08700 -1 osd.6 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:48.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:47.883+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:48.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:47.883+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:48.133 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:47.883+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:48.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:47 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:47.820+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:48.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:47 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:47.820+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:48.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:47 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:47.820+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:48.802 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:48 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:48.694+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:48.802 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:48 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:48.694+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:48.802 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:48 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:48.536+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:48.803 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:48 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:48.536+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:48.803 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:48 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:48.536+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:48.833 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:48 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:48.492+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:48.833 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:48 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:48.492+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:49.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:48 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:48.830+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:49.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:48 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:48.830+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:49.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:48 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:48.830+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:49.382 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:48 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:48.920+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:49.382 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:48 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:48.920+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:49.382 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:48 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:48.920+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:49.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:49.140+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:49.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:49.140+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:49.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:49.140+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:49.864 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:49.542+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:49.865 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:49.542+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:49.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:49.725+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:49.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:49.725+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:49.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:49.532+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:49.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:49.532+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:49.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:49.532+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:50.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:49.861+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:50.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:49.861+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:50.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:49 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:49.861+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:50.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:50.104+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:50.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:50.104+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:50.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:50.104+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:50.382 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:49.939+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:50.382 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:49.939+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:50.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:49 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:49.939+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:50.857 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:50 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:50.562+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:50.858 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:50 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:50.562+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:50.858 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:50 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:50.562+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:50.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:50.743+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:50.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:50.743+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:50.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:50.512+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:50.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:50.512+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:50.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:50.512+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:50.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:50 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:50.512+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:51.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:50 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:50.854+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:51.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:50 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:50.854+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:51.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:50 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:50.854+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:51.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:51.082+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:51.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:51.082+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:51.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:51.082+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:51.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:50.936+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:51.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:50.936+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:51.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:50.936+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:51.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:50 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:50.936+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.640387+0000 front 2024-02-09T23:05:27.640368+0000 (oldest deadline 2024-02-09T23:05:50.540572+0000) 2024-02-09T23:05:51.789 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:51 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:51.504+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:51.790 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:51 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:51.504+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:51.790 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:51 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:51.504+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:51.790 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:51 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:51.504+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:51.874 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:51.599+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:51.875 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:51.599+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:51.875 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:51.599+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:52.142 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:51.871+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:52.143 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:51.871+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:52.143 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:51.871+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:52.143 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:51 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:51.871+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:52.185 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:51 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:51.786+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:52.185 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:51 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:51.786+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:52.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:52 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:52.128+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:52.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:52 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:52.128+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:52.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:52 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:52.128+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:52.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:52 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:52.128+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:52.382 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:51.932+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:02.735317+0000 front 2024-02-09T23:05:02.735334+0000 (oldest deadline 2024-02-09T23:05:28.635474+0000) 2024-02-09T23:05:52.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:51.932+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:11.536980+0000 front 2024-02-09T23:05:11.536931+0000 (oldest deadline 2024-02-09T23:05:35.636693+0000) 2024-02-09T23:05:52.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:51.932+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.337891+0000 front 2024-02-09T23:05:20.337986+0000 (oldest deadline 2024-02-09T23:05:43.238006+0000) 2024-02-09T23:05:52.383 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:51 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6[60502]: 2024-02-09T23:05:51.932+0000 7f2c9eafd700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.640387+0000 front 2024-02-09T23:05:27.640368+0000 (oldest deadline 2024-02-09T23:05:50.540572+0000) 2024-02-09T23:05:52.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:52.789+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:52.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:52.789+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:52.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:52.789+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:52.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:52.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:52.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:52.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:52.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:52.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:52.938 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:52 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:52.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:52.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:52.626+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:52.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:52.626+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:52.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:52.626+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:52.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:52.874+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:52.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:52.874+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:52.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:52.874+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:52.979 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:52 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:52.874+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:53.423 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:53 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:53.142+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:53.424 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:53 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:53.142+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:53.424 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:53 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:53.142+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:53.424 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:53 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:53.142+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:53.691 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:53 smithi003 podman[108784]: 2024-02-09 23:05:53.421347204 +0000 UTC m=+5.690437289 container died 0f90e2b1d7e3b8a82f9ee2f766eaeb03460adfc700c791cb67d09d8ff6930807 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-02-09T23:05:53.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:53.743+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:53.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:53.743+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:53.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:53.743+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:53.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:53.507+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:53.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:53.507+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:53.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:53.507+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:53.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:53 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:53.507+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:53.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:53.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:53.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:53.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:53.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:53.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:53.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:53.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:53.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:53.610+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:53.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:53.610+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:53.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:53 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:53.610+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:54.066 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:53 smithi003 podman[108784]: 2024-02-09 23:05:53.787566016 +0000 UTC m=+6.056656092 container cleanup 0f90e2b1d7e3b8a82f9ee2f766eaeb03460adfc700c791cb67d09d8ff6930807 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-09T23:05:54.066 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:53 smithi003 bash[108784]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6 2024-02-09T23:05:54.381 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:54 smithi003 podman[108801]: 2024-02-09 23:05:54.063867231 +0000 UTC m=+0.642155579 container remove 0f90e2b1d7e3b8a82f9ee2f766eaeb03460adfc700c791cb67d09d8ff6930807 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-09T23:05:54.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:54 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:54.132+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:54.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:54 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:54.132+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:54.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:54 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:54.133+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:54.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:54 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:54.133+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:54.660 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:54 smithi003 podman[108892]: 2024-02-09 23:05:54.306540109 +0000 UTC m=+0.018474201 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:05:54.660 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:54 smithi003 podman[108892]: 2024-02-09 23:05:54.538968642 +0000 UTC m=+0.250902724 container create afe294137c5f9deaf617b5c5ac15294af50264e4b8065e11058969bbb952f88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-09T23:05:54.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:54.738+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:54.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:54.738+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:54.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:54.738+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:54.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:54.547+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:54.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:54.547+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:54.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:54.547+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:54.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:54 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:54.547+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:54.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:54.584+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:54.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:54.584+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:54.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:54.584+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:54.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:54.814+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:54.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:54.814+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:54.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:54.814+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:54.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:54 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:54.814+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:55.064 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:54 smithi003 podman[108892]: 2024-02-09 23:05:54.75858719 +0000 UTC m=+0.470521273 container init afe294137c5f9deaf617b5c5ac15294af50264e4b8065e11058969bbb952f88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:05:55.064 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:54 smithi003 podman[108892]: 2024-02-09 23:05:54.761661667 +0000 UTC m=+0.473595751 container start afe294137c5f9deaf617b5c5ac15294af50264e4b8065e11058969bbb952f88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T23:05:55.064 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:54 smithi003 podman[108892]: 2024-02-09 23:05:54.833439712 +0000 UTC m=+0.545373803 container attach afe294137c5f9deaf617b5c5ac15294af50264e4b8065e11058969bbb952f88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-09T23:05:55.339 INFO:journalctl@ceph.osd.6.smithi003.stdout:Feb 09 23:05:55 smithi003 podman[108892]: 2024-02-09 23:05:55.06228515 +0000 UTC m=+0.774219243 container died afe294137c5f9deaf617b5c5ac15294af50264e4b8065e11058969bbb952f88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-6-deactivate, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD) 2024-02-09T23:05:55.339 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:55 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:55.159+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:55.339 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:55 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:55.159+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:55.339 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:55 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:55.159+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:55.339 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:55 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:55.159+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:55.639 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.6.service' 2024-02-09T23:05:55.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:05:55.682 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-02-09T23:05:55.682 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-02-09T23:05:55.682 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.7 2024-02-09T23:05:55.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:55.718+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:55.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:55.718+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:55.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:55.718+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:55.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 systemd[1]: Stopping Ceph osd.7 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:05:55.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:55.514+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:55.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:55.514+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:55.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:55.514+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:55.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:55.514+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:55.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:55.589+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:55.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:55.589+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:55.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:55.589+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:55.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:55.809+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:55.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:55.809+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:55.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:55.810+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:55.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:55 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:55.810+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:56.435 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:55.993+0000 7f9c4b622700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:05:56.435 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:55.993+0000 7f9c4b622700 -1 osd.7 109 *** Got signal Terminated *** 2024-02-09T23:05:56.435 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:55.993+0000 7f9c4b622700 -1 osd.7 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:05:56.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:56 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:56.178+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:56.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:56 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:56.178+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:56.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:56 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:56.178+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:56.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:56 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:56.178+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:56.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:56.748+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:56.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:56.748+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:56.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:56.748+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:56.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:56.553+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:56.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:56.553+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:56.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:56.553+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:56.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:56 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:56.553+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:56.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:56.597+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:56.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:56.597+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:56.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:56.597+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:56.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:56.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:56.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:56.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:56.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:56.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:56.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:56 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:56.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:57.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:57 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:57.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:57.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:57 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:57.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:57.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:57 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:57.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:57.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:57 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:57.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:57.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:57.702+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:57.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:57.702+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:57.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:57.702+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:57.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:57.540+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:57.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:57.540+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:57.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:57.540+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:57.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:57 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:57.540+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:57.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:57.567+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:57.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:57.567+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:57.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:57.567+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:57.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:57.765+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:57.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:57.765+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:57.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:57.765+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:57.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:57 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:57.765+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:58.381 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:58 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:58.103+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:58.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:58 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:58.103+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:58.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:58 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:58.103+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:58.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:58 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:58.103+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:58.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:58 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:58.103+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:05:58.780 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:58.521+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:58.781 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:58.521+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:58.781 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:58.521+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:58.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:58.678+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:58.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:58.678+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:58.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:58.678+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:58.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:58.678+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:31.824239+0000 front 2024-02-09T23:05:31.824287+0000 (oldest deadline 2024-02-09T23:05:57.724358+0000) 2024-02-09T23:05:58.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:58.561+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:58.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:58.561+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:58.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:58.561+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:58.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:58.561+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:58.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:58 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:58.561+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:05:59.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:58.778+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:59.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:58.778+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:59.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:58.778+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:59.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:58 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:58.778+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:59.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:59.094+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:05:59.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:59.094+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:05:59.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:59.094+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:05:59.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:59.094+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:05:59.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:05:59 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:05:59.094+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:05:59.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:59.713+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:05:59.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:59.713+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:05:59.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:59.713+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:05:59.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:05:59.713+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:31.824239+0000 front 2024-02-09T23:05:31.824287+0000 (oldest deadline 2024-02-09T23:05:57.724358+0000) 2024-02-09T23:05:59.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:59.515+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:05:59.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:59.515+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:05:59.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:59.515+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:05:59.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:59.515+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:05:59.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:05:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:05:59.515+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:05:59.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:59.524+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:05:59.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:59.524+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:05:59.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:05:59.524+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:05:59.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:59.733+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:05:59.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:59.733+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:05:59.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:59.733+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:05:59.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:59.733+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:05:59.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:05:59 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:05:59.733+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:00.381 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:00.110+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:00.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:00.110+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:00.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:00.110+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:00.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:00.110+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:00.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:00 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:00.110+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:00.935 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:06:00.714+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:13.121237+0000 front 2024-02-09T23:05:13.121208+0000 (oldest deadline 2024-02-09T23:05:37.221365+0000) 2024-02-09T23:06:00.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:06:00.714+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:17.221898+0000 front 2024-02-09T23:05:17.221833+0000 (oldest deadline 2024-02-09T23:05:43.122043+0000) 2024-02-09T23:06:00.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:06:00.714+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:27.823212+0000 front 2024-02-09T23:05:27.823261+0000 (oldest deadline 2024-02-09T23:05:51.823700+0000) 2024-02-09T23:06:00.936 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7[54869]: 2024-02-09T23:06:00.714+0000 7f9c48417700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:31.824239+0000 front 2024-02-09T23:05:31.824287+0000 (oldest deadline 2024-02-09T23:05:57.724358+0000) 2024-02-09T23:06:00.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:00.481+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:00.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:00.481+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:00.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:00.481+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:00.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:00.481+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:00.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:00 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:00.481+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:00.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:00.502+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:00.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:00.502+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:00.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:00.502+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:00.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:00.502+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:00.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:00.740+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:00.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:00.740+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:00.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:00.740+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:00.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:00.740+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:00.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:00 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:00.740+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:01.568 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:01.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:01.568 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:01.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:01.568 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:01.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:01.568 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:01.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:01.571 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:01 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:01.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:01.804 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:01 smithi064 podman[74828]: 2024-02-09 23:06:01.526679937 +0000 UTC m=+5.647920253 container died cf4ff0c8317fe71bcf52752d9f703df883f15f39ec5bc0953fd82f4367eb5d3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.2, GIT_CLEAN=True) 2024-02-09T23:06:01.805 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:01 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:01.472+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:01.805 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:01 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:01.472+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:01.805 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:01 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:01.472+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:01.805 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:01 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:01.472+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:01.805 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:01 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:01.472+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:01.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:01.533+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:01.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:01.533+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:01.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:01.533+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:01.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:01.533+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:01.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:01.753+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:01.979 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:01.753+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:01.979 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:01.753+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:01.979 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:01.753+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:01.980 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:01 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:01.753+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:02.147 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:01 smithi064 podman[74828]: 2024-02-09 23:06:01.909872728 +0000 UTC m=+6.031113000 container cleanup cf4ff0c8317fe71bcf52752d9f703df883f15f39ec5bc0953fd82f4367eb5d3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7, ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-290212a, maintainer=Guillaume Abrioux ) 2024-02-09T23:06:02.147 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:01 smithi064 bash[74828]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7 2024-02-09T23:06:02.435 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:02 smithi064 podman[74847]: 2024-02-09 23:06:02.144716279 +0000 UTC m=+0.610164642 container remove cf4ff0c8317fe71bcf52752d9f703df883f15f39ec5bc0953fd82f4367eb5d3c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-02-09T23:06:02.525 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:02 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:02.136+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:02.525 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:02 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:02.136+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:02.525 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:02 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:02.136+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:02.525 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:02 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:02.136+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:02.526 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:02 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:02.136+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:02.783 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:02 smithi064 podman[74942]: 2024-02-09 23:06:02.453806646 +0000 UTC m=+0.105819614 container create 3b9d01e3fb8f98cdd17f320511df5396c318074ca11f69bd22bdaeac8bebc0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-09T23:06:02.783 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:02 smithi064 podman[74942]: 2024-02-09 23:06:02.366201328 +0000 UTC m=+0.018214248 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:06:02.784 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:02 smithi064 podman[74942]: 2024-02-09 23:06:02.781085013 +0000 UTC m=+0.433097942 container init 3b9d01e3fb8f98cdd17f320511df5396c318074ca11f69bd22bdaeac8bebc0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7-deactivate, RELEASE=main-290212a, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-02-09T23:06:02.784 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:02 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:02.436+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:02.784 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:02 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:02.436+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:02.784 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:02 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:02.436+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:02.784 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:02 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:02.436+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:02.785 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:02 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:02.436+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:02.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:02.507+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:02.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:02.507+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:02.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:02.507+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:02.978 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:02.507+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:02.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:02.726+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:02.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:02.726+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:02.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:02.726+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:02.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:02.726+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:02.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:02 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:02.726+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:03.083 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:02 smithi064 podman[74942]: 2024-02-09 23:06:02.784750906 +0000 UTC m=+0.436763833 container start 3b9d01e3fb8f98cdd17f320511df5396c318074ca11f69bd22bdaeac8bebc0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a) 2024-02-09T23:06:03.083 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:02 smithi064 podman[74942]: 2024-02-09 23:06:02.889433059 +0000 UTC m=+0.541445987 container attach 3b9d01e3fb8f98cdd17f320511df5396c318074ca11f69bd22bdaeac8bebc0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7-deactivate, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, RELEASE=main-290212a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:03.364 INFO:journalctl@ceph.osd.7.smithi064.stdout:Feb 09 23:06:03 smithi064 podman[74942]: 2024-02-09 23:06:03.081351865 +0000 UTC m=+0.733364795 container died 3b9d01e3fb8f98cdd17f320511df5396c318074ca11f69bd22bdaeac8bebc0a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, maintainer=Guillaume Abrioux ) 2024-02-09T23:06:03.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:03.146+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:03.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:03.146+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:03.633 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:03.146+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:03.633 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:03.146+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:03.633 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:03 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:03.146+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:03.636 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:03.486+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:03.636 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:03.486+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:03.636 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:03.486+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:03.637 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:03.486+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:03.637 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:03.486+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:03.653 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.7.service' 2024-02-09T23:06:03.698 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:03.699 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-02-09T23:06:03.699 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-02-09T23:06:03.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.8 2024-02-09T23:06:03.755 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.479+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:03.756 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.479+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:03.756 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.479+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:03.756 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.479+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:04.227 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 systemd[1]: Stopping Ceph osd.8 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:04.227 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.992+0000 7f9335d17700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:04.227 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.992+0000 7f9335d17700 -1 osd.8 109 *** Got signal Terminated *** 2024-02-09T23:06:04.227 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:03.992+0000 7f9335d17700 -1 osd.8 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:06:04.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:03.766+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:04.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:03.766+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:04.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:03.766+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:04.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:03.766+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:04.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:03 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:03.766+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:04.631 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:04.179+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:04.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:04.179+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:04.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:04.179+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:04.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:04.179+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:04.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:04 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:04.179+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:04.685 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:04 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:04.480+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:04.685 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:04 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:04.480+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:04.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:04 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:04.480+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:04.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:04 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:04.480+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:04.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:04 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:04.480+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:04.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:04.436+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:04.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:04.436+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:04.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:04.436+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:04.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:04.436+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:05.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:04.769+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:05.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:04.769+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:05.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:04.769+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:05.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:04.769+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:05.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:04 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:04.769+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:05.631 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:05 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:05.143+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:05.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:05 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:05.143+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:05.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:05 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:05.143+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:05.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:05 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:05.143+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:05.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:05 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:05.143+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:05.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:05.411+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:05.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:05.411+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:05.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:05.411+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:05.727 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:05.411+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:05.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:05 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:05.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:05.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:05 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:05.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:05.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:05 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:05.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:05.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:05 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:05.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:05.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:05 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:05.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:06.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:05.737+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:06.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:05.737+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:06.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:05.737+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:06.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:05.737+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:06.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:05 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:05.737+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:06.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:06.111+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:06.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:06.111+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:06.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:06.111+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:06.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:06.111+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:06.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:06 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:06.111+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:06.719 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:06.444+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:06.720 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:06.444+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:06.720 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:06.444+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:06.720 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:06.444+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:06.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:06 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:06.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:06.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:06 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:06.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:06.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:06 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:06.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:06.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:06 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:06.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:06.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:06 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:06.529+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:06.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:06.716+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:06.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:06.716+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:06.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:06.716+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:06.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:06.716+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:06.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:06.716+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:07.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:07.112+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:07.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:07.112+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:07.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:07.113+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:07.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:07.113+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:07.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:07.113+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:07.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:07 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:07.113+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:07.724 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:07.465+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:07.724 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:07.465+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:07.725 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:07.465+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:07.725 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:07.465+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:07.836 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:07.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:07.837 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:07.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:07.837 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:07.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:07.837 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:07.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:07.837 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:07 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:07.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:07.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:07.721+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:07.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:07.721+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:07.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:07.721+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:07.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:07.721+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:07.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:07 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:07.721+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:08.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:08.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:08.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:08.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:08.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:08.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:08.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:08.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:08.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:08.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:08.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:08 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:08.138+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:08.765 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:08.505+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.077156+0000 front 2024-02-09T23:05:12.077139+0000 (oldest deadline 2024-02-09T23:05:36.177052+0000) 2024-02-09T23:06:08.765 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:08.505+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:16.177460+0000 front 2024-02-09T23:05:16.177442+0000 (oldest deadline 2024-02-09T23:05:46.178466+0000) 2024-02-09T23:06:08.765 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:08.505+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.380189+0000 front 2024-02-09T23:05:28.380251+0000 (oldest deadline 2024-02-09T23:05:50.080272+0000) 2024-02-09T23:06:08.765 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8[55431]: 2024-02-09T23:06:08.505+0000 7f9332b0c700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.582763+0000 front 2024-02-09T23:05:34.582728+0000 (oldest deadline 2024-02-09T23:05:59.882589+0000) 2024-02-09T23:06:08.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:08.545+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:08.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:08.545+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:08.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:08.545+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:08.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:08.545+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:08.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:08.545+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:08.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:08 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:08.545+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:09.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:08.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:09.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:08.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:09.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:08.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:09.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:08.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:09.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:08.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:09.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:08 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:08.762+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:09.381 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:09.108+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:09.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:09.108+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:09.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:09.108+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:09.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:09.108+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:09.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:09.108+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:09.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:09 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:09.108+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:09.686 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:09 smithi089 podman[74402]: 2024-02-09 23:06:09.36847401 +0000 UTC m=+5.472651745 container died 1f0102b68da6c01ca5c9fa569026397054abb3d8be8269915d53f105ecb6083e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True) 2024-02-09T23:06:09.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:09.499+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:09.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:09.499+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:09.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:09.499+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:09.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:09.499+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:09.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:09.499+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:09.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:09 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:09.499+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:09.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:09 smithi089 podman[74402]: 2024-02-09 23:06:09.684410995 +0000 UTC m=+5.788588722 container cleanup 1f0102b68da6c01ca5c9fa569026397054abb3d8be8269915d53f105ecb6083e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-09T23:06:09.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:09 smithi089 bash[74402]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8 2024-02-09T23:06:09.977 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:09 smithi089 podman[74418]: 2024-02-09 23:06:09.894031602 +0000 UTC m=+0.521194429 container remove 1f0102b68da6c01ca5c9fa569026397054abb3d8be8269915d53f105ecb6083e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240131) 2024-02-09T23:06:09.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:09.797+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:09.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:09.797+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:09.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:09.797+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:09.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:09.797+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:09.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:09.797+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:09.978 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:09 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:09.797+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:10.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:10.123+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:10.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:10.123+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:10.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:10.123+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:10.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:10.123+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:10.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:10.123+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:10.384 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:10 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:10.123+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:10.582 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:10 smithi089 podman[74508]: 2024-02-09 23:06:10.194852693 +0000 UTC m=+0.017591163 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:06:10.582 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:10 smithi089 podman[74508]: 2024-02-09 23:06:10.295046375 +0000 UTC m=+0.117784851 container create 5eb27fab6253bc965bba43fca9a4edff0addfb6936a2b0aa43fb79e2eb0e5c19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:10.583 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:10 smithi089 podman[74508]: 2024-02-09 23:06:10.505645607 +0000 UTC m=+0.328384063 container init 5eb27fab6253bc965bba43fca9a4edff0addfb6936a2b0aa43fb79e2eb0e5c19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=main-290212a, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.2) 2024-02-09T23:06:10.583 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:10 smithi089 podman[74508]: 2024-02-09 23:06:10.508818352 +0000 UTC m=+0.331556813 container start 5eb27fab6253bc965bba43fca9a4edff0addfb6936a2b0aa43fb79e2eb0e5c19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8-deactivate, ceph=True, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-02-09T23:06:10.583 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:10 smithi089 podman[74508]: 2024-02-09 23:06:10.580509542 +0000 UTC m=+0.403248013 container attach 5eb27fab6253bc965bba43fca9a4edff0addfb6936a2b0aa43fb79e2eb0e5c19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:06:10.832 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:10 smithi089 podman[74508]: 2024-02-09 23:06:10.794308075 +0000 UTC m=+0.617046546 container died 5eb27fab6253bc965bba43fca9a4edff0addfb6936a2b0aa43fb79e2eb0e5c19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux ) 2024-02-09T23:06:10.833 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:10.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:10.833 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:10.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:10.833 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:10.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:10.833 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:10.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:10.833 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:10.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:10.833 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:10 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:10.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:10.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:10.510+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:10.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:10.510+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:10.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:10.510+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:10.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:10.510+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:10.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:10.510+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:10.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:10 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:10.510+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:11.379 INFO:journalctl@ceph.osd.8.smithi089.stdout:Feb 09 23:06:11 smithi089 podman[74534]: 2024-02-09 23:06:11.375569732 +0000 UTC m=+0.576150917 container remove 5eb27fab6253bc965bba43fca9a4edff0addfb6936a2b0aa43fb79e2eb0e5c19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-8-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS) 2024-02-09T23:06:11.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:11.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:11.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:11.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:11.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:11.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.079+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:11.400 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.8.service' 2024-02-09T23:06:11.441 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:11.441 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-02-09T23:06:11.441 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-02-09T23:06:11.441 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.9 2024-02-09T23:06:11.758 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 systemd[1]: Stopping Ceph osd.9 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:11.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:11.478+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:11.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:11.478+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:11.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:11.478+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:11.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:11.478+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:11.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:11.478+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:11.937 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:11 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:11.478+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:11.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:11.858+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:11.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:11.858+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:11.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:11.858+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:11.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:11.858+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:11.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:11.858+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:11.977 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:11.858+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:12.110 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.755+0000 7f3c7b6ad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:12.110 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.755+0000 7f3c7b6ad700 -1 osd.9 109 *** Got signal Terminated *** 2024-02-09T23:06:12.111 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:11 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:11.755+0000 7f3c7b6ad700 -1 osd.9 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:06:12.111 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:12.107+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:12.111 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:12.107+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:12.111 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:12.107+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:12.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:12.107+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:12.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:12.107+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:12.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:12 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:12.107+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:12.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:12.491+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:12.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:12.491+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:12.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:12.491+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:12.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:12.491+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:12.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:12.491+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:12.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:12 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:12.491+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:13.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:12.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:13.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:12.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:13.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:12.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:13.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:12.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:13.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:12.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:13.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:12 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:12.851+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:13.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:13.098+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:13.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:13.098+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:13.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:13.098+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:13.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:13.098+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:13.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:13.098+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:13.383 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:13 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:13.098+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:13.821 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:13.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:13.821 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:13.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:13.821 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:13.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:13.821 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:13.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:13.821 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:13.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:13.822 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:13 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:13.519+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:14.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:13.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:14.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:13.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:14.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:13.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:14.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:13.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:14.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:13.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:14.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:13 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:13.842+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:14.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:14.096+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:14.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:14.096+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:14.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:14.096+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:14.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:14.096+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:14.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:14.096+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:14.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:14 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:14.096+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:14.685 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:14.559+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:14.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:14.559+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:14.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:14.559+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:14.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:14.559+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:14.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:14.559+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:14.686 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:14 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:14.559+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:15.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:14.883+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:15.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:14.883+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:15.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:14.883+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:15.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:14.883+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:15.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:14.883+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:15.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:14.883+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:15.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:15.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:15.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:15.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:15.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:15.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:15.633 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:15 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:15.129+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:52.069620+0000 front 2024-02-09T23:05:52.069675+0000 (oldest deadline 2024-02-09T23:06:14.369687+0000) 2024-02-09T23:06:15.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:15.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:15.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:15.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:15.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:15.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:15.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:15 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:15.558+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:16.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:15.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:16.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:15.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:16.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:15.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:16.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:15.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:16.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:15.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:16.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:15 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:15.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:16.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.060674+0000 front 2024-02-09T23:05:04.060700+0000 (oldest deadline 2024-02-09T23:05:28.760818+0000) 2024-02-09T23:06:16.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.861970+0000 front 2024-02-09T23:05:12.862028+0000 (oldest deadline 2024-02-09T23:05:33.961911+0000) 2024-02-09T23:06:16.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.764560+0000 front 2024-02-09T23:05:20.764530+0000 (oldest deadline 2024-02-09T23:05:45.464485+0000) 2024-02-09T23:06:16.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:28.265911+0000 front 2024-02-09T23:05:28.266045+0000 (oldest deadline 2024-02-09T23:05:51.165971+0000) 2024-02-09T23:06:16.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:32.867114+0000 front 2024-02-09T23:05:32.867256+0000 (oldest deadline 2024-02-09T23:05:57.566926+0000) 2024-02-09T23:06:16.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:44.568408+0000 front 2024-02-09T23:05:44.568502+0000 (oldest deadline 2024-02-09T23:06:06.868220+0000) 2024-02-09T23:06:16.633 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:16 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9[67099]: 2024-02-09T23:06:16.174+0000 7f3c784a2700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:52.069620+0000 front 2024-02-09T23:05:52.069675+0000 (oldest deadline 2024-02-09T23:06:14.369687+0000) 2024-02-09T23:06:16.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:16.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:16.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:16.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:16.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:16.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:16.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:16 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:16.528+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:17.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:16.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:17.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:16.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:17.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:16.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:17.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:16.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:17.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:16.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:17.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:16 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:16.877+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:17.632 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:17 smithi003 podman[109007]: 2024-02-09 23:06:17.313525054 +0000 UTC m=+5.676081281 container died 196d67ecb6dbc4f6cbc8d5efcfa68dfbd37b748e2cb2520d01cbea32daaeda58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-09T23:06:17.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:17.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:17.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:17.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:17.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:17.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:17.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:17 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:17.537+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:17.949 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:17 smithi003 podman[109007]: 2024-02-09 23:06:17.729756399 +0000 UTC m=+6.092312626 container cleanup 196d67ecb6dbc4f6cbc8d5efcfa68dfbd37b748e2cb2520d01cbea32daaeda58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9, org.label-schema.build-date=20240131, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, ceph=True) 2024-02-09T23:06:17.950 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:17 smithi003 bash[109007]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9 2024-02-09T23:06:18.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:17 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:17.894+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:18.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:17 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:17.894+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:18.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:17 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:17.894+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:18.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:17 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:17.894+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:18.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:17 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:17.894+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:18.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:17 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:17.894+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:18.382 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:17 smithi003 podman[109023]: 2024-02-09 23:06:17.947496465 +0000 UTC m=+0.633978374 container remove 196d67ecb6dbc4f6cbc8d5efcfa68dfbd37b748e2cb2520d01cbea32daaeda58 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-09T23:06:18.711 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:18 smithi003 podman[109117]: 2024-02-09 23:06:18.38989522 +0000 UTC m=+0.108909562 container create 9461c7232c55e6771f0e961c8c30f1de9701721d719336a981db38777419a6e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:18.711 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:18 smithi003 podman[109117]: 2024-02-09 23:06:18.298041142 +0000 UTC m=+0.017055455 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:06:18.711 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:18 smithi003 podman[109117]: 2024-02-09 23:06:18.634720018 +0000 UTC m=+0.353734327 container init 9461c7232c55e6771f0e961c8c30f1de9701721d719336a981db38777419a6e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, org.label-schema.license=GPLv2) 2024-02-09T23:06:18.711 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:18 smithi003 podman[109117]: 2024-02-09 23:06:18.638167215 +0000 UTC m=+0.357181516 container start 9461c7232c55e6771f0e961c8c30f1de9701721d719336a981db38777419a6e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-02-09T23:06:18.712 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:18 smithi003 podman[109117]: 2024-02-09 23:06:18.708926186 +0000 UTC m=+0.427940499 container attach 9461c7232c55e6771f0e961c8c30f1de9701721d719336a981db38777419a6e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:06:18.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.511+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:18.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.511+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:18.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.511+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:18.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.511+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:18.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.511+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:18.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.512+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:18.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:18 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:18.512+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:19.132 INFO:journalctl@ceph.osd.9.smithi003.stdout:Feb 09 23:06:18 smithi003 podman[109117]: 2024-02-09 23:06:18.931609886 +0000 UTC m=+0.650624204 container died 9461c7232c55e6771f0e961c8c30f1de9701721d719336a981db38777419a6e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_CLEAN=True, RELEASE=main-290212a) 2024-02-09T23:06:19.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:18.876+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:19.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:18.876+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:19.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:18.876+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:19.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:18.876+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:19.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:18.876+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:19.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:18 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:18.876+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:19.488 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.9.service' 2024-02-09T23:06:19.533 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:19.533 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-02-09T23:06:19.533 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-02-09T23:06:19.533 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.10 2024-02-09T23:06:19.831 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.506+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.506+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.506+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.506+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.506+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.507+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.507+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:19.832 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 systemd[1]: Stopping Ceph osd.10 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:19.833 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.829+0000 7fb157a88700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:19.833 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.829+0000 7fb157a88700 -1 osd.10 109 *** Got signal Terminated *** 2024-02-09T23:06:19.833 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:19 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:19.829+0000 7fb157a88700 -1 osd.10 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:06:20.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:19.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:20.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:19.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:20.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:19.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:20.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:19.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:20.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:19.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:20.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:19.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:20.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:20.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:20.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:20.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:20.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:20.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:20.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:20 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:20.531+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:21.128 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:20.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:21.129 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:20.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:21.129 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:20.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:21.129 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:20.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:21.129 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:20.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:21.129 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:20 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:20.863+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:21.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:21.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:21.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:21.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:21.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:21.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:21.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:21 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:21.509+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:22.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:21.870+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:22.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:21.870+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:22.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:21.870+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:22.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:21.870+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:22.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:21.870+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:22.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:21 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:21.870+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:22.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:22.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:22.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:22.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:22.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:22.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:22.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:22 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:22.548+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:23.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:23.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:23.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:23.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:23.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:23.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:23.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:22 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:22.847+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:23.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:23.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:23.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:23.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:23.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:23.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:23.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:23 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:23.595+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:24.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:24.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:24.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:24.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:24.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:24.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:24.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:23 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:23.829+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:24.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:04.168875+0000 front 2024-02-09T23:05:04.168799+0000 (oldest deadline 2024-02-09T23:05:26.468955+0000) 2024-02-09T23:06:24.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.870659+0000 front 2024-02-09T23:05:12.870759+0000 (oldest deadline 2024-02-09T23:05:37.570881+0000) 2024-02-09T23:06:24.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.372713+0000 front 2024-02-09T23:05:20.372805+0000 (oldest deadline 2024-02-09T23:05:43.872744+0000) 2024-02-09T23:06:24.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.773774+0000 front 2024-02-09T23:05:26.773792+0000 (oldest deadline 2024-02-09T23:05:49.673902+0000) 2024-02-09T23:06:24.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:37.076318+0000 front 2024-02-09T23:05:37.076336+0000 (oldest deadline 2024-02-09T23:05:57.576348+0000) 2024-02-09T23:06:24.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:42.877586+0000 front 2024-02-09T23:05:42.877786+0000 (oldest deadline 2024-02-09T23:06:07.577545+0000) 2024-02-09T23:06:24.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:24 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10[61829]: 2024-02-09T23:06:24.562+0000 7fb150875700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.3:6822 osd.6 since back 2024-02-09T23:05:50.478926+0000 front 2024-02-09T23:05:50.479027+0000 (oldest deadline 2024-02-09T23:06:14.578876+0000) 2024-02-09T23:06:25.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:25.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:25.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:25.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:25.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:25.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:25.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:24 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:24.799+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:25.582 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:25 smithi064 podman[75059]: 2024-02-09 23:06:25.208708362 +0000 UTC m=+5.466831730 container died 5ba7c6fb049a293b7162e539e91f000bdff5419d907666d26cbd3af1de706fc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:25.935 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:25 smithi064 podman[75059]: 2024-02-09 23:06:25.678243397 +0000 UTC m=+5.936366755 container cleanup 5ba7c6fb049a293b7162e539e91f000bdff5419d907666d26cbd3af1de706fc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2) 2024-02-09T23:06:25.936 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:25 smithi064 bash[75059]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10 2024-02-09T23:06:26.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:26.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:26.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:26.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:26.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:26.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:26.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:25 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:25.798+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:26.265 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:25 smithi064 podman[75078]: 2024-02-09 23:06:25.954199627 +0000 UTC m=+0.746763673 container remove 5ba7c6fb049a293b7162e539e91f000bdff5419d907666d26cbd3af1de706fc5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, io.buildah.version=1.33.2) 2024-02-09T23:06:26.519 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:26 smithi064 podman[75171]: 2024-02-09 23:06:26.263302612 +0000 UTC m=+0.114565075 container create f79a146c2e1881131b8145f9a4968fc066f04e7efa0e2223c7b4b0a5f53278eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-02-09T23:06:26.519 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:26 smithi064 podman[75171]: 2024-02-09 23:06:26.166837294 +0000 UTC m=+0.018099750 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:06:26.908 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:26 smithi064 podman[75171]: 2024-02-09 23:06:26.598991051 +0000 UTC m=+0.450253498 container init f79a146c2e1881131b8145f9a4968fc066f04e7efa0e2223c7b4b0a5f53278eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-09T23:06:26.908 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:26 smithi064 podman[75171]: 2024-02-09 23:06:26.60208605 +0000 UTC m=+0.453348491 container start f79a146c2e1881131b8145f9a4968fc066f04e7efa0e2223c7b4b0a5f53278eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10-deactivate, RELEASE=main-290212a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2) 2024-02-09T23:06:26.908 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:26 smithi064 podman[75171]: 2024-02-09 23:06:26.665298976 +0000 UTC m=+0.516561434 container attach f79a146c2e1881131b8145f9a4968fc066f04e7efa0e2223c7b4b0a5f53278eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10-deactivate, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-09T23:06:27.185 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:26 smithi064 podman[75171]: 2024-02-09 23:06:26.906463147 +0000 UTC m=+0.757725600 container died f79a146c2e1881131b8145f9a4968fc066f04e7efa0e2223c7b4b0a5f53278eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131) 2024-02-09T23:06:27.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:27.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:27.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:27.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:27.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:27.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:27.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:26 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:26.826+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:27.529 INFO:journalctl@ceph.osd.10.smithi064.stdout:Feb 09 23:06:27 smithi064 podman[75197]: 2024-02-09 23:06:27.510302492 +0000 UTC m=+0.598386305 container remove f79a146c2e1881131b8145f9a4968fc066f04e7efa0e2223c7b4b0a5f53278eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-09T23:06:27.535 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.10.service' 2024-02-09T23:06:27.581 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:27.581 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-02-09T23:06:27.582 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-02-09T23:06:27.582 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.11 2024-02-09T23:06:27.872 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 systemd[1]: Stopping Ceph osd.11 for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:27.872 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.841+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.869+0000 7fe6a4a02700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.869+0000 7fe6a4a02700 -1 osd.11 109 *** Got signal Terminated *** 2024-02-09T23:06:27.873 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:27 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:27.869+0000 7fe6a4a02700 -1 osd.11 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-02-09T23:06:29.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:29.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:29.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:29.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:29.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:29.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:29.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:28 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:28.857+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:30.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:30.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:30.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:30.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:30.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:30.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:30.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:29 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:29.904+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:31.183 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:31.183 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:31.183 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:31.183 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:31.184 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:31.184 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:31.184 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:31.184 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:30 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:30.918+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6820 osd.8 since back 2024-02-09T23:06:08.883155+0000 front 2024-02-09T23:06:08.883198+0000 (oldest deadline 2024-02-09T23:06:30.583182+0000) 2024-02-09T23:06:32.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6806 osd.0 since back 2024-02-09T23:05:05.268602+0000 front 2024-02-09T23:05:05.268782+0000 (oldest deadline 2024-02-09T23:05:27.568853+0000) 2024-02-09T23:06:32.227 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6804 osd.1 since back 2024-02-09T23:05:12.170928+0000 front 2024-02-09T23:05:12.170792+0000 (oldest deadline 2024-02-09T23:05:33.870963+0000) 2024-02-09T23:06:32.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6804 osd.2 since back 2024-02-09T23:05:20.574296+0000 front 2024-02-09T23:05:20.574282+0000 (oldest deadline 2024-02-09T23:05:46.474497+0000) 2024-02-09T23:06:32.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.3:6814 osd.3 since back 2024-02-09T23:05:26.475200+0000 front 2024-02-09T23:05:26.475173+0000 (oldest deadline 2024-02-09T23:05:51.175132+0000) 2024-02-09T23:06:32.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6812 osd.4 since back 2024-02-09T23:05:34.076470+0000 front 2024-02-09T23:05:34.076420+0000 (oldest deadline 2024-02-09T23:05:59.376384+0000) 2024-02-09T23:06:32.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6812 osd.5 since back 2024-02-09T23:05:43.378194+0000 front 2024-02-09T23:05:43.378224+0000 (oldest deadline 2024-02-09T23:06:07.978954+0000) 2024-02-09T23:06:32.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.64:6820 osd.7 since back 2024-02-09T23:05:57.281123+0000 front 2024-02-09T23:05:57.281239+0000 (oldest deadline 2024-02-09T23:06:22.581167+0000) 2024-02-09T23:06:32.228 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:31 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11[62087]: 2024-02-09T23:06:31.931+0000 7fe6a17f7700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.89:6820 osd.8 since back 2024-02-09T23:06:08.883155+0000 front 2024-02-09T23:06:08.883198+0000 (oldest deadline 2024-02-09T23:06:30.583182+0000) 2024-02-09T23:06:33.556 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:33 smithi089 podman[74626]: 2024-02-09 23:06:33.277525622 +0000 UTC m=+5.507438850 container died d92859fb52e2c04e276eb5d244388cb17474453e86eb0870f389a88255358c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2024-02-09T23:06:33.897 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:33 smithi089 podman[74626]: 2024-02-09 23:06:33.652816827 +0000 UTC m=+5.882730060 container cleanup d92859fb52e2c04e276eb5d244388cb17474453e86eb0870f389a88255358c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11, io.buildah.version=1.33.2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-09T23:06:33.897 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:33 smithi089 bash[74626]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11 2024-02-09T23:06:34.206 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:33 smithi089 podman[74644]: 2024-02-09 23:06:33.895419686 +0000 UTC m=+0.609576726 container remove d92859fb52e2c04e276eb5d244388cb17474453e86eb0870f389a88255358c2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.vendor=CentOS) 2024-02-09T23:06:34.459 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:34 smithi089 podman[74736]: 2024-02-09 23:06:34.204674805 +0000 UTC m=+0.106999842 container create d001ae0d5c6ab8f74070fb6895be8c7e613a124cf597ab356bdc27e8864d7c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-02-09T23:06:34.459 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:34 smithi089 podman[74736]: 2024-02-09 23:06:34.115371332 +0000 UTC m=+0.017696326 image pull c328ab9dcfc7bf907ed3835578853a1e5a54ac5fa4d658ed833b2bc74e1373a6 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58 2024-02-09T23:06:34.459 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:34 smithi089 podman[74736]: 2024-02-09 23:06:34.382830329 +0000 UTC m=+0.285155332 container init d001ae0d5c6ab8f74070fb6895be8c7e613a124cf597ab356bdc27e8864d7c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131) 2024-02-09T23:06:34.459 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:34 smithi089 podman[74736]: 2024-02-09 23:06:34.386101791 +0000 UTC m=+0.288426790 container start d001ae0d5c6ab8f74070fb6895be8c7e613a124cf597ab356bdc27e8864d7c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275) 2024-02-09T23:06:34.460 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:34 smithi089 podman[74736]: 2024-02-09 23:06:34.457138395 +0000 UTC m=+0.359463405 container attach d001ae0d5c6ab8f74070fb6895be8c7e613a124cf597ab356bdc27e8864d7c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=main-290212a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:34.727 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:34 smithi089 podman[74736]: 2024-02-09 23:06:34.677609021 +0000 UTC m=+0.579934015 container died d001ae0d5c6ab8f74070fb6895be8c7e613a124cf597ab356bdc27e8864d7c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-02-09T23:06:35.219 INFO:journalctl@ceph.osd.11.smithi089.stdout:Feb 09 23:06:35 smithi089 podman[74762]: 2024-02-09 23:06:35.202166471 +0000 UTC m=+0.519360927 container remove d001ae0d5c6ab8f74070fb6895be8c7e613a124cf597ab356bdc27e8864d7c24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-osd-11-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-02-09T23:06:35.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@osd.11.service' 2024-02-09T23:06:35.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:35.264 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-02-09T23:06:35.264 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-02-09T23:06:35.268 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.a 2024-02-09T23:06:35.583 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:06:35 smithi003 systemd[1]: Stopping Ceph mds.a for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:35.881 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:06:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a[73656]: 2024-02-09T23:06:35.580+0000 7f6ce2b46700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:35.882 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:06:35 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a[73656]: 2024-02-09T23:06:35.580+0000 7f6ce2b46700 -1 mds.a *** got signal Terminated *** 2024-02-09T23:06:38.132 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:06:37 smithi003 podman[109247]: 2024-02-09 23:06:37.876674868 +0000 UTC m=+2.411969486 container died 6792ec0ee9495e58d4fa68a5ef37c5584a2e3d8ad546bb997954700fb090a00e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-290212a, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux ) 2024-02-09T23:06:38.500 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:06:38 smithi003 podman[109247]: 2024-02-09 23:06:38.238477409 +0000 UTC m=+2.773772033 container cleanup 6792ec0ee9495e58d4fa68a5ef37c5584a2e3d8ad546bb997954700fb090a00e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-290212a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:38.500 INFO:journalctl@ceph.mds.a.smithi003.stdout:Feb 09 23:06:38 smithi003 bash[109247]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-a 2024-02-09T23:06:38.674 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.a.service' 2024-02-09T23:06:38.712 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:38.713 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-02-09T23:06:38.713 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-02-09T23:06:38.713 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.d 2024-02-09T23:06:39.044 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 23:06:38 smithi003 systemd[1]: Stopping Ceph mds.d for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:39.381 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 23:06:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d[72703]: 2024-02-09T23:06:39.041+0000 7f9e24b60700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:39.382 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 23:06:39 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d[72703]: 2024-02-09T23:06:39.041+0000 7f9e24b60700 -1 mds.d *** got signal Terminated *** 2024-02-09T23:06:43.382 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 23:06:43 smithi003 podman[109368]: 2024-02-09 23:06:43.053283864 +0000 UTC m=+4.115302338 container died a32f1d446916f8f37426cbcf06d41e3731265cb8c593876813e5e767889eb2d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-02-09T23:06:43.786 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 23:06:43 smithi003 podman[109368]: 2024-02-09 23:06:43.533253178 +0000 UTC m=+4.595271650 container cleanup a32f1d446916f8f37426cbcf06d41e3731265cb8c593876813e5e767889eb2d6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=main-290212a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-02-09T23:06:43.786 INFO:journalctl@ceph.mds.d.smithi003.stdout:Feb 09 23:06:43 smithi003 bash[109368]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-d 2024-02-09T23:06:43.913 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.d.service' 2024-02-09T23:06:43.948 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:43.948 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-02-09T23:06:43.949 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-02-09T23:06:43.949 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.g 2024-02-09T23:06:44.132 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:06:44 smithi003 systemd[1]: Stopping Ceph mds.g for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:44.631 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:06:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g[73280]: 2024-02-09T23:06:44.294+0000 7fc0302bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:44.632 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:06:44 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g[73280]: 2024-02-09T23:06:44.294+0000 7fc0302bb700 -1 mds.g *** got signal Terminated *** 2024-02-09T23:06:46.131 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:06:45 smithi003 podman[109487]: 2024-02-09 23:06:45.874085862 +0000 UTC m=+1.708954061 container died 1ddc91dd56b1fffc8d7f1a0b9491125d8abbe9312377b529e2b24fa0f28c8e71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131, org.label-schema.vendor=CentOS, RELEASE=main-290212a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-02-09T23:06:46.382 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:06:46 smithi003 podman[109487]: 2024-02-09 23:06:46.237048501 +0000 UTC m=+2.071916699 container cleanup 1ddc91dd56b1fffc8d7f1a0b9491125d8abbe9312377b529e2b24fa0f28c8e71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-02-09T23:06:46.382 INFO:journalctl@ceph.mds.g.smithi003.stdout:Feb 09 23:06:46 smithi003 bash[109487]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-g 2024-02-09T23:06:46.672 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.g.service' 2024-02-09T23:06:46.707 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:46.707 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-02-09T23:06:46.707 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-02-09T23:06:46.707 DEBUG:teuthology.orchestra.run.smithi003:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.j 2024-02-09T23:06:47.035 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 23:06:46 smithi003 systemd[1]: Stopping Ceph mds.j for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:47.382 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 23:06:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j[72322]: 2024-02-09T23:06:47.032+0000 7fa2f63b4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:47.382 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 23:06:47 smithi003 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j[72322]: 2024-02-09T23:06:47.032+0000 7fa2f63b4700 -1 mds.j *** got signal Terminated *** 2024-02-09T23:06:49.235 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 23:06:48 smithi003 podman[109610]: 2024-02-09 23:06:48.970581693 +0000 UTC m=+2.044579485 container died b486efb1188fab0324028bd08a9b45cf4e7ad6743b44bd6dd82dc7552db96fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-02-09T23:06:49.502 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 23:06:49 smithi003 podman[109610]: 2024-02-09 23:06:49.315454309 +0000 UTC m=+2.389452090 container cleanup b486efb1188fab0324028bd08a9b45cf4e7ad6743b44bd6dd82dc7552db96fd5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j, RELEASE=main-290212a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240131, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:49.502 INFO:journalctl@ceph.mds.j.smithi003.stdout:Feb 09 23:06:49 smithi003 bash[109610]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-j 2024-02-09T23:06:49.630 DEBUG:teuthology.orchestra.run.smithi003:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.j.service' 2024-02-09T23:06:49.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:49.668 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-02-09T23:06:49.668 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-02-09T23:06:49.668 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.b 2024-02-09T23:06:49.964 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:06:49 smithi064 systemd[1]: Stopping Ceph mds.b for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:50.238 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:06:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T23:06:49.960+0000 7f9d44e3d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:50.238 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:06:49 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b[67233]: 2024-02-09T23:06:49.960+0000 7f9d44e3d700 -1 mds.b *** got signal Terminated *** 2024-02-09T23:06:54.172 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:06:53 smithi064 podman[75276]: 2024-02-09 23:06:53.878147877 +0000 UTC m=+4.018673732 container died 85309a727c7612cf340e4cf2807b2236a725dab6aa31c7f1f9eadf28d1adeb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b, org.label-schema.build-date=20240131, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-02-09T23:06:54.435 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:06:54 smithi064 podman[75276]: 2024-02-09 23:06:54.26892756 +0000 UTC m=+4.409453407 container cleanup 85309a727c7612cf340e4cf2807b2236a725dab6aa31c7f1f9eadf28d1adeb57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=main-290212a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.license=GPLv2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-02-09T23:06:54.435 INFO:journalctl@ceph.mds.b.smithi064.stdout:Feb 09 23:06:54 smithi064 bash[75276]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-b 2024-02-09T23:06:54.696 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.b.service' 2024-02-09T23:06:54.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:54.739 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-02-09T23:06:54.739 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-02-09T23:06:54.740 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.e 2024-02-09T23:06:55.067 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:54 smithi064 systemd[1]: Stopping Ceph mds.e for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:55.435 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: 2024-02-09T23:06:55.063+0000 7fac93390700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:55.435 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:55 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e[66603]: 2024-02-09T23:06:55.063+0000 7fac93390700 -1 mds.e *** got signal Terminated *** 2024-02-09T23:06:58.185 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:57 smithi064 podman[75410]: 2024-02-09 23:06:57.894998332 +0000 UTC m=+2.925633748 container died d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=main-290212a) 2024-02-09T23:06:58.517 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:58 smithi064 podman[75410]: 2024-02-09 23:06:58.314165146 +0000 UTC m=+3.344800558 container cleanup d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-02-09T23:06:58.517 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:58 smithi064 bash[75410]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e 2024-02-09T23:06:58.752 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.e.service' 2024-02-09T23:06:58.780 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:58 smithi064 podman[75424]: 2024-02-09 23:06:58.515159734 +0000 UTC m=+0.619135528 container remove d178c12357768ebccd00268d7159453bc7863730bb8f246f165b3ddd9551a140 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-e, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.schema-version=1.0, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240131) 2024-02-09T23:06:58.781 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:58 smithi064 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.e.service: Deactivated successfully. 2024-02-09T23:06:58.781 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:58 smithi064 systemd[1]: Stopped Ceph mds.e for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T23:06:58.781 INFO:journalctl@ceph.mds.e.smithi064.stdout:Feb 09 23:06:58 smithi064 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.e.service: Consumed 9.284s CPU time. 2024-02-09T23:06:58.792 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:06:58.793 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-02-09T23:06:58.793 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-02-09T23:06:58.793 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.h 2024-02-09T23:06:59.185 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:06:58 smithi064 systemd[1]: Stopping Ceph mds.h for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:06:59.685 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:06:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h[66914]: 2024-02-09T23:06:59.234+0000 7f86f9ebb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:06:59.685 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:06:59 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h[66914]: 2024-02-09T23:06:59.234+0000 7f86f9ebb700 -1 mds.h *** got signal Terminated *** 2024-02-09T23:07:02.162 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:07:01 smithi064 podman[75531]: 2024-02-09 23:07:01.889194612 +0000 UTC m=+2.873412314 container died be2ffb82557bbdcf887e1592d83ca3b5087884a8d2c93295f55d2ffcadb65141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h, io.buildah.version=1.33.2, RELEASE=main-290212a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240131) 2024-02-09T23:07:02.435 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:07:02 smithi064 podman[75531]: 2024-02-09 23:07:02.242579833 +0000 UTC m=+3.226797523 container cleanup be2ffb82557bbdcf887e1592d83ca3b5087884a8d2c93295f55d2ffcadb65141 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.2, RELEASE=main-290212a) 2024-02-09T23:07:02.436 INFO:journalctl@ceph.mds.h.smithi064.stdout:Feb 09 23:07:02 smithi064 bash[75531]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-h 2024-02-09T23:07:02.678 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.h.service' 2024-02-09T23:07:02.717 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:07:02.717 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-02-09T23:07:02.717 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-02-09T23:07:02.718 DEBUG:teuthology.orchestra.run.smithi064:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.k 2024-02-09T23:07:03.185 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:07:02 smithi064 systemd[1]: Stopping Ceph mds.k for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:07:03.185 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:07:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k[66281]: 2024-02-09T23:07:03.038+0000 7fc3e6f8d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:07:03.185 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:07:03 smithi064 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k[66281]: 2024-02-09T23:07:03.038+0000 7fc3e6f8d700 -1 mds.k *** got signal Terminated *** 2024-02-09T23:07:06.174 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:07:05 smithi064 podman[75654]: 2024-02-09 23:07:05.880231787 +0000 UTC m=+2.936737520 container died 661c32850638bb216cd0a13efcd93cbddd896d1ab7169fa3ff2d8bbd279fd7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240131, org.label-schema.schema-version=1.0) 2024-02-09T23:07:06.435 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:07:06 smithi064 podman[75654]: 2024-02-09 23:07:06.254436744 +0000 UTC m=+3.310942451 container cleanup 661c32850638bb216cd0a13efcd93cbddd896d1ab7169fa3ff2d8bbd279fd7f2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k, org.label-schema.schema-version=1.0, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, RELEASE=main-290212a, org.label-schema.build-date=20240131, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-02-09T23:07:06.435 INFO:journalctl@ceph.mds.k.smithi064.stdout:Feb 09 23:07:06 smithi064 bash[75654]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-k 2024-02-09T23:07:06.601 DEBUG:teuthology.orchestra.run.smithi064:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.k.service' 2024-02-09T23:07:06.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:07:06.638 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-02-09T23:07:06.639 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-02-09T23:07:06.639 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.c 2024-02-09T23:07:06.930 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:06 smithi089 systemd[1]: Stopping Ceph mds.c for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:07:07.189 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c[67015]: 2024-02-09T23:07:06.927+0000 7f78b5a83700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:07:07.189 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:06 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c[67015]: 2024-02-09T23:07:06.927+0000 7f78b5a83700 -1 mds.c *** got signal Terminated *** 2024-02-09T23:07:10.227 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:09 smithi089 podman[74841]: 2024-02-09 23:07:09.884913511 +0000 UTC m=+3.059341933 container died ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c, GIT_CLEAN=True, RELEASE=main-290212a, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-02-09T23:07:10.549 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:10 smithi089 podman[74841]: 2024-02-09 23:07:10.312726742 +0000 UTC m=+3.487155157 container cleanup ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240131, io.buildah.version=1.33.2, org.label-schema.license=GPLv2) 2024-02-09T23:07:10.549 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:10 smithi089 bash[74841]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c 2024-02-09T23:07:10.787 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.c.service' 2024-02-09T23:07:10.811 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:10 smithi089 podman[74867]: 2024-02-09 23:07:10.547145685 +0000 UTC m=+0.655652386 container remove ba0092832da03239192b25783b2d41c623c5bf6e37b5952f94d1b4a5c5ae4ae8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-c, RELEASE=main-290212a, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-02-09T23:07:10.811 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:10 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.c.service: Deactivated successfully. 2024-02-09T23:07:10.812 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:10 smithi089 systemd[1]: Stopped Ceph mds.c for 2bf8a73a-c79c-11ee-95b8-87774f69a715. 2024-02-09T23:07:10.812 INFO:journalctl@ceph.mds.c.smithi089.stdout:Feb 09 23:07:10 smithi089 systemd[1]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.c.service: Consumed 1min 4.912s CPU time. 2024-02-09T23:07:10.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:07:10.825 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-02-09T23:07:10.825 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-02-09T23:07:10.826 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.f 2024-02-09T23:07:11.135 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:07:10 smithi089 systemd[1]: Stopping Ceph mds.f for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:07:11.476 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:07:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f[66706]: 2024-02-09T23:07:11.132+0000 7f82b929d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:07:11.477 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:07:11 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f[66706]: 2024-02-09T23:07:11.132+0000 7f82b929d700 -1 mds.f *** got signal Terminated *** 2024-02-09T23:07:14.152 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:07:13 smithi089 podman[74975]: 2024-02-09 23:07:13.892483792 +0000 UTC m=+2.855575472 container died 60bbd2e371b33bdf9435055a32905b787354e6f2eef19e681ca41b2bab74b336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f, io.buildah.version=1.33.2, org.label-schema.build-date=20240131, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=main-290212a, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-02-09T23:07:14.469 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:07:14 smithi089 podman[74975]: 2024-02-09 23:07:14.24131264 +0000 UTC m=+3.204404328 container cleanup 60bbd2e371b33bdf9435055a32905b787354e6f2eef19e681ca41b2bab74b336 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=main-290212a, io.buildah.version=1.33.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-02-09T23:07:14.469 INFO:journalctl@ceph.mds.f.smithi089.stdout:Feb 09 23:07:14 smithi089 bash[74975]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-f 2024-02-09T23:07:14.600 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.f.service' 2024-02-09T23:07:14.636 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:07:14.636 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-02-09T23:07:14.636 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-02-09T23:07:14.636 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.i 2024-02-09T23:07:14.727 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:07:14 smithi089 systemd[1]: Stopping Ceph mds.i for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:07:15.227 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:07:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i[67337]: 2024-02-09T23:07:14.935+0000 7f21da1d0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:07:15.227 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:07:14 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i[67337]: 2024-02-09T23:07:14.935+0000 7f21da1d0700 -1 mds.i *** got signal Terminated *** 2024-02-09T23:07:18.184 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:07:17 smithi089 podman[75097]: 2024-02-09 23:07:17.886779761 +0000 UTC m=+3.041285403 container died d2f4556cecd8ef3db90ab689eb03dbc2463547c6e3703f05e940332eaf3ce48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240131, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=main-290212a, io.buildah.version=1.33.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-02-09T23:07:18.477 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:07:18 smithi089 podman[75097]: 2024-02-09 23:07:18.303102639 +0000 UTC m=+3.457608274 container cleanup d2f4556cecd8ef3db90ab689eb03dbc2463547c6e3703f05e940332eaf3ce48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i, RELEASE=main-290212a, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240131, ceph=True, CEPH_POINT_RELEASE=) 2024-02-09T23:07:18.477 INFO:journalctl@ceph.mds.i.smithi089.stdout:Feb 09 23:07:18 smithi089 bash[75097]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-i 2024-02-09T23:07:18.733 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.i.service' 2024-02-09T23:07:18.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:07:18.767 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-02-09T23:07:18.767 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-02-09T23:07:18.768 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.l 2024-02-09T23:07:19.083 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:07:18 smithi089 systemd[1]: Stopping Ceph mds.l for 2bf8a73a-c79c-11ee-95b8-87774f69a715... 2024-02-09T23:07:19.477 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:07:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l[66400]: 2024-02-09T23:07:19.080+0000 7f9599a12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-02-09T23:07:19.477 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:07:19 smithi089 ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l[66400]: 2024-02-09T23:07:19.080+0000 7f9599a12700 -1 mds.l *** got signal Terminated *** 2024-02-09T23:07:22.142 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:07:21 smithi089 podman[75219]: 2024-02-09 23:07:21.887091632 +0000 UTC m=+2.908670993 container died d6648ce98dd6b452b38ef29bd6a940a54e38eae42f527b539968cff245673c39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, RELEASE=main-290212a, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-02-09T23:07:22.477 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:07:22 smithi089 podman[75219]: 2024-02-09 23:07:22.240272707 +0000 UTC m=+3.261852059 container cleanup d6648ce98dd6b452b38ef29bd6a940a54e38eae42f527b539968cff245673c39 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:caf5e50b0be996bed0eb0d30a5482e8034654966f88a3565d36aeb446466fb58, name=ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=14dafef590928bb89728de91784b87b8f9bbf275, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240131, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-290212a, org.label-schema.license=GPLv2) 2024-02-09T23:07:22.477 INFO:journalctl@ceph.mds.l.smithi089.stdout:Feb 09 23:07:22 smithi089 bash[75219]: ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715-mds-l 2024-02-09T23:07:22.660 DEBUG:teuthology.orchestra.run.smithi089:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bf8a73a-c79c-11ee-95b8-87774f69a715@mds.l.service' 2024-02-09T23:07:22.694 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-02-09T23:07:22.695 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-02-09T23:07:22.695 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-09T23:07:22.892 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T23:07:25.336 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-09T23:07:25.538 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T23:07:28.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --force --keep-logs 2024-02-09T23:07:28.345 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T23:07:30.837 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T23:07:30.872 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T23:07:30.911 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-02-09T23:07:30.939 INFO:tasks.cephadm:Archiving crash dumps... 2024-02-09T23:07:30.942 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi003/crash 2024-02-09T23:07:30.942 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash -- . 2024-02-09T23:07:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:tar: /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-09T23:07:30.984 INFO:teuthology.orchestra.run.smithi003.stderr:tar: Error is not recoverable: exiting now 2024-02-09T23:07:30.986 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi064/crash 2024-02-09T23:07:30.987 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash -- . 2024-02-09T23:07:31.033 INFO:teuthology.orchestra.run.smithi064.stderr:tar: /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-09T23:07:31.033 INFO:teuthology.orchestra.run.smithi064.stderr:tar: Error is not recoverable: exiting now 2024-02-09T23:07:31.035 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi089/crash 2024-02-09T23:07:31.035 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash -- . 2024-02-09T23:07:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:tar: /var/lib/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/crash: Cannot open: No such file or directory 2024-02-09T23:07:31.074 INFO:teuthology.orchestra.run.smithi089.stderr:tar: Error is not recoverable: exiting now 2024-02-09T23:07:31.075 INFO:tasks.cephadm:Checking cluster log for badness... 2024-02-09T23:07:31.076 DEBUG:teuthology.orchestra.run.smithi003:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-02-09T23:07:31.111 INFO:tasks.cephadm:Compressing logs... 2024-02-09T23:07:31.111 DEBUG:teuthology.orchestra.run.smithi003:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T23:07:31.154 DEBUG:teuthology.orchestra.run.smithi064:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T23:07:31.157 DEBUG:teuthology.orchestra.run.smithi089:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-02-09T23:07:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T23:07:31.182 INFO:teuthology.orchestra.run.smithi003.stderr:: No such file or directory 2024-02-09T23:07:31.183 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.6.log 2024-02-09T23:07:31.183 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.a.log 2024-02-09T23:07:31.183 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-09T23:07:31.183 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.g.log 2024-02-09T23:07:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.j.log 2024-02-09T23:07:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.3.log 2024-02-09T23:07:31.200 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.x.log 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.0.log 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.x.log: 90.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr: 94.5% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.j.log.gz 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-09T23:07:31.201 INFO:teuthology.orchestra.run.smithi003.stderr: 84.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-09T23:07:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log: 87.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-09T23:07:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.9.log 2024-02-09T23:07:31.202 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.d.log 2024-02-09T23:07:31.204 INFO:teuthology.orchestra.run.smithi064.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T23:07:31.204 INFO:teuthology.orchestra.run.smithi064.stderr:: No such file or directory 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.10.log 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.b.log 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.10.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.4.log 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.b.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.1.log 2024-02-09T23:07:31.205 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.k.log 2024-02-09T23:07:31.206 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.4.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.h.log 2024-02-09T23:07:31.206 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.7.log 2024-02-09T23:07:31.206 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.e.log 2024-02-09T23:07:31.206 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.7.log: 90.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-09T23:07:31.206 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.y.log 2024-02-09T23:07:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-02-09T23:07:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:: No such file or directory 2024-02-09T23:07:31.207 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.l.log 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.i.log 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.11.log 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.z.log 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.8.log 2024-02-09T23:07:31.208 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.5.log 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.z.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.f.log 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.audit.log.gz 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.2.log 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log: 87.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-09T23:07:31.209 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.c.log 2024-02-09T23:07:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.2.log: 92.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.z.log.gz 2024-02-09T23:07:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-09T23:07:31.210 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.c.log 2024-02-09T23:07:31.211 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log 2024-02-09T23:07:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.a.log 2024-02-09T23:07:31.212 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.d.log.gz 2024-02-09T23:07:31.212 INFO:teuthology.orchestra.run.smithi003.stderr: 91.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-09T23:07:31.214 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log 2024-02-09T23:07:31.227 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log 2024-02-09T23:07:31.228 INFO:teuthology.orchestra.run.smithi064.stderr: 83.7% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.cephadm.log.gz 2024-02-09T23:07:31.231 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log: 92.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.y.log.gz 2024-02-09T23:07:31.231 INFO:teuthology.orchestra.run.smithi064.stderr: 87.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph.log.gz 2024-02-09T23:07:31.233 INFO:teuthology.orchestra.run.smithi064.stderr:gzip -5 --verbose -- /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.b.log 2024-02-09T23:07:31.236 INFO:teuthology.orchestra.run.smithi089.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.c.log: 90.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-09T23:07:31.274 INFO:teuthology.orchestra.run.smithi064.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log: /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.b.log: 90.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-volume.log.gz 2024-02-09T23:07:31.562 INFO:teuthology.orchestra.run.smithi003.stderr:/var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.a.log: 89.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mgr.x.log.gz 2024-02-09T23:07:31.818 INFO:teuthology.orchestra.run.smithi064.stderr: 92.1% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.b.log.gz 2024-02-09T23:07:32.142 INFO:teuthology.orchestra.run.smithi089.stderr: 92.4% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.c.log.gz 2024-02-09T23:07:32.316 INFO:teuthology.orchestra.run.smithi003.stderr: 94.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.g.log.gz 2024-02-09T23:07:32.542 INFO:teuthology.orchestra.run.smithi003.stderr: 95.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.a.log.gz 2024-02-09T23:07:32.574 INFO:teuthology.orchestra.run.smithi064.stderr: 95.6% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.k.log.gz 2024-02-09T23:07:32.718 INFO:teuthology.orchestra.run.smithi003.stderr: 90.6% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mon.a.log.gz 2024-02-09T23:07:32.737 INFO:teuthology.orchestra.run.smithi089.stderr: 95.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.l.log.gz 2024-02-09T23:07:33.400 INFO:teuthology.orchestra.run.smithi064.stderr: 95.6% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.e.log.gz 2024-02-09T23:07:34.424 INFO:teuthology.orchestra.run.smithi064.stderr: 94.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.4.log.gz 2024-02-09T23:07:34.519 INFO:teuthology.orchestra.run.smithi003.stderr: 93.7% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.0.log.gz 2024-02-09T23:07:34.731 INFO:teuthology.orchestra.run.smithi003.stderr: 94.0% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.9.log.gz 2024-02-09T23:07:34.731 INFO:teuthology.orchestra.run.smithi003.stderr: 93.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.6.log.gz 2024-02-09T23:07:34.898 INFO:teuthology.orchestra.run.smithi089.stderr: 94.1% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.11.log.gz 2024-02-09T23:07:35.488 INFO:teuthology.orchestra.run.smithi003.stderr: 93.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.3.log.gz 2024-02-09T23:07:35.490 INFO:teuthology.orchestra.run.smithi003.stderr: 2024-02-09T23:07:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:real 0m4.322s 2024-02-09T23:07:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:user 0m18.527s 2024-02-09T23:07:35.490 INFO:teuthology.orchestra.run.smithi003.stderr:sys 0m0.603s 2024-02-09T23:07:35.603 INFO:teuthology.orchestra.run.smithi064.stderr: 93.8% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.1.log.gz 2024-02-09T23:07:36.108 INFO:teuthology.orchestra.run.smithi064.stderr: 94.1% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.10.log.gz 2024-02-09T23:07:36.436 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.2.log.gz 2024-02-09T23:07:37.123 INFO:teuthology.orchestra.run.smithi089.stderr: 94.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.5.log.gz 2024-02-09T23:07:37.478 INFO:teuthology.orchestra.run.smithi089.stderr: 94.2% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.8.log.gz 2024-02-09T23:07:37.872 INFO:teuthology.orchestra.run.smithi064.stderr: 94.1% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-osd.7.log.gz 2024-02-09T23:07:48.928 INFO:teuthology.orchestra.run.smithi089.stderr: 93.9% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.f.log.gz 2024-02-09T23:07:50.012 INFO:teuthology.orchestra.run.smithi064.stderr: 93.8% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.h.log.gz 2024-02-09T23:07:50.815 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.c.log.gz 2024-02-09T23:07:52.267 INFO:teuthology.orchestra.run.smithi089.stderr: 93.7% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.i.log.gz 2024-02-09T23:07:52.269 INFO:teuthology.orchestra.run.smithi089.stderr: 2024-02-09T23:07:52.269 INFO:teuthology.orchestra.run.smithi089.stderr:real 0m21.098s 2024-02-09T23:07:52.269 INFO:teuthology.orchestra.run.smithi089.stderr:user 1m18.594s 2024-02-09T23:07:52.269 INFO:teuthology.orchestra.run.smithi089.stderr:sys 0m2.475s 2024-02-09T23:08:13.254 INFO:teuthology.orchestra.run.smithi064.stderr: 93.7% -- replaced with /var/log/ceph/2bf8a73a-c79c-11ee-95b8-87774f69a715/ceph-mds.b.log.gz 2024-02-09T23:08:13.256 INFO:teuthology.orchestra.run.smithi064.stderr: 2024-02-09T23:08:13.256 INFO:teuthology.orchestra.run.smithi064.stderr:real 0m42.084s 2024-02-09T23:08:13.256 INFO:teuthology.orchestra.run.smithi064.stderr:user 1m21.240s 2024-02-09T23:08:13.256 INFO:teuthology.orchestra.run.smithi064.stderr:sys 0m2.348s 2024-02-09T23:08:13.257 INFO:tasks.cephadm:Archiving logs... 2024-02-09T23:08:13.257 DEBUG:teuthology.misc:Transferring archived files from smithi003:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi003/log 2024-02-09T23:08:13.258 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-09T23:08:15.577 DEBUG:teuthology.misc:Transferring archived files from smithi064:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi064/log 2024-02-09T23:08:15.578 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-09T23:08:25.981 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi089/log 2024-02-09T23:08:25.982 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-02-09T23:08:36.022 INFO:tasks.cephadm:Removing cluster... 2024-02-09T23:08:36.023 DEBUG:teuthology.orchestra.run.smithi003:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --force 2024-02-09T23:08:36.224 INFO:teuthology.orchestra.run.smithi003.stdout:Deleting cluster with fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T23:08:36.498 DEBUG:teuthology.orchestra.run.smithi064:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --force 2024-02-09T23:08:36.709 INFO:teuthology.orchestra.run.smithi064.stdout:Deleting cluster with fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T23:08:37.054 DEBUG:teuthology.orchestra.run.smithi089:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bf8a73a-c79c-11ee-95b8-87774f69a715 --force 2024-02-09T23:08:37.245 INFO:teuthology.orchestra.run.smithi089.stdout:Deleting cluster with fsid: 2bf8a73a-c79c-11ee-95b8-87774f69a715 2024-02-09T23:08:37.680 INFO:tasks.cephadm:Removing cephadm ... 2024-02-09T23:08:37.680 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-09T23:08:37.699 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-09T23:08:37.716 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf /home/ubuntu/cephtest/cephadm 2024-02-09T23:08:37.735 INFO:tasks.cephadm:Teardown complete 2024-02-09T23:08:37.735 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-02-09T23:08:37.746 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-02-09T23:08:37.746 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T23:08:37.751 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T23:08:37.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-02-09T23:08:37.869 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-02-09T23:08:37.883 INFO:teuthology.task.clock:Checking final clock skew... 2024-02-09T23:08:37.884 DEBUG:teuthology.orchestra.run.smithi003:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T23:08:37.887 DEBUG:teuthology.orchestra.run.smithi064:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T23:08:37.889 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-02-09T23:08:37.900 INFO:teuthology.orchestra.run.smithi003.stderr:bash: line 1: ntpq: command not found 2024-02-09T23:08:37.902 INFO:teuthology.orchestra.run.smithi064.stderr:bash: line 1: ntpq: command not found 2024-02-09T23:08:37.903 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-02-09T23:08:37.904 INFO:teuthology.orchestra.run.smithi003.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T23:08:37.904 INFO:teuthology.orchestra.run.smithi003.stdout:=============================================================================== 2024-02-09T23:08:37.904 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv01.front.sepia.ceph.com 2 8 377 160 -210us[ -246us] +/- 78ms 2024-02-09T23:08:37.904 INFO:teuthology.orchestra.run.smithi003.stdout:^* hv02.front.sepia.ceph.com 3 7 377 30 -912us[ -960us] +/- 57ms 2024-02-09T23:08:37.905 INFO:teuthology.orchestra.run.smithi003.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 30 +1369us[+1321us] +/- 74ms 2024-02-09T23:08:37.905 INFO:teuthology.orchestra.run.smithi003.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T23:08:37.906 INFO:teuthology.orchestra.run.smithi064.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T23:08:37.906 INFO:teuthology.orchestra.run.smithi064.stdout:=============================================================================== 2024-02-09T23:08:37.906 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 -236us[ -236us] +/- 80ms 2024-02-09T23:08:37.907 INFO:teuthology.orchestra.run.smithi064.stdout:^* hv02.front.sepia.ceph.com 3 7 377 98 -930us[-1025us] +/- 56ms 2024-02-09T23:08:37.907 INFO:teuthology.orchestra.run.smithi064.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +1389us[+1389us] +/- 74ms 2024-02-09T23:08:37.907 INFO:teuthology.orchestra.run.smithi064.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T23:08:37.908 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-02-09T23:08:37.909 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-02-09T23:08:37.909 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 27 -222us[ -222us] +/- 80ms 2024-02-09T23:08:37.909 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv02.front.sepia.ceph.com 3 6 377 29 -911us[ -940us] +/- 57ms 2024-02-09T23:08:37.909 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +1405us[+1377us] +/- 74ms 2024-02-09T23:08:37.909 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-02-09T23:08:37.909 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-02-09T23:08:37.921 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-02-09T23:08:37.922 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-02-09T23:08:37.932 DEBUG:teuthology.orchestra.run.smithi003:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T23:08:38.035 DEBUG:teuthology.orchestra.run.smithi064:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T23:08:38.117 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-02-09T23:08:38.188 DEBUG:teuthology.orchestra.run.smithi003:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T23:08:38.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:08:38.220 DEBUG:teuthology.orchestra.run.smithi064:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T23:08:38.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:08:38.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-02-09T23:08:38.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:08:38.285 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-02-09T23:08:38.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-02-09T23:08:38.308 INFO:teuthology.task.internal:Duration was 2460.486923 seconds 2024-02-09T23:08:38.308 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-02-09T23:08:38.320 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-02-09T23:08:38.321 DEBUG:teuthology.orchestra.run.smithi003:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T23:08:38.323 DEBUG:teuthology.orchestra.run.smithi064:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T23:08:38.325 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-02-09T23:08:38.365 INFO:teuthology.orchestra.run.smithi003.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T23:08:38.368 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T23:08:38.374 INFO:teuthology.orchestra.run.smithi064.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-02-09T23:08:38.715 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-02-09T23:08:38.716 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi003.front.sepia.ceph.com 2024-02-09T23:08:38.716 DEBUG:teuthology.orchestra.run.smithi003:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T23:08:38.736 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi064.front.sepia.ceph.com 2024-02-09T23:08:38.736 DEBUG:teuthology.orchestra.run.smithi064:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T23:08:38.769 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-02-09T23:08:38.770 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-02-09T23:08:38.798 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-02-09T23:08:38.798 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T23:08:38.801 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T23:08:38.812 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-02-09T23:08:38.876 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-02-09T23:08:38.876 DEBUG:teuthology.orchestra.run.smithi003:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T23:08:38.879 DEBUG:teuthology.orchestra.run.smithi064:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T23:08:38.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-02-09T23:08:39.313 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-02-09T23:08:39.325 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-02-09T23:08:39.325 DEBUG:teuthology.orchestra.run.smithi003:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T23:08:39.353 DEBUG:teuthology.orchestra.run.smithi064:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T23:08:39.384 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-02-09T23:08:39.411 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-02-09T23:08:39.424 DEBUG:teuthology.orchestra.run.smithi003:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T23:08:39.426 DEBUG:teuthology.orchestra.run.smithi064:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T23:08:39.429 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-02-09T23:08:39.449 INFO:teuthology.orchestra.run.smithi003.stdout:kernel.core_pattern = core 2024-02-09T23:08:39.455 INFO:teuthology.orchestra.run.smithi064.stdout:kernel.core_pattern = core 2024-02-09T23:08:39.476 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-02-09T23:08:39.504 DEBUG:teuthology.orchestra.run.smithi003:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T23:08:39.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:08:39.518 DEBUG:teuthology.orchestra.run.smithi064:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T23:08:39.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:08:39.540 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-02-09T23:08:39.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-02-09T23:08:39.557 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-02-09T23:08:39.570 INFO:teuthology.task.internal:Transferring archived files... 2024-02-09T23:08:39.571 DEBUG:teuthology.misc:Transferring archived files from smithi003:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi003 2024-02-09T23:08:39.571 DEBUG:teuthology.orchestra.run.smithi003:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T23:08:39.618 DEBUG:teuthology.misc:Transferring archived files from smithi064:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi064 2024-02-09T23:08:39.619 DEBUG:teuthology.orchestra.run.smithi064:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T23:08:39.661 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417/remote/smithi089 2024-02-09T23:08:39.662 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-02-09T23:08:39.700 INFO:teuthology.task.internal:Removing archive directory... 2024-02-09T23:08:39.701 DEBUG:teuthology.orchestra.run.smithi003:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T23:08:39.703 DEBUG:teuthology.orchestra.run.smithi064:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T23:08:39.705 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-02-09T23:08:39.749 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-02-09T23:08:39.764 INFO:teuthology.task.internal:Not uploading archives. 2024-02-09T23:08:39.764 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-02-09T23:08:39.778 INFO:teuthology.task.internal:Tidying up after the test... 2024-02-09T23:08:39.778 DEBUG:teuthology.orchestra.run.smithi003:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T23:08:39.781 DEBUG:teuthology.orchestra.run.smithi064:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T23:08:39.783 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-02-09T23:08:39.792 INFO:teuthology.orchestra.run.smithi003.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 23:08 /home/ubuntu/cephtest 2024-02-09T23:08:39.795 INFO:teuthology.orchestra.run.smithi064.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 23:08 /home/ubuntu/cephtest 2024-02-09T23:08:39.803 INFO:teuthology.orchestra.run.smithi089.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 9 23:08 /home/ubuntu/cephtest 2024-02-09T23:08:39.804 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-02-09T23:08:39.817 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-02-09T23:08:39.842 INFO:teuthology.nuke:Checking targets against current locks 2024-02-09T23:08:39.869 DEBUG:teuthology.nuke:shortname: smithi003 2024-02-09T23:08:39.869 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T23:08:39.894 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi003.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 22:18:55.584249', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnknuxCHqeCcyv4s+OGlpe8Tl9GA6RYMCPgoFCu2QyippZdl+Pw2d3Z7h3vDVnCLx16H2LezgTBUCACXEaIHXM='} 2024-02-09T23:08:39.898 DEBUG:teuthology.nuke:shortname: smithi064 2024-02-09T23:08:39.899 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T23:08:39.921 DEBUG:teuthology.nuke:shortname: smithi089 2024-02-09T23:08:39.921 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-02-09T23:08:39.926 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi064.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 22:18:55.583139', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP68kaqh9+DKptpT9uZPbsRZtpU1kknaEztXtPYckqc9jLQfCK/fjAsQNuqBAKrc+7xJ2N2S2q2W9w981cSuuDQ='} 2024-02-09T23:08:39.955 INFO:teuthology.orchestra.console.smithi003:Power off 2024-02-09T23:08:39.955 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T23:08:39.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-02-09_19:35:09-fs-main-distro-default-smithi/7554417', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-02-09 22:18:55.585176', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPVzyAieWIz1DOzU0iw73w3FWDXUNw4nVIkzpIOvG9scdUtcs9DRg0ScH2fxg/OUY1rZsEKXYdtwNIyvi0lgIf4='} 2024-02-09T23:08:39.982 DEBUG:teuthology.orchestra.console.smithi003:power off output: Chassis Power Control: Down/Off 2024-02-09T23:08:39.983 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:40.009 INFO:teuthology.orchestra.console.smithi064:Power off 2024-02-09T23:08:40.009 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T23:08:40.022 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-02-09T23:08:40.036 DEBUG:teuthology.orchestra.console.smithi064:power off output: Chassis Power Control: Down/Off 2024-02-09T23:08:40.037 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:40.056 INFO:teuthology.orchestra.console.smithi089:Power off 2024-02-09T23:08:40.056 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-02-09T23:08:40.068 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-09T23:08:40.081 DEBUG:teuthology.orchestra.console.smithi089:power off output: Chassis Power Control: Down/Off 2024-02-09T23:08:40.081 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:40.105 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-02-09T23:08:44.024 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:44.069 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:44.106 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:44.138 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-02-09T23:08:44.181 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-09T23:08:44.220 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-02-09T23:08:48.142 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:48.182 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:48.221 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:48.256 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is on 2024-02-09T23:08:48.295 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is on 2024-02-09T23:08:48.334 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is on 2024-02-09T23:08:52.260 DEBUG:teuthology.orchestra.console.smithi003:pexpect command: ipmitool -H smithi003.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:52.295 DEBUG:teuthology.orchestra.console.smithi064:pexpect command: ipmitool -H smithi064.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:52.335 DEBUG:teuthology.orchestra.console.smithi089:pexpect command: ipmitool -H smithi089.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-02-09T23:08:52.370 DEBUG:teuthology.orchestra.console.smithi003:check power output: Chassis Power is off 2024-02-09T23:08:52.406 DEBUG:teuthology.orchestra.console.smithi064:check power output: Chassis Power is off 2024-02-09T23:08:52.446 DEBUG:teuthology.orchestra.console.smithi089:check power output: Chassis Power is off 2024-02-09T23:08:52.472 INFO:teuthology.orchestra.console.smithi003:Power off completed 2024-02-09T23:08:52.508 INFO:teuthology.orchestra.console.smithi064:Power off completed 2024-02-09T23:08:52.547 INFO:teuthology.orchestra.console.smithi089:Power off completed 2024-02-09T23:08:52.654 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/crc wsync/yes} objectstore-ec/bluestore-bitmap omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/automatic export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/dbench}} duration: 2460.4869227409363 failure_reason: 'Command failed (workunit test suites/dbench.sh) on smithi003 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=290212a1bfd33563ba76ee5b7f0b944fdb6c5d90 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=52cff9affa5048c38ff62349d63dfe6b status: fail success: false 2024-02-09T23:08:52.654 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-02-09T23:08:52.741 INFO:teuthology.run:FAIL